Wearable health data collection device, safe driving monitoring system and method

By adopting wearable health data acquisition device and deep learning + FPGA hardware acceleration technology in the intelligent safe driving monitoring system, combined with Retinaface model and chaos system design, the low recognition success rate and inappropriate modules in the existing system are solved. Problem, achieving high accuracy and real-time driver status monitoring and reducing the risk of privacy leakage.

CN114758390BActive Publication Date: 2024-11-26WUHAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210349855.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-02
Publication Date
2024-11-26
Estimated Expiration
2042-04-02

AI Technical Summary

Technical Problem

The existing intelligent safe driving monitoring system has problems such as low recognition success rate, inadequate modules, and lack of supporting APPs when identifying driver status. In addition, the early behavior recognition algorithms based on manual calculations and classifiers cannot achieve real-time and Accuracy requirements.

Method used

Using wearable health data acquisition device and deep learning-based FPGA hardware acceleration technology, combined with Retinaface model for face detection and driving behavior analysis, provides two solutions through cloud detection and local hardware acceleration detection, and uses chaotic system design compression and Encryption algorithms reduce data transmission and protect privacy.

Benefits of technology

Improve the accuracy and real-time nature of driving behavior recognition. The detection time of a single frame is only increased by 13%, the accuracy of inattention detection reaches 91%, the accuracy of fatigue driving detection reaches 87%, and the accuracy of gaze direction detection reaches 83%, The detection frame rate reaches 12FPS, meeting real-time requirements and reducing the risk of privacy leakage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114758390B_ABST
    Figure CN114758390B_ABST
Patent Text Reader

Abstract

The present invention discloses a wearable health data collection device, a safe driving monitoring system and method. The system includes an FPGA onboard device, a wearable health data collection device, a client and a server; by collecting driver videos, face detection and driving behavior analysis are performed; the health data of passengers are collected and the safety status of passengers is analyzed; and data is viewed and vehicle information is managed. The present invention designs a compression and encryption algorithm to reduce the amount of data uploaded images while avoiding privacy leakage problems. The present invention designs an efficient CNN image recognition accelerator based on FPGA. The accelerator is suitable for multiple network models. The present invention deeply studies SSD target detection, quantizes weight parameters, reduces storage scale, and transplants the entire model to the Xilinx FPGA ZYNQ board to verify the performance of the designed accelerator. The FPGA hardware accelerator greatly improves the running speed and reduces energy consumption.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and application technology, and relates to a safe driving monitoring system and method with a complete solution, specifically to a wearable health data acquisition device, a safe driving monitoring system and method based on deep learning, covering face detection algorithms and APP development, onboard devices, wearable devices, hardware acceleration algorithms and other technologies. Background Art

[0002] As the most common and main means of transportation, cars have brought great convenience to travel. However, with the continuous growth of car ownership, the number of traffic accidents each year is also gradually rising, and people's requirements for driving safety are getting higher and higher. According to statistics, a large part of traffic accidents are caused by the subjective factors of drivers, such as drunk driving, fatigue driving, sudden illness, etc. Intelligent safe driving monitoring system refers to a device equipped with advanced sensors, controllers, actuators, etc., integrating modern communication and network technologies, realizing intelligent information exchange and sharing between cars and people, and having complex environmental perception, intelligent decision-making, collaborative control and execution functions. When the car is ready to drive or driving, it can obtain the driver's status at any time, collect data, judge driving behavior, and combine the navigation map data to perform systematic calculations and analysis, so as to make the driver aware of possible dangers in advance, effectively increasing the comfort and safety of car driving.

[0003] Facts have proved that mature and reliable safe driving monitoring systems can provide great guarantees for driving safety. In recent years, the market for safe driving assistance systems has grown rapidly. Originally, such systems were limited to the high-end market, but are now entering the mid-range market. At the same time, many low-tech applications are more common in entry-level passenger cars. Improved new sensor technologies are also creating new opportunities and strategies for system deployment. At the same time, safe driving is expanded to health testing, providing more intimate services for car users and improving the user experience. However, due to the fragmentation of products among various car companies and the low degree of software coordination, most of the existing products have problems such as low intelligence, incompatibility between modules, low recognition success rate, and lack of supporting APPs.

[0004] In recent years, with the continuous breakthroughs in neural networks, the field of deep learning has been fully developed and advanced, and breakthroughs have been made in computer vision and natural language processing. It also shows the possibility of applying deep learning to the field of driving behavior recognition. Early safe driving monitoring systems extracted features through manual calculations and corresponding classifiers, such as SVM and random forests, to establish a behavior recognition algorithm based on gesture basic elements, which could not meet the requirements of real-time and accuracy. However, deep learning, due to its ability to spontaneously learn features, can play a huge role in the field of safe driving monitoring through training with large data sets.

[0005] Object detection is one of the important fields of deep learning. Its task is to find all the objects of interest in the image and determine their categories and locations. It is one of the core issues in the field of computer vision. In the field of object detection, it can be divided into face detection and general object detection. There are often special algorithms for face detection (including face detection, face recognition, recognition of other face attributes, etc.), and there are certain differences from general object detection (recognition), mainly due to the particularity of the face (smaller objects, unclear features between faces, occlusion problems, etc.). General object detection algorithms based on deep learning are mainly divided into two categories: Two stage and One stage. The difference lies in whether to generate an item alternative box (regional proposal, RP). Among them, Two stage has higher recognition accuracy, but slower speed and higher performance requirements, and is not suitable for deployment in the field of driving monitoring. One stage algorithm is more suitable for deployment and debugging of onboard devices due to its lower time and space costs, and is easy to install in embedded devices such as vehicles. Face detection usually requires other additional information, such as the location of facial feature points, the degree of opening of eyes and mouth, etc. Through the feature points, the algorithm can calculate the deflection angle of the face, the direction of the eyes' gaze, and through deeper calculations, the direction of attention and other information can be obtained. Through the degree of opening, the number of times the target yawns or closes the eyes can be determined, which can be used to monitor fatigue driving. Therefore, target detection based on deep learning has greater application prospects and research value in safe driving monitoring.

[0006] FPGA is a customizable hardware circuit structure. The parallel computing mode it provides is consistent with the computing characteristics of CNN, and its reprogrammable characteristics are also suitable for the changeable network structure of neural networks. FPGA has three major advantages: greater parallelism: mainly achieved through concurrency and pipelining; customizable: within the scope of resource constraints, users can implement their own logic circuits; reconfigurable: the internal logic of FPGA can be changed according to demand, reducing development costs. Using FPGA to reuse resources can save more space for the server than using multiple fixed ASIC modules. The key point of FPGA for image processing is that it can perform real-time pipeline operations to achieve the highest real-time performance. Therefore, image processing in application fields with high real-time requirements is basically carried out using FPGA. Summary of the invention

[0007] In order to explore new solutions for intelligent safe driving monitoring systems, the present invention proposes a wearable health data acquisition device, a safe driving monitoring system and method based on deep learning algorithm and FPGA hardware acceleration.

[0008] The technical solution adopted by the device of the present invention is: a wearable health data acquisition device, including a control module, a power module, a temperature detection module, a blood oxygen heart rate detection module, a button module, a buzzer alarm module and a liquid crystal display module;

[0009] The control module is connected and communicated with the power module, temperature detection module, blood oxygen heart rate detection module, key module, buzzer alarm module and liquid crystal display module respectively.

[0010] The technical solution adopted by the system of the present invention is: a safe driving monitoring system, including an FPGA onboard device, a wearable health data acquisition device, a client and a server;

[0011] The FPGA onboard device is arranged in the car cockpit, in front of the driver, and is used to collect the driver's video, perform face detection and positioning, determine the driver's position, and provide a basis for driving behavior analysis;

[0012] The wearable health data collection device is worn on the body of the passenger and is used to collect the passenger's health data to provide a basis for safety status analysis;

[0013] The server is used to store data collected by the FPGA onboard device and the wearable health data collection device, analyze the data, and obtain the driver's driving behavior and the safety status of the passengers;

[0014] The server is connected and communicated with the FPGA onboard device, the wearable health data acquisition device and the client through the network, and the user views the data and manages the vehicle information through the client.

[0015] The technical solution adopted by the method of the present invention is: a safe driving monitoring method, including collecting driver video, performing face detection and analyzing driving behavior; collecting passenger health data, analyzing passenger safety status; and viewing data and managing vehicle information;

[0016] The driver video is collected, face detection is performed and driving behavior is analyzed. The specific implementation process is: the collected driver video image is preprocessed, and then input into the face detection model to obtain a parameter vector Tensor with a length of 220, and face detection and driving behavior analysis are performed; wherein Tensor[0:3] is the regression parameter of the prediction box; Tensor[4] is the confidence that there is a face in the prediction box; Tensor[5:217] is the regression parameter of the facial feature point, and each two bits correspond to the horizontal and vertical coordinates of a feature point; Tensor[218:219] is the predicted value of the opening and closing of the mouth and eyes, with 0 representing closed and 1 representing open;

[0017] The face detection model adopts the Retinaface model, outputs a preliminary feature map through a deep separable convolutional neural network, performs multi-scale feature fusion through an image pyramid FPN module, expands the map receptive field through an SSH module, adjusts the number of channels through a 1x1 convolution, and obtains a parameter vector through an activation function;

[0018] The deep separable convolutional neural network adopts the MobileNet network, including a conv-batchnorm module and a conv-depthwise module. After the image is input, the conv-batchnorm module is first used to perform preliminary calculations, and then the conv-depthwise module is used to increase the network depth; the deep separable convolutional neural network reduces the number of conv-depthwise modules when the image size is larger than a preset value, and then uses a conv-depthwise module with a step size of 2 to halve the size; when the size is not larger than the preset value, the conv-depthwise module with a step size of 1 is continuously used to increase the network depth.

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] 1. Applying deep learning algorithms to the scene of driving behavior recognition, the improved Retinaface can output 106 facial feature points and the opening and closing results of eyes and mouth. After pruning and optimization of the network, the single-frame detection time is only 13% longer than the original version. The accuracy of inattention detection reaches 91%, the accuracy of fatigue driving detection reaches 87%, and the accuracy of gaze direction detection reaches 83%. The accuracy of the target detection network in detecting mobile phones reaches 91%, and the accuracy of detecting cigarettes reaches 90%.

[0021] 2. There are two complete solutions, the difference is cloud detection and local hardware accelerated detection. They have different advantages and disadvantages and different applicable scenarios. Cloud detection has lower requirements on hardware device performance, but relies on network signals; local hardware accelerated detection requires the use of FPGA for parallel computing, which is more expensive, less dependent on signals, and more stable. Supporting wearable devices and mobile APPs have been developed to collect health data through wearable devices, and users can view health data reports and manage devices through APPs.

[0022] 3. Compression and encryption algorithms were designed using chaotic systems to compress and encrypt images that need to be uploaded during cloud detection, reducing the amount of data transmission by an average of 82%. At the same time, encryption can avoid privacy leaks that may occur during data transmission.

[0023] 4. Accelerate the calculation of neural networks through FPGA, and accelerate the calculation of the most important convolution, normalization and activation functions in neural networks through loop unrolling optimization and pipeline acceleration. The detection frame rate reaches 12FPS, which meets the real-time requirements, and has lower cost and higher energy efficiency than GPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a structural block diagram of a wearable health data acquisition device according to an embodiment of the present invention;

[0025] Figure 2 is a system structure block diagram of an embodiment of the present invention;

[0026] Figure 3 It is a structural block diagram of the FPGA onboard device according to an embodiment of the present invention;

[0027] Figure 4 This is a structural diagram of a deep separable convolutional neural network according to an embodiment of the present invention;

[0028] Figure 5 This is a flow chart of face detection model training in an embodiment of the present invention;

[0029] Figure 6 It is a working flow diagram of the Mobilenet module, the feature pyramid network FPN module, and the SSH module of an embodiment of the present invention;

[0030] Figure 7 This is a diagram of the SSH module framework structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0032] Please see Figure 1 , a wearable health data acquisition device provided by the present invention includes a control module, a power module, a temperature detection module, a blood oxygen heart rate detection module, a button module, a buzzer alarm module and a liquid crystal display module;

[0033] The control module is connected and communicated with the power module, the temperature detection module, the blood oxygen heart rate detection module, the button module, the buzzer alarm module and the liquid crystal display module respectively.

[0034] The control module of this embodiment adopts the STM32F103C8T6 core board, and the main control is connected to multiple sensors. The whole includes a power module, a temperature detection module, a blood oxygen heart rate detection module, a button module, a buzzer alarm module, an LCD display module, etc. The peripheral devices include a temperature sensor DS18B20 to detect the temperature change of the passenger in real time, and a MAX30100 wearable photoelectric blood oxygen heart rate sensor IC to detect the blood oxygen and heart rate signals of the passenger's pulse in real time. The data is displayed in real time through the LCD1602 display screen. When the safety data of the passenger exceeds the set value, a DC buzzer is used to alarm in real time to remind the passenger that there is a problem with the safety status.

[0035] The power module of this embodiment supplies 3.3V voltage to the STM32 system in real time through the D2 socket external USB power supply, the power circuit is connected to the chip 3V3 pin, and then a main switch is used to decide in real time whether to start supplying real-time power to the STM32 system.

[0036] This embodiment uses the temperature detection chip DS18B20 commonly used in single-chip microcomputers, which can be used to detect the user's temperature value in real time. The hardware part only includes a power line and a data line (single bus), which can realize two-way communication between the single-chip microcomputer and the DS18B20, and the measurement range is -55℃~+125℃; within the range of -10~+85℃, the accuracy is ±0.5℃, and the display accuracy range is 9-12 bits.

[0037] In this embodiment, a 10K pull-up resistor is connected to the single bus to make the single bus at a high level when idle. In order to ensure the measurement accuracy of the passenger temperature detection module, a connector is prepared outside the module to obtain real-time body temperature by contacting the passenger.

[0038] This embodiment uses Max30100 for heart rate and blood oxygen collection. Max30100 is an integrated pulse blood oxygen and heart rate detection sensor, which includes two LED lights, a photodetector for optimizing optics, and a low-noise analog signal processor, and the other is used to detect pulse blood oxygen and heart rate signals. Its advantages are:

[0039] (1) The MAX30100 chip provides a complete pulse oximeter and heart rate sensor solution that simplifies the design. It integrates LEDs, photo sensors, and high-performance analog devices. It measures 2.8 mm x 2.8 mm x 5.6 mm. It is a 14-pin optically enhanced system-level package.

[0040] (2) The ultra-low power operation of the MAX30100 chip increases battery life and provides programmable sampling rates and ultra-low power consumption for wearable devices;

[0041] (3) The max30100 chip has high SNR measurement performance, provides a flexible integration environment, high sampling rate and can also improve data output capabilities.

[0042] The key module of this embodiment has three keys connected to the PB12 pin, PB13 pin and PB14 pin of the chip in sequence. Among them, the PB12 pin is responsible for calling out the relevant parameters in sequence. The first key press displays the upper and lower limits of the temperature module, the second key press displays the upper and lower limits of the heart rate module, and the third key press displays the upper and lower limits of the blood oxygen module. Secondly, the PB13 pin is responsible for setting the lower limit value, which decreases by one each time the key is pressed. Finally, the PB14 pin is responsible for setting the upper limit value, which increases by one each time the key is pressed.

[0043] The buzzer module of this embodiment is used in conjunction with the button module. When the measured value is lower than the lower limit or higher than the upper limit, the buzzer module will give a real-time alarm to remind the passengers that there is a problem with the safety status.

[0044] The display circuit module of this embodiment adopts LCD1602 module, which can display numbers, symbols, letters (ASCII code), but cannot display Chinese characters and images; each line can display 16 characters and two lines; 5X7 matrix, any character can be displayed by controlling any position. Connect the 14 to 4 pins of LCD1602 with a pull-up resistor and connect them with the PA0 to PA7 and PC15 to PC13 ports of the single-chip computer STM32F103, control the read and write enable of the corresponding port, and write the data into LCD1602; the 3 pin of LCD1602 is controlled by an external potentiometer to control the brightness of LCD1602.

[0045] This embodiment uses STM32F103C8T6 as the main control board.

[0046] Please see Figure 2 , a safe driving monitoring system provided by the present invention includes an FPGA onboard device, a wearable health data acquisition device, a client and a server;

[0047] The FPGA onboard device is installed in the cockpit of the car, in front of the driver, and is used to collect driver video, perform face detection and positioning, determine the driver's position, and provide a basis for driving behavior analysis; the wearable health data acquisition device is worn on the body of the passenger and is used to collect the passenger's health data, providing a basis for safety status analysis; the server is used to save the data collected by the FPGA onboard device and the wearable health data acquisition device, analyze the data, and obtain the driver's driving behavior and the safety status of the passenger; the server is connected and communicated with the FPGA onboard device, the wearable health data acquisition device and the client through the network, and the user views the data and manages the vehicle information through the client.

[0048] Please see Figure 3 The FPGA onboard device of this embodiment includes an OV5640 binocular camera, a 480×720 LCD display, an MQ-3 alcohol sensor, a voice module, a communication module, and a storage module. All sensors are controlled by the dual-core Cortex-A9 microprocessor inside the FPGA.

[0049] The microprocessor is connected to the camera, display, alcohol sensor, voice recognition module, communication module, and storage module respectively, and is used to control the camera, display, alcohol sensor, voice recognition module, communication module, and storage module; the client is used for user registration and login functions, and collects face information during registration; is used for users to add and manage vehicle information; is used for users to add and manage associated users; is used for users to view health information reports of themselves or associated users; the server is a cloud server or a local server;

[0050] This embodiment proposes two sets of technical solutions to meet different application requirements. They are divided into local algorithm implementation based on FPGA hardware acceleration and cloud implementation based on network transmission according to the deployment method of convolutional neural network. Among them, the biggest advantage of local implementation is fast speed and low energy efficiency, which is not affected by bandwidth and network speed, and can effectively avoid privacy leakage. However, the disadvantage is that the cost of FPGA hardware is relatively high and it is only applicable to high-end products. On the contrary, the algorithm implementation of the cloud server is low-cost. The on-board module only needs to transmit the image to the cloud, which can maximize the use of computing resources. The disadvantage is that it is greatly affected by bandwidth and is thus subject to the operating environment. In response to the privacy leakage problem that may occur during data transmission, the present invention designs an encryption algorithm based on a chaotic system to achieve the protection of privacy data.

[0051] The main control board of the vehicle-mounted device in this embodiment is ZYNQ FPGA, which contains FPGA part and ARM hard core. The present invention obtains the data of OV5640 camera through the camera image acquisition IP core, stores it in DDR3 and displays it on the LCD screen through the LCD driver IP core. The data in DDR3 is sent to the 4G module through the serial communication module of PS. The 4G module sends the picture to the server, and the algorithm detection is performed on the server and returned through the serial port. The monitoring module monitors the returned data and alcohol data at the same time, and once an abnormality is found, an alarm is immediately issued through the voice module.

[0052] After the camera of this embodiment collects data, the data timing (row field synchronization timing) of the camera is converted into AXI4-Stream timing through the video in to AXI4 Stream IP core, converted into AXIFULL timing through VDMA, and connected to the HP interface to write DDR3. VDMA also reads the data of DDR3 through AXI-FULL timing and converts it into AXI-Stream timing. The Stream timing output by VDMA is converted into LCD timing through the AXI4-Stream to vedio out IP core, and then drives the LCD. The data after video collection is in DDR3 and can also be sent through the 4G module on the PS side.

[0053] The communication module of this embodiment uses the ATK-GM510_V1.4 (V1.4 is the version number, and the product is identified as ATK-GM510 below). The ATK-GM510 module is equipped with the industrial-grade four-mode LTE Cat.4 module of Gosuncn (GOSUNCN): GM510_C2E_L, working frequency band 13 LTE FDD: B1 / B3 / B5 / B8, LTE TDD: B34 / B38 / B39 / B40 / B41, WCDMA: B1 / B8, GSM: B3 / B8. It can realize voice, SMS, network communication, base station positioning, network card, FTP, HTTP and other functions. The ATK-GM510 module supports RS232 serial port and TTL serial port, with hardware flow control, and supports an ultra-wide working range of 5V to 16V, making the module very convenient to connect to the product.

[0054] The communication module of this embodiment can send text messages, make and receive calls, send HTTP requests, etc. through serial port control. It uses AT command communication and can complete the specified operation by sending AT commands through the serial port. The AT command used in the present invention is ATD, which is used to notify or alarm when dangerous driving behavior is detected.

[0055] This embodiment uses the LU-ASR01 voice module, which is a programmable voice module controlled by a single-chip microcomputer and supports personalized configuration. It contains 8 IOs including uatr and PWM. Multiple wake-up words and command words can be customized. The present invention realizes the following functions:

[0056] (1) When the user is in danger, he shouts out a call for help, and the voice module will automatically make a call through the main control board and send the current coordinates via SMS;

[0057] (2) Connected to the alcohol module, when the alcohol level exceeds the limit, it will automatically alarm and make a phone call;

[0058] (3) When the server returns data indicating that the user is driving fatigued, an alarm will be issued;

[0059] (4) When the wearable device detects abnormal heart rate, blood oxygen, or body temperature, an alarm will be sounded.

[0060] Drunk driving is one of the most important and serious behaviors of dangerous driving. This embodiment uses an alcohol sensor module MQ-3 to detect alcohol concentration and is connected to the main control board. When the alcohol concentration exceeds the standard, a reminder is issued through a voice module and a data transmission module.

[0061] MQ-3 can respond to the alcohol concentration in the air in real time and has a power indicator light and a TTL signal output indicator light. And there are dual-channel signal outputs, namely AO analog output and DO switch quantity TTL level output, which well supports work in different environments. In order to better adapt to driving scenarios, the present invention adjusts the potentiometer of the alcohol sensor. According to the national "Blood and Breath Alcohol Content Thresholds and Inspections for Vehicle Drivers", a driver with an alcohol content of 20-80 mg in 100 ml of blood is considered to be driving under the influence of alcohol, and more than 80 mg is considered to be drunk driving. The present invention has experimentally tested that the most reasonable potentiometer resistance is 17.2Ω, which can be adjusted by rotating the potentiometer. Through testing, the alcohol module successfully identified drunk driving and optimal behavior.

[0062] The communication module of this embodiment uses the ATK-S1216F8-BD-V23 module. This module is a high-performance GPS / Beidou module produced by ALIENTEK. The core uses the S1216F8-BD module of SkyTraq, which has 167 channels, a tracking sensitivity of up to -165dBm, and a measurement output frequency of up to 20Hz. It has the following features:

[0063] (1) The module adopts S1216F8-BD module, which is small in size and has excellent performance.

[0064] (2) The module can set various parameters through the serial port and save them in the internal FLASH, which is easy to use.

[0065] (3) The module has its own IPX interface, which can be connected to various active antennas and has strong adaptability.

[0066] (4) The module is compatible with 3.3V / 5V levels, making it easy to connect to various microcontroller systems.

[0067] (5) The module comes with a rechargeable backup battery that can retain ephemeris data in the event of a power failure.

[0068] This module can obtain a variety of data including accuracy, latitude, altitude, speed, number of satellites used for positioning, number of visible satellites, UTC time and other information. The module can automatically connect to the satellite and communicate with the main control board through the serial port. You only need to parse the data according to the format.

[0069] The FPGA (Field-Programmable Gate Array) of this embodiment is widely used because of its high-speed computing capability, and its parallel computing speed is several orders of magnitude higher than that of traditional serial computing. The heterogeneous ZYNQ platform device launched by Xilinx and the corresponding DNNDK deep learning acceleration library are its important products for edge computing. A computing power of more than 1.4TOPS can be obtained on the ZCU104 development platform.

[0070] The different architectures of the ZYNQ-SOC system-on-chip consist of two components: PS (ARM-based processing system) and PL (FPGA-based programmable logic). The separable design allows the PS and PL parts to run independently of each other, which has great flexibility for design and development. The present invention selects the ZYNQ-based navigator FPGA development board. The present invention selects a development board with rich peripheral resources during the research stage. When it is put into use later, some functions can be castrated, and the cost will be further reduced.

[0071] The client of this embodiment is used in conjunction with the vehicle-mounted device and the wearable device, and is convenient for users to view health data, manage vehicle information, etc. Its basic functions are as follows:

[0072] (1) It has registration and login functions and collects facial information during registration;

[0073] (2) Users add and manage vehicle information;

[0074] (3) User addition and management of associated users;

[0075] (4) Users view their own or associated users’ health information reports.

[0076] The design of the client main interface of this embodiment is roughly divided into three parts. The first part is the blue horizontal bar at the top. The top is fixed at the top using CSS position: fixed, and will not change position as the page scrolls; the second part is the main body in the middle. In the middle is a slot slot, which is used to render the line chart component. The line chart is drawn with Echarts; the third part is the navigation bar at the bottom. The icons in the navigation bar use v-model two-way binding data and will turn blue when selected.

[0077] Regarding the Echarts component, in order to avoid excessive pressure on the server due to user misoperation, the present invention adopts a throttling operation, and only sends a request to the server once within 15 seconds.

[0078] The client user interface design of this embodiment is consistent with the main interface, and is generally divided into three parts: user information at the top, operation options in the middle, and a navigation bar at the bottom. The entire layout uses a flex layout, which can adapt to mobile phones of any size.

[0079] In the client driver list interface design of this embodiment, click Add Driver to jump to the interface for adding drivers. After adding, route will jump to the original page and refresh the data. Click the driver information to jump to the editing interface.

[0080] In the design of the client vehicle details interface of this embodiment, the car icons on this page are v-bind bound to different data, and different vehicle information will be displayed according to the different icons you click. These data are requested from the server through axios during the create period.

[0081] Please see Figure 4-Figure 7 , a safe driving monitoring method provided by the present invention includes collecting driver video, performing face detection and analyzing driving behavior; collecting passenger health data, analyzing passenger safety status; and viewing data and managing vehicle information;

[0082] This embodiment collects driver video, performs face detection and analyzes driving behavior. The specific implementation process is: pre-process the collected driver video image, and then input it into the face detection model to obtain a parameter vector Tensor with a length of 220, perform face detection and analyze driving behavior; wherein Tensor[0:3] is the regression parameter of the prediction box; Tensor[4] is the confidence that there is a face in the prediction box; Tensor[5:217] is the regression parameter of the facial feature point, and each two bits correspond to the horizontal and vertical coordinates of a feature point; Tensor[218:219] is the predicted value of the opening and closing of the mouth and eyes, with 0 representing closed and 1 representing open;

[0083] The face detection model of this embodiment adopts the Retinaface model, outputs a preliminary feature map through a deep separable convolutional neural network, performs multi-scale feature fusion through an image pyramid FPN module, expands the map receptive field through an SSH module, adjusts the number of channels through a 1x1 convolution, and obtains a parameter vector through an activation function;

[0084] Please see Figure 4The deep separable convolutional neural network of this embodiment adopts a MobileNet network, including a conv-batchnorm module and a conv-depthwise module. After the image is input, the conv-batchnorm module is first used for preliminary calculation, and then the conv-depthwise module is used to increase the network depth; the deep separable convolutional neural network reduces the number of conv-depthwise modules when the image size is larger than a preset value, and then uses a conv-depthwise module with a step size of 2 to halve the size; when the size is not larger than the preset value, the conv-depthwise module with a step size of 1 is continuously used to increase the network depth.

[0085] At present, Retinaface has opened two backbone networks, namely MobilenetV1-0.25 and Resnet. Among them, Resnet has better effect, but has higher performance requirements, and the network is complex and not easy to transplant between different environments. Therefore, the present invention uses mobilenet V1-0.25 as the backbone network. The core idea used is depthwise separable convolution, that is, depthwise separable convolution. First, set conv2d (group = True) to perform group convolution to obtain the feature map of the input dimension. Before the fusion operation, a 1*1 convolution layer is used to adjust the number of output channels to reduce the number of operation parameters. In the structure of MobileNet, Conv dw is a group convolution, and a 1x1 convolution will follow after the group convolution for channel processing. The mobilenetV1-0.25 used in this article is a network in which the number of channels of mobilenetV1-1 is compressed to 1 / 4 of the original, and pruned and optimized.

[0086] This embodiment collects driver videos and performs face detection. In the FPGA calculation process, the convolution loop is expanded in two dimensions, namely, the number of input feature maps and the number of output feature maps. The PIPELINE method is adopted to expand and accelerate the input feature channel. The input feature channel acceleration is the same as the convolution module acceleration.

[0087] According to the number of input feature maps, a data is read from the same position on N input feature maps in parallel in each clock cycle, and the product is calculated by the corresponding weight on the convolution kernel. The product results are added through an addition tree with a depth of log2(N) to obtain the partial sum of this convolution and store it. Repeat the above calculation process until all the partial sums required for the convolution are calculated, and then add the partial sums obtained each time through an accumulator to obtain the result of this convolution calculation.

[0088] This expansion method is to simultaneously calculate on each channel of the input feature map and expand the convolution in the dimension of the number of input images. In a general-purpose processor, only one multiplication or addition operation can be performed in one cycle. Taking N channel convolution as an example, the value of the current position of the output feature map requires N multiplications and N-1 additions, that is, 2N-1 cycles. In FPGA, the UNROLL instruction can be used to expand the for loop, improve the parallelism of the program when executing loop iterations, and use the method of exchanging space for time. Each clock cycle reads a data from the same position on the N input feature maps in parallel and calculates the product with the corresponding weight on the convolution kernel. Multiple multiplication operations can be completed, and the product results are added through an addition tree with a depth of log2(N) to obtain the partial sum of this convolution and store it. Repeat the above calculation process until all the partial sums required for the convolution are calculated, and then the partial sums obtained each time are accumulated through an accumulator to obtain the result of this convolution calculation.

[0089] According to the number of output feature maps, a certain data on an input feature map is multiplied by the weights of the corresponding positions of M different convolution kernels in parallel in each clock cycle; the above calculation process is repeated until the products of the corresponding areas of all input feature maps and the corresponding areas of M convolution kernels required for convolution are calculated.

[0090] This expansion method is to calculate the values ​​of multiple output feature maps at the same time, and expand the convolution in the dimension of the number of output images. Just like the expansion of the number of input feature maps. In a general-purpose processor, only one multiplication or addition operation can be performed in one cycle. Taking M convolution kernels as an example, to output M feature maps, it is necessary to calculate the input feature map and the convolution kernel M times, that is, M*One_convolution_period cycles. In FPGA, the UNROLL instruction is used to exchange space for time. Improve the parallelism of the program when executing loop iterations. In each clock cycle, a certain data on an input feature map is multiplied by the weights of the corresponding positions of M different convolution kernels in parallel. Repeat the above calculation process until the product of the corresponding areas of all input feature maps and the corresponding areas of M convolution kernels required for convolution is calculated. This expansion method can obtain the data of the corresponding positions of M output feature maps at one time.

[0091] Therefore, the two expansion methods are combined, that is, data is read from the same position on the N input feature maps in each cycle, and the data read from each input feature map is multiplied by the weights of the corresponding positions of M different convolution kernels.

[0092] This embodiment adopts the PIPELINE method to expand the input feature channel and accelerate the pipeline. The input feature channel acceleration is the same as the convolution module acceleration; the PIPELINE method is adopted. Shorten the overall time. Assuming that each step of the operation requires 1 instruction cycle, the delay of data movement is 3 instruction cycles, and the loop delay requires 6 instruction cycles. After adopting the PIPELINE method, the delay of data movement is 3 instruction cycles, and the loop delay is 4 instruction cycles. When the number of cycles is too many, it can save computing time to a great extent.

[0093] Please see Figure 5 The face detection model used in this embodiment is a trained Retinaface model;

[0094] The training process specifically includes the following steps:

[0095] Step 1: Build a dataset;

[0096] The LaPa dataset with 106 points and a single portrait was used. The face detection model was used to predict and annotate the real boxes of the easily classified data. The minimum confidence level was determined to be 0.955. The data with a confidence level lower than 0.955 were manually annotated and batch processed using a Python program. A self-made set of driving scene photos was added. Opencv was used to convert, scale, and mirror RGB and grayscale images to ensure the applicability of the dataset.

[0097] The data set data contains a list of three elements: train_set (training set), val_set (validation set), and test_set (test set), including 18170 training samples, 2000 validation samples, and 2000 test samples. Each sample contains a portrait image and corresponding annotation information. train_set (training set): used to determine model parameters; val_set (validation set): used to adjust model hyperparameters (such as multiple network structures, the optimal choice of regularization weights); test_set (test set): used to estimate the application effect. In order to avoid the model memory affecting the training effect, the samples are first shuffled. According to the hardware performance, a reasonable batch_size is set, and then the data is converted into np.array format that meets the model input requirements and returned. At the same time, when returning data, the Python generator is set to yield mode to reduce memory usage, and it is encapsulated as a function waiting for calling. Asynchronous data reading is performed through the pytorch framework, and data reading and model training are parallelized to speed up data reading.

[0098] Step 2: Input the data in the training set train_set into the deep separable convolutional neural network in random order to train the face detection model. The ultimate goal is to minimize the function value of the multi-task loss function.

[0099] Please see Figure 5 and Figure 6 The training process of this embodiment is as follows: for the input image data, a list of prior frames on the image is obtained according to the image size, and then the image data is passed through the Mobilenet module, the feature pyramid network FPN module, and the SSH module to obtain a feature map, from which the predicted value of each prior frame is extracted, that is, the position adjustment parameter of each prior frame, the probability of containing a face, the position parameters of the facial feature points, and the prediction results of the opening and closing of the eyes and mouth, and then the loss value loss between these parameters and the true value is calculated for training;

[0100] The multi-task loss function is:

[0101] L=L cls (pi,pi * )+μ1pi * L box (ti,ti * )+μ2pi * L pts (li,li * )+μ3pi * L eop (ei,ei * )+μ4pi * L mop (mi,mi * )+μ3pi * l pixel ;

[0102] Among them, L cls (pi,pi * ) is the face classification loss, which means the loss value calculation for judging whether it is a face. This is a loss function for a classification problem. pi represents the probability of whether the prior box i in the prediction result contains a face. * Represents the real situation. If it is a face, the anchor is 1, otherwise the anchor is 0;

[0103] L box (ti,ti * ) is the face frame regression loss, where ti=(t x , t y , t w , t n ) i ,ti * =(ti *x ,ti * y ,ti * w ,ti * n ) i is the coordinate information of the predicted frame and the coordinate information of the real face frame; where t x , t y is the horizontal and vertical coordinates of the center point of the box, t w , t n is the width and height of the frame; the prediction of the face frame is a regression problem. First, the real frame parameters in the label need to be normalized, and the loss is calculated through the robust regression function smooth_l1_loss;

[0104] pi * L pts (li,li * ) is the key point regression loss, li=(l x1 , l y1 , l x2 , l y2 , ..., l x106 , l y106 );l xi , l yi Represents the corresponding key point coordinates, and the two represent the predicted value and the true value respectively. This loss function is also a regression problem, which requires target normalization and loss calculation through the smooth_l1_loss function;

[0105] L eop (ei,ei * )、L mop (mi,mi * ) is the classification loss of eyes and mouth opening and closing, which means the loss value calculation of judging the opening and closing state of eyes and mouth, which is a classification problem; ei and mi are the predicted probabilities of whether the eyes and mouth are open or not, ei * and * is the actual opening and closing status of the faces in the training set, 1 if open, 0 otherwise;

[0106] This embodiment uses frozen training during the training process. First, the backbone network is frozen for fine-tuning, and then unfreeze training is performed. If the network is trained from 0, the efficiency is low because of too much randomness. Therefore, the present invention chooses to use frozen training. Through the processing of the above steps, the pre-trained weights of the open source model are fully utilized. The feature extraction network will change during unfreeze training, the video memory occupied is large, and all parameters of the network will change.

[0107] This embodiment uses a feature pyramid network FPN module to upsample and superimpose feature maps of different sizes to obtain a new feature map, thereby achieving multi-scale feature fusion;

[0108] The feature pyramid network FPN module can handle the multi-scale change problem in object detection with minimal increase in computational effort, effectively solving such problems that may be caused by different vehicle models and different equipment installation locations in the driving scenarios targeted by the present invention.

[0109] The feature pyramid network FPN module outputs the following three graphs after processing:

[0110] predict1=1x1conv(stage3)

[0111] predict2=1x1conv(stage2)+upsample(1x1conv(predict1))

[0112] predict3=1x1conv(stage1)+upsample(1x1conv(predict2))

[0113] Among them, stage1, stage2, and stage3 represent the graphs output by the three stages of the MobileNet network, 1x1conv represents the convolution function with a convolution kernel size of 1, and upsample represents the upsampling function.

[0114] In the Wilderface dataset, -1 and 1 represent whether the portrait has feature point annotations. The negative part needs to be considered when selecting the activation function. The use of ReLU as the activation function in the open source version Retinaface will cause the gradient of the negative part to be lost. The ratio of positive and negative samples in the dataset is about 5:1. LeakyReLu is selected to increase the gradient sharing of positive samples. After repeated tests, negative_slope=0.35 is selected for the best effect. Taking the training process as an example, the last three effective feature layers of shape obtained by MobilenetV1-0.25 are (80,80,64), (40,40,128) and (20,20,256). Conv2D(kernel_size=1)+BatchNormalization+LeakyReLU are combined into a sequential module, and the number of channels of the three effective feature layers is adjusted. After adjustment, torch.nn.functional.interpolate is used for Upsample, and it is added to the previous layer of the map to achieve feature fusion. Finally, the Conv2D (kernel_size = 3) + BatchNormalization + LeakyReLU module is used to extract features from the processed graph, and the shapes of the three-layer outputs are (80, 80, 20), (40, 40, 20), and (20, 20, 20).

[0115] Please see Figure 7 This embodiment adopts the SSH module, through three parallel structures, using the stacking of 3x3 convolutions to replace the effects of 5x5 and 7x7 convolutions, using two 3x3 convolutions to replace 5x5 convolutions, and using three 3x3 convolutions to replace 7x7 convolutions, reducing the number of parameters. It increases the receptive field of the feature map and compensates for the loss of low semantic features caused by network pruning.

[0116] Each intermediate layer 3x3 convolution module contains three functions: Conv2D+BatchNormalization+LeakyReLU, and the last layer of convolution does not contain an activation function.

[0117] The prediction results of Retinaface include classification prediction results, regression prediction results of the frame and regression prediction results of face key points. The classification prediction results are used to determine whether the prior frame contains an object, and the open source version of Retinaface uses softmax for judgment. In the present invention, a 1x1 convolution is used to adjust the number of channels of the atlas to num_anchorsx 2, which is used to represent the probability that each prior frame contains a face, which is convenient for adjusting positive and negative samples during training, reducing the proportion of negative samples, and accelerating model convergence; the regression prediction results of the frame are used to adjust the prior frame to obtain the prediction frame, and a 1x1 convolution is used to adjust the number of channels of the atlas to num_anchors x 4, which is used to represent the adjustment parameters of each prior frame; the regression prediction results of the face key points are used to adjust the prior frame to obtain the face key points.

[0118] By extracting the three-layer features obtained by the backbone network, adjusting the number of channels, and connecting the results, the prediction results of the network can be obtained.

[0119] The purpose of the embodiment is to identify possible dangerous driving behaviors, among which the identification conditions for fatigue driving are frequent closing of eyes and yawning. The present invention provides two judgment methods. The first is to increase the original network dimension of Retinaface, increase the output parameters, and calculate the judgment result as a score, which has a high accuracy rate, but will increase the calculation time and have high requirements for onboard equipment; the second is to align the face of the portrait through the positions of 106 preset points, eliminate the angle effect, calculate the opening and closing ratio through the Euclidean distance between the feature points near the eyes and mouth, compare it with the preset threshold, and output the judgment result, which has the advantages of fast operation speed and strong robustness.

[0120] Based on the above goals, the present invention has made substantial modifications to the output results of Retinaface. First, the number of facial feature points is increased to 106, and the corresponding parameters and the number of channels are adjusted to 212. In order to increase the loss value, the encoding method of the face prediction result is modified, and the adjustment parameters of the feature point positions directly calculated by the open source version are converted into offset parameters at the positions of the original five feature points. The present invention divides the 106 feature points on the face into five regions, corresponding to the five original points of the eyes, nose, and mouth corners. The feature point position parameters in each region are obtained by offsetting the corresponding regional center point through cascade regression, making full use of the trained model in the open source code, avoiding the randomness of the initial population training, and greatly accelerating the model convergence speed.

[0121] During the training process, this embodiment adopts a priori frame mechanism, presets some anchor points in the picture, and sets two priori frames of fixed sizes with each anchor point as the center.

[0122] The prior frame (Anchor) mechanism, by determining the width and height of the target object in advance, selecting a preset frame of appropriate size, and applying it to each anchor point, can convert the network prediction result from directly pointing to the real frame to pointing to the adjustment parameters of the prior frame to the real frame, reduce the randomness of the operation, and accelerate the convergence of the model. The target object of the present invention is a portrait. According to investigation, the average aspect ratio of the face is 1.43:1. At the same time, the deflection angle of the vehicle during normal driving is within plus or minus 12 degrees, and the average distance from the steering wheel to the driver is 67cm. The final prior frame size is determined to be (287, 210) and (235, 171).

[0123] The backbone network of this embodiment outputs the adjustment parameters for each prior frame, including the adjustment ratio of the center point, width, and height of the prior frame, the adjustment ratio of the horizontal and vertical coordinates of the feature points, and the results of the mouth and eyes opening and closing. Retinaface does not require the input size to be specified, and all parameters have been normalized. Therefore, in the prediction process, the parameters need to be decoded to obtain the final prediction frame parameters.

[0124] After obtaining the final parameters, training and prediction can be performed. When predicting, the parameters can be directly output. The training process needs to process the labels corresponding to the images and then calculate the loss values.

[0125] The driver video collected in this embodiment is transmitted to the server after the image data is encrypted and compressed; the compression processing is performed by graying the image data and performing discrete cosine transform DCT, and the server uses the inverse DCT algorithm to decompress the image; the specific decompression process includes the following steps:

[0126] (1) Generate two Logistic chaotic sequences {H1} and {H2} of length L. The initial value x0 and parameter μ of the chaotic sequence must be consistent with those during encryption;

[0127] (2) Arrange {H1} in descending order and obtain the index sequence h1 of the new position of each element in the sequence;

[0128] (3) Convert the {H2} sequence to a -1, 1 sequence;

[0129] (4) Symbol restoration: perform point multiplication of the sequence A to be decrypted and the sequence {H2} obtained in step (3) to obtain the sequence A1;

[0130] (5) Coordinate restoration: Using the h1 sequence obtained in step (2) as the permutation rule, perform reverse permutation of the element coordinates of the A1 sequence obtained in step (4) to obtain a new one-dimensional sequence, and assign its value to the sequence A;

[0131] (6) looping through steps (4) and (5) count times, with the value of count remaining the same as that used during encryption. At the same time, the value of count is used as one of the keys of the encryption algorithm;

[0132] (7) According to the compression matrix used in encryption, the decrypted one-dimensional sequence is placed in the corresponding position of the two-dimensional matrix, and the rest is filled with zeros to obtain the two-dimensional matrix G. ;

[0133] (8) Generate an 8th-order DCT matrix T=dctmtx(t) using Matlab built-in functions; dctmtx(t) is a discrete cosine transform function;

[0134] (9) Performing an inverse DCT transform on the two-dimensional matrix G to obtain a two-dimensional matrix I, where I is the result of zero padding of the original image during encryption;

[0135] f = @(blockstruct)T′*blockstruct.date*T;

[0136] I=blockproc(G,[t,t],f);

[0137] @(block_struct) is a decorator function, block_struct.data represents the incoming data, that is, a picture or a matrix, blockproc is the function block processing matrix in MATLAB, and t in the formula is the size of the processing matrix; here t uses the same value 8 as in encryption;

[0138] (10) After removing the 0 data value filled in during encryption to enhance the universality of the algorithm, the decrypted image of the one-dimensional encrypted sequence is obtained.

[0139] The following further describes the FPGA hardware acceleration technology solution of this embodiment:

[0140] The target detection applied to the deep convolutional neural network by the present invention has excellent performance. However, the convolutional neural network is a computationally intensive and efficient algorithm. Its parallel computing characteristics are inconsistent with the serial computing characteristics of the general-purpose CPU. It is bound to have efficiency problems when implemented, and other methods are needed to accelerate the neural network. FPGA is a programmable chip whose computing mode is parallel computing. Compared with the general-purpose processor of serial computing, FPGA is more suitable for implementing neural network algorithms that include a large number of parallel computing. As a programmable chip, FPGA can also update the design in time after the convolutional neural network algorithm is improved or changed, which is more in line with the needs of the neural network field. At the same time, FPGA chips have high performance and low power consumption, which is much lower than the power consumption of GPU, and are suitable for application in the vehicle-mounted device of the present invention. At the same time, deploying the deep learning algorithm in the vehicle-mounted device can reduce the amount of data exchange between the device and the server, reduce the dependence on the network, and fundamentally avoid the privacy issues that may arise during data transmission.

[0141] The present invention uses a high-level synthesis tool (HLS) for design. HLS supports the use of multiple high-level languages ​​for FPGA design, such as C, C++, SystemC, etc. In this link, the present invention uses a program written in C++ language to generate register transfer level (RTL) netlist code and Xilinx software IP core through HLS simulation, optimization and synthesis steps.

[0142] The design process can be divided into three parts: design input, simulation synthesis and design output.

[0143] Design input includes source code, test bench, constraint files (Constraints) and optimization directive files (Directive). Source code is the module (i.e. function) we need to accelerate, and test bench is written to test the module we wrote to ensure its correctness. The constraint file is generated by the HLS tool when creating the project, and it is a constraint on the device and clock frequency used in the project; the optimization directive file is generated by the HLS tool after the C++ simulation passes, and it needs to be used in the synthesis process.

[0144] Simulation synthesis consists of two parts. The first part is C++ simulation and C++ / RTL joint simulation, and the second part is C++ synthesis. Simulation is similar to Verilog simulation. You only need to debug the code according to the prompt information until there is no error. C++ synthesis will combine the optimization instruction file to synthesize the source code and generate a synthesis report, and then compare the synthesis results with the requirements. If the requirements do not meet the requirements, adjust the optimization instructions or directly adjust the C++ code until the results meet the requirements.

[0145] After the simulation passes and the comprehensive report meets the requirements, the design can be output to the FPGA as needed.

[0146] The specific design process is:

[0147] Step 1: Create an HLS project;

[0148] Determine the clock cycle and timing margin, use high-level programming languages ​​such as C++ and HLS tools to write programs, design functional modules and test modules. When designing the hardware accelerator of the network, we analyzed that the modules that need to be designed and tested are: input and output modules, convolution modules and pooling modules.

[0149] Step 2: Run C++ simulation and code debugging;

[0150] If the code is correct, use the test module to simulate the function. If the C++ simulation result is inconsistent with the expected result, use the debugger integrated in the HLS tool to debug each functional module or test module, find the problem and debug the code.

[0151] Step 3: Comprehensive design;

[0152] After C++ simulation and debugging, and after ensuring that the functional module and test module codes are correct, they can be synthesized to generate RTL files.

[0153] Step 4: Optimize the design;

[0154] Adjust the design and re-synthesize it based on performance, area, resource utilization, etc. to get a successful hardware acceleration.

[0155] Step 5: Run C++ / RTL co-simulation;

[0156] RTL test code can be automatically converted from C++ test code. If C++ / RTL co-simulation fails, you need to modify the design according to the error message, and re-synthesize and simulate.

[0157] Step 6: Export RTL implementation;

[0158] HLS tools can export designs as IP cores or RTL codes. The exported IP cores include drivers, reference documents, usage examples and other related files.

[0159] Step 7: System integration;

[0160] The generated IP core can be integrated into the system in other development environments, such as Xilinx Vivado.

[0161] One of the main disadvantages of using convolutional neural networks in cloud servers is privacy leakage. This invention successfully improves and solves this disadvantage from the system. Image data needs to be encrypted and compressed before transmission. The data collected by the camera is in RGB888 format. Since the neural network calculation only requires grayscale images, the present invention converts color data into grayscale images through an IP core, reducing the amount of data sent to 1 / 3.

[0162] Since image data has the characteristics of large data volume and high redundancy, designing encryption algorithms using chaotic systems has become a research hotspot in recent years. Chaos is a behavior controlled by nonlinear dynamics, with high sensitivity to initial values ​​and system parameters, white noise, ergodicity and unpredictability. These characteristics of chaos meet the requirements of cryptography. The present invention adopts the encryption idea of ​​combining pixel position scrambling transformation and pixel value substitution transformation, and designs a digital image encryption algorithm based on the insect population model-Logistic chaotic mapping chaos. The inverse affine transformation on the integer domain is introduced, and the algorithm adopts a method combined with a two-dimensional logistic chaotic map to generate multiple groups of chaotic sequences. The pixel scrambling transformation and gray value replacement are controlled by these chaotic sequences.

[0163] Discrete cosine transform (DCT) is a mathematical operation closely related to Fourier transform. In the Fourier series expansion, if the expanded function is an even function, then its Fourier series only contains cosine terms, and then discretization can derive cosine transform, so it is called discrete cosine transform. DCT is an orthogonal transform, which can convert the 8*8 image space expression into the frequency domain. Since the energy M is mainly concentrated in the DC and a few low-frequency transform coefficients, only a small number of data points are needed to represent the image. Therefore, the efficiency of encoding the frequency domain transform coefficients is much higher than directly encoding the spatial domain pixels, thereby achieving the purpose of image compression. The coefficients generated by DCT are easily quantized, so good block compression can be obtained; in addition, DCT has a fast algorithm, such as the use of fast Fourier transform, which can perform efficient operations, so it is easy to implement in both hardware and software; and the DCT algorithm is symmetrical, and the present invention successfully realizes the decompression of images using the inverse DCT algorithm.

[0164] The present invention uses the characteristics of CNN image recognition, deeply analyzes the characteristics of each network layer, and designs an efficient CNN image recognition accelerator based on FPGA. The accelerator is applicable to multiple network models. The present invention deeply studies yolo target detection, quantizes weight parameters, reduces storage scale, and transplants the entire model to the Xilinx FPGA ZYNQ board to verify the performance of the designed accelerator. FPGA hardware accelerator greatly improves the operating speed and reduces energy consumption.

[0165] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A safe driving monitoring method, characterized in that: This includes collecting driver videos, performing face detection and analyzing driving behavior; collecting passengers’ health data and analyzing their safety status; And view data and manage vehicle information; The driver video is collected, face detection is performed, and driving behavior is analyzed. The specific implementation process is: the collected driver video image is preprocessed, and then input into the face detection model to obtain a parameter vector Tensor with a length of 220, and face detection and driving behavior analysis are performed; wherein Tensor[0:3] is the regression parameter of the prediction box; Tensor[4] is the confidence that the prediction box contains a face; Tensor[5:217] is the regression parameter of the facial feature points, where every two digits correspond to the horizontal and vertical coordinates of a feature point; Tensor[218:219] is the predicted value of the mouth and eyes opening and closing, with 0 representing closed and 1 representing open; The face detection model adopts the Retinaface model, outputs a preliminary feature map through a deep separable convolutional neural network, performs multi-scale feature fusion through an image pyramid FPN module, expands the map receptive field through an SSH module, adjusts the number of channels through a 1x1 convolution, and obtains a parameter vector through an activation function; The deep separable convolutional neural network adopts a MobileNet network, including a conv-batchnorm module and a conv-depthwise module. After the image is input, the conv-batchnorm module is first used to perform preliminary calculations, and then the conv-depthwise module is used to increase the network depth; the deep separable convolutional neural network reduces the number of conv-depthwise modules when the image size is larger than a preset value, and then uses a conv-depthwise module with a step size of 2 to halve the size; When the size is not greater than the preset value, the conv-depthwise module with a step size of 1 is continuously used to increase the network depth; The face detection model is a trained Retinaface model; The training process specifically includes the following steps: Step 1: Build a dataset; Using the LaPa dataset with 106 points and a single portrait, we predicted and annotated the ground truth boxes for the easily classified data, determined the minimum confidence level to be 0.955, and manually annotated the data with a confidence level lower than 0.

955. We also added a self-made set of driving scene photos, and performed conversion, scaling, and mirroring operations on RGB and grayscale images to ensure the applicability of the dataset. The dataset is divided into a training set train_set, a validation set val_set, and a test_set test set. Each sample in the dataset contains a portrait image and corresponding annotation information. Step 2: Input the data in the training set train_set into the deep separable convolutional neural network in random order to train the face detection model, with the ultimate goal of minimizing the function value of the multi-task loss function; The training process is as follows: for the input image data, a list of prior frames on the image is obtained according to the image size, and then the image data is passed through the Mobilenet module, the feature pyramid network FPN module, and the SSH module to obtain a feature map, from which the predicted value of each prior frame is extracted, that is, the position adjustment parameters of each prior frame, including the probability of a face, the position parameters of facial feature points, and the prediction results of the opening and closing of eyes and mouth, and then the loss value loss between these parameters and the true value is calculated for training; The multi-task loss function is: L=L cls (more, more * )+μ1pi * THE box (you, you * )+μ2pi * THE pts (there,there * )+μ3pi * THE eop (hey, hey * )+μ4pi * THE mop (me,me * )+μ3pi * the pixel ; Among them, L cls (pi,pi * ) is the face classification loss, indicating the loss value calculation for judging whether it is a face. pi represents the probability of whether the prior box i in the prediction result contains a face. * Represents the real situation. If it is a face, the anchor is 1, otherwise the anchor is 0; L box (ti,ti * ) is the face frame regression loss, where ti=(t x , t y , t w , t n ) i ,ti * =(ti * x ,ti * y ,ti * w ,ti * n ) i is the coordinate information of the predicted frame and the coordinate information of the real face frame; where t x , t y is the horizontal and vertical coordinates of the center point of the box, t w , t n is the width and height of the frame; the prediction of the face frame is a regression problem. First, the real frame parameters in the label need to be normalized, and the loss is calculated through the robust regression function smooth_l1_loss; pi * L pts (li,li * ) is the key point regression loss, li=(l x1 ,l y1 ,l x2 ,l y2 ,…,l x106 , l y106 );l xi , l yi Represents the corresponding key point coordinates, and the two represent the predicted value and the true value respectively. This loss function is also a regression problem, which requires target normalization and loss calculation through the smooth_l1_loss function; L eop (ei,ei * )、L mop (mi,mi * ) is the classification loss of eyes and mouth opening and closing, which means the loss value calculation of judging the opening and closing state of eyes and mouth, which is a classification problem; ei and mi are the predicted probabilities of whether the eyes and mouth are open or not, ei * and * is the actual opening and closing status of the faces in the training set, 1 if open, 0 otherwise; Frozen training is used during the training process. First, the backbone network is frozen for fine-tuning and then unfrozen for training.

2. The safe driving monitoring method according to claim 1, characterized in that: The FPGA onboard device collects driver videos and performs face detection. During the calculation process, the convolution loop is expanded in two dimensions: the number of input feature maps and the number of output feature maps. The PIPELINE method is used to expand and accelerate the input feature channel, and the input feature channel acceleration is the same as the convolution module acceleration. According to the number of input feature maps, a data is read from the same position on N input feature maps in parallel in each clock cycle, and the product is calculated by the corresponding weight on the convolution kernel. The product result is added through an addition tree with a depth of log2(N) to obtain the partial sum of this convolution and store it; Repeat the above calculation process until all the partial sums required for convolution are calculated, and then add up the partial sums obtained each time through an accumulator to get the result of this convolution calculation; According to the number of output feature maps, a certain data on an input feature map is multiplied by the weights of the corresponding positions of M different convolution kernels in parallel in each clock cycle; the above calculation process is repeated until the products of the corresponding areas of all input feature maps and the corresponding areas of M convolution kernels required for convolution are calculated.

3. The safe driving monitoring method according to claim 1, characterized in that: During the training process, the feature pyramid network FPN module is used to upsample and superimpose feature maps of different sizes to obtain new feature maps and realize multi-scale feature fusion; The feature pyramid network FPN module outputs the following three graphs after processing: predict1=1x1conv(stage3) predict2=1x1conv(stage2)+upsample(1x1conv(predict1)) predict3=1x1conv(stage1)+upsample(1x1conv(predict2)) Among them, stage1, stage2, and stage3 represent the graphs output by the three stages of the MobileNet network, 1x1conv represents the convolution function with a convolution kernel size of 1, and upsample represents the upsampling function.

4. The safe driving monitoring method according to claim 1, characterized in that: During the training process, the SSH module, through three parallel structures, uses the stacking of 3x3 convolutions to replace the effects of 5x5 and 7x7 convolutions, uses two 3x3 convolutions to replace 5x5 convolutions, and uses three 3x3 convolutions to replace 7x7 convolutions, thereby reducing the number of parameters, increasing the receptive field of the feature map, and compensating for the loss of low semantic features caused by network pruning.

5. The safe driving monitoring method according to claim 1, characterized in that: During the training process, the prior frame mechanism is adopted, some anchor points are preset in the picture, and two prior frames of fixed sizes are set with each anchor point as the center.

6. The safe driving monitoring method according to any one of claims 1 to 5, characterized in that: The collected driver video is transmitted to the server after the image data is encrypted and compressed; The compression process is to grayscale the image data and perform discrete cosine transform DCT, and the server uses an inverse DCT algorithm to decompress the image; the specific decompression process includes the following steps: (1) Generate two Logistic chaotic sequences {H1} and {H2} of length L. The initial value x0 and parameter μ of the chaotic sequence must be consistent with those during encryption; (2) Arrange {H1} in descending order and obtain the index sequence h1 of the new position of each element in the sequence; (3) Convert the {H2} sequence to a -1, 1 sequence; (4) Symbol restoration: perform point multiplication of the sequence A to be decrypted and the sequence {H2} obtained in step (3) to obtain the sequence A1; (5) Coordinate restoration: Using the h1 sequence obtained in step (2) as the permutation rule, perform reverse permutation of the element coordinates of the A1 sequence obtained in step (4) to obtain a new one-dimensional sequence, and assign its value to the sequence A; (6) looping through steps (4) and (5) count times, with the value of count remaining the same as that used during encryption. At the same time, the value of count is used as one of the keys of the encryption algorithm; (7) According to the compression matrix used in encryption, the decrypted one-dimensional sequence is placed in the corresponding position of the two-dimensional matrix, and the rest is filled with zeros to obtain the two-dimensional matrix G; (8) Generate an 8th-order DCT matrix T=dctmtx(t) using Matlab built-in functions; dctmtx(t) is a discrete cosine transform function; (9) Performing an inverse DCT transform on the two-dimensional matrix G to obtain a two-dimensional matrix I, where I is the result of zero padding of the original image during encryption; f = @(blockstruct)T′*blockstruct.date*T; I = blockproc(G,[t,t],f); @(block_struct) is a decorator function, block_struct.data represents the incoming data, that is, a picture or a matrix, blockproc is the function block processing matrix in MATLAB, and t in the formula is the size of the processing matrix; here t uses the same value 8 as in encryption; (10) After removing the 0 data value filled in during encryption to enhance the universality of the algorithm, the decrypted image of the one-dimensional encrypted sequence is obtained.

7. A safe driving monitoring system, used to implement the method according to any one of claims 1 to 6; characterized in that: Includes FPGA onboard device, wearable health data collection device, client and server; The FPGA onboard device is arranged in the car cockpit, in front of the driver, and is used to collect the driver's video, perform face detection and positioning, determine the driver's position, and provide a basis for driving behavior analysis; The wearable health data collection device is worn on the body of the passenger and is used to collect the passenger's health data to provide a basis for safety status analysis; The server is used to store data collected by the FPGA onboard device and the wearable health data collection device, analyze the data, and obtain the driver's driving behavior and the safety status of the passengers; The server is connected and communicated with the FPGA onboard device, the wearable health data acquisition device and the client through the network, and the user views the data and manages the vehicle information through the client; The server is a cloud server or a local server.

8. The safe driving monitoring system according to claim 7, characterized in that: The FPGA onboard device includes a camera, a display, an alcohol sensor, a voice recognition module, a communication module, a storage module and a microprocessor; The microprocessor is connected to the camera, display, alcohol sensor, voice recognition module, communication module, and storage module respectively, and is used to control the camera, display, alcohol sensor, voice recognition module, communication module, and storage module to work; The client is used for user registration and login functions, and collects facial information during registration; is used for users to add and manage vehicle information; is used for users to add and manage associated users; and is used for users to view health information reports of themselves or associated users.

9. The safe driving monitoring system according to claim 7, characterized in that: The wearable health data collection device includes a control module, a power module, a temperature detection module, a blood oxygen heart rate detection module, a button module, a buzzer alarm module and a liquid crystal display module; The control module is connected and communicated with the power module, temperature detection module, blood oxygen heart rate detection module, key module, buzzer alarm module and liquid crystal display module respectively.