Dangerous driving early warning and braking system and method based on multi-core heterogeneous chip

Abstract

The invention discloses a dangerous driving early warning and braking system based on a multi-core heterogeneous chip, which comprises sensor equipment, the multi-core heterogeneous chip and an executing mechanism, the sensor equipment acquires physiological signals, body reaction signals and vehicle state information of a driver and transmits the physiological signals, the body reaction signals and the vehicle state information to the multi-core heterogeneous chip, and the multi-core heterogeneous chip processes the received information and controls the executing mechanism to execute the dangerous driving early warning and braking. And the execution mechanism gives an alarm or brakes according to the information processing result. According to the method, data from different sensors are effectively integrated, consistency explanation of the mental state of a driver is facilitated, and the false alarm or missed alarm phenomenon caused by a single method is greatly reduced; a multi-core heterogeneous chip is adopted for implementation, different tasks can be processed at the same time, different cores execute different tasks, the data processing efficiency is greatly improved, power consumption is reduced, and the number and cost of chips are reduced; the method has high expandability, and the FPGA can integrate different function IPs required by users, so that the purpose of expanding applications is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent automobiles and relates to a dangerous driving warning and braking system and method based on multi-core heterogeneous chips. Background Art

[0002] With the rapid development of road traffic, the popularity of private cars is increasing, so the importance of safe driving is becoming more and more prominent. Fatigue driving is one of the most common dangerous driving behaviors. Drivers who drive for a long time are prone to dangerous driving, which increases the incidence of traffic accidents. Therefore, we need to warn drivers who are driving fatigued. However, ordinary single-core chips are difficult to handle the large amount of data collected by multiple sensors.

[0003] Multi-core heterogeneity refers to the process of using different CPU cores to complete computing tasks. These cores may include multiple main processors, coprocessors, and accelerators. There are obvious differences between them. The most common one is the combination of general-purpose embedded processors and DSP cores. The starting point of multi-core heterogeneous design is to combine different types of cores together so that hardware computing units with different architectures can better perform different types of tasks, allowing them to perform their respective duties and work synchronously. Summary of the invention

[0004] (I) Purpose of the invention

[0005] The purpose of the present invention is to provide a dangerous driving warning and braking system and method based on multi-core heterogeneous chips, in which the signals collected by different sensors are processed by the multi-core heterogeneous chips, and finally the actuators give alarms or even brakes. Based on the characteristics of the multi-core heterogeneous chips, the system and method can achieve the characteristics of small size, high computing efficiency, high scalability, high real-time performance, etc.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention provides a dangerous driving warning and braking system based on a multi-core heterogeneous chip, which includes: a sensor device, a multi-core heterogeneous chip, and an actuator. The sensor device collects the driver's physiological signals, body reaction signals and vehicle status information and transmits them to the multi-core heterogeneous chip. The multi-core heterogeneous chip processes the received information, and the actuator issues an alarm or brakes according to the information processing results.

[0008] Among them, the sensor equipment includes smart wearable devices, infrared cameras, and millimeter-wave radars; smart wearable devices are used to detect electrocardiogram signals and pulse signals; infrared cameras locate and test the driver's facial condition and analyze whether fatigue driving occurs. The basic process is face detection, eye detection, and mouth detection; millimeter-wave radars obtain the distance between the vehicle and the vehicle in front or obstacles and the relative speed to the vehicle in front in real time.

[0009] Among them, the multi-core heterogeneous chip includes a multi-core DSP module and an FPGA module; the working modules of the multi-core DSP include the high-speed interconnection module SRIO, the inter-core communication module IPC, and the external memory interface EMIF, which have high-performance computing capabilities and are used to process data collected by sensors; the FPGA module includes the data receiving module LVDS and the CAN bus module, which are responsible for transmitting the data collected by the sensors.

[0010] Among them, the multi-core DSP module includes a master core, a first slave core, a second slave core, and a third slave core; the multi-core DSP module adopts a master-slave parallel processing mode and completes communication through an inter-core interrupt IPC mechanism.

[0011] Among them, the first slave core and the second slave core complete data movement and calculation. After the calculation is completed, an inter-core interrupt is sent to the main core to store the data in the shared memory. The main core responds to the interrupt and reads the data from the shared memory. According to the DS evidence theory, it relies on the feedback information of each core to make a decision and obtain a consistent interpretation after multi-sensor fusion. If the main core determines that the driver is fatigued after calculation, it will notify the third slave core through IPC. The third slave core determines whether the current distance to the vehicle in front and the relative speed are within the safety threshold. If so, it will send an alarm signal and control the sound device; if not, it will perform emergency braking to avoid an accident with the vehicle in front.

[0012] Among them, the first slave core receives the driver's physiological characteristic signals uploaded by the smart wearable device through the gigabit network port, including electrocardiogram signals and pulse signals, intercepts multiple time series data fragments, matches the respective target judgment models, obtains two fatigue level assessment results, and stores them in the shared memory.

[0013] Among them, the second slave core obtains the driver's physiological response information of the infrared camera from the FPGA module through the SRIO interface, and then extracts the mouth and eye features, matches them with the preset facial information, obtains the matching degree, and stores it in the shared memory. The FPGA module receives the infrared camera data through the data receiving module LVDS; the third slave core receives the vehicle driving information message sent by the millimeter-wave radar through the CAN bus module, and parses the distance between the vehicle and the front vehicle or obstacle and the relative speed with the front vehicle.

[0014] Among them, the actuator includes a buzzer and an anti-lock braking module.

[0015] The present invention also provides a dangerous driving warning and braking method based on a multi-core heterogeneous chip, comprising the following steps:

[0016] Step 1: Obtain the driver's physiological signals, body reaction signals, the distance between the vehicle and the vehicle in front or obstacles, and the relative speed to the vehicle in front;

[0017] Step 2: Determine whether the driver is driving fatigued based on the driver's physiological signals and body reaction signals; if the driver is driving fatigued, the actuator will sound an alarm;

[0018] Determine whether the distance to the preceding vehicle or obstacle and the relative speed to the preceding vehicle are within the safety threshold; if they are greater than the safety threshold, the actuator is triggered to brake the vehicle;

[0019] If the driver is driving fatigued and the distance to the vehicle in front or obstacles and the relative speed to the vehicle in front are within the safety threshold, the alarm will continue to sound after one minute, triggering the actuator to brake the vehicle.

[0020] Furthermore, in the first step, an infrared camera is installed at the front end of the dashboard of the cab to ensure that the infrared camera can fully cover the driver's facial area and obtain the driver's body reaction signal; the millimeter wave radar is installed inside the front bumper to obtain the distance between the vehicle and the vehicle in front or obstacles and the relative speed to the vehicle in front; the driver wears the smart wearable device to obtain the driver's physiological signals; after the vehicle speed changes from 0 to greater than 0km / h, the infrared camera is started to collect data, and the second slave core receives it as the preset facial information, and the smart wearable device collects electrocardiogram signals and pulse signals as the driver's awake electrocardiogram signals and awake pulse signals; 5 minutes after the infrared camera starts collecting data, the first slave core starts to receive physiological characteristics from the smart wearable device; the third slave core receives millimeter wave radar data.

[0021] (III) Beneficial effects

[0022] The dangerous driving warning and braking system and method based on multi-core heterogeneous chips provided by the above technical solution have the following beneficial effects:

[0023] 1) The present invention effectively integrates data from different sensors, which helps to produce a consistent interpretation of the driver's mental state and greatly reduces the false alarm or missed alarm phenomenon caused by using a single method.

[0024] 2) This system is implemented using multi-core heterogeneous chips, which can process different tasks simultaneously. Different cores perform different tasks, which greatly improves data processing efficiency, reduces power consumption, and reduces the number and cost of chips.

[0025] 3) This system has high scalability, and FPGA can integrate different functional IPs required by users to achieve the purpose of expanding applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic diagram of data collection, processing and decision making in the present invention;

[0027] Figure 2 The present invention is a flow chart of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0029] At present, driver dangerous driving detection technology is divided into three categories: the first category is based on physiological signal detection, the second category is based on body reaction detection technology, and the third category is based on vehicle status information detection technology. The present invention collects three types of data through sensors for comprehensive evaluation. In this embodiment, a multi-core heterogeneous chip is used for data processing and decision-making.

[0030] like Figure 1 As shown, the dangerous driving warning and braking system based on multi-core heterogeneous chips in this embodiment includes: sensor equipment, multi-core heterogeneous chips, and actuators. The sensor equipment collects the driver's physiological signals, body reaction signals and vehicle status information and transmits them to the multi-core heterogeneous chip. The multi-core heterogeneous chip processes the received information, and the actuator issues an alarm or brakes according to the information processing results.

[0031] Among them, the sensor equipment includes smart wearable devices, infrared cameras, and millimeter-wave radars; the smart wearable devices can be smart watches, which are used to detect electrocardiogram signals and pulse signals; the infrared camera locates and tests the driver's facial condition, and analyzes whether fatigue driving occurs. Its basic process is face detection, eye detection, and mouth detection; the millimeter-wave radar obtains the distance between the vehicle and the vehicle in front or obstacles and the relative speed to the vehicle in front in real time.

[0032] The multi-core heterogeneous chip includes a multi-core DSP module and an FPGA module; the working modules of the multi-core DSP include a high-speed interconnect module SRIO, an inter-core communication module IPC, and an external memory interface EMIF, which have high-performance computing capabilities and are used to process data collected by sensors; the FPGA module includes a data receiving module LVDS and a CAN bus module, which are responsible for transmitting data collected by sensors.

[0033] Among them, the multi-core DSP module includes the master core, the first slave core, the second slave core, and the third slave core; the multi-core DSP module adopts the master-slave parallel processing mode and completes communication through the inter-core interrupt IPC mechanism. The first slave core and the second slave core complete data movement and calculation. After the calculation is completed, they send inter-core interrupts to the master core and store the data in the shared memory. The master core responds to the interrupt and reads the data from the shared memory. According to the DS evidence theory, it relies on the feedback information of each core to make a decision and obtain a consistent explanation after multi-sensor fusion. If the master core determines that the driver is fatigued after calculation, it will notify the third slave core through IPC. The third slave core determines whether the current distance and relative speed to the vehicle in front are within the safety threshold. If so, it will send an alarm signal and control the sound device; if not, it will perform emergency braking to avoid accidents with the vehicle in front.

[0034] Among them, the first slave core receives the driver's physiological characteristic signals, namely electrocardiogram signals and pulse signals, uploaded by the smart wearable device through the gigabit network port, intercepts multiple time series data fragments, matches their respective target judgment models, obtains two fatigue level assessment results, and stores them in the shared memory.

[0035] The second slave core obtains the driver's physiological response information of the infrared camera from the FPGA module through the SRIO interface, then extracts the mouth and eye features, matches them with the preset facial information, obtains the matching degree, and stores it in the shared memory. The FPGA module receives the infrared camera data through the data receiving module LVDS.

[0036] Among them, the third slave core receives the vehicle driving information message sent by the millimeter wave radar through the CAN bus module, and analyzes the distance between the vehicle and the front vehicle or obstacle and the relative speed with the front vehicle.

[0037] The master core is mainly used to receive information processed by each slave core, make comprehensive information decisions, and trigger the action of the actuator.

[0038] The multi-core DSP module uses TI's latest 8-core high-performance processor TMS320C6678 chip, and the FPGA module uses Xilinx's Kintex-7 series.

[0039] The actuator includes a buzzer and an anti-lock braking module.

[0040] Reference Figure 2 As shown, the dangerous driving warning and braking method based on multi-core heterogeneous chips in this embodiment includes the following steps:

[0041] Step 1: Obtain the driver's physiological signals, body reaction signals, the distance between the vehicle and the vehicle in front or obstacles, and the relative speed to the vehicle in front;

[0042] Step 2: Determine whether the driver is driving fatigued based on the driver's physiological signals and body reaction signals; if the driver is driving fatigued, the actuator will sound an alarm;

[0043] Determine whether the distance to the preceding vehicle or obstacle and the relative speed to the preceding vehicle are within a safety threshold; if they are greater than the safety threshold, the actuator is triggered to brake the vehicle.

[0044] Among them, when the driver is driving fatigued, the distance to the vehicle in front or obstacles and the relative speed to the vehicle in front are at the safety threshold, the alarm will continue to sound after one minute, triggering the actuator to brake the vehicle.

[0045] In the first step, an infrared camera is installed at the front end of the dashboard in the cab to ensure that the infrared camera can fully cover the driver's facial area and obtain the driver's body reaction signals; the millimeter-wave radar is installed inside the front bumper to obtain the distance between the vehicle and the vehicle in front or obstacles and the relative speed with the vehicle in front; the driver wears the smart wearable device to obtain the driver's physiological signals.

[0046] After the vehicle speed changes from 0 to greater than 0 km / h, the infrared camera is started to collect data, and the second slave core receives it as the preset facial information. The smart wearable device collects the ECG signal and pulse signal as the driver's awake ECG signal and awake pulse signal. Five minutes after the infrared camera starts collecting data, the first slave core starts to receive physiological characteristics from the smart wearable device. The third slave core receives millimeter wave radar data.

[0047] In the second step, after the first and second slave cores complete the calculation, they send an inter-core interrupt to the main core. The main core determines whether the driver is driving fatigued after making a decision. If the detection result is yes, the millimeter-wave radar data is obtained from the third slave core through the inter-core interrupt. If the relative speed and distance to the vehicle in front are less than the safety threshold, the control unit will act to perform emergency braking on the car. If it is greater than the safety threshold, the main core will send a control signal to control the buzzer to issue a fatigue driving reminder.

[0048] The first slave core completes the collection of the driver's physiological information from the smart wearable device received from Ethernet by carrying the LWIP protocol stack.

[0049] FPGA receives the serial differential signal from the infrared camera through its built-in LVDS receiver interface. Inside the FPGA, the received serial data is converted into parallel data through ISERDES and written into DDR. The second slave core moves the data stored in DDR to the shared memory through SRIO. SRIO selects WDMA mode, lane selects 4x, and rate selects 5Gbps. The doorbell is sent when the transmission is completed.

[0050] The inter-core communication module can make full use of the hardware to achieve efficient inter-core communication. In the present invention, it is mainly used for real-time communication between the master core and the slave core. When the master core receives an interrupt and completes the interrupt response, the slave core immediately performs data processing for the next cycle, thereby improving work efficiency.

[0051] The FPGA integrates the CAN bus IP, and the third slave core on the DSP side can read the CAN bus message through the EMIF interface, from which the relative speed and distance to the preceding vehicle can be parsed.

[0052] Since the driver is in a relatively sober state when he starts to drive the car, in this embodiment, face data sampling is performed within 5 minutes after the car is started as preset facial information, so that the image can have a high matching degree after the driver is changed.

[0053] During the infrared image acquisition process, it will be affected by factors such as weather and environmental lighting, resulting in a decrease in image quality. Therefore, it is necessary to preprocess the infrared image, such as denoising and enhancement, to improve the image quality, eliminate or reduce the influence of lighting, and provide the same conditions as much as possible for subsequent matching. The video collected during normal driving also needs to undergo this preprocessing. The video image obtained from the infrared camera is segmented to obtain the driver's eye area and mouth area segmentation map. The key feature points of the face are obtained through the facial feature point detection algorithm. Subsequently, fatigue-related feature parameters such as eye opening and mouth opening are calculated based on the feature point position. The fatigue state judgment threshold is set based on experimental or statistical data, such as the ratio of eye closure time and mouth opening time. The new driver's facial image data is received and analyzed in real time, the above fatigue-related feature parameters are calculated, and compared with the set threshold. After the matching is completed, a number between 0 and 1 is output to represent the degree of fatigue. The larger the value, the greater the degree of fatigue, and the smaller the output, the smaller the degree of fatigue.

[0054] In this embodiment, the awake pulse signal and awake heart rate signal are collected within 5 minutes after the car is started, and compared with the awake value in the subsequent real-time collection.

[0055] Pulse signals and ECG signals are time series, and generally there will be noise. Wavelet threshold denoising can be used. The wavelet coefficients are processed using thresholds at each level to filter out the frequency band where the noise is located, and then the signal is reconstructed. Key feature parameters such as pulse frequency, rhythm, waveform features, and waveform features such as P wave, QRS complex, and T wave of the ECG signal are extracted from the collected pulse and ECG signals for subsequent matching. The pulse and ECG signal features extracted during driving are matched and analyzed with the features during wakefulness. After the matching is completed, two numbers with a value range of 0 and 1 are output, representing the fatigue degree of the two physiological characteristics. The larger the value, the greater the fatigue degree, and the smaller the output, the smaller the fatigue degree.

[0056] The present invention adopts decision-level fusion, that is, each sensor makes a judgment on the target based on the collected data, and then fuses these results. The present invention adopts DS evidence theory as a method of information fusion. This method is to assign different weights to different sensors in different states, and then use the evidence combination method provided by DS evidence theory to improve the target perception effect. The recognition results based on physiological characteristics, based on body reactions, and based on vehicle driving status are fused to improve the perception ability.

[0057] If the decision result is fatigue driving, the system reads the distance to the vehicle in front or the obstacle and the relative speed to the vehicle in front. If it is less than the safety threshold, the system outputs an emergency braking command and the car brakes urgently. If it is greater than the safety threshold, the system issues a fatigue driving alarm for 1 minute. If the system calculates that the driver is still in a fatigue driving state after 1 minute, the car will turn on the double flash and slowly decelerate to 0km / h.

[0058] In summary, the above embodiments of the present invention rely on multi-core heterogeneous chips to process different data from different sensors, and make judgments on the processed data, so as to issue warnings or brakes in time when the driver drives dangerously, thereby reducing the occurrence of traffic accidents.

[0059] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A dangerous driving warning and braking system based on multi-core heterogeneous chips, characterized in that: It includes: sensor equipment, multi-core heterogeneous chips, and actuators. The sensor equipment collects the driver's physiological signals, body reaction signals and vehicle status information and transmits them to the multi-core heterogeneous chip. The multi-core heterogeneous chip processes the received information, and the actuator issues an alarm or brakes according to the information processing results.

2. The dangerous driving warning and braking system based on multi-core heterogeneous chips according to claim 1, characterized in that: The sensor equipment includes smart wearable devices, infrared cameras, and millimeter-wave radars; smart wearable devices are used to detect electrocardiogram signals and pulse signals; infrared cameras locate and test the driver's facial condition and analyze whether fatigue driving occurs. The basic process is face detection, eye detection, and mouth detection; millimeter-wave radars obtain the distance between the vehicle and the vehicle in front or obstacles and the relative speed to the vehicle in front in real time.

3. The dangerous driving warning and braking system based on multi-core heterogeneous chips as claimed in claim 2, characterized in that: The multi-core heterogeneous chip includes a multi-core DSP module and an FPGA module; the working modules of the multi-core DSP include a high-speed interconnect module SRIO, an inter-core communication module IPC, and an external memory interface EMIF, which have high-performance computing capabilities and are used to process data collected by sensors; the FPGA module includes a data receiving module LVDS and a CAN bus module, which are responsible for transmitting data collected by sensors.

4. The dangerous driving warning and braking system based on multi-core heterogeneous chips as claimed in claim 3, characterized in that: The multi-core DSP module includes a master core, a first slave core, a second slave core, and a third slave core; the multi-core DSP module adopts a master-slave parallel processing mode and completes communication through an inter-core interrupt IPC mechanism.

5. The dangerous driving warning and braking system based on multi-core heterogeneous chips as claimed in claim 4, characterized in that: The first slave core and the second slave core complete data movement and calculation. After the calculation is completed, they send an inter-core interrupt to the master core and store the data in the shared memory. The master core responds to the interrupt and reads the data from the shared memory. According to the DS evidence theory, it relies on the feedback information of each core to make a decision and obtain a consistent interpretation after multi-sensor fusion. If the main core determines that the driver is driving fatigued, it will notify the third slave core through IPC. The third slave core determines whether the current distance to the vehicle in front and the relative speed are within the safety threshold. If so, it will send out an alarm signal and control the sound device; if not, it will perform emergency braking to avoid an accident with the vehicle in front.

6. The dangerous driving warning and braking system based on multi-core heterogeneous chips as claimed in claim 5, characterized in that: The first slave core receives the driver's physiological characteristic signals uploaded by the smart wearable device through the gigabit network port, including electrocardiogram signals and pulse signals, intercepts multiple time series data fragments, matches the respective target judgment models, obtains two fatigue level assessment results, and stores them in the shared memory.

7. The dangerous driving warning and braking system based on multi-core heterogeneous chips as claimed in claim 6, characterized in that: The second slave core obtains the driver's physiological response information of the infrared camera from the FPGA module through the SRIO interface, then extracts the mouth and eye features, matches them with the preset facial information, obtains the matching degree, and stores it in the shared memory. The FPGA module receives the infrared camera data through the data receiving module LVDS; the third slave core receives the vehicle driving information message sent by the millimeter-wave radar through the CAN bus module, and parses the distance between the vehicle and the front vehicle or obstacle and the relative speed with the front vehicle.

8. The dangerous driving warning and braking system based on multi-core heterogeneous chips as claimed in claim 7, characterized in that: The actuator includes a buzzer and an anti-lock braking module.

9. A dangerous driving warning and braking method based on multi-core heterogeneous chips, characterized in that: The method is performed based on the system according to claim 8, and the method comprises the following steps: Step 1: Obtain the driver's physiological signals, body reaction signals, the distance between the vehicle and the vehicle in front or obstacles, and the relative speed to the vehicle in front; Step 2: Determine whether the driver is driving fatigued based on the driver's physiological signals and body reaction signals; if the driver is driving fatigued, the actuator will sound an alarm; Determine whether the distance to the preceding vehicle or obstacle and the relative speed to the preceding vehicle are within the safety threshold; if they are greater than the safety threshold, the actuator is triggered to brake the vehicle; If the driver is driving fatigued and the distance to the vehicle in front or obstacles and the relative speed to the vehicle in front are within the safety threshold, the alarm will continue to sound after one minute, triggering the actuator to brake the vehicle.

10. The dangerous driving warning and braking method based on multi-core heterogeneous chips according to claim 9, characterized in that: In the first step, an infrared camera is installed at the front of the dashboard in the cab to ensure that the infrared camera can fully cover the driver's facial area and obtain the driver's body reaction signal; the millimeter wave radar is installed inside the front bumper to obtain the distance between the vehicle and the vehicle in front or obstacles and the relative speed with the vehicle in front; the driver wears the smart wearable device to obtain the driver's physiological signals; After the vehicle speed changes from 0 to greater than 0 km / h, the infrared camera is started to collect data, and the second slave core receives it as the preset facial information. The smart wearable device collects the electrocardiogram signal and pulse signal as the driver's awake electrocardiogram signal and awake pulse signal; 5 minutes after the infrared camera starts collecting data, the first slave core starts to receive the physiological characteristics from the smart wearable device; The third receives millimeter wave radar data from the core.

Images