Driver status monitoring device

Abstract

The present invention relates to a driver status monitoring device, comprising a monitoring box and a lead electroencephalogram cap, wherein the monitoring box comprises a shell, a camera, a signal light, a display screen and an integrated circuit board, the display screen and the camera are both located on the front of the shell, the integrated circuit board is located inside the shell and connected to the camera, the signal light and the display screen, and the integrated circuit board integrates a wireless receiving module and a main chip; the lead electroencephalogram cap comprises a waterproof fabric belt, a monitoring electrode and a monitoring unit, the two ends of the waterproof fabric belt are provided with a connecting unit, the monitoring electrode is distributed on one side in the middle of the waterproof fabric belt, the monitoring unit is located on the other side in the middle of the waterproof fabric belt and connected to the monitoring electrode, and the monitoring unit integrates a wireless sending module, a digital-to-analog converter and a signal amplifier. Compared with the prior art, the present invention can synchronously monitor the driver's brain wave signal, and can more accurately predict psychological activities in combination with face recognition technology, thereby timely reminding the driver and achieving better monitoring effect.

Description

Technical Field

[0001] The invention relates to the field of automobile driving, and in particular to a driver state monitoring device. Background Art

[0002] As the number of cars in the country grows rapidly, traffic accidents are becoming more frequent, seriously damaging people's lives and property safety. Therefore, real-time monitoring and analysis of dangerous driving conditions such as drunkenness, fatigue, anger, and distraction (not paying attention to the road ahead, such as looking at the phone) has become a very important research topic.

[0003] The existing technologies mainly include the following: First, install a breathalyzer or alcohol sensitive element at the driver's position to monitor the alcohol concentration to determine whether the driver is drunk driving; second, monitor fatigue, distraction, or looking at the phone based on the driver's facial image or driving posture recognition. The first technical route can only determine whether the driver is drinking or drunk, and some methods require the driver's active cooperation, which is less feasible. Moreover, if the passengers in the car are drinking, it is easy to cause misjudgment and bring unnecessary trouble. The second technical route is based on image processing and behavior detection, and uses network big data to analyze the driver's facial or body behaviors such as fatigue, closing eyes, and making phone calls. However, for special features such as small eyes and non-obvious expressions, the judgment effect of psychological activities such as distraction is poor, the accuracy is not enough, and it is easy to expose privacy information. Summary of the invention

[0004] The purpose of the present invention is to provide a driver status monitoring device in order to overcome the defects of the above-mentioned prior art. By monitoring the driver's facial expressions, behavioral characteristics and EEG signals respectively, real-time analysis and judgment of the driver's various abnormal states such as drunkenness, fatigue, anger, distraction (not looking at the road ahead, such as looking at a mobile phone) can be performed, and a warning function can be provided to the driver.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A driver status monitoring device comprises a monitoring box and a lead electroencephalogram cap, wherein the monitoring box comprises a shell, a camera, a signal light, a display screen and an integrated circuit board, the display screen and the camera are both located on the front of the shell, the integrated circuit board is located inside the shell and connected to the camera, the signal light and the display screen, and the integrated circuit board integrates a wireless receiving module and a main chip; the lead electroencephalogram cap comprises an elastic band, monitoring electrodes and a monitoring unit, connecting units are provided at both ends of the elastic band, the monitoring electrodes are distributed on one side in the middle of the elastic band, the monitoring unit is located on the other side in the middle of the elastic band and connected to the monitoring electrodes, and the monitoring unit integrates a wireless sending module, a digital-to-analog converter and a signal amplifier.

[0007] Furthermore, the monitoring box also includes speakers, which are arranged on both sides of the shell and connected to the integrated circuit board.

[0008] Furthermore, the elastic band is a waterproof fabric band, and the connection units at both ends of the elastic band are Velcro.

[0009] Furthermore, the display screen is a touch display screen.

[0010] Furthermore, a suction cup is provided at the bottom of the shell for connecting to a center console of a car.

[0011] Furthermore, the elastic band includes at least 8 monitoring electrodes.

[0012] Furthermore, the wireless receiving module and the wireless sending module are a Bluetooth receiving module and a Bluetooth sending module.

[0013] Furthermore, the main chip performs an EEG recognition step, an image recognition step and a comprehensive analysis step, wherein the comprehensive analysis step includes:

[0014] The output results of the EEG recognition step and the image recognition step are obtained respectively for comprehensive analysis;

[0015] If the result of the EEG recognition step is that the driver is in a fatigued state, and the result of the image recognition step is that the driver's eyes are closed for too long, it is comprehensively judged that the driver is in a fatigued state, and the display result is triggered;

[0016] If the result of the EEG recognition step is normal, and the result of the image recognition step is that the driver is holding a phone, it is comprehensively determined that the driver is distracted, triggering the display of the result;

[0017] If the results of the EEG recognition step are inconsistent with those of the image recognition step, the current result is discarded and the display of the result is not triggered.

[0018] Furthermore, the EEG recognition step includes:

[0019] A1. Collect EEG signals through Bluetooth receiving module;

[0020] A2. Perform noise reduction on the collected EEG signals, use a notch filter algorithm to remove mixed power frequency noise in the signal, and then use a 0-50 Hz low-pass filter algorithm to extract the effective band;

[0021] A3. Extract relevant features of the driver's state, use short-time Fourier transform or wavelet transform algorithm to extract the time domain and frequency features of the EEG signal, and use independent component analysis algorithm to extract the driver's head movement and eye movement component features;

[0022] A4, fusing and reducing the extracted features, and inputting them into a classification and discrimination model that has been built and trained, wherein the classification and discrimination model includes a support vector machine model and a long short-term memory neural network model;

[0023] A5. If the output results of the two classification and discrimination modules are consistent, then output the results.

[0024] Furthermore, in step A4, the steps of building and training the support vector machine model and the long short-term memory neural network model are as follows:

[0025] Through experiments, we collected EEG data of drunkenness, fatigue, anger, and distraction, and merged the existing open-source EEG data sets as training and testing data for the model.

[0026] The data is processed for noise reduction and feature extraction to obtain the signal's time domain, frequency domain, head movement and eye movement characteristics;

[0027] The frequency domain features of the signal are used to calculate the power spectral density of each band of the EEG signal through the Welch algorithm, and then the frequency features of each band of the EEG signal are fused with the head movement and eye movement features, and input into the support vector machine model based on the Gaussian kernel function for training and testing. After achieving an acceptable multi-classification accuracy, the model is saved;

[0028] The time domain and frequency domain features are input into the long short-term memory neural network for training and testing, and the model is saved after achieving an acceptable multi-classification accuracy.

[0029] Furthermore, the image recognition step includes:

[0030] B1. Obtain the video captured by the camera, output an image of the video every 5 seconds, identify it, and extract the face image;

[0031] B2. Inputting the facial image into a built and trained image discrimination model, wherein the image discrimination model includes a convolutional neural network model and a dense trajectory algorithm model;

[0032] B3. Compare the expression and behavior recognition results obtained by the two models. If the results are consistent, output them.

[0033] Furthermore, in step B2, the steps of building and training the convolutional neural network model and the dense trajectory algorithm model are as follows:

[0034] Through experiments, we collected images of human facial expressions and behaviors, and merged existing open-source face and behavior datasets as training and testing data for the model.

[0035] Get the video captured by the camera, output an image every 5 seconds, recognize it, and extract the face image;

[0036] After normalization, the facial images are imported into the convolutional neural network for training to recognize facial emotions. After testing, the model is saved after obtaining high multi-classification accuracy.

[0037] The video is input into the dense trajectory algorithm to train the model of behavioral characteristics. After testing and obtaining high multi-classification accuracy, the model is saved.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1) The present invention can synchronously monitor the driver's brain wave signals through the cooperation of the monitoring box and the lead EEG cap. Combined with the face recognition technology, it can make a more accurate prediction of psychological activities, thereby giving timely reminders to the driver, expanding the scope of dangerous state monitoring and achieving better monitoring effects.

[0040] 2) The monitoring box of the present invention is an independent structure, which only has a wireless receiving function but no wireless sending function. All data processing will be performed independently in the monitoring box, and no data will be uploaded to the server, which effectively protects the driver's data privacy.

[0041] 3) The present invention is highly acceptable to drivers, does not affect driving habits, does not bring driving hazards, and can provide safety warning functions to drivers to protect their lives and property.

[0042] 4) The present invention uses multiple steps and multiple algorithm models for comprehensive analysis to distinguish states, thus avoiding misjudgment as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural schematic diagram of the present invention.

[0044] Figure 2 This is a schematic diagram of the back structure of the lead EEG cap.

[0045] Figure 3 Schematic diagram of the process of EEG recognition.

[0046] Figure 4 Schematic diagram of the process of image recognition.

[0047] Figure numerals: 1. Monitoring box, 11. Shell, 12. Camera, 13. Signal light, 14. Display screen, 15. Suction cup, 2. Lead EEG cap, 21. Elastic band, 22. Monitoring electrode, 23. Monitoring unit, 231. Bluetooth sending module, 24. Velcro. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0049] like Figure 1 and Figure 2 As shown, this embodiment provides a driver status monitoring device, including a monitoring box 1 and a lead EEG cap 2.

[0050] The monitoring box 1 includes a housing 11, a camera 12, a signal light 13, a display screen 14, a speaker and an integrated circuit board (not shown in the figure). The housing 11 is a normal rectangular parallelepiped, and a suction cup 15 is provided at the bottom of the housing 11 for connecting to the center console of the car. The display screen 14 and the camera 12 are both located on the front of the housing 11, the camera 12 is located directly above the display screen 14, and the signal lights 13 are arranged on both sides of the camera 12. The display screen 14 adopts a touch display screen 14. The integrated circuit board is located inside the housing 11 and connects the camera 12, the signal light 13, the speaker and the display screen 14, and the integrated circuit board integrates a Bluetooth receiving module and a main chip.

[0051] The lead EEG cap 2 includes an elastic band 21, a monitoring electrode 22 and a monitoring unit 23. The elastic band 21 is made of a waterproof fabric belt, and Velcro 24 is provided at both ends of the elastic band 21, which forms a circular ring structure after being pasted. The monitoring electrode 22 is distributed on one side in the middle of the elastic band 21, and the monitoring unit 23 is located on the other side in the middle of the elastic band 21 and connected to the monitoring electrode 22. There are generally at least 8 monitoring electrodes 22. In this embodiment, 8 monitoring electrodes 22 are used to collect EEG signals in the occipital lobe area of ​​the brain (based on the international 10-20 system, respectively OZ, O1, O2, POZ, PO3, PO4, PO5, PO6 positions), and the electrode type is a dry electrode. The monitoring unit 23 integrates a Bluetooth sending module 231, a digital-to-analog converter and a signal amplifier.

[0052] The brain wave signal detected by the monitoring unit 23 is converted by the signal amplifier and the digital-to-analog converter and then sent through the Bluetooth sending module 231, and is received by the Bluetooth receiving module of the monitoring box 1 and enters the main chip for monitoring.

[0053] The working principle of this embodiment is as follows:

[0054] After the driver is seated, he / she wears the lead EEG cap 2 correctly so that the 8 monitoring electrodes 22 are in full contact with the occipital lobe area of ​​the brain. At this time, the lead EEG cap 2 enters the working state. When the monitoring box 1 is not working, it is in a low-power sleep state. When the Bluetooth connection with the lead EEG cap 2 is successful, the camera 12 is turned on. At this time, the signal light 13 on the right side of the camera 12 of the monitoring box 1 turns green, indicating that the camera 12 enters the working state. When the signal light 13 on the left side of the camera 12 of the smart monitoring box 1 is red, it means that the EEG cap is not worn correctly at this time, and the quality of the collected signal is poor. At this time, the driver needs to adjust the position of the lead EEG cap 2 so that the monitoring electrode 22 is in full contact with the occipital lobe area of ​​the brain (the upper part of the back of the head). When the signal light 13 turns green, it means that the EEG cap is worn correctly and the signal quality is good.

[0055] After the lead EEG cap 2 is put on, the lead EEG cap 2 starts to collect EEG data. The collected raw data is an analog physiological electrical signal in microvolts. Because the EEG signal is weak, it needs to be amplified with high gain by a signal amplifier to highlight the change of the signal. The amplified signal is sampled, quantized, and encoded by an analog-to-digital converter, converted into a digital signal, and then transmitted to the monitoring box 1 through the Bluetooth module. There are many similar products of this type of EEG cap, and the principles are roughly similar, so I will not go into details here. The monitoring box 1 receives the EEG signal through the internal Bluetooth receiving module and transmits it to the main chip for analysis.

[0056] The main chip performs EEG recognition, image recognition and comprehensive analysis steps:

[0057] 1. If Figure 3 As shown, the EEG recognition steps include:

[0058] The first step is to collect EEG signals through the Bluetooth receiving module.

[0059] The second step is to perform noise reduction on the collected EEG signals. A notch filter algorithm is used to remove the mixed power frequency noise in the signal, and then a 0-50 Hz low-pass filter algorithm is used to extract the effective band to improve the reliability of the signal.

[0060] The third step is to extract relevant features of the driver's status. The time domain and frequency features of the EEG signal are extracted using the short-time Fourier transform or wavelet transform algorithm, and the driver's head movement and eye movement component features are extracted using the independent component analysis algorithm.

[0061] The fourth step is to fuse and reduce the dimension of the extracted features and input them into the classification and discrimination model that has been built and trained. The classification and discrimination model includes a support vector machine model and a long short-term memory neural network model. The steps of building and training the support vector machine model and the long short-term memory neural network model are as follows: collect EEG data of drunkenness, fatigue, anger, and distraction through experiments, and merge the existing open source EEG data sets as training and test data for the model; perform noise reduction and feature extraction on the data to obtain the time domain, frequency domain, head movement and eye movement characteristics of the signal; due to the different EEG frequency performances of different human states, EEG signals can be roughly divided into δ band (1-3Hz), θ band (4-7Hz), α band (8-13Hz), β band (14-30Hz), and γ band (31-50Hz). The frequency domain features of the signal are calculated through the Welch algorithm to calculate the power spectral density of each band, and then the frequency features of the five bands are fused with the head movement and eye movement features, and input into the support vector machine algorithm (SVM) based on the Gaussian kernel function for training and testing. Because the features of drunkenness, fatigue, anger, and distraction are different, the model is saved after reaching an acceptable multi-classification accuracy; the time domain and frequency domain features are input into the long short-term memory neural network (LSTM) for training and testing, and the model is saved after reaching an acceptable multi-classification accuracy.

[0062] Step 5: If the output results of the two classification and discrimination modules are consistent, then output the results.

[0063] 2. If Figure 4 As shown, the image recognition step includes:

[0064] The first step is to obtain the video captured by the camera, output an image every 5 seconds, recognize it, and extract the facial image.

[0065] The second step is to input the face image into the built and trained image discrimination model, which includes a convolutional neural network model and a dense trajectory algorithm model. The steps of building and training the convolutional neural network model and the dense trajectory algorithm model are as follows: collect human facial expression and behavior images through experiments, and merge the existing open source data sets of faces and behaviors as the training and test data of the model respectively; obtain the video shot by the camera, output the video with one image every 5 seconds, identify it, and extract the face image; import the facial image into the convolutional neural network after normalization, train it to recognize facial emotions, and save the model after testing to obtain high multi-classification accuracy; input the video into the dense trajectory algorithm (iDT) for model training of behavioral features, and save the model after testing to obtain high multi-classification accuracy.

[0066] The third step is to compare the expression and behavior recognition results obtained by the two models. If the results are consistent, they will be output.

[0067] 3. Comprehensive analysis steps include:

[0068] The output results of the EEG recognition step and the image recognition step are obtained respectively for comprehensive analysis. For example, if the result of the EEG recognition step is that the driver is in a fatigued state, and the result of the image recognition step is that the driver's eyes are closed for too long, it is comprehensively judged that the driver is in a fatigued state. At this time, the display screen of the detection box changes from the date display to "Pay attention to fatigue state", and the voice plays "Pay attention to fatigue state". If the result of the EEG recognition step is normal, and the result of the image recognition step is that the driver is holding a phone, it is comprehensively judged that the driver is dealing with a distracted state. At this time, the screen of the smart detection box changes from the date display to "Pay attention to distracted state", and the voice plays "Pay attention to distracted state". If the results of the EEG recognition step and the image recognition step are inconsistent, the current result is discarded and the next round of recognition and judgment is carried out.

[0069] In summary, the present invention can synchronously monitor the driver's brain wave signals through the cooperation of the monitoring box and the lead EEG cap. Combined with face recognition technology, it can have a more accurate prediction of psychological activity judgment, thereby timely reminding the driver, expanding the scope of dangerous state monitoring, and achieving better monitoring effect. The monitoring box of the present invention is an independent structure, which only has a wireless receiving function and no wireless sending function. All data processing will only be performed independently in the monitoring box, and no data will be uploaded to the server, which effectively protects the driver's data privacy. The present invention is highly acceptable to drivers, does not affect driving habits, does not bring driving risks, can provide drivers with safety warning functions, and protect their life and property safety. The present invention uses multiple steps and multiple algorithm models for comprehensive analysis to distinguish states, avoiding misjudgment as much as possible.

[0070] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A driver status monitoring device, characterized in that: It comprises a monitoring box and a lead EEG cap, wherein the monitoring box comprises a shell, a camera, a signal light, a display screen and an integrated circuit board, the display screen and the camera are both located on the front of the shell, the integrated circuit board is located inside the shell and connected to the camera, the signal light and the display screen, and the integrated circuit board integrates a wireless receiving module and a main chip; the lead EEG cap comprises a waterproof fabric belt, monitoring electrodes and a monitoring unit, both ends of the waterproof fabric belt are provided with connecting units, the monitoring electrodes are distributed on one side in the middle of the waterproof fabric belt, the monitoring unit is located on the other side in the middle of the waterproof fabric belt and connected to the monitoring electrodes, and the monitoring unit integrates a wireless sending module, a digital-to-analog converter and a signal amplifier; The main chip performs an EEG recognition step, an image recognition step and a comprehensive analysis step, wherein the comprehensive analysis step includes: The output results of the EEG recognition step and the image recognition step are obtained respectively for comprehensive analysis; If the result of the EEG recognition step is that the driver is in a fatigued state, and the result of the image recognition step is that the driver's eyes are closed for too long, it is comprehensively judged that the driver is in a fatigued state, and the display result is triggered; If the result of the EEG recognition step is normal, and the result of the image recognition step is that the driver is holding a phone, it is comprehensively determined that the driver is distracted, triggering the display of the result; If the results of the EEG recognition step are inconsistent with those of the image recognition step, the current result is discarded and the display of the result is not triggered; The EEG recognition step comprises: A1. Collect EEG signals through Bluetooth receiving module; A2. Perform noise reduction on the collected EEG signals, use the notch filter algorithm to remove the mixed power frequency noise in the signal, and then use the 0~50Hz low-pass filter algorithm to extract the effective band; A3. Extract relevant features of the driver's state, use short-time Fourier transform or wavelet transform algorithm to extract the time domain and frequency features of the EEG signal, and use independent component analysis algorithm to extract the driver's head movement and eye movement component features; A4, fusing and reducing the extracted features, and inputting them into a classification and discrimination model that has been built and trained, wherein the classification and discrimination model includes a support vector machine model and a long short-term memory neural network model; A5. If the output results of the two classification and discrimination modules are consistent, then output the results; The image recognition step comprises: B1. Obtain the video captured by the camera, output an image of the video every 5 seconds, identify it, and extract the face image; B2. Inputting the facial image into a built and trained image discrimination model, wherein the image discrimination model includes a convolutional neural network model and a dense trajectory algorithm model; B3. Output the expression and behavior recognition results obtained by the two models respectively; In step A4, the steps of building and training the support vector machine model and the long short-term memory neural network model are as follows: Through experiments, we collected EEG data of drunkenness, fatigue, anger, and distraction, and merged the existing open-source EEG data sets as training and testing data for the model. The data is processed for noise reduction and feature extraction to obtain the signal's time domain, frequency domain, head movement and eye movement characteristics; The frequency domain features of the signal are used to calculate the power spectral density of each band of the EEG signal through the Welch algorithm, and then the frequency features of each band of the EEG signal are fused with the head movement and eye movement features, and input into the support vector machine model based on the Gaussian kernel function for training and testing. After achieving an acceptable multi-classification accuracy, the model is saved; Input the time domain and frequency domain features into the long short-term memory neural network for training and testing, and save the model after achieving an acceptable multi-classification accuracy rate; In step B2, the steps of building and training the convolutional neural network model and the dense trajectory algorithm model are as follows: Through experiments, we collected images of human facial expressions and behaviors, and merged existing open-source face and behavior datasets as training and testing data for the model. Get the video captured by the camera, output an image every 5 seconds, recognize it, and extract the face image; After normalization, the facial images are imported into the convolutional neural network for training to recognize facial emotions. After testing, the model is saved after obtaining high multi-classification accuracy. The video is input into the dense trajectory algorithm to train the model of behavioral characteristics. After testing and obtaining high multi-classification accuracy, the model is saved.

2. A driver status monitoring device according to claim 1, characterized in that: The monitoring box also includes speakers, which are arranged on both sides of the shell and connected to the integrated circuit board.

3. A driver status monitoring device according to claim 1, characterized in that: A suction cup is provided at the bottom of the shell for connecting to the center console of the car.

4. The driver status monitoring device according to claim 1, characterized in that: The wireless receiving module and the wireless sending module are a Bluetooth receiving module and a Bluetooth sending module.

Images