Millimeter wave radar and vision fused fatigue detection method

Abstract

The invention discloses a millimeter-wave radar and vision fused fatigue detection method, which relates to the field of safe driving, and comprises the following steps of: acquiring a head video and a vital sign signal of a driver through a camera and a millimeter-wave radar, preprocessing the head video and the vital sign signal, calculating a facial fatigue parameter and a respiratory rate by using a software algorithm, and determining the fatigue of the driver according to the facial fatigue parameter and the respiratory rate. And judging the fatigue state of the driver according to the facial fatigue parameter and the breathing frequency. According to the invention, the accuracy of fatigue detection is improved by using a dual detection mode, and the phenomena of single feature and false detection possibly existing in single fatigue detection are avoided.

Description

Technical Field

[0001] The present invention relates to the field of safe driving technology, and more specifically, to a fatigue detection method integrating millimeter-wave radar and vision. Background Art

[0002] Safe driving technology refers to a series of technologies designed to improve road safety, reduce traffic accidents and enhance driving experience. Drivers often fail to realize that they are in a fatigued driving state due to their working environment and driving time, which leads to major safety hazards. Accurately detecting the driver's fatigue state is an important means of preventing traffic accidents. It not only avoids potential dangerous accidents on road traffic, improves road traffic safety, but also effectively protects the driver's personal safety.

[0003] There are individual differences between different drivers, and a single fatigue detection method based on machine vision is difficult to adapt to all situations. In practical applications, factors such as lighting conditions and facial occlusion may also affect the accuracy and stability of the detection algorithm. By integrating information from multiple different sources, such as physiological signals, vehicle behavior, facial features, etc., a more comprehensive and accurate assessment of the driver's fatigue state can be achieved. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides the following technical solution: a fatigue detection method integrating millimeter wave radar and vision, comprising:

[0005] Step 1. Obtain the driver's head video and vital signs signal through the visible light camera and millimeter wave radar installed in the car; the head video refers to a video containing the driver's head image; the vital signs signal includes breathing information; the visible light camera is used to obtain the driver's head video, and the millimeter wave radar is used to capture the driver's breathing information.

[0006] Step 2. Preprocessing the head video obtained in step 1, wherein the preprocessing includes at least one of denoising, image enhancement, image segmentation or image correction;

[0007] Step 3. Perform face, eye and pupil recognition on the head video, and obtain facial fatigue parameters through software algorithm, where the facial fatigue parameters refer to eye closure time and blinking frequency;

[0008] Step 4. Preprocess the respiratory information of step 1, perform signal separation on the preprocessed data, and finally complete the respiratory rate detection;

[0009] Step 5. Determine the driver's fatigue state based on the facial fatigue parameters and breathing frequency.

[0010] Furthermore, the head video obtained in step 1 is preprocessed, specifically including:

[0011] Use multi-scale Retinex algorithm to eliminate the impact of incident light;

[0012] The head video brightness is improved by redistributing pixel values ​​through histogram equalization, which increases the overall contrast of the head video.

[0013] Remove salt and pepper noise from head videos by median filtering.

[0014] Furthermore, performing face recognition on the head video to obtain the driver's face image specifically includes:

[0015] The head video is identified by using a haar operator to obtain a haar feature, and the haar feature is processed by using a cascade classifier to obtain at least one detection frame.

[0016] Furthermore, according to the fatigue detection method of claim 1, it is characterized in that facial fatigue parameters are acquired through software algorithms, and the facial fatigue parameters refer to eye closure time and blinking frequency, specifically including:

[0017] The driver's eyes are identified in the face image through the Haar classifier integrated in the camera, and a line is drawn along the outer boundary of the eyes to form the eye area;

[0018] The driver's pupil is identified in the human eye area through the Haar classifier, and a line is drawn along the outer boundary of the pupil to form the pupil area;

[0019] On the premise that the face image is recognized, the process of failing to recognize the pupil is recorded as an eye-closing event, and the duration of the eye-closing event is counted to obtain the eye closure time;

[0020] The real-time blinking frequency is calculated through software algorithms;

[0021] The blink frequency expression is:

[0022]

[0023] Among them, t1 is the start time of a detection cycle, t4 is the end time of a detection cycle, c is the total number of blinks in a cycle, and f is the blink frequency.

[0024] Furthermore, the fatigue detection method according to claim 1 is characterized in that the respiratory information of step 1 is preprocessed, the preprocessed data is subjected to signal separation, and finally the respiratory frequency detection is completed, which specifically includes:

[0025] The received beat frequency signal is first subjected to the range dimension FFT to determine the range-bin corresponding to the measured target;

[0026] The phase information corresponding to the target is calculated by the inverse tangent function, and then the real phase of the target is obtained by phase winding;

[0027] Calculate the phase difference between adjacent phases and filter out impulse noise through a sliding mean filtering algorithm;

[0028] The respiratory signal is separated by an IIR bandpass filter, and finally the respiratory frequency is extracted by spectrum estimation. The expression of the respiratory frequency is:

[0029] R = f R *60

[0030] Where R is the respiratory rate, f R is the frequency at the peak of the FFT spectrum.

[0031] Furthermore, the fatigue detection method according to claim 1 is characterized in that judging the driver's fatigue state based on the facial fatigue parameter and the breathing frequency specifically includes:

[0032] When the eyes are closed for more than two seconds or the breathing rate exceeds a predetermined fatigue threshold, the driver is considered to be in a fatigued state.

[0033] The implementation of the present invention includes the following beneficial effects: the present invention combines facial feature detection and driver vital sign detection to monitor and analyze the driver's fatigue state, and can more accurately determine whether the driver is in a fatigue state.

[0034] Compared with the prior art, the present invention has the following advantages and technical effects:

[0035] 1. Non-contact detection method: The present invention uses a variety of sensors, including visible light cameras and millimeter wave radars, to detect and warn the driver's fatigue state in a non-contact manner. This advantage means that the fatigue detection system of the present invention will not have any impact on the driver and will not cause driving obstacles to the driver like traditional wearable detection devices.

[0036] 2. Multi-source information fusion detection: The present invention can achieve a more comprehensive and accurate assessment of the driver's fatigue state by fusing information from multiple different sources, including vital signs and facial features. There are individual differences between different drivers, and a single fatigue detection method based on machine vision is difficult to adapt to all situations. In practical applications, factors such as lighting conditions and facial occlusion may also affect the accuracy and stability of the detection algorithm, while millimeter-wave radar is rarely affected by factors. Therefore, this detection method that fuses multi-source information can accurately detect the driver's fatigue state under complex conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic flow chart of a fatigue detection method combining millimeter wave radar and vision provided in Example 1 of the present invention;

[0038] Figure 2 A schematic diagram of a fatigue detection system integrating millimeter-wave radar and vision provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Embodiment 1: This embodiment provides a fatigue detection method combining millimeter wave radar and vision. Figure 1 The flowchart of the fatigue detection method combining millimeter wave radar and vision in the embodiment includes:

[0041] Step 1. Equipment deployment and acquisition of various driver information;

[0042] In an embodiment of the present invention, the driver's facial video refers to the video of the driver's head captured by a CMOS camera arranged directly facing the driver, and is used as the original image of the driver during driving. In practical applications, the application environment of the fatigue detection system is inside a car, and the driver's range of activities is fixed. The position of the head generally does not change much during driving. Therefore, the camera's shooting angle can be adjusted to face the driver's head position, thereby reducing the difficulty of face recognition. In an embodiment of the present invention, the driver's respiratory physiological information is extracted by an FMCW radar arranged directly facing the driver's chest position. The target detection callback signal in the FMCW radar system depends on the waveform of the FM signal transmitted to the radar site.

[0043] Step 2. Preprocess the head video obtained in step 1;

[0044] Specifically, the present invention uses a multi-scale Retinex algorithm to process the acquired head video at different scales, so as to better capture and process the details and lighting information of the driver's face. Through multi-scale decomposition and reconstruction, the multi-scale Retinex algorithm can enhance the contrast and details of the image while maintaining the true color of the object, and better adapt to different lighting conditions and object types. The operations in the multi-scale Retinex algorithm are similar to the early screen space reflection algorithm. The core idea of ​​the Retinex algorithm is to eliminate the influence of incident light and obtain the true color and texture information of the object under different lighting conditions. In the multi-scale Retinex algorithm, the specific process is shown in the following formula:

[0045]

[0046] Where k is the number of Gaussian kernels. When k = 1, MSR degenerates to SSR. Generally speaking, in order to ensure that the advantages of the high, medium and low scales of the screen space reflection algorithm are taken into consideration, the value of K is usually 3. The convolution kernel size of 15, 80, and 200 can achieve better results. The multi-scale Retinex algorithm chooses to use three Gaussian kernels of different scales for convolution operations, which can obtain three image components of different scales. The coarser scale components capture lower-frequency illumination information, while the finer scale components capture higher-frequency details. This can better process video images at different scales while combining the advantages of components of different scales. The weights of the three equally divided convolutions are evenly distributed to each incident image of different scales. During the addition process, the image components of different scales are weighted and synthesized. In order to obtain more accurate image enhancement results during weighted synthesis, appropriate weights are given to each scale according to the contribution of components of different scales.

[0047] The brightness of the head video is improved by redistributing pixel values ​​through histogram equalization, and the overall contrast of the head video is increased. The histogram refers to a method used to describe the distribution of image pixel values. The salt and pepper noise in the head video is removed by median filtering. The head video is processed using a median filter with a window size of 3*3 to remove the salt and pepper noise and weaken the Gaussian noise to a certain extent.

[0048] Step 3. Perform face, eye and pupil recognition on the head video, and obtain facial fatigue parameters through software algorithm, where the facial fatigue parameters refer to eye closure time and blinking frequency;

[0049] In an embodiment of the present invention, face, eye and pupil recognition are performed on the head video, and facial fatigue parameters are acquired through software algorithms. The facial fatigue parameters include eye closure time and blinking frequency, including: the driver's face and eye recognition are realized through the Haar classifier encapsulated in the openmv firmware, and the Haar classifier encapsulated in the openmv firmware imports a special Haar operator for recognizing human eyes and faces;

[0050] In the embodiment of the present invention, facial fatigue parameters are collected by software algorithms. Facial fatigue parameters refer to eye closure time and blinking frequency, including: the software algorithm used in the present invention is to identify and locate the pupil, calculate the area of ​​the pupil and track its size changes, and finally determine the blinking state of the driver's pupil according to the size of the detected area of ​​the driver's pupil. Wherein pupil recognition is to detect the darkest area in the human eye image, and then determine whether it is the driver's pupil according to the size of the area; on the premise of recognizing the face image, the process of not recognizing the pupil is recorded as an eye closure event, and the duration of the eye closure event is counted to obtain the eye closure time; the blinking frequency is calculated by the blinking frequency detection algorithm. The blinking frequency detection algorithm is similar to the PERCLOS algorithm. The degree of eye closure is detected in a detection cycle t1 to t4. The PERCLOS algorithm of the P80 standard is also referred to. The degree of eye closure reaches 80% as the eye closure standard, and the pupil area of ​​the real-time human eye is detected. When the degree of eye closure exceeds 80% during each blink, it is counted as one blink. The blinking frequency expression is:

[0051]

[0052] Among them, t1 is the start time of a detection cycle, t4 is the end time of a detection cycle, c is the total number of blinks in a cycle, and f is the blink frequency.

[0053] Step 4, preprocessing the respiratory information of step 1, performing signal separation on the preprocessed data, and finally completing respiratory rate detection;

[0054] In an embodiment, the breathing information of step 1 is preprocessed, the preprocessed data is subjected to signal separation, and finally the breathing frequency detection is completed, including: the FMCW signal emitted by the millimeter wave radar is used to calculate the echo signal according to Maxwell electromagnetic field theory, wherein the FMCW signal refers to a sinusoidal signal whose frequency changes linearly with time, and the echo signal expression is:

[0055]

[0056] Among them, f c is the starting frequency, T Cis the frequency modulation period, B is the frequency modulation bandwidth, c is the speed of light, t d is the delay time, A R is the amplitude of the echo signal;

[0057] The echo signal is received by the receiving antenna, input into the low noise amplifier, and then mixed with the local transmission signal through the mixer, and then passed through the low-pass filter to obtain the intermediate frequency signal. Perform the distance dimension FFT transformation on the echo signal to determine the corresponding distance range, calculate the corresponding phase information through the positive and negative tangent functions, and then perform the unwrapping operation on the obtained phase value to obtain the real phase value. The method is: use each phase to subtract the previous phase in turn. If the result is greater than Π, then subtract 2Π from the current phase value; if the result is less than Π, then add 2Π to the current phase value; the rest of the cases are not processed; by subtracting the continuous phase value, perform the phase difference operation on the unfolded phase, which helps to enhance the heartbeat signal and eliminate any phase drift, where the phase value refers to the frequency value formed by the phase change; filter out the pulse noise through the sliding mean filtering algorithm, separate the respiratory signal with the IIR bandpass filter, and finally extract the respiratory frequency through spectrum estimation. The expression of the respiratory frequency is:

[0058] R = f R *60

[0059] Where R is the respiratory rate, f R is the frequency at the peak of the FFT spectrum.

[0060] Step 5. Determine the driver's fatigue state based on the facial fatigue parameters and breathing frequency.

[0061] In an embodiment, based on the facial fatigue parameters and breathing frequency, the driver's fatigue state is determined, including: Under normal circumstances, the normal blinking frequency of a person is 16 times / minute to 20 times / minute, and each blink lasts about 3 seconds. If a person is in a fatigued state, his blinking frequency will increase accordingly. The present invention selects 30 times / minute as the blinking threshold for fatigue driving judgment and 2s as the eye closing time threshold for fatigue driving judgment according to actual conditions; when the facial fatigue parameter is higher than the threshold or the breathing frequency drops by more than 30% of the normal state, the driver is judged to be in a fatigued state. The dual detection method improves the accuracy of fatigue detection, avoids the false detection phenomenon that may exist in single-aspect features and word fatigue detection, and effectively avoids the false alarm phenomenon of fatigue driving during the driver's driving process, thereby ensuring the driver's safe driving state.

[0062] Example 2: Please refer to Figure 2As shown, the part not described in detail in this embodiment is described in Example 1, which provides a fatigue detection method integrating millimeter wave radar and vision, including a sensor data acquisition module, a data preprocessing module, a feature recognition module and a fatigue judgment module. Among them, the sensor data acquisition module is responsible for collecting the head video and vital sign signals of the driver in the vehicle. The data preprocessing module is responsible for preprocessing the collected raw data. The feature recognition module is used to obtain the driver's facial fatigue parameters and breathing frequency. The fatigue judgment module is used to judge the driver's fatigue state based on the facial fatigue parameters and breathing frequency.

[0063] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A fatigue detection method integrating millimeter wave radar and vision, characterized in that: The method comprises the following steps: Step 1. Obtain the driver's head video and vital signs signal through the visible light camera and millimeter wave radar installed in the car; the head video refers to a video containing the driver's head image; the vital signs signal includes breathing information; the visible light camera is used to obtain the driver's head video, and the millimeter wave radar is used to capture the driver's breathing information. Step 2. Preprocessing the head video obtained in step 1, wherein the preprocessing includes at least one of denoising, image enhancement, image segmentation or image correction; Step 3. Perform face, eye and pupil recognition on the head video, and obtain facial fatigue parameters through software algorithm, where the facial fatigue parameters refer to eye closure time and blinking frequency; Step 4. Preprocess the respiratory information of step 1, perform signal separation on the preprocessed data, and finally complete the respiratory rate detection; Step 5. Determine the driver's fatigue state based on the facial fatigue parameters and breathing frequency.

2. The fatigue detection method of the millimeter wave radar and vision fusion according to claim 1 is characterized in that The head video obtained in step 1 is preprocessed, specifically including: Use multi-scale Retinex algorithm to eliminate the impact of incident light; The head video brightness is improved by redistributing pixel values ​​through histogram equalization, which increases the overall contrast of the head video. Remove salt and pepper noise from head videos by median filtering.

3. The fatigue detection method according to claim 1, characterized in that: The face, eye and pupil recognition is performed on the head video, specifically including: The head video is identified by using a haar operator to obtain a haar feature, and the haar feature is processed by using a cascade classifier to obtain at least one detection frame.

4. The fatigue detection method according to claim 1, characterized in that: The facial fatigue parameters are collected through software algorithms. Facial fatigue parameters refer to the eye closure time and blinking frequency, including: The driver's eyes are identified in the face image through the Haar classifier integrated in the camera, and a line is drawn along the outer boundary of the eyes to form the eye area; The driver's pupil is identified in the human eye area through the Haar classifier, and a line is drawn along the outer boundary of the pupil to form the pupil area; On the premise that the face image is recognized, the process of failing to recognize the pupil is recorded as an eye-closing event, and the duration of the eye-closing event is counted to obtain the eye closure time; The real-time blinking frequency is calculated through software algorithms; The blink frequency expression is: Among them, t1 is the start time of a detection cycle, t4 is the end time of a detection cycle, c is the total number of blinks in a cycle, and f is the blink frequency.

5. The fatigue detection method according to claim 1, characterized in that: Preprocessing the respiratory information of step 1, performing signal separation on the preprocessed data, and finally completing the respiratory frequency detection, specifically includes: The received beat frequency signal is first subjected to the range dimension FFT to determine the range-bin corresponding to the measured target; The phase information corresponding to the target is calculated by the inverse tangent function, and then the real phase of the target is obtained by phase winding; Calculate the phase difference between adjacent phases and filter out impulse noise through a sliding mean filtering algorithm; The respiratory signal is separated by an IIR bandpass filter, and finally the respiratory frequency is extracted by spectrum estimation. The expression of the respiratory frequency is: R=f R *60 Where R is the respiratory rate, f R is the frequency at the peak of the FFT spectrum.

6. The fatigue detection method according to claim 1, characterized in that: Based on the facial fatigue parameters and breathing frequency, the driver's fatigue state is determined, specifically including: When the facial fatigue parameter or the breathing rate exceeds a predetermined fatigue threshold, the driver is considered to be in a fatigue state.

Images