Multi-feature fusion fatigue driving monitoring method and system based on infrared band

Abstract

The invention discloses a multi-feature fusion fatigue driving monitoring method and system based on an infrared band. The method comprises the following steps: acquiring a face image; marking key points in the face image through a face detection model; performing eye aspect ratio calculation, eye data filtering and eye adaptive threshold correction according to the key points of the eyes to realize blink frequency calculation; calculating the aspect ratio of the mouth according to the key points of the mouth, filtering the data of the mouth, and correcting a self-adaptive threshold value to realize the judgment of yawning; pulse signals are extracted according to the key points of the face, noise reduction processing is conducted on the pulse signals, and heart rate calculation is conducted; and performing multiple-set canonical correlation analysis according to the blinking frequency, the yawning possibility and the heart rate to judge the fatigue driving degree. According to the method, three indexes including the heart rate, the blinking frequency and the yawning are fused into one parameter capable of reflecting the fatigue degree of the driver through multi-set canonical correlation analysis, and then the accuracy of fatigue driving judgment is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automobile driving control, and particularly relates to a method and system for monitoring fatigue driving based on multi-feature fusion of infrared wave bands. BACKGROUND

[0002] The existing fatigue driving monitoring system is mainly based on visual recognition or contact / non-contact sensors. The system based on visual recognition mainly collects the image of the face by the camera, identifies the facial features of the person through technology, and then calculates the index reflecting the fatigue degree of the driver through these facial features. The system based on contact / non-contact sensors mainly collects the physiological characteristics of the human body such as breathing rate and heart rate by sensors, and comprehensively obtains the fatigue degree of the driver. However, the existing contact / non-contact sensors cannot fuse the physiological characteristics of the human body into an index, thereby constructing the relationship between the index and fatigue driving.

[0003] In addition, the existing contact / non-contact sensors all use the IPPG (Imaging Photoplethysmography) technology, obtain the image gray value of the forehead and nose of the face, separate the periodic signal from it by using Fourier transform, take the part with the strongest energy in the frequency spectrum as the frequency of the heartbeat, and thus obtain the heart rate. However, this way is not accurate after obtaining the curve of the change of the pixel gray value with time, because the change of the pixel caused by the heart rate signal is very weak and is easily disturbed by the environment. SUMMARY

[0004] The technical problem to be solved by the present application is that in order to solve the various deficiencies of the existing fatigue driving monitoring system, the present application provides a method and system for monitoring fatigue driving based on multi-feature fusion of infrared wave bands.

[0005] In order to solve the above technical problem, the embodiment of the present application provides a method for monitoring fatigue driving based on multi-feature fusion of infrared wave bands, comprising the following steps: S1: collecting the face image by a quantum dot infrared camera; S2: constructing a face detection model, and labeling the key points in the face image by the face detection model; S3: calculating the eye aspect ratio, filtering the eye data, and correcting the adaptive threshold value of the eye to realize the calculation of the blinking frequency according to the key points of the eye; S4: calculating the mouth aspect ratio, filtering the mouth data, and correcting the adaptive threshold value to realize the judgment of yawning according to the key points of the mouth; S5: extracting a pulse signal according to the key points of the face, denoising the pulse signal, and calculating a heart rate; S6: judging the degree of fatigue driving according to multiple set canonical correlation analysis and a fatigue driving classification model based on a support vector machine on the blink frequency, the possibility of yawning, and the heart rate.

[0006] Preferably, in the step S1, the quantum dot infrared camera is a quantum dot infrared camera with a response waveband of 1550 nm, and the quantum dot infrared camera is under a 1550 nm wavelength light supplement lamp for face image acquisition.

[0007] Preferably, the step S2 specifically comprises: S21: converting the image of the frame into a gray image after the quantum dot infrared camera acquires an image of a frame; S22: detecting the image using a trained Dlib model, and inputting the image into a face detection model for key point detection; S23: outputting 68 key point coordinates of the face and an image after face position labeling and key point labeling.

[0008] Preferably, the step S3 comprises: S31: calculating an eye aspect ratio according to the key points of the eyes; the eye aspect ratio EAR is calculated according to the following formula: ; Wherein, A1 and A4 are end points at the corners of the eyes; A5 and A6 are three equal points on the upper edge of the eyes; A3 and A2 are three equal points on the lower edge of the eyes. S32: performing eye filtering processing on the eye aspect ratio calculation data; the eye filtering is Kalman filtering; S33: constructing a model of an eye adaptive threshold; inputting the data after eye filtering processing into the model of the eye adaptive threshold to obtain blink frequency data; The formula of the model of the eye adaptive threshold is: ; ; ; Wherein, the threshold is the model of the eye adaptive threshold.

[0009] Preferably, the model of the eye adaptive threshold comprises a storage library; the storage library is used to store the EAR data after filtering processing.

[0010] Preferably, the step S4 comprises: S41: Calculate the mouth aspect ratio (MAR) based on key points of the mouth; the formula for calculating the mouth aspect ratio (MAR) is: ; Where P1 and P7 are the endpoints at the corners of the mouth; P3 and P5 are the highest points of the upper lip edge; P4 is the center point of the upper lip edge; P2 is the midpoint between P1 and P3, and P6 is the midpoint between P5 and P7, with P2 and P6 located at the upper lip edge; P8-P 12 The points P6-P2 are the intersections perpendicular to the edge of the lower lip; P 13 P 14 Located at the junction of the upper and lower lips when they are closed, and line segment P1P7 is longer than line segment P. 13 P 14 Length 0.8cm~1.2cm; S42: Perform mouth-view filtering on the calculated mouth aspect ratio data; the mouth-view filtering is Kalman filtering; S43: Construct a model for an adaptive threshold for the mouth; input the filtered data of the mouth into the model for the adaptive threshold for the mouth to determine whether a yawn has occurred; The mouth adaptive threshold model determines whether mouth opening and closing is speaking or yawning based on the data after mouth filtering. It also determines whether it is yawning by the number of mouth opening and closing times in one cycle and the average opening time of the mouth each time it opens in one cycle.

[0011] Preferably, step S5 includes: S51: Using the coordinates of some key points at the eyebrows, eyes, and nose, mark two rectangular areas on the forehead and cheeks, calculate the average grayscale value of all pixels within these two rectangular areas, and use it as the grayscale data for this frame of the image. S52: Use image photoplethysmography scanning technology to scan grayscale data, obtain raw signal data, and perform noise reduction processing on the raw signal data to obtain intermediate signal data; S53: Using the cross-correlation method, with the standard heart rate signal as the basis function, the frequency of the original heart rate signal is determined.

[0012] Preferably, the noise reduction process in step S52 includes: Detrending involves subtracting the fitted linear trend term from the original signal data. The sliding window method removes the mean by selecting 15 data points from the original signal data, calculating the average value of the data points within the window, and subtracting the mean of the window from each point to obtain the mean-removed data. A bandpass filter is used to block frequency components that are not within the heart rate range from passing through; Wavelet decomposition is performed using the basis functions of wavelets; The basis functions of the wavelet are: ; Where t is time.

[0013] Preferably, step S53 includes: Continuous standard heart rate signals were obtained using linear interpolation. ; For continuous standard heart rate signals Time shifting is performed to align the phase of the continuous standard heart rate signal with the sampled signal, thus obtaining the time-shifted signal. : ; Take the normalized inner product of the time-shifted signal and the sampled signal: ; Select the maximum value as the inner product result after cross-correlation at that frequency; Frequency shifting of time-shifted signals: ; Obtain the inner product results at different frequencies, compare the inner product results at different periods, and the maximum inner product value corresponds to... T The T The period of the raw heart rate signal, the raw heart rate signal ; Wherein, x[n] is the sampled signal; T The period of the continuous standard heart rate signal after frequency shift; fps τ is the video frame rate; k is the time shift parameter; n is the discrete function independent variable; and τ is the time shift step size.

[0014] This invention also provides a system for multi-feature fusion-based fatigue driving monitoring in the infrared band. The system is used to execute the aforementioned method for multi-feature fusion-based fatigue driving monitoring in the infrared band, including: A supplemental lighting module is used to provide a 1550nm infrared light environment; The imaging module is used to capture images under a 1550nm wavelength fill light. The calculation module is used to annotate key points in the face image through the face detection model, and to obtain blink frequency, yawn probability and heart rate data through key points in the face image, and to determine the degree of fatigue driving based on multiset canonical correlation analysis of blink frequency, yawn probability and heart rate. The computation module includes a recognition unit, an eye unit, a mouth unit, a heart rate unit, and a fatigue determination unit. The recognition unit constructs a face detection model and annotates key points in the face image using this model. The eye unit calculates blink frequency by performing eye aspect ratio calculation, eye data filtering, and adaptive threshold correction based on key points. The mouth unit calculates mouth aspect ratio, filters mouth data, and performs adaptive threshold correction based on key points to determine yawning. The heart rate unit extracts pulse signals from key points on the face, performs noise reduction on the pulse signals, and calculates heart rate. The fatigue determination unit determines the degree of fatigue driving based on multiset canonical correlation analysis of blink frequency, yawning probability, and heart rate, and uses a fatigue driving classification model based on support vector machines.

[0015] Implementing the embodiments of the present invention has the following beneficial effects: (1) In this embodiment of the invention, multiset canonical correlation analysis is used to perform feature fusion, and the three indicators of heart rate, blinking frequency and yawning are fused into a parameter that can reflect the driver's fatigue level. Then, a fatigue driving prediction model is established by using a support vector machine based on radial basis kernel function and the fatigue driving level, thereby improving the accuracy of fatigue driving judgment.

[0016] (2) In this embodiment of the invention, the standard heart rate signal is used as a "probe" and the mathematical method of calculating the normalized inner product is used to directly find the original heart rate signal contained in the original signal data, thereby making the obtained original heart rate signal more accurate.

[0017] (3) In this embodiment of the invention, an eye-adaptive threshold model is constructed to selectively extract the EAR value and update the EAR threshold in real time. This method is less affected by individual differences and time-period fluctuations, and has stronger robustness and accuracy.

[0018] (4) The present invention uses a 1550nm wavelength supplement light and an infrared camera corresponding to the 1550nm band to collect images, which solves the problems that visible light cameras are greatly affected by ambient light and cannot work at night, and that infrared supplement lights with wavelengths of 850nm or 940nm are not safe for human eyes. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of a method for monitoring fatigue driving based on multi-feature fusion in the infrared band provided by the present invention; Figure 2 This invention provides a flowchart for obtaining key points in a face image. Figure 3 A flowchart for obtaining blink frequency provided by the present invention; Figure 4 A schematic diagram illustrating the changes in EAR values ​​at key eye points during blinking, provided by the present invention. Figure 5 A flowchart of the repository update process provided by this invention; Figure 6 This is a schematic diagram of the EAR curve and adaptive threshold curve provided by the present invention; Figure 7 The flowchart for determining yawning provided by the present invention; Figure 8 A schematic diagram of key points of the mouth provided by the present invention; Figure 9 The flowchart for obtaining heart rate provided by this invention; Figure 10 The flowchart for determining the degree of driver fatigue provided by this invention; Figure 11 The present invention provides a system structure diagram for fatigue driving monitoring based on multi-feature fusion in the infrared band. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See Figure 1 This invention provides a method for fatigue driving monitoring based on multi-feature fusion in the infrared band. The method specifically includes: S1: Facial image acquisition via quantum dot infrared camera; S2: Construct a face detection model and use the face detection model to annotate key points in the face image; S3: Based on key points of the eye, calculate the aspect ratio of the eye, filter eye data, and perform adaptive threshold correction to calculate the blink frequency; S4: Calculate the aspect ratio of the mouth based on key points, filter mouth data, and perform adaptive threshold correction to determine yawning. S5: Extract pulse signals based on key facial points, perform noise reduction on the pulse signals, and calculate heart rate; S6: Determine the degree of fatigue driving based on multiset canonical correlation analysis of blinking frequency, yawning probability, and heart rate, and a fatigue driving classification model based on support vector machine.

[0023] In step S1, the quantum dot infrared camera is a quantum dot infrared camera with a response wavelength of 1550nm, and the quantum dot infrared camera is used to acquire face images under a 1550nm wavelength supplementary light.

[0024] This invention employs multiset canonical correlation analysis (MCAA) for feature fusion, combining heart rate, blink rate, and yawning frequency into a single parameter reflecting driver fatigue levels. A fatigue prediction model is then established using a support vector machine based on radial basis function (RBF) and the fatigue level, thereby improving the accuracy of fatigue judgment. Furthermore, this invention utilizes a 1550nm wavelength supplemental lighting system alongside an infrared camera with a corresponding 1550nm wavelength for image acquisition. This addresses the issues of visible light cameras being susceptible to ambient light interference and unable to operate at night, as well as the safety concerns of 850nm or 940nm wavelength infrared supplemental lighting for the human eye.

[0025] See Figure 2 Step S2 specifically includes: S21: After the quantum dot infrared camera acquires a frame of image, the image of that frame is converted into a grayscale image; S22: Use the trained Dlib model to detect the image and feed it into the face detection model for key point detection; S23: Output the coordinates of 68 key points of the face and the image after face location and key point annotation.

[0026] In this embodiment of the invention, the face detection model used is the Dlib model. Because the infrared wavelength used is quite special, it is necessary to re-collect data to train the model.

[0027] The model training process included: acquiring a large number of volunteer faces under a 1550nm illumination using a quantum dot infrared camera with a 1550nm response band, resulting in a raw infrared face dataset of approximately 6000 images under 1550nm illumination, which was then expanded. The expanded dataset was then labeled with face locations and 68 facial landmarks, and input into the Dlib model for training, yielding a face recognition model and a facial landmark recognition model, respectively.

[0028] See Figure 3 ,Figure 4 Step S3 includes: S31: Calculate the eye's aspect ratio (EAR) based on key points of the eye area; the formula for calculating the eye's aspect ratio (EAR) is: .

[0029] The EAR signal can characterize the degree of eye opening and closing, and can thus be used to determine whether blinking has occurred. A1 and A4 are the endpoints at the corners of the eyes; A5 and A6 are the points where the upper edge of the eye is divided into three equal parts; and A3 and A2 are the points where the lower edge of the eye is divided into three equal parts.

[0030] S32: Perform eye filtering processing on the eye aspect ratio calculation data; the eye filtering is Kalman filtering.

[0031] The EAR signal is a real-time updated signal. Given the significant achievements of Kalman filtering in the tracking field and its excellent online computational performance, this invention chooses this method to implement signal filtering.

[0032] Kalman filtering is a recursive algorithm, so necessary initial parameters must be provided when using the model. These parameters include the initial state. x (0), initial error covariance P (0), process noise Q and measurement noise R This study empirically sets parameters based on experimental results: .

[0033] S33: Construct a model for an eye-adaptive threshold; input the filtered data into the model for the eye-adaptive threshold to obtain blink frequency data; The model formula for the eye adaptive threshold is: ; ; .

[0034] Among them, the thresholdThis is a model for an eye-adaptive threshold. This model overcomes the problems of high false positive rates and weak adaptability of EAR values ​​to individual and time-varying variations when using a fixed threshold to determine eye closure. It includes a storage repository. This repository stores filtered EAR data. x represents the sum of the 5 most recent EAR values ​​in the repository, and y represents the sum of the 15 most recent EAR values ​​in the repository, i.e., the threshold is only related to the 15 most recent EAR values ​​in the repository. Considering that EAR values ​​closer to the blink waveform are more important, a coefficient β is designed. To avoid detecting excessive noise and reduce false positives, a weight α is applied to reduce the threshold amplitude.

[0035] See Figure 5 The repository selectively stores EAR values. count The EAR (Earning Frame) is the number of frames that are consecutively less than a threshold. Let N represent the minimum number of frames that the EAR is consecutively less than the threshold during a blink. The blink waveform will be identified. Since the threshold is an estimate of the EAR value under normal conditions, the EAR value during the blink process should not be included in the threshold calculation. Therefore, the repository should stop updating when a possible blink occurs. When encountering a long blink waveform, the threshold should not remain constant throughout the blink, as the EAR when the eye reopens may be less than the threshold. To minimize the occurrence of such situations, a parameter M is used to limit the threshold when... At this time, the repository and threshold continue to be updated, and the threshold decreases due to the smaller EAR involved in the calculation.

[0036] like Figure 6 As shown, the adaptive threshold can still track the signal well even when the signal curve fluctuates, and can effectively detect blink waveforms. According to the threshold model formula, the proposed adaptive threshold is only related to the most recent EAR value, so it is not only unaffected by individual differences and time-based fluctuations, but also robust to facial expressions.

[0037] See Figure 7 , Figure 8 Step S4 includes: S41: Calculate the mouth aspect ratio (MAR) based on key points of the mouth; the formula for calculating the mouth aspect ratio (MAR) is: .

[0038] Where P1 and P7 are the endpoints at the corners of the mouth; P3 and P5 are the highest points of the upper lip edge; P4 is the center point of the upper lip edge; P2 is the midpoint between P1 and P3, and P6 is the midpoint between P5 and P7, with P2 and P6 located at the upper lip edge; P8-P 12The points P6-P2 are the intersections perpendicular to the edge of the lower lip; P 13 P 14 Located at the junction of the upper and lower lips when they are closed, and line segment P1P7 is longer than line segment P. 13 P 14 Length 0.8cm~1.2cm.

[0039] S42: Perform mouth-view filtering on the calculated mouth aspect ratio data; the mouth-view filtering is Kalman filtering.

[0040] The mouth filtering and eye filtering processes are processed in the same way.

[0041] S43: Construct a model for mouth adaptive threshold; input the filtered mouth data into the model for mouth adaptive threshold to determine whether yawning has occurred.

[0042] The changes in mouth MAR (mouth irritation and yawning) are mainly caused by two situations: speaking and yawning. The mouth opening time and degree of opening differ significantly between these two situations. Generally, when speaking, the mouth opens and closes only slightly, with MAR increasing and decreasing repeatedly, a continuous process of opening and closing. In contrast, the mouth changes during yawning are more uniform, generally with a larger opening and closing degree and a longer duration for each MAR change. Therefore, the mouth adaptive threshold model determines whether mouth opening and closing constitutes speaking or yawning based on the filtered mouth data, and further determines whether it is a yawn by the number of mouth opening and closing cycles within a period and the average opening time of the mouth during each opening within that period. In this embodiment, the period is 10 seconds. In other embodiments, this period can be changed according to different circumstances.

[0043] See Figure 9 Step S5 includes: S51: Using the coordinates of some key points at the eyebrows, eyes, and nose, mark two rectangular areas on the forehead and cheeks, and calculate the average grayscale value of all pixels within these two rectangular areas as the grayscale data for this frame of the image.

[0044] Since the forehead and cheeks are less affected by these periodic signals, these two areas were selected as Regions of Interest (ROIs) for extracting heart rate signals. Using 68 keypoints obtained from face recognition, and the coordinates of some keypoints at the eyebrows, eyes, and nose, two rectangular regions—the forehead and cheeks—were marked. The average grayscale value of all pixels within these two rectangular regions was calculated as the ROI grayscale value for that frame of the image.

[0045] S52: Use image photoplethysmography (IPPG) to scan grayscale data to obtain raw signal data, and perform noise reduction on the raw signal data to obtain intermediate signal data.

[0046] IPPG detects blood flow fluctuations by capturing changes in light intensity on the skin surface using a camera. The signal consists of a DC component, composed of ambient light, static reflections from the skin tissue, and device noise. An AC component, generated by periodic changes in blood flow caused by the heartbeat, has a smaller amplitude, typically only 1%-5% of the DC component. Therefore, the AC component in the IPPG signal is easily overwhelmed by noise from the DC component, requiring preprocessing to separate the effective information.

[0047] The noise reduction process in step S52 includes: Detrending involves subtracting the fitted linear trend term from the original signal data. The sliding window method removes the mean by selecting 15 data points from the original signal data, calculating the average value of the data points within the window, and subtracting the mean of the window from each point to obtain the mean-removed data. A bandpass filter is used to block frequency components that are not within the heart rate range from passing through; Wavelet decomposition is performed using the basis functions of wavelets; The basis functions of the wavelet are: .

[0048] Where t is time.

[0049] S53: Using the cross-correlation method, with the standard heart rate signal as the basis function, the frequency of the original heart rate signal is determined.

[0050] Step S53 includes: Continuous standard heart rate signals were obtained using linear interpolation. ; For continuous standard heart rate signals Time shifting is performed to align the phase of the continuous standard heart rate signal with the sampled signal, thus obtaining the time-shifted signal. : ; Take the normalized inner product of the time-shifted signal and the sampled signal: ; Select the maximum value as the inner product result after cross-correlation at that frequency; Frequency shifting of time-shifted signals: ; Obtain the inner product results at different frequencies, compare the inner product results at different periods, and the maximum inner product value corresponds to... TThe T The period of the raw heart rate signal, the raw heart rate signal .

[0051] Wherein, x[n] is the sampled signal. T This is the period of the continuous standard heart rate signal after frequency shifting. fps τ is the video frame rate. k is the time-shift parameter, and n is the independent variable of the discrete function. τ is the time-shift step size. The number of values ​​for k and n is the same as the number of samples. For different time-shift parameters k, the time-shifted signal... f k The values ​​of [n] also differ, resulting in different calculations. The largest value among these results is selected as the inner product result after cross-correlation at that frequency. This inner product result will then be compared with those at other frequencies to calculate the frequency.

[0052] In this embodiment of the invention, the period of the continuous standard heart rate signal obtained by linear interpolation is 1 s, and τ is 0.01 s. In other embodiments, adjustments can be made as needed.

[0053] In the normalized inner product of the time-shifted signal and the sampled signal, since the sampled data x[n] is a discrete signal, the time-shifted signal in the inner product should also be a discrete signal. Therefore, the time-shifted signal is directly derived from... The time-frequency shifted discrete signal obtained from sampling is the same as the length of the sampled data x[n].

[0054] This invention uses a standard heart rate signal as a "probe" and employs a mathematical method of calculating the normalized inner product to directly find the original heart rate signal contained in the original signal data, thereby making the obtained original heart rate signal more accurate.

[0055] See Figure 10 Step S6 consists of multiset canonical correlation analysis (MCAA) calculation and a fatigue driving classification model based on support vector machines. MCAA is used to fuse feature layers into a single index, while the support vector machine-based fatigue driving classification model is used to construct the relationship between this index and fatigue driving. Ultimately, the degree of fatigue driving is displayed through blink frequency, yawning probability, and heart rate.

[0056] See Figure 11 The present invention also provides a system for fatigue driving monitoring based on infrared band multi-feature fusion. The system for fatigue driving monitoring based on infrared band multi-feature fusion is used to execute the above-mentioned method for fatigue driving monitoring based on infrared band multi-feature fusion, and includes: a supplementary lighting module 10, an imaging module 20 and a computing module.

[0057] The supplementary lighting module 10 is used to provide a 1550nm infrared light environment. The imaging module 20 is used to capture images under a 1550nm wavelength supplementary light. The computing module is used to annotate key points in the face image through a face detection model, and to obtain blink frequency, yawn probability, and heart rate data through the key points in the face image. Based on multiset canonical correlation analysis of blink frequency, yawn probability, and heart rate, the module determines the degree of driver fatigue.

[0058] The computation module includes a recognition unit 310, an eye unit 320, a mouth unit 330, a heart rate unit 340, and a fatigue determination unit 350. The recognition unit 310 constructs a face detection model and annotates key points in the face image using this model. The eye unit 320 calculates the blink frequency based on key points in the eyes, performs eye data filtering, and applies adaptive threshold correction. The mouth unit 330 calculates the mouth aspect ratio based on key points in the mouth, performs mouth data filtering, and applies adaptive threshold correction to determine yawning. The heart rate unit 340 extracts pulse signals from key points in the face, performs noise reduction on the pulse signals, and calculates the heart rate. The fatigue determination unit 350 determines the degree of fatigue driving based on multiset canonical correlation analysis of blink frequency, yawning probability, and heart rate, and uses a fatigue driving classification model based on support vector machines.

[0059] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for fatigue driving monitoring based on multi-feature fusion in the infrared band, characterized in that, Includes the following steps: S1: Facial image acquisition via quantum dot infrared camera; S2: Construct a face detection model and use the face detection model to annotate key points in the face image; S3: Based on key points of the eye, calculate the aspect ratio of the eye, filter eye data, and perform adaptive threshold correction to calculate the blink frequency; S4: Calculate the aspect ratio of the mouth based on key points, filter mouth data, and perform adaptive threshold correction to determine yawning. S5: Extract pulse signals based on key facial points, perform noise reduction on the pulse signals, and calculate heart rate; S6: Determine the degree of fatigue driving based on multiset canonical correlation analysis of blinking frequency, yawning probability, and heart rate, and a fatigue driving classification model based on support vector machine.

2. The method for fatigue driving monitoring based on multi-feature fusion in the infrared band according to claim 1, characterized in that, In step S1, the quantum dot infrared camera is a quantum dot infrared camera with a response band of 1550nm, and the quantum dot infrared camera is used to acquire face images under a 1550nm wavelength supplementary light.

3. The method for fatigue driving monitoring based on multi-feature fusion in the infrared band according to claim 1, characterized in that, Step S2 specifically includes: S21: After the quantum dot infrared camera acquires a frame of image, the image of that frame is converted into a grayscale image; S22: Use the trained Dlib model to detect the image and feed it into the face detection model for key point detection; S23: Output the coordinates of 68 key points of the face and the image after face location and key point annotation.

4. The method for fatigue driving monitoring based on multi-feature fusion in the infrared band according to claim 1, characterized in that, Step S3 includes: S31: Calculate the eye's aspect ratio (EAR) based on key points of the eye area; the formula for calculating the eye's aspect ratio (EAR) is: ； Wherein, A1 and A4 are the endpoints at the corners of the eyes; A5 and A6 are the points where the upper edge of the eye is divided into three equal parts; A3 and A2 are the points where the lower edge of the eye is divided into three equal parts. S32: Perform eye filtering processing on the eye aspect ratio calculation data; the eye filtering is Kalman filtering; S33: Construct a model for an eye-adaptive threshold; input the filtered data into the model for the eye-adaptive threshold to obtain blink frequency data; The model formula for the eye adaptive threshold is: ； ； ； Among them, the threshold This is a model for eye-adaptive thresholding.

5. The method for fatigue driving monitoring based on multi-feature fusion in the infrared band according to claim 4, characterized in that, The model for the eye-adaptive threshold includes a repository; the repository is used to store filtered EAR data.

6. The method for fatigue driving monitoring based on multi-feature fusion in the infrared band according to claim 1, characterized in that, Step S4 includes: S41: Calculate the mouth aspect ratio (MAR) based on key points of the mouth; the formula for calculating the mouth aspect ratio (MAR) is: ； Where P1 and P7 are the endpoints at the corners of the mouth; P3 and P5 are the highest points of the upper lip edge; P4 is the center point of the upper lip edge; P2 is the midpoint between P1 and P3, and P6 is the midpoint between P5 and P7, with P2 and P6 located at the upper lip edge; P8-P 12 The points P6-P2 are the intersections perpendicular to the edge of the lower lip; P 13 P 14 Located at the junction of the upper and lower lips when they are closed, and line segment P1P7 is longer than line segment P. 13 P 14 Length 0.8cm~1.2cm; S42: Perform mouth-view filtering on the calculated mouth aspect ratio data; the mouth-view filtering is Kalman filtering; S43: Construct a model for an adaptive threshold for the mouth; input the filtered data of the mouth into the model for the adaptive threshold for the mouth to determine whether a yawn has occurred; The mouth adaptive threshold model determines whether mouth opening and closing is speaking or yawning based on the data after mouth filtering. It also determines whether it is yawning by the number of mouth opening and closing times in one cycle and the average opening time of the mouth each time it opens in one cycle.

7. The method for fatigue driving monitoring based on multi-feature fusion in the infrared band according to claim 1, characterized in that, Step S5 includes: S51: Using the coordinates of some key points at the eyebrows, eyes, and nose, mark two rectangular areas on the forehead and cheeks, calculate the average grayscale value of all pixels within these two rectangular areas, and use it as the grayscale data for this frame of the image. S52: Use image photoplethysmography scanning technology to scan grayscale data, obtain raw signal data, and perform noise reduction processing on the raw signal data to obtain intermediate signal data; S53: Using the cross-correlation method, with the standard heart rate signal as the basis function, the frequency of the original heart rate signal is determined.

8. The method for fatigue driving monitoring based on multi-feature fusion in the infrared band according to claim 7, characterized in that, The noise reduction process in step S52 includes: Detrending involves subtracting the fitted linear trend term from the original signal data. The sliding window method removes the mean by selecting 15 data points from the original signal data, calculating the average value of the data points within the window, and subtracting the mean of the window from each point to obtain the mean-removed data. A bandpass filter is used to block frequency components that are not within the heart rate range from passing through; Wavelet decomposition is performed using the basis functions of wavelets; The basis functions of the wavelet are: ； Where t is time.

9. The method for fatigue driving monitoring based on multi-feature fusion in the infrared band according to claim 7, characterized in that, Step S53 includes: Continuous standard heart rate signals were obtained using linear interpolation. ; For continuous standard heart rate signals Time shifting is performed to align the phase of the continuous standard heart rate signal with the sampled signal, thus obtaining the time-shifted signal. : ； Take the normalized inner product of the time-shifted signal and the sampled signal: ； Select the maximum value as the inner product result after cross-correlation at that frequency; Frequency shifting of time-shifted signals: ； Obtain the inner product results at different frequencies, compare the inner product results at different periods, and the maximum inner product value corresponds to... T The T The period of the raw heart rate signal, the raw heart rate signal ; Wherein, x[n] is the sampled signal; T The period of the continuous standard heart rate signal after frequency shift; fps τ is the video frame rate; k is the time shift parameter; n is the discrete function independent variable; and τ is the time shift step size.

10. A system for monitoring fatigue driving based on multi-feature fusion in the infrared band, characterized in that, The system for multi-feature fusion fatigue driving monitoring based on the infrared band is used to execute any one of the methods for multi-feature fusion fatigue driving monitoring based on the infrared band according to claims 1-9, including: A supplemental lighting module is used to provide a 1550nm infrared light environment; The imaging module is used to capture images under a 1550nm wavelength fill light. The calculation module is used to annotate key points in the face image through the face detection model, and to obtain blink frequency, yawn probability and heart rate data through key points in the face image, and to determine the degree of fatigue driving based on multiset canonical correlation analysis of blink frequency, yawn probability and heart rate. The computation module includes a recognition unit, an eye unit, a mouth unit, a heart rate unit, and a fatigue determination unit. The recognition unit constructs a face detection model and annotates key points in the face image using this model. The eye unit calculates blink frequency by performing eye aspect ratio calculation, eye data filtering, and adaptive threshold correction based on key points. The mouth unit calculates mouth aspect ratio, filters mouth data, and performs adaptive threshold correction based on key points to determine yawning. The heart rate unit extracts pulse signals from key points on the face, performs noise reduction on the pulse signals, and calculates heart rate. The fatigue determination unit determines the degree of fatigue driving based on multiset canonical correlation analysis of blink frequency, yawning probability, and heart rate, and uses a fatigue driving classification model based on support vector machines.

Images