A multi-modal fatigue driving detection method based on thermal imaging

Abstract

The application discloses a kind of multi-modal fatigue driving detection methods based on thermal imaging, comprising: using thermal imaging equipment to obtain driver face image data, using YOLO v8 model to carry out face and key point detection, extract the region of interest at forehead and nose, and calculate the mean of image pixel gray level to obtain heartbeat and respiratory signal;Signal processing is carried out to the physiological signal obtained, and the physiological characteristics related to fatigue are extracted, and fatigue and non-fatigue data set is constructed to train fatigue detection model;The thermal imaging image of fatigue and non-fatigue is extracted by face key point, and a two-classification model is trained to detect fatigue;The decision fusion of multi-model is carried out to face fatigue feature model and physiological characteristic model, and the fatigue state of driver is judged using multi-modal synthesis.The application improves the accuracy and robustness of fatigue driving detection model by the decision fusion of multi-model, can adapt to more complex light environment and individual difference, effectively improves the precision of fatigue driving recognition.

Description

Technical Field

[0001] This invention relates to the field of fatigue driving detection technology, and in particular to a multimodal fatigue driving detection method based on thermal imaging. Background Technology

[0002] With traffic safety issues becoming increasingly serious, fatigued driving has become one of the leading causes of traffic accidents worldwide. To improve road safety, various fatigue driving detection technologies and products have emerged. Currently, most fatigue driving detection products on the market use facial fatigue feature recognition methods based on computer vision technology. These methods analyze the driver's facial features to determine if the driver is fatigued. This approach uses a regular optical camera to capture facial images of the driver, then combines computer vision and deep learning to assess driver fatigue.

[0003] Fatigue driving detection methods based on facial feature extraction can be mainly divided into two categories: feature extraction methods based on the shape of facial features and local facial feature extraction methods based on deep learning.

[0004] Feature extraction method based on facial features

[0005] This method primarily uses a face recognition model or keypoint detection model to detect the driver's face and label multiple key points on the face. Based on this keypoint information, the system can further calculate the opening and closing of the eyes and mouth, as well as the angles of head posture, such as the EAR value of the eyes (eye opening and closing), the MAR value of the mouth (mouth opening and closing), and the pitch, yaw, and roll of the head (head posture angles). These facial features are used to determine whether the driver is fatigued. The advantages of this method are its simplicity, low computational requirements, and ease of implementation. However, it also has some significant limitations:

[0006] The method is greatly affected by the lighting environment. In particular, the recognition accuracy is low and the recognition effect is poor in environments such as low light, backlight or glare.

[0007] Issues with missing facial features: If the driver wears sunglasses, a mask, hair accessories, or other obstructions, their facial features may be obscured, making it impossible to accurately identify fatigue levels.

[0008] Capturing dynamic facial changes is difficult: During actual driving, the driver's facial expressions and postures are constantly changing. Therefore, the positioning of key facial points is unstable, which may lead to inaccurate calculation of EAR and MAR values ​​for the eyes and mouth.

[0009] Individual differences: The size of a driver's eyes and mouth varies considerably. If a uniform threshold is used to judge fatigue, the results may be inconsistent among different individuals, resulting in poor generalization ability.

[0010] Deep learning-based local facial feature extraction methods

[0011] Another approach is based on deep learning-based local feature extraction. This method uses deep learning algorithms such as convolutional neural networks to extract local features of the eyes and mouth from facial images. The specific process includes: locating the eye and mouth regions using facial landmarks or other facial features; then feeding the image information of these local regions into a trained classifier to detect the opening and closing states of the eyes and mouth. Common indicators include the frequency of yawning and blinking (such as the PERCLOS value), which are often used to determine driver fatigue. The advantage of this method is that it does not rely on facial landmarks and can extract effective features under complex lighting conditions, resulting in high recognition accuracy. However, this method also has the following drawbacks:

[0012] Incomplete datasets: Most methods simply label the state of the eyes and mouth as open, closed, open mouth, or closed mouth, lacking the support of facial data with multiple sample features, resulting in weak generalization ability of the trained models.

[0013] The classifier training has high requirements: This method requires a large amount of labeled data to train the classifier, and the quality of the dataset has a significant impact on the performance of the classifier. If the dataset is not comprehensive or diverse enough, the final recognition effect may be unsatisfactory.

[0014] Light environment dependence: Although this method can cope with complex lighting environments to some extent, it is still affected by strong lighting environment and factors such as missing facial features, which leads to a decrease in the success rate of fatigue recognition.

[0015] In summary, existing fatigue driving detection products based on facial features mainly rely on computer vision technology. While they can provide relatively accurate fatigue recognition results under certain conditions, their accuracy drops significantly under extreme lighting conditions (such as low light, backlight, and glare) and when drivers wear sunglasses, masks, or hair accessories that obscure facial features. Furthermore, a single facial feature often cannot comprehensively reflect a driver's fatigue state, leading to limitations in practical applications. Therefore, combining multiple physiological signals and fusing multimodal information to improve detection accuracy has become an important direction in current fatigue driving detection technology research. Summary of the Invention

[0016] This invention addresses the shortcomings of existing technologies by providing a multimodal fatigue driving detection method based on thermal imaging. It employs thermal imaging equipment combined with signal processing and computer vision technologies to acquire the driver's physiological characteristics and facial fatigue features in a non-contact manner, and utilizes multimodal information to improve the accuracy of fatigue driving detection.

[0017] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0018] A multimodal fatigue driving detection method based on thermal imaging includes the following steps:

[0019] a) Use thermal imaging equipment to acquire a dataset of thermal imaging images with the driver's face, and use this dataset to train a YOLO v8 model for detecting faces and their key point information;

[0020] b) Perform face and facial landmark detection on each frame of the video stream to obtain face bounding boxes and facial landmark information. Further extract the regions of interest on the forehead and nose using the facial landmark information. Calculate the average pixel grayscale value of the regions of interest in each frame to obtain the original heartbeat and breathing signals (time series signals). Through signal processing, finally obtain physiological signals including heartbeat and breathing signals.

[0021] c) Perform fatigue-related feature extraction processing on the physiological signals obtained in step b), extract fatigue-related physiological features, and establish fatigue and non-fatigue datasets based on these physiological features to train a multi-physiological feature-driven fatigue driving detection model, including:

[0022] A fatigue driving detection model based on heartbeat signal characteristics: This model identifies fatigue driving by analyzing the characteristics of the driver's heartbeat signal (the characteristics of heart rate variability (HRV) in the time and frequency domains, as well as the sample entropy of the heartbeat signal).

[0023] A fatigue driving detection model based on respiratory signal features: This model uses six respiratory signal features of the driver (such as respiratory rate, amplitude, amplitude standard deviation, sample entropy, etc.) to determine whether a fatigue state exists.

[0024] A fatigue driving detection model based on the fusion of heartbeat and respiratory signal features.

[0025] d) Analyze the facial fatigue features of each frame of the image, extract the thermal imaging face image dataset of fatigued and non-fatigue using facial key point information, and use the dataset to train the VGG19 binary classification model to obtain a fatigue driving detection model based on facial fatigue features. The driver is judged to be fatigued by analyzing the facial fatigue features of the driver.

[0026] e) The fatigue driving detection model based on facial fatigue features and the fatigue driving detection model driven by multiple physiological features are fused into a multi-model decision-making system to determine the driver's fatigue state using multiple modalities.

[0027] Furthermore, the process for obtaining the heartbeat signal described in step b) is as follows:

[0028] S1.1. Based on the periodic temperature changes of human skin, select a region of interest (ROI) 1, which is located on the driver's forehead. Use the average pixel grayscale values ​​of this region to form a time series as the original heartbeat signal.

[0029] S1.2. Use the YOLO v8 model to perform face detection on each frame of the thermal imaging video stream, obtain face bounding boxes and facial key point information, and determine the forehead region as the region of interest (ROI1) based on this information.

[0030] S1.3 Convert the region of interest (ROI)1 of each frame image into a grayscale image.

[0031] S1.4 Calculate the average pixel grayscale value of the region of interest (ROI1) in each frame of grayscale image, generate the time series of the heartbeat signal, and form the original heartbeat signal x1.

[0032] S1.5. Perform trend removal on the original heartbeat signal x1 using the HP filtering method to remove the trend component, resulting in the trend-removed heartbeat signal x2.

[0033] S1.6. Perform wavelet threshold denoising on the trend-removed heartbeat signal x2. Use the sym8 wavelet basis, select a fixed threshold, and decompose into 2 layers to obtain the denoised heartbeat signal x3.

[0034] S1.7. Use a bandpass filter to filter the denoised heartbeat signal x3 to obtain the final heartbeat signal x4.

[0035] Furthermore, the procedure for acquiring respiratory signals described in step b) is as follows:

[0036] S2.1. Select the nostril area as the region of interest (ROI) based on facial key point information;

[0037] S2.2. Convert the ROI2 of each frame image into a grayscale image and calculate the average grayscale value of the pixels in that region to obtain the original breathing signal y1.

[0038] S2.3. Use the Otsu thresholding algorithm to generate a binary image of the nostril region and create a mask. Only calculate the gray-scale mean of the mask region to obtain a high signal-to-noise ratio respiratory signal.

[0039] S2.4 Perform Empirical Mode Decomposition (EMD) on the original respiratory signal y1 to remove trend components and high-frequency noise, and obtain the final respiratory signal y2.

[0040] Further, step c) extracts fatigue-related physiological characteristics, specifically as follows:

[0041] Fatigue-related features were extracted from respiratory signals, including:

[0042] Respiratory cycle: calculated by measuring the time interval between adjacent peaks in the respiratory waveform;

[0043] Respiratory amplitude: calculated by the difference between the maximum and minimum values ​​of the respiratory waveform in each cycle;

[0044] Other fatigue characteristics include: respiratory rate, standard deviation of respiratory cycle, standard deviation of respiratory amplitude, and sample entropy, totaling six characteristics.

[0045] The above features were obtained through statistical analysis of 2-minute respiratory signals and used to construct dataset B, which includes two categories of data: fatigue and non-fatigue.

[0046] Extracting fatigue-related features from heart rate signals, including:

[0047] The features extracted from heart rate variability (HRV) in the time domain include heart rate (HR), heart rate interval (RR), mean of all RRs (Mean), standard deviation of all RRs (SDNN), root mean square value of the difference between all adjacent RRs (rMSSD), standard deviation of the mean of RR intervals within every 2 minutes (SDANN), and percentage of the total number of heartbeats with an interval difference of more than 50 milliseconds (pNN50).

[0048] The frequency domain feature extraction of HRV includes: high frequency (HF: the number of variations in the normal heartbeat interval in the high frequency range, referred to as high frequency), low frequency (LF: the number of variations in the normal heartbeat interval in the low frequency range, referred to as low frequency), low frequency / high frequency, and the sample entropy of the heartbeat signal.

[0049] The above features were obtained by performing time-domain and frequency-domain analysis on 2-minute heartbeat signals and used to construct dataset C, which includes two types of data: fatigue and non-fatigue.

[0050] Furthermore, in step c), the fatigue driving detection model based on respiratory signal features is trained on dataset B using a random forest (RF) model.

[0051] The fatigue driving detection model based on heartbeat signal features is trained on dataset C using an SVM model.

[0052] Furthermore, step c) also includes a fatigue driving detection model based on physiological characteristics. This model combines features from respiratory and heartbeat signals, performing feature-level fusion to concatenate the respiratory and heartbeat signal features into a new feature vector. This new feature vector is then used to construct fatigue and non-fatigue driving datasets E. Random Forest (RF) is then used to train dataset E.

[0053] Furthermore, step d) of establishing a fatigue driving detection model based on facial fatigue features specifically includes:

[0054] The YOLO v8 model is used to obtain fatigued and non-fatigue face images in each frame of thermal imaging video stream, and each frame is labeled as fatigued and non-fatigue, thus forming a fatigued and non-fatigue thermal imaging face dataset. The VGG19 image binary classification model is trained using the thermal imaging face dataset, and finally a fatigue driving detection model based on facial fatigue features is obtained.

[0055] Furthermore, step e) specifically includes:

[0056] This paper proposes a multi-model decision fusion method that integrates a facial feature-based fatigue driving detection model and a physiological feature-based fatigue driving detection model. The DS evidence theory based on the mean K coefficient method is used for multi-model decision fusion. Through decision fusion, the prediction results from the above-mentioned different models are fused to obtain a multi-modal fatigue driving detection model. This model is used to determine the driver's fatigue state and provide the driver with real-time fatigue reminders.

[0057] Compared with the prior art, the advantages of the present invention are as follows:

[0058] 1. High-precision fatigue driving detection: This invention effectively improves the accuracy and robustness of fatigue driving detection by combining facial fatigue characteristics with multiple physiological characteristics.

[0059] 2. Multimodal fusion improves recognition reliability: The multimodal decision fusion method based on DS evidence theory is adopted to fuse the model based on facial fatigue features with the model based on physiological signals. This method can comprehensively consider the fatigue information brought about by facial changes and physiological signal changes, which greatly improves the robustness and anti-interference ability of the system and reduces the misjudgment rate and overfitting risk caused by a single modality.

[0060] 3. High adaptability and unaffected by gender: The various thermal imaging datasets in this invention are derived from subjects of multiple genders, and the fatigue characteristics of facial thermal imaging are independent of gender. This makes the detection method highly versatile and adaptable to the fatigue state detection of drivers of different genders.

[0061] 4. High real-time performance, suitable for real-world driving scenarios: By using the YOLO v8 model for face and key point detection, this invention can acquire the driver's facial state information in real time, making it suitable for real-time fatigue monitoring. Combined with the physiological feature analysis of heart rate and respiratory signals, this method can detect fatigue in a short time and issue timely warnings to ensure driving safety.

[0062] 5. Low cost and non-invasive: This invention utilizes existing thermal imaging equipment and common physiological signal acquisition methods (such as heart rate and respiratory rate), requiring no additional invasive equipment or complex hardware configuration. The system's low cost and high efficiency give it significant market application potential, especially suitable for practical applications such as vehicle fatigue monitoring systems and long-distance driving scenarios. Attached Figure Description

[0063] Figure 1 This is a flowchart of the multimodal fatigue driving detection method according to an embodiment of the present invention;

[0064] Figure 2 This is an illustration of a face thermal imaging diagram according to an embodiment of the present invention;

[0065] Figure 3 This is a diagram showing the detection results of the YOLO v8 model in an embodiment of the present invention;

[0066] Figure 4 This is a schematic diagram of the region of interest in an embodiment of the present invention;

[0067] Figure 5 This is a flowchart of the heartbeat signal acquisition process according to an embodiment of the present invention;

[0068] Figure 6 This is a schematic diagram of the original heartbeat signal in an embodiment of the present invention;

[0069] Figure 7 This is a schematic diagram of the heartbeat signal after removing the trend according to an embodiment of the present invention;

[0070] Figure 8 This is a schematic diagram of the heartbeat signal after wavelet threshold processing according to an embodiment of the present invention;

[0071] Figure 9 This is a schematic diagram of a heartbeat signal according to an embodiment of the present invention;

[0072] Figure 10 This is a thermal image of exhalation in an embodiment of the present invention;

[0073] Figure 11 This is a thermal image of the inhalation process according to an embodiment of the present invention;

[0074] Figure 12 This is a flowchart of the respiratory signal acquisition process according to an embodiment of the present invention;

[0075] Figure 13This is a mask diagram of the nostrils in an embodiment of the present invention;

[0076] Figure 14 This is a schematic diagram of the original respiratory signal in an embodiment of the present invention;

[0077] Figure 15 This is a schematic diagram of EMD breakdown according to an embodiment of the present invention;

[0078] Figure 16 This is a schematic diagram of respiratory signals according to an embodiment of the present invention;

[0079] Figure 17 This is a structural diagram of respiratory signal features related to fatigue in an embodiment of the present invention;

[0080] Figure 18 This is a structural diagram of heartbeat signal characteristics related to fatigue in an embodiment of the present invention;

[0081] Figure 19 This is a schematic diagram of the feature fusion structure of an embodiment of the present invention;

[0082] Figure 20 This is a flowchart of the decision fusion process for multiple models in an embodiment of the present invention. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0084] like Figure 1 As shown, this invention provides a multimodal fatigue driving detection method based on thermal imaging, comprising the following steps:

[0085] I. Acquiring thermal imaging images with human faces

[0086] Thermal imaging images of people with faces are acquired using a thermal imaging camera. The thermal imaging camera is MAG62 from Jugo Technology, with a resolution of 640*480, an output video frame rate of 25Hz, a temperature measurement range of -20~150℃, and a temperature measurement accuracy of 0.01℃.

[0087] II. Annotating thermal images containing human faces

[0088] Annotation tools were used to annotate thermal imaging images containing faces. The annotations included a bounding box for face detection and 16 facial landmarks. A total of 5000 images were annotated, consisting of approximately 50 images of faces from different angles each from about 100 participants, forming thermal imaging image dataset A. Specific annotation details are as follows... Figure 2 As shown.

[0089] III. Training a YOLO v8 model for detecting faces and facial landmarks in thermal imaging frames.

[0090] The YOLO v8 model was trained using a pre-labeled dataset A, which consisted of 5000 thermal images with faces. The training was performed in 300 epochs, primarily to train the YOLO v8 model for face detection and 16 facial landmark detection models. The training results are as follows: Figure 3 As shown.

[0091] IV. Use the trained YOLO v8 model to detect faces and facial landmarks in frames of the video stream.

[0092] A pre-trained YOLO v8 model was used to detect faces and facial landmarks in frames of a video stream. Based on the facial landmark information, regions of interest (ROIs) were obtained at the forehead and nose. The obtained ROIs at the forehead (ROI1) are 40 pixels wide and 25 pixels high, and the obtained ROIs at the nose (ROI2) are 30 pixels wide and 15 pixels high. The ROIs are shown below. Figure 4 As shown.

[0093] V. Heartbeat Signal Acquisition Based on Thermal Imaging Video Stream

[0094] 5.1 Selection of Region of Interest

[0095] Because the human body is a natural biological infrared emitter, its surface continuously radiates infrared heat energy, primarily concentrated in the long-infrared band with wavelengths of 9.312-9.464 μm. Based on human physiology, the skin receives signals from the brain's control center to maintain a relatively constant body temperature. Due to blood flow, skin temperature undergoes relatively small fluctuations. Heat exchange exists between blood vessels and surrounding tissues, regulating skin temperature. These small temperature changes are most pronounced in superficial blood vessels. The periodic changes in skin temperature correspond to the periodic variations in heart rate. Therefore, the pixel grayscale values ​​of the corresponding regions in a thermal imaging image also change periodically. By calculating the average pixel grayscale value of the region of interest in each frame of the thermal imaging video stream, a time series of pixel grayscale values ​​can be formed, which is the original heartbeat signal. Thus, by observing skin changes in specific areas, such as the neck and forehead where heat exchange is significant, heart rate can be extracted. Since the neck area is generally not exposed in daily life, the forehead area is chosen as the region of interest (ROI1) for obtaining the heartbeat signal.

[0096] The thermal imaging camera used in this study outputs video frames at a rate of 25Hz, which means it outputs 25 frames per second. This means the sampling frequency of the heartbeat signal is 25Hz. Since the human heart beats no more than 200 times per minute, the maximum frequency of the heartbeat signal is approximately 3.334Hz, which conforms to the Nyquist sampling theorem.

[0097] 5.2 The specific heartbeat signal acquisition process is as follows: Figure 5 As shown, the details are as follows:

[0098] First, the trained YOLO v8 model is used to detect each frame in the video stream to obtain face bounding boxes and facial landmark information. The region of interest (ROI) 1 on the forehead is then obtained through the facial landmark information.

[0099] Then, the region of interest (ROI) 1 acquired from each frame of the thermal imaging video stream is converted into a grayscale image using the formula: Gd = 0.299R + 0.587G + 0.114B, where R, G, B and Gd are the red, green and blue component values ​​of the pixel before conversion and the grayscale value after conversion, respectively.

[0100] Next, the average pixel grayscale value in the grayscale image of ROI1 for each frame is calculated to obtain the original discrete heartbeat signal x1 (time series), as shown below. Figure 6 As shown.

[0101] Secondly, because the time series obtained by taking the grayscale mean from the ROI1 region contains not only periodic terms that fluctuate according to a certain pattern, but also trend components that steadily increase or decrease over time according to a certain rule, the presence of these trend components causes high-amplitude disturbances in the time series, making the series non-stationary. Since the trend term of time series x1 is not necessarily a straight line, the HP filtering method removes the trend from the original discrete heartbeat signal x1. This method uses the principle of symmetric moving average to separate data with increasing or decreasing trend terms from randomly changing time series data. The trend-removed heartbeat signal x2 is shown below. Figure 7 As shown.

[0102] Because thermal imaging video acquisition is inevitably affected by various light interferences and other noises, effective signal processing methods are needed to suppress noise. Wavelet transform can effectively distinguish between signals and noise because signal energy is usually concentrated in some large wavelet coefficients, while noise energy is distributed throughout the wavelet domain. Furthermore, wavelet decomposition provides multi-resolution analysis of signals, allowing processing at different scales, thus better capturing local signal features. By selecting appropriate wavelet bases and thresholds, effective denoising can be performed based on different types of signals and noise characteristics. A suitable threshold can retain the main features of the signal while removing noise components. Therefore, based on the characteristics of wavelet threshold denoising and the characteristics of discrete signals in this invention, after multiple experiments, the sym8 wavelet base was used, a fixed threshold was selected, and the decomposition layer was two. Wavelet threshold denoising was performed on the trend-removed heartbeat signal x2 to obtain the heartbeat signal x3. The heartbeat signal x3 after wavelet threshold denoising is shown below. Figure 8 As shown.

[0103] Finally, under normal circumstances, a person's heart rate will vary slightly under different conditions. For example, the heart rate range in a resting state is (60-100) beats / min, while the heart rate increases during exercise. Therefore, a bandpass filter with an upper cutoff frequency of 40 beats / min and a lower cutoff frequency of 200 beats / min can be selected to perform bandpass filtering on the wavelet threshold-denoised heartbeat signal x3 to obtain the final heartbeat signal x4. The heartbeat signal x4 is as follows: Figure 9 As shown.

[0104] VI. Acquisition of Respiratory Signals Based on Thermal Imaging

[0105] 6.1 Selection of Region of Interest

[0106] Human respiration is the process of gas exchange between the body and the external environment. When a person inhales or exhales, the airflow entering or exiting the nose causes changes in the temperature inside the nasal cavity. Specifically, during inhalation, as air enters the body through the nasal cavity, the temperature inside the nasal cavity drops slightly. This is because outside air is usually cooler than body temperature, and when it enters the nasal cavity, it absorbs some heat, causing the temperature inside the nasal cavity to decrease. At the same time, the mucous membranes and blood vessels in the nasal cavity also warm the inhaled air, bringing it closer to body temperature. During exhalation, as air is expelled from the lungs, the temperature inside the nasal cavity rises slightly. This is because exhaled air contains moisture and other substances, which release heat during expulsion, raising the temperature inside the nasal cavity. Furthermore, the mucous membranes and blood vessels in the nasal cavity also cool the exhaled air to maintain the normal function of the nasal cavity. Figure 10 , 11As shown, the periodic temperature changes around the nasal cavity caused by the breathing process create an opportunity for thermal imaging equipment to measure the respiratory rate. Therefore, the nostril area is selected as the region of interest (ROI2). The respiratory signal is extracted by utilizing the physical phenomenon that the periodic temperature changes of the exhaled gas near the nostrils caused by the airflow passing through the nasal cavity during human breathing, which in turn causes periodic changes in the grayscale value of the thermal imaging image.

[0107] 6.2 The specific respiratory signal acquisition process is as follows: Figure 12 As shown, the details are as follows:

[0108] First, the trained YOLO v8 model is used to detect each frame in the video stream to obtain face bounding boxes and facial landmark information. Then, the region of interest (ROI) at the nose is obtained using the facial landmark information.

[0109] Then, the region of interest (ROI)2 acquired from each frame of the thermal imaging video stream is converted into a grayscale image using the formula: Gd = 0.299R + 0.587G + 0.114B, where R, G, B, and Gd are the red, green, and blue component values ​​of the pixel before conversion and the grayscale value after conversion, respectively.

[0110] Next, the average pixel grayscale value in the grayscale image of ROI2 for each frame is calculated to obtain the original discrete breathing signal y1 (time series).

[0111] By tracking the region of interest (ROI), the average grayscale value in each frame can be obtained. In this case, ROI2 contains a large area surrounding the nose. If the average grayscale value of the pixels in ROI2 is directly calculated by averaging this area representing the breathing signal, the calculated average grayscale value will include most of the areas with constant grayscale, thus incorporating noise. Therefore, to improve the signal-to-noise ratio of the breathing signal y1, a mask focusing on the significantly changing areas of the ROI is needed when calculating the average grayscale value. To find this mask, the first step is to accumulate 200 thermal imaging video frames containing both inhalation and exhalation. Then, using the Otsu thresholding algorithm (an automatic image thresholding method), a threshold is found in the grayscale image with the largest variance in the histogram. A binary image of the nostril region is created, and a mask is built using this image. The mask region is then segmented, and the pixel grayscale values ​​are calculated for the segmented mask region. The mask is shown below. Figure 13 As shown.

[0112] A mask is obtained by performing Otsu thresholding on 200 frames of video. Then, only the average pixel grayscale value of the mask area is calculated to finally obtain the original breathing signal y1, as shown below. Figure 14 As shown.

[0113] Empirical Mode Decomposition (EMD), as a fundamental time-frequency domain processing method, has the significant advantage of directly analyzing unknown signals without requiring detailed analysis or research. It can perform hierarchical division based on the information's inherent time scale without human intervention, making it theoretically well-suited for decomposing nonlinear and non-stationary signals. Therefore, the EMD algorithm has a unique advantage in processing non-stationary and nonlinear data. In this invention, the original signal acquired by the thermal imaging camera is a complex signal that is nonlinear, irregular, non-stationary, and contains various noises. Therefore, the EMD algorithm can effectively decompose and reconstruct it, removing high-frequency signals to obtain low-frequency breathing signals. The results of EMD decomposition of the original breathing signal y1 are as follows: Figure 15 As shown.

[0114] Finally, the signal after EMD decomposition is filtered, removing the trend component, high-frequency components, and low-frequency components. The remaining components are then reconstructed to obtain the respiratory signal y2. The respiratory signal y2 is as follows: Figure 16 As shown.

[0115] VII. Extraction of Fatigue-Related Heartbeat and Respiratory Signal Features

[0116] Studies have shown that during the transition from alertness to fatigue while driving, a driver's breathing undergoes certain changes, manifested in a gradual increase in respiratory amplitude and a gradual increase in respiratory cycle length. Furthermore, the fluctuation range of respiratory cycle and amplitude is larger in fatigued driving, while breathing is more stable and smooth when the driver is alert. Therefore, the time interval between two adjacent peaks in the respiratory waveform is defined as the respiratory waveform period, and the difference between the maximum and minimum values ​​within each period is defined as the respiratory amplitude. The respiratory cycle and amplitude can be accurately extracted from the respiratory signal. Statistical analysis of the mean period and mean amplitude of the respiratory signal is used to study the respiratory characteristics of drivers under fatigue. Other fatigue characteristics include: respiratory rate, standard deviation of respiratory cycle, standard deviation of respiratory amplitude, and sample entropy, totaling six features. Dataset B is constructed using the selected physiological features. The data in the dataset are divided into fatigued and non-fatigue categories. Each data point is constructed using features obtained from statistical analysis of a 2-minute respiratory signal. Specific features are as follows: Figure 17 As shown.

[0117] Regarding the extraction of fatigue-related heartbeat signals, early research has focused on extracting temporal and frequency domain features of 2-minute heartbeat signals based on heart rate variability (HRV). HRV features are then used to determine if a driver is fatigued. The temporal features of HRV primarily extract heart rate (HR), heart rate interval (RR), the mean of all RRs (Mean), the standard deviation of all RRs (SDNN), the root mean square of the differences between all adjacent RRs (rMSSD), the standard deviation of the mean RR interval over every 2 minutes (SDANN), and the percentage of heartbeats with an interval difference exceeding 50 milliseconds (pNN50). In the frequency domain, the main features extracted are high-frequency (HF: the variability of normal heartbeat intervals within the high-frequency range), low-frequency (LF: the variability of normal heartbeat intervals within the low-frequency range), low-frequency / high-frequency, and the sample entropy of the heartbeat signal. Dataset C is constructed using the physiological characteristics selected above. The data in this dataset is divided into two categories: fatigued and non-fatigue. Each data point is constructed using features derived from the analysis of a 2-minute heartbeat signal. Specific features are as follows: Figure 18 As shown.

[0118] VIII. Establishment of a Fatigue Driving Detection Model

[0119] 8.1 Fatigue Driving Detection Model Based on Facial Fatigue Features

[0120] Because facial features appear significantly different in thermal imaging when the human body is fatigued (due to lack of sleep) versus awake (due to sufficient sleep), this difference manifests in infrared thermograms as asymmetrical or uneven temperature distribution on the forehead (either unilaterally or bilaterally), and a larger heated area around the eyes compared to the normal state. Furthermore, the facial infrared thermal imaging of sleep deprivation is gender-independent. Therefore, thermal imaging equipment can be used to acquire facial images of drivers in different states, thereby establishing a dataset D of fatigued and non-fatigued facial images. Data in dataset D is derived from virtual driving experiments involving 25 participants: 1 hour of awake driving and 1.5 hours of fatigued driving; 15 participants were male and 10 were female.

[0121] The VGG19 convolutional neural network model was trained on dataset D and validated on the validation set, achieving an accuracy of 78.67%. Observation of the facial thermal imaging images in the dataset revealed that the current thermal imager outputs images with a resolution of 640*480. Due to the low resolution, the images contain limited texture information, thus failing to fully utilize the feature extraction capabilities of the VGG19 convolutional network. Therefore, adaptive histogram equalization was performed on images from both fatigued and non-fatigue face datasets to enhance image details, allowing the VGG19 network to capture more facial features and improve the final classification accuracy. After adaptive histogram equalization, the VGG19 model achieved an accuracy of 93.89%.

[0122] 8.2 Fatigue Driving Detection Model Based on Respiratory Signal Features

[0123] Dataset B, constructed using physiological features extracted from 2-minute respiratory signals described in Section 7 (data in this dataset comes from virtual driving experiments involving 25 participants, including 1 hour of sober driving and 1.5 hours of fatigued driving; 15 males and 10 females), was then used to train a binary classification machine learning model. To reduce the risk of overfitting and improve the model's generalization ability, 5-fold cross-validation was employed during training to obtain the most suitable binary classification model. The machine learning models used in the training included Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Decision Tree (DT), and Random Forest (RF).

[0124] By performing 5-fold cross-validation training on the above four models, the training results are shown in Table 1. The accuracy and F1 score of different folds are calculated to obtain the final average accuracy and average F1 score. From the above two indicators, it can be seen that the RF model is more suitable for binary classification of respiratory features of fatigue and non-fatigue.

[0125] Table 1

[0126]

[0127] 8.3 Fatigue Driving Detection Model Based on Heartbeat Signal Features

[0128] Using the physiological features extracted from the 2-minute heartbeat signal in Section 7, dataset C was constructed (the data in this dataset all came from the virtual driving experiment data of 25 subjects, including 1 hour of sober driving and 1.5 hours of fatigued driving, 15 males and 10 females). This dataset C was used to train a binary classification model for machine learning, thereby training a model that can distinguish whether the driver is fatigued by using the features extracted from the heartbeat signal in the time domain and frequency domain.

[0129] By training four models—DT, KNN, RF, and SVM—using 5-fold cross-validation, the training results are shown in Table 2. The accuracy and F1 score of different folds were calculated to obtain the final average accuracy and average F1 score. From the above two indicators, it can be seen that the SVM model is more suitable for binary classification of heartbeat signal features of fatigue and non-fatigue.

[0130] Table 2

[0131]

[0132] 8.4 Fatigue Driving Detection Model Based on Physiological Characteristics

[0133] To improve the classification accuracy between fatigue and non-fatigue, and to enhance the robustness of the model, fatigue features related to respiratory and heart rate signals extracted from the same 2-minute thermal imaging video stream were fused at the feature level. Six features of the respiratory signal were directly concatenated with the HRV (Heart Rate Value) features of the heart rate signal in both the time and frequency domains to form a new feature vector. Each feature vector was then labeled, establishing a fatigue and non-fatigue driving dataset E. The specific feature relationships are as follows: Figure 19 As shown.

[0134] Using dataset E, four models—DT, KNN, RF, and SVM—were trained with 5-fold cross-validation, and the results are shown in Table 3. The accuracy and F1 score of different folds were calculated to obtain the final average accuracy and average F1 score. From the above two indicators, it can be seen that the RF model is more suitable for detecting fatigue driving using physiological features.

[0135] Table 2

[0136]

[0137] 8.5 Multimodal Fatigue Driving Detection Model

[0138] To improve the accuracy of fatigue driving detection, enhance the robustness of the model, reduce the error and overfitting risk of single models trained based on facial features and physiological features, and integrate the advantages of multiple modalities to improve overall prediction performance, this invention performs multi-model decision fusion on fatigue driving detection models based on facial fatigue features, respiratory signal features, heartbeat signal features, and physiological features.

[0139] This invention employs a multi-model decision fusion method based on Dempster's evidence theory. When using Dempster's evidence theory for multi-model decision-level fusion, it is typically necessary to calculate the conflict coefficient K between different model pairs. The K value measures the conflict relationship between different evidence elements; the larger K is, the greater the conflict. When K=1, the Dempster fusion rule is ineffective. The K value mainly measures the conflict between local focal elements, but it fails to consider the reliability differences between communicating focal elements, easily falling into the trap of self-conflict. Ultimately, this prevents the Dempster fusion rule from fusing information and easily produces results that defy common sense. Therefore, to address these anomalies, this invention uses the mean K coefficient method. The multi-model decision structure of Dempster's evidence theory based on the mean K coefficient method is as follows: Figure 20 As shown, the accuracy of the final multimodal fatigue driving detection model obtained through data training is 0.975.

[0140] The methods described above according to the invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be stored as software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the multimodal fatigue driving detection method described herein. Furthermore, when a general-purpose computer accesses the code used to implement the processes shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the processes shown herein.

[0141] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the implementation methods of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A multi-modal fatigue driving detection method based on thermal imaging, characterized in that, The method comprises the following steps: a) acquiring a thermal imaging image dataset with the driver's face using a thermal imaging device, and training a YOLOv8 model using the dataset for detecting human faces and key point information thereof; b) detecting human faces and human face key points in each frame of the video stream to obtain face frames and face key point information, further extracting a region of interest at the forehead and nose based on the face key point information, and obtaining original heartbeat and respiration signals by calculating the pixel gray mean value of the region of interest in each frame of the image, and finally obtaining physiological signals including heartbeat signals and respiration signals through signal processing; c) performing fatigue-related feature extraction processing on the physiological signals obtained in step b) to extract physiological features related to fatigue, and establishing a fatigue and non-fatigue dataset based on the physiological features for training a multi-physiological feature driven fatigue driving detection model, including: a fatigue driving detection model based on heartbeat signal features: recognizing fatigue driving by analyzing the heartbeat signal features of the driver; a fatigue driving detection model based on respiration signal features: using the respiration signal features of the driver to determine whether there is a fatigue state; a fatigue driving detection model based on fusion of heartbeat signal and respiration signal features; d) analyzing the face fatigue features of each frame of image, extracting a fatigue and non-fatigue thermal imaging face image dataset using the face key point information, and training a VGG19 binary classification model using the dataset to obtain a fatigue driving detection model based on face fatigue features, and determining whether the driver is fatigued by analyzing the face fatigue features of the driver; e) performing multi-model decision fusion of the fatigue driving detection model based on face features and the fatigue driving detection model based on physiological features, using the DS evidence theory based on the mean K coefficient method for multi-model decision fusion, and finally obtaining a multi-modal fatigue driving detection model by decision fusion of the prediction results from the above multiple different models, and using the model to determine the fatigue state of the driver to provide real-time fatigue reminders for the driver.

2. The multi-modal fatigue driving detection method according to claim 1, wherein the process of obtaining the heartbeat signal in step b) is as follows: S1.1, according to the periodic change of human skin temperature, selecting a region of interest ROI1, which is located at the forehead of the driver, and forming a time series by the pixel gray mean value of the region as the original heartbeat signal; S1.2, using the YOLOv8 model to detect human faces in each frame of the thermal imaging video stream to obtain face frames and face key point information, and determining the forehead region as the region of interest ROI1 based on the information; S1.3, converting the region of interest ROI1 of each frame of image into a gray scale image; S1.4, calculating the pixel gray mean value of the region of interest ROI1 in each frame of the gray scale image to generate a time series of the heartbeat signal, forming the original heartbeat signal x1; S1.5, removing the trend of the original heartbeat signal x1, using the HP filtering method to remove the trend component, and obtaining the heartbeat signal x2 after removing the trend; ​ S1.6, wavelet threshold denoising is performed on the heartbeat signal x2 after trend removal, a sym8 wavelet basis is used, a fixed threshold is selected, and the number of decomposition layers is 2, to obtain a denoised heartbeat signal x3; S1.7, a band-pass filter is used to filter the denoised heartbeat signal x3, to obtain a final heartbeat signal x4.

3. The multi-modal fatigue driving detection method of claim 1, wherein: The process of obtaining the breathing signal in step b) is as follows: S2.1, selecting a nostril region as a region of interest ROI2 based on face key point information; S2.2, converting the ROI2 of each frame of image into a gray-scale image, and calculating the pixel gray-scale mean value of the region to obtain an original breathing signal y1; S2.3, using the Otsu threshold algorithm to generate a binary image of the nostril region and create a mask, and only calculating the gray-scale mean value of the mask region to obtain a high signal-to-noise ratio breathing signal; S2.4, performing empirical mode decomposition EMD on the original breathing signal y1 to remove the trend component and high-frequency noise, to obtain a final breathing signal y2.

4. The multi-modal fatigue driving detection method of claim 1, wherein: In step c), the fatigue-related physiological features are extracted as follows: The fatigue-related features extracted from the breathing signal include: Breathing period: the time interval between adjacent peaks in the breathing waveform is calculated; Breathing amplitude: the difference between the maximum value and the minimum value of the breathing waveform in each period is calculated; The remaining fatigue features include breathing frequency, breathing period standard deviation, breathing amplitude standard deviation, and sample entropy, totaling 6 features; The above features are obtained by statistical analysis of 2-minute breathing signals, and are used to construct a data set B including fatigue and non-fatigue data; The fatigue-related features extracted from the heartbeat signal include: The time-domain features of heart rate variability HRV include heart rate HR, heartbeat interval R-R, mean value of all R-R, standard deviation of all R-R SDNN, root mean square value of all adjacent R-R differences rMSSD, and standard deviation of the average value of R-R interval every 2 minutes SDANN. The percentage of the number of adjacent R-R interval differences exceeding 50 milliseconds to the total number of heartbeats pNN50; The frequency-domain features of HRV include high frequency HF, low frequency LF, low frequency / high frequency, and sample entropy of the heartbeat signal; The above features are obtained by time-domain and frequency-domain analysis of 2-minute heartbeat signals, and are used to construct a data set C including fatigue and non-fatigue data.

5. The multi-modal fatigue driving detection method of claim 4, wherein: In step c), the fatigue driving detection model based on the breathing signal features uses a random forest RF model to train the data set B; The fatigue driving detection model based on the heartbeat signal features uses a SVM model to train the data set C.

6. The multi-modal fatigue driving detection method of claim 5, wherein: The fatigue driving detection model based on physiological characteristics in step c) combines the features of the respiratory signal and the heartbeat signal for feature-level fusion; the respiratory signal features and the heartbeat signal features are spliced into a new feature vector, and a new feature vector is used to establish a driving data set E of fatigue and non-fatigue; and the data set E is trained using a random forest RF.

7. The multi-modal fatigue driving detection method according to claim 1, characterized in that: The fatigue driving detection model based on facial fatigue features in step d) specifically comprises: YOLOv8 model is used to obtain the fatigue and non-fatigue human face images in each frame of the thermal imaging video stream, and each frame of image is labeled as fatigue and non-fatigue, thereby forming a thermal imaging image human face data set of fatigue and non-fatigue, a VGG19 image binary classification model is trained using the thermal imaging image human face data set, and finally a fatigue driving detection model based on facial fatigue features is obtained.

Images