Multi-modal fatigue driving detection method based on thermal imaging

Abstract

The invention discloses a multi-mode fatigue driving detection method based on thermal imaging, and the method comprises the steps: obtaining the facial image data of a driver through a thermal imaging device, carrying out the detection of a human face and key points through a YOLO v8 model, extracting regions of interest at the forehead and the nose, and calculating the gray average value of image pixels to obtain heartbeat and breathing signals; performing signal processing on the acquired physiological signals, extracting physiological features related to fatigue, and constructing fatigue and non-fatigue data sets for training a fatigue detection model; fatigue and non-fatigue thermal imaging images are extracted through face key points, and a binary classification model is trained for fatigue detection; and performing multi-model decision fusion on the facial fatigue feature model and the physiological feature model, and comprehensively judging the fatigue state of the driver by using multiple modes. According to the method, the accuracy and robustness of the fatigue driving detection model are improved through decision fusion of multiple models, the method can adapt to relatively complex light environments and individual differences, and the accuracy of fatigue driving recognition is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue driving detection, and in particular to a multi-modal fatigue driving detection method based on thermal imaging. Background Art

[0002] As traffic safety issues become increasingly serious, fatigue driving has become one of the main causes of traffic accidents around the world. In order to improve road safety, various fatigue driving detection technologies and products have emerged. At present, most fatigue driving detection products on the market use facial fatigue feature recognition methods based on computer vision technology, which analyze the driver's facial features to determine whether he is in a fatigued state. This method uses ordinary optical cameras to collect the driver's facial image, and then combines computer vision and deep learning to judge the driver's fatigue.

[0003] Fatigue driving detection methods based on facial feature extraction can be mainly divided into two categories: feature extraction methods based on facial features and shapes and local facial feature extraction methods based on deep learning.

[0004] Feature extraction method based on facial features

[0005] This method mainly detects the driver's face through a face recognition model or a key point detection model, and calibrates multiple key points on the face. Based on these key point information, the system can further calculate the degree of opening and closing of the eyes and mouth, as well as the angle of the head posture, such as the EAR value of the eyes (eye opening and closing), the MAR value of the mouth (mouth opening and closing), and the Pitch, Yaw and Roll of the head (head posture angle), and judge whether the driver is fatigued through these facial features. The advantages of this method are simple program implementation, low computing power requirements, and easy implementation. However, it also has the following obvious limitations:

[0006] Great influence of light environment: This method is greatly affected by lighting conditions, especially in dark, backlit or glare environments, the recognition accuracy is low and the recognition effect is poor.

[0007] Facial feature missing problem: If the driver wears sunglasses, masks, hair accessories and other obstructions, facial features may be blocked, resulting in inaccurate identification of fatigue status.

[0008] Difficulty in capturing dynamic facial changes: During actual driving, the driver's facial expressions and postures are constantly changing. Therefore, the positioning of facial key points is unstable, which may lead to inaccurate calculation of the EAR and MAR values ​​of the eyes and mouth.

[0009] Individual difference problem: The size of drivers’ eyes and mouths varies greatly. If a unified threshold is used to judge fatigue status, it may lead to inconsistent results for different individuals and poor generalization ability.

[0010] Local facial feature extraction method based on deep learning

[0011] Another method is local feature extraction based on deep learning. Through relevant deep learning algorithms such as convolutional neural networks, local features of the eyes and mouth are extracted from facial images. The specific process includes: using facial key points or other facial features to locate the eye and mouth areas, and then sending the image information of these local areas to the trained classifier to detect the opening and closing status of the eyes and mouth. Common indicators include the frequency of yawning and blinking (such as PERCLOS value), which are often used to determine whether the driver is tired. The advantage of this method is that it does not rely on facial key points, and can extract effective features under complex lighting conditions, with high recognition accuracy. However, this method also has the following shortcomings:

[0012] The dataset is not comprehensive: Most methods simply label the state of the eyes and mouth as open, closed, open or closed, and lack the support of facial data with multi-sample features, resulting in weak generalization ability of the trained model.

[0013] High requirements for classifier training: This method requires a large amount of labeled data to train the classifier, and the quality of the dataset has an important impact on the performance of the classifier. If the dataset is not comprehensive or diverse enough, the final recognition effect may not be ideal.

[0014] Light environment dependence: Although this method can cope with complex lighting environments to a certain extent, it is still affected by factors such as strong lighting environment and missing facial features, resulting in a decrease in the success rate of fatigue recognition.

[0015] In summary, existing facial feature-based fatigue driving detection products mainly rely on computer vision technology. Although they can provide relatively accurate fatigue recognition results under certain conditions, the recognition accuracy is low under extreme lighting conditions (such as dim light, backlight, glare) and when the driver is wearing sunglasses, masks, hair accessories and other facial features are missing. In addition, a single facial feature often cannot fully reflect the driver's fatigue state, resulting in certain limitations in the actual application of existing methods. Therefore, combining multiple physiological signals, fusing multimodal information, and improving detection accuracy have become an important direction for current research on fatigue driving detection technology. Summary of the invention

[0016] In view of the defects of the prior art, the present invention provides a multi-modal fatigue driving detection method based on thermal imaging. The thermal imaging device is combined with signal processing technology and computer vision technology to obtain the driver's physiological characteristics and facial fatigue characteristics through a non-contact method, and the multi-modal information is used to improve the accuracy of fatigue driving detection.

[0017] In order to achieve the above invention object, the technical solution adopted by the present invention is as follows:

[0018] A multimodal fatigue driving detection method based on thermal imaging, comprising the following steps:

[0019] a) Using thermal imaging equipment to obtain a thermal imaging image dataset with the driver's face, and using the dataset to train a YOLO v8 model for detecting faces and their key point information;

[0020] b) Performing face and face key point detection on each frame of the video stream to obtain face frame and facial key point information, further extracting the region of interest at the forehead and nose through the facial key point information, and obtaining the original heartbeat and respiratory signals (time series signals) by calculating the pixel grayscale mean of the region of interest in each frame of the image, and finally obtaining physiological signals including heartbeat signals and respiratory signals through signal processing;

[0021] c) performing fatigue-related feature extraction processing on the physiological signal obtained in step b), extracting the fatigue-related physiological features, and establishing fatigue and non-fatigue data sets based on the physiological features for training a fatigue driving detection model driven by multiple physiological features, including:

[0022] Fatigue driving detection model based on heartbeat signal characteristics: Fatigue driving is identified by analyzing the driver's heartbeat signal characteristics (characteristics of heart rate variability (HRV) in the time domain and frequency domain, and sample entropy of the heartbeat signal).

[0023] Fatigue driving detection model based on breathing signal characteristics: The driver's six breathing signal characteristics (such as breathing frequency, amplitude, amplitude standard deviation, sample entropy, etc.) are used to determine whether there is a fatigue state.

[0024] Fatigue driving detection model based on the fusion of heartbeat signal and breathing signal features.

[0025] d) Analyze the facial fatigue features of each frame image, use the facial key point information to extract the fatigue and non-fatigue thermal imaging face image dataset, and use the dataset to train the VGG19 binary classification model to obtain a fatigue driving detection model based on facial fatigue features. By analyzing the facial fatigue features of the driver, whether he is fatigued or not is determined;

[0026] e) The fatigue driving detection model based on facial fatigue features and the fatigue driving detection model driven by multiple physiological features are integrated into a multi-model decision-making process to use multiple modalities to judge the driver's fatigue state.

[0027] Furthermore, the process of obtaining the heartbeat signal in step b) is as follows:

[0028] S1.1. According to the periodic change of human skin temperature, a region of interest ROI1 is selected, which is located at the forehead of the driver. A time series is formed by the mean grayscale value of pixels in the region as the original heartbeat signal.

[0029] S1.2. Use the YOLO v8 model to perform face detection on each frame of the thermal imaging video stream, obtain the face frame and facial key point information, and based on this information, determine the forehead area as the region of interest ROI1.

[0030] S1.3, convert the region of interest ROI1 of each frame image into a grayscale image.

[0031] S1.4. Calculate the mean grayscale value of the pixel in the region of interest ROI1 in each frame of the grayscale image, generate a time series of the heartbeat signal, and form the original heartbeat signal x1.

[0032] S1.5. Detrending the original heartbeat signal x1, using the HP filtering method to remove the trend component, and obtaining the detrended heartbeat signal x2.

[0033] S1.6. Perform wavelet threshold denoising on the heartbeat signal x2 after trend removal, use the sym8 wavelet basis, select a fixed threshold, and decompose the number of layers to 2 to obtain the denoised heartbeat signal x3.

[0034] S1.7. Use a bandpass filter to filter the denoised heartbeat signal x3 to obtain a final heartbeat signal x4.

[0035] Furthermore, the process of obtaining the respiratory signal in step b) is as follows:

[0036] S2.1, selecting the nostril area as the region of interest ROI2 based on the facial key point information;

[0037] S2.2, converting the ROI2 of each frame image into a grayscale image, and calculating the mean grayscale value of the pixels in the area to obtain the original respiratory signal y1;

[0038] S2.3, using the Otsu threshold algorithm to generate a binary image of the nostril area and create a mask, only calculating the grayscale mean of the mask area to obtain a respiratory signal with a high signal-to-noise ratio;

[0039] S2.4. Perform empirical mode decomposition (EMD) on the original respiratory signal y1 to remove trend components and high-frequency noise to obtain the final respiratory signal y2.

[0040] Furthermore, in step c), physiological characteristics related to fatigue are extracted as follows:

[0041] Extract fatigue-related features from the breathing signal, including:

[0042] Respiratory cycle: by calculating the time interval between adjacent peaks in the respiratory waveform;

[0043] Respiratory amplitude: by calculating the difference between the maximum and minimum values ​​of the respiratory waveform in each cycle;

[0044] The other fatigue features are: breathing frequency, breathing cycle standard deviation, breathing amplitude standard deviation, sample entropy, a total of 6 features

[0045] The above features are obtained by statistically analyzing the 2-minute breathing signal and are used to construct dataset B, which includes fatigue and non-fatigue data.

[0046] Extract fatigue-related features from heartbeat signals, including:

[0047] The features extracted from heart rate variability (HRV) in the time domain include heart rate (HR), heart beat interval (RR), the mean value (Mean) of all RRs, the standard deviation (SDNN) of all RRs, the root mean square value of the difference between all adjacent RRs (rMSSD), the standard deviation of the mean value of RR intervals within every 2 minutes (SDANN), and the percentage of the number of adjacent RR intervals with a difference of more than 50 milliseconds in the total number of heartbeats (pNN50).

[0048] The frequency domain feature extraction of HRV includes: high frequency (HF: the variation of normal heartbeat intervals within the high frequency range, referred to as high frequency), low frequency (LF: the variation of normal heartbeat intervals within the low frequency range, referred to as low frequency), low frequency / high frequency, and sample entropy of heartbeat signals.

[0049] The above features are obtained by analyzing the 2-minute heartbeat signal in the time domain and frequency domain, and are used to construct dataset C, which includes fatigue and non-fatigue data.

[0050] Furthermore, in step c), the fatigue driving detection model based on the breathing signal feature is trained on the dataset B using the random forest (RF) model.

[0051] The fatigue driving detection model based on heartbeat signal features uses the SVM model to train the dataset C.

[0052] Furthermore, step c) also includes a fatigue driving detection model based on physiological characteristics, combining the features of the breathing signal and the heartbeat signal, performing feature-level fusion, splicing the breathing signal features and the heartbeat signal features into a new feature vector, and using the new feature vector to establish a fatigue and non-fatigue driving data set E. The data set E is trained using random forest (RF).

[0053] Furthermore, in step d), establishing a fatigue driving detection model based on facial fatigue features specifically includes:

[0054] The YOLO v8 model is used to obtain fatigue and non-fatigue facial images in each frame of thermal imaging video stream, and each frame of the image is annotated as fatigue and non-fatigue, thereby forming a fatigue and non-fatigue thermal imaging image face dataset. The thermal imaging image face dataset is used to train the VGG19 image binary classification model, and finally a fatigue driving detection model based on facial fatigue features is obtained.

[0055] Furthermore, step e) specifically includes:

[0056] The fatigue driving detection model based on facial features and the fatigue driving detection model based on physiological features are subjected to multi-model decision fusion, and the DS evidence theory based on the mean K coefficient method is used for multi-model decision fusion. Through decision fusion, the prediction results from the above multiple different models are fused, and finally a fatigue driving detection model based on multimodality is obtained. This model is used to judge the driver's fatigue state and provide the driver with real-time fatigue reminder.

[0057] Compared with the prior art, the advantages of the present invention are:

[0058] 1. High-precision fatigue driving detection: The present invention effectively improves the accuracy and robustness of fatigue driving detection by combining facial fatigue features with multiple physiological features.

[0059] 2. Multimodal fusion improves recognition reliability: A multimodal decision fusion method based on DS evidence theory is adopted to fuse the model based on facial fatigue features with the model based on physiological signals. This can comprehensively consider the fatigue information brought about by facial changes and physiological signal changes, greatly improve the system's robustness and anti-interference ability, and reduce the misjudgment rate and overfitting risk brought by a single modality.

[0060] 3. Strong adaptability and not affected by gender: The various thermal imaging data sets in the present invention are derived from subjects of multiple different genders, and the fatigue characteristics of facial thermal imaging are independent of gender, which makes the detection method highly versatile and can adapt to the fatigue status detection of drivers of different genders.

[0061] 4. Strong real-time performance, suitable for actual driving scenarios: Through the YOLO v8 model for face and key point detection, the present invention can obtain the driver's facial status information in real time, which is suitable for real-time fatigue monitoring. Combined with the physiological characteristics analysis of heartbeat and breathing signals, this method can detect fatigue in a short time and issue an alarm in time to ensure driving safety.

[0062] 5. Low cost and non-invasive: The present invention utilizes existing thermal imaging equipment and common physiological signal acquisition methods (such as heart rate and respiratory rate), without the need for additional invasive equipment or complex hardware configuration. The low cost and high efficiency of the system make it have great market application potential, especially suitable for practical applications such as vehicle fatigue monitoring systems and long-term driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a flow chart of a multi-modal fatigue driving detection method according to an embodiment of the present invention;

[0064] Figure 2 This is a face thermal imaging diagram annotation diagram according to an embodiment of the present invention;

[0065] Figure 3 This is a graph of the detection results of the yolo v8 model according to an embodiment of the present invention;

[0066] Figure 4 is a schematic diagram of a region of interest according to an embodiment of the present invention;

[0067] Figure 5 This is a flow chart of obtaining a heartbeat signal according to an embodiment of the present invention;

[0068] Figure 6 is a schematic diagram of an original heartbeat signal according to an embodiment of the present invention;

[0069] Figure 7 is a schematic diagram of a heartbeat signal after trend removal according to an embodiment of the present invention;

[0070] Figure 8 is a schematic diagram of a heartbeat signal after wavelet threshold processing according to an embodiment of the present invention;

[0071] Fig. 9 This is a schematic diagram of a heartbeat signal according to an embodiment of the present invention;

[0072] Fig.10 is a thermal image of an embodiment of the present invention during exhalation;

[0073] Fig.11 is a thermal imaging diagram of an embodiment of the present invention during inhalation;

[0074] Fig.12 is a flow chart of obtaining a respiratory signal according to an embodiment of the present invention;

[0075] Fig.13is a mask image at the nostril of an embodiment of the present invention;

[0076] Fig.14 is a schematic diagram of an original breathing signal according to an embodiment of the present invention;

[0077] Fig.15 is a schematic diagram of EMD decomposition according to an embodiment of the present invention;

[0078] Fig.16 is a schematic diagram of a breathing signal according to an embodiment of the present invention;

[0079] Fig.17 is a diagram of a characteristic structure of a respiratory signal related to fatigue according to an embodiment of the present invention;

[0080] Fig.18 is a characteristic structure diagram of a heartbeat signal related to fatigue according to an embodiment of the present invention;

[0081] Fig.19 It is a schematic diagram of the feature fusion structure of an embodiment of the present invention;

[0082] Fig. 20 It is a decision fusion flow chart of multiple models in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0084] like Figure 1 As shown, the present invention provides a multi-modal fatigue driving detection method based on thermal imaging, comprising the following steps:

[0085] 1. Get thermal imaging images with human faces

[0086] Thermal imaging images with human faces are obtained using a thermal imaging camera, which is MAG62 from JuGe Technology. The resolution is 640*480, the external video frame rate is 25Hz, the temperature measurement range is -20~150℃, and the temperature measurement accuracy is 0.01℃.

[0087] 2. Labeling Thermal Images with Faces

[0088] The thermal imaging images with faces were annotated using annotation tools, including a bounding box for face detection and 16 facial key points. The number of annotated images was 5,000, from about 100 subjects, each with about 50 images of faces from different angles, thus forming a thermal imaging image dataset A. The specific annotation content is as follows: Figure 2 shown.

[0089] 3. Training the YOLO v8 model for detecting faces and facial key points in thermal imaging frames

[0090] The YOLO v8 model is trained using the labeled dataset A, where the training dataset is 5,000 thermal imaging images with faces, and the training rounds are 300 times. The trained YOLO v8 model is mainly used to detect faces and 16 facial key point detection models. The training results are as follows: Figure 3 shown.

[0091] 4. Use the trained YOLO v8 model to detect faces and facial key points in frames in the video stream

[0092] The trained YOLO v8 model is used to detect faces and facial key points in the frames of the video stream. Based on the facial key point information, the regions of interest at the forehead and nose are obtained. The forehead region of interest (ROI1) is 40 pixels wide and 25 pixels high, and the nose region of interest (ROI2) is 30 pixels wide and 15 pixels high. Figure 4 shown.

[0093] 5. Heartbeat signal acquisition based on thermal imaging video stream

[0094] 5.1. Selection of Region of Interest

[0095] Since the human body is a natural biological infrared emission source, the surface of the human body can continuously radiate infrared heat energy to the outside world. The energy is mainly concentrated in the long infrared band with a wavelength of 9.312-9.464μm. According to the physiological characteristics of the human body, the human skin receives signals from the brain control center to maintain a relatively constant body temperature. Due to the flow of blood in the body, the temperature of the human skin will change relatively slightly. There is heat exchange between the human blood vessels and the surrounding tissues of the blood vessels. The purpose of regulating skin temperature is achieved through heat exchange. Small changes in skin temperature are most prominent in superficial blood vessels, and these temperature changes are The periodic changes consistent with the heart rate, due to the periodic changes in the skin surface temperature, the pixel grayscale value of the area corresponding to the thermal imaging image also changes periodically. Therefore, by calculating the pixel grayscale mean of the area of ​​interest in each frame of the thermal imaging video stream, a time series of pixel grayscale mean values ​​can be formed, and this time series is the original heartbeat signal; therefore, by observing the skin changes in specific areas, such as the neck, forehead and other areas with obvious heat exchange, the heart rate can be extracted; because in daily life, the neck area is generally not exposed, so the forehead area is selected as the region of interest ROI1 for obtaining the heartbeat signal.

[0096] The thermal imaging camera used in this study outputs a video frame rate of 25Hz, that is, 25 frames are output per second, that is, the sampling frequency of the heartbeat signal is 25Hz. Since the human heart beats no more than 200 times per minute, the maximum frequency of the heartbeat signal is about 3.334Hz, which conforms to the Nyquist sampling theorem.

[0097] 5.2. The specific process of obtaining heartbeat signals is as follows: Figure 5 As shown, the details are as follows:

[0098] First, the trained YOLO v8 model is used to detect each frame in the video stream to obtain the face frame and face key point information, and the region of interest ROI1 at the forehead is obtained through the face key point information.

[0099] Then, the region of interest ROI1 obtained in each frame of the thermal imaging video stream is converted into a grayscale image using the formula: Gd=0.299R+0.587G+0.114B, where R, G, B and Gd are the red, green and blue component values ​​of the pixel before conversion and the grayscale value after conversion, respectively.

[0100] Then calculate the mean grayscale value of the pixels in the grayscale image of ROI1 of each frame to obtain the original discrete heartbeat signal x1 (time series), as shown in Figure 6 shown.

[0101] Secondly, since the time series in which the grayscale mean is taken through the ROI1 area contains not only periodic items that fluctuate according to a certain rule, but also trend components that steadily increase or decrease according to a certain rule over time, the existence of the trend component causes high-amplitude disturbances in the time series, making the series non-stationary. Since the trend item of the time series x1 is not necessarily a straight line, the HP filtering method is used to remove the trend of the original discrete heartbeat signal x1. This method adopts the principle of symmetric data moving average, aiming to separate the data with increasing or decreasing trend items in the randomly changing time series data. The heartbeat signal x2 after detrending is as follows Figure 7 shown.

[0102] Since it is inevitable to be affected by various light interferences and other noises when collecting thermal imaging videos, it is necessary to adopt effective signal processing methods to suppress noise, and wavelet transform can effectively distinguish between signals and noise, because the energy of the signal is usually concentrated in some large wavelet coefficients, while the noise energy is distributed in the entire wavelet domain. In addition, wavelet decomposition provides multi-resolution analysis of the signal, allowing the signal to be processed at different scales, so as to better capture the local characteristics of the signal. By selecting appropriate wavelet basis and threshold, effective denoising can be performed according to different types of signal and noise characteristics. The appropriate threshold can retain the main characteristics of the signal while removing the noise component. Therefore, according to the characteristics of wavelet threshold denoising and the characteristics of discrete signals in the present invention, after many experiments, the sym8 wavelet basis is adopted, the fixed threshold is selected, and the decomposition layer number is 2. The heartbeat signal x2 after trend removal is subjected to wavelet threshold denoising to obtain the heartbeat signal x3. The heartbeat signal x3 after wavelet threshold denoising is as follows: Figure 8 shown.

[0103] Finally, under normal circumstances, a person's heart rate will be slightly different in different situations. For example, the heart rate range in a calm state is (60-100) times / min, and the heart rate increases in a state of exercise. Therefore, a bandpass filter with an upper cutoff frequency of 40 times / min and a lower cutoff frequency of 200 times / min can be selected to perform bandpass filtering on the heartbeat signal x3 after wavelet threshold denoising to obtain the final heartbeat signal x4. The heartbeat signal x4 is as follows: Fig. 9 shown.

[0104] 6. Acquisition of respiratory signals based on thermal imaging

[0105] 6.1. Selection of Region of Interest

[0106] Human breathing refers to the process of gas exchange between the body and the external environment. When the human body inhales or exhales, the airflow entering or flowing out of the human nose will cause changes in the temperature inside the nasal cavity. Specifically, when inhaling, when air enters the body through the nasal cavity, the temperature inside the nasal cavity will drop slightly. This is because the air outside is usually lower than body temperature. When it enters the nasal cavity, it absorbs some heat, causing the temperature inside the nasal cavity to drop; at the same time, the mucous membrane and blood vessels in the nasal cavity will also warm the inhaled air to make it close to body temperature; when exhaling, when the air is discharged from the lungs to the body, the temperature inside the nasal cavity will rise slightly. This is because the exhaled air contains moisture and other substances. These substances release heat during the exhalation process, which increases the temperature inside the nasal cavity. In addition, the mucous membrane and blood vessels in the nasal cavity will also cool the exhaled air to maintain the normal function of the nasal cavity. Such as Fig.10 , 11As shown, the periodic temperature changes around the nasal cavity caused by the breathing process create an opportunity for the thermal imaging device to measure the respiratory frequency. Therefore, the nostril area is selected as the region of interest ROI2, and the physical phenomenon that the airflow through the nasal cavity during human breathing causes the periodic changes in the temperature of the exhaled gas in the area near the nostrils, thereby causing the periodic changes in the grayscale value of the thermal imaging image, is used to extract the respiratory signal.

[0107] 6.2. Specific breathing signal acquisition process is as follows Fig.12 As shown, the details are as follows:

[0108] First, use the trained YOLO v8 model to detect each frame in the video stream to obtain the face frame and face key point information, and obtain the region of interest ROI at the nose through the face key point information.

[0109] Then, the region of interest ROI2 obtained in each frame of the thermal imaging video stream is converted into a grayscale image using the formula: Gd=0.299R+0.587G+0.114B, where R, G, B and Gd are the red, green and blue component values ​​of the pixel before conversion and the grayscale value after conversion, respectively.

[0110] Secondly, the mean grayscale value of pixels in the grayscale image of ROI2 of each frame is calculated to obtain the original discrete respiratory signal y1 (time series).

[0111] By tracking the region of interest, the average grayscale value in each frame of the image can be obtained. At this time, the ROI2 region includes a large area surrounding the nose. If the area representing the respiratory signal is directly averaged when calculating the pixel grayscale mean of the ROI2 region, the calculated grayscale mean includes most of the areas with unchanged grayscale, thereby taking the noise part into account. Therefore, in order to improve the signal-to-noise ratio of the respiratory signal y1, a mask focusing on the area with obvious changes in the region of interest is required when calculating the pixel grayscale mean. In order to find the mask, the first step is to accumulate some thermal imaging video frames, and the number of frames is set to 200 frames. These frames include inhalation and exhalation actions. Then, the Otsu threshold algorithm is used, which is an automatic image threshold method. The threshold is found from the grayscale image with the largest variance in the histogram, and a binary image of the nostril area is created. The mask is established through the binary image of the nostril area, and then the mask area is segmented. The pixel grayscale is calculated for the segmented mask area. The mask is as follows Fig.13 shown.

[0112] The mask is obtained by performing Otsu threshold processing on the 200-frame video, and then only the pixel grayscale mean of the mask area is calculated, and finally the original respiratory signal y1 is obtained, as shown in Fig.14 shown.

[0113] Since empirical mode decomposition (EMD) is a basic time-frequency domain processing method, its most prominent advantage is that it does not need to conduct more specific analysis and research on an unknown signal, but can be directly analyzed, and can make specific hierarchical divisions according to the time scale of the information itself without human intervention. The theory is more suitable for the decomposition of nonlinear and unstable signals. Therefore, the EMD algorithm has an inherent advantage in processing unstable and nonlinear data. In the present invention, the original signal collected by the thermal imaging camera is a complex signal that is nonlinear, irregular, unstable and contains various noises. Therefore, the EMD algorithm can be used to decompose and reorganize it well, remove high-frequency signals, and obtain low-frequency breathing signals. By performing EMD decomposition on the original breathing signal y1, the results are as follows: Fig.15 shown.

[0114] Finally, the signal after EMD decomposition is screened, the trend part is removed, the high-frequency part and the low-frequency part are removed, and the remaining part is reorganized to obtain the breathing signal y2. The breathing signal y2 is as follows Fig.16 shown.

[0115] 7. Extraction of heartbeat and breathing signal features related to fatigue

[0116] Studies have shown that in the process of driving from sober to fatigued, the driver's breathing changes to a certain extent, which is reflected in the gradual increase in breathing amplitude and the gradual increase in breathing cycle. In addition, the fluctuation range of breathing cycle and breathing amplitude is large in the state of fatigue driving, while when driving in a sober state, the driver's breathing is relatively stable and smooth. Therefore, the time interval between two adjacent peaks in the breathing waveform is defined as the period of the breathing waveform, and the difference between the maximum and minimum values ​​in the breathing waveform in each period is taken as the amplitude of breathing. The breathing cycle and breathing amplitude can be accurately extracted through the extracted breathing signal; the cycle mean and amplitude mean of the breathing signal are statistically analyzed to study the breathing characteristics of the driver during fatigue driving. The remaining fatigue characteristics are: breathing frequency, breathing cycle standard deviation, and breathing amplitude standard deviation, sample entropy, a total of 6 features. The above-screened physiological features are used to construct data set B. The data in the data set are divided into fatigue and non-fatigue categories. Each piece of data is constructed by the features obtained by statistical analysis of a breathing signal with a duration of 2 minutes. The specific features are as follows: Fig.17 shown.

[0117] In terms of fatigue-related heartbeat signal extraction, according to early studies, the time domain and frequency domain features of the 2-minute heartbeat signal in the heart rate variability (HRV) are extracted, and the features of HRV are used to determine whether the driver is in a fatigued state; the HRV time domain features mainly extract heart rate (HR), heartbeat interval (RR), the mean value (Mean) of all RRs, the standard deviation of all RRs (SDNN), the root mean square value of the difference between all adjacent RRs (rMSSD), the standard deviation of the mean value of RR intervals within every 2 minutes (SDANN), and the percentage of the number of adjacent RR intervals with a difference of more than 50 milliseconds in the total number of heartbeats (pNN50). The frequency domain features of HRV mainly extract high frequency (HF: the variation of normal heartbeat intervals in the high frequency range, referred to as high frequency), low frequency (LF: the variation of normal heartbeat intervals in the low frequency range, referred to as low frequency), low frequency / high frequency, and sample entropy of the heartbeat signal. The above selected physiological features are used to construct the data set C. The data in the data set are divided into two categories: fatigue and non-fatigue. Each piece of data is constructed by calculating and analyzing the features of the heartbeat signal with a duration of 2 minutes. The specific features are as follows: Fig.18 shown.

[0118] 8. Establishment of Fatigue Driving Detection Model

[0119] 8.1. Fatigue driving detection model based on facial fatigue features

[0120] Since the facial features of the human body when fatigued (lack of sleep) and awake (sufficient sleep) are greatly different in thermal imaging, this difference is reflected in the infrared thermal image as unilateral or bilateral asymmetric and uneven temperature distribution in the forehead area, and a larger hot area in the eye area than in the normal state. In addition, the infrared thermal imaging expression of the face of insufficient sleep is not related to gender, so thermal imaging equipment can be used to obtain facial images of drivers in different states, thereby establishing a fatigue and non-fatigue facial image dataset D. The data in dataset D are all from the virtual driving experiment data of 25 subjects, including 1 hour of awake driving and 1.5 hours of fatigue driving, 15 males and 10 females.

[0121] The VGG19 convolutional neural network model is used to train the dataset D. After verifying the validation set, the model accuracy reaches 78.67%. By observing the thermal imaging images of faces in the dataset, the output image resolution of the thermal imager currently used is 640*480. Due to the low resolution and less texture information in the image, the VGG19 convolutional network's ability in feature extraction has not been fully utilized. Therefore, the images in the fatigue and non-fatigue face datasets are processed by adaptive histogram equalization. The purpose is to enhance the details of the image so that the VGG19 network can obtain more facial features, thereby improving the final classification accuracy of the network. After the images in the dataset are processed by adaptive histogram equalization, the accuracy of the VGG19 model reaches 93.89%.

[0122] 8.2 Fatigue driving detection model based on breathing signal characteristics

[0123] The dataset B constructed by extracting the physiological features of the 2-minute breathing signal in Section 7 (the data of this dataset are all from the virtual driving experiment data of 25 subjects, including 1 hour of sober driving and 1.5 hours of fatigue driving, 15 males and 10 females), and then the dataset B is used to train the machine learning binary classification model. In order to reduce the risk of overfitting and improve the generalization ability of the model, 5-fold cross validation is used in the model training process to finally obtain the most suitable binary classification model; the machine learning models involved in the training mainly include support vector machine (SVM), K nearest neighbor (KNN), decision tree (DT), and random forest (RF).

[0124] By performing 5-fold cross-validation training on the above four models, the training results shown in Table 1 are obtained. The precision and F1 score of different folds are calculated to obtain the final average precision and average F1 score. From the above two indicators, it can be seen that the RF model is more suitable for binary classification of fatigue and non-fatigue breathing features.

[0125] Table 1

[0126]

[0127] 8.3. Fatigue driving detection model based on heartbeat signal characteristics

[0128] The dataset C constructed by using the physiological features extracted from the 2-minute heartbeat signal in Section 7 (the data in this dataset are all from the virtual driving experiment data of 25 subjects, including 1 hour of sober driving and 1.5 hours of fatigue driving, with 15 males and 10 females) is used to train a binary classification model for machine learning, thereby training a model that can use the features extracted from the heartbeat signal in the time domain and frequency domain to distinguish whether the driver is fatigued.

[0129] By performing 5-fold cross-validation training on the four models of DT, KNN, RF, and SVM, the training results shown in Table 2 are obtained. The accuracy and F1 score of different folds are calculated to obtain the final average accuracy and average F1 score. From the above two indicators, it can be seen that the SVM model is more suitable for binary classification of fatigue and non-fatigue heartbeat signal features.

[0130] Table 2

[0131]

[0132] 8.4. Fatigue driving detection model based on physiological characteristics

[0133] In order to improve the classification accuracy of fatigue and non-fatigue and improve the robustness of the model, the fatigue features of breathing and heartbeat signals extracted from the same 2-minute thermal imaging video stream are fused at the feature level, and the six features of the breathing signal and the HRV features of the heartbeat signal in the time domain and frequency domain are directly spliced ​​to form a new feature vector, and each feature vector is labeled to establish the fatigue and non-fatigue driving dataset E. The specific feature relationship is as follows Fig.19 shown.

[0134] Using data set E, DT, KNN, RF, and SVM models were trained with 5-fold cross validation, and the results shown in Table 3 were obtained. The accuracy and F1 score of different folds were calculated to obtain the final average accuracy and average F1 score. From the above two indicators, it can be seen that the RF model is more suitable for detecting fatigue driving using physiological characteristics.

[0135] Table 2

[0136]

[0137] 8.5. Multimodal Fatigue Driving Detection Model

[0138] In order to improve the accuracy of fatigue driving detection, enhance the robustness of the model, reduce the error and overfitting risk of single models trained based on facial features and physiological features, and integrate the advantages of multiple modalities to improve the overall prediction performance, the present invention performs multi-model decision fusion on a fatigue driving detection model based on facial fatigue features, a fatigue driving detection model based on breathing signal features, a fatigue driving detection model based on heartbeat signal features, and a fatigue driving detection model based on physiological features.

[0139] The present invention adopts a multi-model decision fusion method based on DS evidence theory. When using DS evidence theory to fuse the decision layer of multiple models, it is usually necessary to calculate the conflict coefficient K between different model pairs. The K value is used to measure the conflict relationship between elements of different evidence bodies. The larger the K, the greater the conflict. When K=1, the Dempster fusion rule is in an invalid state. The K value mainly measures the conflict between local focal elements, but fails to consider the reliability difference between the focal elements, and is prone to fall into the trap of self-conflict, which ultimately makes the Dempster fusion rule unable to fuse information and easily produces counterintuitive results. Therefore, in order to solve the above abnormal phenomena, the present invention adopts the mean K coefficient method. The multi-model decision structure of DS evidence theory based on the mean K coefficient method is as follows: Fig. 20 As shown in the figure, the accuracy of the final multimodal fatigue driving detection model obtained through data training is 0.975.

[0140] The method according to the present invention described above can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the multimodal fatigue driving detection method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the processing shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the processing shown here.

[0141] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A multi-modal fatigue driving detection method based on thermal imaging, characterized in that: The following steps are involved: a) Using thermal imaging equipment to obtain a thermal imaging image dataset with the driver's face, and using the dataset to train a YOLO v8 model for detecting faces and their key point information; b) performing face and face key point detection on each frame of the video stream to obtain face frame and facial key point information, further extracting the region of interest at the forehead and nose through the facial key point information, and obtaining the original heartbeat and respiratory signals by calculating the pixel grayscale mean of the region of interest in each frame of the image, and finally obtaining the physiological signals including the heartbeat signal and the respiratory signal through signal processing; c) performing fatigue-related feature extraction processing on the physiological signal obtained in step b), extracting the fatigue-related physiological features, and establishing fatigue and non-fatigue data sets based on the physiological features for training a fatigue driving detection model driven by multiple physiological features, including: Fatigue driving detection model based on heartbeat signal characteristics: Fatigue driving can be identified by analyzing the driver's heartbeat signal characteristics; Fatigue driving detection model based on breathing signal characteristics: The driver's breathing signal characteristics are used to determine whether there is a fatigue state; Fatigue driving detection model based on the fusion of heartbeat signal and breathing signal features; d) Analyze the facial fatigue features of each frame image, use the facial key point information to extract the fatigue and non-fatigue thermal imaging face image dataset, and use the dataset to train the VGG19 binary classification model to obtain a fatigue driving detection model based on facial fatigue features. By analyzing the facial fatigue features of the driver, whether he is fatigued or not is determined; e) The fatigue driving detection model based on facial fatigue features and the fatigue driving detection model driven by multiple physiological features are integrated into a multi-model decision-making process to use multiple modalities to judge the driver's fatigue state.

2. The multimodal fatigue driving detection method according to claim 1, characterized in that: The process of obtaining the heartbeat signal in step b) is as follows: S1.

1. According to the periodic change of human skin temperature, a region of interest ROI1 is selected, the region is located at the driver's forehead, and a time series is formed by the mean grayscale value of pixels in the region as the original heartbeat signal; S1.2, use the YOLO v8 model to perform face detection on each frame of the thermal imaging video stream, obtain the face frame and facial key point information, and determine the forehead area as the region of interest ROI1 based on this information; S1.3, converting the region of interest ROI1 of each frame image into a grayscale image; S1.4, calculating the mean grayscale value of pixels in the region of interest ROI1 of each frame of the grayscale image, generating a time series of the heartbeat signal, and forming the original heartbeat signal x1; S1.5, remove the trend of the original heartbeat signal x1 by using the HP filtering method to remove the trend component, and obtain the heartbeat signal x2 after the trend is removed; S1.6, perform wavelet threshold denoising on the heartbeat signal x2 after trend removal, use the sym8 wavelet basis, select a fixed threshold, and decompose the number of layers to 2 to obtain the denoised heartbeat signal x3; S1.

7. Use a bandpass filter to filter the denoised heartbeat signal x3 to obtain a final heartbeat signal x4.

3. The multimodal fatigue driving detection method according to claim 1, characterized in that: The process of obtaining the respiratory signal in step b) is as follows: S2.1, selecting the nostril area as the region of interest ROI2 based on the facial key point information; S2.2, converting the ROI2 of each frame image into a grayscale image, and calculating the mean grayscale value of the pixels in the area to obtain the original respiratory signal y1; S2.3, using the Otsu threshold algorithm to generate a binary image of the nostril area and create a mask, only calculating the grayscale mean of the mask area to obtain a respiratory signal with a high signal-to-noise ratio; S2.

4. Perform empirical mode decomposition (EMD) on the original respiratory signal y1 to remove trend components and high-frequency noise to obtain the final respiratory signal y2.

4. The multimodal fatigue driving detection method according to claim 1, characterized in that: In step c), physiological characteristics related to fatigue are extracted, as follows: Extract fatigue-related features from the breathing signal, including: Respiratory cycle: by calculating the time interval between adjacent peaks in the respiratory waveform; Respiratory amplitude: by calculating the difference between the maximum and minimum values ​​of the respiratory waveform in each cycle; The other fatigue features are: breathing frequency, breathing cycle standard deviation, breathing amplitude standard deviation, sample entropy, a total of 6 features The above features are obtained by statistically analyzing the 2-minute breathing signals and used to construct dataset B, which includes fatigue and non-fatigue data; Extract fatigue-related features from heartbeat signals, including: The features extracted from HRV in the time domain include heart rate HR, heart beat interval RR, mean value of all RRs Mean, standard deviation SDNN of all RRs, root mean square value of all adjacent RR differences rMSSD, standard deviation SDANN of the mean value of RR intervals within every 2 minutes, and percentage pNN50 of the number of adjacent RR intervals with a difference of more than 50 milliseconds in the total number of heart beats; The frequency domain features of HRV mainly extract high frequency HF, low frequency LF, low frequency / high frequency, and sample entropy of heartbeat signal; The above features are obtained by analyzing the 2-minute heartbeat signal in the time domain and frequency domain, and are used to construct dataset C, which includes fatigue and non-fatigue data.

5. The multimodal fatigue driving detection method according to claim 4, characterized in that: In step c), the fatigue driving detection model based on the breathing signal feature is trained on the dataset B using the random forest RF model; The fatigue driving detection model based on heartbeat signal features uses the SVM model to train the dataset C.

6. The multimodal fatigue driving detection method according to claim 5, characterized in that: Step c) also includes a fatigue driving detection model based on physiological characteristics, combining the characteristics of the breathing signal and the heartbeat signal to perform feature-level fusion; splicing the breathing signal features and the heartbeat signal features into a new feature vector, and using the new feature vector to establish a fatigue and non-fatigue driving data set E; using random forest RF to train the data set E.

7. The multimodal fatigue driving detection method according to claim 1, characterized in that: The step d) of establishing a fatigue driving detection model based on facial fatigue features specifically includes: The YOLO v8 model is used to obtain fatigue and non-fatigue facial images in each frame of thermal imaging video stream, and each frame of the image is annotated as fatigue and non-fatigue, thereby forming a fatigue and non-fatigue thermal imaging image face dataset. The thermal imaging image face dataset is used to train the VGG19 image binary classification model, and finally a fatigue driving detection model based on facial fatigue features is obtained.

8. The multimodal fatigue driving detection method according to claim 1, characterized in that: Step e) specifically comprises: The fatigue driving detection model based on facial features and the fatigue driving detection model based on physiological features are subjected to multi-model decision fusion, and the DS evidence theory based on the mean K coefficient method is used for multi-model decision fusion. Through decision fusion, the prediction results from the above multiple different models are fused, and finally a fatigue driving detection model based on multimodality is obtained. This model is used to judge the driver's fatigue state and provide the driver with real-time fatigue reminder.

Images