Video non-contact fatigue detection method fusing facial features and physiological features
Through the video contactless fatigue detection method that fuses facial and physiological characteristics, the LSTM-MHA model is used to automatically identify critical moments, solving the problems that are difficult to reveal in the existing methods of weak signal and coordinated evolution laws, and achieving high-precision fatigue state monitoring.
Patent Information
- Application Number
- CN202510613278.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-13
- Publication Date
- 2025-08-05
AI Technical Summary
The existing non-contact fatigue detection methods have problems such as weak signal, time synchronization and data heterogeneity in multimodal feature fusion. It is difficult for traditional methods to reveal the coordinated evolution law of physiological signals and facial behavior, and lack adaptability, resulting in insufficient detection accuracy.
A video contactless fatigue detection method that combines facial features and physiological characteristics is adopted to obtain key feature points through face detection, and a time sequence information of facial and physiological characteristics is modeled based on long-term and short-term memory networks and multi-head attention mechanisms, and critical moments are automatically identified and given higher weights, to construct an LSTM-MHA fatigue detection model.
It realizes high-precision continuous monitoring of driver fatigue status without wearing sensors, improves detection accuracy and adaptability, is suitable for actual driving scenarios, and reveals the coordinated evolution law of physiological signals and facial behavior.
Smart Images
Figure CN120419965A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of fatigue detection, and in particular relates to a video non-contact fatigue detection method that integrates facial features and physiological features. Background Art
[0002] Fatigue is a significant factor affecting human health and work efficiency, especially in driving environments, where fatigue can lead to accidents. This makes fatigue detection of great practical significance. From a physiological perspective, human physiological characteristics are objective indicators of fatigue. However, traditional fatigue detection methods based on physiological characteristics often use contact sensors. This not only affects driver comfort but can also affect normal operation during long-term monitoring, presenting significant limitations. Therefore, non-contact fatigue detection methods can detect fatigue without the need for wearable equipment, effectively overcoming the limitations of contact-based fatigue detection.
[0003] Existing non-contact fatigue detection methods primarily rely on facial video information, analyzing dynamic changes in the driver's eyes, mouth, and head to extract behavioral features related to fatigue. For example, eye aspect ratio, blink frequency, mouth aspect ratio, and head posture angle can all be used as important indicators for measuring fatigue. Because these methods do not require any sensors, they are convenient and practical, and are widely used in driving safety monitoring scenarios. To further improve detection accuracy, some studies have also attempted to fuse facial features with physiological signals (such as heart rate and heart rate variability) to comprehensively assess fatigue status using multimodal information. In multimodal fusion research, the fusion of facial and physiological features typically involves combining non-contact and contact signals. Even if both methods are non-contact, they can still face issues such as time synchronization and data heterogeneity. In the feature selection stage, existing methods mainly select key features based on manual experience or prior knowledge, and filter features using traditional statistical methods such as principal component analysis and maximum correlation minimum redundancy to extract a limited set of features for model training.
[0004] However, existing methods still have room for improvement in processing multimodal fatigue signatures. For one thing, extracting effective fatigue signatures from weak physiological signals in a non-contact manner remains a challenge. Furthermore, traditional video-based non-contact fatigue detection methods struggle to effectively reveal the co-evolution between physiological signals and facial behavior during fatigue. Furthermore, feature selection methods based on experience or static rules struggle to cope with complex scenarios characterized by large individual differences and diverse fatigue manifestations. They lack the ability to adapt to the dynamic evolution of fatigue states and struggle to identify the key moments in a time series that best represent fatigue. Summary of the Invention
[0005] In order to address the shortcomings of the above-mentioned existing technologies, the present invention proposes a video non-contact fatigue detection method that integrates facial features and physiological features, so as to automatically identify and focus on the key moments in facial features and physiological characteristics that best represent the fatigue state, thereby realizing effective non-contact fatigue detection and providing technical support for driver fatigue monitoring in real driving environments.
[0006] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:
[0007] The non-contact fatigue detection method of the present invention, which integrates facial features and physiological features, is characterized in that it is performed according to the following steps:
[0008] Step 1: Perform face detection on the subject’s video image to obtain the first Collection of key feature points of the frame , and according to the key feature points , get the facial region of interest of the frame ;
[0009] Step 2: Based on the collection of key feature points Extract Facial features of the frame , and in the facial region of interest Extraction Physiological characteristics of frames ;
[0010] Step 3: Construct a fatigue detection model, including feature preprocessing module, temporal feature encoding module, attention mechanism module and classification module, and and Processing is performed to obtain the predicted fatigue state sequence ,in, Indicates the The fatigue state vector of each time step.
[0011] The non-contact fatigue detection method for video that integrates facial features and physiological features according to the present invention is also characterized in that step 1 comprises:
[0012] Step 1.1: Use facial feature point detection algorithm to locate the first Frame video image The position of the facial landmarks is obtained Collection of key feature points of the frame ;
[0013] Step 1.2: Collection of key feature points In the process, the local area surrounded by the key feature points of the eyes and mouth is removed, and the remaining part is retained as the first facial region of interest of the frame .
[0014] Furthermore, the step 2 includes:
[0015] Step 2.1: According to Collection of key feature points of the frame , extract the Facial features that represent changes in facial state in frames , thereby constructing the original facial feature sequence ,in, represents the dimension of the extracted facial features, and T represents the total number of frames;
[0016] Step 2.2: According to facial region of interest of the frame The color change information of The pulse wave signal of the frame is Frame pulse wave signal, calculate the The heart rate and heart rate variability characteristics of the frame are used as the first Physiological characteristics of frames , and then construct the original physiological feature sequence ,in, Represents the dimension of the extracted physiological features.
[0017] Furthermore, the step 3 includes:
[0018] Step 3.1: The feature preprocessing module includes: a normalization processing unit and a feature fusion unit, and and Processing is performed to obtain a fusion feature sequence ;
[0019] Step 3.2: The temporal feature encoding module is to use the long short-term memory network LSTM to Processing is performed to obtain the enhanced time series feature sequence ,in, For the The enhanced time series feature vector is N, where N represents the total number of time steps.
[0020] Step 3.3: The attention mechanism module uses the attention mechanism to Processing to obtain attention feature sequence ;
[0021] Step 3.4: The classification module is a two-layer fully connected layer Perform fatigue state classification and obtain the predicted The fatigue state vector of time steps .
[0022] Furthermore, the step 3.1 includes:
[0023] Step 3.1.1: Normalization processing unit converts facial feature sequence and physiological characteristic sequences Perform normalization to obtain the normalized facial feature sequence and normalized physiological feature sequence ;
[0024] Step 3.1.2: Feature fusion unit will and After splicing on the feature dimension, a fusion feature sequence is formed ,in, Indicates the dimension of fused features.
[0025] Furthermore, the step 3.2 includes:
[0026] Step 3.2.1: According to the set time step length ,Will Divide into N time-step fusion feature subsequences ,in, Indicates the The time steps contained Frame fusion features, , Indicates rounding up;
[0027] Step 3.2.2: Fusion features of time steps Input to LSTM for processing and output The hidden state vector of time steps , thus obtaining the hidden state vector sequence .
[0028] Furthermore, the step 3.3 includes:
[0029] Step 3.3.1: Use the multi-head attention mechanism to Processing is performed to construct the query matrix , key matrix Sum Matrix ,in, Represents the query matrix query vector or key matrix The dimension of the key vector, Representation value matrix The dimension of the median vector. From the query matrix Take out the query vector corresponding to the time step , and calculate With the key matrix The dot product scores of all key vectors in form the attention energy sequence ;in, Indicates the The attention energy value of time steps;
[0030] Step 3.3.2: Use the softmax function to Perform normalization to obtain the attention weight sequence { ,in, Indicates the The attention weights of time steps. Then the value matrix The median vector is weighted to obtain Attention feature representation of time steps ,in, Representation value matrix Middle The value vector of the row, thus obtaining the attention feature sequence .
[0031] Furthermore, the step 3.4 includes:
[0032] Step 3.4.1: First fully connected layer pair After feature transformation, it is processed by nonlinear activation function to obtain the first The intermediate feature vector of time steps , thus forming an intermediate feature sequence ;
[0033] Step 3.4.2: Second fully connected layer pair Processing to obtain the predicted The fatigue state vector of time steps , thus forming a predicted fatigue state sequence , where 𝐶 is the number of categories of fatigue status.
[0034] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the video non-contact fatigue detection method, and the processor is configured to execute the program stored in the memory.
[0035] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium. The computer program is characterized in that when the computer program is run by a processor, the steps of the video non-contact fatigue detection method are executed.
[0036] Compared with the prior art, the present invention has the following beneficial effects:
[0037] 1. This invention proposes a video-based, non-contact fatigue detection method that integrates facial and physiological features. This method can simultaneously capture the driver's facial and physiological features in video without the need for wearing any sensors. This method uses a camera, a single-source, non-contact sensing device, to achieve higher-precision continuous perception and intelligent judgment of fatigue status. Compared to traditional detection methods that rely on contact sensors, this method effectively addresses the significant interference with user behavior caused by wearing the device and the poor comfort of long-term use. Furthermore, the simultaneous acquisition of dual-modal features by single-source non-contact sensing effectively improves system performance, making it suitable for continuous fatigue monitoring tasks in actual driving scenarios.
[0038] 2. The present invention simultaneously models the temporal information of facial features and physiological features, which can effectively capture the dynamic changes of facial behavioral changes in eye, mouth and head posture and physiological signals in the time dimension, revealing the co-evolution law between physiological signals and facial behavior under fatigue state, and solving the problem of independent processing of facial features and physiological features and lack of temporal dependency modeling in existing methods.
[0039] 3. The present invention identifies and dynamically weights the key moments in the time series that best represent the fatigue state, improving the model's discriminative ability by assigning higher weights to these moments. It also uses parallel attention heads to perform feature perception on different time segments, automatically capturing key physiological and behavioral changes during the occurrence, aggravation, or relief of fatigue, effectively overcoming the static and lack of selectivity shortcomings of traditional feature weighting strategies. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 Flowchart of the present invention;
[0041] Figure 2 This is a map showing the specific locations of key feature points for eye, mouth, and head posture estimation in the present invention;
[0042] Figure 3 This is the structural diagram of the LSTM-MHA fatigue detection model of the present invention. DETAILED DESCRIPTION
[0043] In this example, a video non-contact fatigue detection method that integrates facial features and physiological features is developed. By detecting key feature points on the face, facial features of the eyes, mouth, and head posture are extracted based on these key points. The pulse wave signal of the facial area is analyzed in combination with the POS algorithm to extract physiological features in a non-contact manner. The facial features and physiological features are then input into the LSTM-MHA fatigue detection model to detect the fatigue state. The specific flow chart is shown below. Figure 1 As shown, the following steps are included:
[0044] Step 1: Perform face detection on the subject’s video image to obtain the first Collection of key feature points of the frame , and according to the key feature points , get the facial region of interest of the frame ;
[0045] Step 1.1: Use facial feature point detection algorithm to locate the first Frame video image The position of the facial landmarks is obtained Collection of key feature points of the frame , where the key points used for eye, mouth and head pose estimation are as follows Figure 2 As shown;
[0046] Step 1.2: Collection of key feature points In the process, the local area surrounded by the key feature points of the eyes and mouth is removed, and the remaining part is retained as the first facial region of interest of the frame .
[0047] Step 2: Based on the collection of key feature points Extract Facial features of the frame , and in the facial region of interest Extraction Physiological characteristics of frames ;
[0048] Step 2.1: According to Collection of key feature points of the frame , extract the Facial features that represent changes in facial state in frames , thereby constructing the original facial feature sequence ,in, Represents the dimension of the extracted facial features. In this example, is 5, T represents the total number of frames;
[0049] Step 2.2: According to facial region of interest of the frame The color change information of The pulse wave signal of the frame is The pulse wave signal of the frame is analyzed in time domain and frequency domain to calculate the The heart rate and heart rate variability characteristics of the frame are used as the first Physiological characteristics of frames , and then construct the original physiological feature sequence ,in, Represents the dimension of the extracted physiological features. In this example, is 5.
[0050] Step 3: Construct a fatigue detection model, denoted as LSTM-MHA, as follows Figure 3 As shown in the figure, it includes: feature preprocessing module, temporal feature encoding module, attention mechanism module and classification module, and and Processing is performed to obtain the predicted fatigue state sequence ,in, Indicates the The fatigue state vector of each time step.
[0051] Step 3.1: The feature preprocessing module includes: normalization processing unit and feature fusion unit, and and Processing is performed to obtain a fusion feature sequence ;
[0052] Step 3.1.1: In order to prevent the features of a certain modality from dominating the fatigue features, the normalization processing unit converts the facial feature sequence and physiological characteristic sequences Perform normalization to obtain the normalized facial feature sequence and normalized physiological feature sequence ;
[0053] Step 3.1.2: Feature fusion unit will and After splicing on the feature dimension, a fusion feature sequence is formed ,in, Indicates the dimension of fusion features. In this example, is 10, providing unified data input for subsequent time series feature extraction.
[0054] Step 3.2: The temporal feature encoding module uses the long short-term memory network LSTM to Processing is performed to obtain the enhanced time series feature sequence ,in, For the The enhanced time series feature vector is N, which represents the total number of time steps. LSTM is used to model the temporal information of facial features and physiological features.
[0055] Step 3.2.1: According to the set time step length ,Will Divide into N time-step fusion feature subsequences ,in, Indicates the The time steps contained Frame fusion features, , Indicates rounding up. In this example, Set to 900 frames.
[0056] Step 3.2.2: Fusion features of time steps The input is processed into an LSTM with a hidden layer of 128, aiming to capture the long-term and short-term dependencies in time series data, reveal the co-evolutionary law between physiological signals and facial behaviors under fatigue conditions, and output the first The hidden state vector of time steps , thus obtaining the hidden state vector sequence .
[0057] Step 3.3: The attention mechanism module uses the attention mechanism to Processing to obtain attention feature sequence ;
[0058] Step 3.3.1: Use the multi-head attention mechanism to The attention mechanism automatically identifies the importance of the current time step in the global sequence and gives higher weights to more representative time segments, thereby improving the model's ability to focus on key fatigue moments. In this example, the number of MHA heads is set to 4 to construct the query matrix , key matrix Sum Matrix ,in, Represents the query matrix query vector or key matrix The dimension of the key vector, Representation value matrix The dimension of the median vector. From the query matrix Take out the query vector corresponding to the time step , and calculate With the key matrix The dot product scores of all key vectors in form the attention energy sequence ;in, Indicates the The attention energy value of time steps;
[0059] Step 3.3.2: Use the softmax function to Perform normalization to obtain the attention weight sequence { ,in, Indicates the The attention weights of time steps. Then the value matrix The median vector is weighted to obtain Attention feature representation of time steps ,in, Representation value matrix Middle The value vector of the row, thus obtaining the attention feature sequence .
[0060] Step 3.4: The classification module is to use two layers of fully connected layers to Perform fatigue state classification and obtain the predicted The fatigue state vector of time steps ;
[0061] Step 3.4.1: First fully connected layer pair After feature transformation, it is processed by nonlinear activation function to obtain the first The intermediate feature vector of time steps , thus forming an intermediate feature sequence ,In this example, the number of units in the first fully connected layer is set to 128;
[0062] Step 3.4.2: Second fully connected layer pair Processing to obtain the predicted The fatigue state vector of time steps , thus forming a predicted fatigue state sequence , where 𝐶 is the number of fatigue state categories. In this example, the number of units in the second fully connected layer is set to 32, and a Dropout layer with a dropout probability of 0.3 is introduced before the fully connected layer to prevent overfitting.
[0063] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.
[0064] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.
[0065] To verify the effectiveness of the method presented in this paper, we conducted experiments on the public dataset UTA-RLDD. All subjects were divided into 10 groups, each containing 6 subjects. Nine of these groups were used as training sets, and the remaining group was used as a test set. Furthermore, 20% of the samples in the training set were used as a validation set to evaluate the performance of the fatigue detection model. Using 10-fold cross-validation, the above process was repeated 10 times to ensure that each group of samples was included in the test set, effectively reducing the randomness of the experimental results.
[0066] The present invention was tested on the UTA-RLDD database with three typical fatigue detection models: support vector machine (SVM), LSTM, and one-dimensional convolutional neural network (1D-CNN). The performance was evaluated using five performance indicators: accuracy, precision, recall, F1, and area under the ROC curve (AUC). The experimental results are shown in Table 1.
[0067] Table 1 Comparison of experimental results of fatigue detection models on the UTA-RLDD dataset
[0068]
[0069] Table 1 shows that the LSTM-MHA model demonstrates significantly superior overall performance compared to other models. The LSTM-MHA model achieves an accuracy of 93.71%, an 8.6 percentage point improvement over the best comparison model, 1D-CNN. Its precision is 90.62%, a 10.41 percentage point improvement over SVM. Its recall and F1 scores are 87.08% and 88.63%, respectively, exceeding the LSTM baseline model by 13.87 and 12.29 percentage points. Furthermore, the LSTM-MHA achieves an AUC of 98.10%, significantly higher than both 1D-CNN and traditional LSTM models. This demonstrates the LSTM-MHA model's superior robust classification capabilities in class imbalance scenarios.
Claims
1. A video non-contact fatigue detection method integrating facial features and physiological features, characterized in that: The steps are as follows: Step 1: Perform face detection on the subject’s video image to obtain the first Collection of key feature points of the frame , and according to the key feature points , get the facial region of interest of the frame ; Step 2: Based on the collection of key feature points Extract Facial features of the frame , and in the facial region of interest Extraction Physiological characteristics of frames ; Step 3: Construct a fatigue detection model, including feature preprocessing module, temporal feature encoding module, attention mechanism module and classification module, and and Processing is performed to obtain the predicted fatigue state sequence ,in, Indicates the The fatigue state vector of each time step.
2. The video non-contact fatigue detection method integrating facial features and physiological features according to claim 1 is characterized in that: The step 1 comprises: Step 1.1: Use facial feature point detection algorithm to locate the first Frame video image The position of the facial landmarks is obtained Collection of key feature points of the frame ; Step 1.2: Collection of key feature points In the process, the local area surrounded by the key feature points of the eyes and mouth is removed, and the remaining part is retained as the first facial region of interest of the frame .
3. The video non-contact fatigue detection method integrating facial features and physiological features according to claim 1 is characterized in that: The step 2 includes: Step 2.1: According to Collection of key feature points of the frame , extract the Facial features that represent changes in facial state in frames , thereby constructing the original facial feature sequence ,in, represents the dimension of the extracted facial features, and T represents the total number of frames; Step 2.2: According to facial region of interest of the frame The color change information of The pulse wave signal of the frame is Frame pulse wave signal, calculate the The heart rate and heart rate variability characteristics of the frame are used as the first Physiological characteristics of frames , and then construct the original physiological feature sequence ,in, Represents the dimension of the extracted physiological features.
4. The video non-contact fatigue detection method integrating facial features and physiological features according to claim 1 is characterized in that: The step 3 comprises: Step 3.1: The feature preprocessing module includes: a normalization processing unit and a feature fusion unit, and and Processing is performed to obtain a fusion feature sequence ; Step 3.2: The temporal feature encoding module is to use the long short-term memory network LSTM to Processing is performed to obtain the enhanced time series feature sequence ,in, For the The enhanced time series feature vector is N, where N represents the total number of time steps. Step 3.3: The attention mechanism module uses the attention mechanism to Processing to obtain attention feature sequence ; Step 3.4: The classification module is a two-layer fully connected layer Perform fatigue state classification and obtain the predicted The fatigue state vector of time steps .
5. The video non-contact fatigue detection method integrating facial features and physiological features according to claim 4 is characterized in that: The step 3.1 includes: Step 3.1.1: Normalization processing unit converts facial feature sequence and physiological characteristic sequences Perform normalization to obtain the normalized facial feature sequence and normalized physiological feature sequence ; Step 3.1.2: Feature fusion unit will and After splicing on the feature dimension, a fusion feature sequence is formed ,in, Indicates the dimension of fused features.
6. The video non-contact fatigue detection method integrating facial features and physiological features according to claim 4 is characterized in that: The step 3.2 includes: Step 3.2.1: According to the set time step length ,Will Divide into N time-step fusion feature subsequences ,in, Indicates the The time steps contained Frame fusion features, , Indicates rounding up; Step 3.2.2: Fusion features of time steps Input to LSTM for processing and output The hidden state vector of time steps , thus obtaining the hidden state vector sequence .
7. The video non-contact fatigue detection method integrating facial features and physiological features according to claim 4 is characterized in that: The step 3.3 includes: Step 3.3.1: Use the multi-head attention mechanism to Processing is performed to construct the query matrix , key matrix Sum Matrix ,in, Represents the query matrix query vector or key matrix The dimension of the key vector, Representation value matrix The dimension of the median vector. From the query matrix Take out the query vector corresponding to the time step , and calculate With the key matrix The dot product scores of all key vectors in form the attention energy sequence ;in, Indicates the The attention energy value of time steps; Step 3.3.2: Use the softmax function to Perform normalization to obtain the attention weight sequence { ,in, Indicates the The attention weights of time steps. Then the value matrix The median vector is weighted to obtain Attention feature representation of time steps ,in, Representation value matrix Middle The value vector of the row, thus obtaining the attention feature sequence .
8. The video non-contact fatigue detection method integrating facial features and physiological features according to claim 5 is characterized in that: The step 3.4 includes: Step 3.4.1: First fully connected layer pair After feature transformation, it is processed by nonlinear activation function to obtain the first The intermediate feature vector of time steps , thus forming an intermediate feature sequence ; Step 3.4.2: Second fully connected layer pair Processing to obtain the predicted The fatigue state vector of time steps , thus forming a predicted fatigue state sequence , where 𝐶 is the number of categories of fatigue status.
9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the video non-contact fatigue detection method according to any one of claims 1 to 8, and the processor is configured to execute the program stored in the memory.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video non-contact fatigue detection method according to any one of claims 1 to 8 are executed.