Cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning
By using multi-view enhancement and adversarial learning methods, a 3D convolution and Transformer architecture is constructed, combined with a gradient reversal layer and a domain discriminator network, which solves the problems of individual differences and cross-domain inconsistency in driver fatigue detection, achieves accurate fatigue state detection and reliable uncertainty estimation, and is suitable for in-vehicle systems with multi-source data.
Patent Information
- Application Number
- CN202510814444.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-18
- Publication Date
- 2025-09-19
AI Technical Summary
Existing driver fatigue detection technologies have difficulty achieving accurate detection and robustness in the face of individual differences, cross-domain inconsistencies, and labeling inconsistencies, especially in vehicle-mounted systems with limited computing resources.
Using multi-view enhancement and adversarial learning methods, by constructing 3D convolution and Transformer architectures, combining gradient reversal layers and domain discriminator networks, a generalizable loss function is designed to output category predictions and prediction uncertainties of fatigue states.
It improves the generalization ability of the model in cross-domain environments, reduces the impact of noise, provides reliable uncertainty estimation, improves the reliability and robustness of the system, and is suitable for different data collection environments and annotation standards.
Smart Images
Figure CN120673384A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of driver status monitoring, and in particular to a cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning. Background Art
[0002] Deep learning-driven computer vision methods have made significant progress in detecting driver fatigue, but these methods still face multiple challenges in practical application. For example, different individuals exhibit different patterns of facial feature changes and physiological responses when fatigued. This individual difference makes it difficult for models trained on a specific population to accurately identify the fatigue state of new individuals, resulting in limited generalization capabilities in practical applications. Cross-domain inconsistency also significantly affects model performance. Existing fatigue detection datasets vary greatly in their collection environments. These environmental differences make it difficult to directly transfer models trained in one setting to other environments, severely limiting the practicality of fatigue detection systems.
[0003] On a technical level, existing applications for driver fatigue detection face severe challenges. While data augmentation methods can alleviate environmental differences to a certain extent, they struggle to capture the temporal characteristics of fatigue progression and have limited effectiveness when processing video sequence data. Feature learning techniques such as feature normalization can help reduce domain differences, but may also eliminate key individual-specific fatigue patterns, reducing the model's sensitivity to subtle fatigue signals. Meta-learning strategies typically rely on static features and assume consistent labeling, conditions that are often not met in fatigue detection scenarios, resulting in poor performance. While complex deep learning models can provide high detection accuracy, their computational cost and memory requirements often exceed the capabilities of onboard systems. Lightweight models, while computationally efficient, lack accuracy when dealing with complex environmental changes and individual differences. In practical applications, fatigue detection systems need to run in real time with limited computing resources while maintaining robustness to environmental changes. This requirement presents a significant gap between the performance of existing technologies and that of existing technologies.
[0004] Overall, existing approaches struggle to effectively handle view conflicts in multi-view data, cannot accurately estimate the uncertainty of classification results, and have limited generalization capabilities in cross-domain scenarios. Their inability to handle noisy labels makes the models susceptible to mislabeling and makes it difficult to maintain stable performance under mixed labeling standards. These technical bottlenecks severely restrict the deployment and promotion of fatigue detection systems in real-world safety-critical applications. There is an urgent need to develop novel architectures that can simultaneously address the multiple challenges of cross-subject, cross-domain, and inconsistent labeling. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning. The detection method models the output of the neural network as fuzzy membership and uses entropy to model the uncertainty of classification; it reduces the impact of conflicting views in multi-view fusion, while considering view-specific uncertainty and inter-view conflict.
[0006] In order to solve the above technical problems, the present invention is implemented in the following ways: A cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning specifically includes the following steps: S1, multi-view data preprocessing and enhancement; S2, build 3D convolution and Transformer architecture for feature extraction; S3, handles the gradient reversal layer and domain discriminator network in the adversarial learning module; S4. Construct a generalizable loss function to identify the reliability of ambiguous driver states; S5. Optimize the joint adversarial loss and classification loss; S6. After the model training is completed, the multi-view data containing noise and misaligned views in the test phase is input into the trained model, and the category prediction results and prediction uncertainty estimation of the fatigue state are output.
[0007] Furthermore, the specific method of step S1 is as follows: The input facial video dataset consists of V views of S samples, represented as , where v s Represents the Sth sample from different domains; for each video frame, extract N f facial key points to form a feature space for the facial area of fatigue detection , where T represents the number of frames in each video; Multi-view data augmentation model for facial fatigue signals: Taking the driver’s face video V as the starting point, extract Dimensional facial key points, for each input sequence, the expression of the basic key points is as follows: Among them, F represents the original view, represents the position of the i-th facial key point in image coordinates; Creating facial feature views of the driving scene, the feature views include a base view, an occlusion view, and a mirror view, The base view F retains the original facial key points f i The dynamic characteristics of , are expressed as follows: in, represents the position of the i-th facial key point in image coordinates; Blocking View The dynamic features that simulate real-world occlusions such as steering wheel interaction or head rotation are expressed as follows: Mirror View The dynamic features of asymmetric fatigue expressions such as unilateral eye rubbing are retained, and the expression is as follows: Augmented views are combined into a connected feature matrix ,in Depending on the number of enhanced views, when the above three views are used, the expression is as follows: .
[0008] Furthermore, the specific method of step S2 is as follows: 3D convolution and Transformer architecture optimized for fatigue time pattern are used for feature extraction, and the processing dimension is changed from input data shape Start with B, where B represents the batch size, and reshape to For 3D operations; through two consecutive 3D convolution and max pooling layer pairs, hierarchical features are extracted and the output is reshaped into For Transformer processing, Represents the new dimension after feature change; For temporal feature learning, learnable category tokens and position encodings are used to enhance flattened features. The enhanced features are processed by Transformer blocks. Each block implements a 4-head attention mechanism, a feedforward network, layer normalization, and a residual connection. Dropout regularization technology is applied to each block to improve model robustness. The processed features are then input into the adversarial learning module and the classification module for processing.
[0009] Furthermore, the specific method of step S3 is as follows: An adversarial learning module that processes in-car camera changes, including a gradient reversal layer (GRL) and a domain discriminator network; The gradient reversal layer dynamically adjusts the feature alignment by the gradient coefficient λ, which is used for the gradual domain shift between data, and its expression is as follows: Where h and l represent the upper and lower bounds of vehicle camera adaptation, α represents the adaptation rate between different driving datasets, n represents the batch index, and N represents the maximum number of iterations. During backpropagation, the gradient reversal layer applies a -λ value to drive the feature distribution towards cross-agent convergence. The domain discriminator network is a two-layer network used to process 2N f The domain characteristics of the dimension are used to distinguish data from different scenarios. The expression is as follows: Where D(x) represents the domain discriminator network, x represents the input feature vector, and represents the weight matrix, and represents the bias term, represents the rectified linear unit activation function; The discriminator's objective is to learn the loss function through adversarial The implementation is expressed as follows: in, Represents the batch index, feature vector Extracted from the feature processing module, D represents the number of driving data domains, represents the domain label of sample i, represents the predicted probability of the discriminator for domain d.
[0010] Furthermore, the specific method of step S4 is as follows: Threshold Cutting off the loss value of uncertain samples commonly seen in subjective evaluation is very important for predicting Below threshold The sample i will be Set it to 0 to prevent the sample from affecting the model parameters during back propagation; Basic loss function The expression is as follows: in, Represents the true label The predicted probability of is a positive number less than 1, smaller The value increases the sensitivity to early fatigue indicators, the classification loss function The expression is as follows: in, Represents a sample The binary weights of Represents iteration The classification model parameters, represents the cutoff constant; During the training process, the model is iteratively updated with the classification loss function , detection model parameters and sample weights , the specific expression is as follows: in, Indicates the samples; Automatically filter the fuzzy samples by iterative updating, and weight them Zero, while retaining reliable samples, the final model parameters For binary fatigue detection, it has three key advantages: noise reduction through truncation, training stability through bounded loss of progressive fatigue mode, and parameter Adaptive sensitivity of control.
[0011] Furthermore, the specific method of step S5 is as follows: For driving scenarios, we combine adversarial loss and classification loss optimization. The specific expression is as follows: in, represents the dynamic weight factor, represents the classification loss function, represents the adversarial learning loss function, Dynamic weight factor Starting from 0.0001 and increasing it allows the model to first establish reliable feature extraction and then optimize the domain-invariant representation.
[0012] Compared with the prior art, the present invention has the following beneficial effects: This paper develops a multi-view key point enhancement strategy that can systematically generate facial features, effectively solves the inconsistency problem caused by different camera angles and subject motion, and improves the generalization ability of the model across different subjects; proposes an adversarial domain module for learning cross-domain invariant features, which learns features by minimizing the differences between the source domain and the target domain, effectively reducing the impact of label standard and recording scene inconsistency on model performance; through the setting of a robust and generalizable loss function, it efficiently handles label noise and inconsistency in the evaluation process, and performs well in both subjective and objective evaluation scenarios, significantly improving model performance.
[0013] The present invention achieves accurate detection of cross-domain driver fatigue status, while providing reliable uncertainty estimation, significantly improving the reliability and robustness of the system in practical applications; the method is suitable for processing multi-source data from different data acquisition environments, different camera settings and different annotation standards, providing a new technical solution for driver status monitoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 Schematic diagram of the process of cross-domain fatigue detection method of the present invention. DETAILED DESCRIPTION
[0015] The specific implementation of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples.
[0016] To address the three major challenges in driver fatigue detection: cross-subject inconsistency, cross-domain inconsistency, and label standard inconsistency, we propose a technical solution that integrates multi-view enhancement, adversarial learning, and a generalizable loss function. This method can effectively handle conflicting information between different views, accurately estimate the uncertainty of classification results, and maintain good generalization capabilities in cross-domain scenarios. The technical solution of this invention is as follows: like Figure 1 As shown, a cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning includes the following steps: S1. Multi-view data preprocessing and enhancement. The specific methods are as follows: The input facial video dataset consists of V views of S samples, represented as , where v s Represents the Sth sample from different domains; for each video frame, extract N f facial landmarks , focusing on facial areas related to fatigue detection (eyes, mouth, head posture), forming a feature space , where T represents the sequence length, i.e. the number of frames in each video; To handle cross-subject inconsistencies (such as skin and motion feature differences) and camera angle variations, a multi-view data augmentation model for facial fatigue signals is proposed: Taking the driver’s face video V as the starting point, extract dimensional facial key points, Represents the number of facial key points. Using key points can focus on important facial movements while avoiding problems caused by lighting changes. For each input sequence, the expression of the basic key points is as follows: Among them, F represents the original view, represents the position of the i-th facial key point in image coordinates; Creating facial feature views of the driving scene, the feature views include a base view, an occlusion view, and a mirror view, The base view F retains the original facial key points f i The dynamic characteristics of , are expressed as follows: in, represents the position of the i-th facial key point in image coordinates; Blocking View The dynamic features that simulate real-world occlusions such as steering wheel interaction or head rotation are expressed as follows: Mirror View The dynamic features of asymmetric fatigue expressions such as unilateral eye rubbing are retained, and the expression is as follows: Augmented views are combined into a connected feature matrix ,in Depending on the number of enhanced views, when the above three views are used, the expression is as follows: at this time 6. F is the basic view, To block the view, In this way, the model can learn robust feature representations to different viewpoints and occlusions.
[0017] S2. Build a 3D convolution and Transformer architecture for feature extraction. The specific method is as follows: 3D convolution and Transformer architecture optimized for fatigue time pattern are used for feature extraction, and the processing dimension is changed from input data shape Start with B, where B represents the batch size, and reshape to For 3D operations; through two consecutive 3D convolution and max pooling layer pairs, hierarchical features are extracted and the output is reshaped into For Transformer processing, Represents the new dimension after feature change, where and is the dimension produced by the 3D convolution operation.
[0018] For temporal feature learning, the flattened features are enhanced using learnable category tokens and positional encodings. The enhanced features are then Each block implements a 4-head attention mechanism, a feedforward network, layer normalization, and a residual connection. Dropout regularization is applied to each block to improve the robustness of the model. The processed features are then input into the adversarial learning module and the classification module for processing.
[0019] S3. Design the gradient reversal layer and domain discriminator network in the adversarial learning module. The specific method is as follows: To address the generalization challenges brought about by differences in fatigue assessment standards across different data collection environments and domains, this paper proposes an adversarial learning module for in-car camera changes. The adversarial learning module includes a gradient reversal layer (GRL) and a domain discriminator network. The gradient reversal layer dynamically adjusts the feature alignment by the gradient coefficient λ, which is used for the gradual domain shift between data, and its expression is as follows: Where h and l represent the upper and lower bounds of vehicle camera adaptation, α represents the adaptation rate between different driving datasets, n represents the batch index, and N represents the maximum number of iterations. During backpropagation, the gradient reversal layer applies a -λ value to drive the feature distribution towards cross-agent convergence. The domain discriminator network is a two-layer network used to process 2N f The domain characteristics of the dimension are used to distinguish data from different scenarios. The expression is as follows: Where D(x) represents the domain discriminator network, x represents the input feature vector, and represents the weight matrix, and represents the bias term, represents the rectified linear unit activation function. To enhance the robustness of the model to changes in the specifications of the in-vehicle camera, the present invention seamlessly integrates the dropout mechanism.
[0020] The discriminator's objective is to learn the loss function through adversarial The implementation is expressed as follows: in, Represents the batch index, feature vector Extracted from the feature processing module, D represents the number of driving data domains, represents the domain label of sample i, represents the predicted probability of the discriminator for domain d. The proposed method uses the discriminator to distinguish different domains, while GRL reverses the gradient during backpropagation, obfuscates the domain boundaries and learns domain-invariant representations of fatigue patterns.
[0021] S4. Construct a generalizable loss function to identify the reliability of the fuzzy driver state. The specific method is as follows: To handle inconsistent fatigue labels between self-reports and others' assessments, a generalizable loss function is designed to identify the reliability of ambiguous driver states. When labels may be unreliable (for example, during the transition period between alert and fatigued states), the proposed model automatically reduces their influence during training.
[0022] Threshold The loss value of common uncertain samples in truncated subjective evaluation (KSS scale inconsistency) is important for prediction Below threshold The sample i will be Set it to 0 to prevent the sample from affecting the model parameters during back propagation; Basic loss function The expression is as follows: in, Represents the true label The predicted probability of Is a positive number less than 1, which controls the response of the loss function to the transition fatigue state; The value increases the sensitivity to early fatigue indicators, the classification loss function The expression is as follows: in, Representation sample The binary weights of Represents iteration The classification model parameters, represents the cutoff constant, which provides a gradual sample selection mechanism to gradually identify and filter out the fuzzy transition states and mislabeled samples that are common in driver fatigue datasets.
[0023] During training, the model iteratively updates three key components: classification loss , detection model parameters and sample weights , the specific expression is as follows: in, Indicates the samples, automatically filter the fuzzy samples through iterative updates, and weight them Zero, while retaining reliable samples, the final model parameters For binary fatigue detection, it has three key advantages: noise reduction through truncation, training stability through bounded loss of progressive fatigue mode, and parameter Adaptive sensitivity of control.
[0024] S5. Optimize the joint adversarial loss and classification loss. The specific method is as follows: For driving scenarios, we combine adversarial loss and classification loss optimization. The specific expression is as follows: in, represents the dynamic weight factor, represents the classification loss function, Represents the adversarial learning loss function, the final loss function of this combined target Robust fatigue detection across domains is ensured by improving the loss function Dealing with noisy labels by adversarial learning loss function Handling domain shift. Dynamic weighting factor Starting from 0.0001 and increasing it allows the model to first establish reliable feature extraction and then optimize the domain-invariant representation.
[0025] S6. After model training is complete, the multi-view data containing noise and misaligned views from the test phase is fed into the trained model, which outputs a fatigue state category prediction and an estimate of the prediction uncertainty. Specifically, the model provides two key outputs for each test sample: the category prediction and the prediction uncertainty. This allows the system to not only identify fatigue states but also assess the reliability of the prediction, which is particularly important for safety-critical applications.
[0026] The complete technical solution formed by the above steps in the present invention realizes the accurate detection of cross-domain driver fatigue status, provides reliable uncertainty estimation, and significantly improves the reliability and robustness of the system in practical applications; it is also suitable for processing multi-source data from different data acquisition environments, different camera settings, and different annotation standards, providing a new technical solution for the field of driver status monitoring.
[0027] The above description is merely an embodiment of the present invention. It is stated again that, for a person skilled in the art, several improvements can be made to the present invention without departing from the principles of the present invention, and these improvements are also included in the scope of protection of the claims of the present invention.
Claims
1. A cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning, characterized by: The specific steps include: S1, multi-view data preprocessing and enhancement; S2, build 3D convolution and Transformer architecture for feature extraction; S3, handles the gradient reversal layer and domain discriminator network in the adversarial learning module; S4. Construct a generalizable loss function to identify the reliability of ambiguous driver states; S5. Optimize the joint adversarial loss and classification loss; S6. After the model training is completed, the multi-view data containing noise and misaligned views in the test phase is input into the trained model, and the category prediction results and prediction uncertainty estimation of the fatigue state are output.
2. The cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning according to claim 1 is characterized by: The specific method of step S1 is as follows: The input facial video dataset consists of V views of S samples, represented as , where v s Represents the Sth sample from different domains; for each video frame, extract N f facial key points to form a feature space for the facial area of fatigue detection , where T represents the number of frames in each video; Multi-view data augmentation model for facial fatigue signals: Taking the driver’s face video V as the starting point, extract Dimensional facial key points, for each input sequence, the expression of the basic key points is as follows: Among them, F represents the original view, represents the position of the i-th facial key point in image coordinates; Creating facial feature views of the driving scene, the feature views include a base view, an occlusion view, and a mirror view, The base view F retains the original facial key points f i The dynamic characteristics of , are expressed as follows: in, represents the position of the i-th facial key point in image coordinates; Blocking View The dynamic features for real-world occlusion situations that simulate steering wheel interaction or head rotation are expressed as follows: Mirror View The dynamic characteristics of the asymmetric fatigue expression with unilateral eye rubbing are retained, and the expression is as follows: Augmented views are combined into a connected feature matrix ,in Depending on the number of enhanced views, when the above three views are used, the expression is as follows: 。 3. The cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning according to claim 1 is characterized by: The specific method of step S2 is as follows: 3D convolution and Transformer architecture optimized for fatigue time pattern are used for feature extraction, and the processing dimension is changed from input data shape Start with B, where B represents the batch size, and reshape to For 3D operations; through two consecutive 3D convolution and max pooling layer pairs, hierarchical features are extracted and the output is reshaped into For Transformer processing, Represents the new dimension after feature change; For temporal feature learning, learnable category tokens and position encodings are used to enhance the flattened features. The enhanced features are processed by Transformer blocks. Each block implements a 4-head attention mechanism, a feedforward network, layer normalization, and a residual connection. Dropout regularization is applied to each block, and the processed features are then input into the adversarial learning module and the classification module for processing.
4. The cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning according to claim 1 is characterized by: The specific method of step S3 is as follows: An adversarial learning module that processes in-car camera changes includes a gradient reversal layer and a domain discriminator network. The gradient reversal layer dynamically adjusts the feature alignment by the gradient coefficient λ, which is used for the gradual domain shift between data, and its expression is as follows: Where h and l represent the upper and lower bounds of vehicle camera adaptation, α represents the adaptation rate between different driving datasets, n represents the batch index, and N represents the maximum number of iterations. During backpropagation, the gradient reversal layer applies a -λ value to drive the feature distribution towards cross-agent convergence. The domain discriminator network is a two-layer network used to process 2N f The domain characteristics of the dimension are used to distinguish data from different scenarios. The expression is as follows: Where D(x) represents the domain discriminator network, x represents the input feature vector, and represents the weight matrix, and represents the bias term, represents the rectified linear unit activation function; The discriminator's objective is to learn the loss function through adversarial The implementation is expressed as follows: in, Represents the batch index, feature vector Extracted from the feature processing module, D represents the number of driving data domains, represents the domain label of sample i, represents the predicted probability of the discriminator for domain d.
5. The cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning according to claim 1 is characterized by: The specific method of step S4 is as follows: Threshold Cutting off the loss value of uncertain samples commonly seen in subjective evaluation is very important for predicting Below threshold The sample i will be Set it to 0 to prevent the sample from affecting the model parameters during back propagation; Basic loss function The expression is as follows: in, Represents the true label The predicted probability of is a positive number less than 1, smaller The value increases the sensitivity to early fatigue indicators, the classification loss function The expression is as follows: in, Representation sample The binary weights of Represents iteration The classification model parameters, represents the cutoff constant; During the training process, the model is iteratively updated with the classification loss function , detection model parameters and sample weights , the specific expression is as follows: in, Indicates the samples.
6. The cross-domain driver fatigue detection method based on multi-view enhancement and adversarial learning according to claim 1 is characterized by: The specific method of step S5 is as follows: For driving scenarios, we combine adversarial loss and classification loss optimization. The specific expression is as follows: in, represents the dynamic weight factor, represents the improved loss function, represents the adversarial learning loss function.