Double-flow feature embedded driving fatigue detection system and method based on transfer learning

Abstract

The invention provides a double-flow feature embedded driving fatigue detection system based on transfer learning, and relates to the technical field of machine vision and intelligent traffic, and the system comprises a facial feature extraction module which is used for receiving a driver facial image collected by a vehicle-mounted camera in an actual driving environment; the head posture estimation module is used for processing the head posture data set to capture the fatigue behavior performance of the driver; the migration fusion module is used for performing learnable fusion on the face visual feature vector and the head posture feature vector and converting the fusion into high-dimensional fusion feature representation; and the state classification and prompt feedback module performs multi-class classification, outputs a fatigue detection result, and is used for real-time driving state judgment and early warning. According to the method, the end-to-end heterogeneous feature embedding framework is constructed by fusing the facial representation and the three-dimensional head posture features. According to the method, efficient fusion of cross-source data features in a unified embedding space is realized, and high-reliability perception and early warning of driving fatigue behaviors are ensured.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and intelligent transportation technology, and in particular to a dual-stream feature embedding driving fatigue detection system and a dual-stream feature embedding driving fatigue detection method based on transfer learning. Background Technology

[0002] Fatigue driving is a major contributing factor to traffic accidents. When drivers are fatigued, their attention significantly decreases, visual perception and reaction speed are noticeably weakened, and the risk of operational errors increases significantly. This physiological and psychological state makes it difficult for drivers to respond to emergencies in a timely manner, thus significantly increasing the probability of traffic accidents. This phenomenon is particularly prominent in the transportation industry, especially for drivers driving at night, who are more prone to drowsiness and fatigue, further impairing their judgment and decision-making abilities. This not only threatens the driver's personal safety but may also have a serious impact on other road users.

[0003] With the rapid development of deep learning and computer vision technologies, these advanced technologies have provided important support for building efficient, accurate, and real-time fatigue warning systems. Such systems can continuously monitor the driver's fatigue state and issue timely warnings, thereby effectively preventing potential traffic accidents. Existing fatigue driving detection methods mainly include those based on driver physiological signals, facial features, and vehicle driving parameters. Due to the advantages of non-contact and high accuracy, facial feature-based methods have become a research hotspot.

[0004] However, this method still faces a series of challenges in practical applications.

[0005] First, existing datasets often focus on a single modality, such as facial images and fatigue state labels, but their feature representation is insufficient in complex driving environments. Although some studies in recent years have attempted to introduce head pose information to assist fatigue detection, existing multidimensional datasets are self-built and not publicly available, which limits the widespread application and validation in this field. Second, in fatigue driving detection, small eyes, such as squinting and smiling, are easily misjudged as fatigue states. If not handled properly, the existence of ambiguous samples will seriously affect the reliability of the model.

[0006] Therefore, a highly accurate and robust non-contact fatigue detection system is needed to solve the above-mentioned technical problems. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a dual-stream feature embedding driving fatigue detection system and method based on transfer learning, which solves the problems of low robustness of the model in complex scenarios and low overall detection accuracy in the prior art.

[0008] To achieve the above and other related objectives, the present invention provides a dual-stream feature embedding driving fatigue detection system based on transfer learning, comprising:

[0009] The facial feature extraction module is used to receive driver facial images captured by the vehicle-mounted camera in actual driving conditions;

[0010] The head pose estimation module processes the head pose dataset to capture driver fatigue behavior.

[0011] The transfer fusion module is used to learnably fuse facial visual feature vectors with head pose feature vectors, converting them into a high-dimensional fused feature representation.

[0012] The status classification and feedback module is used to perform multi-class classification on high-dimensional fusion features, output fatigue detection results, and provide real-time driving status judgment and warning.

[0013] The head pose estimation module adopts the FSANet architecture, which has a dual-stream multi-scale feature extraction mechanism to fully capture the local and global information of head pose. At the same time, a pixel-level attention weight mapping module is introduced to enhance the response capability to key head regions.

[0014] By adopting the above technical solution, an end-to-end heterogeneous feature embedding framework is constructed by fusing facial representations and three-dimensional head pose features. This method introduces a module-level transfer learning strategy to achieve efficient fusion of cross-source data features in a unified embedding space, ensuring highly reliable perception and early warning of driving fatigue behavior. Furthermore, the dual-stream architecture enables the system to perceive both facial micro-expressions and macro-head movements, significantly improving the robustness of the system.

[0015] This invention also provides a dual-stream feature embedding method for driving fatigue detection based on transfer learning, and a dual-stream feature embedding driving fatigue detection system based on transfer learning, comprising the following steps:

[0016] S1. Facial visual feature extraction and weighted training: The facial feature extraction module extracts dynamic visual features related to fatigue through the YOLOv11n-cls lightweight detection network.

[0017] S2. Head pose estimation and regression: The head pose estimation module uses a regression model based on the FSANet architecture to output the angle values ​​of the driver's head in the pitch, yaw and roll directions, capturing the driver's fatigue behavior.

[0018] S3. Transfer Fusion: The two modules in steps S1 and S2, which are pre-trained on training datasets from different sources, are fused in a learnable manner through the transfer fusion module, and converted into a unified high-dimensional fusion feature representation through trainable linear mapping parameters and nonlinear activation functions.

[0019] S4. Classification Decision and Feedback: Based on the high-dimensional fusion feature representation obtained in step S3, the state classification and prompt feedback module uses a fully connected neural network combined with the Softmax function to perform multi-class classification and output fatigue detection results.

[0020] By adopting the above technical solution, this method ensures the independence of feature extraction and the unity of feature fusion. Each step is interconnected, which not only guarantees the purity of single-modal features, but also solves the training problem under small sample data by using transfer learning, thus ensuring the convergence speed and detection accuracy of the algorithm under limited data.

[0021] In one embodiment of the present invention, in step S1, the facial feature extraction module introduces a weighted training strategy, including the following steps:

[0022] S11. Divide the samples in the training set into normal samples and blurred samples; blurred samples include facial image samples with squinting, smiling, and eye openings smaller than a preset width threshold.

[0023] S12. In the model training process, a hierarchical weighted cross-entropy loss function is introduced. The loss value of fuzzy samples is multiplied by the weight coefficient ω to improve the learning intensity of the model for fuzzy samples.

[0024] S13. Determine the optimal weight parameter value ω through experiments to enhance the model's ability to discriminate fuzzy fatigue features in complex visual scenes.

[0025] By adopting the above technical solution, introducing a weighted training strategy, and classifying normal samples into fuzzy samples, the model's ability to distinguish fuzzy fatigue features can be significantly improved. This directly reduces the false alarm rate caused by changes in driver facial expressions in practical applications, and significantly improves the driver's experience.

[0026] In one embodiment of the present invention, the head pose estimation module in step S2 adopts the FSANet architecture and includes:

[0027] Based on this architecture, a dual-stream multi-scale feature extraction mechanism is provided to fully capture local and global information of head pose. At the same time, a pixel-level attention weight mapping module is introduced to enhance the response capability to key head regions. Finally, a multi-stage classification stepwise regression learning module is used to achieve high-precision 3D pose estimation of pitch, yaw and roll angles. The multi-stage classification stepwise regression learning module adopts the Soft Stagewise Regression module.

[0028] By adopting the above technical solutions, compared with traditional direct regression methods, traditional head pose estimation often relies on facial key points. Once the face is occluded (such as by a mask or water cup), key point detection will fail. The FSANet architecture directly aggregates spatial structure information from the feature map for regression, without relying on key points. Therefore, it can still maintain high accuracy even when the face is partially occluded. The SSR module, through the strategy of "classification first and then regression", transforms continuous angle prediction into staged probability distribution prediction, which solves the problem of unstable continuous values ​​in direct regression of neural networks. It can accurately capture the driver's slight drowsy nodding movements and improve the sensitivity of fatigue behavior capture.

[0029] In one embodiment of the present invention, the migration fusion module in step S3 includes:

[0030] During training, the pre-trained parameters of the facial visual feature extraction module and the head pose estimation module in steps S1 and S2 are frozen. Only the fusion and subsequent classification modules are jointly fine-tuned to achieve unified embedding and end-to-end adaptation of heterogeneous features. The facial feature vector extracted in step S1 and the head pose feature vector obtained in step S2 are concatenated to form a joint feature representation. This representation is then transformed using a set of trainable linear mapping matrices and nonlinear activation functions to obtain high-dimensional fusion features.

[0031] By adopting the above technical solution and freezing these parameters, this invention only needs to train a lightweight fusion layer on a small fatigue dataset to achieve the alignment of heterogeneous features in a unified embedding space. This not only speeds up the training process but also ensures the model's generalization ability on specific fatigue detection tasks. It retains the powerful feature extraction capabilities learned by the pre-trained model on large-scale datasets while avoiding overfitting due to the small amount of data in the target domain, and significantly reduces the computational resources required for training.

[0032] In one embodiment of the present invention, the feature mapping process in step S3 can be expressed as follows: In the formula, This refers to the driver's facial input image captured by the vehicle's onboard camera. This indicates that step S1 extracts the data based on the input image. facial features This indicates that step S2 obtains the result based on the input image. Head posture characteristics, This indicates a feature concatenation operation. It is a learnable linear mapping matrix. It is a non-linear activation function. This represents the final high-dimensional fusion feature.

[0033] By adopting the above technical solution, the design of this linear mapping matrix combined with the nonlinear activation function can automatically learn the weight relationship between visual features and pose features, ensuring that feature vectors of different dimensions can be projected into the same manifold space for interaction.

[0034] In one embodiment of the present invention, in step S4, a multi-layer fully connected neural network structure is adopted for the fused feature vector obtained in step S3, combined with normalization processing, nonlinear activation function and Dropout strategy to improve generalization ability, and the high-dimensional fused features are mapped to the probability distribution of three driving states, namely awake, drowsy and yawning, through the Softmax function at the output end to complete the fatigue state classification.

[0035] By adopting the above technical solutions, S4 employs a multi-layer fully connected network combined with the Dropout strategy to further enhance the model's non-linear expressive ability. Dropout randomly discards neurons during training, effectively preventing the model from over-relying on certain specific features (overfitting). This allows the system to maintain consistently high classification accuracy when facing drivers of different genders, ages, and races. The Softmax function maps the output to a probability distribution, enabling the system to provide not only a "yes / no" judgment but also a "confidence level," facilitating smoothing processing in subsequent logic layers.

[0036] In one embodiment of the present invention, in step S4, the system outputs a corresponding prompt sound based on the predicted category obtained from the classification. The prompt sound is not triggered when the driver is awake, a short warning sound is triggered when the driver is drowsy, and a longer prompt sound is triggered when the driver is yawning, so as to enhance the intelligent reminder and intervention effect on driver fatigue behavior.

[0037] By adopting the above technical solution, this graded intervention mechanism conforms to human factors engineering. It can provide a slight reminder when there is mild fatigue to prevent interference with driving, and provide strong intervention when severe fatigue characteristics (such as continuous yawning) appear, so as to maximize driving safety and reflect the intelligence and humanization of the system.

[0038] In one embodiment of the present invention, the YOLOv11n-cls lightweight detection network used in step S1 includes a C3k2 module, an SPPF module, and a C2PSA module; wherein, the C3k2 module is based on a cross-stage local network structure and uses a variable convolution kernel size to extract features at different depth levels; the SPPF module is used to perform spatial pyramid pooling on the feature map to fuse multi-scale information; the C2PSA module introduces a spatial attention mechanism to enhance the saliency of fatigue-related feature extraction by enhancing the feature response of the eye and mouth regions.

[0039] In one embodiment of the present invention, the Soft Stagewise Regression module transforms the continuous angle regression problem into a staged classification and refinement regression problem. It predicts the discrete interval to which each angle belongs and the relative offset within the interval, including the discrete interval to which pitch angle, yaw angle, and roll angle belong and the relative offset within the interval. It calculates the expected value of the probability distribution to obtain the final pitch angle, yaw angle, and roll angle values, thereby solving the problem of discontinuity in angle prediction.

[0040] As described above, the dual-stream feature embedding driving fatigue detection system and method based on transfer learning of the present invention has the following beneficial effects:

[0041] 1. Significantly improves detection accuracy and robustness: By constructing a dual-stream feature embedding model, it effectively integrates two complementary types of information: facial texture details (extracted by the YOLOv11n-cls lightweight detection network) and head geometric pose (extracted by the FSANet architecture). Compared with single-modal detection, this method is more robust under changes in lighting, partial facial occlusion (such as wearing a mask), or atypical poses, and significantly reduces the false negative rate and false positive rate.

[0042] 2. Effectively solves the pain point of misjudgment of ambiguous samples: By introducing a weighted training strategy and a hierarchical weighted loss function, the model's learning weights for ambiguous fatigue samples such as squinting, smiling, and small eyes are specifically enhanced, which improves the misjudgment problem of traditional algorithms in these boundary cases and makes the system more in line with the needs of real driving scenarios.

[0043] 3. Overcoming the barriers to heterogeneous data fusion: By adopting a module-level transfer learning strategy and freezing the pre-trained backbone network and fine-tuning the fusion layer, the problem of facial features and pose features coming from different datasets and lacking corresponding relationships has been successfully solved. This enables the system to build a powerful multimodal detection system using existing high-quality single-modal datasets.

[0044] 4. Achieving lightweight and real-time performance: The system adopts the YOLOv11n-cls lightweight detection network and FSANet architecture as its basic architecture. Combined with the parameter freezing strategy, the number of model parameters and computational complexity are greatly reduced, enabling the system to be deployed on vehicle embedded platforms with limited computing power, achieving low-latency real-time inference and meeting the stringent requirements of driving safety for response speed.

[0045] 5. Provides intelligent hierarchical intervention: The system can not only identify "fatigue / non-fatigue", but also subdivide "drowsy" and "yawning" states, and trigger different levels of voice alarms accordingly. This hierarchical feedback mechanism is more humanized and can provide appropriate intervention based on the degree of danger, improving the driver experience and the practical value of the system. Attached Figure Description

[0046] Figure 1 The diagram shows a schematic of the facial fatigue feature extraction module based on the improved YOLOv11n-cls lightweight detection network in an embodiment of the present invention.

[0047] Figure 2 The diagram shows the structure of the head pose estimation module based on the FSANet architecture in an embodiment of the present invention.

[0048] Figure 3 The diagram shown is a schematic representation of feature fusion and classification based on transfer learning mechanism in an embodiment of the present invention. Detailed Implementation

[0049] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0050] Please see Figures 1 to 3 It should be understood that the structures, proportions, sizes, etc., illustrated in the accompanying drawings are merely for illustrative purposes to aid those skilled in the art and to facilitate understanding and reading. They are not intended to limit the scope of the invention and therefore have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to size, without affecting the effectiveness and purpose of the invention, should still fall within the scope of the technical content disclosed in this invention. Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity and not intended to limit the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention's implementation.

[0051] Example 1:

[0052] like Figure 1 , Figure 2 As shown, this embodiment provides a dual-stream feature embedding driving fatigue detection system based on transfer learning.

[0053] include:

[0054] The facial feature extraction module is used to receive driver facial images captured by the vehicle-mounted camera in actual driving environment. It is responsible for extracting the region of interest (ROI) of the face from the video stream captured by the vehicle-mounted camera and mapping it into a high-dimensional feature vector. It is the system's entry point for perceiving driver facial details (such as eye opening and closing, mouth shape changes, and facial muscle relaxation).

[0055] In this embodiment, the facial feature extraction module uses the YOLOv11n-cls lightweight detection network model as its core architecture, which includes the following key substructures:

[0056] Stem layer: Fast downsampling reduces image resolution and decreases subsequent computation.

[0057] C3k2 module: This is the core component of YOLOv11 and an improvement on the C2f module. It improves the richness of feature extraction while maintaining lightweight by introducing combinations of convolutional kernels of different sizes (such as 1x1 and 3x3) and dynamically adjusting the receptive field in the residual connections. Compared with the traditional Bottleneck structure, C3k2 makes more efficient use of gradient flow information.

[0058] C2PSA module (Convolutional block with Parallel Spatial Attention): This module introduces a spatial attention mechanism. In the deeper layers of feature extraction, C2PSA can adaptively assign higher weights to key facial regions (such as around the eyes and mouth) while suppressing interfering features from the background (such as seats and car windows).

[0059] SPPF (Spatial Pyramid Pooling - Fast): Located at the end of the backbone network, it achieves the fusion of multi-scale features through three parallel 5x5 max pooling layers, enabling the model to adapt to facial scale changes caused by the driver's head moving forward and backward.

[0060] The YOLOv11n-cls lightweight detection network extracts dynamic visual features related to fatigue, which are then processed by a global average pooling layer to output a facial feature vector. .

[0061] The head pose estimation module processes the head pose dataset to capture the driver's fatigue behavior and can calculate the driver's three-dimensional head pose angles in real time, namely pitch angle (nodding / tilting), yaw angle (turning left and right), and roll angle (tilting left and right).

[0062] In this embodiment, the core architecture of the head pose estimation module adopts FSANet (Fine-Grained Structure Aggregation Network).

[0063] Dual-stream input: The network is designed with two parallel feature extraction paths, each using a different convolutional kernel configuration, to capture features of different frequencies in the image (such as texture details and contour structures).

[0064] Capsule Fusion: This technique uses a capsule network strategy to aggregate features from two streams. By employing a dynamic routing mechanism between capsules, it preserves the spatial hierarchy of features.

[0065] Pixel-wise Attention: Automatically generates an attention heatmap, highlighting regions that contribute to pose estimation (such as the nose and ears) while ignoring useless background.

[0066] Soft Stagewise Regression (SSR): This method divides the angular range (e.g., -99° to +99°) into multiple overlapping bins. The network predicts the probability distribution and offset within each bin, and finally calculates the expected value to obtain the precise angle. This method is more stable and converges faster than direct MSE regression, and outputs a pose feature vector. .

[0067] The transfer fusion module is used to learnably fuse facial visual feature vectors and head pose feature vectors, converting them into a high-dimensional fusion feature representation. It can align and fuse the facial features output by the facial feature extraction module and the pose features output by the head pose estimation module.

[0068] In this embodiment, the migration fusion module architecture design adopts a lightweight multilayer perceptron (MLP):

[0069] Input layer: Accepts the concatenated vector [ ; ];

[0070] Hidden layers: contain 1-2 fully connected (Dense Layers), each followed by a Batch Normalization (BN) layer and an activation function (such as ReLU or SiLU).

[0071] Dropout layer: Add Dropout (dropout rate set to 0.3-0.5) between hidden layers to randomly deactivate some neurons, preventing the model from over-relying on features of a certain modality and enhancing robustness.

[0072] The status classification and feedback module performs multi-class classification, outputs fatigue detection results, and is used for real-time driving status judgment and early warning. It makes a final judgment based on fused features and executes feedback.

[0073] The classifier maps the output of the fusion module to the probabilities of three categories: awake, drowsy, and yawning.

[0074] This module uses logic control to smooth the probability sequence over time (e.g., using sliding window filtering) to prevent output jumps. Based on the determined state category, control commands are sent via the CAN bus or audio interface to trigger the playback of different audio files.

[0075] Example 2:

[0076] This embodiment provides a detection method for a dual-stream feature embedding driving fatigue detection system based on transfer learning as described in Embodiment 1, including the following steps:

[0077] Step S1: Facial visual feature extraction and weighted training;

[0078] Based on the facial feature extraction module and existing fatigue behavior datasets, an improved lightweight YOLOv11n-cls network classification model is introduced to extract dynamic visual features highly correlated with fatigue states, thereby achieving efficient recognition of facial fatigue behavior and extracting fatigue-related dynamic visual features.

[0079] Image preprocessing: The video frames captured by the vehicle camera are processed by face detection (such as using RetinaFace or MTCNN) to crop out the facial region, and then resized to 224*224 pixels and normalized.

[0080] Feature extraction: The preprocessed image is input into a network with YOLOv11n-cls lightweight detection network as the backbone. The image passes through the Stem layer, stacked C3k2 modules, C2PSA attention module, and SPPF module in sequence. Finally, the feature vector is extracted in the GAP layer before the classification head. ;

[0081] Because the model faces difficulties in classifying "boundary samples" such as slight squinting and small eye openings during training,

[0082] In step S1, the facial feature extraction module introduces a weighted training strategy. In this strategy, each sample in the training set is assigned a corresponding loss weight according to the degree of blurriness of its visual features. Blurred samples are given higher training attention, thereby strengthening the model's ability to distinguish complex samples and improving its robustness in recognizing fatigue behavior in real-world scenarios.

[0083] The weighted training strategy includes the following steps:

[0084] S11. Sample division: Divide the samples in the training set into "normal samples" and "fuzzy samples". Samples with squinting smiles, small eye fissures, etc. are classified as "fuzzy samples", and the rest of the images are classified as "normal samples".

[0085] S12. In the model training process, a hierarchical weighted cross-entropy loss function is introduced. For fuzzy samples, the loss is multiplied by the weight coefficient ω to improve its learning strength, while for normal samples, the original loss value is maintained to avoid overfitting.

[0086] The loss function takes the following form:

[0087]

[0088] In the formula, N represents the number of samples, and ω is the weighting coefficient. For fuzzy samples, For normal samples, , and represents the cross-entropy loss of the i-th sample.

[0089] S13. Determine the optimal weight parameter value ω through experiments to enhance the model's ability to discriminate fuzzy fatigue features in complex visual scenes.

[0090] Step S2: Head pose estimation and regression;

[0091] The head pose estimation module employs a regression model based on the FSANet architecture to analyze the changes in the driver's head in the three directions of pitch, yaw, and roll frame by frame. This enables the modeling of typical fatigue behaviors such as looking down, tilting the head, and drowsiness during driving, and outputs the angle values ​​of the driver's head in the three directions of pitch, yaw, and roll to capture the driver's fatigue behavior.

[0092] The head pose estimation module adopts the FSANet architecture. The FSANet network structure employs a dual-branch multi-scale feature extraction path, which features a dual-stream multi-scale feature extraction mechanism to fully capture local and global information of the head pose. Simultaneously, a pixel-level attention weight mapping module is introduced to enhance the responsiveness to key regions. Finally, a multi-stage classification stepwise regression learning module is used to achieve high-precision 3D pose estimation of pitch, yaw, and roll angles. This multi-stage classification stepwise regression learning module employs the Soft Stagewise Regression module.

[0093] The Soft Stagewise Regression module transforms the continuous angle regression problem into a staged classification and refinement regression problem. It obtains the final pitch, yaw, and roll angles by predicting the discrete interval to which each angle belongs and the relative offset within the interval, and calculating the expected value of the probability distribution, in order to solve the problem of discontinuity in angle prediction.

[0094] Processing flow: Input the same facial image (or an extended region containing the head) into the FSANet architecture;

[0095] SSR regression mechanism: The FSANet architecture internally calculates the angle through Soft Stagewise Regression and sets the predicted pitch angle. to its scope Divided into The model outputs the probability for each interval. and the adjustment value within the range The final angle prediction value is:

[0096] ;

[0097] in, It is the first The central value of each interval, this mechanism makes angle prediction probabilistic and more resistant to interference.

[0098] Output: Head pose features .

[0099] Step S3: Migration and fusion;

[0100] To effectively integrate facial visual features and 3D head pose features from heterogeneous sources with significant differences in modal structure, thereby improving the overall perception of driving fatigue behavior, the system introduces a module-level transfer learning strategy through a transfer fusion module. Combined with a feature embedding fusion mechanism, it constructs a unified feature representation space across modalities and tasks. The two modules in steps S1 and S2, which are pre-trained on training datasets from different sources, are fused in a learnable manner through the transfer fusion module. The fused features are then converted into a unified high-dimensional fusion feature representation through trainable linear mapping parameters and nonlinear activation functions.

[0101] The transfer fusion module in step S3 includes: freezing the pre-trained facial visual feature extraction module and head pose estimation module in steps S1 and S2 during training, and only fine-tuning the fusion and classification parts to achieve unified embedding and end-to-end adaptation of heterogeneous features; concatenating the facial feature vector extracted in step S1 with the head pose feature vector obtained in step S2 to form a joint feature representation, and transforming it through a set of trainable linear mapping matrices and nonlinear activation functions to obtain high-dimensional fusion features;

[0102] Training strategy:

[0103] 1. Pre-training: The YOLOv11n-cls lightweight detection network is pre-trained on a face dataset to obtain visual feature extraction capabilities; the FSANet architecture is pre-trained on a head pose dataset to obtain head pose perception capabilities.

[0104] 2. Freeze: When jointly training on the driving fatigue dataset, freeze the convolutional layer parameters of the two networks (Setting requires_grad = False) so that they are used only as fixed feature extractors.

[0105] 3. Fine-tuning: Gradient updates are performed only on the parameters of the newly added "fusion layer" and "classification layer";

[0106] Feature concatenation: The facial feature vector output from step S1 is then processed. and the head feature vector output in step S2 Concatenate along the channel dimension to obtain the joint vector. , ];

[0107] Feature mapping: The input is fed into a fully connected layer for feature fusion and mapping. This feature fusion and mapping process can be formally expressed as:

[0108]

[0109] In the formula, Indicates facial features, Indicates head posture characteristics. It is a learnable linear mapping matrix. It is a non-linear activation function;

[0110] First, sub-feature extraction modules are constructed for the two modalities respectively, using a facial feature extractor and a head pose parameter regressor, and are pre-trained independently on datasets from different sources to ensure that the two sub-modules can fully learn and extract the most discriminative potential representation information in their respective modalities.

[0111] After pre-training is completed, the main parameters of the above sub-modules are frozen in the subsequent joint training stage, and only a few fusion layers are fine-tuned, so as to ensure the stability of transfer learning, reduce parameter redundancy and speed up convergence.

[0112] During the fusion process, the system uses tensor splicing to integrate two modal features from different sources along the channel dimension to form a unified input structure. Then, the spliced ​​feature tensor is sent into the fusion mapping network for cross-modal feature unification.

[0113] This mapping network consists of a series of learnable linear mapping matrices and nonlinear activation units such as ReLU. Its role is to alleviate problems such as dimensional misalignment, scale inconsistency, and uneven distribution in the original modality space, and to improve the expressive power and semantic consistency of features through nonlinear transformation, thereby effectively constructing joint embedded features containing multimodal complementary information.

[0114] The core advantage of this module-level transfer fusion strategy lies in its seamless transition from pre-trained feature extraction to feature space embedding. While preserving the original semantic structure of each modality, it breaks down modal barriers through an embedding alignment mechanism, constructing a joint feature representation with stronger discriminative power and higher robustness, thus providing a solid foundation for the final classification of driving fatigue behavior. Experimental results show that the model using this fusion strategy exhibits better convergence speed and classification accuracy under multimodal data input, especially demonstrating good generalization ability under non-ideal conditions such as occlusion and blurring.

[0115] Step S4: Classification decision-making and hierarchical feedback;

[0116] The state classification and feedback module, based on the fused feature representation in step S3, uses a fully connected neural network combined with the Softmax function to perform multi-class classification and output fatigue detection results;

[0117] Classification:

[0118] The fused feature vector is input into the output layer, and after normalization by the Softmax function, the result is obtained. ;

[0119] Decision logic:

[0120] like Determined to be conscious

[0121] like And sustained frame rate threshold If so, it is determined to be drowsy. In this embodiment, It can be 15 detectives;

[0122] like And sustained frame rate threshold If it is yawning, then it is determined to be yawning. In this embodiment, It can be 30.

[0123] Feedback Execution:

[0124] Awake: System silence, reset counters;

[0125] Drowsiness (High Risk): Immediately triggers a high-frequency buzzer alarm and emits a continuous alert tone;

[0126] Yawning (medium risk): Play a voice prompt and detect a short, sharp sound to remind the driver to adjust their state;

[0127] The fused feature vector will be fed into the fatigue classification and feedback module as input. This module has a multi-layer fully connected neural network structure and is supplemented by Batch Normalization and Dropout mechanisms, which can effectively prevent overfitting while improving the model's expressive power.

[0128] Finally, the Softmax output layer categorizes the driver's state into three types: "awake", "yawning", and "drowsy". Based on the recognition results, the system automatically activates different levels of voice prompt mechanisms: no prompts are given for the awake state, short prompts are given for yawning, and continuous prompts are given for drowsy state, thereby providing timely risk intervention for the driver.

[0129] The core innovation of the entire driver fatigue detection system lies in the integration of a dual-stream transfer learning feature fusion mechanism and a confidence-based weighted training strategy, which fundamentally solves key problems such as the difficulty of heterogeneous modality fusion and the inaccurate identification of samples with ambiguous boundaries.

[0130] The system constructs two feature extraction paths: facial visual flow and 3D head pose flow. These paths are pre-trained based on the improved YOLOv11n-cls lightweight detection network and the FSANet architecture, respectively. Subsequently, through a module-level transfer learning strategy, feature sharing and unified embedding are achieved while maintaining the expressive power of sub-modules, effectively improving the consistency and discriminative ability of cross-modal fusion.

[0131] Meanwhile, to improve the model's ability to perceive subtle signs of fatigue (such as squinting, smiling, and narrowed eye openings), a hierarchical weighted loss mechanism based on sample ambiguity is introduced during training, significantly enhancing the model's robustness and sensitivity in fuzzy feature recognition. The overall system architecture follows lightweight and modular principles, facilitating deployment on edge computing platforms such as in-vehicle control units. The network design controls computational resource overhead while ensuring feature expressiveness, achieving low-latency, real-time inference performance. This method demonstrates excellent adaptability and stability in various driving environments, possessing high practical value and broad engineering potential.

[0132] Example 3:

[0133] This embodiment provides a hardware system for a dual-stream feature embedding driving fatigue detection system based on transfer learning as described in Embodiment 1, including:

[0134] Hardware platform: The system is deployed on the NVIDIA Jetson Orin Nano edge computing module, which provides approximately 40 TOPS of AI computing power, sufficient to support real-time inference of a dual-stream network.

[0135] Image acquisition: A 1080P global shutter near-infrared camera is used. The camera is installed on the A-pillar of the driver's seat or in front of the steering wheel and is equipped with a 940nm infrared fill light to ensure that clear grayscale facial images can still be obtained in a completely dark environment at night. The infrared light is invisible and will not interfere with the driver's vision.

[0136] Inference optimization: The model is accelerated using the TensorRT engine. The YOLOv11n-cls lightweight detection network model is exported to ONNX format and converted to TensorRT Engine (FP16 accuracy). The FSANet architecture model is also converted to TensorRT Engine.

[0137] The fusion module, as a lightweight MLP, is implemented directly through CUDA kernel functions or merged into the main network.

[0138] Performance metrics: On Orin Nano, the end-to-end inference latency of the dual-stream system is controlled within 30ms, meeting the requirements for real-time monitoring. The total number of model parameters is less than 5MB, resulting in low memory usage.

[0139] In summary, this invention improves the model's perception accuracy for fuzzy samples through a weighted training mechanism and addresses the modal gap in heterogeneous feature fusion through a module transfer strategy. Ultimately, it constructs a multimodal driver fatigue detection system that is structurally efficient, features uniformly represented, and easy to deploy. This system not only achieves low-latency, real-time inference in an in-vehicle environment but also exhibits excellent noise resistance and stability in complex visual interference scenarios. It provides a reliable and practical engineering solution for driver state monitoring in intelligent driving assistance systems, possessing promising prospects for widespread application and industrialization value.

[0140] Therefore, this invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0141] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A dual-stream feature embedding driving fatigue detection system based on transfer learning, characterized in that, include: The facial feature extraction module is used to receive driver facial images captured by the vehicle-mounted camera in actual driving conditions; The head pose estimation module processes the head pose dataset to capture driver fatigue behavior. The transfer fusion module is used to learnably fuse facial visual feature vectors with head pose feature vectors, converting them into a high-dimensional fused feature representation. The status classification and feedback module is used to perform multi-class classification on high-dimensional fusion features, output fatigue detection results, and provide real-time driving status judgment and warning. The head pose estimation module adopts the FSANet architecture, which has a dual-stream multi-scale feature extraction mechanism to fully capture the local and global information of head pose. At the same time, a pixel-level attention weight mapping module is introduced to enhance the response capability to key head regions.

2. A driving fatigue detection method based on transfer learning and dual-stream feature embedding, characterized in that, The dual-stream feature embedding driving fatigue detection system based on transfer learning as described in claim 1, wherein the dual-stream feature embedding driving fatigue detection method based on transfer learning includes: S1. Facial visual feature extraction and weighted training: The facial feature extraction module extracts dynamic visual features related to fatigue through the YOLOv11n-cls lightweight detection network. S2. Head pose estimation and regression: The head pose estimation module uses a regression model based on the FSANet architecture to output the angle values ​​of the driver's head in the pitch, yaw and roll directions, capturing the driver's fatigue behavior. S3. Transfer Fusion: The two modules in steps S1 and S2, which are pre-trained on training datasets from different sources, are fused in a learnable manner through the transfer fusion module, and converted into a unified high-dimensional fusion feature representation through trainable linear mapping parameters and nonlinear activation functions. S4. Classification Decision and Feedback: Based on the high-dimensional fusion feature representation obtained in step S3, the state classification and prompt feedback module uses a fully connected neural network combined with the Softmax function to perform multi-class classification and output fatigue detection results.

3. The dual-stream feature embedding driving fatigue detection method based on transfer learning according to claim 2, characterized in that: In step S1, the facial feature extraction module introduces a weighted training strategy, including the following steps: S11. Divide the samples in the training set into normal samples and blurred samples; the blurred samples include facial image samples with squinting, smiling, and eye openings smaller than a preset width threshold. S12. In the model training process, a hierarchical weighted cross-entropy loss function is introduced. The loss value of fuzzy samples is multiplied by the weight coefficient ω to improve the learning intensity of the model for fuzzy samples. S13. Determine the optimal weight parameter value ω through experiments to enhance the model's ability to discriminate fuzzy fatigue features in complex visual scenes.

4. The dual-stream feature embedding driving fatigue detection method based on transfer learning according to claim 2, characterized in that: The head pose estimation module in step S2 adopts the FSANet architecture and includes: Based on this architecture, a dual-stream multi-scale feature extraction mechanism is provided to fully capture local and global information of head pose. At the same time, a pixel-level attention weight mapping module is introduced to enhance the response capability to key head regions. Finally, a multi-stage classification stepwise regression learning module is used to achieve high-precision 3D pose estimation of pitch, yaw and roll angles. The multi-stage classification stepwise regression learning module adopts the Soft Stagewise Regression module.

5. The dual-stream feature embedding driving fatigue detection method based on transfer learning according to claim 4, characterized in that: The Soft Stagewise Regression module transforms the continuous angle regression problem into a staged classification and refinement regression problem. It predicts the discrete interval to which each angle belongs and the relative offset within the interval, including the discrete interval to which pitch angle, yaw angle, and roll angle belong and the relative offset within the interval. The expected value of the probability distribution is calculated to obtain the final pitch, yaw, and roll angle values.

6. The dual-stream feature embedding driving fatigue detection method based on transfer learning according to claim 2, characterized in that: The migration and fusion module in step S3 includes: During training, the pre-trained parameters of the facial visual feature extraction module and the head pose estimation module in steps S1 and S2 are frozen. Only the fusion and subsequent classification modules are jointly fine-tuned to achieve unified embedding and end-to-end adaptation of heterogeneous features. The facial feature vector extracted in step S1 and the head pose feature vector obtained in step S2 are concatenated to form a joint feature representation. This representation is then transformed using a set of trainable linear mapping matrices and nonlinear activation functions to obtain high-dimensional fusion features.

7. The dual-stream feature embedding driving fatigue detection method based on transfer learning according to claim 6, characterized in that: In step S3, the feature mapping process can be represented as follows: ; In the formula, This refers to the driver's facial input image captured by the vehicle's onboard camera. This indicates that step S1 extracts the data based on the input image. facial features This indicates that step S2 obtains the result based on the input image. Head posture characteristics, This indicates a feature concatenation operation. It is a learnable linear mapping matrix. It is a non-linear activation function. This represents the final high-dimensional fusion feature.

8. The dual-stream feature embedding driving fatigue detection method based on transfer learning according to claim 2, characterized in that: In step S4, a multi-layer fully connected neural network structure is used for the fused feature vector obtained in step S3. Normalization processing, non-linear activation function and Dropout strategy are combined to improve the generalization ability. At the output end, the high-dimensional fused features are mapped to the probability distribution of three driving states: awake, drowsy and yawning through the Softmax function to complete the fatigue state classification.

9. The dual-stream feature embedding driving fatigue detection method based on transfer learning according to claim 8, characterized in that: In step S4, the system outputs corresponding prompts based on the predicted categories obtained from the classification. No prompts are triggered when the driver is awake, a short warning sound is triggered when the driver is drowsy, and a longer prompt sound is triggered when the driver is yawning, so as to enhance the intelligent reminder and intervention effect on driver fatigue behavior.

10. The dual-stream feature embedding driving fatigue detection method based on transfer learning according to claim 2, characterized in that: The YOLOv11n-cls lightweight detection network used in step S1 includes a C3k2 module, an SPPF module, and a C2PSA module. The C3k2 module is based on a cross-stage local network structure and uses a variable kernel size to extract features at different depth levels. The SPPF module is used to perform spatial pyramid pooling on the feature map to fuse multi-scale information. The C2PSA module introduces a spatial attention mechanism, which enhances the saliency of fatigue-related feature extraction by improving the feature response to the eye and mouth regions.

Images