A lightweight fatigue driving detection model and method for a complex occlusion scene

Abstract

The application discloses a kind of light fatigue driving detection model and method for complex occlusion scene, including feature extraction backbone network module, mixed encoder module and decoder module.Feature extraction backbone network module includes light benchmark network and residual channel-space attention module RCSAB, for enhancing the local feature representation capability of key fatigue area such as eye, mouth under occlusion scene;Mixed encoder module includes adaptive multi-scale feature modulation fusion module MFM and occlusion perception modulation module OAM, respectively for realizing the dynamic fusion of cross-scale feature, inhibiting occlusion noise propagation, and according to global occlusion state, feature is adaptively modulated;Decoder module includes transformer decoder, for the target query and fatigue state prediction of enhanced feature.The application can improve the precision and robustness of fatigue driving detection under complex occlusion scene under the premise of keeping model lightweight, meet the real-time deployment needs of vehicle-mounted edge device.

Description

Technical Field

[0001] This invention relates to the field of fatigue driving detection technology, specifically to a lightweight fatigue driving detection model and method for complex obstruction scenarios, which is particularly suitable for real-time detection of fatigue status when drivers wear sunglasses, masks, and the steering wheel obstructs key areas of their face. Background Technology

[0002] Fatigue driving is one of the core risk factors causing road traffic accidents. Drivers often experience partial or complete obstruction of key facial areas (such as the eyes and mouth) due to sunglasses, masks, or steering wheels, posing a significant challenge to accurate fatigue detection. Detection methods based on driver facial features are more feasible due to their high accuracy and lack of interference with normal driving. Therefore, researching a detection method that is highly robust under occlusion conditions and meets the requirements for in-vehicle deployment has become a major need in the field of driver assistance.

[0003] With the development of deep learning, convolutional neural networks (CNNs) have become the mainstream approach for fatigue driving detection due to their feature extraction capabilities. Existing technologies utilize multi-task cascaded networks and self-built EM-CNNs to accurately extract regions of interest (ROIs) for the eyes, and combine these with PERCLOS and mouth opening indicators to achieve accurate fatigue identification. Multiple lightweight networks independently extract facial key points and poses, enabling fatigue detection even when the driver is wearing sunglasses or a mask. Introducing a self-calibrating illumination enhancement module and a lightweight backbone network into YOLOv5s significantly improves detection accuracy in low-light environments. Other solutions improve the YOLOv10 algorithm by utilizing multi-scale dilated convolutions and global grouped coordinate attention mechanisms to effectively enhance the ability to lock onto small targets. Although continuously optimized CNN architectures show significant improvements in real-time performance and local perception, their inherent local receptive fields struggle to effectively extract scattered facial key features under extremely severe occlusion, and they also lack the ability to model the semantics of global occlusion.

[0004] To overcome the aforementioned shortcomings, the Transformer architecture and its variants, capable of modeling long-distance dependencies, have been gradually introduced into the field of fatigue detection. Some techniques utilize global self-attention mechanisms for modeling, effectively overcoming interference from wearing sunglasses and complex lighting conditions. To further balance local perception and global modeling, new combinations integrating local feature extraction with global attention mechanisms have emerged. For example, by cascading standard Transformer encoders, multi-head attention mechanisms are used to capture the spatial global dependencies of facial features, effectively addressing local occlusion; convolutional and attention fusion modules are designed to achieve deep fusion of local receptive fields and global context; lightweight visual Transformer variants are introduced, significantly improving the ability to recognize subtle facial expressions and hand overlap occlusion; and by combining depthwise separable convolutions with low-complexity cross-attention mechanisms, efficient inference on the vehicle side is achieved with extremely low parameter counts. Although the above-mentioned architectures integrating self-attention mechanisms have significant advantages in global perception and occlusion resistance, occlusion noise still diffuses to deeper features during cross-scale feature stitching, and the model's responsiveness to key local structures such as the eyes and mouth is significantly weakened after lightweighting.

[0005] Based on the above-mentioned architecture of integrating self-attention mechanism, RT-DETR, as an end-to-end detection framework that combines accuracy and real-time performance, can better adapt to vehicle-mounted devices with limited computing power. However, its application in complex occlusion scenarios still has the following defects: (1) The backbone network has insufficient feature perception ability for fatigue discrimination areas such as eyes and mouth, and lightweighting further weakens the representation ability of mid-layer features; (2) The efficient hybrid encoder completes feature fusion through cross-scale feature static splicing, which cannot filter out noise caused by occlusion, resulting in a decrease in the quality of multi-scale facial feature fusion; (3) The model lacks a global occlusion state perception and control mechanism, and cannot dynamically adjust feature response weights according to facial occlusion. Invalid features will interfere with the decoding process, reducing detection accuracy and robustness. Summary of the Invention

[0006] To overcome the shortcomings of the existing technology, this invention provides a lightweight fatigue driving detection model and method for complex occlusion scenarios. While maintaining the model's lightweight nature, it enhances the perception capability of key local areas in occlusion scenarios, suppresses occlusion noise interference in cross-scale feature fusion, and adaptively adjusts feature responses according to the global occlusion state, thereby improving the accuracy and robustness of fatigue driving detection in complex occlusion scenarios.

[0007] The present invention adopts the following technical solution to solve the technical problem.

[0008] This invention discloses a lightweight fatigue driving detection model for complex occlusion scenarios, including a feature extraction backbone network module, a hybrid encoder module, and a decoder module; The feature extraction backbone network module is used to introduce a residual channel-spatial attention mechanism to enhance the local perception ability of key fatigue feature areas of the eyes and mouth; the feature extraction backbone network module includes a lightweight baseline network and a residual channel-spatial attention module RCSAB; The hybrid encoder module is used to achieve dynamic filtering and adaptive fusion of cross-scale features through dynamic weights, and to predict the occlusion state based on the global context to dynamically adjust the feature response; the hybrid encoder module includes an adaptive multi-scale feature modulation and fusion module (MFM) and an occlusion-aware modulation module (OAM). The decoder module is used to perform target query and iterative decoding on the enhanced features and output fatigue state detection results; the decoder module includes a Transformer decoder.

[0009] The lightweight fatigue driving detection model for complex occlusion scenarios of the present invention is also characterized by: The lightweight baseline network is obtained by pruning and compressing the RT-DETR model.

[0010] The RCSAB module adopts a residual connection structure, including a local feature extraction unit, a channel attention unit, and a dynamic spatial attention unit; The local feature extraction unit includes two cascaded 3×3 convolutions, a batch normalization layer, and a Leaky ReLU activation function; The dynamic spatial attention unit is used to generate dynamic convolution kernels in real time based on input features to achieve content-adaptive spatial modulation.

[0011] The MFM module is used to sequentially perform channel-level adaptive modulation on input features of different scales, generate cross-scale dynamic fusion weights based on global context information, and obtain multi-scale fusion features through weighted summation.

[0012] The OAM module includes an occlusion state sensing unit and a feature linear modulation unit; The occlusion state perception unit is used to predict the occlusion category probability through global average pooling and a lightweight convolutional network. The feature linear modulation unit is used to generate scaling and offset coefficients for the channel dimension based on the predicted occlusion probability, and modulates the original features in the form of residuals.

[0013] In the lightweight benchmark network, the RCSAB module is embedded in the mid-layer feature extraction stage of the lightweight benchmark network.

[0014] This invention also discloses a lightweight fatigue driving detection method for complex occlusion scenarios, which includes the following steps: Step 1: Feature extraction step; Extract multi-scale feature maps of the input driver's facial image through a lightweight backbone network embedded with RCSAB modules; Step 2: Multi-scale noise-resistant fusion step; The MFM module is used to dynamically filter and adaptively weight and fuse feature maps of different scales to obtain multi-scale fused features; Step 3: Global Occlusion Modulation Step; The global occlusion state is predicted based on the global context of the fused features using the OAM module, and the features are then linearly modulated at the channel level. Step 4: Decoding and output step; Input the modulated features into the Transformer decoder, and output the detection results of the driver's fatigue state after iterative decoding.

[0015] The lightweight fatigue driving detection method of the present invention for complex occlusion scenarios is also characterized by: In step 1, the RCSAB module performs preliminary feature extraction on the input features by performing two 3×3 convolutions, batch normalization, and Leaky ReLU activation in sequence. Then, it sequentially passes through channel attention calibration and dynamic spatial attention modulation, and finally fuses with the input features through residual connections.

[0016] In step 2, the MFM module first performs channel modulation on the input features at each scale, then aggregates the modulated features, generates dynamic fusion weights for each scale based on the global context of the aggregated features, and finally performs weighted summation on the features at each scale according to the dynamic fusion weights to obtain the fused features.

[0017] In step 3, the OAM module first extracts the macroscopic semantics of the features through global average pooling and predicts the probability distribution of the occlusion state; then it generates the scaling factor and offset factor of the channel dimension according to the probability distribution; finally, it modulates the original features in the form of residuals.

[0018] In step 4, the Transformer decoder uses an intersection-union-ratio (IU)-aware query selection strategy to filter the initial query vector and outputs the detection result after multiple rounds of iterative decoding.

[0019] Compared with existing technologies, the beneficial effects of this invention are reflected in: (1) Design the RCSAB module, which strengthens the model’s local perception of key fatigue feature areas such as eyes and mouth by combining channel attention and dynamic spatial attention mechanisms. This effectively overcomes the defect that the response capability to local key structures is weakened after lightweighting. At the same time, dynamic spatial attention generates dynamic convolution kernels in real time according to input features, and accurately locks the discriminative regions that are not occluded under occlusion conditions. (2) The MFM module is designed to replace the traditional static stitching method. It performs adaptive modulation of input features at different scales at the channel level and generates cross-scale dynamic fusion weights based on the global context to realize dynamic selection and adaptive fusion of features, effectively suppress the propagation of occlusion noise in cross-scale feature fusion, and improve the fusion quality of multi-scale facial features. (3) Design the OAM module, extract macro semantics and predict the probability distribution of occlusion state through global average pooling, and generate the scaling coefficient and offset coefficient of the channel dimension accordingly. The feature is linearly modulated in the form of residual, so that the model can dynamically adjust the feature response weight according to the global occlusion state and enhance the decoder's adaptability to occlusion scene. (4) The three modules RCSAB, MFM and OAM work together to achieve a model mAP50 of 91.4% with only 10.2M parameters and 41.8GFLOPs of computation, which is 2.1 percentage points higher than the original RT-DETR model. In the fatigue driving scenario with occlusion, the mAP50 reaches 94.0%, which is significantly better than mainstream target detection models such as YOLO series, RT-DETR series and D-FINE. It achieves an effective balance between detection accuracy, lightweight and anti-occlusion robustness, and meets the requirements of vehicle edge deployment. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of the detection model in an embodiment of the lightweight fatigue driving detection model and method for complex occlusion scenarios of the present invention.

[0021] Figure 2 This is a structural diagram of the RCSAB module in the detection model of an embodiment of the present invention.

[0022] Figure 3 This is a structural diagram of the MFM module in the detection model of this invention.

[0023] Figure 4 This is a structural diagram of the OAM module in the detection model of an embodiment of the present invention.

[0024] Figure 5 This is a scatter plot comparing the detection performance of embodiments of the present invention.

[0025] Figure 6 This is a comparison diagram of detection heatmaps from an embodiment of the present invention.

[0026] The present invention will be further described below through specific embodiments and in conjunction with the accompanying drawings. Detailed Implementation

[0027] See Figures 1-6This invention discloses a lightweight fatigue driving detection model and method for complex occlusion scenarios. The detection model includes a feature extraction backbone network module, a hybrid encoder module, and a decoder module. The feature extraction backbone network module is used to introduce a residual channel-spatial attention mechanism to enhance the local perception ability of key fatigue feature areas of the eyes and mouth; the feature extraction backbone network module includes a lightweight baseline network and a residual channel-spatial attention module RCSAB; The hybrid encoder module is used to achieve dynamic filtering and adaptive fusion of cross-scale features through dynamic weights, and to predict the occlusion state based on the global context to dynamically adjust the feature response; the hybrid encoder module includes an adaptive multi-scale feature modulation and fusion module (MFM) and an occlusion-aware modulation module (OAM). The decoder module is used to perform target query and iterative decoding on the enhanced features and output fatigue state detection results; the decoder module includes a Transformer decoder.

[0028] This invention presents a lightweight fatigue driving detection model for complex occlusion scenarios. Its lightweight baseline network is obtained by pruning and compressing the RT-DETR model, significantly reducing the number of parameters and computational cost while laying the foundation for on-vehicle edge deployment. The feature extraction backbone network module embeds an RCSAB module in the mid-layer feature extraction stage of the lightweight baseline network to enhance the perception of key fatigue discrimination regions such as the eyes and mouth in occlusion scenarios. It also effectively overcomes the weakening of local feature representation capabilities caused by lightweighting by combining channel attention and dynamic spatial attention mechanisms, maintaining low computational cost. The hybrid encoder module incorporates an MFM module, which uses dynamic weights to dynamically select and adaptively fuse cross-scale features to suppress the propagation of occlusion noise during feature fusion. An OAM module uses global context to predict the occlusion state and performs channel-level adaptive modulation of features to dynamically adjust feature response weights according to the global occlusion state. The decoder module employs a Transformer decoder, using an intersection-union-ratio (IUGR)-aware query selection strategy to filter high-quality initial query vectors and outputs highly robust driver fatigue state detection results after multiple rounds of iterative decoding. While maintaining the lightweight characteristics of the model, this invention effectively improves the accuracy and robustness of fatigue driving detection in complex occlusion scenarios, and is suitable for real-time detection and intelligent early warning in vehicle edge deployment environments.

[0029] In specific implementation, the lightweight benchmark network includes multiple convolutional layers; the RCSAB module is embedded in the mid-layer feature extraction stage of the lightweight benchmark network.

[0030] like Figure 1As shown, the lightweight baseline network takes a 640×640×3 original driver facial image as input, processes it through multiple convolutional layers and an RCSAB module, and outputs multi-scale feature maps to the hybrid encoder module. The feature extraction backbone network module combines a lightweight feature extraction structure with a residual channel-spatial attention mechanism, which reduces the number of model parameters and computational complexity while enhancing the feature discrimination of key fatigue discrimination regions such as the eyes and mouth in occluded scenes.

[0031] In specific implementation, the RCSAB module adopts a residual connection structure, including a local feature extraction unit, a channel attention unit, and a dynamic spatial attention unit.

[0032] The local feature extraction unit includes two cascaded 3×3 convolutions, a batch normalization layer, and a Leaky ReLU activation function.

[0033] like Figure 2 As shown, the RCSAB module first performs preliminary feature extraction and smoothing on the input features through a local feature extraction unit. Then, it inputs the features into a channel attention unit, performing adaptive weight allocation based on the channel dimension to enhance channels containing key facial information and suppress irrelevant background noise. Next, the enhanced features are input into a dynamic spatial attention unit. This unit first generates a dynamic convolution kernel in real-time based on the input features using a lightweight kernel generator, then performs dynamic convolution on the channel-averaged spatial descriptor. After activation by the Sigmoid function, a spatial attention weight map is generated and multiplied element-wise with the input features to achieve content-adaptive spatial modulation. Finally, the modulated features are fused with the original input features through residual connections and output. Through this structure, the RCSAB module can adaptively adjust the spatial attention region according to the current occlusion situation. Even when facial key points are partially occluded, it can still accurately lock the unoccluded discriminative region, significantly improving the robustness of fatigue detection under occlusion conditions.

[0034] like Figure 1 As shown, the multi-scale feature map output by the lightweight baseline network enters the hybrid encoder module. In the hybrid encoder module, the MFM module is placed before the OAM module, and the output features of the MFM module serve as the input features of the OAM module.

[0035] In specific implementation, the MFM module is used to sequentially perform channel-level adaptive modulation on input features of different scales, generate cross-scale dynamic fusion weights based on global context information, and obtain multi-scale fusion features through weighted summation.

[0036] like Figure 3As shown, the MFM module receives input features from adjacent scales. First, it independently modulates the input features at the channel level using global average pooling and a lightweight mapping function to evaluate the effectiveness of each feature under the current sample conditions. Then, it aggregates the modulated features and generates cross-scale dynamic fusion weights based on the global context information of the aggregated features using a multilayer perceptron. Finally, it performs a weighted summation of the features at each scale according to the generated dynamic weights to obtain the final multi-scale fused features. Through this process, the MFM module can adaptively adjust the relative weights of features at different scales during the fusion process according to the overall state of the input features, thereby effectively suppressing occlusion noise during multi-scale information interaction and achieving more stable feature fusion.

[0037] In specific implementation, the OAM module includes an occlusion state sensing unit and a feature linear modulation unit.

[0038] The occlusion state perception unit is used to predict the occlusion category probability through global average pooling and a lightweight convolutional network. The feature linear modulation unit is used to generate scaling and offset coefficients for the channel dimension based on the predicted occlusion probability, and modulates the original features in the form of residuals.

[0039] like Figure 4 As shown, the OAM module first performs global average pooling on the input features through an occlusion state awareness unit to extract macroscopic semantics. Then, a lightweight network consisting of two 1×1 convolutions predicts the occlusion state probability distribution of the current sample. Next, a feature linear modulation unit generates scaling and offset coefficients for the channel dimensions through a linear mapping layer based on the predicted occlusion probability distribution. Finally, the original features are modulated in residual form, outputting the modulated features. To ensure stability during the initial training phase, the linear layers corresponding to the scaling and offset coefficients employ a zero-initialization strategy, ensuring the module initially behaves as an identity mapping. Through this mechanism, the OAM module enables the model to perceive the current macroscopic occlusion pattern and adaptively adjust the feature representation, effectively eliminating occlusion interference and ensuring that the features input to the decoder have a high signal-to-noise ratio.

[0040] like Figure 1 As shown, the modulated features output by the OAM module enter the decoder module. In specific implementation, the decoder module includes a Transformer decoder, which uses an intersection-union-ratio (IU)-aware query selection strategy to filter the initial query vector and outputs the detection result after multiple rounds of iterative decoding.

[0041] This invention also discloses a lightweight fatigue driving detection method for complex occlusion scenarios, which includes the following steps: Step 1: Feature extraction step; Extract multi-scale feature maps of the input driver's facial image through a lightweight backbone network embedded with RCSAB modules; Step 2: Multi-scale noise-resistant fusion step; The MFM module is used to dynamically filter and adaptively weight and fuse feature maps of different scales to obtain multi-scale fused features; Step 3: Global Occlusion Modulation Step; The global occlusion state is predicted based on the global context of the fused features using the OAM module, and the features are then linearly modulated at the channel level. Step 4: Decoding and output step; Input the modulated features into the Transformer decoder, and output the detection results of the driver's fatigue state after iterative decoding.

[0042] In specific implementation, in step 1, the RCSAB module performs preliminary feature extraction on the input features by performing two 3×3 convolutions, batch normalization, and Leaky ReLU activation in sequence, and then performs channel attention calibration and dynamic spatial attention modulation in sequence, and finally fuses with the input features through residual connections.

[0043] When the lightweight backbone network extracts features from the original driver's facial image, it follows formulas (1) to (5). Let the feature mapping input to the RCSAB module be... The features are first processed through two concatenated 3×3 convolution operations, followed by batch normalization (BN) and Leaky ReLU activation functions, to output the feature matrix. This enables preliminary extraction of local spatial information and feature smoothing. (1) In formula (1), Leaky ReLU is chosen instead of standard ReLU in order to avoid excessive suppression of weak negative feature responses when the lightweight model has limited channels, thereby effectively preserving the facial discrimination information and structural context that become weak due to interference in occluded scenes.

[0044] Subsequently, features Adaptive weight allocation is performed within the channel attention unit: (2) In Equation (2), CA represents the channel attention operation, which is used to enhance the channel containing key facial information while suppressing irrelevant background noise interference.

[0045] Channel Enhanced Features The data is fed into the dynamic spatial attention unit. This unit first generates dynamic convolutional kernels using a lightweight kernel generator. : (3) In formula (3), GAP represents global average pooling, Conv1 and Conv2 represent 1×1 convolutions, and δ represents the ReLU activation function. The generated initial dimensions... for Subsequently, it was reshaped in the spatial dimension into Dynamic convolution kernels.

[0046] At the same time, by analyzing the channel dimension Calculate the average to obtain the spatial descriptor. Utilizing the reshaped dynamic convolution kernel right Dynamic convolution operations are performed, followed by activation using the Sigmoid function, to generate the final spatial attention weight map. : (4) Finally, output features The original feature map is obtained by element-wise multiplication of the spatial weight map: (5) In formula (5), This represents element-wise multiplication. By introducing a dynamic spatial attention mechanism with sample awareness, the RCSAB module can adaptively adjust the spatial attention region according to the current occlusion situation, enabling the model to accurately lock the unoccluded discriminative region through the weights generated by the dynamic kernel even when facial key points are partially occluded.

[0047] In specific implementation, in step 2, the MFM module first performs channel modulation on the input features at each scale, then aggregates the modulated features, generates dynamic fusion weights for each scale based on the global context of the aggregated features, and finally performs weighted summation on the features at each scale according to the dynamic fusion weights to obtain the fused features.

[0048] Let the input features of adjacent scales be respectively ,in This represents the upsampled high-level features. MFM first performs channel-level adaptive modulation on the input features within their respective scales; that is, for each layer of input features... Channel-level context information is extracted using global average pooling, and modulation weights are generated using a lightweight mapping function. (6) In formula (6), This indicates element-wise multiplication. It is the Sigmoid activation function. This is the ReLU activation function. This process independently modulates features at different scales, achieving preliminary filtering of their internal information.

[0049] After completing intra-scale modulation, MFM further adaptively models the relative contributions between features at different scales. First, the modulated features are aggregated: (7) Subsequently, cross-scale fusion weights are generated based on the global contextual information of the fusion features: (8) In formula (8), Represents the dynamic weight coefficients assigned to features at different scales, satisfying .

[0050] Finally, the multi-scale fusion features are obtained through a weighted summation method: (9) Through the above process, the MFM module can adaptively adjust the relative weights of features at different scales in the fusion process according to the overall state of the input features, thereby achieving more stable feature fusion in the process of multi-scale information interaction and effectively suppressing the propagation of occlusion noise.

[0051] In specific implementation, in step 3, the OAM module first extracts the macroscopic semantics of the features through global average pooling and predicts the probability distribution of the occlusion state; then it generates the scaling factor and offset factor of the channel dimension according to the probability distribution; finally, it modulates the original features in the form of residuals.

[0052] Let the characteristics of the input OAM module be: The OAM module first extracts the macroscopic semantics of the feature map using global average pooling, and then predicts the probability distribution of the occlusion state through a lightweight convolutional network. : (10) (11) In formulas (10) and (11), This represents the output of the state sensor. The total number of predefined occlusion states (in this embodiment) (These correspond to no occlusion, eye occlusion, and mouth occlusion, respectively). The ReLU activation function is used. This calculation process compresses the spatial dimension through GAP, extracts global statistical information containing the receptive field of the entire image, and then uses a classification network to output the probability response of the current sample in different occlusion categories, enabling the model to perceive global occlusion.

[0053] To achieve adaptive state adjustment at the feature level, the OAM module introduces a feature linear modulation mechanism, which generates scaling coefficients for the channel dimension based on the predicted occlusion probability distribution P using a linear mapping function. With offset coefficient : (12) In formula (12), The weight matrix of the linear mapping layer. For bias terms, This indicates a spatial dimension reshaping operation, used to expand two-dimensional channel features into a form similar to the input features. A four-dimensional tensor that matches the spatial dimensions.

[0054] Finally, feature modulation is applied to the original features in the form of residuals to obtain the output features. : (13) In formula (13), This represents the feature map after occlusion-sensing modulation. and These represent element-wise addition and element-wise multiplication, respectively. This formula performs feature modulation by introducing residual connections, generating parameters... and Apply fine-tuning to the original features To ensure stability during the initial training phase, the linear layers related to the modulation parameter generator all employ a zero-initialization strategy, meaning that the weight matrix is ​​initialized to zero during the initial training phase. With bias term All values ​​are initialized to zero, making the module behave approximately like an identity mapping. .

[0055] In specific implementation, in step 4, the Transformer decoder uses an intersection-union-ratio (IUU)-aware query selection strategy to filter the initial query vector, and outputs the detection result after multiple rounds of iterative decoding.

[0056] To verify the effectiveness of the lightweight fatigue driving detection method for complex occlusion scenarios in this invention, experiments were conducted using a self-built complex occlusion fatigue driving dataset. This dataset is a mixture of data from the publicly available dataset YawDD (see Abtahi S, Omidyeganeh M, Shirmohammadi S, et al. YawDD: a yawning detection dataset [C] / / Proceedings of the 5th ACM Multimedia Systems Conference. 2014: 24-28.) and the self-built dataset. It covers situations where drivers wear sunglasses, masks, and the steering wheel, causing partial occlusion of key facial areas under both strong and normal lighting conditions. The annotations are divided into six categories: no occlusion fatigue, no occlusion fatigue, eye occlusion fatigue, eye occlusion fatigue, mouth occlusion fatigue, and mouth occlusion fatigue. The dataset is divided according to driver ID. The training set contains 12,035 images, the validation set contains 5,533 images, and the test set contains 1,749 images, with an overall ratio of 6:3:1.

[0057] The trained weights were tested on the test set, and the detection results were compared with those of YOLOv8n, YOLOv8s, YOLOv8m (see Jocher Glenn.Ultralytics yolov8.https: / / github.com / ultralytics / ultralytics,2023), YOLOv9s (see Wang CY, Yeh IH, Liao H YM.YOLOv9:Learning what you want to learn using programmable gradient information[C] / / European Conference on Computer Vision.Cham:Springer NatureSwitzerland,2024:1-21.), YOLOv10n, YOLOv10s, YOLOv10m (see Wang A, Chen H, Liu L, et al.YOLOv10:Real-time end-to-end object detection[J].Advances in Neural Information Processing) Systems, 2024, 37:107984-108011.), YOLO11n, YOLO11s, YOLO11m algorithms (see Khanam R, Hussain M. YOLOv11: An overview of the key architectural enhancements[J]. arXiv preprint arXiv:2410.17725,2024.), YOLOv12n, YOLOv12s, YOLOv12m algorithms (see Tian Y, Ye Q, Doermann D. YOLOv12: Attention-centric real-time object detectors[J]. Advances in Neural Information Processing Systems, 2026, 38:78433-78457.), YOLO26n, YOLO26m algorithms (see Sapkota R, Cheppally RH, Sharda A, et al.).YOLO26: Key architectural enhancements and performance benchmarking for real-time object detection[J].arXiv preprintarXiv:2509.25164,2025.), RTDETR-resnet50, RTDETR-resnet101, RT-DETR-l algorithm (for details, see Zhao Y, Lv W, Xu S, et al. DETRs beat YOLOs on real-time object detection[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2024:16965-16974.), D-FINE algorithm (see Peng Y, Li H, Wu P, et al. D-FINE: Redefine regression task of DETRs as fine-grained distribution refinement[C] / / International Conference on Learning Representations. 2025, 2025: 44015-44031.), and the ME-YOLOv8 algorithm (see Debsi A, Ling G, Al-Mahbashi M, et al. Driverdistraction and fatigue detection in images using ME-YOLOv8 algorithm[J]. IET Intelligent Transport Systems, 2024, 18(10): 1910-1930.) were compared using objective data on accuracy (P), recall (R), average precision (mAP50 / mAP50-95), number of model parameters (Params), and gigabit floating-point computation (GFLOPs). P represents the proportion of samples predicted as positive that are actually positive, R represents the probability of samples that are actually positive being predicted as positive, and mAP50 / mAP50-95 represents the average detection precision when the IoU of the bounding box is greater than or equal to 0.5 / 0.5:0.95, as shown in Tables 1 and 2 below.

[0058] Table 1 Comparison of experimental results for different detection models As shown in Table 1, the method of this invention achieves an mAP50 of 91.4% with only 10.2M parameters and 41.8 GFLOPs of computation, significantly outperforming all the lightweight models compared. Compared to the YOLO11m, which has higher accuracy in the YOLO series, the method of this invention achieves an mAP50 1.3 percentage points higher while reducing the number of parameters by approximately 49% and the computation by approximately 38%. Compared to the baseline model RT-DETR-1, the method of this invention improves the mAP50 by 2.1 percentage points while reducing the number of parameters by approximately 68% and the computation by approximately 60%.

[0059] Compared to high-performance models such as RT-DETR-1, although they perform better in some accuracy metrics, these models generally suffer from excessive parameter count and computational complexity, making them unsuitable for deployment in computationally limited environments such as automotive environments. In contrast, the method of this invention significantly reduces model complexity while maintaining detection accuracy, making it more suitable for practical engineering applications.

[0060] Based on the above analysis, the method of this invention enhances local perception through the RCSAB module, suppresses fusion noise through the MFM module, and achieves global occlusion state perception and channel-level feature modulation through the OAM module. The three work together to effectively alleviate the accuracy loss caused by lightweighting in fatigue driving detection tasks, and exhibit more stable and reliable detection performance in complex occlusion scenarios, thus having high engineering application value.

[0061] Table 2 Comparative experimental results of different detection models under occlusion fatigue conditions Key facial areas such as the eyes and mouth are important indicators of fatigue behaviors such as closing the eyes and yawning. However, occlusions such as sunglasses, masks, and steering wheels can easily lead to the loss of these features, posing a challenge for fatigue detection. To test the robustness of the model under different occlusion types, Table 2 shows the performance comparison of each model in the occlusion fatigue category on the same dataset. The experimental results show that the method of this embodiment still maintains an mAP50 of 94.0% under this extreme condition. Even compared with high-precision models (such as YOLO11m, RT-DETR-1, and RT-DETR-resnet101) with significantly more parameters, the detection accuracy of the method of this embodiment remains the highest, demonstrating its good feature recognition and discrimination capabilities even when facing complex local occlusions.

[0062] Figure 5This is a bubble scatter plot comparing the performance of the method of this invention and some comparative models in terms of parameter quantity, accuracy, and mAP50, where the bubble size represents the number of parameters. The figure clearly shows that the method of this invention achieves the highest accuracy and mAP50 under the condition of a smaller number of parameters, fully verifying the superior balance between lightweight design and high precision achieved by this invention.

[0063] Figure 6 This is a comparison of Grad-CAM++ heatmaps of the method and some comparative models under different occlusion conditions, such as sunglasses and masks. The visualization shows that conventional models like YOLOv8n are highly susceptible to strong visual interference from objects like sunglasses and masks. Their high-response areas (red areas) are often concentrated on the surfaces of occluders that lack fatigue semantic information, such as sunglasses lenses or masks, exhibiting attentional shifts or fragmented distribution, significantly weakening their focus on fatigue-related actions. In contrast, the method of this invention demonstrates significant occlusion perception and region removal capabilities: when faced with large-area occlusions such as sunglasses or masks, the model can accurately identify visual interference and classify it as invalid information, applying feature suppression and presenting these areas as cool-colored regions in the heatmap; simultaneously, it positions the high-response red areas in key regions that represent fatigue-related actions—when the driver is yawning, the red area in the heatmap focuses on the open jaw and mouth contour; when drowsy, the red area is concentrated at the edge of the closed eyelids. The visualization results fully demonstrate the synergistic effectiveness of the various modules of this invention: the model not only successfully cuts off the propagation of occlusion noise in cross-scale feature fusion, but also dynamically captures and utilizes the remaining effective features when local key features are occluded, thereby achieving a high-confidence fatigue state determination.

[0064] Based on a comprehensive analysis of both subjective and objective detection results, the method of this invention, through the coordinated operation of three modules—RCSAB, MFM, and OAM—achieves a model mAP50 of 91.4% with only 10.2M parameters and 41.8 GFLOPs of computation, and even reaches 94.0% in fatigue driving scenarios with occlusion. This significantly outperforms mainstream target detection models such as the YOLO series, RT-DETR series, and D-FINE, effectively balancing detection accuracy, lightweight design, and robustness against occlusion, and can meet the real-time fatigue driving detection requirements in vehicle edge deployment environments.

[0065] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0066] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A lightweight fatigue driving detection model for complex occlusion scenarios, characterized in that, It includes a feature extraction backbone network module, a hybrid encoder module, and a decoder module; The feature extraction backbone network module is used to generate dynamic convolution kernels in real time based on input features through the dynamic spatial attention mechanism in the embedded residual channel-spatial attention module RCSAB to achieve content-adaptive spatial modulation and enhance the local perception ability of key fatigue feature regions of the eyes and mouth in occluded scenes; the feature extraction backbone network module includes a lightweight baseline network and a residual channel-spatial attention module RCSAB. The hybrid encoder module is used to generate cross-scale dynamic fusion weights based on the global context of aggregated features through the adaptive multi-scale feature modulation and fusion module MFM, so as to realize the dynamic selection and adaptive fusion of cross-scale features to suppress occlusion noise. It also uses the occlusion-aware modulation module OAM to predict the occlusion state based on the macroscopic semantics extracted by global average pooling, and generates channel dimension scaling and offset coefficients to linearly modulate the features in the form of residuals. The hybrid encoder module includes the adaptive multi-scale feature modulation and fusion module MFM and the occlusion-aware modulation module OAM. The decoder module is used to perform target query and iterative decoding on the enhanced features and output fatigue state detection results; the decoder module includes a Transformer decoder.

2. The lightweight fatigue driving detection model for complex occlusion scenarios according to claim 1, characterized in that, The lightweight baseline network includes multiple convolutional layers. The lightweight baseline network is obtained by performing structured pruning and compression on the RT-DETR model: by compressing the feature layers in the backbone network and adjusting the encoder-decoder depth and number of channels, the number of parameters and computational cost are reduced to meet the real-time requirements of vehicle edge deployment. The problem of decreased discriminative power of mid-layer features after lightweight compression is addressed by the RCSAB module embedded in the mid-layer feature extraction stage of the backbone network. The RCSAB module is embedded in the mid-level feature extraction stage of the lightweight baseline network. The embedding position is after the mid-level convolutional layer and before the high-level feature output. It is used to enhance the representation ability of mid-level features for key fatigue discrimination regions such as eyes and mouth in occluded scenes and to make up for the decrease in the discriminative power of mid-level features caused by lightweight compression. The RCSAB module adopts a residual connection structure, including a local feature extraction unit, a channel attention unit, and a dynamic spatial attention unit. The local feature extraction unit consists of two cascaded 3×3 convolutions, a batch normalization layer, and a LeakyReLU activation function. The dynamic spatial attention unit includes a dynamic convolution kernel generation branch and a spatial weight generation branch. The dynamic convolution kernel generation branch is used to generate dynamic convolution kernels in real time based on input features, and the spatial weight generation branch is used to perform dynamic convolution operations on spatial descriptors and activate and generate a spatial attention weight map.

3. The lightweight fatigue driving detection model for complex occlusion scenarios according to claim 1, characterized in that, The MFM module includes multiple parallel intra-scale modulation branches, feature aggregation units, and dynamic weight generation units; each intra-scale modulation branch is used to perform channel-level adaptive modulation on the input features of the corresponding scale. The feature aggregation unit is used to aggregate the modulated features; The dynamic weight generation unit is used to generate dynamic fusion weights at various scales based on the global context of aggregated features.

4. A lightweight fatigue driving detection model for complex occlusion scenarios as described in claim 1, characterized in that, The OAM module includes an occlusion state sensing unit and a feature linear modulation unit; The occlusion state perception unit is composed of a global average pooling layer and a lightweight convolutional network connected in series, and is used to predict the probability of occlusion category. The feature linear modulation unit is used to generate scaling and offset coefficients for the channel dimension based on the predicted occlusion probability, and modulates the original features in the form of residuals.

5. A lightweight fatigue driving detection method for complex occlusion scenarios, implemented based on the model described in any one of claims 1-4, characterized in that, Includes the following steps: Step 1: Feature extraction. Multi-scale feature maps of the input driver's facial image are extracted using a lightweight backbone network embedded with RCSAB modules. Step 2: Multi-scale noise-resistant fusion, dynamically filtering and adaptively weighting the feature maps at different scales to obtain multi-scale fused features; Step 3: Global occlusion modulation. Predict the global occlusion state based on the global context of the fused features, and perform channel-level linear modulation on the features. Step 4: Decoding output. Input the modulated features into the Transformer decoder, and output the detection results of the driver's fatigue state after iterative decoding.

6. A lightweight fatigue driving detection method for complex occlusion scenarios according to claim 5, characterized in that, In step 1, let the input feature be... After local spatial feature extraction through two 3×3 convolutions, batch normalization, and Leaky ReLU activation, the output feature matrix is ​​obtained. After channel attention calibration, the output is... ; In dynamic spatial attention modulation, After global average pooling, a dynamic convolution kernel is generated through 1×1 convolution. Simultaneously, the spatial descriptor is obtained by averaging the channel dimensions. With dynamic convolution kernels For space descriptors Perform convolution operations and activate the spatial weight map using the Sigmoid function. Spatial weight map Input features Element-wise multiplication is used to perform spatial modulation and output features. Modulated features Through residual connection and module input Output the sum of each element.

7. A lightweight fatigue driving detection method for complex occlusion scenarios according to claim 5, characterized in that, In step 2, let the adjacent scale input features be... The MFM module performs channel modulation on the input features at each scale: channel-level context is extracted through global average pooling, modulation weights are generated by a lightweight mapping function and sigmoid activation, and then multiplied element-wise with the original features to complete intra-scale modulation, outputting the modulation weights. Features modulated at each scale are summed and aggregated to form... Aggregation features Dynamic fusion weights at various scales are generated by a multilayer perceptron. ,satisfy Normalization constraints; finally, modulated features at each scale. According to the corresponding dynamic fusion weight Weighted summation, outputting multi-scale fused features .

8. A lightweight fatigue driving detection method for complex occlusion scenarios according to claim 5, characterized in that, In step 3, let the input feature be... The OAM module processes input features Global average pooling is performed to compress spatial dimensions and extract macroscopic semantics. The occlusion state probability distribution is then predicted using a lightweight convolutional network. ; The linear mapping layer will convert the probability distribution Scaling factor mapped to channel dimension and offset coefficient After spatial reshaping, it is consistent with the input features. Dimension matching; finally, the original features are modulated channel-by-channel in residual form and output. The scaling factor and offset coefficient The corresponding linear layers employ a zero-initialization strategy, ensuring that the module behaves as an identity mapping during the initial training phase. .

9. A lightweight fatigue driving detection method for complex occlusion scenarios according to claim 5, characterized in that, The detection model is trained using transfer learning, with the RT-DETR pre-trained model used as the initial weights. During training, a fatigue driving dataset containing three scenarios—unobstructed, eye-occluded, and mouth-occluded—was used for supervised learning. Random erasure, Mosaic, and Mixup data augmentation strategies were combined to improve the model's generalization ability to complex occlusion scenarios. In the later stages of training, Mosaic data augmentation was turned off, and mixed precision training was used to optimize the model parameters to improve the model's convergence speed and detection accuracy.

Images