A driver dangerous driving detection method based on structure perception state space convolution network

Abstract

A driver dangerous driving detection method based on structure perception state space convolutional network belongs to the technical field of intelligent traffic safety and computer vision, and particularly relates to a driver state detection method based on improved YOLOv11. The method comprises the following steps: acquiring an input image containing a driver region, and performing size scaling and normalization preprocessing on the input image; inputting the preprocessed image into an improved YOLOv11s target detection network; in the backbone network and feature fusion neck network of the network, replacing part of the standard convolutional layers with SWConv convolutional modules to expand the effective receptive field; compared with the prior art, the improved method can improve the detection accuracy and robustness under complex illumination, occlusion and slender small target scenes while maintaining high inference efficiency.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and computer vision technology, specifically to a driver state detection method and system based on an improved YOLOv11s network, which can be used in driver monitoring systems for various vehicles such as passenger cars and commercial vehicles. Background Technology

[0002] With the development of intelligent driving and active safety technologies, driver state monitoring systems are playing an increasingly important role in preventing fatigued and distracted driving. Existing driver state monitoring methods are mostly based on facial key point localization, eye opening and closing estimation, and manually designed feature extraction, which are relatively sensitive to changes in lighting, occlusion, and posture, and lack robustness.

[0003] In recent years, YOLO series object detection methods have been widely used in driver status monitoring scenarios, enabling real-time detection of targets such as driver faces, mobile phones, cigarettes, and seat belts. However, most existing methods directly adopt general object detection network structures, lacking targeted structural optimization for slender and small targets such as mobile phones, cigarettes, seat belts, and small slits around the eyes in in-vehicle scenes. Furthermore, they are prone to missed detections and false detections in nighttime, backlighting, and partially occluded scenarios.

[0004] To improve detection performance, existing methods have attempted to introduce large convolutional kernels, attention mechanisms, or Transformer structures. However, large convolutional kernels typically increase the number of parameters and computational cost significantly, and self-attention structures often have high complexity, which is detrimental to maintaining inference efficiency. Therefore, it is necessary to provide a driver state detection method that maintains high inference efficiency while also taking into account large receptive field modeling and slender structure enhancement. Summary of the Invention

[0005] The purpose of this invention is to provide a driver state detection method based on an improved YOLOv11, in order to solve the problems of insufficient detection accuracy and robustness of existing methods in complex lighting, occlusion, and slender small target scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: Based on the YOLOv11s target detection network, in the backbone network and the feature fusion neck network, some standard convolutional layers are replaced with SWConv convolutional modules to expand the effective receptive field; a structure-aware state space attention module is set on the high-level feature map of the feature fusion neck network to enhance the topological connectivity and texture continuity features of slender targets such as seat belts, mobile phones, cigarettes, and eyes; the enhanced multi-scale features are input into the detection head, the detection results of driver-related targets are output, and the dangerous driving state is further determined.

[0007] Furthermore, the driver's condition is determined based on the detection results: When a mobile phone is detected simultaneously in the driver's area, it can be determined that the user is using a mobile phone while driving; when a cigarette is detected, it can be determined that the user is smoking. When a seatbelt is missing or its condition is abnormal, it can be determined that the driver is not wearing a seatbelt; when the driver's eyes are closed or show signs of fatigue, it can be determined that the driver is fatigued. The corresponding driver condition recognition result is then output.

[0008] Compared with the prior art, the present invention has the following beneficial effects: First, the effective receptive field of mid-to-high-level features is expanded by using the SWConv convolution module, which improves the ability to perceive the contextual information of the driver's upper body region and slender targets. Second, the geometric continuity and long-range dependencies of slender targets are explicitly modeled by the structure-aware state-space attention module, which improves the detection accuracy of targets such as seat belts, mobile phones, cigarettes and fine cracks in the eyes. Third, while maintaining the YOLOv11 multi-scale detection framework, the overall robustness in complex scenarios is improved.

[0009] While maintaining high inference speed, this method can effectively improve the detection accuracy (mean accuracy, mAP) and recall rate of key slender small targets such as mobile phones, cigarettes, seat belts, and eye slits under challenging conditions such as complex lighting and partial occlusion. This improves the robustness and reliability of the entire driver state detection system and provides more accurate technical support for assisted driving safety warnings. Attached Figure Description

[0010] Figure 1 This is a flowchart of the driver state detection method in an embodiment of the present invention.

[0011] Figure 2 This is a schematic diagram of the improved YOLOv11 overall network structure in an embodiment of the present invention.

[0012] Figure 3 This is a schematic diagram of the key positions of the skeleton and neck of the improved YOLOv11 based on SWConv in an embodiment of the present invention.

[0013] Figure 4 This is a schematic diagram of the structure-aware state space attention module in an embodiment of the present invention.

[0014] Figure 5 This is a schematic diagram of the detection results of the method described in the embodiments of the present invention in a real-world scenario. Detailed Implementation

[0015] Example 1: Overall Method Flow Step S1: Construction and preprocessing of multi-scenario driver state image dataset A driver state image dataset covering various real-world driving scenarios was constructed. These scenarios include those with drastic lighting changes, partial target occlusion, and scenarios containing elongated small targets. Partial target occlusion scenarios include hands obscuring the face and steering wheels obscuring the body. Elongated small targets include objects such as a driver holding a mobile phone, a cigarette, a seatbelt diagonally across the chest, and the elongated area around the eyes, which indicates fatigue.

[0016] The acquired or obtained image data is preprocessed, including at least size standardization, illumination normalization, and data augmentation operations. At the same time, targets in the image such as the driver's face, eye area, mobile phone, cigarette, and seat belt are labeled at the bounding box level, and the driver's state category is labeled according to the needs of the detection task to form a dataset for model training and validation.

[0017] Step S2: Construction of a driver state detection network based on the improved YOLOv11 Using the standard YOLOv11 network as the basic detection framework, improvements were made to its backbone feature extraction and feature fusion neck section to construct a lightweight target detection network suitable for driver state detection. The improved network enhances its feature representation capabilities for small, elongated targets such as mobile phones, cigarettes, seat belts, and the narrow area around the eyes by introducing the SWConv convolutional module and the SAVSS structure-aware visual state space module.

[0018] Step S2.1: Improvement of backbone and neck convolutional structures based on SWConv In some standard convolutional locations of the YOLOv11s network, the SWConv convolutional module is used to replace the original standard convolutions to enhance the network's ability to model long-distance spatial relationships and the overall morphology of slender structures.

[0019] The SWConv convolutional module extracts multi-scale spatial information from input features through parallel standard depthwise separable convolutional branches and dilated depthwise separable convolutional branches. It then performs feature fusion through pointwise convolution, expanding the effective receptive field while maintaining low parameter count and computational complexity. Through these improvements, the network can better represent the overall geometric structure and contextual information of targets such as the long side of a mobile phone, the extension direction of a seatbelt, the shape of a cigarette stick, and the slender region of the eye.

[0020] Step S2.2: High-level feature structure perception enhancement based on SAVSS In the feature fusion neck of the YOLOv11s network, the SAVSS structure-aware visual state space module is embedded at the high-level feature scale to enhance the structure of high semantic features. Preferably, the SAVSS module is set on the branches corresponding to the feature maps with downsampling steps of 16 and 32, that is, at the fusion output positions of the P4 and P5 high-level feature maps.

[0021] The SAVSS module is used to explicitly model the topological connectivity and texture continuity of slender structural targets on a two-dimensional feature map. Based on a selective state-space model, this module rearranges the feature map into a one-dimensional feature sequence according to a preset scanning path, and achieves continuous modeling of multi-directional structural information through state recursion. The scanning path includes at least horizontal and diagonal scanning paths, thereby enhancing the network's ability to perceive the continuous structure of targets such as seat belts, cigarettes, mobile phone edges, and slender regions of the eye.

[0022] Step S2.3: Multi-scale feature fusion and detection output The multi-scale features extracted from the backbone network are fused in the neck network in a top-down and bottom-up manner to form a multi-scale feature map for detection.

[0023] After the high-level feature map is processed by SWConv and SAVSS, the enhanced features and low-level detail features are input together into the YOLOv11s multi-scale detection head to output the location and category information of targets such as driver's face, eye area, mobile phone, cigarette, and seat belt.

[0024] Through the above improvements, the network can take into account both high-level semantic information and local structural details while maintaining the lightweight characteristics of YOLOv11s, thereby improving the detection accuracy and stability in complex lighting, partial occlusion, and slender small target scenarios.

[0025] Step S3: Model Training and Parameter Optimization The improved YOLOv11s network constructed in step S2 was trained end-to-end using the dataset constructed in step S1 to obtain the trained network weight parameters.

[0026] During training, the loss function includes at least bounding box regression loss, classification loss, and target confidence loss. Preferably, higher loss weights are assigned to small or elongated target samples such as mobile phones, cigarettes, seat belt buckles, and the narrow eye area to improve the model's learning ability for these types of targets. By iteratively optimizing the parameters of the SWConv module, SAVSS module, and detection head, the network gradually converges on the training and validation sets, obtaining a target model suitable for driver state detection tasks.

[0027] Step S4: Driver Status Detection and Result Output The image of the driver to be detected is input into the trained improved YOLOv11s network. The backbone network, feature fusion neck and multi-scale detection head are used to complete feature extraction, structure enhancement and target detection in sequence, and output the detection results of targets such as driver's face, eye area, mobile phone, cigarette, seat belt.

[0028] Furthermore, the driver's condition is determined based on the detection results: When a mobile phone is detected simultaneously in the driver's area, it can be determined that the user is using a mobile phone while driving; when a cigarette is detected, it can be determined that the user is smoking. When a seatbelt is missing or its condition is abnormal, it can be determined that the driver is not wearing a seatbelt; when the driver's eyes are closed or show signs of fatigue, it can be determined that the driver is fatigued. The corresponding driver condition recognition result is then output.

[0029] Example 2: Improved YOLOv11 network structure based on SWConv In this embodiment, the YOLOv11 object detection network is selected as the basic framework. While keeping the overall multi-scale detection structure unchanged, some standard convolutions in the backbone feature extraction module and the feature fusion neck module are replaced, and the SWConv convolution module is used to achieve lightweight large receptive field modeling.

[0030] The SWConv convolutional module is used to achieve approximate modeling of a larger receptive field without significantly increasing the kernel size. For an input feature with C channels and X input feature map, the SWConv convolutional module includes: a first depthwise separable convolutional branch, a second dilated depthwise separable convolutional branch, and a pointwise convolutional fusion layer. The first depthwise separable convolutional branch uses a 3×3 convolutional kernel to perform local neighborhood aggregation on the input feature, and the output is denoted as Y1; the second dilated depthwise separable convolutional branch uses a 3×3 convolutional kernel with a preset dilation rate d to perceive the spatial context over a larger range, and the output is denoted as Y2; then Y1 and Y2 are summed or concatenated by channel, and channel fusion is performed through 1×1 pointwise convolution to obtain the fused feature Yf.

[0031] The fused features, after batch normalization and nonlinear activation, are residually concatenated with the input features X to obtain the output features X^ of the SWConv convolutional module. Its expression can be written as: in, W represents the convolution operation, W1×1 represents the pointwise convolution weights, BN represents the batch normalization operation, and σ represents the nonlinear activation function.

[0032] In the network structure, the SWConv convolutional module is used to replace some standard convolutions in the feature layers of the backbone network with downsampling strides of 8 and 16 to enhance the contextual receptive field for slender small targets such as eyes, mobile phones, cigarettes, and seat belts. At the same time, in the high-level feature branches of the feature fusion neck network, the SWConv convolutional module is also used to replace some standard convolutions to further integrate high-level semantic information and mid-level texture information.

[0033] Example 3: Structure-Aware State-Space Attention Module In this embodiment, the structure-aware state space attention module is set on the high-level feature maps P4 and P5 of the feature fusion neck network, and is located after feature fusion and before the detection head, in order to enhance the feature representation ability of slender targets such as seat belts, mobile phones, cigarettes and eyes.

[0034] The structure-aware state-space attention module includes a gated bottleneck convolution submodule and a structure-aware state-space submodule. The gated bottleneck convolution submodule performs channel compression and morphological feature extraction on the input features through 1×1 dimensionality reduction convolution, 3×3 depthwise convolution, and 1×1 dimensionality increase convolution, and uses a gating function to modulate the passing ratio of features in different channels, thereby highlighting the relevant channels of slender structural targets.

[0035] For the input feature sequence w(t), the structure-aware state-space submodule adopts a continuous-time state-space model: Where z(t) is the hidden state, u(t) is the output feature, and A, B, C, and D are learnable parameter matrices. The continuous-time state-space model is discretized according to the time step Δ, resulting in: , The corresponding discrete recursive formula is: , Where wk represents the input feature at the k-th position, zk represents the corresponding hidden state, uk represents the module output, and R, S, ΔP0, and ΔQ0 are parameter matrices learned through training.

[0036] To adapt to the elongated, tilted, and variable-directional geometric structures of targets such as seat belts, mobile phones, and cigarettes in driver status monitoring scenarios, this embodiment introduces a structure-aware scanning strategy in the structure-aware state space submodule. Specifically, the high-level feature map is rearranged into a one-dimensional feature sequence along at least two directions, preferably including horizontal and diagonal directions. Discrete state space recursion is applied to the input sequence on each path, and the output sequences of different paths are remapped back to two-dimensional space and fused in the channel dimension to form a structure-enhanced feature map. Through this strategy, the hidden state can propagate continuously along the target structure, establishing long-range dependencies, thereby improving the network's response strength and topological consistency to elongated targets.

[0037] Example 4: Training and State Determination During training, the improved network was trained end-to-end using a dataset containing labeled targets such as driver's face, eyes, mobile phone, cigarette, seat belt buckle, and seat belt body. The loss function included bounding box regression loss, classification loss, and confidence loss, with higher weights assigned to small targets with areas below a preset threshold to improve the detection accuracy of mobile phones, cigarettes, and partially obscured seat belt targets.

[0038] During inference, the network outputs detection bounding boxes and category information for driver-related targets. Then, based on the detection results, a state determination is made: if the driver is detected and seatbelt-related targets are missing or not closed, it is determined that the driver is not wearing a seatbelt; if a mobile phone target is detected simultaneously with the driver's head area and meets spatial association conditions, it is determined that the driver is making or receiving a phone call while driving; if a cigarette target is detected and meets spatial association conditions with the driver's mouth or hand area, it is determined that the driver is smoking; if closed-eye state is detected in multiple consecutive frames, it is determined that the driver is experiencing eye strain.

Claims

1. A method for detecting dangerous driving by drivers based on a structure-aware state-space convolutional network, characterized in that, Includes the following steps: S1, acquire an input image containing the driver's area, and perform size scaling and normalization preprocessing on the input image; S2, input the preprocessed image into the improved YOLOv11s object detection network; S3, in the backbone network and feature fusion neck network of the improved YOLOv11s object detection network, some standard convolutional layers are replaced with SWConv convolutional modules to expand the effective receptive field and obtain enhanced multi-scale features. S4, a structure-aware state space attention module is set on the high-level feature map of the feature fusion neck network to enhance the features of the slender structural target; S5 inputs the enhanced multi-scale features into the detection head and outputs the detection results of the driver's face, eyes, mobile phone, cigarette and seat belt related targets; S6. Based on the detection results, determine whether the driver is not wearing a seat belt, making or receiving phone calls while driving, smoking, or in a state of fatigue with eyes closed, and output the determination result.

2. The method for detecting dangerous driving by drivers based on structure-aware state-space convolutional networks according to claim 1, characterized in that, The improved YOLOv11 target detection network includes a backbone feature extraction module, a feature fusion neck module, and a multi-scale detection head.

3. The method for detecting dangerous driving by a driver based on a structure-aware state-space convolutional network according to claim 1 or 2, characterized in that, The SWConv convolutional module includes a first depthwise separable convolutional branch and a second dilated depthwise separable convolutional branch set in parallel, and a pointwise convolutional fusion layer set after the parallel branches; the first depthwise separable convolutional branch uses a 3×3 convolutional kernel, and the second dilated depthwise separable convolutional branch uses a 3×3 convolutional kernel with a preset dilation rate; the output of the parallel branches is fused by pointwise convolution and then residually connected with the input features to obtain the output features of the SWConv convolutional module; the SWConv convolutional module is used to replace some standard convolutions in the backbone network feature layers with downsampling strides of 8 and 16, and to replace some standard convolutions in the high-level feature branches in the feature fusion neck network.

4. The method for detecting dangerous driving by a driver based on a structure-aware state-space convolutional network according to any one of claims 1 to 3, characterized in that, The structure-aware state-space attention module includes a gated bottleneck convolution submodule and a structure-aware state-space submodule. The gated bottleneck convolution submodule performs channel compression and morphological feature extraction on the input features through 1×1 dimensionality reduction convolution, 3×3 depthwise convolution, and 1×1 dimensionality increase convolution, and modulates the passing ratio of different channel features through a gating function. The structure-aware state-space submodule rearranges the input feature map into a one-dimensional feature sequence along at least two directions, and scans and updates the feature sequences in each direction based on the state-space recursive structure to obtain structural feature maps in different directions. The structural feature maps are then fused to form a slender structure enhancement feature map.

5. The method for detecting dangerous driving by a driver based on a structure-aware state-space convolutional network according to any one of claims 1 to 8, characterized in that, The detection results include at least one or more of the following: driver's face, eyes open, eyes closed, mobile phone, cigarette, seat belt buckle, and seat belt body.

6. The method for detecting dangerous driving by a driver based on a structure-aware state-space convolutional network according to any one of claims 1 to 5, characterized in that, During training, higher weights are given to the detection loss of small targets with an area below a preset threshold in order to improve the detection accuracy of targets such as mobile phones, cigarettes and partially obscured seat belts.

Images