Transform-YOLO-based driver fatigue detection optimization method

Abstract

The invention discloses a driver fatigue detection optimization method based on a Transform-YOLO (Young Line Offset). Comprising the following steps of data acquisition, data preprocessing, target detection, key region feature extraction, multi-modal feature fusion and fatigue state classification. The method comprises the following steps: firstly, detecting a face and key areas in a video stream of a driver through a YOLOv11 model, wherein the face and key areas comprise postures of eyes, a mouth and a head; the method comprises the following steps: firstly, extracting a target detection result, then extracting spatio-temporal characteristics by utilizing a key region attention (KRA) module, and splicing the target detection result and the spatio-temporal characteristics in combination with a multi-modal characteristic fusion technology; and finally, realizing fatigue state judgment through a classifier. According to the method, the problems of calculation redundancy and insufficient real-time performance of the traditional Transform in feature modeling of the driver face key areas (eyes and mouths) are solved.

Description

Technical Field

[0001] This invention relates to an optimized method for driver fatigue detection based on Transformer-YOLO, specifically, a method for driver fatigue detection based on the fusion of key region attention Transformer and YOLOv11. Background Technology

[0002] With the accelerating pace of automotive intelligence, real-time and accurate detection of driver fatigue has become a core requirement of intelligent driving safety systems. Vision-based detection technologies, by capturing dynamic changes in key areas of the driver's face (such as the degree of eye closure and mouth opening / closing), have become the most promising solutions. Among them, the YOLO series of object detection models are widely used in in-vehicle real-time systems due to their balance between detection speed and accuracy.

[0003] Currently, existing fatigue detection models suffer from severe computational redundancy when directly incorporating the Transformer's global self-attention mechanism. The core criteria for determining driver fatigue focus on the area around the eyes and mouth, but traditional Transformers perform global correlation modeling of all pixels in the entire image, resulting in significant computational resource consumption in non-critical regions. At the feature fusion level, the traditional YOLOv11 multi-scale feature pyramid network is weakly adaptable to features in key regions. The size of a driver's face varies considerably across video frames; when facial pixels account for less than 5%, detailed features in small target areas such as the area around the eyes and mouth are easily ignored by shallow convolutional layers, while deep semantic features suffer from location inaccuracies due to their large receptive fields. Limitations in training strategies further exacerbate the model's robustness issues. Traditional detection models use uniformly weighted classification and regression losses without supervising signal enhancement for key regions such as the area around the eyes and mouth. In extreme scenarios such as low light at night or when the driver is wearing sunglasses, the model is prone to misclassifying textures in non-critical regions as fatigue features, leading to a higher false alarm rate.

[0004] In summary, existing technologies have significant shortcomings in terms of efficient modeling of key fatigue features, optimization of computational resources, and adaptability to complex scenarios. There is an urgent need for a lightweight improvement scheme that combines domain prior knowledge to enhance the feature representation capabilities of the periorbital and oral regions while maintaining real-time detection efficiency, so as to meet the actual needs of vehicle safety systems for high precision and low latency. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to propose an optimized method for driver fatigue detection based on Transformer-YOLO, which solves the problems of computational redundancy and insufficient robustness in complex scenarios of existing technologies by using lightweight design and key area feature enhancement, and achieves high-precision real-time fatigue detection.

[0006] Technical solution: The driver fatigue detection optimization method based on Transformer-YOLO described in this invention includes the following steps:

[0007] (1) Data acquisition;

[0008] (2) Data preprocessing;

[0009] (3) Target detection;

[0010] (4) Key region feature extraction;

[0011] (5) Multimodal feature fusion;

[0012] (6) Classification of fatigue state.

[0013] Furthermore, in step (1), data acquisition refers to: acquiring driver video stream data and dividing the video stream into non-overlapping frame sequences;

[0014] Specifically, the process involves acquiring a video dataset of drivers in various states, including normal driving, eyes closed, yawning, and head tilting. The video stream is then segmented into non-overlapping frame sequences, with each sequence of 5 frames forming a temporal unit {T1, T2, ..., T...}. n}, with adjacent units spaced 2 frames apart.

[0015] Furthermore, in step (2), data preprocessing involves processing each frame of the image; the specific steps are as follows:

[0016] (21) Apply an adaptive histogram equalization algorithm to each frame of the image for illumination normalization, and output the time-series frame sequence after illumination normalization. Equation (1) is defined as the histogram equalization algorithm:

[0017] I out (x,y)=CLAHE(I in (x,y),N tiles =8,C limit =2) (1)

[0018] (22) The established timestamp index table can assist in subsequent temporal correlation analysis of time series features or screening of motion-blurred frames, recording the acquisition time t of each frame. i and motion ambiguity coefficient b i Equation (2) is defined as the motion fuzzy coefficient b. i :

[0019]

[0020] Furthermore, in step (3), target detection refers to: using the YOLOv11 model to detect the face region in the video frame in real time, and outputting the face bounding box and key region coordinates; wherein, the key region includes the eye region, the mouth region and the head pose angle.

[0021] Furthermore, step (3) specifically includes the following steps:

[0022] (31) Load the pre-trained YOLOv11 model and match it with the size of the pre-processed first frame image; the loaded model then processes the illumination-normalized temporal frame sequence, detects the face region in the video frame in real time, and outputs the face bounding box, key region coordinates and head pose angle.

[0023] (32) YOLOv11 detection is performed on the first frame of the temporal unit. The model extracts and analyzes features of the first frame image through its internal multi-layer network structure.

[0024] (33) Output eye heatmap H using YOLOv11 model eye H mouth ;

[0025] (34) The model outputs the head pose Euler angles (α, β, γ), which reflect the head pose from different dimensions; Equation (3) defines the method for calculating the head Euler angles:

[0026]

[0027] (35) After the YOLOv11 model performs preliminary detection on the first frame image, multiple face bounding boxes are generated. At this time, nonmaximum suppression is used to filter and merge these candidate boxes.

[0028] (36) The ROI tracking algorithm is used to process non-first frames; Equation (4) is defined as the ROI tracking algorithm:

[0029]

[0030] Furthermore, the key region feature extraction in step (4) refers to: cropping the key region image based on the coordinates output in step (2) and inputting it into the key region attention Transformer module for spatiotemporal feature extraction.

[0031] Furthermore, step (4) specifically includes the following steps:

[0032] (41) Based on the heatmap coordinates output in step (33), crop the eye area.

[0033] (42) Input the ROI images of 5 consecutive frames into the key region attention Transformer module;

[0034] (43) Temporal modeling uses a Transformer encoder with L=3 layers and FFN=512 feedforward network;

[0035] (44) Output spatiotemporal feature vectors Equation (5) is defined as the method for calculating spatiotemporal feature vectors:

[0036] f st =AvgPool(TransformerEncoder(z0)) (5).

[0037] Furthermore, in step (5), multimodal feature fusion refers to concatenating the head pose angle output by YOLOv11 with the spatiotemporal feature vector extracted by the key region attention Transformer to form fused features.

[0038] Furthermore, step (5) specifically includes the following steps:

[0039] (51) The spatiotemporal features of step (44) are spliced ​​with the head pose angle of step (34);

[0040] (52) Filter effective features through the feature selection layer;

[0041] (53) Dynamically standardize feature fusion.

[0042] Furthermore, the fatigue state classification in step (6) refers to: inputting the fused features into the classifier and outputting the fatigue state classification result; specifically, it includes the following steps:

[0043] (61) Input the features processed in step (53) into the two-level classifier, and then judge them; Equation (6) is defined as the fatigue judgment rule:

[0044]

[0045] (62) When FatigueFlag=1 is detected three times in a row, an alarm is triggered.

[0046] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: Through a key-region-oriented lightweight Transformer design, this invention focuses the self-attention mechanism on core fatigue feature areas such as the driver's eye area and mouth, significantly reducing computational overhead while maintaining real-time detection efficiency, effectively adapting to the computing power limitations of in-vehicle embedded devices; the key-region spatiotemporal feature extraction and multimodal feature fusion strategies enhance the detection capability of subtle fatigue features and improve feature recognition under different resolutions, poses, and lighting conditions; targeted optimizations for complex scenarios such as occlusion and low light significantly enhance the model's adaptability to special situations such as drivers wearing sunglasses and masks, reducing false positives and false negatives. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the overall operation of the present invention;

[0048] Figure 2 This is a flowchart of the feature splicing and standardization process of the present invention;

[0049] Figure 3 This is a diagram of the Transformer-YOLO network architecture of the present invention. Detailed Implementation

[0050] The specific technical solution of the present invention will be further described in detail below with reference to specific examples.

[0051] As shown in the figure, the specific operation steps of the driver fatigue detection optimization method based on Transformer-YOLO described in this invention are as follows:

[0052] Step (1), Data Acquisition: Acquire driver video stream data and divide the video stream into non-overlapping frame sequences;

[0053] Step (2), Data preprocessing: Process the image of each frame;

[0054] Step (3): Use the YOLOv11 model to detect the face region in the video frame in real time and output the face bounding box and key region coordinates; wherein, the key region includes the eye region, mouth region and head pose angle;

[0055] Step (4), Key Region Feature Extraction: Based on the coordinates output in step (2), crop the key region image and input it into the key region attention Transformer module for spatiotemporal feature extraction;

[0056] Step (5), Multimodal feature fusion: The head pose angle output by YOLOv11 is concatenated with the spatiotemporal feature vector extracted by the key region attention Transformer to form a fused feature;

[0057] Step (6) Fatigue state classification: Input the fused features into the classifier and output the fatigue state classification result.

[0058] Step (1) specifically involves: acquiring a video dataset of drivers in normal driving, eye-closed, yawning, and head-turning states; dividing the video stream into non-overlapping frame sequences, with each 5 frames constituting a temporal unit {T1, T2, ..., T...}. n}, with adjacent units spaced 2 frames apart.

[0059] Step (2) includes the following steps:

[0060] (21) Apply the Adaptive Histogram Equalization (CLAHE) algorithm to each frame of the image for illumination normalization, and output the illumination-normalized temporal frame sequence to optimize the image illumination consistency and provide a more robust input for subsequent processing; Equation (1) is defined as the histogram equalization algorithm:

[0061] I out (x,y)=CLAHE(I in (x,y),N tiles =8,C limit =2) (1)

[0062] In the formula, N tiles Represents the number of blocks, C limit Indicates the contrast limit threshold;

[0063] (22) The established timestamp index table can assist in subsequent analysis of the temporal correlation of time-series features or the filtering of motion-blurred frames, improving the adaptability of the process to dynamic acquisition scenarios; record the acquisition time t of each frame. i and motion ambiguity coefficient b i Equation (2) is defined as the motion fuzzy coefficient b. i :

[0064]

[0065] In the formula, M represents the number of frames into which the video is divided in step (1). It is the gradient of the M-frame image, representing the gradient of the k-th frame image at pixel (x,y), reflecting the changes in image edges.

[0066] Step (3) includes the following steps:

[0067] (31) Load the pre-trained YOLOv11 model (its backbone network is CSPDarknet53-Tiny), which matches the size of the pre-processed first frame image and can be directly input; then the model loaded in step (21) processes the time-series frame sequence normalized by illumination, detects the face region in the video frame in real time, and outputs the face bounding box, key region coordinates (including eye region and mouth region) and head pose angle.

[0068] (32) YOLOv11 detection is performed on the first frame of the temporal unit; the model extracts and analyzes features from the first frame image through its internal multi-layer network structure; in terms of face detection, the model can accurately locate the position of the face in the image and output the coordinates of the face bounding box (x1, y1, x2, y2), where (x1, y1) is the coordinate of the upper left corner of the bounding box, (x2, y2) is the coordinate of the lower right corner of the bounding box, and the face detection is only performed when the confidence level p face The detection result will only be retained if the value is ≥0.95, in order to ensure the accuracy of face region recognition;

[0069] (33) Output eye heatmap H using YOLOv11 model eye H mouth The heatmap has a dimension of 56×56. Through different numerical distributions, it clearly presents the position and feature information of the eye area in the image. It can reflect key details such as the outline of the eyes and the position of the mouth, providing an important basis for the subsequent extraction of fatigue features related to the eyes.

[0070] (34) The model outputs the Euler angles (α, β, γ) of the head pose. These three angles reflect the head pose from different dimensions. Equation (3) defines the method for calculating the Euler angles of the head:

[0071]

[0072] In the formula, α usually represents the pitch angle, which measures the degree of head rotation up and down; β represents the yaw angle, which reflects the state of head rotation left and right; and γ is the roll angle, which shows the rotation of the head around its own longitudinal axis.

[0073] (35) After the YOLOv11 model performs preliminary detection on the first frame image, it will generate multiple possible face bounding boxes. At this time, non-maximum suppression (NMS) is used to filter and merge these candidate boxes. The NMS threshold is set to 0.5. When the IoU of two boxes is ≥0.5, they are judged as highly overlapping redundant boxes. At this time, the confidence score (p) of each box is retained. face The highest-valued bounding box is selected, and other overlapping boxes are removed. After NMS processing, only non-overlapping or low-overlapping bounding boxes with a confidence level of p are retained. faceA face bounding box with a value ≥0.95 ensures that the output (x1, y1, x2, y2) is the unique and most accurate face region coordinates;

[0074] (36) For non-first frames, the ROI tracking algorithm is used for processing. The tracking algorithm "inherits" the position from the previous frame to reduce repeated detection; Equation (4) is defined as the ROI tracking algorithm:

[0075]

[0076] In the formula, B represents the face bounding box in the current frame; t-1 The bounding box coordinates of the face in the previous frame (time t-1) are represented by ΔB; ΔB represents the displacement increment of the bounding box, i.e., the change in position from t-1 to frame t; OpticalFlow(I t-1 ,I t ) represents the optical flow algorithm, which calculates pixel motion between two adjacent frames.

[0077] Step (4) includes the following steps:

[0078] (41) Based on the heatmap coordinates output in step (33), crop the eye area. and mouth area

[0079] (42) Input the ROI images of 5 consecutive frames into the key region attention Transformer module; first divide each frame of ROI into non-overlapping 8×8 pixel blocks, with an embedding dimension D = 128; Equation (5) is defined as the pixel block method that distinguishes different spatial locations and time steps:

[0080]

[0081] In the formula, The learnable projection matrix maps each pixel block vector to a 128-dimensional embedding space, and then concatenates the embedding vectors of all pixel blocks to form a sequence.

[0082] (43) Temporal modeling uses a Transformer encoder with 3 layers and FFN = 512 feedforward network;

[0083] (44) Output spatiotemporal feature vectors Equation (6) is defined as the method for calculating spatiotemporal feature vectors:

[0084] f st =AvgPool(TransformerEncoder(z0)) (6).

[0085] Step (5) includes the following steps:

[0086] (51) The spatiotemporal features of step (44) are concatenated with the head posture angle of step (34). By concatenating the two complementary features, a more comprehensive representation of the driver's state is obtained, thereby improving the accuracy of fatigue detection. Equation (7) is defined as the feature concatenation method:

[0087]

[0088] In the formula, The spatiotemporal feature vector in step (44) is represented by (α,β,γ), which are the Euler angles of the head attitude, representing the pitch angle, yaw angle, and roll angle, respectively; the fused f fusion It is typically used as input to a fully connected classifier to predict fatigue state;

[0089] (52) Valid features are selected through a feature selection layer. The feature selection layer automatically selects valid features through a trainable mask M, thereby improving the generalization ability and interpretability of the model; Equation (8) is defined as the method for selecting valid features through the feature selection layer:

[0090] f selected =f fusion ⊙M,M∈{0,1} 259 (8)

[0091] In the formula, M represents the binary mask generated by the threshold function;

[0092] (53) Dynamic standardization is applied to feature fusion. Dynamic standardization adaptively adjusts the feature distribution through sliding window statistics to enhance the robustness of the model to interference factors such as illumination changes and pose fluctuations. Equation (9) is defined as the dynamic standardization method:

[0093]

[0094] feature f i Convert to zero mean, unit variance form Eliminate feature distribution drift during long-term driving; among which and This represents the sliding window statistic, which is the mean of the i-th feature at time t. and standard deviation Calculated based on data from the most recent T=30 seconds.

[0095] Step (6) includes the following steps:

[0096] (61) Input the features processed in step (53) into a two-level classifier. The first level is a 3-layer MLP classifier, which outputs the original score vector. Corresponding to three categories (awake, fatigued, moderately fatigued); Equation (10) is defined as s:

[0097]

[0098] The second level is the temporal smoothing layer, which performs a weighted average of the scores of the current frame t and the two frames preceding it, and calculates the weighted score of the first 5 frames. Equation (11) defines the weighted score s. t :

[0099]

[0100] Then, it is judged according to the fatigue judgment rule, which is defined as the fatigue judgment rule in equation (12):

[0101]

[0102] The formula counts the number of "showing signs of fatigue" among the five indicators; FatigueFlag = 1 indicates a "fatigue state"; FatigueFlag = 0 indicates a "non-fatigue state"; k = 0 is the average eye closure degree; k = 1 is the abnormality of blinking frequency; k = 2 is the deviation of head pitch angle; k = 3 is the average mouth opening degree; k = 4 is the attention distraction index; The value is the normalized value, and 0.8 is the preset "fatigue tendency threshold". If at least 3 of the 5 indicators show fatigue tendency, then FatigueFlag = 1; if less than 3 of the 5 indicators show fatigue tendency, then FatigueFlag = 0.

[0103] (62) When FatigueFlag=1 is detected three times in a row, an alarm is triggered;

[0104] Input the real-time fatigue status flag FatigueFlag from step (61), maintain a continuous fatigue counter Counter, initialized to 0. If FatigueFlag = 1 in the current frame, then Counter = Counter + 1; if FatigueFlag = 0, then Counter = 0. When fatigue status is detected three times consecutively, an alarm is triggered immediately. The prediction results are shown in Table 1:

[0105] Table 1 Prediction Results

[0106] Number of tests FatigueFlag Counter Alarm status 1 1 1 Not triggered 2 1 2 Not triggered 3 1 3 Trigger alarm 4 1 4 Continuous alarm 5 0 0 Stop alarm

Claims

1. An optimized method for driver fatigue detection based on Transformer-YOLO, characterized in that, Includes the following steps: (1) Data acquisition; (2) Data preprocessing; (3) Target detection; (4) Key region feature extraction; (5) Multimodal feature fusion; (6) Classification of fatigue state.

2. The driver fatigue detection optimization method based on Transformer-YOLO according to claim 1, characterized in that, In step (1), data acquisition refers to: acquiring driver video stream data and dividing the video stream into non-overlapping frame sequences; Specifically, the process involves acquiring a video dataset of drivers in various states, including normal driving, eyes closed, yawning, and head tilting. The video stream is then segmented into non-overlapping frame sequences, with each sequence of 5 frames forming a temporal unit {T1, T2, ..., T...}. n }, with adjacent units spaced 2 frames apart.

3. The driver fatigue detection optimization method based on Transformer-YOLO according to claim 1, characterized in that, In step (2), data preprocessing involves processing the image of each frame. The specific operating steps are as follows: (21) Apply an adaptive histogram equalization algorithm to each frame of the image for illumination normalization, and output the time-series frame sequence after illumination normalization. Equation (1) is defined as the histogram equalization algorithm: Yo out (x,y)=CLAHE(I in (x,y),N tiles =8,C limit =2) (1) (22) The established timestamp index table can assist in subsequent temporal correlation analysis of time series features or screening of motion-blurred frames, recording the acquisition time t of each frame. i and motion ambiguity coefficient b i Equation (2) is defined as the motion fuzzy coefficient b. i :

4. The driver fatigue detection optimization method based on Transformer-YOLO according to claim 1, characterized in that, In step (3), target detection refers to using the YOLOv11 model to detect the face region in the video frame in real time and output the face bounding box and key region coordinates. The key regions include the eye region, mouth region and head pose angle.

5. The driver fatigue detection optimization method based on Transformer-YOLO according to claim 4, characterized in that, Step (3) specifically includes the following steps: (31) Load the pre-trained YOLOv11 model and match it with the size of the pre-processed first frame image; the loaded model then processes the illumination-normalized temporal frame sequence, detects the face region in the video frame in real time, and outputs the face bounding box, key region coordinates and head pose angle. (32) YOLOv11 detection is performed on the first frame of the temporal unit. The model extracts and analyzes features of the first frame image through its internal multi-layer network structure. (33) Output eye heatmap H using YOLOv11 model eye H mouth ; (34) The model outputs the head pose Euler angles (α,β,γ), which reflect the head pose from different dimensions. Equation (3) defines the method for calculating the head Euler angles: (35) After the YOLOv11 model performs preliminary detection on the first frame image, multiple face bounding boxes are generated. The candidate boxes are filtered and merged by nonmaximum suppression. (36) For non-first frames, the ROI tracking algorithm is used for processing. Equation (4) defines the ROI tracking algorithm as follows:

6. The driver fatigue detection optimization method based on Transformer-YOLO according to claim 1, characterized in that, In step (4), key region feature extraction refers to cropping the key region image based on the coordinates output in step (2) and inputting it into the key region attention Transformer module for spatiotemporal feature extraction.

7. The driver fatigue detection optimization method based on Transformer-YOLO according to claim 6, characterized in that, Step (4) specifically includes the following steps: (41) Based on the heatmap coordinates output in step (33), crop the eye area. and mouth area (42) Input the ROI images of 5 consecutive frames into the key region attention Transformer module; (43) Temporal modeling uses a Transformer encoder with L=3 layers and FFN=512 feedforward network; (44) Output spatiotemporal feature vectors Equation (5) is defined as the method for calculating spatiotemporal feature vectors: f st =AvgPool(TransformerEncoder(z0)) (5)。 8. The driver fatigue detection optimization method based on Transformer-YOLO according to claim 1, characterized in that, In step (5), multimodal feature fusion refers to concatenating the head pose angle output by YOLOv11 with the spatiotemporal feature vector extracted by the key region attention Transformer to form fused features.

9. The driver fatigue detection optimization method based on Transformer-YOLO according to claim 8, characterized in that, Step (5) specifically includes the following steps: (51) The spatiotemporal features of step (44) are spliced ​​with the head pose angle of step (34); (52) Filter effective features through the feature selection layer; (53) Dynamically standardize feature fusion.

10. The driver fatigue detection optimization method based on Transformer-YOLO according to claim 1, characterized in that, The fatigue state classification in step (6) refers to: inputting the fused features into the classifier and outputting the fatigue state classification result; specifically, it includes the following steps: (61) Input the features processed in step (53) into the two-level classifier, and then judge them; Equation (6) is defined as the fatigue judgment rule: (62) When FatigueFlag=1 is detected three times in a row, an alarm is triggered.

Images