Fatigue driving detection intelligent early warning method based on YOLOv12 deep neural network and multi-modal feature fusion

Abstract

The invention relates to a fatigue driving detection intelligent early warning method based on YOLOv12 deep neural network and multi-modal feature fusion. According to the method, an improved YOLOv12 facial feature extraction module, an LSTM-Transform time sequence modeling module and an intelligent early warning decision module are included, and organic combination of high-precision recognition of the facial fatigue state of a driver and time sequence dependence modeling is achieved by constructing a hybrid neural network architecture and adopting collaborative optimization of a feature pyramid network and a C2f module. A bidirectional LSTM-Transform time sequence prediction network is innovatively designed, and intelligent fusion of fatigue development trend prediction and critical point early warning is realized through long-distance time sequence dependence modeling guided by a multi-head self-attention mechanism. According to the method, the technical problems that a traditional fatigue detection method is not high in precision and weak in time sequence modeling capacity are effectively solved, mAP at 0.5 reaches 97.9% on a 14,859 annotated image data set and is remarkably improved compared with a traditional method, experimental results show that the model has excellent performance in objective evaluation indexes such as the precision rate of 96.2% and the recall rate of 96.9% and subjective evaluation of actual application scenes, and the method has a good application prospect. The method achieves a remarkable effect in fatigue state recognition and early warning decision, and has a remarkable technical effect and industrial application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, computer vision, deep learning and intelligent traffic methods, in particular to a fatigue driving intelligent detection method based on an improved YOLOv12 deep neural network architecture, which realizes fatigue state detection and early warning by improving the feature pyramid network, C2f module and bottleneck residual block of YOLOv12, and combining LSTM-Transformer time series modeling. BACKGROUND

[0002] With the rapid development of intelligent traffic methods and the increasing strictness of road safety supervision, fatigue driving detection technology plays an important role in traffic safety guarantee, accident prevention and intelligent driving assistance. However, due to the complexity of fatigue state and the limitation of detection device performance, the fatigue features obtained often have problems such as low precision, poor real-time performance and high false alarm rate. The root of these problems lies in the multi-factor coupling characteristics of fatigue driving and the weak changes of physiological signals, which leads to a significant decrease in detection accuracy and seriously affects the subsequent warning and intervention applications. Therefore, how to effectively improve the accuracy and real-time performance of fatigue driving detection has become a hot research issue.

[0003] Multi-modal fatigue detection technology aims to recognize fatigue state from multi-dimensional data such as facial expressions, physiological signals and driving behaviors, and has made significant progress in intelligent traffic methods in recent years. However, the particularity of fatigue driving makes it difficult for traditional single-modal detection methods to be directly applied: on the one hand, the representation mode of fatigue state is complex, including facial micro-expression changes, eye state fluctuations and physiological signal abnormalities; on the other hand, existing methods have difficulty in effectively balancing multi-modal feature fusion and time series dependent modeling, resulting in suboptimal detection results. Although methods based on convolutional neural networks and recurrent neural networks perform well in fatigue detection tasks, they still face challenges in complex driving scenarios: CNN methods have difficulty in modeling long-term temporal dependencies, resulting in insufficient fatigue trend prediction ability, and although they can enhance feature extraction by stacking convolutional layers, they are limited by the local receptive field of convolution kernels, and the weak fatigue signal capture effect is limited. SUMMARY

[0004] The present application proposes a fatigue driving intelligent detection method based on improved YOLOv12 and time series modeling to solve the core technical problems of low detection accuracy, weak feature extraction ability and insufficient time series modeling ability in existing fatigue driving detection technology. The method improves the feature pyramid network, C2f module and bottleneck residual block of YOLOv12 to build a high-precision facial fatigue state detection model, and realizes fatigue trend prediction and intelligent early warning by combining LSTM-Transformer time series modeling.

[0005] A fatigue driving intelligent detection method based on YOLOv12 and time series modeling, the method comprising the following steps:

[0006] Step 1: Construct a fatigue driving detection dataset containing 14,859 labeled images covering 6 driving state categories;

[0007] Step 2: Construct an improved YOLOv12 face feature extraction network containing a feature pyramid network, a C2f module and a Bottleneck residual block;

[0008] Step 3: Construct an LSTM-Transformer time series modeling network for time series analysis and trend prediction of continuous detection results;

[0009] Step 4: Set the input size to 640x640 pixels, use 100 training rounds and optimized training parameter configuration;

[0010] Step 5: Evaluate the model performance using standard indicators such as mAP, precision, recall, etc., and verify the effectiveness of the method;

[0011] Step 6: Implement four-level hierarchical early warning decision-making based on time series prediction results for intelligent early warning.

[0012] Preferably, the step 1 dataset construction is specifically:

[0013] A fatigue driving detection dataset is established containing 14,859 labeled images covering 6 driving state categories: safe driving, dangerous driving, fatigue driving, yawning, distraction, and drinking water. The dataset is divided into training set, validation set and test set according to the proportions of 80.4%, 12.9% and 6.6%. YOLO format is used for bounding box labeling, and each image contains target position coordinates and category labels.

[0014] Preferably, the step 2 improved YOLOv12 face feature extraction network containing a feature pyramid network, a C2f module and a Bottleneck residual block is specifically:

[0015] In the feature pyramid network design stage, first input the driver's face image through standardization preprocessing mapping to 640x640 pixel size, then through the Bottom-up self-bottom-up path for multi-level feature extraction. The multi-scale feature fusion calculation formula of the feature pyramid network is:

[0016] P i =Conv 1×1 (C i )+Upsample(P i+1 )

[0017] Where P iPyramid feature of the i-th layer, C i Output of the i-th layer of the backbone network, Conv 1×1 1x1 convolution operation.

[0018] In the C2f module improvement stage, the input features are first subjected to convolutional transformation for preliminary feature processing, and then the channels are divided into two parallel processing paths through the segmentation operation. The forward propagation calculation formula of the C2f module is:

[0019] F C2f = Concat (F1, Bottleneck n (F2))

[0020] Where F1 and F2 represent the two features after segmentation, Bottleneck n n serial Bottleneck residual blocks.

[0021] In the Bottleneck residual block design stage, a bottleneck layer design of 1x1→3x3→1x1 convolution sequence is adopted, and the gradient disappearance problem of deep network is alleviated through identity mapping residual connection. The mathematical expression of the residual connection is:

[0022] F output = F(x, {W i}) + x

[0023] Where F(x, {W i}) represents the residual mapping function, and x represents the input feature.

[0024] The improved bottleneck layer calculation process is:

[0025]

[0026] The mathematical expression of the SiLU activation function is:

[0027]

[0028] The network adopts the YOLO standard data format, the input image size is 640x640 pixels, and the output contains 6 state categories of detection results.

[0029] Preferably, the step 2 constructs an improved YOLOv12 face feature extraction network comprising a feature pyramid network, a C2f module and a Bottleneck residual block, specifically:

[0030] In the feature pyramid network design stage, first, the driver's face image is inputted and mapped to 640x640 pixel size through standardization preprocessing, and then multi-level feature extraction is performed through the Bottom-up path; the Top-down path starts from the high-level feature, and through upsampling and element-by-element addition operation, the deep semantic information is fused into the shallow feature to realize multi-scale target detection.

[0031] In the C2f module improvement stage, the input features are first processed by convolution transformation, and then divided into two parallel processing paths through segmentation operation, one of which directly transmits to keep the original feature information, and the other one is processed by the Bottleneck residual block for deep feature extraction, and finally the two features are fused by splicing.

[0032] In the Bottleneck residual block design stage, the bottleneck layer design of 1x1→3x3→1x1 convolution sequence is adopted, and the gradient disappearance problem of deep network is relieved by identity mapping residual connection.

[0033] Preferably, the step 3 of constructing the LSTM-Transformer time series modeling network is specifically:

[0034] A hybrid time series prediction network is constructed to perform time series analysis and trend prediction on the continuous frame results of YOLOv12 detection. This module contains two key components, LSTM gate unit and Transformer time series attention mechanism.

[0035] In the time series feature preparation stage, the detection results of continuous frames are organized into time series feature sequences:

[0036] X t =[P t,1 ,P t,2 ,…,P t,6 ,Conf t ]

[0037] Where P t,i is the probability of the i-th state category of the t-th frame, and Conf t is the detection confidence.

[0038] The LSTM time series encoder adopts bidirectional LSTM to process the time series feature sequence, and the hidden state dimension is 256. The calculation process of the forward LSTM is:

[0039]

[0040]

[0041]

[0042]

[0043] where σ denotes the sigmoid function, denotes element-wise multiplication, and W and b denote the weight matrix and bias vector, respectively.

[0044] The Transformer attention layer establishes long-range temporal dependencies through multi-head self-attention mechanism. The multi-head attention calculation formula is:

[0045] MultiHead(Q, K, V) = Concat(head1, …, head h )W O

[0046]

[0047] where the calculation formula of the attention mechanism is:

[0048]

[0049] The multi-task prediction head includes three prediction branches: fatigue degree prediction, trend analysis, and critical point early warning.

[0050] Preferably, the step 4 training parameter configuration is specifically:

[0051] The training adopts 100 rounds, a batch size of 16, and uses pre-trained YOLOv12s weights for initialization. The loss function includes a bounding box loss, a classification loss, and a distributed focal loss. The Adam optimizer is used with an initial learning rate of 0.001 and a cosine annealing scheduling strategy.

[0052] Preferably, the step 5 performance evaluation is specifically:

[0053] Standard target detection evaluation indicators are used, including mean average precision (mAP) at different IoU thresholds to evaluate detection accuracy, and precision and recall to measure the accuracy and integrity of the detection results. The method achieves an mAP@0.5 of 97.9%, a precision of 96.2%, and a recall of 96.9% on 1,922 validation set images.

[0054] Preferably, the step 6 four-level hierarchical early warning decision is specifically:

[0055] The early warning method adopts a four-level hierarchical architecture: green safe level, yellow attention level, orange warning level, and red danger level, and realizes intelligent management of fatigue state through a progressive intervention strategy. The comprehensive score is calculated by weighted combination of fatigue level, critical point probability, and trend score, with weight coefficients of 0.4, 0.3, and 0.3, respectively. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 Flow chart for overall architecture of the method of the present application

[0057] Figure 2 Network structure diagram of YOLOv12 feature pyramid for the present application

[0058] Figure 3 Module structure diagram of C2f for the present application

[0059] Figure 4 Bottleneck residual block structure diagram for the present application

[0060] Figure 5 Internal structure diagram of LSTM gating unit for the present application

[0061] Figure 6 Architecture diagram of Transformer temporal attention mechanism for the present application

[0062] Figure 7 Model training convergence curve diagram for the present application

[0063] Figure 8 Example diagram of verification set detection results for the present application DETAILED DESCRIPTION

[0064] The present application is further described below in conjunction with the accompanying drawings and examples. However, the present application can be implemented in many different ways and should not be construed as being limited to the embodiments shown; rather, these embodiments provide those skilled in the art with a full and enabling disclosure of how to make and use the embodiments, which satisfy applicable legal requirements.

[0065] A fatigue driving intelligent detection method based on YOLOv12 and time series modeling is described in detail below in conjunction with specific embodiments.

[0066] The method specifically includes the following steps:

[0067] Step 1: Construct a fatigue driving detection dataset, containing 14,859 labeled images, covering 6 driving state categories;

[0068] A public driver monitoring dataset is used, containing 14,859 labeled images, covering 6 driving state categories: safe driving, dangerous driving, fatigue driving, yawning, distraction, and drinking water. The dataset is divided into training set, validation set, and test set according to the proportions of 80.4%, 12.9%, and 6.6%. Various data augmentation techniques are used to expand the training samples, including horizontal flipping, HSV color jittering, Mosaic enhancement, random erasing, etc., to improve the model generalization ability.

[0069] Step 2: Improved YOLOv12 face feature extraction network containing feature pyramid network, C2f module and bottleneck residual block is constructed as shown in Figure 2 、 Figure 3 、 Figure 4

[0070] In the feature pyramid network design stage, as shown in Figure 2 , first input the driver face image through standardization preprocessing mapping to 640x640 pixel size. The image standardization formula is:

[0071]

[0072] Where μ = [0.485, 0.456, 0.406] and σ = [0.229, 0.224, 0.225] are the mean and standard deviation of ImageNet dataset respectively.

[0073] The feature extraction process of bottom-up path is:

[0074] C2 = Conv 3×3 (Conv 6×6 (I norm ))

[0075] C i = DownSample(C i-1 ), i ∈ {3, 4, 5}

[0076] The feature fusion calculation formula of top-down path is:

[0077] P5 = Conv 1×1 (C5)

[0078] P i = Conv 1×1 (C i )+ Upsample(P i+1 ), i ∈ {4, 3, 2}

[0079] In the C2f module improvement stage, as shown in Figure 3 , the mathematical expression of input feature segmentation operation is:

[0080] F1, F2 = Split(Conv(X), dim = 1)

[0081] Where Split operation equally divides the channel dimension into two parts. The complete forward propagation of C2f module is:

[0082] Y1 = F1

[0083] ​

[0084] F C2f =Conv(Concat([Y1,Y2],dim=1))

[0085] where n represents the number of bottleneck residual blocks, and represents the series operation.

[0086] In the bottleneck residual block design stage, as shown in Figure 4 , a 1x1→3x3→1x1 convolution sequence bottleneck layer design is adopted. The detailed calculation process of the bottleneck block is as follows:

[0087] F1=SiLU(BN(Conv 1×1 (x)))

[0088] F2=SiLU(BN(Conv 3×3 (F1)))

[0089] F3=BN(Conv 1×1 (F2))

[0090] F output =SiLU(F3+x)

[0091] where BN represents the batch normalization operation, and the residual connection alleviates the gradient vanishing problem of deep network.

[0092] The mathematical expression and derivative of the SiLU activation function are as follows:

[0093]

[0094]

[0095] The network output layer adopts a decoupling detection head, and the classification and regression branches are calculated respectively:

[0096] P cls =σ(Conv cls (F neck ))

[0097] P reg =Conv reg (F neck )

[0098] where F neck represents the neck network output feature, and σ represents the sigmoid activation function.

[0099] Step 3: Construct the LSTM-Transformer time series modeling network to perform time series analysis and trend prediction on the continuous detection results, as shown in Figure 5 ,Figure 6 as shown;

[0100] A hybrid time series prediction network is constructed to perform time series analysis and trend prediction on the results of consecutive frames of YOLOv12 detection. This module contains two key components: LSTM gate unit and Transformer time series attention mechanism.

[0101] As shown in Figure 5 , the LSTM unit contains three gating mechanisms: forget gate, input gate and output gate. The method adopts a bidirectional LSTM network with a hidden state dimension of 256. The final hidden state of the bidirectional LSTM is calculated as:

[0102]

[0103] where and represent the hidden states of the forward and backward LSTMs, respectively.

[0104] As shown in Figure 6 , the Transformer time series attention mechanism further captures long-range time series dependencies based on LSTM encoding. The position encoding calculation formula is:

[0105]

[0106]

[0107] where pos represents the position, i represents the dimension index, and d model represents the model dimension.

[0108] Based on the LSTM-Transformer hybrid architecture, three prediction branches are designed. The calculation formula of the fatigue level prediction branch is:

[0109] FatigueLevel = σ(W f ·F temporal +b f ) × 6

[0110] where F temporal represents the time series features, and the output range 0-6 corresponds to 6 fatigue levels.

[0111] Step 4: Implement four-level hierarchical early warning decision-making based on time series prediction results for intelligent early warning;

[0112] A hierarchical early warning method was constructed to realize intelligent management and progressive intervention of fatigue state. The early warning method adopted a four-level hierarchical architecture: green safety level, yellow attention level, orange warning level and red danger level, and realized intelligent management of fatigue state through progressive intervention strategy. The method was based on the comprehensive score of fatigue level, critical point probability and trend score, with weight coefficients of 0.4, 0.3 and 0.3, respectively.

[0113] Step 5: The performance of the model was evaluated by using standard indicators such as mAP, precision and recall, and the effectiveness of the method was verified.

[0114] The performance was verified by using standard target detection evaluation indicators. The method achieved excellent performance of mAP@0.5 of 97.9%, precision of 96.2% and recall of 96.9% on 1,922 validation set images, which proved the effectiveness of the fatigue driving detection method based on YOLOv12 and time series modeling.

[0115] Step 6: Set the mixed loss function to optimize network parameters.

[0116] The method used a multi-level mixed loss function to optimize the overall performance, and the total loss function was:

[0117] L total =L detection +L temporal

[0118] The detection loss is:

[0119] L detection =λ1L box +λ2L cls +λ3L dfl

[0120] The time series modeling loss is:

[0121] L temporal =λ4L fatigue +λ5L trend +λ6L critical

[0122] The time series consistency loss calculation formula is:

[0123]

[0124] The Adam optimizer parameter update formula is:

[0125] m t =β1m t-1 +(1-β1)g t

[0126]

[0127]

[0128] Experimental results and performance verification:

[0129] The training convergence curve as shown in Figure 7 indicates that the model performance indicators have been significantly improved: the precision rate has increased from 65% to 96.2%, the recall rate has increased from 60% to 96.9%, the mAP@0.5 has increased from 65% to 97.9%, and the mAP@0.5-0.95 has increased from 45% to 87.3%, proving the effectiveness of the method.

[0130] The validation set detection results as shown in Figure 8 show the detection effect of the optimized YOLOv12 model on the actual validation set, and the method can accurately identify different degrees of fatigue state, and has good practicability and reliability.

[0131] The above is only a preferred embodiment of the fatigue driving intelligent detection method based on YOLOv12 and time series modeling, and the protection scope of the method is not limited to the above-mentioned embodiments, and any technical solution under the same idea belongs to the protection scope of the present application. It should be noted that for those skilled in the art, some improvements and changes without departing from the principles of the present application should also be considered as the protection scope of the present application. The present application can be widely applied to intelligent traffic methods, automobile safety auxiliary driving, logistics transportation safety supervision and other related fields, and has important practical value and popularization prospect.

Claims

1. A fatigue driving intelligent detection method based on YOLOv12 and time series modeling, characterized in that, Comprise the following steps: Step 1: build a fatigue driving detection dataset containing 14,859 labeled images, covering 6 driving state categories; Step 2: build a fatigue state detection model using the YOLOv12 deep neural network architecture, and realize automatic recognition of the driver's face state through target detection method; Step 3: design an LSTM-Transformer time series modeling network to analyze the time series and trend of the continuous detection results; Step 4: configure the optimized training parameters, including 100 training rounds, 640x640 image size and adaptive learning rate scheduling; Step 5: realize the model performance evaluation system, and use mAP, precision, recall and other standard indicators to quantify the detection effect; Step 6: realize the hierarchical warning of fatigue state through multi-level warning control theory, and establish an intelligent intervention decision engine.

2. The method of claim 1, wherein, The dataset construction is specifically: A fatigue driving detection dataset is established, containing 14,859 labeled images, covering 6 driving state categories: safe driving (Safe Driving), dangerous driving (Dangerous Driving), sleepy driving (Sleepy Driving), yawning (Yawn), distraction (Distracted), and drinking (Drinking). The dataset is divided into training set (11,952 images), validation set (1,922 images) and test set (985 images) according to the ratio of 80.4%, 12.9% and 6.6%. The YOLO format is used for bounding box labeling, and each image contains target position coordinates and class labels. The data is strictly verified and processed in the format.

3. The method of claim 1, wherein, The YOLOv12 feature extraction module in step 2 is specifically: A fatigue driving state detection method based on YOLOv12 target detection network is constructed, which realizes the detection of driver face area and 6 state classification by using end-to-end deep learning method. YOLOv12 network architecture improvement: an improved single-stage detection framework is adopted, which contains the following core components: Improved C2f module forward propagation: F (l) = C2f (l) (F (l-1) ) where F (l) represents the feature map of the l-th layer, C2f l represents the improved C2f module of the l-th layer. Bottleneck residual connection: F output = F(x, {W i}) + x where F(x, {W i}) denotes a residual mapping function, x denotes input features, and the gradient flow is enhanced by a residual connection. SiLU activation function: Replace the traditional ReLU activation function to improve the non-linear expression ability and gradient flow characteristics. The network uses YOLO standard data format, the input image size is 640x640 pixels, and the output contains 6 state categories of detection results. The pre-trained YOLOv12s weight is used for initialization, combined with bounding box regression loss, classification loss and distributed focal loss for multi-task joint optimization. The method realizes mAP@0.5 of 97.9%, precision of 96.2% and recall of 96.9% on 1,922 validation set images.

4. The method of claim 1, wherein, The model training process is specifically: 100 training rounds, batch size 16, using pre-trained YOLOv12s weight for initialization. Use Adam optimizer, initial learning rate 0.001, cosine annealing scheduling strategy. The loss function weight is set to bounding box loss 7.5, classification loss 0.5 and distributed focal loss 1.

5.

5. The method of claim 1, wherein, The LSTM-Transformer time series modeling in step 3 is as follows: The face fatigue features extracted by YOLOv12 are analyzed based on a hybrid neural network architecture. The time series feature sequence construction formula is: X t = [P t,1 , P t,2 ,..., P t,6 , Conf t ] where P t,i is the probability of the ith state class of the tth frame, Conf t is the detection confidence. The LSTM time series encoder uses a bidirectional LSTM to process the time series feature sequence. The forward LSTM calculation process is: f t = σ(W f · [h t-1 , x t ]+ b f ) i t = σ(W i · [h t-1 , x t ]+ b i ) C t = f t ⊙ C t-1 + i t ⊙ tanh(W C · [h t-1 , x t ] + b C ) h t = o t ⊙ tanh(C t ) The final hidden state of the bidirectional LSTM is: The Transformer attention layer establishes long-distance time series dependencies through a multi-head self-attention mechanism: MultiHead(Q, K, V) = Concat(head1,..., head h )W O The multi-task prediction head includes three prediction branches. The fatigue level prediction formula is: FatigueLevel = σ(W f ·F temporal +b f ) x 6 6. The method of claim 1, wherein, The early warning decision module in step 6 is: This module integrates the detection results of YOLOv12 and the LSTM-Transformer time series prediction results to realize real-time monitoring and grading warning of the driver's fatigue state. The warning module includes fatigue level quantification prediction, trend prediction, and critical point warning mechanism, respectively outputting fatigue level, trend vector, and warning probability, and adopts a weighted combination method for multi-task optimization.

7. The method of claim 1, wherein, The multi-level warning control theory is as follows: The warning method adopts a four-level hierarchical warning architecture: green safety level, yellow attention level, orange warning level, and red danger level, and realizes intelligent management of fatigue state through a progressive intervention strategy. The comprehensive score is calculated by weighted combination of fatigue level, critical point probability, and trend score, with weight coefficients of 0.4, 0.3, and 0.3 respectively, and warning thresholds set at 0.3, 0.6, and 0.

8. The progressive intervention strategy includes: normal monitoring, slight reminder, obvious warning and suggestion to rest, and forced intervention and suggestion to stop.

8. The method of claim 1, wherein, The intelligent warning decision engine is implemented as follows: The intelligent decision engine adopts a dual-module architecture design, integrating YOLOv12 face detector and LSTM-Transformer time series prediction model, and realizes high-precision fatigue state judgment and intelligent warning decision generation through time series information fusion. The detection and prediction results are combined through weighted fusion, and the rule engine generates the optimal decision based on the prediction results, time series trend, and critical point probability, and adjusts the intervention intensity and mode selection adaptively according to the fatigue probability.

9. An intelligent fatigue driving detection method based on YOLOv12 and time series modeling, characterized by: This method uses YOLOv12 target detection network architecture combined with LSTM-Transformer time series modeling to build fatigue driving state detection and trend prediction model. Through the training of 14,859 labeled images, the automatic detection and 6-class state classification (safe driving, dangerous driving, fatigue driving, yawning, distraction, and drinking) of the driver's face area are realized, and the fatigue development trend prediction and critical point warning are realized through time series modeling. The method uses an input size of 640x640 pixels, and after 100 rounds of training optimization, it achieves an mAP@0.5 of 97.9%, an accuracy of 96.2%, and a recall rate of 96.9% on 1,922 validation set images, proving the effectiveness and practical value of the fatigue driving detection method based on deep learning and time series modeling.

Images