A lightweight method for detecting and tracking fatigue driving

By introducing Mobilenetv3-small backbone and ECA attention mechanism in the YOLOv5s network model, combined with the DeepSort algorithm, the problems of low detection accuracy and large model parameters in the existing fatigue driving detection methods are solved, and efficient in vehicle-mounted edge equipment is achieved. Accurate fatigue driving status recognition.

CN118230296BActive Publication Date: 2024-11-05JIANGSU FENGDUN TECHNOLOGY ENGINEERING CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410324420.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-21
Publication Date
2024-11-05
Estimated Expiration
2044-03-21

AI Technical Summary

Technical Problem

The existing fatigue driving detection methods have problems such as low detection accuracy, large model parameters, and insufficient consideration of facial information loss. Especially in scenarios where vehicle edge equipment deployment and high real-time and accuracy requirements are high, there is a lack of effective solutions. .

Method used

The lightweight YOLOv5s network model combined with the DeepSort algorithm is adopted to enhance context information attention by introducing the Mobilenetv3-small backbone and ECA attention mechanism, and optimize prediction box regression using Wise-EIoU Loss, fusing continuous eye-closing frames and yawning frames. Fatigue driving detection model was constructed with PERCLOS scores.

Benefits of technology

It realizes the significant reduction in model size while maintaining high accuracy, improves detection robustness, reduces the probability of facial information loss, meets the deployment needs of on-board edge equipment, and improves the accuracy of identification of fatigue driving status.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118230296B_ABST
    Figure CN118230296B_ABST
Patent Text Reader

Abstract

The present invention discloses a lightweight fatigue driving detection and tracking method, which includes obtaining a driver image to be identified; inputting the driver image to be identified into a pre-built YOLOv5s network model to obtain a facial feature target frame; using the Deepsort algorithm to continuously track and optimize the facial feature target frame to obtain label information; and determining the driver's driving state including fatigue driving state and normal driving state based on the label information, the number of consecutive eye closing frames of the driver, the number of consecutive yawning frames of the driver, and the calculated PERCLOS score. This model can reduce the probability of driver facial information loss during the detection process, and significantly reduce the model size while maintaining high accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a lightweight fatigue driving detection and tracking method, and belongs to the technical field of vehicle networking intelligent detection related to fatigue driving detection improved by a detection algorithm based on deep learning. Background Art

[0002] In today's society, with the continuous advancement of urbanization and the accelerated pace of life, people's dependence on transportation has increased significantly. Although the popularity of cars makes travel more convenient, it also inevitably brings traffic accidents. Fatigue driving is usually caused by the driver's long-term driving tasks, which leads to a decline in his driving ability, control and judgment ability, thus posing a serious threat to the life safety of himself and the surrounding pedestrians. Therefore, studying an efficient fatigue driving detection method is of great significance to traffic safety.

[0003] At present, fatigue driving detection is mainly divided into two detection methods, namely active detection and passive detection. Active detection is mainly based on the driver's self-perception, relying on the driver's self-feeling and self-evaluation to judge the fatigue level. This method is convenient and intuitive, but human subjective feelings cannot accurately judge the real driving state. Passive detection methods can be divided into three methods. The first passive detection method is to detect fatigue driving based on the driver's physiological signals. This method can achieve a high detection accuracy, but the collection of physiological signals requires direct contact with the driver's body, which is easy to affect the driver's operation and the cost of wearable equipment is high. The second passive detection method is to detect fatigue driving based on the vehicle's driving trajectory. This method does not require direct contact with the driver, but is easily affected by road conditions and weather changes, and has low accuracy and real-time performance. The third passive detection method is a computer vision detection method that uses a camera to collect the driver's facial feature information and analyzes the category features through a deep learning network to determine the fatigue state. This method is easy to deploy and can take into account both real-time and accuracy. It is currently the main hotspot in the research of fatigue driving detection.

[0004] Early computer vision-based fatigue driving detection was mainly determined by a single feature. Wang et al. proposed a fatigue monitoring method for driver mouth movement. By continuously monitoring and tracking the driver's mouth movement, the mouth is classified and a warning is issued when the driver is fatigued. Alioua et al. proposed a support vector machine (SVM) and circular Hough transform (CHT) mouth detection algorithm based on face extraction. Tao et al. used six key points to locate the driver's eyes, and then input the extracted histogram of oriented gradients (HOG) features into the SVM classifier to identify the eye state. Zhuang et al. proposed an eye opening fatigue detection method based on pupil and iris segmentation. This method extracts eye images through the Dlib algorithm, uses a segmentation network to extract pupil and iris features in the eye image, and uses a decision network to evaluate the eye opening. Finally, the PERCLOS value is calculated based on the eye opening to predict the driver's state. The method is tested on the NTHU-DDD dataset. However, the above methods only detect fatigue through a single feature, do not take into account the driver's multiple fatigue states, and have poor robustness.

[0005] In response to these problems, researchers have proposed a fatigue driving detection method based on multi-feature fusion. Zheng et al., considering the uncertainty of driving environment and driver fatigue state, proposed a MAX-MIN detection algorithm based on deep learning. The face is recognized by shufflenet network, and the coordinates of facial key points are obtained by Dlib key point algorithm, and finally the values ​​of EAR and MAR are obtained. Liu et al. extracted a fatigue detection algorithm based on multi-features of face. The algorithm uses MTCNN to locate the driver's eyes and mouth, and then extracts dynamic and static features from the local face image. Finally, the extracted features are combined with a dual-channel neural network to classify the image data. Ji et al. proposed a fatigue driving detection algorithm based on multi-index fusion and state recognition network. The face and facial key points are detected by MTCNN, the driver's eye and mouth images are extracted, and the image features are extracted by the state recognition network. Finally, the extracted features are fused to establish a fatigue judgment model to analyze the driver's fatigue state. Zhao et al. used MTCNN for face detection and key point positioning, and then classified the state of eyes and mouth. The proposed EM-CNN algorithm detects the state of eyes and mouth from the key parts extracted by features. Finally, the driver's fatigue state is identified by combining MAR, PERCLOS and mouth opening rate. You et al. proposed a driver fatigue detection algorithm based on face motion information entropy. The algorithm used the improved YOLOv3-tiny to locate the driver's facial information in different scenarios, and then used the Dlib algorithm and the coordinates of facial feature points to construct FFT (face feature triangle). The constructed FFT information was used to create a face feature vector. Finally, the face feature vector was extracted, and the facial motion information entropy was extracted through a sampler. The information entropy was compared with the set threshold to determine the driver's fatigue state.

[0006] As deep learning networks continue to be optimized and improved, the number of detection model parameters has also increased significantly, which is not conducive to the deployment of vehicle-mounted edge devices. In addition, the above studies lack consideration of the problem of facial information loss caused by the driver turning his head during the detection process, resulting in poor continuous detection. Summary of the invention

[0007] The technical problem to be solved by the present invention is to overcome the defects of the prior art, provide a lightweight fatigue driving detection and tracking method, obtain the driver image to be identified; input the driver image to be identified into the pre-built YOLOv5s network model to obtain the face feature target frame; use the Deepsort algorithm to continuously track and optimize the face feature target frame to obtain label information; based on the label information, the number of continuous eye closing frames of the driver, the number of continuous yawning frames of the driver and the calculated PERCLOS score, determine the driver's driving state including fatigue driving state and normal driving state. First, the present invention introduces a lightweight Mobilenetv3-small to reconstruct the YOLOv5s backbone. In order to cope with the significant decrease in detection accuracy after the lightweight backbone, the original SE attention mechanism of Mobilenetv3-small is replaced with the ECA attention mechanism, avoiding the problem of dimensionality reduction of the SE attention mechanism and improving the detection accuracy. Then, the C3 module in the neck network is combined with the Triplet Attention Module (TAM attention mechanism) so that the YOLOv5s network model can strengthen the attention to contextual information in the learning task and capture more complex associations and dependencies. In addition, the problem of large error in prediction box regression calculation using Wise-EIoU Loss is used to improve the robustness of model detection. Finally, based on the improved YOLOv5s lightweight face feature detection algorithm and DeepSort facial feature tracking algorithm, the number of continuous eye closure frames, the number of continuous yawning frames, and the PERCLOS score multi-feature fusion are used to construct a fatigue driving detection model.

[0008] Preferably, the present invention provides a lightweight fatigue driving detection and tracking method, comprising:

[0009] Acquire a driver image to be identified;

[0010] Input the driver image to be identified into the pre-built YOLOv5s network model to obtain the facial feature target frame;

[0011] Use the Deepsort algorithm to continuously track and optimize the facial feature target frame to obtain label information;

[0012] Based on the label information, the number of consecutive eye-closing frames of the driver, the number of consecutive yawning frames of the driver and the calculated PERCLOS score, the driver's driving state, including fatigue driving state and normal driving state, is determined.

[0013] Preferably, the pre-built YOLOv5s network model includes a Backbone module, a Neck module and a Head module connected in sequence, wherein the Backbone module includes a Conv2d layer, a first MobileNet_ECA layer, a first MobileNet_Block layer, a second MobileNet_Block layer, a second MobileNet_ECA layer, a third MobileNet_ECA layer, a fourth MobileNet_ECA layer, a fifth MobileNet_ECA layer, a sixth MobileNet_ECA layer, a seventh MobileNet_ECA layer, an eighth MobileNet_ECA layer, a ninth MobileNet_ECA layer and an SPPF layer connected in sequence;

[0014] The first MobileNet_ECA layer, the second MobileNet_ECA layer, the third MobileNet_ECA layer, the fourth MobileNet_ECA layer, the fifth MobileNet_ECA layer, the sixth MobileNet_ECA layer, the seventh MobileNet_ECA layer, the eighth MobileNet_ECA layer, and the ninth MobileNet_ECA layer introduce the ECANet channel attention mechanism, including a global average pooling layer, a one-dimensional convolution layer, and a normalization layer connected in sequence.

[0015] Preferably, the Neck module includes a second CBS K1-S1 layer, a second Upsample layer, a second Concat layer, a first CTA layer, a first CBS K1-S1 layer, a first Upsample layer, a first Concat layer, a second CTA layer, a first CBS k3-s2 layer, a third Concat layer, a third CTA layer, a second CBS k3-s2 layer, a fourth Concat layer, and a fourth CTA layer.

[0016] The second CBS K1-S1 layer, the second Upsample layer, the second Concat layer, the first CTA layer, the first CBS K1-S1 layer, the first Upsample layer, and the first Concat layer are connected in sequence. The second CTA layer, the first CBS k3-s2 layer, the third Concat layer, the third CTA layer, the second CBS k3-s2 layer, the fourth Concat layer, and the fourth CTA layer are connected in sequence.

[0017] The output end of the second MobileNet_Block layer is connected to the input end of the first Concat layer, the output end of the sixth MobileNet_ECA layer is connected to the second Concat layer, the output end of the SPPF layer is connected to the second CBS K1-S1 layer, the output end of the first Concat layer is connected to the input end of the second CTA layer, the output end of the first CBS K1-S1 layer is connected to the input end of the third Concat layer, the output end of the second CBS K1-S1 layer is connected to the input end of the fourth Concat layer, and the output ends of the second CTA layer, the third CTA layer and the fourth CTA layer are connected to the Head module.

[0018] Preferably, the SPPF layer includes a first CBS layer, a first MaxPool2d layer, a second MaxPool2d layer, a third MaxPool2d layer, a first Contact layer, and a second CBS layer connected in sequence, and an output end of the first CBS layer, an output end of the first MaxPool2d layer, and an output end of the second MaxPool2d layer are connected to an input end of the first Contact layer;

[0019] The first CTA layer, the second CTA layer, the third CTA layer and the fourth CTA layer each include the seventh CBS layer, the fifth CBS layer, the Bottleneck layer, the TripletAtt layer, the second Contact layer and the sixth CBS layer.

[0020] The fifth CBS layer, the Bottleneck layer, the TripletAtt layer, the second Contact layer and the sixth CBS layer are connected in sequence, the output end of the fifth CBS layer is connected to the input end of the seventh CBS layer, and the output end of the seventh CBS layer is connected to the input end of the second Contact layer;

[0021] The Bottleneck layer includes the third CBS layer, the fourth CBS layer and the Add layer connected in sequence, and the first CBS layer, the second CBS layer, the third CBS layer, the fourth CBS layer, the fifth CBS layer, the sixth CBS layer and the seventh CBS layer include the Conv layer, the BN layer and the SiLU layer connected in sequence.

[0022] Prior to this, the pre-built YOLOv5s network model uses Wise-EIoU Loss to calculate the rectangle loss value.

[0023] Prioritize the pre-built YOLOv5s network model, including:

[0024] Construct a YOLOv5s network model including a Backbone module, a Neck module, and a Head module connected in sequence;

[0025] The YOLOv5s network model is trained using the public datasets YawDD, CEW and Drozy. The public datasets YawDD, CEW and Drozy include pictures of male drivers with eyes open, eyes closed, yawning, non-yawning, normal driving, fatigued driving, female drivers with eyes open, eyes closed, yawning, non-yawning, normal driving and fatigued driving.

[0026] Preferably, a lightweight fatigue driving detection and tracking system comprises:

[0027] A driver image acquisition module, used to acquire an image of a driver to be identified;

[0028] A facial feature target frame acquisition module is used to input the driver image to be identified into a pre-built YOLOv5s network model to obtain a facial feature target frame;

[0029] The label information acquisition module is used to use the Deepsort algorithm to continuously track and optimize the facial feature target frame to obtain label information;

[0030] The driver driving state determination module is used to determine the driver's driving state including fatigue driving state and normal driving state based on label information, the number of consecutive eye closing frames of the driver, the number of consecutive yawning frames of the driver and the calculated PERCLOS score.

[0031] Preferably, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the program.

[0032] Preferably, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0033] The beneficial effects achieved by the present invention are:

[0034] The present invention combines the improved YOLOv5s target detection algorithm and DeepSort tracking algorithm to propose a lightweight fatigue driving detection and tracking method. First, the present invention introduces a lightweight Mobilenetv3-small to reconstruct the YOLOv5s backbone. In order to cope with the significant drop in detection accuracy after the lightweight backbone, the original SE attention mechanism of Mobilenetv3-small is replaced with the ECA attention mechanism, which avoids the problem of dimension reduction of the SE attention mechanism and improves the detection accuracy. Then, the C3 module in the neck network is combined with the Triplet Attention Module (TAM attention mechanism) to enable the YOLOv5s network model to strengthen the attention to contextual information in the learning task and capture more complex associations and dependencies. In addition, the problem of large errors in the prediction box regression calculation using Wise-EIoU Loss is improved to improve the robustness of model detection. Finally, based on the improved YOLOv5s lightweight face feature detection algorithm and DeepSort facial feature tracking algorithm, the number of continuous eye closure frames, the number of continuous yawning frames, and the PERCLOS score multi-feature fusion are used to construct a fatigue driving detection model. The experimental results of the present invention show that the model can reduce the probability of driver facial information loss during the detection process, and significantly reduce the model size while maintaining high accuracy, which can meet the deployment requirements of vehicle-mounted edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0036] Figure 1 This is the structure diagram of YOLOv5s;

[0037] Figure 2 It is a structural diagram of deep convolution;

[0038] Figure 3 It is the structural diagram of point-by-point convolution;

[0039] Figure 4 This is the structure diagram of the SENet module;

[0040] Figure 5 This is the structure diagram of ECANet;

[0041] Figure 6 This is the structure diagram of Mobilenet_ECA;

[0042] Figure 7 It is a structural diagram of the triplet network structure;

[0043] Figure 8 It is a structural diagram of the YOLOv5s network model in the present invention;

[0044] Fig. 9 It is a flow chart of fatigue driving detection;

[0045] Fig.10 It is a schematic diagram of the mAP comparison between the YOLOv5s before improvement and the present invention;

[0046] Fig.11 This is the precision-recall curve of YOLOv5s before improvement;

[0047] Fig.12 It is a curve diagram of the precision-recall rate of the present invention. DETAILED DESCRIPTION

[0048] To facilitate the technical solution of the application, some concepts involved in the application are first explained below.

[0049] YOLOv5 is a single-stage target detection algorithm. It combines many advantages of previous generations of YOLO series and has good accuracy and inspection speed in target detection tasks. YOLOv5 controls the depth of the model and the number of channels between layers through depth factors and width factors. It is divided into YOLOv5-N, YOLOv5-S, YOLOv5-M, YOLOv5-L and YOLOv5-X. YOLOv5 is divided into four parts, namely Input, Backbone, Neck and Head. Input means inputting 640×640×3 pictures. This stage includes an image preprocessing stage, which is to scale the input image to the input size of the network and perform normalization and other operations. Backbone is usually used to extract some feature representations, which not only uses the C3 (CSPDarknet53) structure, but also the Focus structure. The Neck part uses the classic FPN and PAN design, which can further improve the diversity and robustness of the features. Head is used to complete the output of the target detection results at the end. This paper comprehensively considers the relationship between detection accuracy and model size, and takes YOLOv5s as the benchmark model. The network structure of YOLOv5s is as follows: Figure 1 As shown in Figure 2, the algorithm performs better in practical application scenarios and has lower computational cost.

[0050] YOLOv5s uses the CSPDarknet backbone, which has advantages in accuracy, but the computational cost is relatively high. Considering the limitations of the current computing costs of vehicle-mounted edge devices, the present invention replaces the original YOLOv5s facial feature extraction backbone with Mobilenetv3-small, reducing the amount of calculation and parameters, achieving lightweight models, and facilitating the deployment of edge devices. Mobilenetv3-small mainly uses depthwise separable convolution, which greatly reduces the amount of calculation and parameters of depthwise separable convolution compared to ordinary convolution. The number of parameters of depthwise separable convolution P dwconv and the amount of computation G dwconv The calculation formula is as follows:

[0051]

[0052] The number of parameters of ordinary convolution P conv and the amount of computation G conv The calculation formula is as follows:

[0053]

[0054] In the above formula: N is the number of convolution kernels; H and W are the height and width of the output feature map respectively; M i is the number of channels of the input feature map; K s is the convolution kernel size. In summary, the number of parameters and computational complexity of depthwise separable convolution is reduced by:

[0055]

[0056] Depthwise convolution can be divided into two processes: depth-wise convolution and pointwise convolution. The specific structure is as follows: Figure 2 ,3. In depth-wise convolution, one convolution kernel processes one channel, and the number of convolution kernels is the same as the channels of the previous layer. Then the output feature maps are spliced ​​to get the final output. In pointwise convolution, the size of the convolution kernel is 1×1×M, where M is the number of channels of the previous layer. Pointwise convolution can change the number of output channels on the one hand, and on the other hand, it can perform channel fusion on the feature maps output by channel-by-channel convolution.

[0057] See also Figure 1The present application discloses a lightweight fatigue driving detection and tracking method, comprising: obtaining a driver image to be identified; inputting the driver image to be identified into a pre-built YOLOv5s network model to obtain a facial feature target frame; performing continuous target tracking optimization on the facial feature target frame using a Deepsort algorithm to obtain label information; based on the label information, the number of consecutive eye closing frames of the driver, the number of consecutive yawning frames of the driver and the calculated PERCLOS score, determining the driver's driving state including a fatigue driving state and a normal driving state.

[0058] In an embodiment of the present application, the pre-built YOLOv5s network model includes a Backbone module, a Neck module and a Head module connected in sequence, wherein the Backbone module includes a Conv2d layer, a first MobileNet_ECA layer, a first MobileNet_Block layer, a second MobileNet_Block layer, a second MobileNet_ECA layer, a third MobileNet_ECA layer, a fourth MobileNet_ECA layer, a fifth MobileNet_ECA layer, a sixth MobileNet_ECA layer, a seventh MobileNet_ECA layer, an eighth MobileNet_ECA layer, a ninth MobileNet_ECA layer and an SPPF layer connected in sequence;

[0059] The first MobileNet_ECA layer, the second MobileNet_ECA layer, the third MobileNet_ECA layer, the fourth MobileNet_ECA layer, the fifth MobileNet_ECA layer, the sixth MobileNet_ECA layer, the seventh MobileNet_ECA layer, the eighth MobileNet_ECA layer, and the ninth MobileNet_ECA layer introduce the ECANet channel attention mechanism, including a global average pooling layer, a one-dimensional convolution layer, and a normalization layer connected in sequence.

[0060] In the embodiment of the present application, the Neck module includes a second CBS K1-S1 layer, a second Upsample layer, a second Concat layer, a first CTA layer, a first CBS K1-S1 layer, a first Upsample layer, a first Concat layer, a second CTA layer, a first CBS k3-s2 layer, a third Concat layer, a third CTA layer, a second CBS k3-s2 layer, a fourth Concat layer, and a fourth CTA layer.

[0061] The second CBS K1-S1 layer, the second Upsample layer, the second Concat layer, the first CTA layer, the first CBS K1-S1 layer, the first Upsample layer, and the first Concat layer are connected in sequence. The second CTA layer, the first CBS k3-s2 layer, the third Concat layer, the third CTA layer, the second CBS k3-s2 layer, the fourth Concat layer, and the fourth CTA layer are connected in sequence.

[0062] The output end of the second MobileNet_Block layer is connected to the input end of the first Concat layer, the output end of the sixth MobileNet_ECA layer is connected to the second Concat layer, the output end of the SPPF layer is connected to the second CBS K1-S1 layer, the output end of the first Concat layer is connected to the input end of the second CTA layer, the output end of the first CBS K1-S1 layer is connected to the input end of the third Concat layer, the output end of the second CBS K1-S1 layer is connected to the input end of the fourth Concat layer, and the output ends of the second CTA layer, the third CTA layer and the fourth CTA layer are connected to the Head module.

[0063] In the embodiment of the present application, the SPPF layer includes a first CBS layer, a first MaxPool2d layer, a second MaxPool2d layer, a third MaxPool2d layer, a first Contact layer, and a second CBS layer connected in sequence, and the output end of the first CBS layer, the output end of the first MaxPool2d layer, and the output end of the second MaxPool2d layer are connected to the input end of the first Contact layer;

[0064] The first CTA layer, the second CTA layer, the third CTA layer and the fourth CTA layer each include the seventh CBS layer, the fifth CBS layer, the Bottleneck layer, the TripletAtt layer, the second Contact layer and the sixth CBS layer.

[0065] The fifth CBS layer, the Bottleneck layer, the TripletAtt layer, the second Contact layer and the sixth CBS layer are connected in sequence, the output end of the fifth CBS layer is connected to the input end of the seventh CBS layer, and the output end of the seventh CBS layer is connected to the input end of the second Contact layer;

[0066] The Bottleneck layer includes the third CBS layer, the fourth CBS layer and the Add layer connected in sequence, and the first CBS layer, the second CBS layer, the third CBS layer, the fourth CBS layer, the fifth CBS layer, the sixth CBS layer, and the seventh CBS layer include the Conv layer, the BN layer and the SiLU layer connected in sequence. In the embodiment of the present application, the pre-built YOLOv5s network model uses Wise-EIoU Loss to calculate the rectangular box loss value.

[0067] In an embodiment of the present application, a YOLOv5s network model is pre-constructed, including: constructing a YOLOv5s network model including a Backbone module, a Neck module and a Head module connected in sequence; using public data sets YawDD, public data sets CEW and public data sets Drozy to train the YOLOv5s network model, the public data sets YawDD, public data sets CEW and public data sets Drozy including pictures of male drivers with eyes open, pictures of male drivers with eyes closed, pictures of male drivers yawning, pictures of male drivers without yawning, pictures of male drivers driving normally, pictures of male drivers with fatigued driving, pictures of female drivers with eyes open, pictures of female drivers with eyes closed, pictures of female drivers with yawning, pictures of female drivers without yawning, pictures of female drivers driving normally and pictures of female drivers with fatigued driving.

[0068] In an embodiment of the present application, a lightweight fatigue driving detection and tracking system includes: a driver image acquisition module, used to acquire a driver image to be identified; a facial feature target frame acquisition module, used to input the driver image to be identified into a pre-built YOLOv5s network model to obtain a facial feature target frame; a label information acquisition module, used to use a Deepsort algorithm to perform continuous target tracking optimization on the facial feature target frame to obtain label information; a driver driving state determination module, used to determine the driver's driving state including a fatigue driving state and a normal driving state based on the label information, the number of continuous eye closing frames of the driver, the number of continuous yawning frames of the driver and the calculated PERCLOS score.

[0069] In an embodiment of the present application, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0070] In an embodiment of the present application, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0071] Mobilenetv3-small also includes the SENet module, which includes two operations: compression and excitation. The specific structure is as follows Figure 4As shown in the figure, H, W, and C represent the height, width, and number of channels of the feature map. First, the input feature map [C, H, W] is squeezed (average pooled), and the size of the pooled feature map is [C, 1, 1]. Then, the Excitation operation is performed, and the input vector is processed through two fully connected layers. Finally, the Scale operation is performed, and the calculated channel weights are multiplied by the two-dimensional matrix of the corresponding channel of the original feature map to obtain the output result.

[0072] Conventional MobileNetV3-Small uses the SENet channel attention mechanism. The dimension reduction of the SENet channel attention mechanism will bring side effects to the channel attention mechanism, and the efficiency of capturing the dependencies between all channels is not high, which may cause the loss of facial feature extraction. For this reason, the present invention introduces the ECANet channel attention mechanism. Compared with the fully connected layer of the SENet channel attention mechanism, the ECANet channel attention mechanism directly adds a 1×1 convolution layer after the global average pooling layer to replace the fully connected layer, avoiding the dimension reduction problem of the SENet channel attention mechanism, more effectively capturing cross-channel interactions, and the ECANet channel attention mechanism only involves a small number of parameters compared to the SENet channel attention mechanism. The brief structure of ECANet is as follows: Figure 5 shown.

[0073] In order to make the MobileNet_ECA module more effective in extracting facial fatigue features, first given the aggregated features [C, 1, 1] obtained by average pooling, the MobileNet_ECA module generates channel weights by performing a one-dimensional convolution with a convolution kernel size of k, and finally multiplies the normalized channel weights and the original input feature map channel by channel to generate a weighted feature map. The convolution kernel k is adaptively determined by mapping the channel dimension C:

[0074]

[0075] In the formula, k represents the convolution kernel size, C represents the number of channels, ∥ odd Indicates that k can only be an odd number, γ is used to change the number of channels C, and b is used to change the ratio of the convolution kernel and size. The present invention selects k as 3 through experimental analysis, which can fully integrate the interaction between some channels, reduce the number of parameters while maintaining performance. The improved MobileNet_ECA module mainly implements channel separable convolution and ECA channel attention mechanism and residual connection. The structure of the Mobilenet_ECA module is as follows: Figure 6 shown.

[0076] The overall network structure of the improved MobilenetV3-small is shown in Table 1.

[0077] Table 1.Improved network structure of MobilenetV3-small

[0078]

[0079]

[0080] In Table 1, the first column Input represents the input size, and the first row of 640 2 Indicates the length and width of the input feature, and 3 represents the number of input feature channels. The second column Operator indicates the type and size of the convolution. [conv2d, 3×3] indicates a convolution of size 3×3, bneck indicates the use of depthwise separable convolution, and ECAbneck indicates the use of ECA attention mechanism and depthwise separable convolution. The third column exp size indicates the number of channels after the dimension increase of the first layer 1×1 convolution of bneck. The fourth column ECA indicates whether the MobileNet_ECA module is used. The fifth column AF (activation function) indicates the type of activation function used. Among them, HS represents the hard-swish activation function, and RE represents the use of the ReLU activation function. Although the lightweight backbone provides favorable conditions for the deployment of vehicle-mounted edge devices, its detection accuracy will also be affected while the number of parameters and calculations are reduced. Integrating the attention mechanism into the deep learning network can enhance the network performance. The attention mechanism is a core technology in deep learning. It can highly focus on important information and ignore irrelevant information in the network. The attention mechanism enables the model to dynamically learn different inputs, thereby continuously adjusting the weights to select important information from a large amount of scattered information. In order to enhance the network's attention to key facial features, the present invention introduces the TAM attention mechanism, which is an almost parameter-free triple attention mechanism. Compared with attention mechanisms such as CBAM and SE, which combine dimensionality reduction functions in calculations, this dimensionality reduction is redundant for capturing dependencies between channels. The TAM attention mechanism can establish dependencies between dimensions without performing dimensionality reduction operations, which means that the TAM attention mechanism can obtain richer target features at negligible computational costs, and can better understand the contextual relationships between targets, helping the model to better identify and locate targets and improve detection accuracy.

[0081] like Figure 7 As shown in the figure, the TAM attention mechanism is a three-branch structure that interacts across dimensions to calculate the attention weights. The top branch calculates the attention weights of the channel dimension C and the spatial dimension W, the middle part calculates the attention weights of the channel dimension C and the spatial dimension H, and the last branch integrates the interaction between the spatial dimensions H and W. When the input tensor is χ(C×H×W), the specific operations of the three branches are as follows:

[0082] The first branch rotates the input χ 90° counterclockwise along the H axis (Permute) to obtain the rotation tensor then After Z-pool operation, we get the tensor Z-pool performs average pooling and maximum pooling in the W dimension, retaining rich features while reducing the network depth. Its mathematical formula is as follows:

[0083] Z-pool = [MaxPool 0d (χ),AvgPool 0d (x)]

[0084] Where, 0 d represents the 0th dimension of average pooling and maximum pooling. Then, the pooled tensor Through a standard convolutional layer of size 7×7 and a normalization layer with an intermediate output dimension of (1×H×C), the attention weight is finally generated through a Sigmoid activation layer. The attention weight is rotated 90° along the H axis to match the shape of the input tensor χ.

[0085] In the second branch, the input χ is rotated 90° counterclockwise along the W axis to obtain the rotation tensor After Z-pool operation, we get the tensor Then, the attention weights are generated through a 7×7 standard convolution layer, a Batch Norm layer, and a Sigmoid activation function. The attention weights are rotated 90° along the W axis to have the same shape as the input tensor χ.

[0086] Similarly, in the last branch, the input χ is channel-pooled to obtain the output tensor χ3 (2×H×W). Then it passes through a 7×7 standard convolution layer, a Batch Norm layer, and a Sigmoid activation function to generate attention weights. The attention weights are connected to the residual to obtain the output tensor (C×H×W).

[0087] Finally, the output tensors (C×H×W) generated by each branch are aggregated using the average method.

[0088] In the context of fatigue driving detection, the neck part performs feature fusion and refinement on the key facial features extracted by the backbone part, which facilitates accurate feature detection in the head part. The present invention embeds the TAM attention mechanism into the C3 module of the neck part to form a new CTA module. The CTA module solves the problem of context information loss in traditional methods of calculating single-dimensional attention weights by integrating cross-dimensional information interaction, thereby enabling the network to capture more complex associations and dependencies in learning tasks.

[0089] The conventional YOLOv5s network model uses CIoU Loss to calculate the rectangular box loss value. The specific calculation formula of CIoU Loss is as follows:

[0090] L=L obj +L cls +L bbox

[0091] Where L obj Represents the loss of prediction confidence, L cls represents the classification loss, L bbox Represents the loss of predicting the position of the rectangular box. CIoU Loss adds the overlapping area, center distance, and aspect ratio of the rectangular box into the calculation at the same time, which improves the stability of training and speeds up convergence. However, CIoU Loss does not really represent the true difference between the height of the rectangular box and its confidence, which will affect the accuracy of the algorithm in evaluating the similarity of the bounding box.

[0092] Aiming at the problem of CIoU Loss, the present invention uses Wise-EIoU Loss instead of CIoU Loss. Wise-EIoU can be divided into attention-based Wise-EIoU v1, Wise-EIoU v2 with monotonic focusing mechanism (FM), and Wise-EIoU v3 with dynamic non-monotonic FM. Wise-EIoU v3 can assign smaller gradient gains to anchor boxes with large outliers, effectively preventing large gradient losses in low-quality samples. This makes full use of the non-monotonic adjustment potential of static FM, reduces the penalty for distance and aspect ratio of low-quality samples, speeds up the convergence of the model, and improves the extraction performance of local facial features.

[0093] Therefore, the present invention adopts the loss function of the Wise-EIoU v3 reconstruction baseline model, which effectively balances the impact of high-quality and low-quality sample differences on the model and improves the performance of extracting local features of the face. The intersection over union (IoU) loss between the ground truth box and the predicted box can be obtained, expressed as L IoU :

[0094]

[0095] Then define the penalty factor R WIoU For low-quality anchor boxes L IoU On this basis, the gradient gain r is used to adjust the influence of the focus of the constraint box regression on the quality of the anchor box, thereby obtaining the loss L of WIoU v3. IoU :

[0096]

[0097] L WIoUv3 =rRWIoU L IoU

[0098] In the equation, the outlier The hyperparameters δ and α are used to control

[0099] When the driving environment changes or the driver's head swings and causes the detection image to lose frames, the facial feature detection algorithm will have short-term failure and misjudgment, missed fatigue features and other problems, all of which will lead to abnormal fatigue recognition and failure to accurately identify the fatigue state, which may cause traffic accidents. Therefore, the present invention uses the DeepSort algorithm with facial feature target tracking to continuously track and optimize the key features of the face, such as opening and closing the eyes and the mouth, so as to improve the accuracy of key feature recognition.

[0100] The DeepSort algorithm predicts the detection frames corresponding to its determined Tracks and uncertain Tracks through Kalman filtering. The detection frames of the determined Tracks and the facial feature target frames are cascade matched. As long as the Tracks detection frame matches the facial feature target frame, the movement information of the eyes, mouth features and head of the facial feature target in the facial feature target frame will be saved. There are two possible results after cascade matching. The first is that the Tracks detection frame and the facial feature target frame are successfully paired, which means that the previous frame and the next frame are tracked successfully. The corresponding facial feature target frame is updated with the corresponding Tracks variable through Kalman filtering, and the label information is directly output. The second is that the facial feature target frame and the Tracks detection frame are mismatched. At this time, the previous uncertain Tracks detection frame, the mismatched Tracks detection frame and the facial feature target frame are IOU matched, and then the cost matrix C (cost matrix) is calculated based on the result of IOU matching. The calculation formula is:

[0101] C = 1-IOU

[0102] Finally, all cost matrices are used as input to the Hungarian algorithm to obtain linear matching results, and finally the label information of the previous frame is retained for output.

[0103] The improved face feature tracking model is as follows Figure 8As shown in the figure, when the driver is in a fatigued state, the time it takes for his eyes to go from open to closed and then back to open will become longer, and the time of closing eyes will become longer within a certain period of time, the number of blinks will also decrease, and the number of yawns will increase. At this time, the driver's fatigue state will be very obvious. Therefore, the length of time the eyes are closed and the number of yawns can reflect the driver's fatigue state to a certain extent. PERCLOS refers to the time ratio of the number of frames with closed eyes to the total number of frames in a unit time frame. In the specified unit time frame t0, the PERCLOS score of closed eyes is calculated based on the percentage of the number of frames with closed eyes in the unit time frame:

[0104]

[0105] In the above formula, P e represents the PERCLOS score with eyes closed, t e1 and t e2 Represents the total number of eye-closing frames in the unit time frame t0.

[0106] In order to more accurately determine whether the driver is in a fatigue state, the present invention integrates the number of continuous eye closure frames, the number of continuous yawning frames and the PERCLOS score for judgment, and sets a fatigue threshold. It is determined that within a 75-frame unit time frame (t0=75 frames), if the driver's eye PERCLOS score is lower than 0.15, then he is in an awake state; when the PERCLOS score is greater than 0.15 and less than 0.30, and the number of continuous yawning frames at this time exceeds 30 frames and less than 60 frames, he is in a mild fatigue state; when the number of continuous yawning frames exceeds 60 frames, he is in a complete fatigue state; when the PERCLOS score is greater than 0.30, he is in a complete fatigue state. At the same time, when the driver closes his eyes for more than 20 frames in a row, he is also in a complete fatigue state. The fatigue driving detection process is as follows: Fig. 9 First, the input image is subjected to the YOLO algorithm to obtain the target detection frame, and then the continuous target tracking prediction is performed through Deepsort. Finally, the driver's driving state is judged by combining the number of continuous eye closure frames, the number of continuous yawning frames and the PERCLOS score, and the two states of fatigue and normal are output.

[0107] In order to solve the problem of insufficient diversity and insufficient sample size of the current public fatigue driving detection datasets, the present invention adopts multiple comprehensive datasets, including the public datasets YawDD, CEW and Drozy, which contain datasets of pictures of male drivers with eyes open and closed, female drivers with eyes open and closed, yawning, non-yawning, normal driving and fatigue driving.

[0108] The present invention captures a picture of the public data set YawDD simulated driving video data set at a frequency of 20 frames, and then adds the public data set CEW and the public data set Drozy, and uses the closed-eye state data set in the public data set CEW to make up for the problem of too few closed-eye features in the public data set YawDD, and the night scene data set of the public data set Drozy makes up for the problem of insufficient night driving scenes in the public data set YawDD, thereby enhancing the robustness and generalization ability of the model. The data set of the present invention has a total of 7321 images, and uses Labelimg to annotate the self-built data set to generate 1441 yawn bounding box labels, 4271 non_yawn bounding box labels, 4687 open_eye bounding box labels, and 2304 close_eye bounding box labels. There are a total of 4 categories and 12703 driver facial feature bounding box labels. The data set is divided into a training set and a verification set at a ratio of 8:2, with a total of 5486 images in the training set and a total of 1466 images in the verification set. Fig.10 A visual representation of the dataset is shown in .

[0109] The experiment of the present invention uses 640×640 pixels as the input image size. For the training parameters, the batch size is selected as 8, the training epoch is set to 100, the momentum is fixed to 0.937, and the initial learning rate is set to 0.001. The specific experimental conditions are shown in Table 2.

[0110] Table 2. Experimental conditions

[0111] Configuration Details CPU I5-12500H GPU RTX 3060 Memory 16 GB Deep learning framework Pytorch 1.13.0 Python 3.8 CUDA 11.7

[0112] In order to objectively evaluate the performance of the model, this paper uses mean average precision (mAP), floating point operations (FLOPs), parameters (Params) and model size as evaluation indicators. mAP is an indicator for evaluating the accuracy of the model. The higher the mAP, the higher the accuracy of the model and the stronger the overall performance. FLOPs, Params and Size are indicators for evaluating the lightweight of the model. The lower these three indicators are, the lighter the model is. The calculation formula of mAP is as follows:

[0113]

[0114] In the above formula, P represents precision, R represents recall, TP represents the number of correctly predicted positive samples, FP represents the number of negative samples classified as positive, TN represents the number of correctly predicted negative samples, FN represents false negative samples, and n represents the total number of detected categories. In the present invention, n is 4.

[0115] Table 3 shows the experimental comparison results of introducing Mobilenetv3-small lightweight backbone into YOLOv5s and replacing SE attention mechanism with ECA attention mechanism. It can be seen from Table 3 that after the original YOLOv5s replaced the Mobilenetv3-small lightweight backbone, Params decreased significantly by 49.8%, FLOPs decreased significantly by 63.2%, but mAP decreased by 0.9%. When the present invention replaces the original SE attention mechanism of Mobilenetv3-small with the ECA attention mechanism, the application of the ECA module under different convolution kernel sizes K is improved compared with the original lightweight backbone, and Params is also reduced by 12.8% compared with the original lightweight backbone, indicating that applying different attention mechanisms in the lightweight backbone can continue to reduce the size of the model, and the appropriate attention mechanism will increase the detection accuracy. At the same time, comparing the effects of different convolution kernel sizes K on the ECA module, the mAP is the highest when the convolution kernel size is 3, reaching 94.4%. Therefore, the present invention selects Mobilenet-ECA to reconstruct the YOLOv5s backbone part as the best choice.

[0116] Table 3. Comparative experiments of improved Mobilenet-ECA backbone network and different convolution kernel sizes

[0117]

[0118] In order to verify whether the improvement proposed in the present invention is effective for the performance of facial feature detection, the present invention designs a set of ablation experiments and performs them under the same experimental conditions. The specific results are shown in Table 4.

[0119] Table 4. Experimental results of ablation

[0120]

[0121] By introducing the improved Mobilenet-ECA lightweight backbone into the original YOLOv5s network model, Params is significantly reduced by 56.3%, and the model size and FLOPs are reduced by 54.1% and 63.9% respectively. It can be seen that the lightweight degree of the model is significantly improved at this time, but due to the sharp reduction in model size, mAP also drops by 0.1%. The 0.1% accuracy loss is acceptable for the introduction of the improved lightweight backbone. By introducing the contextual information enhancement module CTA, mAP is improved by 0.8% compared with the model after the introduction of the improved lightweight backbone, and the improvement effect is obvious. Due to the almost parameter-free attention mechanism used in the present invention, Params, FLOPs and model size are almost unchanged at this time. Using Wise-EIOU to replace the CIoU loss function, mAP is improved by 0.2% compared with the model after the introduction of the improved Mobilenet-ECA backbone. After integrating the above three improvements, the improved algorithm has a 1.0% increase in mAP, a 56.3% decrease in Params, a 63.2% decrease in FLOPs, and a 54.0% decrease in model size compared to the original YOLOv5s model, indicating that the improved model has stronger detection performance than the original YOLOv5s model and is more lightweight.

[0122] like Fig.11 The following is a comparison of mAPs of various categories before and after the improvement. As can be seen from the figure, each feature category has been improved compared to the original YOLOv5s. In particular, the mAP of the closed-eye state, which is easy to judge the driver's fatigue, has been greatly improved by 3.0%, which is a significant improvement.

[0123] In order to verify the impact of the C3 module incorporating different attention mechanisms on the performance of the YOLOv5s algorithm after introducing the improved Mobilenet-ECA lightweight backbone, several different types of attention mechanisms are selected for comparative analysis. The experimental results are shown in Table 5.

[0124] Table 5. Comparison of different attention mechanisms

[0125]

[0126] It can be seen from the table that the Triplet attention introduced in the present invention has the best mAP performance for the context enhancement module CTA compared with several common attention mechanisms, and its Params is also the lowest, which meets the requirements of the lightweight algorithm of the present invention.

[0127] In order to evaluate the effect of the improved lightweight face feature detection algorithm of the present invention, we selected 7 representative fatigue driving detection algorithms and compared them with our improved face feature detection algorithm, namely: YOLOv3-Tiny, YOLOv4-Tiny, YOLOv5s, YOLOv5n, YOLOv7-Tiny, YOLOv8s, SSD. All models were trained for 100 epochs on the same dataset to ensure the consistency of the evaluation algorithm. The evaluation indicators include mAP@0.5, Params, FLOPs, Size. The comparison results are shown in Table 6.

[0128] Table 6. Comparative test results

[0129]

[0130]

[0131] As can be seen from Table 6, the number of parameters, FLOPs and model size of our algorithm are much smaller than YOLOv3-Tiny, YOLOv5s, YOLOv7-Tiny, and SSD, which are 3.07M, 5.8G, and 6.30MB respectively, and the mAP is improved compared with the above algorithms, by 7.3%, 1.0%, 0.1%, and 6.3% respectively. Although the FLOPs of our algorithm is slightly higher than those of YOLOv4-Tiny and YOLOv5n, the mAP is greatly improved compared with these two algorithms, by 5.8% and 2.6% respectively. Compared with the latest target detection algorithm YOLOv8s, the mAP of our algorithm has decreased by 0.4%, but its number of parameters, FLOPs, and model size have been greatly reduced compared with the YOLOv8s algorithm, by 72.6%, 80.1%, and 73.4% respectively. In summary, our algorithm not only has significant advantages in terms of lightweight, but also has excellent detection accuracy, ensuring the feasibility requirements of deploying on-board edge devices. From Fig.12 From the figure, we can see the changes in the PR curve of YOLOv5s before and after the improvement. The area occupied by the curve is called AP (average precision). The higher the AP, the better the performance. From the two figures, we can see that the AP value of each category of the improved model is higher than that of the original YOLOv5s.

[0132] In order to verify the feasibility and effectiveness of the improved models for fatigue detection in real driving scenarios, the final recognition rate of the YAWDD driving detection video dataset is used as the evaluation index, and the detection video dataset is divided into two driving states: fatigue and non-fatigue. It contains 72 groups of male driving samples and 81 groups of female driving samples, totaling 153 groups of video sets with an average duration of 22 seconds. The recognition results before and after facial feature tracking are used as the basis for the effectiveness of the tracking algorithm. The experimental results are shown in Table 7.

[0133] Table 7. Blink and yawn recognition results before and after tracking

[0134]

[0135] As can be seen from Table 7, the introduction of facial feature tracking algorithm has greatly improved the accuracy of blink recognition, and the blink recognition rate in non-fatigue and fatigue states has increased by 4.7% and 9.2% respectively. The recognition rate of driver yawning has limited improvement, only 2%, because the yawning process is different from the normal state, and this category is clearly distinguished. The two are combined for the final fatigue detection and recognition. Table 8 shows the fatigue driving recognition results before and after tracking.

[0136] Table 8. Fatigue driving recognition results before and after tracking

[0137]

[0138] It can be seen from Table 8 that the recognition rate of the facial feature tracking algorithm for the fatigue and non-fatigue states of male and female drivers reached 97.4%, which is 5.2% higher than the recognition rate before tracking. The recognition accuracy has been significantly improved, which verifies the effectiveness of the improved model for fatigue driving detection and recognition, thereby compensating for the continuous detection failure caused by the driver's head shaking and environmental changes.

[0139] The visualization process of this detection experiment is as follows Fig. 9 As shown in the figure, the recognition and facial feature classification results of fatigue driving state and normal driving state are shown respectively, and the yellow checkbox is the DeepSort tracking box. Fig. 9 It can be seen that the detection and tracking algorithm can accurately detect and track the state of the driver's eyes and mouth, and make effective identification statistics and driving status judgments for the driver's blinking and yawning behaviors during driving. Even when the female driver wears glasses and the strong light makes the front of the glasses look blurry, the detection and tracking algorithm can recognize the female driver's closed eyes and judge it as fatigue. Through the above experiments, the effectiveness and robustness of the improved fatigue driving detection and tracking method are proved, which meets the expectations of the experimental design.

[0140] The large number of model parameters is not conducive to the deployment of vehicle-mounted edge devices, and the loss of driver facial information leads to poor continuous detection. The present invention proposes a fatigue driving detection method that integrates DeepSort and lightweight YOLOv5s. First, the present invention introduces a lightweight Mobilenetv3-small to reconstruct the YOLOv5s backbone. In order to cope with the significant drop in detection accuracy after the lightweight backbone, the original SE attention mechanism of Mobilenetv3-small is replaced with the ECA attention mechanism, which avoids the problem of dimension reduction of the SE attention mechanism and improves the detection accuracy. Then, the CTA module is used in the neck network to strengthen feature fusion, so that the network can strengthen the focus on contextual information in learning tasks and capture more complex associations and dependencies. In addition, Wise-EIOU is used to optimize the problem of large error in CIoU in prediction box regression calculation, so as to improve the robustness of model detection. Through comparative experiments, the improved face feature detection algorithm of the present invention is improved by 1.0% in mAP, 56.3% in Params, 63.2% in FLOPs, and 54.0% in model size compared with the original YOLOv5s. The results show that the improved facial feature detection model has stronger detection performance than the original YOLOv5s model, and the model is more lightweight. Finally, the facial feature detection and DeepSort facial feature tracking are combined to optimize the driver's continuous classification tracking, and the designed fatigue discrimination mechanism is used to test 153 simulated driving video sets. The results show that the recognition rate of driver fatigue and non-fatigue states reached 97.4%, which is 5.2% higher than before facial feature tracking, proving the effectiveness of facial feature tracking for continuous detection of fatigue driving.

[0141] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0142] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention invented herein. This application is intended to cover any variations, uses or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art that are not invented by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0143] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above are only specific implementation methods of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. A lightweight fatigue driving detection and tracking method, characterized in that: include: Acquire a driver image to be identified; Input the driver image to be identified into the pre-built YOLOv5s network model to obtain the facial feature target frame; Use the Deepsort algorithm to continuously track and optimize the facial feature target frame to obtain label information; Based on the label information, the number of consecutive eye-closing frames of the driver, the number of consecutive yawning frames of the driver, and the calculated PERCLOS score, the driver's driving state, including fatigue driving state and normal driving state, is determined; The pre-built YOLOv5s network model includes a Backbone module, a Neck module, and a Head module connected in sequence, wherein the Backbone module includes a Conv2d layer, a first MobileNet_ECA layer, a first MobileNet_Block layer, a second MobileNet_Block layer, a second MobileNet_ECA layer, a third MobileNet_ECA layer, a fourth MobileNet_ECA layer, a fifth MobileNet_ECA layer, a sixth MobileNet_ECA layer, a seventh MobileNet_ECA layer, an eighth MobileNet_ECA layer, a ninth MobileNet_ECA layer, and an SPPF layer connected in sequence; The first MobileNet_ECA layer, the second MobileNet_ECA layer, the third MobileNet_ECA layer, the fourth MobileNet_ECA layer, the fifth MobileNet_ECA layer, the sixth MobileNet_ECA layer, the seventh MobileNet_ECA layer, the eighth MobileNet_ECA layer, and the ninth MobileNet_ECA layer introduce the ECANet channel attention mechanism, including a global average pooling layer, a one-dimensional convolution layer, and a normalization layer connected in sequence; The Neck module includes the second CBS K1-S1 layer, the second Upsample layer, the second Concat layer, the first CTA layer, the first CBSK1-S1 layer, the first Upsample layer, the first Concat layer, the second CTA layer, the first CBS k3-s2 layer, the third Concat layer, the third CTA layer, the second CBS k3-s2 layer, the fourth Concat layer, and the fourth CTA layer. The second CBS K1-S1 layer, the second Upsample layer, the second Concat layer, the first CTA layer, the first CBS K1-S1 layer, the first Upsample layer, and the first Concat layer are connected in sequence. The second CTA layer, the first CBS k3-s2 layer, the third Concat layer, the third CTA layer, the second CBS k3-s2 layer, the fourth Concat layer, and the fourth CTA layer are connected in sequence. The output end of the second MobileNet_Block layer is connected to the input end of the first Concat layer, the output end of the sixth MobileNet_ECA layer is connected to the second Concat layer, the output end of the SPPF layer is connected to the second CBS K1-S1 layer, the output end of the first Concat layer is connected to the input end of the second CTA layer, the output end of the first CBS K1-S1 layer is connected to the input end of the third Concat layer, the output end of the second CBS K1-S1 layer is connected to the input end of the fourth Concat layer, and the output ends of the second CTA layer, the output ends of the third CTA layer and the output ends of the fourth CTA layer are connected to the Head module; The SPPF layer includes a first CBS layer, a first MaxPool2d layer, a second MaxPool2d layer, a third MaxPool2d layer, a first Contact layer, and a second CBS layer connected in sequence, wherein an output end of the first CBS layer, an output end of the first MaxPool2d layer, and an output end of the second MaxPool2d layer are connected to an input end of the first Contact layer; The first CTA layer, the second CTA layer, the third CTA layer and the fourth CTA layer each include the seventh CBS layer, the fifth CBS layer, the Bottleneck layer, the TripletAtt layer, the second Contact layer and the sixth CBS layer. The fifth CBS layer, the Bottleneck layer, the TripletAtt layer, the second Contact layer and the sixth CBS layer are connected in sequence, the output end of the fifth CBS layer is connected to the input end of the seventh CBS layer, and the output end of the seventh CBS layer is connected to the input end of the second Contact layer; The Bottleneck layer includes the third CBS layer, the fourth CBS layer and the Add layer connected in sequence, and the first CBS layer, the second CBS layer, the third CBS layer, the fourth CBS layer, the fifth CBS layer, the sixth CBS layer and the seventh CBS layer include the Conv layer, the BN layer and the SiLU layer connected in sequence; Pre-built YOLOv5s network model, including: Construct a YOLOv5s network model including a Backbone module, a Neck module, and a Head module connected in sequence; The YOLOv5s network model is trained using the public datasets YawDD, CEW and Drozy. The public datasets YawDD, CEW and Drozy include pictures of male drivers with eyes open, eyes closed, yawning, non-yawning, normal driving, fatigued driving, female drivers with eyes open, eyes closed, yawning, non-yawning, normal driving and fatigued driving.

2. A lightweight fatigue driving detection and tracking method according to claim 1, characterized in that: The pre-built YOLOv5s network model uses Wise-EIoU Loss to calculate the rectangular box loss value.

3. A lightweight fatigue driving detection and tracking system based on the method of claim 1, characterized in that: include: A driver image acquisition module, used to acquire an image of a driver to be identified; A facial feature target frame acquisition module is used to input the driver image to be identified into a pre-built YOLOv5s network model to obtain a facial feature target frame; The label information acquisition module is used to use the Deepsort algorithm to continuously track and optimize the facial feature target frame to obtain label information; The driver driving state determination module is used to determine the driver's driving state including fatigue driving state and normal driving state based on label information, the number of consecutive eye closing frames of the driver, the number of consecutive yawning frames of the driver and the calculated PERCLOS score.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Citation Information

Patent Citations

  • Fatigue driving detection method and system based on facial feature fusion

    CN110532887A

  • Coal foreign matter detection system and method based on improved YOLOv5 algorithm

    CN115410039A

  • MPE-YOLOv5-based gesture recognition method

    CN116052266A

  • Method for identifying bird species related to bird-related fault of transformer substation

    CN116229380A

  • Motorcycle helmet detection and tracking method based on improved YOLOv5DeepSORT algorithm

    CN116645322A