Lightweight fatigue driving detection method based on improved yov8

Abstract

The invention provides a lightweight fatigue driving detection method based on improved yolov8. The lightweight fatigue driving detection method comprises the step of performing improvement on the basis of a yolov8 neural network model. In a backbone layer, MobileNetv3-Small is used as a facial feature extraction backbone lightweight network model, and an SE attention mechanism module is introduced, so that the model is lightweight; a CCFM lightweight cross-scale feature fusion module is introduced into a check layer to integrate features of different scales through fusion operation, so that the detection capability of small-scale objects is enhanced; then, real-time id tracking is carried out on the facial features in combination with a bottom-port tracking algorithm, and the problems of target shielding, deformation and the like are effectively solved; facial features and Dlib key points are combined, and fatigue is comprehensively judged according to data features such as yawn number, nodding total number, longest continuous eye closing time and PERCLOS in unit time; according to the method, the configuration requirement of the vehicle-mounted terminal is met, meanwhile, the face recognition precision is effectively improved, and the robustness is high.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue driving state detection, and in particular to a lightweight fatigue driving detection method based on improved YOLO V8. Background Art

[0002] According to official statistics from the national database, by the end of June 2024, the number of cars in use will reach 345 million, accounting for 76.37% of the total number of motor vehicles. While private cars and public transportation have made people's travel more convenient, the incidence of traffic accidents has also gradually increased. According to statistics from the Ministry of Transport, about 1.19 million people die in traffic accidents in my country every year. 35%-45% of them are caused by driver fatigue. If fatigue warnings can be issued to drivers before they show behavioral awareness such as drowsiness and inattention, the probability of accidents will be greatly reduced. Therefore, it is particularly important to vigorously develop driver fatigue detection technology and continuously improve detection accuracy.

[0003] At present, the technical directions of fatigue driving detection are mainly divided into three categories: based on the physiological characteristics of the driver, based on the driving characteristics of the vehicle, and based on the facial characteristics of the driver. The detection technology based on physiological characteristics mainly obtains the driver's EEG, EMG, ECG and other signals to judge the degree of fatigue. Its accuracy is relatively high, but to obtain these data, it is necessary to wear professional equipment during driving, which will cause inconvenience to the driver and affect driving, so this technology is not very applicable at present; the detection technology based on vehicle driving characteristics is to judge the degree of driver fatigue driving based on the angular velocity of the vehicle steering wheel, driving speed, engine speed and other data, but the driver's personal driving habits and differences in car attributes will affect the data characteristics, which will affect the detection accuracy and cause misjudgment; the third type of detection technology based on the driver's facial characteristics mainly uses the degree of opening and closing of the driver's eyes and mouth, and the tilting arc of the head as the judgment factors of whether the driver is sleepy. This detection method has the advantages of real-time, non-contact and easy to implement, and will not cause discomfort to the driver. Therefore, the detection technology based on facial features has become the mainstream technology for determining fatigue driving detection in recent years.

[0004] At present, there has been a certain amount of research accumulation in the field of fatigue driving detection technology based on facial features at home and abroad. Reference 1 uses an improved SSD (Single shot multibox detector) network to obtain the positioning information of the eye and mouth areas, and combines the 68 key points of the face and the aspect ratio of the eyes and mouth to jointly judge the driving fatigue state. The final detection accuracy is about 97.2%; Reference 2 uses the MediaPipe detection model to locate the key points of the face, realize the dynamic extraction of fatigue features, and obtain four evaluation indicators: the number of blinks per unit time (PERCLOS), the duration of yawning, the presence of drowsiness, and the approximate entropy of face swing, for real-time evaluation. Although both of the above methods have achieved high detection accuracy, they have not taken into account the deployment requirements of vehicle-mounted terminals. The model has a large number of parameters and calculations, which is not conducive to terminal deployment. In order to reduce the hardware configuration requirements, Reference 3 uses deep separable convolution and visual attention mechanism to build the EfficientDet facial image feature extraction network; uses a bidirectional feature pyramid network and k-means prior box clustering method to build the EfficientDet driver state detection network, and the final average precision and recall rate reach 97.92% and 96.75% respectively. However, it does not consider that the loss of target information caused by short-term occlusion, instantaneous steering and other problems will reduce the accuracy of continuous detection. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a lightweight fatigue driving detection method based on improved yolov8, which can accurately identify the driver's fatigue state in the real-time video stream while meeting the lightweight deployment requirements of the vehicle-mounted terminal, and effectively improves the robustness of the driver's facial feature extraction and the reliability of continuous detection.

[0006] In order to achieve the above object, the present invention provides a lightweight fatigue driving detection method based on improved yolov8, which specifically includes the following steps:

[0007] Step S1: collect, screen and annotate facial features of facial fatigue data to create a data set;

[0008] Step S2: Improve the original yolov8 network and use the data set to train the target detection model to obtain the fatigue driving detection model;

[0009] Step S3: combining the model of step S2 with the Dlib key points, and judging the driver's fatigue state according to features such as eye aspect ratio, mouth aspect ratio, and head swing features;

[0010] Step S4: Connect the obtained fatigue driving detection model to the real-time video stream for real-time detection, determine whether the driver in the video is fatigue driving, and process according to the detection result to obtain the detection result.

[0011] Furthermore, in step S1, facial fatigue data is collected, screened, and facial features are annotated, and the steps are as follows:

[0012] Step S11: constructing a data set including a real driving yawning data set YawDD and a self-made data set; the self-made data set is mainly obtained through network collection, camera shooting, and network public data sets, including people of different skin colors, eyes open and closed, image data in various scenes such as buses and classrooms;

[0013] Step S12: Screen the data and delete low-quality data;

[0014] Step S13: For the filtered image data, labelimg software is used to perform facial annotation on the dataset.

[0015] Furthermore, the step S1 also includes framing and classifying the data set, processing the YawDD video data by capturing a picture every 20 frames, manually annotating the data set using the Labelimg annotation tool, framing the facial features contained in the image with an annotation box, and then outputting the target category and location information, and the annotation format is the Pascal VOC format.

[0016] Furthermore, the specific steps of step S2 are:

[0017] Step S21: Introduce the MobileNetv3-Small structure into the Backbone layer of the yolov8 target detection algorithm, use MobileNetv3 to replace all convolutions of the Backbone layer, and introduce the SE attention mechanism;

[0018] Step S22: Introduce the CCFM module into the Neck part of yolov8 and rebuild the module structure of the Neck part;

[0019] Step S23: In view of the low average accuracy caused by the short-term loss of facial features, the face detection algorithm may have detection failure or misjudgment problems, and the facial dynamics are continuously tracked and optimized in combination with bot-sort tracking detection;

[0020] Step S24: Introduce the data set in step S1 into the improved yolov8 model for model training.

[0021] Furthermore, in the step S2, the single-stage target detection model yolov8 of the yolo series is selected as the target detection model. The single-stage target detection model yolov8 uses the C2f module to replace the C3 module of YOLOv5 on the backbone network, and uses DFL and Anchor-Free forms on the classification regression loss function, uses BCE Loss as the classification Loss, and uses DFL Loss and CIoU Loss as the regression loss, and uses the Task-Aligned Assigner positive and negative sample matching method, and the Mosaic operation is closed in the last 10 epochs.

[0022] Furthermore, the steps of model training in step S24 are:

[0023] Step S241: Divide the data set prepared in step S1 into a training set, a validation set, and a test set, and ensure that the labeled image data contains the category and bounding box coordinate information of the target object;

[0024] Step S242: modifying the model configuration file in the improved yolov8 code to specify the parameters of the model;

[0025] Step S243: Use the training set and the model configuration file to train the model, and adjust the parameter settings of the learning rate, optimizer and loss function during the training process so that the training outputs the optimal fatigue driving detection model.

[0026] Furthermore, the specific steps of step S3 are:

[0027] Step S31: According to the Dlib 68 key point positioning, the eye aspect ratio, mouth aspect ratio, and head posture features are obtained, and the driver's fatigue state is reflected according to the blinking frequency per unit time, the longest continuous eye closure time, PERCLOS, the number of yawns, and the total number of nods;

[0028] Step S32: When EAR is less than 0.2, it is judged that the driver blinks once; if the driver blinks more than 20 times within a unit time of 60 seconds, it is judged that the driver is in a fatigue state; when the longest continuous eye closure time reaches 0.8 seconds, it is judged that the driver is in a fatigue state; when PERCLOS is greater than or equal to 0.2, it is judged that the driver is in a fatigue state;

[0029] Step S33: When MAR is greater than 0.6, it is determined that the driver yawns once; if the driver yawns more than 3 times within a unit time of 60 seconds, it is determined that the driver is in a fatigue state;

[0030] Step S34: When the pitch is greater than 30°, it is considered that one head-down behavior has occurred; if the head-down behavior occurs more than three times in a unit of time, it is determined that the driver is in a fatigue state;

[0031] Combined with the changing rules of each feature in steps S32-S34, a fusion judgment is performed; in the process of fusion judgment of each feature, when one of the features fails, the other features can still be used as fatigue indicators.

[0032] Furthermore, in step S3, the facial judgment of fatigue driving characteristics is mainly based on the eye aspect ratio, mouth aspect ratio and head features; the opening and closing state of the eyes is quantified by calculating the ratio of the vertical key point distance to the horizontal key point distance of the eyes; and the Euclidean distances corresponding to the two eyes are calculated respectively using geometric methods to obtain the aspect ratio value of the eyes at any time.

[0033] Furthermore, when the eyes are always open, the EAR will remain in a relatively stable range; when blinking or closing the eyes occurs, the EAR value will gradually decrease and approach 0; among them, the eyes can be considered closed when the degree of closure exceeds 80%.

[0034] Compared with the prior art, the present invention has the following beneficial effects: a single-stage target detection model of the YOLO series is selected as the target detection and recognition model, and the original YOLOv8 model structure is improved to obtain an improved YOLOv8 model structure, and corresponding ablation experiments and comparative experiments are carried out to obtain a fatigue driving detection model with higher precision and lower model volume; in order to improve the robustness of the driver's facial feature extraction and the reliability of continuous detection, and consider the lightweight deployment requirements of the vehicle-mounted terminal; on the basis of the YOLOv8n target detection algorithm, MobileNetv3-Small is used as the feature extraction backbone to achieve lightweight model; SE attention module and CCFM lightweight cross-scale feature fusion module are introduced, and at the same time, combined with the bot-sort tracking algorithm, an efficient tracking algorithm for real-time facial feature detection is realized, and the accuracy of the detection result is effectively improved; combined with Dlib key point positioning, the driver's fatigue degree is judged according to the longest continuous eye closure time, the total number of yawns per unit time, the total number of nods, and PERCLOS data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an overall flow chart of the improved yolov8 lightweight fatigue driving detection method of the present invention;

[0036] Figure 2 It is a schematic diagram of the steps of the present invention;

[0037] Figure 3It is the network structure diagram of yolov8 of the present invention;

[0038] Figure 4 This is the network structure diagram of the improved yolov8 of the present invention;

[0039] Figure 5 This is a schematic diagram of the labelimg labeled sample of the present invention;

[0040] Figure 6 Different labeled sample graphs of the data set of the present invention;

[0041] Figure 7 Comparison of the recall rate accuracy curves of the original YOLOv8 and the improved YOLOv8 of the present invention

[0042] Figure 8 is an identification diagram of a normal driving state of the present invention;

[0043] Fig. 9 This is an effect recognition diagram of the present invention under a camera. Specific implementation plan

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0046] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments can be combined with each other. The present application will be described in detail with reference to the accompanying drawings.

[0047] like Figure 1 -like Fig. 9 The present invention proposes a lightweight fatigue driving detection method based on improved yolov8, which can accurately identify the driver's fatigue state in the real-time video stream while meeting the lightweight deployment requirements of the vehicle-mounted terminal, and effectively improves the robustness of the driver's facial feature extraction and the reliability of continuous detection.

[0048] Specifically, Figure 1-Figure 4 As shown, in order to achieve the above purpose, the present invention provides a lightweight fatigue driving detection method based on improved yolov8, comprising the following steps:

[0049] Step S1: collect, screen and annotate facial features of facial fatigue data to create a data set;

[0050] Step S2: Improve the original yolov8 network and use the data set to train the target detection model to obtain the fatigue driving detection model;

[0051] Step S3: combining the improved target detection model with the Dlib key points to determine the driver's fatigue state based on features such as eye aspect ratio, mouth aspect ratio, and head swing features;

[0052] Step S4: Connect the obtained fatigue driving detection model to the real-time video stream for real-time detection, determine whether the driver in the video is fatigue driving, and process according to the detection result to obtain the detection result.

[0053] By adopting the above technical solution, data is collected to create a data set; then a suitable target recognition model is selected and improved as needed, and the data set is placed in the improved network model and trained to obtain a detection model; the model integrates the bot-sort face tracking algorithm to improve the continuity of target detection accuracy; combined with Dlib facial key point positioning, fatigue judgment analysis is performed according to different judgment results, which improves the accuracy of facial feature recognition of the target detection model and enhances the robustness of the model.

[0054] Preferably, if Figure 5 -like Figure 6 As shown, in step S1, the data set mainly consists of a real driving yawning data set YawDD and a self-made data set. The YawDD data set mainly provides a real in-car environment during the day, which is suitable for detecting whether the driver is driving fatigued by the driver's yawning frequency. It lacks a night shooting data set and only contains one scene of yawning; the self-made data set is mainly collected through the Internet and captured by a camera, including fatigue data of people of different skin colors, eyes open and closed, buses, classrooms and other scenes, which makes up for the problem of single or missing categories of some face data in the YawDD data set and improves the robustness of model training. The YawDD data set is mainly an important data set for yawn detection related research. It consists of two groups of driver videos with different facial features. These videos record a variety of scenes, including people driving normally, yawning, talking, laughing and other behavioral states. These videos are shot in a real environment, and their lighting conditions will change, and there is no audio information in the video. These different behavioral performances, video formats, shooting environments and many other factors together constitute a complete YawDD data set.

[0055] Furthermore, the step S1 also includes framing and classifying the data set. The present invention processes the YawDD video data by intercepting a picture every 20 frames, uses the Labelimg annotation tool to manually annotate the data set, frames the facial features contained in the image with an annotation frame, and then outputs the target category and position information, and the annotation format is the PascalVOC format. The data set contains 1873 faces, 1244 open mouths, 609 closed mouths, 2889 open eyes, 449 closed eyes, and 205 glasses, with a total of 7269 facial feature data annotated in 6 categories. The data sample contains photos of drivers with different skin colors, different ages, wearing different decorations, and driving environments under different lighting conditions in the car.

[0056] Preferably, YOLOv8 is selected as the core algorithm for face detection in step S2. As a target detection model using an anchor-free split Ultralytics head, YOLOv8 has high detection accuracy and real-time efficiency. Based on the previous version, YOLOv8 introduces a C2f structure with more gradient flows to replace the C3 structure in YOLOv5, achieving further lightweighting, making it the best choice for detection and recognition of various objects in a wide range of applications. In view of the problems that the YOLOv8 network may miss detections, misdetections, and weak continuous detection when the face is lost due to occlusion or teleportation in real-time detection scenarios, and the large number of parameters, improvements have been made considering the lightweight deployment requirements of vehicle-mounted terminals.

[0057] Furthermore, the improvement of the yolov8 neural network in step S2 mainly consists of three parts. The first part is to introduce the lightweight MobileNetv3-Small structure into the Backbone layer, and use MobileNetv3 to replace all modules of the Backbone layer. It contains the SE (Squeeze-and-Excitation Networks) attention mechanism, which greatly reduces the number of model parameters and the amount of calculation while ensuring the efficient extraction of features in the image, and has the accuracy of target detection; the second part introduces the cross-scale feature fusion module CCFM (Cross-Scale Feature Fusion Module), optimizes the neck network (neck) efficient hybrid encoder, redesigns the structure of the encoder, and reconstructs the module structure of the Neck part, integrates features of different scales through fusion operations, and inserts a fusion block composed of convolutional layers in the fusion path. The function of the fusion block is to fuse adjacent features into a new feature. The fusion block contains N RepBlocks. The outputs of the two paths are fused by element addition to enhance the model's adaptability to scale changes and the ability to detect small-scale objects; the third part combines the bot-sort tracking algorithm to continuously track and optimize facial dynamics. It retains the low-score detection frame while ensuring the tracking of the high-score detection frame; improves the accuracy of prediction frame prediction by improving the state vector of the Kalman filter and using the camera motion compensation method; and also proposes a cosine distance fusion method for IoU and ReID to improve the robustness between detection and trajectory.

[0058] In the step S2, the single-stage target detection model yolov8 of the yolo series is selected as the target detection model. The single-stage target detection model yolov8 uses the C2f module to replace the C3 module of YOLOv5 on the backbone network, and uses DFL and Anchor-Free forms on the classification regression loss function, uses BCE Loss as the classification Loss, and uses DFL Loss and CIoU Loss as the regression loss, and uses the Task-Aligned Assigner positive and negative sample matching method, and the Mosaic operation is closed in the last 10 epochs.

[0059] Preferably, the specific steps of step S2 are:

[0060] Step S21: Introduce the MobileNetv3-Small structure into the Backbone layer of the yolov8 target detection algorithm, use MobileNetv3 to replace all convolutions of the Backbone layer, and introduce the SE attention mechanism;

[0061] Step S22: Introduce the CCFM module into the Neck part of yolov8 and rebuild the module structure of the Neck part;

[0062] Step S23: In view of the low average accuracy caused by the short-term loss of facial features, the face detection algorithm may have problems such as detection failure or misjudgment, and the facial dynamics are continuously tracked and optimized in combination with bot-sort tracking detection;

[0063] Step S24: Introduce the data set in step S1 into the improved yolov8 model and perform model training.

[0064] Preferably, if Figure 7 The figure shows the comparison of the recall rate accuracy curve of the original yolov8 model and the improved model. In order to verify the effectiveness of the lightweight backbone, fusion module and face tracking for face detection, an ablation experiment was conducted on the algorithm based on the improved YOLOv8n. The results of facial feature detection before and after the improvement are shown in Table 1. After integrating the MobileNetv3-Small lightweight backbone, the floating point calculation amount, parameter amount and model volume are greatly reduced, and the lightweight effect is obvious, but the average precision mean and recall rate also decreased by 1.7% and 1.4% respectively, which is a large loss. After integrating CCFM and BoT-SORT, the overall performance has been improved and the model volume is lighter.

[0065] Table 1 Ablation experiment results

[0066]

[0067] From the comprehensive chart, we can see that the recall rate of this model has increased by 1.1% compared with YOLOv8n, which is mainly reflected in the three categories of closed_eye, open_mouth and closed_mouth. In particular, the recall rate of the closed_eye category has increased by 9.3% to 89%. Compared with YOLOv8n, the average precision has increased by 0.4%, which is mainly reflected in the two categories of open_eye and open_mouth. While the recall rate and average precision have increased, the volume and computational complexity of the model have also decreased by 32.3% and 18.3% respectively, verifying the feasibility of the model to accurately extract facial features while being lightweight.

[0068] Table 2 Comparison results of category data before and after improvement

[0069]

[0070] Further, in step S3, since the driver's eye state characteristics are one of the keys to fatigue driving judgment. By analyzing the shape and opening degree of the eyes, the driver's fatigue degree can be effectively identified. Among them, the eye aspect ratio is an important indicator, and the opening and closing state of the eyes is quantified by calculating the ratio of the vertical key point distance of the eyes to the horizontal key point distance; similarly, the mouth feature also plays an important role in fatigue judgment. When a person is in a state of fatigue, the frequency of yawning will increase significantly. Due to physical fatigue, the brain and body are hypoxic, and it is necessary to open the mouth and yawn to inhale more oxygen to regulate body functions. The behavior of yawning is often accompanied by fatigue and lack of attention. Therefore, by observing the opening and closing amplitude of the mouth and the number of yawns, a person's fatigue state can be effectively evaluated; affected by the physiological state, when a person is in a state of fatigue, he often nods or tilts his head unconsciously, so head posture estimation is used as an indicator for determining fatigue detection. Therefore, the present invention combines the facial feature model with the Dlib key point, and reflects the driver's fatigue state according to the blinking frequency per unit time, the longest continuous eye closure time, PERCLOS, the number of yawns, and the total number of nods.

[0071] Preferably, the specific steps of step S3 are:

[0072] Step S31: According to the Dlib 68 key point positioning, the eye aspect ratio, mouth aspect ratio, and head posture features are obtained, and the driver's fatigue state is reflected based on the blinking frequency per unit time, the longest continuous eye closure time, PERCLOS, the number of yawns, and the total number of nods.

[0073] Step S32: When EAR is less than 0.2, it is judged that the driver blinked once. If the driver blinks more than 20 times within a unit time of 60 seconds, he is judged to be in a fatigue state; when the longest continuous eye closure time reaches 0.8 seconds, he is judged to be in a fatigue state; when PERCLOS is greater than or equal to 0.2, he is judged to be in a fatigue state;

[0074] Step S33: When MAR is greater than 0.6, it is determined that the driver yawns once. If the driver yawns more than 3 times within a unit time of 60 seconds, it is determined that the driver is in a fatigue state;

[0075] Step S34: When the pitch is greater than 30°, it is considered that the head-down behavior has occurred. If the head-down behavior occurs more than three times in a unit of time, it is determined that the driver is in a fatigue state.

[0076] Step S35: Combine the changing rules of each feature in steps S32-S34 to make a fusion judgment. In the multi-feature fusion judgment process, when one of the features fails, the other features can still be used as indicators for judging fatigue.

[0077] Preferably, in step S3, the facial judgment on fatigue driving characteristics is mainly based on the eye aspect ratio, mouth aspect ratio and head features. Eye Aspect Ratio (EAR) is the ratio of the length to the width of the eye, and the opening and closing state of the eyes is quantified by calculating the ratio of the vertical key point distance to the horizontal key point distance of the eye. When the driver feels tired, the degree of closure of the eyelids will increase, resulting in a decrease in the EAR value. The Euclidean distance corresponding to the two eyes is calculated using a geometric method to obtain the aspect ratio value of the eyes at any time. The calculation formula of EAR is as follows. When the eyes are always in an open state, EAR will remain in a relatively stable range; when blinking or closing the eyes occurs, the EAR value will gradually decrease and approach 0; among them, the degree of closure of the eyes exceeding 80% can be regarded as a closed state. The formula is as follows.

[0078]

[0079]

[0080] Currently, the most reliable method for judging eye fatigue is PERCLOS (Percentage of Eye Closure). It was proposed by the Carnegie Mellon Institute and is defined as the time that the eyes are closed for a certain proportion (70% or 80%) per unit time (usually 1 minute or 30 seconds). The calculation formula of PERCLOS is as follows.

[0081]

[0082] Mouth Aspect Ratio (MAR) refers to the ratio of the length to the width of the mouth, which is used to determine whether the mouth is open or closed. Similar to EAR, when the mouth is closed, the MAR value will approach 0. The two corresponding Euclidean distances are calculated using geometric methods to obtain the aspect ratio of the mouth at any time. The calculation formula of MAR is as follows.

[0083]

[0084] When people are in a state of fatigue, they often nod or tilt their heads unconsciously, so head posture estimation is used as an indicator for fatigue detection. In order to accurately obtain the three-dimensional spatial angle information of the driver's head, we draw the 12 axes of the cube according to the driver's facial features, convert the rotation matrix into a rotation vector, convert the 2D image key points into a 3D face model, and calculate the posture changes (pitch, yaw, roll) in three-dimensional space. Because this process is very strict, in order to determine the three-dimensional spatial angle information of the head, it is necessary to understand the inherent parameters of the camera, thereby reducing the perception error of perspective distortion. First, use Dlib to locate and extract 2D face key points. According to the position of the key points, the 3D face model is matched to obtain the corresponding 3D face model; then the relationship between the 3D face key points and the 2D face key points is calculated according to the PNP algorithm in OpenCV; finally, the rotation matrix is ​​used to calculate the Euler angle.

[0085] The present invention selects a single-stage target detection model of the YOLO series as a target detection and recognition model, improves the original YOLOv8 model structure, obtains an improved YOLOv8 model structure, and performs corresponding ablation experiments and comparative experiments to obtain a fatigue driving detection model with higher precision and lower model volume; in order to improve the robustness of the driver's facial feature extraction and the reliability of continuous detection, and consider the lightweight deployment requirements of the vehicle-mounted terminal; on the basis of the YOLOv8n target detection algorithm, MobileNetv3-Small is used as the feature extraction backbone to achieve the lightweight of the model; the SE attention module and the CCFM lightweight cross-scale feature fusion module are introduced, and the bot-sort tracking algorithm is combined to realize an efficient tracking algorithm for real-time facial feature detection, which effectively improves the accuracy of the detection result; combined with Dlib key point positioning, the driver's fatigue degree is judged according to the longest continuous eye closure time, the total number of yawns per unit time, the total number of nods, and PERCLOS data.

[0086] It can be seen from the experimental results that for driver fatigue driving detection technology, taking into account the requirements of lightweight models, continuous detection and robustness, the present invention has achieved optimization in both accuracy and model volume compared with the original model after a series of improvements and optimizations. While the number of parameters is reduced by 34.5%, the average precision mean and recall rate of the improved model are increased by 1.1% and 0.4% respectively. It can realize real-time detection of small cameras in the car, does not distinguish between races, and has a high detection rate for drivers of all ages and skin colors, effectively reducing the incidence of traffic accidents caused by fatigue driving.

[0087] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A lightweight fatigue driving detection method based on improved yolov8, characterized in that: The specific steps include: Step S1: collect, screen and annotate facial features of facial fatigue data to create a data set; Step S2: Improve the original yolov8 network and use the data set to train the target detection model to obtain the fatigue driving detection model; Step S3: combining the model of step S2 with the Dlib key points, and judging the driver's fatigue state according to features such as eye aspect ratio, mouth aspect ratio, and head swing features; Step S4: Connect the obtained fatigue driving detection model to the real-time video stream for real-time detection, determine whether the driver in the video is fatigue driving, make a judgment based on the fatigue state, and obtain the detection result.

2. According to claim 1, a method for lightweight fatigue driving detection based on improved yolov8 is characterized in that: In step S1, facial fatigue data is collected, screened, and facial features are annotated, and the steps are as follows: Step S11: constructing a data set including a real driving yawning data set YawDD and a self-made data set; the self-made data set is mainly obtained through network collection, camera shooting, and network public data sets, including people of different skin colors, eyes open and closed, image data in various scenes such as buses and classrooms; Step S12: Screen the data and delete low-quality data; Step S13: For the filtered image data, labelimg software is used to perform facial annotation on the dataset.

3. A method for lightweight fatigue driving detection based on improved yolov8 according to claim 2, characterized in that: The step S1 also includes framing and classifying the data set. The YawDD video data is processed by capturing a picture every 20 frames. The data set is manually labeled using the Labelimg labeling tool. The facial features contained in the image are framed with a labeling frame, and then the target category and location information are output. The labeling format is the Pascal VOC format.

4. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 1 is characterized in that: The specific steps of step S2 are: Step S21: Introduce the MobileNetv3-Small structure into the Backbone layer of the yolov8 target detection algorithm, use MobileNetv3 to replace all convolutions of the Backbone layer, and introduce the SE attention mechanism; Step S22: Introduce the CCFM module into the Neck part of yolov8 and rebuild the module structure of the Neck part; Step S23: In view of the low average accuracy caused by the short-term loss of facial features, the face detection algorithm may have detection failure or misjudgment problems, and the facial dynamics are continuously tracked and optimized in combination with bot-sort tracking detection; Step S24: Introduce the data set in step S1 into the improved yolov8 model for model training.

5. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 1 is characterized in that: In the step S2, the single-stage target detection model yolov8 of the yolo series is selected as the target detection model. The single-stage target detection model yolov8 uses the C2f module to replace the C3 module of YOLOv5 on the backbone network, and uses DFL and Anchor-Free forms on the classification regression loss function, uses BCE Loss as the classification Loss, and uses DFL Loss and CIoU Loss as the regression loss, and uses the Task-Aligned Assigner positive and negative sample matching method, and the Mosaic operation is closed in the last 10 epochs.

6. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 4 is characterized in that: The steps of model training in step S24 are: Step S241: Divide the data set prepared in step S1 into a training set, a validation set, and a test set, and ensure that the labeled image data contains the category and bounding box coordinate information of the target object; Step S242: modifying the model configuration file in the improved yolov8 code to specify the parameters of the model; Step S243: Use the training set and the model configuration file to train the model, and adjust the parameter settings of the learning rate, optimizer and loss function during the training process so that the training outputs the optimal fatigue driving detection model.

7. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 1 is characterized in that: The specific steps of step S3 are: Step S31: According to the Dlib 68 key point positioning, the eye aspect ratio, mouth aspect ratio, and head posture features are obtained, and the driver's fatigue state is reflected according to the blinking frequency per unit time, the longest continuous eye closure time, PERCLOS, the number of yawns, and the total number of nods; Step S32: When EAR is less than 0.2, it is judged that the driver blinks once; if the driver blinks more than 20 times within a unit time of 60 seconds, it is judged that the driver is in a fatigue state; when the longest continuous eye closure time reaches 0.8 seconds, it is judged that the driver is in a fatigue state; when PERCLOS is greater than or equal to 0.2, it is judged that the driver is in a fatigue state; Step S33: When MAR is greater than 0.6, it is determined that the driver yawns once; if the driver yawns more than 3 times within a unit time of 60 seconds, it is determined that the driver is in a fatigue state; Step S34: When the pitch is greater than 30°, it is considered that one head-down behavior has occurred; if the head-down behavior occurs more than three times in a unit of time, it is determined that the driver is in a fatigue state; Combined with the changing rules of each feature in steps S32-S34, a fusion judgment is performed; in the process of fusion judgment of each feature, when one of the features fails, the other features can still be used as fatigue indicators.

8. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 1 is characterized in that: The facial judgment on fatigue driving characteristics in step S3 is mainly based on the eye aspect ratio, mouth aspect ratio and head features; the opening and closing state of the eyes is quantified by calculating the ratio of the vertical key point distance to the horizontal key point distance of the eyes; the Euclidean distances corresponding to the two eyes and the mouth are calculated respectively using a geometric method to obtain the aspect ratio of the eyes and the mouth at any time.

9. A method for lightweight fatigue driving detection based on improved yolov8 according to claim 8, characterized in that: When the eyes are always open, EAR will remain within a relatively stable range; when blinking or closing the eyes, the EAR value will gradually decrease and approach 0; among them, the eyes can be considered closed when the degree of closure exceeds 80%; the same applies to MAR.

Images