Lightweight fatigue driving detection method based on improved yolov8

Abstract

The present invention provides a lightweight fatigue driving detection method based on an improved yolov8, comprising: improving the yolov8 neural network model. In the backbone layer, MobileNetv3‑Small is used as the backbone lightweight network model for facial feature extraction, and the SE attention mechanism module is introduced to make the model lightweight; in the neck layer, a CCFM lightweight cross-scale feature fusion module is introduced to integrate features of different scales through a fusion operation, thereby enhancing the detection capability of small-scale objects; combined with the bot‑sort tracking algorithm, real-time ID tracking of facial features is performed to effectively handle problems such as target occlusion and deformation; facial features are combined with Dlib key points, and fatigue is comprehensively determined based on data features such as the number of yawns per unit time, the total number of nods, the longest continuous eye closure time, and PERCLOS; while meeting the configuration requirements of the vehicle terminal, the present invention effectively improves the accuracy of facial recognition and has strong robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue driving state detection, and in particular to a lightweight fatigue driving detection method based on improved YOLOv8. Background Art

[0002] According to official national database statistics, by the end of June 2024, the number of cars on the road will reach 345 million, accounting for 76.37% of the total motor vehicle population. While private cars and public transportation have made travel more convenient, the incidence of traffic accidents has also been steadily increasing. Driver fatigue is responsible for 35%-45% of these accidents. If fatigue warnings can be issued before drivers become aware of drowsiness or inattention, the probability of accidents will be significantly reduced. Therefore, it is crucial to vigorously develop driver fatigue detection technology and continuously improve detection accuracy.

[0003] Currently, fatigue detection technologies fall into three main categories: those based on driver physiological characteristics, those based on vehicle driving characteristics, and those based on driver facial features. Physiologically based detection technologies primarily assess driver fatigue by acquiring signals from the driver's electroencephalogram (EEG), electromyography (EMG), and electrocardiogram (ECG). While these technologies offer high accuracy, acquiring these signals requires specialized equipment, which can be inconvenient and disruptive to the driver's driving. Therefore, this technology is currently limited in practical application. Vehicle driving characteristics based detection technologies assess driver fatigue based on data such as steering wheel angular velocity, driving speed, and engine speed. However, individual driving habits and vehicle characteristics can affect these data characteristics, which can affect detection accuracy and lead to misjudgments. The third category, facial feature-based detection technologies, primarily assess driver drowsiness by determining the degree of eye and mouth opening and the tilt of the head. This detection method offers advantages such as real-time, contactless, and easy implementation, without causing discomfort to the driver. Consequently, facial feature-based detection has become the mainstream technology for fatigue detection in recent years.

[0004] Currently, research on facial feature-based fatigue driving detection technology has been extensively conducted both domestically and internationally. Reference 1 utilizes an improved SSD (Single Shot Multibox Detector) network to obtain localization information for the eye and mouth regions, combining 68 key facial points and the eye and mouth aspect ratios to discriminate driver fatigue status, achieving a final detection accuracy of approximately 97.2%. Reference 2 utilizes the MediaPipe detection model to locate facial key points and dynamically extract fatigue features. Four evaluation metrics are derived: the number of blinks per unit time (PERCLOS), yawn duration, drowsiness, and approximate entropy of facial wiggling, for real-time evaluation. While both methods achieve high detection accuracy, they fail to consider the deployment requirements of in-vehicle terminals. The model's large number of parameters and computational complexity make it difficult to deploy. To reduce hardware requirements, Reference 3 uses depthwise separable convolution and visual attention mechanisms to construct the EfficientDet facial image feature extraction network. It also uses a bidirectional feature pyramid network and k-means prior box clustering to construct the EfficientDet driver state detection network, achieving an average precision and recall of 97.92% and 96.75%, respectively. However, this approach fails to account for the loss of target information due to short-term occlusions and transient steering, which can reduce the accuracy of continuous detection. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention proposes a lightweight fatigue driving detection method based on the improved yolov8, which can accurately identify the driver's fatigue state in the real-time video stream while meeting the lightweight deployment requirements of the vehicle-mounted terminal, and effectively improve the robustness of the driver's facial feature extraction and the reliability of continuous detection.

[0006] In order to achieve the above objectives, the present invention provides a lightweight fatigue driving detection method based on an improved yolov8, which specifically includes the following steps:

[0007] Step S1: Collect, filter and annotate facial features of facial fatigue data to create a dataset;

[0008] Step S2: Improve the network of the original yolov8 and use the dataset to train the target detection model to obtain the fatigue driving detection model;

[0009] Step S3: Combine the model from step S2 with the Dlib key points to determine the driver's fatigue status based on features such as eye aspect ratio, mouth aspect ratio, and head swing characteristics;

[0010] Step S4: The obtained fatigue driving detection model is connected to the real-time video stream for real-time detection, and it is determined whether the driver in the video is fatigue driving. The detection result is processed according to the detection result to obtain the detection result.

[0011] Furthermore, in step S1, facial fatigue data is collected, screened, and facial features are annotated, as follows:

[0012] Step S11: Constructing a dataset including a real driving yawning dataset YawDD and a self-made dataset; the self-made dataset is mainly obtained through online collection, camera shooting, and online public datasets, and contains image data of people of different skin colors, with eyes open and closed, in various scenes such as buses and classrooms;

[0013] Step S12: Screen the data and delete low-quality data;

[0014] Step S13: For the filtered image data, use labelimg software to perform facial annotation on the dataset.

[0015] Furthermore, step S1 also includes framing and classifying the data set, processing the YawDD video data by capturing a picture every 20 frames, manually annotating the data set using the Labelimg annotation tool, framing the facial features contained in the image with annotation boxes, and then outputting the target category and location information in the Pascal VOC format.

[0016] Furthermore, the specific steps of step S2 are:

[0017] Step S21: Introduce the MobileNetv3-Small structure into the Backbone layer of the yolov8 target detection algorithm, use MobileNetv3 to replace all convolutions in the Backbone layer, and introduce the SE attention mechanism;

[0018] Step S22: Introduce the CCFM module into the Neck part of yolov8 and rebuild the module structure of the Neck part;

[0019] Step S23: To address the problem of low average accuracy due to short-term loss of facial features, which may cause detection failure or misjudgment in the face detection algorithm, continuous tracking optimization of facial dynamics is performed in combination with bot-sort tracking detection;

[0020] Step S24: Introduce the data set in step S1 into the improved yolov8 model for model training.

[0021] Furthermore, in step S2, the single-stage target detection model yolov8 of the yolo series is selected as the target detection model. The single-stage target detection model yolov8 uses the C2f module to replace the C3 module of YOLOv5 on the backbone network, and uses DFL and Anchor-Free forms on the classification and regression loss functions, uses BCE Loss as the classification Loss, and uses DFL Loss and CIoU Loss as the regression loss, and uses the Task-Aligned Assigner positive and negative sample matching method, and at the same time, the Mosaic operation is closed in the last 10 epochs.

[0022] Furthermore, the steps of model training in step S24 are:

[0023] Step S241: Split the dataset generated in step S1 into a training set, a validation set, and a test set, and ensure that the labeled image data contains the target object category and bounding box coordinate information;

[0024] Step S242: modifying the model configuration file in the improved yolov8 code to specify the parameters of the model;

[0025] Step S243: Use the training set and model configuration file to train the model, and adjust the parameter settings of the learning rate, optimizer, and loss function during the training process so that the training outputs the optimal fatigue driving detection model.

[0026] Furthermore, the specific steps of step S3 are:

[0027] Step S31: Based on the Dlib 68 keypoint positioning, the eye aspect ratio, mouth aspect ratio, and head posture features are obtained. The blink frequency per unit time, the longest continuous eye closure time, PERCLOS, the number of yawns, and the total number of nods are used to reflect the driver's fatigue status.

[0028] Step S32: When EAR is less than 0.2, it is determined that the driver blinked once; if the driver blinks more than 20 times within a unit time of 60 seconds, it is determined that the driver is in a fatigue state; when the longest continuous eye closure time reaches 0.8 seconds, it is determined that the driver is in a fatigue state; when PERCLOS is greater than or equal to 0.2, it is determined that the driver is in a fatigue state;

[0029] Step S33: When MAR is greater than 0.6, it is determined that the driver yawns once; if the driver yawns more than three times within a unit time of 60 seconds, it is determined that the driver is in a fatigue state;

[0030] Step S34: When the pitch is greater than 30°, it is considered that the head-down behavior has occurred once; if the head-down behavior occurs more than three times in a unit of time, it is determined that the driver is in a fatigue state;

[0031] Combine the changing rules of each feature in steps S32-S34 to perform fusion judgment; in the process of fusion judgment of each feature, when one of the features fails, the other features can still be used as fatigue indicators.

[0032] Furthermore, in step S3, the facial judgment of fatigue driving characteristics is mainly based on the eye aspect ratio, mouth aspect ratio and head features; the opening and closing state of the eyes is quantified by calculating the ratio of the vertical key point distance to the horizontal key point distance of the eyes; and the Euclidean distance corresponding to the two eyes is calculated using a geometric method to obtain the aspect ratio value of the eyes at any time.

[0033] Furthermore, when the eyes are always open, the EAR will remain within a relatively stable range; when blinking or closing the eyes occurs, the EAR value will gradually decrease and approach 0; among them, the eyes can be considered closed if the degree of closure exceeds 80%.

[0034] Compared with the prior art, the present invention has the following beneficial effects: a single-stage target detection model of the YOLO series is selected as the target detection and recognition model, and the original YOLOv8 model structure is improved to obtain an improved YOLOv8 model structure, and corresponding ablation experiments and comparative experiments are carried out to obtain a fatigue driving detection model with higher precision and lower model volume; in order to improve the robustness of driver facial feature extraction and the reliability of continuous detection, and considering the lightweight deployment requirements of vehicle-mounted terminals; on the basis of the YOLOv8n target detection algorithm, MobileNetv3-Small is adopted as the feature extraction backbone to achieve lightweight model; SE attention module and CCFM lightweight cross-scale feature fusion module are introduced, and at the same time, combined with the bot-sort tracking algorithm, an efficient tracking algorithm for real-time facial feature detection is realized, which effectively improves the accuracy of detection results; combined with Dlib key point positioning, the driver's fatigue level is judged according to data such as the longest continuous eye closure time, the total number of yawns per unit time, the total number of nods, and PERCLOS. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an overall flow chart of the improved yolov8 lightweight fatigue driving detection method of the present invention;

[0036] Figure 2 It is a schematic diagram of the steps of the present invention;

[0037] Figure 3This is the network structure diagram of yolov8 of the present invention;

[0038] Figure 4 This is the network structure diagram of the improved yolov8 of the present invention;

[0039] Figure 5 This is a schematic diagram of the labelimg labeled sample of the present invention;

[0040] Figure 6 Different labeled sample images of the data set of the present invention;

[0041] Figure 7 Comparison of the recall accuracy curves of the original YOLOv8 and the improved YOLOv8 of the present invention

[0042] Figure 8 is an identification diagram of a normal driving state of the present invention;

[0043] Figure 9 This is an effect recognition diagram of the present invention under the camera. Specific implementation plan

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0046] It should be noted that, in the absence of conflict, the embodiments and features and technical solutions in the present invention can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings.

[0047] like Figure 1 -like Figure 9 The present invention proposes a lightweight fatigue driving detection method based on the improved yolov8, which can meet the lightweight deployment requirements of the vehicle-mounted terminal, accurately identify the driver's fatigue state in the real-time video stream, and effectively improve the robustness of the driver's facial feature extraction and the reliability of continuous detection.

[0048] Specifically, such as Figure 1-Figure 4 As shown, in order to achieve the above purpose, the present invention provides a lightweight fatigue driving detection method based on improved yolov8, comprising the following steps:

[0049] Step S1: Collect, filter and annotate facial features of facial fatigue data to create a dataset;

[0050] Step S2: Improve the network of the original yolov8 and use the dataset to train the target detection model to obtain the fatigue driving detection model;

[0051] Step S3: Combining the improved object detection model with Dlib key points, the driver’s fatigue status is determined based on features such as eye aspect ratio, mouth aspect ratio, and head swing characteristics.

[0052] Step S4: The obtained fatigue driving detection model is connected to the real-time video stream for real-time detection, and it is determined whether the driver in the video is fatigue driving. The detection result is processed according to the detection result to obtain the detection result.

[0053] Using the above technical solution, data is collected to create a dataset. Then, a suitable target recognition model is selected and improved as needed. The dataset is placed into the improved network model and trained to obtain a detection model. The model integrates the bot-sort face tracking algorithm to improve the continuity of target detection accuracy. Combined with Dlib facial key point positioning, fatigue judgment analysis is performed based on different judgment results, which improves the accuracy of the target detection model in facial feature recognition and enhances the robustness of the model.

[0054] Preferably, if Figure 5 -like Figure 6 As shown in step S1, the dataset primarily consists of the YawDD dataset, a real-world driving yawning dataset, and a self-produced dataset. The YawDD dataset primarily provides a realistic daytime in-car environment and is suitable for detecting driver fatigue by yawning frequency. However, it lacks a nighttime dataset and only includes yawning scenarios. The self-produced dataset, collected online and captured with a camera, includes fatigue data from people of different skin colors, with eyes open or closed, and in various scenarios, such as buses and classrooms. This addresses the issue of some facial data being single or missing in the YawDD dataset and improves the robustness of model training. The YawDD dataset is a key dataset for yawn detection research. It consists of two sets of driver videos with different facial features. These videos capture a variety of scenes, including normal driving, yawning, talking, laughing, and other behaviors. These videos are captured in real-world environments with varying lighting conditions and lack audio information. These diverse behavioral expressions, video formats, and shooting environments all contribute to the complete YawDD dataset.

[0055] Furthermore, the step S1 also includes framing and classifying the data set. The present invention processes the YawDD video data by intercepting a picture every 20 frames, uses the Labelimg annotation tool to manually annotate the data set, frames the facial features contained in the image with annotation frames, and then outputs the target category and position information. The annotation format is PascalVOC format. The data set contains 1873 faces (face), 1244 open mouths (open_mouth), 609 closed mouths (closed_mouth), 2889 open eyes (open_eye), 449 closed eyes (closed_eye), and 205 glasses (sunglasses), for a total of 7269 facial feature data labeled in 6 categories. The data samples include photos of drivers with different skin colors, different ages, wearing different decorations, and driving environments under different lighting conditions in the car.

[0056] Preferably, YOLOv8 is selected as the core algorithm for face detection in step S2. As a target detection model that uses an anchor-free split Ultralytics head, YOLOv8 has high detection accuracy and real-time efficiency. Based on the previous version, YOLOv8 introduced a C2f structure with more gradient flows to replace the C3 structure in YOLOv5, achieving further lightweighting, making it the best choice for detection and recognition of various objects in a wide range of applications. In view of the fact that the YOLOv8 network may miss detections, make false detections, and have weak continuous detection when the face is lost due to occlusion or teleportation in real-time detection scenarios, and the number of parameters is large, improvements have been made considering the lightweight deployment requirements of vehicle-mounted terminals.

[0057] Furthermore, the improvements to the YoLoV8 neural network in step S2 consist of three main parts. The first part introduces a lightweight MobileNetv3-Small structure into the Backbone layer, replacing all modules in the Backbone layer with MobileNetv3. This includes the SE (Squeeze-and-Excitation Networks) attention mechanism, significantly reducing the number of model parameters and computational complexity while ensuring efficient feature extraction from images and achieving accurate object detection. The second part introduces the Cross-Scale Feature Fusion Module (CCFM), which optimizes the neck network's efficient hybrid encoder, redesigns the encoder structure, and reconstructs the module structure of the neck. Features of different scales are integrated through a fusion operation, and a fusion block consisting of convolutional layers is inserted into the fusion path. The fusion block fuses adjacent features into a new feature. The fusion block contains N RepBlocks, and the outputs of the two paths are fused through element-wise addition, enhancing the model's adaptability to scale changes and its ability to detect small-scale objects. The third part combines the bot-sort tracking algorithm to optimize continuous tracking of facial movements. It retains low-score detection frames while ensuring high-score detection frame tracking; improves the accuracy of prediction frame prediction by improving the state vector of the Kalman filter and using the camera motion compensation method; and also proposes a cosine distance fusion method for IoU and ReID to improve the robustness between detection and trajectory.

[0058] In step S2, the single-stage target detection model yolov8 of the yolo series is selected as the target detection model. The single-stage target detection model yolov8 uses the C2f module to replace the C3 module of YOLOv5 on the backbone network, and uses DFL and Anchor-Free forms on the classification regression loss function, uses BCE Loss as the classification Loss, and uses DFL Loss and CIoU Loss as the regression loss, and uses the Task-Aligned Assigner positive and negative sample matching method. At the same time, the Mosaic operation is closed in the last 10 epochs.

[0059] Preferably, the specific steps of step S2 are:

[0060] Step S21: Introduce the MobileNetv3-Small structure into the Backbone layer of the yolov8 target detection algorithm, use MobileNetv3 to replace all convolutions in the Backbone layer, and introduce the SE attention mechanism;

[0061] Step S22: Introduce the CCFM module into the Neck part of yolov8 and rebuild the module structure of the Neck part;

[0062] Step S23: To address the problems of low average accuracy due to short-term loss of facial features, which may cause detection failure or misjudgment in the face detection algorithm, continuous tracking optimization of facial dynamics is performed in combination with bot-sort tracking detection;

[0063] Step S24: Introduce the data set in step S1 into the improved yolov8 model and perform model training.

[0064] Preferably, if Figure 7 The figure shows a comparison of the recall-accuracy curves of the original YOLOv8 model and the improved model. To verify the effectiveness of the lightweight backbone, fusion module, and face tracking for face detection, an ablation experiment was conducted on the algorithm based on the improved YOLOv8n. The results of facial feature detection before and after the improvement are shown in Table 1. After integrating the MobileNetv3-Small lightweight backbone, the floating-point computation load, parameter count, and model size were significantly reduced, achieving a significant lightweighting effect. However, the mean average precision and recall rate also decreased by 1.7% and 1.4%, respectively, representing significant losses. After integrating CCFM and BoT-SORT, overall performance was improved, and the model size was even lighter.

[0065] Table 1 Ablation experiment results

[0066]

[0067] As shown in the chart, our model achieves a 1.1% increase in recall compared to YOLOv8n, primarily in the closed_eye, open_mouth, and closed_mouth categories. In particular, the closed_eye category's recall increases by 9.3%, reaching 89%. The average precision (APM) of our model also increases by 0.4%, primarily in the open_eye and open_mouth categories. While these increases in recall and APM are accompanied by a 32.3% and 18.3% decrease in model size and computational complexity, respectively, demonstrating the feasibility of our model's lightweight yet accurate facial feature extraction.

[0068] Table 2 Comparison results of category data before and after improvement

[0069]

[0070] Furthermore, in step S3, since the driver's eye state characteristics are one of the key factors in determining fatigue driving, analyzing the shape and degree of eye opening and closing can effectively identify the driver's fatigue level. The eye aspect ratio is a key indicator, quantifying the eye's openness and closing state by calculating the ratio of the vertical distance between the eye's key points to the horizontal distance between the key points. Similarly, mouth features also play an important role in fatigue assessment. When a person is fatigued, the frequency of yawning increases significantly. Physical fatigue leads to oxygen deprivation in the brain and body, necessitating yawning with a wide mouth to absorb more oxygen to regulate bodily functions. Yawning is often accompanied by fatigue and lack of concentration. Therefore, observing the mouth's opening and closing amplitude and the number of yawns can effectively assess a person's fatigue state. Due to physiological factors, when a person is fatigued, they often nod or tilt their head unconsciously, so head posture estimation is used as an indicator for fatigue detection. Therefore, the present invention combines facial feature models with Dlib key points to reflect the driver's fatigue state based on blink frequency per unit time, longest continuous eye closure time, PERCLOS, the number of yawns, and the total number of nods.

[0071] Preferably, the specific steps of step S3 are:

[0072] Step S31: Based on the Dlib 68 key point positioning, the eye aspect ratio, mouth aspect ratio, and head posture features are obtained. The driver's fatigue state is reflected based on the blink frequency per unit time, the longest continuous eye closure time, PERCLOS, the number of yawns, and the total number of nods.

[0073] Step S32: When EAR is less than 0.2, it is determined that the driver blinked once. If the driver blinks more than 20 times within a unit time of 60 seconds, it is determined that the driver is in a fatigue state; when the longest continuous eye closure time reaches 0.8 seconds, it is determined that the driver is in a fatigue state; when PERCLOS is greater than or equal to 0.2, it is determined that the driver is in a fatigue state;

[0074] Step S33: When MAR is greater than 0.6, it is determined that the driver yawned once. If the driver yawns more than three times within a unit time of 60 seconds, it is determined that the driver is in a fatigue state;

[0075] Step S34: When the pitch is greater than 30°, it is considered that the driver has lowered his head once. If the driver lowers his head three or more times in a unit of time, it is determined that the driver is in a fatigued state.

[0076] Step S35: Combine the changing patterns of each feature in steps S32-S34 to perform fusion judgment. In the multi-feature fusion judgment process, when one feature fails, the other features can still be used as indicators to determine fatigue.

[0077] Preferably, in step S3, the facial judgment of fatigue driving characteristics is mainly based on the eye aspect ratio, mouth aspect ratio and head features. The eye aspect ratio (EAR) is the ratio of the length to the width of the eye, and the opening and closing state of the eyes is quantified by calculating the ratio of the vertical key point distance to the horizontal key point distance of the eye. When the driver feels tired, the degree of closure of the eyelids will increase, resulting in a decrease in the EAR value. The Euclidean distance corresponding to the two eyes is calculated using a geometric method to obtain the aspect ratio value of the eyes at any time. The calculation formula of EAR is as follows. When the eyes are always open, EAR will remain within a relatively stable range; when blinking or closing the eyes occurs, the EAR value will gradually decrease and approach 0; among them, the degree of closure of the eyes exceeds 80% and can be regarded as a closed state. The formula is as follows.

[0078]

[0079]

[0080] Currently, the most reliable method for determining eye fatigue is PERCLOS (Percentage of Eye Closure). Developed by the Carnegie Mellon University Institute, it is defined as the time per unit of time (typically 1 minute or 30 seconds) when the eyes are closed for a certain percentage (70% or 80%). The PERCLOS calculation formula is as follows.

[0081]

[0082] The Mouth Aspect Ratio (MAR) is the ratio of the mouth's length to its width, used to determine whether the mouth is open or closed. Similar to the EAR, when the mouth is closed, the MAR value approaches 0. Using geometric methods, the Euclidean distance between two corresponding values ​​is calculated to obtain the mouth's aspect ratio at any given moment. The MAR calculation formula is as follows.

[0083]

[0084] When people are fatigued, they often nod or tilt their heads unconsciously. Therefore, head pose estimation is used as a metric for fatigue detection. To accurately obtain the 3D spatial angle information of the driver's head, we mapped the 12 axes of a cube based on the driver's facial features. By converting the rotation matrix into rotation vectors, we transformed the 2D image keypoints into a 3D facial model and calculated the 3D pose changes (pitch, yaw, and roll). Because this process is very rigorous, determining the 3D spatial angle information of the head requires understanding the inherent parameters of the camera to reduce the perceived error caused by perspective distortion. First, we use Dlib to locate and extract 2D facial keypoints. Based on the keypoint positions, we perform 3D facial model matching to obtain the corresponding 3D facial model. Then, we use the PNP algorithm in OpenCV to calculate the relationship between the 3D facial keypoints and the 2D facial keypoints. Finally, we use the rotation matrix to calculate the Euler angles.

[0085] The present invention selects a single-stage target detection model of the YOLO series as the target detection and recognition model, improves the original YOLOv8 model structure to obtain an improved YOLOv8 model structure, and conducts corresponding ablation experiments and comparative experiments to obtain a fatigue driving detection model with higher precision and lower model volume; in order to improve the robustness of driver facial feature extraction and the reliability of continuous detection, and considering the lightweight deployment requirements of vehicle-mounted terminals; on the basis of the YOLOv8n target detection algorithm, MobileNetv3-Small is adopted as the feature extraction backbone to achieve lightweight model; SE attention module and CCFM lightweight cross-scale feature fusion module are introduced, and at the same time combined with the bot-sort tracking algorithm, an efficient tracking algorithm for real-time facial feature detection is realized, which effectively improves the accuracy of detection results; combined with Dlib key point positioning, the driver's fatigue level is judged according to data such as the longest continuous eye closure time, the total number of yawns per unit time, the total number of nods, and PERCLOS.

[0086] From the experimental results, it can be seen that for driver fatigue driving detection technology, taking into account the problems of lightweight model requirements, continuous detection and robustness, the present invention has achieved optimization in both accuracy and model volume compared with the original model after a series of improvements and optimizations. While the number of parameters is reduced by 34.5%, the average precision mean and recall rate of the improved model have increased by 1.1% and 0.4% respectively. It can realize real-time detection of small cameras in the car, can not distinguish between races, and has a high detection rate for drivers of all ages and skin colors, effectively reducing the incidence of traffic accidents caused by fatigue driving.

[0087] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A lightweight fatigue driving detection method based on improved yolov8, characterized in that: The specific steps include: Step S1: Collect, filter and annotate facial features of facial fatigue data to create a dataset; Step S2: Improve the network of the original yolov8 and use the dataset to train the target detection model to obtain the fatigue driving detection model; Step S3: Combine the model from step S2 with the Dlib key points and determine the driver's fatigue status based on the eye aspect ratio, mouth aspect ratio, and head swing features; Step S4: Connecting the obtained fatigue driving detection model to the real-time video stream for real-time detection, determining whether the driver in the video is fatigue driving, and making a judgment based on the fatigue state to obtain a detection result; The specific steps of step S2 are: Step S21: Introduce the MobileNetv3-Small structure into the Backbone layer of the yolov8 target detection algorithm, use MobileNetv3 to replace all convolutions in the Backbone layer, and introduce the SE attention mechanism; Step S22: Introduce the CCFM module into the Neck part of yolov8 and rebuild the module structure of the Neck part; Step S23: To address the problem of low average accuracy due to short-term loss of facial features, which may cause detection failure or misjudgment in the face detection algorithm, continuous tracking optimization of facial dynamics is performed in combination with bot-sort tracking detection; Step S24: Introduce the data set in step S1 into the improved yolov8 model for model training.

2. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 1 is characterized in that: In step S1, facial fatigue data is collected, screened, and facial features are annotated. The steps are as follows: Step S11: Constructing a dataset including a real driving yawning dataset YawDD and a self-made dataset; the self-made dataset is obtained by collecting data from the Internet, taking pictures with a camera, and using public datasets on the Internet, and contains image data of people of different skin colors, with eyes open or closed, and in various scenes such as buses and classrooms; Step S12: Screen the data and delete low-quality data; Step S13: For the filtered image data, use labelimg software to perform facial annotation on the dataset.

3. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 2 is characterized in that: The step S1 also includes framing and classifying the data set. The YawDD video data is processed by capturing a picture every 20 frames. The data set is manually labeled using the Labelimg labeling tool. The facial features contained in the image are framed with labeling boxes, and then the target category and location information are output. The labeling format is Pascal VOC format.

4. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 1 is characterized in that: In step S2, the single-stage target detection model yolov8 of the yolo series is selected as the target detection model. The single-stage target detection model yolov8 uses the C2f module to replace the C3 module of YOLOv5 on the backbone network, and uses DFL and Anchor-Free forms on the classification and regression loss functions, uses BCE Loss as the classification Loss, and uses DFL Loss and CIoU Loss as the regression loss, and uses the Task-Aligned Assigner positive and negative sample matching method. At the same time, the Mosaic operation is closed in the last 10 epochs.

5. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 1 is characterized in that: The steps of model training in step S24 are: Step S241: Split the dataset generated in step S1 into a training set, a validation set, and a test set, and ensure that the labeled image data contains the target object category and bounding box coordinate information; Step S242: modifying the model configuration file in the improved yolov8 code to specify the parameters of the model; Step S243: Use the training set and model configuration file to train the model, and adjust the parameter settings of the learning rate, optimizer, and loss function during the training process so that the training outputs the optimal fatigue driving detection model.

6. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 1 is characterized in that: The specific steps of step S3 are: Step S31: Based on the Dlib 68 keypoint positioning, the eye aspect ratio, mouth aspect ratio, and head posture features are obtained. The blink frequency per unit time, the longest continuous eye closure time, PERCLOS, the number of yawns, and the total number of nods are used to reflect the driver's fatigue status. Step S32: When EAR is less than 0.2, it is determined that the driver blinked once; if the driver blinks more than 20 times within a unit time of 60 seconds, it is determined that the driver is in a fatigue state; when the longest continuous eye closure time reaches 0.8 seconds, it is determined that the driver is in a fatigue state; when PERCLOS is greater than or equal to 0.2, it is determined that the driver is in a fatigue state; Step S33: When MAR is greater than 0.6, it is determined that the driver yawns once; if the driver yawns more than three times within a unit time of 60 seconds, it is determined that the driver is in a fatigue state; Step S34: When the pitch is greater than 30°, it is considered that the head-down behavior has occurred once; if the head-down behavior occurs more than three times in a unit of time, it is determined that the driver is in a fatigue state; Combine the changing rules of each feature in steps S32-S34 to perform fusion judgment; in the process of fusion judgment of each feature, when one of the features fails, the other features can still be used as fatigue indicators.

7. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 1 is characterized in that: The facial judgment of fatigue driving characteristics in step S3 is based on the eye aspect ratio, mouth aspect ratio and head features; the opening and closing state of the eyes is quantified by calculating the ratio of the vertical key point distance to the horizontal key point distance of the eyes; and the Euclidean distance corresponding to the two eyes and the mouth is calculated using a geometric method to obtain the aspect ratio value of the eyes and mouth at any time.

8. The method for lightweight fatigue driving detection based on improved yolov8 according to claim 7 is characterized in that: When the eyes are always open, EAR will remain within a relatively stable range; when blinking or closing the eyes occurs, the EAR value will gradually decrease and approach 0; among them, the eyes can be considered closed if the degree of closure exceeds 80%; the same applies to MAR.

Images