A face covering driver drowsiness recognition method based on eye local features

Abstract

The application provides a driver drowsiness recognition method based on facial occlusion and eye local features, comprising an eye region detection module, an eye key point extraction module and a drowsiness state discrimination module. First, a labeled data set containing occlusion scenes such as masks and hats is constructed, and the eye region detection model and the key point extraction model are trained respectively, wherein the key point training needs to convert the labeled coordinates to the eye local image coordinate system. In the detection stage, the eye region image is obtained continuously and stably by combining the region detection model with the tracking algorithm; the eye region image is input into the key point extraction model, and the eye key point coordinates are obtained through the heat map analysis. Finally, the drowsiness is discriminated based on the eye key points and time sequence features. The application focuses on the eye local features, avoids the interference of facial occlusion on feature extraction, does not need the traditional whole face detection path, and significantly improves the accuracy of driver drowsiness recognition in the occlusion scene.

Description

Technical Field

[0001] This invention relates to the field of drowsiness recognition, and more particularly to a method for recognizing driver drowsiness in cases of facial occlusion based on local eye features. Background Technology

[0002] With the continuous development of the global transportation industry, traffic accidents have become one of the main factors threatening life and property safety. Statistics show that road traffic injuries are the leading cause of death for people aged 5 to 29, with fatigue driving playing a key role. Data indicates that fatigue driving accounts for approximately 14% to 20% of general traffic accidents, rising to 43% in major traffic accidents, and reaching as high as 37% in scenarios involving large trucks and highways. Further data shows that approximately 40% of major traffic accidents involving heavy trucks are directly related to fatigue driving.

[0003] Fatigue driving refers to a decline in a driver's alertness due to prolonged continuous driving or insufficient sleep, leading to a series of behaviors that affect safe driving, such as inattention, slow reaction time, and impaired judgment. Due to its highly concealed and sudden nature, fatigue driving has become a major threat to public safety. Therefore, developing a technological system capable of accurately monitoring driver drowsiness is of significant practical importance for preventing traffic accidents and protecting lives.

[0004] However, existing driver drowsiness detection technologies primarily rely on facial feature extraction. They typically capture facial images using cameras and employ computer vision algorithms to analyze indicators such as eyelid closure, yawning frequency, and head posture to determine driver drowsiness. The core of these methods lies in acquiring complete facial information. However, in real-world driving scenarios, facial occlusion is a common problem—for example, drivers may wear masks or hats, or their faces may be partially obscured due to lighting conditions or camera angles. These situations severely interfere with the accuracy of facial landmark localization, leading to a significant drop in the accuracy of traditional methods. Therefore, achieving reliable driver drowsiness detection under conditions of facial occlusion has become a major challenge in the field of drowsiness detection. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a method for driver drowsiness recognition when the face is obscured. This method aims to solve the problem of decreased accuracy in traditional drowsiness recognition methods due to obstructions such as masks and hats in real-world driving scenarios. The method is optimized for driver facial obstruction, employing eye region detection, robust key point extraction, and temporal state discrimination to accurately identify driver drowsiness even in complex obstructed scenarios.

[0006] The first aspect of this invention provides a method for recognizing driver drowsiness based on facial occlusion using local eye features, comprising: S1. Construct a labeled dataset containing facial occlusion scenes, and train the eye region detection model and the eye key point extraction model respectively; among them, the training of the eye key point extraction model requires transforming the labeled coordinates to the local eye image coordinate system; S2. Construct an eye region detection module. The eye region detection module adopts a strategy that combines a trained eye region detection model with a tracking algorithm to obtain continuous and stable eye region images. S3. Construct an eye key point extraction module that uses an eye key point extraction model. Input the eye region image into the trained eye key point extraction model and obtain the coordinates of the eye key points through heatmap analysis. S4. Construct a drowsy state discrimination module, which judges the driver's drowsy state based on the coordinates of key eye points and time sequence features.

[0007] Further, step S1 involves constructing a labeled dataset containing facial occlusion scenes, and training the eye region detection model and the eye keypoint extraction model, respectively, including: S11. Collect facial images of the driver in scenarios where the face is obscured by masks, hats, etc., and annotate the bounding box of the eye area and six key points of the eye. S12. Transform the coordinates of the labeled eye key points from the original whole face image coordinate system to the eye local image coordinate system to generate eye key point training samples; use the labeled eye region bounding boxes as eye region detection training samples. S13. Train the eye region detection model and the eye key point extraction model using eye region detection training samples and eye key point training samples, respectively.

[0008] Further, the eye region detection module in step S2 includes an eye region detection model and a KCF tracker; the eye region detection module adopts a strategy combining a trained eye region detection model with a tracking algorithm to obtain continuous and stable eye region images, including: S21. The YOLOv8s model is used as the eye region detection model and combined with the KCF tracker to form a detection-tracking module. S22. The YOLOv8s model is started in the first frame and every 5 frames to detect the eye region. The detection confidence threshold is set to 0.5. If the result is lower than the threshold, the detection result is discarded. The KCF tracker is used to update the eye region position in non-detection frames. S23. Adaptively expand and constrain the eye region obtained by detection or tracking; S24. Cropping and expanding the eye region image to obtain a continuous and stable eye region image sequence.

[0009] Furthermore, the adaptive expansion ratio is 10%, that is, expanding 10% upwards, downwards, leftwards, and rightwards; the boundary constraint is used to ensure that the expanded area does not exceed the range of the original image.

[0010] Further, in step S3, the eye key point extraction module uses the UNet model as the eye key point extraction model; the step of inputting the eye region image into the trained eye key point extraction model and obtaining the coordinates of the eye key points through heatmap analysis includes: S31. Preprocess the input eye region image by converting it into a PyTorch tensor and normalizing it according to the mean and standard deviation of the ImageNet dataset. S32. Input the preprocessed image into the UNet model for prediction and output the heat map corresponding to the six key points. S33. Perform peak analysis on the heatmap of each key point to obtain the key point coordinates in the heatmap coordinate system, and map the key point coordinates from the heatmap size back to the original eye region image size to obtain accurate eye key point coordinates.

[0011] Further, the drowsiness state discrimination module in step S4 performs a driver drowsiness state discrimination process based on eye key point coordinates and time sequence features, including: S41. Calculate the eye aspect ratio (EAR) based on the coordinates of six key eye points. The calculation formula is as follows: In the formula, P1 represents the left and right outer corners of the eyes, P4 represents the left and right inner corners of the eyes, P2 represents the left edge of the upper eyelid, P3 represents the right edge of the upper eyelid, P5 represents the right edge of the lower eyelid, and P6 represents the left edge of the lower eyelid. S42. Based on the eye aspect ratio, blink detection is performed. When EAR < 0.16, it is determined that the eyes are closed. If the interval between the current time of closing the eyes and the last time of closing the eyes is less than 0.5 seconds, it is considered a continuation of the same blink, and only the duration is updated. Otherwise, it is recorded as a new blink (the list is reset and the current time is stored in the blink frequency list). At the same time, single-frame drowsiness state is determined. When EAR < 0.2, the current frame is determined to be in a drowsy state. S43. Introduce temporal features and majority voting mechanism, maintain a sliding window with a length of 20 frames, count the number of times the drowsy state appears in the window, and output the state with the most occurrences as the final judgment result; at the same time, combine the blink frequency index for comprehensive judgment to further improve the accuracy of drowsiness recognition.

[0012] A second aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor as described above in a method for recognizing driver drowsiness in facial occlusion based on local eye features.

[0013] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for recognizing driver drowsiness based on facial occlusion using local eye features.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1) Strong anti-occlusion recognition capability: This invention is specifically designed for facial occlusion scenarios. It adopts a cascaded detection and key point heatmap prediction model, which effectively overcomes the interference of occlusion objects such as masks and sunglasses on eye feature extraction, and significantly improves the accuracy of drowsiness recognition under occlusion conditions.

[0015] 2) High detection and tracking stability: The detection and tracking strategy combines YOLOv8s and KCF tracking algorithms, and with adaptive expansion of the eye region and boundary constraints, it achieves stable tracking of continuous frames while ensuring detection accuracy, avoiding feature loss caused by occlusion or rapid movement.

[0016] 3) Robustness of key point localization: The key point heatmap generated by the UNet model is more robust to interference such as occlusion and lighting changes compared to the method of directly predicting coordinates. The model training is more stable and the key point localization is more accurate.

[0017] 4) Smooth and reliable state determination: The introduction of timing features and majority voting mechanism in the drowsiness determination stage effectively filters out noise interference caused by false detection in a single frame, improves the continuity and stability of state determination, and avoids frequent changes in system state. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the steps of a method for recognizing driver drowsiness through facial occlusion based on local eye features.

[0019] Figure 2 This diagram illustrates the training process of the eye region detection model and the eye key point extraction model.

[0020] Figure 3 This is a schematic diagram of the network structure during the testing phase of the method in this embodiment.

[0021] Figure 4 This is a schematic diagram of the eye region detection module.

[0022] Figure 5This is a schematic diagram of the eye key point extraction module. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0024] Please see Figure 1 , Figure 1 A flowchart illustrating the steps of a method for recognizing driver drowsiness based on facial occlusion using local eye features, as provided in this embodiment of the invention, includes the following steps: S1. Construct a labeled dataset containing facial occlusion scenes, and train the eye region detection model and the eye key point extraction model respectively; among them, the training of the eye key point extraction model requires transforming the labeled coordinates to the local eye image coordinate system; S2. Construct an eye region detection module. The eye region detection module adopts a strategy that combines a trained eye region detection model with a tracking algorithm to obtain continuous and stable eye region images. S3. Construct an eye key point extraction module that uses an eye key point extraction model. Input the eye region image into the trained eye key point extraction model and obtain the coordinates of the eye key points through heatmap analysis. S4. Construct a drowsy state discrimination module, which judges the driver's drowsy state based on the coordinates of key eye points and time sequence features.

[0025] In its specific implementation, this invention first trains the eye region detection model and the eye key point extraction model using a labeled dataset. The training process is as follows: Figure 2 As shown; subsequently, the input video stream was analyzed through an eye region detection module, an eye key point extraction module, and a drowsiness state discrimination module to conduct a drowsiness state recognition test. The network structure diagram during the test phase is shown in the figure. Figure 3 As shown in the attached figures. The model training and testing process will be explained in detail below with reference to the accompanying figures.

[0026] I. Model Training Process In one specific implementation of this embodiment, the process of constructing the labeled dataset and training the eye region detection model and the eye key point extraction model is as follows: Figure 2 As shown, the specific steps include: S11. Collect facial images of drivers wearing masks, hats, or other face coverings, and annotate the bounding boxes of the eye region and six key eye points to obtain the original annotation file. The six key eye points include the left and right outer corners of the eyes, the left and right inner corners of the eyes, the left and right edges of the upper eyelids, and the left and right edges of the lower eyelids. Convert the original annotation file into an eye region annotation file and an eye key point annotation file, respectively.

[0027] S12. Use the bounding boxes of the eye regions in the eye region annotation file as training samples for eye region detection; transform the coordinates of the eye key points in the eye key point annotation file from the original whole-face image coordinate system to the eye local image coordinate system to generate eye key point training samples. The specific coordinate transformation method is as follows: Let the coordinates of the upper left corner of the eye region bounding box be (x... min ,y min If the original keypoint coordinates are (x, y), then the keypoint coordinates in the transformed eye local image coordinate system are (xx, y). min yy min ).

[0028] S13. Train the eye region detection model and the eye keypoint extraction model using eye region detection training samples and eye keypoint training samples, respectively. The eye region detection model uses YOLOv8s and is implemented on an Nvidia GeForce GTX 1660 Ti graphics card using the PyTorch deep learning framework. The training parameters are set as follows: initial learning rate 0.01, weight decay 0.0005, batch size 16, and number of training epochs 300. The eye keypoint extraction model uses UNet and is implemented on an Nvidia GeForce RTX 3050 graphics card using PyTorch. The training parameters are set as follows: initial learning rate 0.001, batch size 16, and number of training epochs 200. A Gaussian heatmap is generated for each keypoint as the training objective, and the Gaussian kernel sigma parameter is set to 3.

[0029] II. Testing Process During the testing phase, the input video stream first passes through the eye region detection module to obtain a continuous and stable sequence of eye region images. Then, the eye region images are input into the eye keypoint extraction module, which obtains precise eye keypoint coordinates through heatmap analysis. Finally, the drowsiness state discrimination module outputs the driver's drowsiness state based on the eye keypoints and temporal features. The network structure during the testing phase is as follows: Figure 3 As shown.

[0030] In one specific embodiment of this example, the eye region detection module employs a strategy combining a trained eye region detection model with a tracking algorithm to acquire continuous and stable eye region images, specifically including the following steps: S21, such as Figure 4As shown, the eye region detection module constructed in this embodiment uses the YOLOv8s model as the eye region detection model and combines it with the KCF tracker to form a detection-tracking module. The YOLOv8s model used includes a backbone network, a neck network, and a head; the backbone network is based on the CSPDarknet architecture and consists of 4 C2f modules and 1 SPPF module; the neck network adopts an FPN-PAN feature fusion structure; and the head adopts a decoupled head.

[0031] S22. In the specific implementation process, the YOLOv8s model is started for eye region detection in the first frame and every 5 frames. The detection confidence threshold is set to 0.5. If the result is lower than the threshold, the detection result is discarded. In non-detection frames, the KCF tracker is used to update the eye region position of the current frame according to the position of the previous frame to ensure that the eye region tracking of consecutive frames is successful.

[0032] S23. Adaptively expand the detected or tracked eye region by 10%, that is, expand the original bounding box size by 10% in all directions (up, down, left, and right), and add boundary constraints to ensure that the expanded region does not exceed the original image range, thus avoiding the loss of eye features due to the tracking box being too narrow.

[0033] S24. Cropping and expanding the eye region image to obtain a continuous and stable eye region image sequence for subsequent key point detection.

[0034] In one specific implementation of this embodiment, the eye key point extraction module uses the UNet model to generate a heatmap, and obtains the coordinates of the eye key points through heatmap analysis, specifically including the following steps: S31, such as Figure 5 As shown, the eye keypoint extraction module constructed in this embodiment uses the UNet model as the eye keypoint extraction model. The UNet model consists of an encoder and a decoder. The encoder performs downsampling four times (max pooling + convolutional units), and the decoder performs upsampling four times (transposed convolution + convolutional units). It also fuses feature information from different levels through skip connections.

[0035] S32. Preprocess the input eye region image by converting it into a PyTorch tensor and normalizing it according to the mean and standard deviation of the ImageNet dataset to convert the image into the model input format.

[0036] S33. Input the preprocessed eye region image into the trained UNet model for prediction, and output a heatmap corresponding to six key points. Compared with direct coordinate regression, the heatmap mechanism is more robust to occlusion and has more stable training.

[0037] S34. Perform peak analysis on the heatmap of each key point to find the location of the maximum value as the key point coordinates in the heatmap coordinate system; then map the key point coordinates from the heatmap size back to the original eye region image size to obtain the accurate eye key point coordinates.

[0038] In one specific implementation of this embodiment, the drowsiness detection module determines the driver's drowsiness state based on the coordinates of key eye points and temporal features, specifically including the following steps: S41. Calculate the eye aspect ratio (EAR) based on the coordinates of six key eye points. The calculation formula is as follows: (1) In the formula, P1 represents the left and right outer corners of the eyes, P4 represents the left and right inner corners of the eyes, P2 represents the left edge of the upper eyelid, P3 represents the right edge of the upper eyelid, P5 represents the right edge of the lower eyelid, and P6 represents the left edge of the lower eyelid.

[0039] S42. Based on the eye aspect ratio, blink detection is performed. When EAR < 0.16, it is determined that the eyes are closed. If the interval between the current time of closing the eyes and the last time of closing the eyes is less than 0.5 seconds, it is considered a continuation of the same blink, and only the duration is updated. Otherwise, it is recorded as a new blink (the list is reset and the current time is stored in the blink frequency list). At the same time, single-frame drowsiness state determination is performed. When EAR < 0.2, the current frame is determined to be in a drowsy state.

[0040] S43. Introducing temporal features and a majority voting mechanism, a sliding window with a length of 20 frames is maintained. The number of times the drowsy state appears within the window is counted, and the state with the most occurrences is used as the final drowsy judgment result output, thereby effectively filtering out single-frame noise interference and improving the stability of state judgment. At the same time, a comprehensive judgment is made by combining the blink frequency index. The blink frequency is calculated based on the number of blinks within a 4-second time window, with the unit being blinks / minute, further improving the accuracy of drowsiness recognition.

[0041] This invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of the steps of a facial occlusion driver drowsiness recognition method based on local eye features as described in any of the above embodiments.

[0042] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a method for recognizing driver drowsiness based on local eye features as described in any of the above embodiments.

[0043] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions conceived without inventive effort should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims.

[0044] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for recognizing driver drowsiness due to facial occlusion based on local eye features, characterized in that, include: S1. Construct a labeled dataset containing facial occlusion scenes, and train the eye region detection model and the eye key point extraction model respectively; among them, the training of the eye key point extraction model requires transforming the labeled coordinates to the local eye image coordinate system; S2. Construct an eye region detection module. The eye region detection module adopts a strategy that combines a trained eye region detection model with a tracking algorithm to obtain continuous and stable eye region images. S3. Construct an eye key point extraction module that uses an eye key point extraction model. Input the eye region image into the trained eye key point extraction model and obtain the coordinates of the eye key points through heatmap analysis. S4. Construct a drowsy state discrimination module, which judges the driver's drowsy state based on the coordinates of key eye points and time sequence features.

2. The method according to claim 1, characterized in that, Step S1 involves constructing a labeled dataset containing facial occlusion scenes, and training the eye region detection model and the eye keypoint extraction model, respectively, including: S11. Collect facial images of the driver in scenarios where the face is obscured by masks, hats, etc., and annotate the bounding box of the eye area and six key points of the eye. S12. Transform the coordinates of the labeled eye key points from the original whole face image coordinate system to the eye local image coordinate system to generate eye key point training samples; use the labeled eye region bounding boxes as eye region detection training samples. S13. Train the eye region detection model and the eye key point extraction model using eye region detection training samples and eye key point training samples, respectively.

3. The method according to claim 1, characterized in that, Step S2 describes an eye region detection module that includes an eye region detection model and a KCF tracker. The eye region detection module employs a strategy combining a trained eye region detection model with a tracking algorithm to acquire continuous and stable eye region images, including: S21. The YOLOv8s model is used as the eye region detection model and combined with the KCF tracker to form a detection-tracking module. S22. The YOLOv8s model is started in the first frame and every 5 frames to detect the eye region. The detection confidence threshold is set to 0.

5. If the result is lower than the threshold, the detection result is discarded. The KCF tracker is used to update the eye region position in non-detection frames. S23. Adaptively expand and constrain the eye region obtained by detection or tracking; S24. Cropping and expanding the eye region image to obtain a continuous and stable eye region image sequence.

4. The method according to claim 3, characterized in that, The adaptive expansion ratio is 10%, that is, expanding 10% upwards, downwards, leftwards, and rightwards; the boundary constraint is used to ensure that the expanded area does not exceed the range of the original image.

5. The method according to claim 1, characterized in that, Step S3 describes an eye keypoint extraction module that uses the UNet model as the eye keypoint extraction model; the step of inputting the eye region image into the trained eye keypoint extraction model and obtaining the eye keypoint coordinates through heatmap analysis includes: S31. Preprocess the input eye region image by converting it into a PyTorch tensor and normalizing it according to the mean and standard deviation of the ImageNet dataset. S32. Input the preprocessed image into the UNet model for prediction and output the heat map corresponding to the six key points. S33. Perform peak analysis on the heatmap of each key point to obtain the key point coordinates in the heatmap coordinate system, and map the key point coordinates from the heatmap size back to the original eye region image size to obtain accurate eye key point coordinates.

6. The method according to claim 1, characterized in that, Step S4, the drowsiness discrimination module, performs a driver drowsiness discrimination process based on eye key point coordinates and time-series features, including: S41. Calculate the eye aspect ratio (EAR) based on the coordinates of six key eye points. The calculation formula is as follows: In the formula, P1 represents the left and right outer corners of the eyes, P4 represents the left and right inner corners of the eyes, P2 represents the left edge of the upper eyelid, P3 represents the right edge of the upper eyelid, P5 represents the right edge of the lower eyelid, and P6 represents the left edge of the lower eyelid. S42. Based on the eye aspect ratio, blink detection is performed. When EAR < 0.16, it is determined that the eyes are closed. If the interval between the current time of closing the eyes and the time of closing the eyes is less than 0.5 seconds, it is considered as a continuation of the same blink, and only the duration is updated. Otherwise, it is recorded as a new blink. At the same time, single-frame drowsiness state is determined. When EAR < 0.2, the current frame is determined to be in a drowsy state. S43. Introduce temporal features and majority voting mechanism, maintain a sliding window with a length of 20 frames, count the number of times the drowsy state appears in the window, and output the state with the most occurrences as the final judgment result; at the same time, combine the blink frequency index for comprehensive judgment to further improve the accuracy of drowsiness recognition.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the driver drowsiness recognition method based on facial occlusion as described in any one of claims 1-6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the driver drowsiness recognition method based on facial occlusion as described in any one of claims 1-6.

Images