Lightweight human face fatigue detection method

Abstract

The invention relates to the field of fatigue driving detection and lightweight, and discloses a lightweight fatigue driving detection method, which comprises the following steps of: 1, sampling from a video according to a fixed time interval of 1 frame per second by using a public data set YawDD, and selecting an eye closing data set Close Eyes in the line Wild as a supplement; 2, selecting YOLOv8n-Pose as a reference model, and optimizing the YOLOv8n-Pose to obtain a fatigue driving detection model; 3, quantifying and deploying the fatigue driving detection model; 4, a result output by the fatigue driving detection model is a key point position, and key points comprise eyes and a mouth; judging driver fatigue according to the position coordinates of the key points; the method is suitable for low-power-consumption and high-real-time industrial scenes such as forklift driving fatigue detection.

Description

Technical Field

[0001] This invention relates to the fields of fatigue driving detection and lightweighting, specifically a lightweight fatigue driving detection method. Background Technology

[0002] To prevent fatigued driving, common detection methods currently include physiological characteristic-based detection, driving behavior-based detection, and vehicle status-based detection. Physiological characteristic-based detection assesses fatigue by monitoring the driver's physiological indicators such as heart rate, blood pressure, brain waves, and eye movements. While it can monitor driver physiological changes in real time and detect fatigue promptly, the equipment may cause discomfort to the driver, affecting driving, and the collection and storage of personal physiological information poses a risk of privacy breaches. Vehicle status-based detection indirectly assesses driver fatigue by monitoring vehicle conditions such as lane departure, vehicle swaying, and driving trajectory. However, vehicle status is affected by various factors, including road conditions and vehicle performance, which may lead to misjudgments.

[0003] In summary, current forklift driver fatigue detection methods have the following shortcomings: (1) Some high-precision fatigue detection devices require specialized sensors and algorithms, resulting in high equipment costs and making them difficult to deploy widely on industrial vehicles such as forklifts. (2) The collection and storage of information may involve the risk of privacy leaks, causing concern and resistance from drivers. (3) The forklift operating environment is complex and variable, such as changes in light and vibration interference, which may affect the accuracy and robustness of the detection system. Therefore, developing a lightweight fatigue detection method that can detect driver status in real time while reducing computational burden and improving the generalization ability of the model has significant market demand and application prospects. This method can solve the shortcomings of current technology, especially in high-safety and high-reliability application scenarios, and has broad application value. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the present invention aims to provide a fatigue driving detection method using an improved deep learning model, thereby addressing the deficiencies in existing technologies and providing a safe, efficient, and miniaturized solution suitable for embedded devices.

[0005] This invention includes the following steps:

[0006] A lightweight method for detecting facial fatigue includes the following steps:

[0007] Step 1: Using the public dataset YawDD, sample 1 frame per second from the video at fixed time intervals. To avoid missing keyframes of crucial fatigue behaviors, the Closed Eyes in the Wild dataset is selected as a supplement. The specific steps are as follows:

[0008] Step 1.1: Filter the selected dataset, removing duplicate or redundant images; and perform grayscale conversion and ROI (Region of Interest) processing on the selected images to reduce computational overhead and enhance robustness;

[0009] Step 1.2: Use the Labelimg annotation tool to manually annotate the selected dataset, outline the facial features in the images with bounding boxes, and then output the target category and location information. The annotation format is PascalVOC format, which is then converted to the YOLO format required by the model.

[0010] Step 2: Select YOLOv8n-Pose as the baseline model and perform optimization. The specific steps are as follows:

[0011] Step 2.1: By replacing the CSPDarknet backbone module in the original YOLOv8n-Pose network with the MobileNetV4 backbone network, the Backbone interface of the original YOLOv8n-Pose backbone module is kept consistent during the network replacement process to ensure seamless integration with the subsequent neck network and Pose Head for key point detection. To improve the accuracy of face detection and facial key point localization tasks, and to meet the deployment requirements in scenarios with limited computing resources, the feature map downsampling structure in the backbone network was optimized, including adjusting the size and position of the pooling window to retain more key information. At the same time, a balance analysis of accuracy and computational cost was performed on feature maps at different resolutions to ensure that the prediction of facial key points such as eyes and mouth has a lower computational load while maintaining high accuracy.

[0012] Step 2.2: Improvement of the detection head, specifically:

[0013] Step 2.2.1: The detector head in the YOLOv8n-Pose network structure was improved by replacing it with a lightweight Lite-GNSC-Head structure; the floating-point Softmax module in the original GNSC-Head was reconstructed; a binary approximation integer Softmax function was adopted to replace the original floating-point Softmax function. The approximation form is more suitable for implementation in integer computing environments, reducing the dependence on high-cost computations and improving the inference efficiency and compatibility of the model on resource-constrained platforms; the binary approximation integer Softmax function is adopted:

[0014]

[0015] Where z i The original logit value for the i-th category live location, where T represents the temperature coefficient, z j Logit values ​​for all categories;

[0016] Step 2.2.2: Replace the traditional standard convolution operation with a depthwise separable convolution structure: First, break down the fully connected relationships between all input channels and the convolution kernel in the standard convolution, and apply an independent K×K convolution kernel to each input channel to extract spatial features within the channel; this operation does not introduce cross-channel interaction, significantly reducing the amount of computation; then, after the depthwise convolution, introduce a 1×1 convolution kernel to linearly combine all channels to achieve information fusion between channels;

[0017] Step 2.3: After replacing the backbone with MobileNetV4, the model is significantly lighter. To avoid a decrease in accuracy, the following accuracy compensation strategy, an attention mechanism, is adopted. The specific steps are as follows:

[0018] Step 2.3.1: Introduce the Efficient-CBAM module in the middle and end of the MobileNetV4 backbone. This module consists of compressed channel attention and spatial attention, which enhances the ability to focus on local facial regions such as eyes, mouth, and nose tip. The channel attention module uses 1×1 convolution to approximate the fully connected layer, reducing parameters. The output has the same number of channel weights as the original feature channels. The spatial attention module takes the feature map after channel merging as input and uses the two-dimensional feature map after channel fusion as input. It uses convolution operations to extract discriminative key regions in the feature map and generates a spatial attention weight map to enhance the feature representation ability of key locations.

[0019] Step 3: Quantify and deploy the fatigue driving detection model, as follows:

[0020] Step 3.1: Save the trained fatigue driving detection model as best.pt. The model needs to be integrated into the fatigue driving detection module. Convert the model to ONNX format. Use the TensorFlow Lite toolchain to perform integer quantization on the model, converting the floating-point model to an 8-bit integer model. During quantization, statistical calibration is performed using the processed dataset. The scaling factor and zero-point of the tensor are calculated, and the following quantization formula is applied:

[0021]

[0022] Where x is a floating-point number, q is the quantized integer value, s is the scaling factor, and z is the zero point;

[0023] In practical deployment, the quantized model uses integer matrix multiplication to complete feature extraction and key point prediction. Its core convolutional inference process can be described as follows:

[0024] Y = s x ·s w(Conv(X q -z x W q -z w ))+z Y (3)

[0025] Where X q and W q s are the quantized integers of the input activation and weights, respectively. x With s w z is the corresponding scaling factor. x With z w ,z y These represent the zero-point offsets of the input, weights, and output, respectively.

[0026] Step 3.2: Deploy the quantized model to the fatigue driving detection module, export the quantized model as a .tflite format, and further use a transcoding tool to convert it into a C language array form .h file for the embedded system to call; embed it into the ESP-IDF project, and complete model loading and inference through the TensorFlow Lite Micro engine;

[0027] Step 4: The model outputs key points for the positions of the eyes and mouth; driver fatigue is determined based on the coordinates of these key points, with the following specific steps:

[0028] Step 4.1: Calculate the aspect ratio EAR of the eye according to equation (4):

[0029]

[0030] For the left eye, P1 represents the outer corner of the left eye, P2 and P3 represent the upper middle and inner sides of the upper eyelid, respectively, P4 represents the inner corner of the left eye, and P5 and P6 represent the inner and lower middle sides of the lower eyelid, respectively. For the right eye, P1 represents the outer corner of the right eye, P2 and P3 represent the upper middle and inner sides of the upper eyelid, respectively, P4 represents the inner corner of the right eye, and P5 and P6 represent the inner and lower middle sides of the lower eyelid, respectively.

[0031] Step 4.2: If EAR is less than E0, then the eyes are determined to be closed, where E0 represents the threshold for determining the aspect ratio of the eyes to be closed;

[0032] Step 4.3: Calculate the driver's eye fatigue state value f according to formula (5). e If f e >f e0 If so, it is determined that the driver is in a state of fatigue;

[0033]

[0034] Among them, t eT represents the number of frames with eyes closed within a given detection time. e f represents the total number of frames during the detection time. e0 Threshold for judging eye fatigue;

[0035] Step 4.4: Calculate the length-to-width ratio (MAR) of the mouth according to equation (6):

[0036] Among them, M1 represents the left corner of the mouth, M2 and M3 represent the middle and upper sides of the upper lip, M4 represents the right corner of the mouth, and M5 and M6 represent the middle and lower sides of the lower lip.

[0037] Step 4.5: If MAR is less than M0, then the mouth is determined to be closed, where M0 represents the threshold of the mouth length-to-width ratio for determining whether the mouth is closed;

[0038] Step 4.6: Calculate the driver's mouth fatigue value f according to formula (7). m If f m >f m0 This indicates that the driver is fatigued;

[0039] The duration of a typical human yawn is 3-5 seconds; in this algorithm, the unit detection time is set to 30 seconds, and more than two yawns within 30 seconds are considered fatigue driving; therefore, this paper will use f m The parameter threshold is set to 0.4, meaning that when the yawning frequency per unit time is greater than 0.4, the driver is judged to be in a state of fatigue; the formula is:

[0040]

[0041] Among them, t m T represents the frequency of yawning per unit of time. m f represents the unit detection time. m0 This indicates the threshold for judging eye fatigue.

[0042] The design concept of this invention is as follows:

[0043] Based on the YOLOv8n-Pose framework, the CSPDarknet backbone module in the YOLOv8n-Pose network is replaced with the MobileNetV4 backbone network. The YOLOv8n-Pose detection head is modified to a Lite-GNSC-Head structure, and the original floating-point Softmax is replaced with a binary approximate integer Softmax function; traditional standard convolution operations are replaced with depthwise separable convolution structures, significantly reducing computational cost. Efficient-CBAM modules are introduced in the middle and end of the MobileNetV4 backbone to enhance the ability to focus on local facial regions such as the eyes, mouth, and nose tip. Finally, TensorFlow Lite Micro is used to complete the model quantization and embedding deployment, making it suitable for low-power, real-time industrial scenarios such as forklift driver fatigue detection.

[0044] By employing the above-described technology, compared with existing technologies, the beneficial effects of this invention are as follows: it can detect fatigue conditions in real time and issue timely warnings without affecting the driver's driving. It can still achieve high-precision fatigue motion detection under low-resolution image input conditions, possesses localized processing capabilities, requires no network connection or cloud dependency, and is suitable for edge scenarios with strict privacy and latency requirements, such as vehicles and head-mounted devices. Attached Figure Description

[0045] Figure 1 Images for the dataset;

[0046] Figure 2 The dataset consists of images after grayscale processing;

[0047] Figure 3 for Figure 2 Image after ROI processing;

[0048] Figure 4 This is the coordinate map of the face after model detection. Detailed Implementation

[0049] The following detailed embodiments illustrate the specific implementation of the forklift driver fatigue detection method of the present invention. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0050] The forklift driver fatigue detection method of this invention first selects YOLOv8n-Pose as the baseline model for optimization. The CSPDarknet backbone module in the YOLOv8n-Pose network is replaced with a MobileNetV4 backbone network. The YOLOv8n-Pose detection head is modified to a Lite-GNSC-Head structure, and the original floating-point Softmax is replaced with a binary approximate integer Softmax function; traditional standard convolution operations are replaced with depthwise separable convolution structures, significantly reducing computational load. Efficient-CBAM modules are introduced in the middle and end of the MobileNetV4 backbone to enhance the ability to focus on local facial regions such as the eyes, mouth, and nose tip. The trained model is then lightweighted and embedded into a microcontroller for use. The forklift driver's state is monitored in real time via a camera, and an audible warning is issued when the model determines that the driver is fatigued.

[0051] Specifically, the steps include the following:

[0052] Step 1: Using the public dataset YawDD, sample 1 frame per second from the video at fixed time intervals. To avoid missing keyframes of crucial fatigue behaviors, the Closed Eyes in the Wild dataset is selected as a supplement. The specific steps are as follows:

[0053] Step 1.1: Filter the selected dataset, removing duplicate or redundant images; and perform grayscale conversion and ROI (Region of Interest) processing on the selected images to reduce computational overhead and enhance robustness;

[0054] Step 1.2: Use the Labelimg annotation tool to manually annotate the selected dataset, outline the facial features in the images with bounding boxes, and then output the target category and location information. The annotation format is PascalVOC format, which is then converted to the YOLO format required by the model.

[0055] Step 2: Select YOLOv8n-Pose as the baseline model and perform optimization. The specific steps are as follows:

[0056] Step 2.1: By replacing the CSPDarknet backbone module in the original YOLOv8n-Pose network with the MobileNetV4 backbone network, the Backbone interface of the original YOLOv8n-Pose backbone module is kept consistent during the network replacement process to ensure seamless integration with the subsequent neck network and Pose Head for key point detection. To improve the accuracy of face detection and facial key point localization tasks, and to meet the deployment requirements in scenarios with limited computing resources, the feature map downsampling structure in the backbone network is optimized. The size and position of the pooling window are adjusted according to the important regions of the feature map to retain more key information. At the same time, a balance analysis of accuracy and computational cost is performed on feature maps at different resolutions to ensure that the prediction of facial key points such as eyes and mouth has a lower computational load while maintaining high accuracy.

[0057] Step 2.2: Improvement of the detection head, specifically:

[0058] Step 2.2.1: The detector head in the YOLOv8n-Pose network structure was improved by replacing it with a lightweight Lite-GNSC-Head structure; the floating-point Softmax module in the original GNSC-Head was reconstructed; a binary approximation integer Softmax function was adopted to replace the original floating-point Softmax function. The approximation form is more suitable for implementation in integer computing environments, reducing the dependence on high-cost computations and improving the inference efficiency and compatibility of the model on resource-constrained platforms; the binary approximation integer Softmax function is adopted:

[0059]

[0060] Where z i The original logit value for the i-th category live location, where T represents the temperature coefficient, z j Logit values ​​for all categories;

[0061] Step 2.2.2: Replace the traditional standard convolution operation with a depthwise separable convolution structure: First, break down the fully connected relationships between all input channels and the convolution kernel in the standard convolution, and apply an independent K×K convolution kernel to each input channel to extract spatial features within the channel; this operation does not introduce cross-channel interaction, significantly reducing the amount of computation; then, after the depthwise convolution, introduce a 1×1 convolution kernel to linearly combine all channels to achieve information fusion between channels;

[0062] Step 2.3: After replacing the backbone with MobileNetV4, the model is significantly lighter. To avoid a decrease in accuracy, the following accuracy compensation strategy, an attention mechanism, is adopted. The specific steps are as follows:

[0063] Step 2.3.1: Introduce the Efficient-CBAM module in the middle and end of the MobileNetV4 backbone. This module consists of compressed channel attention and spatial attention, which enhances the ability to focus on local facial regions such as eyes, mouth, and nose tip. The channel attention module uses 1×1 convolution to approximate the fully connected layer, reducing parameters. The output has the same number of channel weights as the original feature channels. The spatial attention module takes the feature map after channel merging as input and uses the two-dimensional feature map after channel fusion as input. It uses convolution operations to extract discriminative key regions in the feature map and generates a spatial attention weight map to enhance the feature representation ability of key locations.

[0064] Step 3: Quantify and deploy the fatigue driving detection model, as follows:

[0065] Step 3.1: Save the trained fatigue driving detection model as best.pt. The model needs to be integrated into the fatigue driving detection module. Convert the model to ONNX format. Use the TensorFlow Lite toolchain to perform integer quantization on the model, converting the floating-point model to an 8-bit integer model. During quantization, statistical calibration is performed using the processed dataset. The scaling factor and zero-point of the tensor are calculated, and the following quantization formula is applied:

[0066]

[0067] Where x is a floating-point number, q is the quantized integer value, s is the scaling factor, and z is the zero point;

[0068] In practical deployment, the quantized model uses integer matrix multiplication to complete feature extraction and key point prediction. Its core convolutional inference process can be described as follows:

[0069] Y = s x ·s w (Conv(X q -z x W q -z w ))+z Y (3)

[0070] Where X q and W q s are the quantized integers of the input activation and weights, respectively. x With s w z is the corresponding scaling factor. x With z w ,z y These represent the zero-point offsets of the input, weights, and output, respectively.

[0071] Step 3.2: Deploy the quantized model to the fatigue driving detection module, export the quantized model as a .tflite format, and further use a transcoding tool to convert it into a C language array form .h file for the embedded system to call; embed it into the ESP-IDF project, and complete model loading and inference through the TensorFlow Lite Micro engine;

[0072] Step 4: The model outputs key points for the positions of the eyes and mouth; driver fatigue is determined based on the coordinates of these key points, with the following specific steps:

[0073] Step 4.1: Calculate the aspect ratio EAR of the eye according to equation (4):

[0074]

[0075] For the left eye, P1 represents the outer corner of the left eye, P2 and P3 represent the upper middle and inner sides of the upper eyelid, respectively, P4 represents the inner corner of the left eye, and P5 and P6 represent the inner and lower middle sides of the lower eyelid, respectively. For the right eye, P1 represents the outer corner of the right eye, P2 and P3 represent the upper middle and inner sides of the upper eyelid, respectively, P4 represents the inner corner of the right eye, and P5 and P6 represent the inner and lower middle sides of the lower eyelid, respectively.

[0076] Step 4.2: If EAR is less than E0, then the eyes are determined to be closed, where E0 represents the threshold for determining the aspect ratio of the eyes to be closed; in this embodiment, referring to the P80 standard in the PERCLOS standard, E0 = 0.2 is set.

[0077] Step 4.3: Calculate the driver's eye fatigue state value f according to formula (5). e If f e >f e0 If so, it is determined that the driver is in a state of fatigue;

[0078]

[0079] Among them, t e T represents the number of frames with eyes closed within a given detection time. e f represents the total number of frames during the detection time. e0 This represents the threshold for judging eye fatigue; in the embodiment, f is set. e0 =0.5;

[0080] Step 4.4: Calculate the length-to-width ratio (MAR) of the mouth according to equation (6):

[0081]

[0082] Among them, M1 represents the left corner of the mouth, M2 and M3 represent the middle and upper sides of the upper lip, M4 represents the right corner of the mouth, and M5 and M6 represent the middle and lower sides of the lower lip.

[0083] Step 4.5: If MAR is less than M0, then the mouth is determined to be closed, where M0 represents the threshold of the mouth length-to-width ratio for determining mouth closure; in this embodiment, referring to the P80 standard in the PERCLOS standard, M0 = 0.2 is set.

[0084] Step 4.6: Calculate the driver's mouth fatigue value f according to formula (7). m If f m >f m0 This indicates that the driver is fatigued;

[0085] The duration of a typical human yawn is 3-5 seconds; in this algorithm, the unit detection time is set to 30 seconds, and more than two yawns within 30 seconds are considered fatigue driving; therefore, this paper will use f m The parameter threshold is set to 0.4, meaning that when the yawning frequency per unit time is greater than 0.4, the driver is judged to be in a state of fatigue; the formula is:

[0086]

[0087] Among them, t m T represents the frequency of yawning per unit of time. m f represents the unit detection time. m0 This represents the threshold for judging eye fatigue; in this embodiment, f is set. m0 =0.4.

[0088] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms described in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A lightweight method for detecting facial fatigue, characterized in that, Includes the following steps: Step 1: Using the public dataset YawDD, sample from the video at fixed time intervals of 1 frame per second, and select the Closed Eyes in the Wild dataset as a supplement; Step 2: Select YOLOv8n-Pose as the baseline model and optimize it to obtain the fatigue driving detection model; Step 3: Quantify and deploy the fatigue driving detection model; Step 4: The fatigue driving detection model outputs the key point locations, including the eyes and mouth; Driver fatigue can be determined based on the location coordinates of key points.

2. The lightweight face fatigue detection method according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 1.1: Filter the selected dataset, removing duplicate or redundant images; and perform grayscale conversion and ROI (Region of Interest) processing on the selected images to reduce computational overhead and enhance robustness; Step 1.2: Use the Labelimg annotation tool to manually annotate the selected dataset, outline the facial features in the images with annotation boxes, and then output the target category and location information. The annotation format is Pascal VOC format, which is then converted to the YOLO format required by the model.

3. The lightweight face fatigue detection method according to claim 1, characterized in that, The specific steps of step 2 are as follows: Step 2.1: Replace the CSPDarknet backbone module in the original YOLOv8n-Pose network with the MobileNetV4 backbone network. During the network replacement process, the Backbone interface of the backbone module in the original YOLOv8n-Pose will be kept consistent to ensure seamless integration with the subsequent neck network and key point detection Pose Head. To improve the accuracy of face detection and facial landmark localization tasks, and to meet the deployment requirements in scenarios with limited computing resources, the feature map downsampling structure in the backbone network was optimized, including adjusting the size and position of the pooling window to retain more information. Step 2.2: Improvement of the detection head, specifically: Step 2.2.1: The detection head part in the YOLOv8n-Pose network structure was improved by replacing it with a lightweight Lite-GNSC-Head structure; the floating-point Softmax module in the original GNSC-Head was reconstructed; and the original Softmax function based on floating-point operations was replaced with a replacement function based on binary approximation integer Softmax. The binary approximation integer Softmax function is as follows: Where z i The original logit value for the i-th category live location, where T represents the temperature coefficient, z j Logit values ​​for all categories; Step 2.2.2: Replace the traditional standard convolution operation with a depthwise separable convolution structure to decouple the spatial and channel dimensions in the original convolution: First, use Depthwise convolution to extract spatial features independently on each input channel, and then use Pointwise (1×1) convolution to linearly combine the outputs of each channel, thereby achieving cross-channel information fusion. Step 2.3: After replacing the backbone with MobileNetV4, the model is significantly lighter. To avoid a decrease in accuracy, the following accuracy compensation strategy, an attention mechanism, is adopted. The specific steps are as follows: Step 2.3.1: Introduce Efficient-CBAM modules in the middle and end of the MobileNetV4 backbone. The Efficient-CBAM module includes a compressed channel attention module and a spatial attention module. The compressed channel attention module uses 1×1 convolutions to approximate fully connected layers, reducing parameters. It outputs channel weights with the same number of original feature channels. The spatial attention module takes the two-dimensional feature map after channel fusion as input, uses convolution operations to extract discriminative regions in the feature map, and generates a spatial attention weight map to enhance the feature representation ability of key locations.

4. The lightweight face fatigue detection method according to claim 1, characterized in that, The specific steps of step 3 are as follows: Step 3.1: Save the trained fatigue driving detection model as best.pt. The model needs to be integrated into the fatigue driving detection module. Convert the model to ONNX format. The TensorFlow Lite toolchain was used to perform integer quantization on the model, converting the floating-point model into an 8-bit integer model. During the quantization process, a pre-processed dataset was used for statistical calibration. The scaling factor and zero-point of the tensor were calculated, and the following quantization formula was applied: Where x is a floating-point number, q is the quantized integer value, s is the scaling factor, and z is the zero point; In practical deployment, the quantized model uses integer matrix multiplication to complete feature extraction and key point prediction. Its core convolutional inference process is described as follows: Y=s x ·s w (Conv(X q -z x ,W q -z w ))+z Y (3) Where X q and W q s are the quantized integers of the input activation and weights, respectively. x With s w z is the corresponding scaling factor. x With z w ,z y These represent the zero-point offsets of the input, weights, and output, respectively. Step 3.2: Deploy the quantized model to the fatigue driving detection module, export the quantized model as a .tflite format, and further use a transcoding tool to convert it into a C language array form .h file for the embedded system to call; embed it into the ESP-IDF project, and complete model loading and inference through the TensorFlow Lite Micro engine.

5. The lightweight face fatigue detection method according to claim 1, characterized in that, The specific steps of step 4 are as follows: Step 4.1: Calculate the aspect ratio EAR of the eye according to equation (4): For the left eye, P1 represents the outer corner of the left eye, P2 and P3 represent the upper middle and inner sides of the upper eyelid, respectively, P4 represents the inner corner of the left eye, and P5 and P6 represent the inner and lower middle sides of the lower eyelid, respectively. For the right eye, P1 represents the outer corner of the right eye, P2 and P3 represent the upper middle and inner sides of the upper eyelid, respectively, P4 represents the inner corner of the right eye, and P5 and P6 represent the inner and lower middle sides of the lower eyelid, respectively. Step 4.2: If EAR is less than E0, then the eyes are determined to be closed, where E0 represents the threshold for determining the aspect ratio of the eyes to be closed; Step 4.3: Calculate the driver's eye fatigue state value f according to formula (5). e If f e >f e0 If so, it is determined that the driver is in a state of fatigue; Among them, t e T represents the number of frames with eyes closed within a given detection time. e f represents the total number of frames during the detection time. e0 Threshold for judging eye fatigue; Step 4.4: Calculate the length-to-width ratio (MAR) of the mouth according to equation (6): Among them, M1 represents the left corner of the mouth, M2 and M3 represent the middle and upper sides of the upper lip, M4 represents the right corner of the mouth, and M5 and M6 represent the middle and lower sides of the lower lip. Step 4.5: If MAR is less than M0, then the mouth is determined to be closed, where M0 represents the threshold of the mouth length-to-width ratio for determining whether the mouth is closed; Step 4.6: Calculate the driver's mouth fatigue value f according to formula (7). m If f m >f m0 This indicates that the driver is fatigued; Among them, t m T represents the frequency of yawning per unit of time. m f represents the unit detection time. m0 This indicates the threshold for judging eye fatigue.

Images