Fatigue driving detection method based on adaptive facial motion feature threshold

Abstract

This invention discloses a fatigue driving detection method based on adaptive facial motion feature thresholds, specifically including the following steps: Step 1, acquiring real-time video of the driver's driving process and loading the video stream into image frames; Step 2, identifying the label information, confidence level, and location information in the image frames loaded in Step 1; Step 3, cyclically reading the image frames loaded in Step 1 and calculating EAR and MAR; Step 4, adaptively calculating EART and MART based on the EAR and MAR obtained in Step 3 at different time periods; Step 5, counting the number of blinks and yawns; Step 6, determining whether the driver is fatigued in the current time period based on the number of blinks and yawns counted in Step 5, and then continuing to cyclically execute Step 1 to determine the fatigue state in the next time period. This method solves the problems of low robustness and low reliability caused by using a fixed fatigue detection threshold in existing technologies.

Description

Technical Field

[0001] This invention belongs to the field of image processing methods and relates to a fatigue driving detection method based on adaptive facial motion feature thresholds. Background Technology

[0002] When drivers maintain focus on driving for extended periods without sufficient rest, their physiological and psychological functions gradually become imbalanced. This leads to a significant decline in their attention to and control of the vehicle, as well as their reaction time to unexpected events. In severe cases, this can result in temporary sleep deprivation, causing the vehicle to become completely out of control; this phenomenon is known as drowsy driving.

[0003] Currently, domestic and international methods for detecting driver fatigue mainly fall into two categories: fatigue detection based on physiological characteristics and fatigue detection based on facial features. Among them, fatigue detection algorithms based on facial features determine whether a driver is fatigued by using a fixed threshold to determine the state of their eyes and mouth. However, each driver's eye and mouth features are different, and existing algorithms do not take into account these individual facial differences. This method of using a uniform, fixed threshold to determine fatigue status leads to defects such as low robustness and low reliability in the detection results. Summary of the Invention

[0004] The purpose of this invention is to provide a fatigue driving detection method based on adaptive facial motion feature thresholds, which solves the problems of low robustness and low reliability caused by using fixed fatigue detection thresholds in the prior art.

[0005] The technical solution adopted in this invention is a fatigue driving detection method based on adaptive facial action feature thresholds, which specifically includes the following steps:

[0006] Step 1: Acquire real-time video of the driver's driving process and load the video stream into image frames;

[0007] Step 2: Use the YOLO V5 network to identify the label information, confidence level, and location information (outer frame) in the image frame loaded in Step 1. Use the label information to determine whether the driver has sat in the driver's seat. If the driver has not sat in the driver's seat, continue to execute Step 1 until the driver starts to sit in the driver's seat before executing Step 3.

[0008] Step 3: Read the image frames loaded in Step 1 in a loop, obtain the facial feature location information for each frame, and calculate the eye aspect ratio (EAR) and mouth aspect ratio (MAR).

[0009] Step 4: Based on the aspect ratio of the eye (EAR) and the aspect ratio of the mouth (MAR) obtained in Step 3, adaptively calculate the eye aspect ratio threshold (EART) and the mouth aspect ratio threshold (MART) at different time periods.

[0010] Step 5: Count the number of blinks and yawns based on the length-to-width ratio thresholds of the eyes and mouth obtained in Step 4.

[0011] Step 6: Based on the number of blinks and yawns counted in Step 5, determine whether the current period is fatigued. Then, continue to repeat Step 1 to determine the fatigue status for the next period.

[0012] The invention is further characterized in that,

[0013] The specific process of step 1 is as follows:

[0014] Step 1.1: Obtain the video source using OpenCV's video acquisition functions;

[0015] Step 1.2: After obtaining the video source of the driver's driving process in Step 1.1, set the timer period, connect the slot function of the timer period overflow, and use it to display one frame of video. After loading it into an image frame, read each frame of the video.

[0016] The specific implementation method for step 2 is as follows:

[0017] Step 2.1: Keep the original aspect ratio of the image frame loaded in Step 1, and fill the remaining part with gray to obtain a new image frame;

[0018] Step 2.2 converts the non-contiguous memory-stored arrays in the new image frames obtained in Step 2.1 into contiguous memory-stored NumPy arrays, which makes the running speed faster;

[0019] Step 2.3: Convert the NumPy array processed in Step 2.2 into a PyTorch tensor, with both sharing memory;

[0020] Step 2.4: Process the tensor obtained in Step 2.3 through the YOLO V5 network, and extract the highest score among all scores as the final result.

[0021] Step 2.5: Extract label information, confidence score, and location information (outline) from the final result of Step 2.4;

[0022] Step 2.6: Determine whether the driver has sat in the driver's seat based on the tag information obtained in Step 2.5. If no tag information indicating that the driver has sat in the driver's seat is detected, continue to execute Step 1 until the tag information indicating that the driver has sat in the driver's seat is detected before executing Step 3.

[0023] The specific process of step 3 is as follows:

[0024] Step 3.1: Use the Dlib face landmark detector to process and obtain the facial landmark feature information in the image frame loaded in Step 1;

[0025] Step 3.2: Convert the facial key point feature information obtained in Step 3.1 into array format. Extract the coordinates of the left and right eyes and mouth based on the indices of facial landmarks such as the left and right eyes and mouth. Calculate the corresponding Euclidean distance using the coordinates, and obtain the aspect ratio of the left and right eyes and mouth using this Euclidean distance. The mathematical formulas for calculating the aspect ratio of the eyes and mouth are as follows: (1) and (2):

[0026]

[0027]

[0028] In the formula, EAR (eye aspect ratio) represents the length-to-width ratio of the eye, and p1, p2, p3, p4, p 5, p6 represents the coordinates of six key points around the eyes; MAR (mouth aspect ratio) represents the length-to-width ratio of the mouth, m1, m2, m3, m4, m 5, m6 represents the coordinates of the six key points on the mouth.

[0029] The specific process of step 4 is as follows:

[0030] Step 4.1: Create arrays EYE_EAR and MOUTH_MAR to store the aspect ratios of the eyes and mouth, respectively; create a threshold calculation sequence array ThresholdSort, an integer parameter ThresholdCount for the number of threshold calculations, and a boolean parameter Threshold for whether to calculate the threshold; create an array ERA for the eye aspect ratios and an array MARA for the mouth aspect ratios; create an integer parameter Roll for the current time frame count; create an integer parameter imageCount for the number of image frames; 1 second of image is 12 frames, and imageCount ranges from 120 to 180; when the driver first sits in the driver's seat, set the ThresholdCount parameter to 1 and begin extracting the image frames of the imageCount count. The aspect ratios of the eyes (EAR) and mouth (MAR) of the Count frame image are stored in the arrays EYE_EAR and MOUTH_MAR, respectively. To ensure the rationality and reliability of video selection, five random numbers are generated from 50 seconds to 1800 seconds, and these five second numbers are sorted in ascending order and stored in the array ThresholdSort. When the video reaches the frame number corresponding to the five second numbers, the aspect ratios of the eyes (EAR) and mouth (MAR) of the imageCount frame image are extracted again and stored in the arrays EYE_EAR and MOUTH_MAR, respectively. The parameter ThresholdCount is then set to 2, 3, 4, 5, and 6 in sequence. The parameter Threshold is set to True at the frame number corresponding to the (imageCount+1)th frame and the frame number corresponding to the five randomly generated second numbers.

[0031] Step 4.2: When the Threshold parameter is True, the Roll parameter is incremented by 1 for each image frame loaded, and the array EYE_EAR stored in Step 4.1 is sorted in descending order, and the array MOUTH_MAR is sorted in ascending order. The length of both arrays EYE_EAR and MOUTH_MAR is imageCount * ThresholdCount.

[0032] Step 4.3: To prevent malicious driver manipulation that could lead to inaccurate data, and to calculate the threshold using only data with larger eye and smaller mouth sizes, the first 10% and last 50% of the data in the imageCount*ThresholdCount frame data are removed, and only 40% of the data in the middle region is used for calculation. The first 10% and last 50% of the data in the sorted arrays EYE_EAR and MOUTH_MAR from Step 4.2 are then removed and stored in the eye aspect ratio array ERA and the mouth aspect ratio array MARA.

[0033] Step 4.4: Based on the eye length-to-width ratio array ERA and the mouth length-to-width ratio array MARA obtained in Step 4.3, calculate the eye and mouth length-to-width ratio thresholds. The specific mathematical formulas for the calculation are as follows (3)(4):

[0034] EART=H mean (EARA)*0.5 (3)

[0035] MART=H mean (MARA)*2 (4)

[0036] In the formula, EART (Eye aspect ratio threshold) represents the eye aspect ratio threshold, MART (Mouthaspect ratio threshold) represents the mouth aspect ratio threshold, and H... mean () represents the operation of calculating the mean, EARA represents the eye length-to-width ratio array, and MARA represents the mouth length-to-width ratio array.

[0037] The specific process of step 5 is as follows:

[0038] Create an integer parameter `Counter` for the number of times eyes are closed, an integer parameter `RollEye` for the number of blinks in the current time period, and an integer parameter `Total` for the total number of blinks; create an integer parameter `mCounter` for the number of times mouths are opened, an integer parameter `RollMouth` for the number of yawns in the current time period, and an integer parameter `mTotal` for the total number of yawns. Based on the eye aspect ratio thresholds `EART` and `MART` obtained in step 4, and the eye aspect ratios `EAR`, `MAR`, and label information, confidence level, and location information (outline) obtained in steps 2 and 3, determine the current facial movement state.

[0039] When the eye aspect ratio EAR is less than the eye aspect ratio threshold EART obtained in step 4, and driver tag information is detected in the current image frame in step 3, the parameter Counter is incremented by 1, and the parameter RollEye is incremented by 1. When the parameter Counter>=2, it means that one blink activity has been completed, the parameter Total is incremented by 1, and the parameter Counter is reset.

[0040] When the mouth length-to-width ratio is greater than the driver's mouth length-to-width ratio threshold obtained in step 4, parameter mCounter is incremented by 1, and parameter RollMouth is incremented by 1. Since a yawn lasts for 2-3 seconds, to prevent the driver's speech from being mistaken for a yawn, when parameter mCounter>=3, it means that a yawn activity has been completed, parameter mTotal is incremented by 1, and parameter mCounter is reset.

[0041] The specific process of step 6 is as follows:

[0042] The fatigue state of the current time period is judged based on the blink frame count parameter RollEye and yawn frame count parameter RollMouth obtained in step 5. In this experiment, each time period is set as imageCount frame image. When the parameter Roll = imageCount, the PERCLOS (Percentage of EyeClosure) value of the imageCount frame image of the current time period is calculated. The mathematical formula for calculating PERCLOS is as follows (5):

[0043]

[0044] In the formula, N eye N represents the number of frames with eyes closed in the current time period, which is the RollEye parameter counted in step 5. mouth N represents the number of frames in the current time period when the mouth is open, which is the RollMouth parameter calculated in step 5. total This represents the total number of frames in the current time period; in the current experiment, it is the imageCount frame.

[0045] When PERCLOS is greater than a certain value M, it indicates that the current state is fatigued. In this experiment, M is set to 38%. If it is less than M, it indicates that the current state is awake. After the judgment is completed, the frame count parameter Roll, blink frame count parameter RollEye, and mouth opening frame count parameter RollMouth of the time period are reset. Then, step 1 is repeated to judge the fatigue state of the next time period.

[0046] The beneficial effects of this invention are:

[0047] Compared to traditional facial motion-based fatigue driving detection methods, this invention proposes an adaptive facial motion feature threshold. By extracting image frames from six different time periods to calculate the threshold, the rationality and reliability of the threshold are ensured. Furthermore, after each image frame extraction, the latest calculated threshold is used to determine the fatigue state, making fatigue driving detection more accurate. Traditional facial motion-based fatigue driving detection methods use fixed eye-mouth aspect ratio thresholds to determine the state. However, the normal aspect ratios of the eyes and mouth vary from person to person, making a uniform threshold unsuitable and reducing reliability and robustness. This invention adaptively calculates the current driver's eye-mouth aspect ratio threshold and compares it with the current ratio to determine the open-eye and open-mouth states. The PERCLOS value is then calculated based on the number of blinks and yawns. This method provides a more accurate PERCLOS value and more precise fatigue state assessment. Attached Figure Description

[0048] Figure 1 This is a framework diagram of a fatigue driving detection method based on adaptive facial motion feature thresholds according to the present invention.

[0049] Figure 2 This is a schematic diagram of key eye points in a fatigue driving detection method based on adaptive facial motion feature thresholds according to the present invention.

[0050] Figure 3 This is a schematic diagram of the key points of the mouth in a fatigue driving detection method based on adaptive facial motion feature thresholds according to the present invention.

[0051] Figure 4 This is a network structure diagram of a fatigue driving detection method based on adaptive facial action feature thresholds according to the present invention. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0053] This invention relates to a fatigue driving detection method based on adaptive facial motion feature thresholds, such as... Figure 1 As shown, the specific steps include the following:

[0054] Step 1: Acquire real-time video of the driver's driving process and load the video stream into image frames;

[0055] The specific process of step 1 is as follows:

[0056] Step 1.1: Obtain the video source using OpenCV's video acquisition functions;

[0057] Step 1.2: After obtaining the video source of the driver's driving process in Step 1.1, set the timer period, connect the slot function of the timer period overflow, and use it to display one frame of video. After loading it into an image frame, read each frame of the video.

[0058] Step 2: Use the YOLO V5 network to identify the label information, confidence level, and location information (outer frame) in the image frame loaded in Step 1. Use the label information to determine whether the driver has sat in the driver's seat. If the driver has not sat in the driver's seat, continue to execute Step 1 until the driver starts to sit in the driver's seat before executing Step 3.

[0059] The specific implementation method for step 2 is as follows:

[0060] Step 2.1: Keep the original aspect ratio of the image frame loaded in Step 1, and fill the remaining part with gray to obtain a new image frame;

[0061] Step 2.2 converts the non-contiguous memory-stored arrays in the new image frames obtained in Step 2.1 into contiguous memory-stored NumPy arrays, which makes the running speed faster;

[0062] Step 2.3: Convert the NumPy array processed in Step 2.2 into a PyTorch tensor, with both sharing memory;

[0063] Step 2.4: Process the tensor obtained in Step 2.3 using the YOLO V5 network, and extract the highest score among all scores as the final result; for example... Figure 4 As shown, the YOLO V5 network structure of this invention mainly consists of four parts: input, backbone, neck, and prediction. The input consists of a 608x608x3 image, which passes through various modules, including Focus, CBL, CSP1, CSP2, and SPP, ultimately outputting three feature layers. The backbone is used to abstract and extract image features. The neck then outputs features at different sizes, enabling better detection of targets of different sizes. Finally, the prediction part maps the final features.

[0064] Step 2.5: Extract label information, confidence score, and location information (outline) from the final result of Step 2.4;

[0065] Step 2.6: Determine whether the driver has sat in the driver's seat based on the tag information obtained in Step 2.5. If no tag information indicating that the driver has sat in the driver's seat is detected, continue to execute Step 1 until the tag information indicating that the driver has sat in the driver's seat is detected before executing Step 3.

[0066] Step 3: Read the image frames loaded in Step 1 in a loop, obtain the facial feature location information for each frame, and calculate the eye aspect ratio (EAR) and mouth aspect ratio (MAR).

[0067] The specific process of step 3 is as follows:

[0068] Step 3.1: Use the Dlib face landmark detector to process and obtain the facial landmark feature information in the image frame loaded in Step 1;

[0069] Step 3.2: Convert the facial key point feature information obtained in Step 3.1 into array format. Extract the coordinates of the left and right eyes and mouth based on the indices of facial landmarks such as the left and right eyes and mouth. Calculate the corresponding Euclidean distance using the coordinates, and obtain the aspect ratio of the left and right eyes and mouth using this Euclidean distance. The mathematical formulas for calculating the aspect ratio of the eyes and mouth are as follows: (1) and (2):

[0070]

[0071]

[0072] In the formula, EAR (eye aspect ratio) represents the length-to-width ratio of the eye, and p1, p2, p3, p4, p 5, p6 represents the coordinates of six key points around the eyes; MAR (mouth aspect ratio) represents the length-to-width ratio of the mouth, m1, m2, m3, m4, m 5, m6 represents the coordinates of six key points on the mouth. The specific locations are as follows: Figure 2-3 As shown.

[0073] Step 4: Based on the aspect ratio of the eye (EAR) and the aspect ratio of the mouth (MAR) obtained in Step 3, adaptively calculate the eye aspect ratio threshold (EART) and the mouth aspect ratio threshold (MART) at different time periods.

[0074] The specific process of step 4 is as follows:

[0075] Step 4.1: Create arrays EYE_EAR and MOUTH_MAR to store the aspect ratios of the eyes and mouth, respectively; create a threshold calculation sequence array ThresholdSort, an integer parameter ThresholdCount for the number of threshold calculations, and a boolean parameter Threshold for whether to calculate the threshold; create an array ERA for the eye aspect ratios and an array MARA for the mouth aspect ratios; create an integer parameter Roll for the current time frame count; create an integer parameter imageCount for the number of image frames; 1 second of images is 12 frames, and imageCount is 120; when the driver first sits in the driver's seat, set the ThresholdCount parameter to 1, and start extracting the image frames of the imageCount count. The aspect ratios of the eyes (EAR) and mouth (MAR) of each frame are stored in the arrays EYE_EAR and MOUTH_MAR, respectively. To ensure the rationality and reliability of video selection, five random numbers are generated from 50 seconds to 1800 seconds, and these five second numbers are sorted in ascending order and stored in the array ThresholdSort. When the video reaches the frame number corresponding to the five second numbers, the aspect ratios of the eyes (EAR) and mouth (MAR) of the image of frame ImageCount are extracted again and stored in the arrays EYE_EAR and MOUTH_MAR, respectively. The parameter ThresholdCount is then set to 2, 3, 4, 5, and 6 in sequence. The parameter Threshold is set to True at the frame number corresponding to the (imageCount+1)th frame (frame 121) and the frame number corresponding to the five randomly generated second numbers.

[0076] Step 4.2: When the Threshold parameter is True, the Roll parameter is incremented by 1 for each image frame loaded, and the array EYE_EAR stored in Step 4.1 is sorted in descending order, and the array MOUTH_MAR is sorted in ascending order. The lengths of both arrays EYE_EAR and MOUTH_MAR are imageCount * ThresholdCount (120 * ThresholdCount at this time).

[0077] Step 4.3: To prevent malicious driver manipulation that could lead to inaccurate data, and to calculate the threshold using only data with larger eye and smaller mouth areas, the first 10% and last 50% of the data are removed from the imageCount*ThresholdCount (which is 120*ThresholdCount) frame data, and only 40% of the data in the middle region is used for calculation. The first 10% and last 50% of the data are also removed from the sorted arrays EYE_EAR and MOUTH_MAR from Step 4.2, and then stored in the eye aspect ratio array ERA and the mouth aspect ratio array MARA.

[0078] Step 4.4: Based on the eye length-to-width ratio array ERA and the mouth length-to-width ratio array MARA obtained in Step 4.3, calculate the eye and mouth length-to-width ratio thresholds. The specific mathematical formulas for the calculation are as follows (3)(4):

[0079] EART=H mean (MARA)*0.5 (3)

[0080] MART=H mean (MARA)*2 (4)

[0081] In the formula, EART (Eye aspect ratio threshold) represents the eye aspect ratio threshold, MART (Mouthaspect ratio threshold) represents the mouth aspect ratio threshold, and H... mean () represents the operation of calculating the mean, EARA represents the eye length-to-width ratio array, and MARA represents the mouth length-to-width ratio array.

[0082] Step 5: Count the number of blinks and yawns based on the length-to-width ratio thresholds of the eyes and mouth obtained in Step 4.

[0083] The specific process of step 5 is as follows:

[0084] Create an integer parameter `Counter` for the number of times eyes are closed, an integer parameter `RollEye` for the number of blinks in the current time period, and an integer parameter `Total` for the total number of blinks; create an integer parameter `mCounter` for the number of times mouths are opened, an integer parameter `RollMouth` for the number of yawns in the current time period, and an integer parameter `mTotal` for the total number of yawns. Based on the eye aspect ratio thresholds `EART` and `MART` obtained in step 4, and the eye aspect ratios `EAR`, `MAR`, and label information, confidence level, and location information (outline) obtained in steps 2 and 3, determine the current facial movement state.

[0085] When the eye aspect ratio EAR is less than the eye aspect ratio threshold EART obtained in step 4, and driver tag information is detected in the current image frame in step 3, the parameter Counter is incremented by 1, and the parameter RollEye is incremented by 1. When the parameter Counter>=2, it means that one blink activity has been completed, the parameter Total is incremented by 1, and the parameter Counter is reset.

[0086] When the mouth length-to-width ratio is greater than the driver's mouth length-to-width ratio threshold obtained in step 4, parameter mCounter is incremented by 1, and parameter RollMouth is incremented by 1. Since a yawn lasts for 2-3 seconds, to prevent the driver's speech from being mistaken for a yawn, when parameter mCounter>=3, it means that a yawn activity has been completed, parameter mTotal is incremented by 1, and parameter mCounter is reset.

[0087] Step 6: Based on the number of blinks and yawns counted in Step 5, determine whether the current period is fatigued. Then, continue to repeat Step 1 to determine the fatigue status for the next period.

[0088] The specific process of step 6 is as follows:

[0089] The fatigue state of the current time period is judged based on the blink frame count parameter RollEye and yawn frame count parameter RollMouth obtained in step 5. In this experiment, each time period is set as imageCount frames (120 frames are selected at this time). When the parameter Roll = imageCount (Roll = 120), the eye closure rate PERCLOS (Percentage of Eye Closure) value of the imageCount frames (120 frames are selected at this time) of the current time period is calculated. The mathematical formula for calculating PERCLOS is as follows (5):

[0090]

[0091] In the formula, N eye N represents the number of frames with eyes closed in the current time period, which is the RollEye parameter counted in step 5. mouth N represents the number of frames in the current time period when the mouth is open, which is the RollMouth parameter calculated in step 5. total This represents the total number of frames in the current time period, which is imageCount frames in the current experiment (120 frames in the current experiment);

[0092] When PERCLOS is greater than a certain value M, it indicates that the current state is fatigued. In this experiment, M is set to 38%. If it is less than M, it indicates that the current state is awake. After the judgment is completed, the frame count parameter Roll, blink frame count parameter RollEye, and mouth opening frame count parameter RollMouth of the time period are reset. Then, step 1 is repeated to judge the fatigue state of the next time period.

Claims

1. A fatigue driving detection method based on adaptive facial motion feature thresholds, characterized in that: Specifically, the steps include the following: Step 1: Acquire real-time video of the driver's driving process and load the video stream into image frames; Step 2: Use the YOLO V5 network to identify the label information, confidence level and location information in the image frame loaded in Step 1, and use the label information to determine whether the driver has sat in the driver's seat. If the driver has not sat in the driver's seat, continue to execute Step 1 until the driver starts to sit in the driver's seat before executing Step 3. Step 3: Read the image frames loaded in Step 1 in a loop, obtain the facial feature location information for each frame, and calculate the eye aspect ratio EAR and mouth aspect ratio MAR. Step 4: Adaptively calculate the eye length-to-width ratio thresholds EART and MART based on the eye length-to-width ratio EAR and mouth length-to-width ratio MAR obtained in Step 3 at different time periods. The specific process of step 4 is as follows: Step 4.1: Create arrays EYE_EAR and MOUTH_MAR to store the aspect ratios of the eyes and mouth, respectively; create a threshold calculation sequence array ThresholdSort, an integer parameter ThresholdCount for the number of threshold calculations, and a boolean parameter Threshold for whether to calculate the threshold; create an array ERA for the eye aspect ratios and an array MARA for the mouth aspect ratios; create an integer parameter Roll for the current time frame count; create an integer parameter imageCount for the number of image frames; 1 second of image is 12 frames, and imageCount ranges from 120 to 180; when the driver first sits in the driver's seat, set the ThresholdCount parameter to 1 and begin extracting the image frames of the imageCount count. The aspect ratios of the eyes (EAR) and mouth (MAR) of the Count frame image are stored in the arrays EYE_EAR and MOUTH_MAR, respectively. To ensure the rationality and reliability of video selection, five random numbers are generated from 50 seconds to 1800 seconds, and these five second numbers are sorted in ascending order and stored in the array ThresholdSort. When the video reaches the frame number corresponding to the five second numbers, the aspect ratios of the eyes (EAR) and mouth (MAR) of the imageCount frame image are extracted again and stored in the arrays EYE_EAR and MOUTH_MAR, respectively. The parameter ThresholdCount is then set to 2, 3, 4, 5, and 6 in sequence. The parameter Threshold is set to True at the frame number corresponding to the (imageCount+1)th frame and the frame number corresponding to the five randomly generated second numbers. Step 4.2: When the Threshold parameter is True, the Roll parameter is incremented by 1 for each image frame loaded, and the array EYE_EAR stored in Step 4.1 is sorted in descending order, and the array MOUTH_MAR is sorted in ascending order. The length of both arrays EYE_EAR and MOUTH_MAR is imageCount * ThresholdCount. Step 4.3: To prevent malicious driver manipulation that could lead to inaccurate data, and to calculate the threshold using only data with larger eye and smaller mouth sizes, the first 10% and last 50% of the data in the imageCount*ThresholdCount frame data are removed, and only 40% of the data in the middle region is used for calculation. The first 10% and last 50% of the data in the sorted arrays EYE_EAR and MOUTH_MAR from Step 4.2 are then removed and stored in the eye aspect ratio array EARA and the mouth aspect ratio array MARA. Step 4.4: Based on the eye length-to-width ratio array ERA and the mouth length-to-width ratio array MARA obtained in Step 4.3, calculate the eye and mouth length-to-width ratio thresholds. The specific mathematical formulas for the calculation are as follows (3)(4): EARTH=H mean (EAR)*0.5 (3) MART=H mean (MARA)*2 (4) In the formula, EART represents the eye length-to-width ratio threshold, MART represents the mouth length-to-width ratio threshold, and H... mean () represents the operation of calculating the mean, EARA represents the eye length-to-width ratio array, and MARA represents the mouth length-to-width ratio array. Step 5: Count the number of blinks and yawns based on the length-to-width ratio thresholds of the eyes and mouth obtained in Step 4. Step 6: Based on the number of blinks and yawns counted in Step 5, determine whether the current period is fatigued. Then, continue to repeat Step 1 to determine the fatigue status for the next period.

2. The fatigue driving detection method based on adaptive facial motion feature threshold according to claim 1, characterized in that: The specific process of step 1 is as follows: Step 1.1: Obtain the video source using OpenCV's video acquisition functions; Step 1.2: After obtaining the video source of the driver's driving process in Step 1.1, set the timer period, connect the slot function of the timer period overflow, and use it to display one frame of video. After loading it into an image frame, read each frame of the video.

3. The fatigue driving detection method based on adaptive facial motion feature threshold according to claim 2, characterized in that: The specific implementation method of step 2 is as follows: Step 2.1: Keep the original aspect ratio of the image frame loaded in Step 1, and fill the remaining part with gray to obtain a new image frame; Step 2.2: Convert the arrays stored in non-contiguous memory in the new image frames obtained in Step 2.1 into NumPy arrays stored in contiguous memory. Step 2.3: Convert the NumPy array processed in Step 2.2 into a PyTorch tensor, with both sharing memory; Step 2.4: Process the tensor obtained in Step 2.3 through the YOLO V5 network, and extract the highest score among all scores as the final result. Step 2.5: Extract label information, confidence score, and location information from the final result of Step 2.4; Step 2.6: Determine whether the driver has sat in the driver's seat based on the tag information obtained in Step 2.

5. If no tag information indicating that the driver has sat in the driver's seat is detected, continue to execute Step 1 until the tag information indicating that the driver has sat in the driver's seat is detected before executing Step 3.

4. The fatigue driving detection method based on adaptive facial motion feature threshold according to claim 3, characterized in that: The specific process of step 3 is as follows: Step 3.1: Use the Dlib face landmark detector to process and obtain the facial landmark feature information in the image frame loaded in Step 1; Step 3.2: Convert the facial key point feature information obtained in Step 3.1 into array format. Extract the coordinates of the left and right eyes and mouth based on the indices of the facial landmarks of the left and right eyes and mouth. Calculate the corresponding Euclidean distance using the coordinates, and obtain the aspect ratio of the left and right eyes and mouth using this Euclidean distance. The mathematical formulas for calculating the aspect ratio of the eyes and mouth are as follows: (1) and (2): In the formula, EAR represents the length-to-width ratio of the eye, and p1, p2, p3, p4, p 5, p6 represents the coordinates of six key points around the eyes; MAR represents the aspect ratio of the mouth, m1, m2, m3, m4, m 5, m6 represents the coordinates of the six key points on the mouth.

5. The fatigue driving detection method based on adaptive facial motion feature threshold according to claim 1, characterized in that: The specific process of step 5 is as follows: Create an integer parameter for the number of times eyes are closed (Counter), an integer parameter for the number of blinks in the current time period (RollEye), and an integer parameter for the total number of blinks (Total); create an integer parameter for the number of times mouths are opened (mCounter), an integer parameter for the number of yawns in the current time period (RollMouth), and an integer parameter for the total number of yawns (mTotal); based on the eye aspect ratio thresholds EART and MART obtained in step 4, and the eye aspect ratios EAR and MAR, mouth aspect ratios MAR, label information, confidence level, and location information obtained in steps 2 and 3, determine the current facial movement state; When the eye aspect ratio EAR is less than the eye aspect ratio threshold EART obtained in step 4 and driver tag information is detected in the current image frame in step 3, the parameter Counter is increased by 1 and the parameter RollEye is increased by 1; when the parameter Counter>=2, it means that one blinking activity has been completed, the parameter Total is increased by 1, and the parameter Counter is reset. When the mouth length-to-width ratio is greater than the driver's mouth length-to-width ratio threshold obtained in step 4, parameter mCounter is increased by 1 and parameter RollMouth is increased by 1. To prevent the driver's speech from being misjudged as yawning, when parameter mCounter>=3, it means that a yawning activity has been completed, parameter mTotal is increased by 1, and parameter mCounter is reset.

6. The fatigue driving detection method based on adaptive facial motion feature threshold according to claim 5, characterized in that: The specific process of step 6 is as follows: The fatigue state in the current time period is determined based on the blink frame count parameter RollEye and yawn frame count parameter RollMouth obtained in step 5. Each time period is set as imageCount frames. When the parameter Roll = imageCount, the PERCLOS value of the closed eye rate of the imageCount frames in the current time period is calculated. The mathematical formula for calculating PERCLOS is as follows (5): In the formula, N eye N represents the number of frames with eyes closed in the current time period, which is the RollEye parameter counted in step 5. mouth N represents the number of frames in the current time period when the mouth is open, which is the RollMouth parameter calculated in step 5. total This represents the total number of frames in the current time period; in the current experiment, it is the imageCount frame. When PERCLOS is greater than a certain value M, it indicates that the person is currently in a state of fatigue. If it is less than M, it means that the person is currently awake. After the judgment is completed, the frame count parameter Roll, blink frame count parameter RollEye, and mouth opening frame count parameter RollMouth of the period are reset. Then, step 1 is repeated to judge the fatigue state of the next period.

Images