Fatigue driving real-time monitoring method based on deep learning and computer vision

Abstract

The invention discloses a fatigue driving real-time monitoring method based on deep learning and computer vision, and the method comprises the steps: capturing a video from a camera through employing OpenCV, reading each frame from the captured video stream, and processing the image frames; recognizing a human face from the real-time video stream and extracting key feature points of the human face; the eye closing degree is evaluated by calculating the geometric proportion of the eyes, and the mouth opening degree is evaluated by calculating the geometric proportion of the mouth; acquiring a head posture from the real-time video stream and calculating a posture angle; whether the eye closing state continues for a period of time, whether the mouth opening state continues for a period of time and whether the head lowering state continues for a period of time are judged, so that whether the user is in fatigue driving or not is recognized; and displaying information of the picture frame rate, the facial feature points, the geometric proportion and the head posture angle in real time, displaying a fatigue warning when a fatigue alarm is triggered, receiving a data updating request of a user, and updating the data in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a real-time monitoring method for fatigue driving based on deep learning and computer vision. Background Art

[0002] With the development of modern transportation, driver fatigue driving has become one of the main causes of traffic accidents. In order to improve road safety and reduce accidents caused by fatigue driving, driver fatigue status monitoring technology has emerged. Fatigue monitoring technology mainly analyzes the driver's physiological signals, behavioral characteristics and vehicle dynamic data to evaluate the driver's fatigue level in real time and issue a warning when necessary.

[0003] Driver fatigue status monitoring systems are mainly divided into two methods: direct monitoring and indirect monitoring. Direct monitoring evaluates the fatigue status by analyzing the driver's facial, eye and head movements and other physiological signals, including blinking frequency, eye closure time, head posture, etc. Indirect monitoring infers the driver's fatigue level by analyzing the driver's operating behavior and vehicle status, such as steering wheel use, vehicle speed, acceleration, etc.

[0004] Internationally, Mercedes-Benz's Attention Assist system, Toyota's Driver Monitor, Ford's Driver Alert System, etc. are all representatives of fatigue monitoring technology. These systems use existing vehicle sensors or additional cameras and infrared sensors, combined with advanced algorithms, to achieve real-time monitoring of driver fatigue.

[0005] Domestic research started late but has developed rapidly. Tsinghua University, Tongji University and other universities have achieved remarkable results in this field, and some research results have been commercialized. For example, Tsinghua University's fatigue driving warning system analyzes the driver's facial information to determine the fatigue state, and has been applied to Zotye M11, M12 and Shaanxi Automobile heavy trucks.

[0006] Although existing fatigue monitoring technologies have made some progress in improving road safety, they still have some significant shortcomings and challenges:

[0007] (1) Poor environmental adaptability: Vision-based fatigue monitoring systems perform poorly in strong or weak light environments, and it is difficult to ensure that visual signals that meet quality requirements can be continuously obtained under various lighting conditions.

[0008] (2) Balance between real-time performance and accuracy: Although deep learning technology provides high-precision monitoring, its algorithm takes up a lot of memory when running, which may lead to poor real-time performance and difficulty in integration into the controller of the vehicle's existing electronic control system.

[0009] (3) Hardware cost and deployment complexity: Sensor-based physiological signal monitoring systems often require wearable devices, which are inconvenient to use and may affect the natural behavior of individuals. In addition, some high-precision monitoring systems require the installation of additional hardware such as cameras and infrared sensors, which increases vehicle costs and the complexity of system deployment.

[0010] (4) Data privacy and security issues: With the development of fatigue monitoring technology, which involves the collection of driver physiological and behavioral data, how to ensure data security and user privacy has become an important issue.

[0011] (5) Ignoring individual differences: Different drivers have different fatigue manifestations. Existing technologies may not fully consider individual differences, resulting in the lack of universality of monitoring results. Summary of the invention

[0012] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a real-time monitoring method for fatigue driving based on deep learning and computer vision, which is a more accurate, reliable and user-friendly fatigue monitoring solution, and can significantly improve road safety and the driver's driving experience.

[0013] The technical solution of the present invention is: the real-time monitoring method of fatigue driving based on deep learning and computer vision comprises the following steps:

[0014] (1) Use OpenCV to capture video from the camera, read each frame from the captured video stream, and process the image frame;

[0015] (2) Recognize faces from real-time video streams and extract key facial features;

[0016] (3) The degree of eye closure is assessed by calculating the geometric proportions of the eyes, and the degree of mouth openness is assessed by calculating the geometric proportions of the mouth;

[0017] (4) Obtain head posture from real-time video stream and calculate posture angle;

[0018] (5) By determining whether the eyes are closed for a period of time, the mouth is open.

[0019] To determine whether the head-down state lasts for a period of time, and to identify whether the user is driving fatigued;

[0020] (6) Displaying the frame rate, facial feature points, geometric proportions, and head posture angles in real time; displaying a fatigue warning when a fatigue alarm is triggered; and updating the data in real time upon receiving a user's request for updated data;

[0021] Wherein said step (2) comprises the following sub-steps:

[0022] (2.1) Use the detector function in the DLib deep learning model, which is based on

[0023] HOG features plus linear classifier detect the face area in the image;

[0024] (2.2) In the detected face area, use the predictor function in the DLib library to extract 68 key feature points of the face.

[0025] The beneficial technical effects of the present invention are as follows:

[0026] 1. High accuracy: By integrating multiple facial features and head posture indicators, the accuracy of fatigue status judgment is improved.

[0027] 2. Real-time: The system can monitor and analyze the individual's fatigue status in real time and issue warnings in time.

[0028] 3. Non-invasive: No need to wear any equipment, and it will not affect the individual's natural behavior.

[0029] 4. Environmental adaptability: The system can operate stably under a variety of environmental conditions and has strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flow chart of a real-time monitoring method for fatigue driving based on deep learning and computer vision according to the present invention is shown.

[0031] Figure 2 A schematic diagram of a real-time monitoring method for fatigue driving based on deep learning and computer vision according to the present invention is shown.

[0032] Figure 3 The feature point index of the eye is shown.

[0033] Figure 4 The eye aspect ratio is shown over time.

[0034] Figure 5 The feature point index of the mouth is shown. DETAILED DESCRIPTION

[0035] like Figure 1 , 2 As shown, this real-time monitoring method for fatigue driving based on deep learning and computer vision includes the following steps:

[0036] (1) Use OpenCV to capture video from the camera and read each

[0037] One frame, and process the image frame;

[0038] (2) Recognize faces from real-time video streams and extract key facial features;

[0039] (3) The degree of eye closure is assessed by calculating the geometric proportions of the eyes, and the degree of mouth openness is assessed by calculating the geometric proportions of the mouth;

[0040] (4) Obtain head posture from real-time video stream and calculate posture angle;

[0041] (5) Determine whether the mouth is open by determining whether the eyes are closed for a period of time

[0042] Whether it lasts for a period of time, to determine whether the head-down state lasts for a period of time, to identify

[0043] Whether the user is driving while fatigued;

[0044] (6) Real-time display of frame rate, facial feature points, geometric proportions, head posture angle information, display of fatigue warning when fatigue alarm is triggered, and receiving updated data from users

[0045] Request,real-time update of data;

[0046] Wherein said step (2) comprises the following sub-steps:

[0047] (2.1) Use the detector function in the DLib deep learning model, which is based on

[0048] HOG features plus linear classifier detect the face area in the image;

[0049] (2.2) In the detected face area, use the predictor function in the DLib library to extract 68 key feature points of the face.

[0050] The beneficial technical effects of the present invention are as follows:

[0051] 1. High accuracy: By integrating multiple facial features and head posture indicators, the accuracy of fatigue status judgment is improved.

[0052] 2. Real-time: The system can monitor and analyze the individual's fatigue status in real time and issue warnings in time.

[0053] 3. Non-invasive: No need to wear any equipment, and it will not affect the individual's natural behavior.

[0054] 4. Environmental adaptability: The system can operate stably under a variety of environmental conditions and has strong robustness.

[0055] Preferably, the step (1) comprises the following sub-steps:

[0056] (1.1) Flip the captured frame horizontally;

[0057] (1.2) Scale the frame to a width of 720 pixels;

[0058] (1.3) Convert the captured frames from BGR color space to grayscale space.

[0059] Preferably, in the step (2.1), a grayscale image is first extracted from the video frame captured by the camera; then, the grayscale image is input into the detector function of DLib to obtain a rectangular area of ​​the face.

[0060] Preferably, in the step (2.2), for each detected rectangular face area, the predictor function is used to obtain feature points; then the detection results of DLib are converted into a NumPy array for subsequent processing; the feature point indexes of specific parts are obtained, and then the feature points of these parts are extracted from the NumPy array, and for each feature point, a small circle is drawn.

[0061] Preferably, in step (3), the degree of eye closure is evaluated by using an ordered dictionary in the imutils library to associate the index of the feature point with the corresponding facial part, obtain the feature point of the specific part through the index, and obtain the feature point index of the eye of the specific part (such as Figure 3 As shown), then extract the feature points of these parts from the NumPy array; the eye aspect ratio EAR formula is:

[0062]

[0063] Among them, P1, P2, ... P6 are the coordinates of the feature points of the contour of a single eye, and ||*|| represents the Euclidean norm, which is used to calculate the distance between two vectors;

[0064] A sharp drop in the eye's aspect ratio indicates a blink.

[0065] Preferably, in step (3), the mouth opening degree is evaluated by using an ordered dictionary in the imutils library to associate the index of the feature point with the corresponding facial part, obtain the feature point of the specific part through the index, obtain the index of the feature point of the specific part of the mouth, and then extract the feature points of these parts from the NumPy array (such as Figure 5 Yawning detection based on MAR algorithm uses DLib to extract 6 feature points of the mouth, and the coordinates of these 6 feature points are used to calculate the degree of mouth opening when yawning. The MAR formula of the mouth aspect ratio is:

[0066]

[0067] Among them, Y i Represents the ordinate of the feature point, X i The horizontal coordinate represents the feature point. The subscripts 49, 51, 53, 55, 57, and 59 are the six mouth feature points in the mouth contour extracted by DLib.

[0068] Preferably, the step (4) comprises the following sub-steps:

[0069] (4.1) Face key point detection: Use the MediaPipe deep learning model to detect facial key points, including the corners of the eyes, the tip of the nose, and the corners of the mouth;

[0070] (4.2) 3D model matching, matching the detected 2D facial key points with the pre-defined 3D facial model, which provides the corresponding position of each key point in the head space;

[0071] (4.3) Solve the transformation relationship. Specifically, the posture of an object relative to the camera is represented by a rotation matrix and a translation matrix. The key points of the face in the image are projected onto the three-dimensional face model. The rotation and translation matrices from the 3D model points to the 2D image points are calculated using the stereo matching algorithm PnP. The Euler angle is solved based on the two-dimensional and three-dimensional coordinate transformation relationship matrices.

[0072] (4.4) Calculate the attitude angle. According to the rotation matrix, use the Euler angle transformation to calculate the pitch angle, yaw angle and roll angle of the head.

[0073] Preferably, the step (5) comprises the following sub-steps:

[0074] (5.1) Setting an eye closure threshold to determine whether the eyes are closed. If the vertical distance of the eyes is lower than the eye closure threshold, the eyes are closed. Setting an eye closure continuous frame number threshold. If the continuous frame number exceeds the eye closure continuous frame number threshold, it means that the eyes have been closed for a period of time. If both of the above conditions are met, the user is identified as fatigued driving and a warning is displayed.

[0075] (5.2) Setting a mouth opening threshold to determine whether the mouth is open. If the vertical distance of the mouth is higher than the mouth opening threshold, it indicates that the mouth is open. Setting a mouth opening continuous frame number threshold. If the number of continuous frames exceeds the mouth opening continuous frame number threshold, it indicates that the mouth is open for a period of time, that is, a yawn. Setting a yawn frequency threshold. If the number of yawns reaches the set threshold, the user is identified as fatigued driving and a warning is displayed.

[0076] (5.3) Set the head lowering threshold to judge the head state. If the head pitch angle is greater than the head lowering threshold, it indicates the head lowering state. Set the head lowering continuous frame number threshold. If the continuous frame number exceeds the head lowering continuous frame number threshold,

[0077] Indicates that the head-down state lasts for a period of time; if the above two conditions are met at the same time, the user is identified as fatigued driving and a warning appears;

[0078] (5.4) When the warning appears for a certain period of time, the cycle ends, all values ​​are reset to zero, and fatigue monitoring starts again.

[0079] Preferably, in step (5), the eye closure threshold is 0.2, the eye closure continuous frame number threshold is 40, the mouth opening threshold is 0.5, the mouth opening continuous frame number threshold is 30, the head lowering threshold is 15, and the head lowering continuous frame number threshold is 60.

[0080] In order to verify the technical effect of the present invention, a series of program experimental simulations were carried out. The experimental results are as follows:

[0081] Run the program in non-debug mode.

[0082] When the vertical distance ratio of the subject's eyes is lower than 0.2 and the number of consecutive frames exceeds 40, a warning appears and the screen displays "SLEEP!!!".

[0083] When the vertical distance ratio of the test subject's mouth is higher than 0.5 and the number of consecutive frames exceeds 30, the system will recognize the user as yawning.

[0084] If the user yawns three times, the system will identify the user as fatigued driving and issue a warning. The screen will display "SLEEP!!!", then all values ​​will be cleared and fatigue monitoring will be restarted.

[0085] When the test subject's head pitch angle is greater than 15 degrees and the number of consecutive frames exceeds 60, the system will identify the user as fatigued driving and a warning will appear, with the screen displaying "SLEEP!!!".

[0086] The experimental results show that:

[0087] The average response time of the system is less than 1 second, and it can realize the function of real-time warning of fatigue monitoring through eyes, mouth, and head posture, which can meet the needs of real-time monitoring. The invention effectively improves the accuracy and real-time performance of fatigue monitoring by adopting advanced algorithm models and technical solutions, while providing a user-friendly interactive interface, which has significant technical advantages and practical value. Experimental simulation verification analysis further proves the potential of the present invention in improving driving safety.

[0088] With the continuous advancement of technology, fatigue monitoring systems will become more accurate and efficient, and can provide stronger protection for traffic safety. Through the following implementation cases, it can be seen that the application of fatigue monitoring technology in different scenarios can help improve driving safety, reduce traffic accidents caused by fatigue driving, and have significant social and economic benefits.

[0089] The present invention has a wide range of practical application scenarios and can be applied to a variety of environments where driver fatigue status needs to be monitored to improve safety and reduce the risk of accidents. The following are some specific embodiments that demonstrate the effects and functions that this solution can bring.

[0090] Example 1 Public Transportation

[0091] On buses, taxis or school buses, fatigue monitoring systems can ensure the safety of passengers and drivers. The system determines whether the driver is tired by analyzing the driver's facial expressions and behaviors, such as yawning and frequent blinking. Once signs of fatigue are detected, the system can alert the driver through sound or vibration and notify the dispatch center to take appropriate measures.

[0092] Example 2 Intelligent Transportation System

[0093] In intelligent traffic management systems, fatigue monitoring technology can be combined with traffic monitoring cameras to analyze the behavior of road users in real time. This can not only be used to remind individual drivers, but also for traffic flow management and accident prevention strategy formulation.

[0094] Example 3 Special Operation Vehicle

[0095] Fatigue monitoring of engineering vehicle drivers is particularly important in construction sites, mining areas or other special working environments. By installing a fatigue monitoring system on these vehicles, the driver's status can be monitored in real time. For example, when the system detects that the driver's blinking frequency is abnormal or the eyes are closed for too long, it will immediately issue a warning to remind the driver to take a rest or the monitoring center will intervene to guide the driver to stop the vehicle safely to protect the safety of workers and equipment.

[0096] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A real-time monitoring method for fatigue driving based on deep learning and computer vision, characterized by: It includes the following steps: (1) Use OpenCV to capture video from the camera, read each frame from the captured video stream, and process the image frame; (2) Recognize faces from real-time video streams and extract key facial features; (3) The degree of eye closure is assessed by calculating the geometric proportions of the eyes, and the degree of mouth openness is assessed by calculating the geometric proportions of the mouth; (4) Obtain head posture from real-time video stream and calculate posture angle; (5) Identify whether the user is driving fatigued by determining whether the eyes are closed for a period of time, whether the mouth is open for a period of time, and whether the head is lowered for a period of time; (6) Displaying the frame rate, facial feature points, geometric proportions, and head posture angles in real time; displaying a fatigue warning when a fatigue alarm is triggered; and updating the data in real time upon receiving a user's request for updated data; Wherein said step (2) comprises the following sub-steps: (2.1) Use the detector function in the DLib deep learning model, which detects the face area in the image based on the HOG feature plus a linear classifier; (2.2) In the detected face area, use the predictor function in the DLib library to extract 68 key feature points of the face.

2. The real-time monitoring method for fatigue driving based on deep learning and computer vision according to claim 1 is characterized in that: The step (1) comprises the following sub-steps: (1.1) Flip the captured frame horizontally; (1.2) Scale the frame to a width of 720 pixels; (1.3) Convert the captured frames from BGR color space to grayscale space.

3. The real-time monitoring method for fatigue driving based on deep learning and computer vision according to claim 2 is characterized in that: In the step (2.1), a grayscale image is first extracted from the video frame captured by the camera; then, the grayscale image is input into the detector function of DLib to obtain a rectangular area of ​​the face.

4. The real-time monitoring method for fatigue driving based on deep learning and computer vision according to claim 3 is characterized in that: In the step (2.2), for each detected rectangular face area, the predictor function is used to obtain feature points; then the detection result of DLib is converted into a NumPy array for subsequent processing; the feature point index of a specific part is obtained, and then the feature points of these parts are extracted from the NumPy array, and for each feature point, a small circle is drawn.

5. The real-time monitoring method for fatigue driving based on deep learning and computer vision according to claim 4 is characterized in that: In step (3), the degree of eye closure is assessed: Using an ordered dictionary in the imutils library, associate the index of the feature points with their corresponding facial parts, obtain the feature points of a specific part by index, obtain the index of the feature points of the eyes in a specific part, and then extract the feature points of these parts from the NumPy array; The eye aspect ratio EAR formula is: Among them, P1, P2, ... P6 are the coordinates of the feature points of the contour of a single eye, and ||*|| represents the Euclidean norm, which is used to calculate the distance between two vectors; A sharp drop in the eye's aspect ratio indicates a blink.

6. The real-time monitoring method for fatigue driving based on deep learning and computer vision according to claim 5 is characterized in that: In step (3), the degree of mouth opening is evaluated: Use an ordered dictionary in the imutils library to associate the index of the feature point with its corresponding facial part, obtain the feature point of a specific part by index, obtain the index of the feature point of the mouth in a specific part, and then extract the feature points of these parts from the NumPy array; Yawn detection based on the MAR algorithm uses DLib to extract the six feature points of the mouth. The coordinates of these six feature points are used to calculate the degree of mouth opening when yawning. The MAR formula for the aspect ratio of the mouth is: Among them, Y i Represents the ordinate of the feature point, X i The horizontal coordinate represents the feature point. The subscripts 49, 51, 53, 55, 57, and 59 are the six mouth feature points in the mouth contour extracted by DLib.

7. The real-time monitoring method for fatigue driving based on deep learning and computer vision according to claim 4 is characterized in that: The step (4) comprises the following sub-steps: (4.1) Face key point detection: Use the MediaPipe deep learning model to detect facial key points, including the corners of the eyes, the tip of the nose, and the corners of the mouth; (4.2) 3D model matching, matching the detected 2D facial key points with the pre-defined 3D facial model, which provides the corresponding position of each key point in the head space; (4.3) Solve the transformation relationship. Specifically, the posture of an object relative to the camera is represented by a rotation matrix and a translation matrix. The key points of the face in the image are projected onto the three-dimensional face model. The rotation and translation matrices from the 3D model points to the 2D image points are calculated using the stereo matching algorithm PnP. The Euler angle is solved based on the two-dimensional and three-dimensional coordinate transformation relationship matrices. (4.4) Calculate the attitude angle. According to the rotation matrix, use the Euler angle transformation to calculate the pitch angle, yaw angle and roll angle of the head.

8. The real-time monitoring method for fatigue driving based on deep learning and computer vision according to claim 4 is characterized in that: The step (5) comprises the following sub-steps: (5.1) Setting an eye closure threshold to determine whether the eyes are closed. If the vertical distance of the eyes is lower than the eye closure threshold, the eyes are closed. Setting an eye closure continuous frame number threshold. If the continuous frame number exceeds the eye closure continuous frame number threshold, it means that the eyes have been closed for a period of time. If both of the above conditions are met, the user is identified as fatigued driving and a warning is displayed. (5.2) Setting a mouth opening threshold to determine whether the mouth is open. If the vertical distance of the mouth is higher than the mouth opening threshold, it indicates that the mouth is open. Setting a mouth opening continuous frame number threshold. If the number of continuous frames exceeds the mouth opening continuous frame number threshold, it indicates that the mouth is open for a period of time, that is, a yawn. Setting a yawn frequency threshold. If the number of yawns reaches the set threshold, the user is identified as fatigued driving and a warning is displayed. (5.3) Set the head lowering threshold to judge the head state. If the head pitch angle is greater than the head lowering threshold, it indicates the head lowering state. Set the head lowering continuous frame number threshold. If the continuous frame number exceeds the head lowering continuous frame number threshold, Indicates that the state of bowing the head lasts for a period of time; if both of the above conditions are met at the same time If the device is not working properly, the user will be identified as fatigued driving and a warning will appear; (5.4) When the warning appears for a certain period of time, the cycle ends, all values ​​are reset to zero, and fatigue monitoring starts again.

9. The real-time monitoring method for fatigue driving based on deep learning and computer vision according to claim 5 is characterized in that: In step (5), the eye closure threshold is 0.2, the eye closure continuous frame number threshold is 40, the mouth opening threshold is 0.5, the mouth opening continuous frame number threshold is 30, the head lowering threshold is 15, and the head lowering continuous frame number threshold is 60.

Images