Driver state detection system and driver monitoring system
The driver state detection system, which integrates facial recognition, facial state detection, error detection, and head posture recognition modules, solves the problem of inaccurate detection caused by individual differences among drivers, and achieves higher accuracy and safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN CSR TIMES ELECTRIC VEHICLE
- Filing Date
- 2024-10-30
- Publication Date
- 2026-05-01
AI Technical Summary
Existing driver monitoring systems have low accuracy in facial detection when dealing with individual differences among drivers, resulting in a high false alarm rate, which poses a safety hazard, especially in commercial vehicle environments.
The system employs a face recognition module, a face state detection module, a state error detection module, a face feature correction module, a driver posture recognition module, and a driving state determination module. It improves the accuracy of face state detection through error detection and feature correction, and makes a comprehensive judgment by combining the driver's head posture recognition results.
It improves the accuracy of driver status detection, overcomes the inaccuracy caused by facial differences between different drivers, captures changes in the driver's head posture, reduces false alarm rate, and improves safety.
Smart Images
Figure CN121963154A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, specifically to a driver state detection system and a driver monitoring system. Background Technology
[0002] Globally, a large number of traffic accidents each year are caused by driver fatigue, distraction, or physical discomfort. Real-time monitoring of driver status can effectively reduce the occurrence of such accidents. Market demand and regulations in various countries have driven the rapid development of driver monitoring systems (DMS).
[0003] Based on overall needs and costs, there are currently various DMS system solutions on the market. However, most solutions use fixed facial recognition models for facial state detection, failing to consider the interference of individual differences between drivers on facial state detection. This results in a high false alarm rate and low accuracy in detecting driver state under different drivers and environments, posing safety hazards. Compared to passenger vehicles, commercial vehicles have higher complexity in vehicle management, driving environment, and operation, and longer driving time, requiring more accurate detection of driver state. Relying solely on facial state detection is insufficient to identify situations where drivers are distracted (such as turning their heads or making phone calls), posing traffic safety hazards. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to provide a driver status detection system and a driver monitoring system to improve the accuracy of driver status detection.
[0005] In a first aspect, the present invention provides a driver state detection system, which includes a face recognition module, a face state detection module, and a face tracking module. In addition, the driver state detection system also includes a state error detection module, a facial feature correction module, a driver posture recognition module, and a driving state determination module.
[0006] The state error detection module is used to acquire facial feature points output by the face recognition module, and perform error detection on the facial state detection results output by the face state detection module according to the pre-configured correspondence between facial feature points and facial states, to obtain the state error detection result, and send the state error detection result to the facial feature correction module; the state error detection result is either error present or error-free, and the facial state includes eye state and mouth state.
[0007] The facial feature correction module is used to correct the facial state discrimination parameters of the facial state detection module based on facial feature points when the state error detection result is incorrect. The facial state discrimination parameters include the eye aspect ratio threshold, the pupil offset ratio threshold, and the mouth aspect ratio threshold.
[0008] The driver posture recognition module is used to input the collected driving video into the head posture recognition model for recognizing the driver's head posture, obtain the driver head posture recognition result, and input the driver head posture recognition result into the driving state determination module;
[0009] The driving state determination module is used to detect the driver's state based on the facial state detection results input from the facial state detection module and the driver's head posture recognition results, and obtain the driver's state detection results.
[0010] Optionally, facial feature points include a first periorbital feature point, a second periorbital feature point, a third periorbital feature point, a fourth periorbital feature point, a fifth periorbital feature point, a sixth periorbital feature point, a first pupil feature point, a second pupil feature point, a first mouth feature point, a second mouth feature point, a third mouth feature point, a fourth mouth feature point, a fifth mouth feature point, and a sixth mouth feature point.
[0011] Optionally, the facial state discrimination parameters of the facial state detection module can be corrected based on facial feature points, including:
[0012] Calculate the driver's actual eye aspect ratio, actual pupil offset ratio, and actual mouth aspect ratio based on facial feature points;
[0013] Based on the actual eye aspect ratio, actual pupil offset ratio, and actual mouth aspect ratio, the thresholds for eye aspect ratio, pupil offset ratio, and mouth aspect ratio are corrected respectively.
[0014] Optionally, the formula for calculating the actual eye aspect ratio is as follows:
[0015]
[0016] Where EAR represents the actual eye aspect ratio, P1, P2, P3, P4, P5, and P6 represent the first, second, third, fourth, fifth, and sixth periorbital feature points, respectively, and norm(·) represents the Euclidean distance function.
[0017] The formula for calculating the pupillary deflection ratio is as follows:
[0018]
[0019] Wherein, PSR represents the actual pupillary deviation ratio, PSR xThe horizontal pupillary deflection ratio component, PSR y This represents the vertical pupillary deflection ratio component, (P) Pupil_center .x,P Pupil_center .y) represents the first pupil feature point, (P Eye_ center .x,P Eye_center .y) represents the feature point of the second pupil;
[0020] The formula for calculating the actual mouth width-to-height ratio is as follows:
[0021]
[0022] Wherein, MOU represents the actual mouth width-to-height ratio, and B1, B2, B3, B4, B5, and B6 represent the first, second, third, fourth, fifth, and sixth mouth feature points, respectively.
[0023] Optionally, the driver's head posture recognition results include the driver's head roll angle, pitch angle, and yaw angle.
[0024] Optionally, the driver posture recognition module is not coupled with the face recognition module, face state detection module, face tracking module, error detection module, and face feature correction module.
[0025] Optionally, the face recognition module uses an infrared 2D camera.
[0026] Optionally, the face tracking module performs face tracking based on the Tracker algorithm.
[0027] Secondly, the present invention provides a driver monitoring system, including the driver status detection system described above.
[0028] The beneficial effects of this invention are:
[0029] The driver state detection system provided by this invention uses a state error detection module to detect errors in the facial state detection results. When errors exist in the facial state detection results, the facial feature correction module can correct the facial state discrimination parameters of the facial state detection module, thereby obtaining accurate facial state detection results. This overcomes the defect of inaccurate driver facial state detection caused by facial differences between different drivers, which is beneficial to improving the accuracy of driver state detection. In addition, the driving state determination module combines the facial state detection results and the driver's head posture recognition results to perform driver state detection, which can additionally capture changes in the driver's head posture, further improving the accuracy of driver state detection. Attached Figure Description
[0030] Figure 1This is a schematic diagram of the driver status detection system in one embodiment of this application;
[0031] Figure 2 This is a schematic diagram showing the location of eye feature points in one embodiment of this application;
[0032] Figure 3 This is a schematic diagram showing the location of mouth feature points in one embodiment of this application;
[0033] Figure 4 This is a schematic diagram of FBE error in one embodiment of this application. Detailed Implementation
[0034] To address the inaccuracy of traditional driver monitoring systems in detecting driver state, this invention provides a driver state detection system and a driver monitoring system. The driver state detection system uses a state error detection module to detect errors in facial state detection results. When errors exist in the facial state detection results, a facial feature correction module corrects the facial state discrimination parameters of the facial state detection module, thereby obtaining accurate facial state detection results. This overcomes the inaccuracy caused by facial differences between different drivers, improving the accuracy of driver state detection. Furthermore, the driving state determination module combines facial state detection results with driver head posture recognition results for driver state detection, additionally capturing changes in the driver's head posture, further enhancing the accuracy of driver state detection.
[0035] The driver status detection system provided by the present invention will be described below.
[0036] like Figure 1 As shown, the driver state detection system includes a face recognition module 101, a face state detection module 102, a face tracking module 103, a state error detection module 104, a facial feature correction module 105, a driver posture recognition module 106, and a driving state determination module 107. The modules in the flowchart are all software algorithm modules integrated into the intelligent domain controller SOC hardware platform, and all require infrared 2D camera sensor hardware to function.
[0037] The following is a description of each module in the driver status detection system.
[0038] The face recognition module 101 is used to capture driving videos and perform face recognition on the driving videos and extract facial feature points of the target face.
[0039] Specifically, in one embodiment of the present invention, the face recognition module 101 captures driving video using an infrared 2D camera. On the one hand, although depth cameras (3D cameras) can achieve three-dimensional detection of faces and body postures, their commercial costs are high, while planar 2D cameras, combined with professional datasets and trained specific algorithms, can achieve driver state detection. On the other hand, compared with infrared 2D cameras, depth cameras (3D cameras) produce richer colors in their images, but they are also more susceptible to changes in lighting, such as changes in brightness when entering or exiting tunnels, lack of lighting, and complex background elements, which greatly increases the dimensionality of subsequent data processing and the complexity of the algorithms.
[0040] The following describes the process of face recognition module 101 performing face recognition and extracting facial feature points of the target face.
[0041] For face recognition, in one embodiment of the present invention, firstly, the driving video is subjected to grayscale conversion and pixel scaling to obtain multiple image frames; then, the multiple image frames are input into a pre-trained face detector model to obtain multiple face bounding boxes in each image frame; finally, using an integral image, the face in the largest face bounding box is identified as the target face, which represents the driver's face.
[0042] To extract facial feature points from a target face, the identified target face can be input into a pre-trained Dlib model to obtain multiple facial feature points. In an embodiment of the present invention, the facial feature points include a first peri-eye feature point, a second peri-eye feature point, a third peri-eye feature point, a fourth peri-eye feature point, a fifth peri-eye feature point, a sixth peri-eye feature point, a first pupil feature point, a second pupil feature point, a first mouth feature point, a second mouth feature point, a third mouth feature point, a fourth mouth feature point, a fifth mouth feature point, and a sixth mouth feature point.
[0043] The patent utilizes the correspondence between 68 feature points detected by the Dlib facial feature point library and physical organs, including feature point sets for the eyes and mouth. Specifically, this patent uses six feature point sets for the left and right eyes (corresponding to Dlib feature points 36-41 / 42-47). Figure 2 P1-P6 (corresponding to the first, second, third, fourth, fifth, and sixth periorbital feature points, respectively); while the pupil center feature point P... Eye_center It is the center position of the fitted ellipse of 6 eye feature points; the eye region image is converted into a binary image by grayscale thresholding, the pupil is detected as a black region, and the center of the pupil P is... Pupil_centerIt is its geometric center; the mouth feature points used in the patent are selected from the Dlib mouth outer lip contour feature points (the six mouth feature points 48, 50, 52, 54, 56, and 58 in the Dlib feature point map correspond to...). Figure 3 P1-P6 correspond to the first, second, third, fourth, fifth, and sixth mouth feature points, respectively.
[0044] This correspondence was chosen because these points are evenly distributed in the eye and mouth areas, which can accurately capture the local deformation of these organs, thereby improving the accuracy and robustness of facial recognition and analysis tasks.
[0045] The face state detection module 102 is used to perform face state detection based on the facial feature points output by the face recognition module 101 and its own pre-set face state discrimination parameters.
[0046] The aforementioned facial state discrimination parameters include the eye aspect ratio threshold, the pupil offset ratio threshold, and the mouth aspect ratio threshold. The eye aspect ratio threshold is used to determine the open / closed state of the eyes, the pupil offset ratio threshold is used to determine whether the eyes are looking straight ahead, and the mouth aspect ratio threshold is used to determine the degree of opening and closing of the mouth. Each threshold can be initialized based on general human facial data.
[0047] The face tracking module 103 is used to track the target face after the face recognition module 101 identifies it, so as to obtain the driver's facial state in real time. In an embodiment of the present invention, the face tracking module 103 performs face tracking based on the Tracker algorithm.
[0048] The following describes the process of face tracking module 103 performing face tracking based on the Tracker algorithm in an embodiment of the present invention.
[0049] Specifically, the face recognition cache frames and real-time captured driving video frames are used as inputs to the Tracker algorithm. The median flow tracker in the Tracker uniformly sprinkles points in the target box of frame t (t≤n) and samples them as feature points. The pyramid LK optical flow method is used to track the corresponding positions of the feature points in frame t+1. Starting from the initial position Pt of a tracking point at time t, the tracker tracks the position P(t+k) that generates time (t+k). Then, it tracks the predicted position P't of time t in reverse from position P(t+k). Finally, the Euclidean distance between the initial position Pt and the predicted position P't is used as the FBE error of the tracker at time t (e.g., ...). Figure 4(As shown); the half of the points with the smallest FBE error are selected as the best tracking points. The position and scale of the bounding box in frame t+1 are calculated based on the coordinate changes of these points. Let di represent the displacement of a point and dm represent the median displacement. The residual can be defined as |di-dm|. If the residual is greater than 10 pixels, the tracking is considered to have failed, and face detection is performed again. If the tracking is successful, the output of the result will combine the offsets of all remaining points with high confidence. The remaining points are used to estimate the target's movement. The median of the spatial dimension movement of each point is used as the target's movement. For each feature point, the ratio of its movement to the previous point is calculated. The median of the ratios of all points is used as the change in scale of the current image relative to the previous image. Then, the face target is updated on the current image.
[0050] The state error detection module 104 is used to acquire the facial feature points output by the face recognition module 101, and perform error detection on the facial state detection results output by the face state detection module 102 according to the pre-configured correspondence between facial feature points and facial state, to obtain the state error detection result, and send the state error detection result to the face feature correction module 105.
[0051] In embodiments of the present invention, the above-mentioned state error detection result is either error present or error-free, and the facial state includes eye state and mouth state. For example, the eye state is open eyes, closed eyes, or pupil deviation, and the mouth state is open mouth or closed mouth.
[0052] In one embodiment of the present invention, the coordinates of each feature point can be used to calculate the eye aspect ratio (EAR), pupil offset ratio (PSR), and mouth aspect ratio (MOU). By comparing the mouth and eye aspect ratio thresholds, pupil offset ratio thresholds, and mouth aspect ratio thresholds, the open / closed state of the eyes, whether the human eye is looking straight ahead, and whether the driver is yawning can be obtained.
[0053] It should be noted that when the facial state result obtained based on the above correspondence is inconsistent with the facial state detection result output by the facial state detection module 102, the state error detection result is determined to have an error; otherwise, the state error detection result is determined to have no error. When the state error detection result is without error, it is not necessary to correct the facial state discrimination parameters of the facial state detection module 102 through the facial feature correction module 105. At this time, the facial state detection result output by the facial state detection module 102 can be input into the driving state determination module 107. When the state error detection result has an error, it is necessary to correct the facial state discrimination parameters of the facial state detection module 102 through the facial feature correction module 105, use the corrected facial feature correction module 105 to obtain a new facial state detection result, and use the state error detection module 104 for judgment again until the state error detection result output by the state error detection module 104 is without error.
[0054] This measure can overcome the shortcomings of inaccurate driver facial state detection caused by facial differences between different drivers, and is conducive to improving the accuracy of driver state detection.
[0055] The facial feature correction module 105 is used to correct the facial state discrimination parameters of the facial state detection module 102 based on facial feature points when the state error detection result is that there is an error.
[0056] In embodiments of the present invention, facial state discrimination parameters include eye aspect ratio threshold, pupil offset ratio threshold, and mouth aspect ratio threshold.
[0057] The following describes the process by which the facial feature correction module 105 corrects the facial state discrimination parameters of the facial state detection module 102 based on facial feature points, specifically including steps I to II.
[0058] Step I: Calculate the driver's actual eye aspect ratio, actual pupil offset ratio, and actual mouth aspect ratio based on facial feature points.
[0059] Specifically, the formula for calculating the actual aspect ratio of the eye is as follows:
[0060]
[0061] Where EAR represents the actual eye aspect ratio, P1, P2, P3, P4, P5, and P6 represent the first, second, third, fourth, fifth, and sixth periorbital feature points, respectively, and norm(·) represents the Euclidean distance function.
[0062] The formula for calculating the pupillary deflection ratio is as follows:
[0063]
[0064] Wherein, PSR represents the actual pupillary deviation ratio, PSR x The horizontal pupillary deflection ratio component, PSR y This represents the vertical pupillary deflection ratio component, (P) Pupil_center .x,P Pupil_center .y) represents the first pupil feature point, (P Eye_ center .x,P Eye_center .y) represents the feature point of the second pupil;
[0065] The formula for calculating the actual mouth width-to-height ratio is as follows:
[0066]
[0067] Wherein, MOU represents the actual mouth width-to-height ratio, and B1, B2, B3, B4, B5, and B6 represent the first, second, third, fourth, fifth, and sixth mouth feature points, respectively.
[0068] Step II: Based on the actual eye aspect ratio, actual pupil offset ratio, and actual mouth aspect ratio, the threshold values for eye aspect ratio, pupil offset ratio, and mouth aspect ratio are corrected respectively.
[0069] Based on the actual eye aspect ratio, actual pupil offset ratio, and actual mouth aspect ratio, the thresholds for eye aspect ratio, pupil offset ratio, and mouth aspect ratio are corrected, including:
[0070] The three correction methods share the same overall approach. Taking EAR eye feature correction as an example:
[0071] An EAR linear SVM classifier is created based on the EAR threshold EARthresh_base for the open / closed eye state. For an image / video cache with fps of n, the classifier scans the image cache frames in groups of ±n / 5 frames, filters out the open / closed eye state, filters out the fully open / fully closed eye state, and then aligns the state data into groups by timestamp (total number of groups is K), and records EARopen[i] and EARclosed[i] of the corresponding group respectively.
[0072] The threshold calculation function fthresh() and Bezier correction are used to calculate EARthresh[i], the average value EARthresh_avg, and the final corrected threshold EARthresh_new for each group:
[0073] EAR thresh [i] = f thresh(EAR open [i],EAR closed [i],EAR thresh_base )
[0074]
[0075] Taking the above calculation approach for EARthresh_new as an example, calculate PSRthresh_new and MOUthresh_new respectively.
[0076] The driver posture recognition module 106 is used to input the collected driving video into the head posture recognition model for recognizing the driver's head posture, obtain the driver head posture recognition result, and input the driver head posture recognition result into the driving state determination module 107.
[0077] The above driver head posture recognition results include the driver's head roll angle, pitch angle, and yaw angle.
[0078] The process of obtaining the driver's head posture recognition results is explained below.
[0079] In an embodiment of the present invention, a fine-grained head pose estimation deep learning model is used. This model does not rely on the alignment of facial feature points with the 3D head standard. Instead, it directly uses a multi-loss network from the image to divide the angle range into 66 bits. Softmax is applied to the 66 categories, and the expectation of the result is calculated to obtain the three Euler angles of the head pose: roll, pitch, and yaw, thereby predicting the head pose.
[0080] The model is trained simultaneously from three perspectives, with each perspective incurring a loss. The loss is primarily generated by the Bin classification and integral regression components. H(·) represents the classification cross-entropy loss function, Mse(·) represents the mean squared error loss function, and y represents the true yaw, pitch, and roll angles labeled in the training data. This represents the model's prediction result for the corresponding angle of a given input image.
[0081] However, the model first performs Bin classification and then integral regression. Bin classification is a coarse classification, which introduces additional errors that are amplified during integral regression, limiting the model's predictive performance. Therefore, the following solution is used:
[0082] Based on different classification scales, the angle range is refined from a fixed 66 bits to more precise bits, and each bit calculates its own cross-entropy loss function. In the integral regression component, the result of the bit classification is used to calculate the expectation and regression loss. The regression loss and multiple classification losses are combined into a total loss. Each angle has such a combined loss and shares the previous convolutional layers. Therefore: In practice, the image frame is imported into the head pose recognition model trained using the AFLW2000 dataset, and the predicted three Euler angles and the average error are output.
[0083] It should be noted that, in the embodiments of the present invention, the driver posture recognition module 106 is not coupled to the face recognition module 101, the face state detection module 102, the face tracking module 103, the error detection module, and the face feature correction module 105.
[0084] The driving state determination module 107 is used to perform driver state detection based on the facial state detection result and the driver head posture recognition result input by the facial state detection module 102, and obtain the driver state detection result.
[0085] First, facial and head pose status recognition was performed simultaneously, and the results are as follows:
[0086] Facial state recognition: Eye state: By detecting eye feature points and calculating EAR (eye aspect ratio), it can determine whether the driver's eyes are closed, thereby identifying fatigue or drowsiness.
[0087] Mouth position: Analyzing the opening and closing of the mouth is used to detect whether the driver is yawning, which is a common sign of fatigue.
[0088] Facial expressions: Through facial expression recognition models, states such as drowsiness, distraction, and tension can be identified.
[0089] Head posture recognition: Euler angles are used to estimate the head rotation angles, including yaw (left and right head turn), pitch (up and down head tilt), and roll (head tilt). These angles are used to determine the driver's gaze direction.
[0090] Direction of gaze: By combining head posture and direction of eye gaze, it can be determined whether the driver is paying attention to the road and whether there is any distraction (such as looking at a mobile phone or the outside world).
[0091] Head position and movement: If the driver's head is detected to be deviating from the normal driving position for an extended period of time (such as frequently looking down), it may indicate that the driver is inattentive or fatigued.
[0092] The driver's condition monitoring results, based on a comprehensive analysis of the above facial and head posture status, include:
[0093] Fatigue state:
[0094] Prolonged eye closure, frequent yawning, and heavy eyelids indicate that the driver may be fatigued.
[0095] Continuous naps or short periods of sleep.
[0096] Distracted state:
[0097] The gaze is off the road, the head turns frequently, and the direction of the gaze is inconsistent with what is in front for a long time.
[0098] The driver may be looking at their phone, entertainment system, or surroundings.
[0099] State of focus:
[0100] The driver kept his eyes on the road ahead, focusing his head and eyes on the driving operation.
[0101] The absence of distraction or fatigue indicates that the driver is in a safe driving state.
[0102] Dangerous situation:
[0103] An emergency warning is required if the driver exhibits extreme fatigue (eyes closed for more than a certain threshold), distraction, or sudden abnormal behavior (such as abruptly turning their head).
[0104] Driver condition monitoring results can be used in the following ways:
[0105] Driving safety monitoring:
[0106] Fatigue alert: When the system detects signs of driver fatigue, it can remind the driver to rest through sound, vibration, or other means to avoid traffic accidents.
[0107] Distraction alert: If the system detects driver distraction (such as looking at a mobile phone or chatting with passengers), it can issue a warning to draw the driver's attention and prevent negligent driving.
[0108] Interaction with autonomous driving systems:
[0109] Takeover Request: In semi-autonomous driving mode, when the system detects that the driver is not focused on driving (such as being distracted for a long time), it will issue a warning in advance and request the driver to take over the vehicle.
[0110] Disabled driver detection: If a driver loses control of the vehicle due to extreme fatigue, health problems, or other reasons, the system can automatically slow down, stop, or even send out a distress signal.
[0111] Behavioral data analysis:
[0112] Driving behavior optimization: Record driver status detection results and analyze their long-term driving behavior habits to provide a basis for personalized driving suggestions, insurance assessments, etc.
[0113] Accident liability determination: In the event of a traffic accident, the records from the driver status monitoring system can serve as reference data for the accident investigation, helping to analyze whether the driver is fatigued or distracted, thereby assisting in determining accident liability.
[0114] Driver assistance function integration:
[0115] Adaptive driving mode: Adjusts the vehicle's driving mode based on the driver's condition. For example, it automatically activates driver assistance functions to reduce driving burden when driver fatigue is detected.
[0116] Warning system optimization: Based on the status detection results, adjust the intensity or type of different warnings (such as audible and visual alerts) to improve the driver's response efficiency to warnings.
[0117] As can be seen from the above, the driver state detection system provided by the present invention performs error detection on the facial state detection results through the state error detection module 104. When there is an error in the facial state detection results, the facial state discrimination parameters of the facial state detection module 102 can be corrected by the facial feature correction module 105, thereby obtaining accurate facial state detection results. This can overcome the defect of inaccurate driver facial state detection caused by facial differences between different drivers, and is conducive to improving the accuracy of driver state detection. In addition, the driving state determination module 107 combines the facial state detection results and the driver head posture recognition results to perform driver state detection, which can additionally capture changes in the driver's head posture, which is conducive to improving the accuracy of driver state detection.
[0118] The driver monitoring system provided by the present invention includes the driver state detection system disclosed in any of the above embodiments, and has the same beneficial effects as the driver state detection system described in any of the above embodiments, which will not be repeated here.
[0119] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.
[0120] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.
Claims
1. A driver state detection system, comprising a face recognition module, a face state detection module, and a face tracking module, characterized in that, The driver state detection system also includes a state error detection module, a facial feature correction module, a driver posture recognition module, and a driving state determination module. The state error detection module is used to acquire the facial feature points output by the face recognition module, and perform error detection on the facial state detection result output by the face state detection module according to the pre-configured correspondence between facial feature points and facial state, to obtain the state error detection result, and send the state error detection result to the facial feature correction module. The state error detection result is either error present or error-free, and the facial state includes the eye state and the mouth state. The facial feature correction module is used to correct the facial state discrimination parameters of the facial state detection module based on the facial feature points when the state error detection result is incorrect; the facial state discrimination parameters include the eye aspect ratio threshold, the pupil offset ratio threshold, and the mouth aspect ratio threshold. The driver posture recognition module is used to input the collected driving video into a head posture recognition model for recognizing the driver's head posture, obtain the driver head posture recognition result, and input the driver head posture recognition result into the driving state determination module; The driving state determination module is used to perform driver state detection based on the facial state detection result input by the facial state detection module and the driver's head posture recognition result, and obtain the driver state detection result.
2. The driver status detection system according to claim 1, characterized in that, The facial feature points include a first periorbital feature point, a second periorbital feature point, a third periorbital feature point, a fourth periorbital feature point, a fifth periorbital feature point, a sixth periorbital feature point, a first pupil feature point, a second pupil feature point, a first mouth feature point, a second mouth feature point, a third mouth feature point, a fourth mouth feature point, a fifth mouth feature point, and a sixth mouth feature point.
3. The driver status detection system according to claim 2, characterized in that... The facial state discrimination parameters of the facial state detection module are corrected based on the facial feature points, including: Calculate the driver's actual eye aspect ratio, actual pupil offset ratio, and actual mouth aspect ratio based on the facial feature points. Based on the actual eye aspect ratio, the actual pupil offset ratio, and the actual mouth aspect ratio, the threshold values for eye aspect ratio, pupil offset ratio, and mouth aspect ratio are corrected respectively.
4. The driver status detection system according to claim 3, characterized in that, The formula for calculating the actual eye aspect ratio is as follows: Wherein, EAR represents the aspect ratio of the actual eye, P1, P2, P3, P4, P5, P6 represent the first, second, third, fourth, fifth, and sixth periorbital feature points, respectively, and norm(·) represents the Euclidean distance function. The formula for calculating the pupil offset ratio is as follows: Wherein, PSR represents the actual pupil offset ratio, PSR x The horizontal pupillary deflection ratio component, PSR y This represents the vertical pupillary deflection ratio component, (P) Pupil_center .x,P Pupil_center .y) represents the first pupil feature point, (P Eye_center .x,P Eye_center .y) represents the feature point of the second pupil; The formula for calculating the actual mouth width-to-height ratio is as follows: Wherein, MOU represents the actual mouth width-to-height ratio, and B1, B2, B3, B4, B5, and B6 represent the first, second, third, fourth, fifth, and sixth mouth feature points, respectively.
5. The driver status detection system according to claim 4, characterized in that, The driver's head posture recognition results include the driver's head roll angle, pitch angle, and yaw angle.
6. The driver status detection system according to claim 1, characterized in that, The driver posture recognition module is not coupled with the face recognition module, the face state detection module, the face tracking module, the error detection module, and the face feature correction module.
7. The driver status detection system according to claim 1, characterized in that, The face recognition module uses an infrared 2D camera.
8. The driver status detection system according to claim 1, characterized in that, The face tracking module performs face tracking based on the Tracker algorithm.
9. A driver monitoring system, characterized in that, Includes the driver status detection system as described in any one of claims 1-8.