A driver driving behavior detection system based on face recognition

Abstract

The present application relates to the technical field of image detection, and more particularly to a driver driving behavior detection system based on face recognition, comprising: a data acquisition module, which acquires behavior images and determines the adaptability to complex light environment according to the number of key point losses in the behavior images; a data processing module, which processes the behavior images to output face bounding boxes and a plurality of key point coordinates, and performs cross-frame identity association caching on the face target according to the number of losses of the face bounding boxes; a behavior analysis module, which extracts facial features from standardized face images, performs behavior recognition to obtain recognition results, and performs sample enhancement on pseudo-labels of five sense organs time sequence motion constraints according to the number of missed detections of fatigue state signals; a decision confirmation module, which confirms abnormal behaviors; and an early warning control module, which generates early warning information according to the abnormal confirmation results and identity authentication results, and executes early warning strategies. The present application improves the accurate recognition capability of driver face images during driving.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a driver behavior detection system based on face recognition. Background Technology

[0002] In existing technologies, driver fatigue detection typically employs a scheme based on a single facial key point model: a single-stage face detection model is used to locate the face and extract facial key points. Based on these key points, the eye opening ratio and mouth opening ratio are calculated and compared with preset fixed thresholds. When the eye opening ratio is lower than the eye-closed threshold, it is determined to be closed eyes; when the mouth opening ratio is higher than the mouth-opening threshold, it is determined to be yawning; and when the head deflection angle exceeds a fixed threshold, it is determined to be inattentiveness. An alarm is triggered immediately after any fatigue feature is detected in a single frame image. Furthermore, face recognition and fatigue detection are usually deployed as independent systems and are not related to each other. However, the above solutions have many shortcomings: fixed thresholds cannot adapt to the differences in physiological characteristics of different drivers. Drivers with small eyes may have a low EAR when their eyes are open, which can easily be misjudged as closed eyes. Drivers with large eyes may still have an EAR higher than the threshold when their eyes are closed, leading to missed alarms. When wearing glasses, lens reflections interfere with key point positioning, further aggravating false alarms. Single-frame real-time alarms cannot distinguish between normal blinking and true fatigue-induced eye closure, and frequent false alarms affect the driving experience. Glasses reflections cause the EAR to be systematically low, and the system lacks self-correcting capabilities. Fatigue detection and identity authentication are independent of each other, making it impossible to confirm whether the detected person is an authorized driver, and also impossible to detect the safety risk of unauthorized personnel replacing the driver. The judgment of head posture and eye and mouth status is disconnected. When the head is tilted, perspective distortion causes a low EAR on one side, which can be misjudged as closed eyes. At the same time, head tilting also triggers an inattention alarm, resulting in a double false alarm.

[0003] Chinese Patent Publication No. CN116152786A discloses a method for monitoring the driving behavior of construction crane operators based on deep learning. The method includes: 1) acquiring real-time images of the cab via a camera to obtain facial region images and facial key point coordinates, and performing facial recognition and identity verification based on a deep convolutional neural network; 2) cropping images of the eyes and mouth using facial key point coordinates, identifying the state change features of the eyes and mouth using a deep convolutional neural network, and determining whether the driver is experiencing fatigue driving behavior based on PERCLOS and PMOT parameters; 3) acquiring facial key point coordinates... After marking, the pitch, yaw, and roll angles of the driver's head posture are estimated through matrix transformation, and the operating status of the construction crane is integrated to determine whether the field of vision has deviated; 4) The driver's current actions and posture are estimated based on the skeleton network, and key object information and the operating status of the construction crane are integrated to further determine whether the driver is distracted; 5) The detected facial recognition, fatigue driving, field of vision deviation estimation, and distraction behavior information are sent to the management platform. The management platform pushes SMS and video to major alarms that occur multiple times in a short period of time, enabling managers to comprehensively manage the construction site. It can be seen that the deep learning-based construction crane driver behavior monitoring method, due to its use of a fixed threshold to judge fatigue state, cannot adapt to individual differences, and the single-frame real-time judgment lacks continuous verification over time, resulting in insufficient accurate recognition of driver facial images during driving. Summary of the Invention

[0004] To address this, the present invention provides a driver behavior detection system based on facial recognition, which overcomes the problems in the prior art where the use of fixed thresholds to determine fatigue status fails to adapt to individual differences and the single-frame instantaneous determination lacks continuous verification over time, resulting in insufficient accurate recognition of driver facial images during driving.

[0005] To achieve the above objectives, the present invention provides a driver behavior detection system based on facial recognition, comprising: The data acquisition module is used to collect images of the driver's behavior during driving, and to determine whether the adaptability to complex lighting environments meets the requirements based on the number of lost key points in the behavior images. The data processing module is used to process the behavior image to output the face bounding box and several key point coordinates, so as to perform cross-frame identity association caching of the face target based on the number of times the face bounding box is lost. The behavior analysis module is used to extract facial features from standardized face images and perform behavior recognition to obtain recognition results. It then uses the results to enhance the pseudo-labels of the temporal motion constraints of the facial features based on the number of missed detections of fatigue state signals. The decision confirmation module is used to verify the time continuity of the fatigue state signal and the attention deviation signal to confirm abnormal behavior. The early warning control module is used to generate early warning information based on the confirmation result of the abnormal behavior and the identity authentication result, so as to execute the corresponding early warning strategy.

[0006] Furthermore, the data processing module is used to input the behavioral image into the main detection model and output the face bounding box and the coordinates of several key points; The data processing module is used to correct the original face region into a standardized face image based on the coordinates of the key points using an affine transformation algorithm.

[0007] Furthermore, the behavior analysis module is used to input the standardized face image into the key point detection model and output several facial key point coordinates and three-dimensional head pose parameters. The behavior analysis module is used to input the standardized face image into the face feature extraction model, output the face feature vector, and compare it with the driver face database to confirm the driver's identity authentication result.

[0008] Furthermore, the behavior analysis module is used to dynamically update the current driver's closed-eye and open-mouth judgment thresholds based on historical EAR / MAR data when the head posture is within the frontal range. The behavior analysis module is used to compare the real-time EAR / MAR with a fixed threshold and an adaptive threshold respectively. When any of the eye-closing or mouth-opening judgment conditions are met, the corresponding fatigue state signal is output. The behavior analysis module is used to directly output an attention deviation signal when the head posture angle exceeds the range of the frontal face, and skips the eye and mouth state discrimination.

[0009] Furthermore, the decision confirmation module is used to record the corresponding timestamp when abnormal behavior is detected for the first time, and to determine whether the duration of the abnormal behavior exceeds a preset time threshold during the duration of the abnormal behavior, and to confirm the corresponding abnormal behavior only when the threshold is exceeded. The decision confirmation module is used to collect multiple frames of images at preset intervals during the continuous process of abnormal behavior to form a sequence of evidence of abnormal behavior. The decision confirmation module is used to establish a sliding window for the closed-eye state and cache the closed-eye detection results for several consecutive frames. The decision confirmation module is used to calculate the deviation between the current EAR value and the adaptive threshold when the historical frame determines that the eyes are closed and the current frame detects that the eyes are open. If the deviation is less than the preset range, it is determined to be eye glare interference, the closed eye state is maintained and threshold self-repair is performed.

[0010] Furthermore, the data acquisition module is used to determine whether the adaptability to complex lighting environments meets the requirements based on whether the number of lost key points in the behavioral image is less than or equal to a preset number of lost points. The data acquisition module is used to determine that the ability to adapt to complex lighting environments does not meet the requirements if the number of lost key points in the behavioral image is greater than the preset number of lost points.

[0011] Furthermore, the data processing module is used to determine that the stable tracking capability of the face bounding box does not meet the requirements if the number of times the face bounding box is lost is greater than a preset first number of times. The data processing module is used to perform cross-frame identity association caching of the face target based on the fact that the number of times the face bounding box is lost is greater than the preset first number of times of loss and less than the preset second number of times of loss. The data processing module is used to preliminarily determine that the fatigue state accurate detection capability does not meet the requirements based on the number of times the face bounding box is lost being greater than or equal to the preset second number of times it is lost, and to determine whether the fatigue state accurate detection capability meets the requirements based on the number of times the fatigue state signal is missed.

[0012] Furthermore, the data processing module is used to assign a unique identifier to the face target detected in consecutive video frames, and bind and store the identifier with the corresponding driving state cache; The data processing module is used to keep the identity identifier alive when the face bounding box is lost for a short time, and to restore the binding status data through the identity identifier when the same target is detected again.

[0013] Furthermore, the behavior analysis module is used to determine whether the fatigue state accurate detection capability meets the requirements based on whether the number of missed detections of the fatigue state signal is less than or equal to the preset number of missed detections. The behavior analysis module is used to determine that the fatigue state accuracy detection capability does not meet the requirements if the number of missed detections of the fatigue state signal is greater than the preset number of missed detections, and to perform sample augmentation on the pseudo-labels of the temporal motion constraints of the facial features.

[0014] Furthermore, the behavior analysis module is used to establish temporal constraint relationships by utilizing the motion trends of key facial features within continuous video frames, and to generate pseudo-label extended samples for the collected frontal face samples. The behavior analysis module is used to combine the pseudo-label extended samples with real-time frontal face samples to participate in the extreme value statistical update of the adaptive threshold.

[0015] Compared with existing technologies, the beneficial effects of the present invention are as follows: The system of the present invention, by setting a data acquisition module, can collect images of the driver's behavior during driving, providing a basic data source for subsequent face detection and state recognition; by setting a data processing module, it can perform face detection and key point extraction on the behavior images, and use an affine transformation algorithm to correct face images from different angles and positions into standardized face images, eliminating image differences caused by posture changes during driving, and providing a standardized input image with a unified format for face recognition and fatigue detection; by setting a behavior analysis module, it can extract facial features from the standardized face images to output driving state features, and perform real-time behavior recognition based on the driving state features to obtain recognition results; by setting a decision confirmation module, it can verify the temporal continuity of fatigue state signals and attention deviation signals, confirming the corresponding abnormal behavior only when the abnormal behavior continues to exceed a preset time threshold, and correcting the abrupt changes in the closed-eye state under glasses-wearing scenarios; by setting an early warning control module, it can generate early warning information based on the abnormal behavior confirmation results and identity authentication results, and execute corresponding early warning strategies, thereby improving the accurate recognition capability of driver face images during driving.

[0016] Furthermore, the system described in this invention determines its adaptability to complex lighting environments by setting a preset number of lost points. Strong light from car windows and direct midday sunlight can cause facial overexposure, leading to oversaturation in localized areas of the facial image. In these areas, the key point detection model cannot extract effective facial texture information, resulting in the loss of key points around the eyes and mouth. By determining the adaptability to complex lighting environments, the system's stability in detecting key facial areas under complex lighting conditions such as strong light and backlight can be quantified. This avoids abnormal extraction of driving state features due to missing key points, improves the accuracy of fatigue driving behavior recognition, and further enhances the accurate recognition of driver facial images during driving.

[0017] Furthermore, the system of the present invention performs cross-frame identity association caching of facial targets by setting a preset first loss number and a preset second loss number. Due to vehicle bumps and vibrations, the face continuously shifts in the image, making it impossible to maintain continuous and stable face tracking, resulting in frequent loss and re-detection of the face bounding box and frequent clearing of the state cache. By performing cross-frame identity association caching of facial targets, a unique identity identifier can be assigned to the same driver target, and the closed-eye state sliding window and fatigue timer are bound to the identity identifier. When the face bounding box is lost for a short time, the identity identifier remains alive and maintains the bound state data. When the same target is detected again, the identity identifier is used to complete the same source matching, directly inheriting the closed-eye state window and fatigue timer value before the loss, realizing seamless continuation of state data, avoiding interruption of driving state cache and repeated reset of fatigue timer caused by bumps, thereby ensuring the continuity of fatigue detection state under bumpy scenarios, and further improving the accurate recognition capability of driver face images during driving.

[0018] Furthermore, the system described in this invention enhances the pseudo-labels of the temporal motion constraints of the facial features by setting a preset number of missed detections. Due to insufficient effective frontal face samples collected in the early stage of system startup, it is impossible to obtain enough data on the opening and closing of the eyes and mouth under the frontal face posture to support statistical learning. The adaptive threshold convergence is slow, resulting in fluctuations in the EAR and MAR judgment boundaries, leading to missed detections of fatigue driving. The fatigue state signal cannot truly reflect the driver's actual fatigue state. By enhancing the pseudo-labels of the temporal motion constraints of the facial features, the temporal constraint relationship can be established by utilizing the motion trend of key points of the facial features within continuous video frames. Pseudo-label extended samples that conform to the motion law are generated for the collected frontal face samples. Low-weight pseudo-label extended samples are generated for small-angle candidate samples after temporal interpolation and posture compensation to assist in updating. The pseudo-label extended samples are jointly used with the high-quality frontal face samples collected in real time to participate in the extreme value statistics update of the adaptive threshold, expanding the available sample size in the startup stage, suppressing disordered oscillations of the EAR and MAR judgment boundaries, accelerating the convergence of the adaptive threshold, improving the accuracy of fatigue state detection in the system startup stage, and further improving the accurate recognition capability of driver face images during driving. Attached Figure Description

[0019] Figure 1 This is an overall structural block diagram of the driver driving behavior detection system based on face recognition according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the process by which the driver behavior detection system based on face recognition determines the adaptability to complex lighting environments according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the cross-frame identity association and caching process for a face target in a driver behavior detection system based on face recognition, according to an embodiment of the present invention. Figure 4 This is a flowchart illustrating the sample augmentation process of pseudo-labels constrained by the temporal motion of facial features in the driver behavior detection system based on face recognition, according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0021] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0022] Please see Figure 1 The diagram shown is an overall structural block diagram of a driver behavior detection system based on face recognition according to an embodiment of the present invention. The system described in this embodiment includes: The data acquisition module is used to collect images of the driver's behavior during driving, and to determine whether the adaptability to complex lighting environments meets the requirements based on the number of lost key points in the behavior images. The data processing module is used to process the behavior image to output the face bounding box and several key point coordinates, so as to perform cross-frame identity association caching of the face target based on the number of times the face bounding box is lost. The behavior analysis module is used to extract facial features from standardized face images and perform behavior recognition to obtain recognition results. It then uses the results to enhance the pseudo-labels of the temporal motion constraints of the facial features based on the number of missed detections of fatigue state signals. The decision confirmation module is used to verify the time continuity of the fatigue state signal and the attention deviation signal to confirm abnormal behavior. The early warning control module is used to generate early warning information based on the confirmation result of the abnormal behavior and the identity authentication result, so as to execute the corresponding early warning strategy.

[0023] Specifically, behavioral images include cockpit environment images, driver's face area images, and images of driver's head posture changes.

[0024] Specifically, driving state characteristics include eye state characteristics, mouth state characteristics, and head posture characteristics.

[0025] Specifically, the identification results include fatigue state signals, attention deviation signals, and identity authentication results.

[0026] Specifically, fatigue state signals include closed-eye state signals, open-mouth state signals, and combined fatigue state signals.

[0027] Specifically, the attention deviation signal is a signal that quantifies the degree of deviation between the driver's line of sight and the direction of the driving task by measuring the head posture angle. It is actively output when the posture exceeds the range of the face, while the eye and mouth fatigue judgment is blocked to avoid the superposition of false alarms.

[0028] Specifically, abnormal behaviors include abnormal behaviors such as closing eyes, opening mouth, and attention deviating from the expected behavior.

[0029] Specifically, the warning information includes fatigue driving warnings, attention deviation warnings, and identity abnormality warnings.

[0030] Specifically, the early warning strategies include voice prompts, vibration alerts, and remote reporting.

[0031] In this embodiment, the system described in this invention, by setting a data acquisition module, can collect behavioral images of the driver during driving, providing a basic data source for subsequent face detection and state recognition; by setting a data processing module, it can perform face detection and key point extraction on the behavioral images, and use an affine transformation algorithm to correct face images from different angles and positions into standardized face images, eliminating image differences caused by posture changes during driving, and providing a standardized input image with a unified format for face recognition and fatigue detection; by setting a behavior analysis module, it can extract facial features from the standardized face images to output driving state features, and perform real-time behavior recognition based on the driving state features to obtain recognition results; by setting a decision confirmation module, it can verify the time continuity of fatigue state signals and attention deviation signals, confirming the corresponding abnormal behavior only when the abnormal behavior continues to exceed a preset time threshold, and repairing the abrupt changes in the closed-eye state in the case of wearing glasses; by setting an early warning control module, it can generate early warning information based on the abnormal behavior confirmation results and identity authentication results, and execute corresponding early warning strategies, thereby improving the accurate recognition capability of driver face images during driving.

[0032] Specifically, the data processing module is used to input the behavior image into the main detection model and output the face bounding box and the coordinates of several key points; The data processing module is used to correct the original face region into a standardized face image based on the coordinates of the key points using an affine transformation algorithm.

[0033] Specifically, the behavior analysis module is used to input the standardized face image into the key point detection model and output the coordinates of several facial key points and three-dimensional head pose parameters. The behavior analysis module is used to input the standardized face image into the face feature extraction model, output the face feature vector, and compare it with the driver face database to confirm the driver's identity authentication result.

[0034] Specifically, the behavior analysis module is used to dynamically update the current driver's closed-eye and open-mouth judgment thresholds based on historical EAR / MAR data when the head posture is within the frontal range. The behavior analysis module is used to compare the real-time EAR / MAR with a fixed threshold and an adaptive threshold respectively. When any of the eye-closing or mouth-opening judgment conditions are met, the corresponding fatigue state signal is output. The behavior analysis module is used to directly output an attention deviation signal when the head posture angle exceeds the range of the frontal face, and skips the eye and mouth state discrimination.

[0035] Specifically, EAR is the aspect ratio of the eye, a quantitative indicator of the degree of eye opening and closing calculated based on the coordinates of key points around the eye.

[0036] Specifically, MAR stands for Mouth Aspect Ratio, a quantitative indicator of the degree of mouth opening and closing calculated based on the coordinates of key points on the mouth.

[0037] In this embodiment, fatigue detection can also be performed using end-to-end deep learning fatigue detection, physiological signal fusion fatigue detection, and cloud-based inference distributed solutions. Among them, end-to-end deep learning fatigue detection: train an end-to-end fatigue detection network, directly input face images, and output fatigue type classification and fatigue degree regression values. This approach saves the intermediate key point extraction and manual rule design, but has weak interpretability and requires a large amount of labeled data for training. Fatigue detection through physiological signal fusion: By increasing the acquisition of physiological signals and performing multimodal fusion and discrimination of visual features and physiological signals, visual features provide intuitive fatigue representations, while physiological signals provide objective indicators of physiological state. The complementarity of the two can significantly improve the reliability of detection. A distributed cloud-based inference solution: This replaces local TensorRT inference by uploading face images to a cloud server for inference analysis, while the local machine is only responsible for image acquisition and result display. This solution can use larger-scale models to achieve higher accuracy and facilitates model iteration and updates, but it has high requirements for network bandwidth and latency, making it unsuitable for scenarios with strict real-time requirements.

[0038] Specifically, the decision confirmation module is used to record the corresponding timestamp when abnormal behavior is detected for the first time, and to determine whether the duration of the abnormal behavior exceeds a preset time threshold during the period of the abnormal behavior. The corresponding abnormal behavior is confirmed only when the threshold is exceeded. The decision confirmation module is used to collect multiple frames of images at preset intervals during the continuous process of abnormal behavior to form a sequence of evidence of abnormal behavior. The decision confirmation module is used to establish a sliding window for the closed-eye state and cache the closed-eye detection results for several consecutive frames. The decision confirmation module is used to calculate the deviation between the current EAR value and the adaptive threshold when the historical frame determines that the eyes are closed and the current frame detects that the eyes are open. If the deviation is less than the preset range, it is determined to be eye glare interference, the closed eye state is maintained and threshold self-repair is performed.

[0039] Specifically, the preset interval is a time parameter for periodically acquiring multiple frames of images from a continuous video stream and storing them in an evidence sequence according to a preset time interval or frame interval during the continuous process of abnormal behavior.

[0040] In this embodiment, for the acquisition of multi-frame discrete photos, video clips can also be used as evidence for acquisition; The evidence collection process for video clips involves directly recording short video clips during the time delay and storing the videos as evidence. This approach can present the dynamic process of fatigue behavior more completely, and its storage efficiency may be better than that of multiple independent images.

[0041] In this embodiment, for the sliding window tracking and glasses reflection adaptive repair mechanism in the closed-eye state, Kalman filtering state tracking, exponential moving average (EMA) threshold repair, glasses detection pre-judgment, and infrared dual-mode closed-eye discrimination can also be used for repair. The Kalman filter state tracking involves smoothing the EAR value sequence using a Kalman filter. The state variable of the Kalman filter is the "true degree of eye closure," and the observation variable is the "EAR value with noise." The true eye closure state is estimated in real time through a prediction-update step, naturally suppressing EAR value fluctuations caused by reflections. This scheme does not require explicit differentiation between "reflection frames" and "true eye-closed frames," as the filtering process automatically achieves smoothing. Threshold correction using Exponential Moving Average (EMA): Instead of a fixed step correction of +0.003 each time, an exponential moving average method is used: m_left_eye_min = α × tanlefteye + (1-α) × m_left_eye_min, where α is a smoothing coefficient (e.g., 0.05). When reflective interference is detected, EMA naturally brings the minimum value slightly closer to the current observation value. The correction speed is proportional to the degree of interference, avoiding over-correction or under-correction that may occur with a fixed step size. Pre-detection of eyeglasses: A glasses-wearing detection step is added before fatigue detection. When glasses are detected on the driver, the closed-eye threshold is automatically increased by a compensation amount, or the method for determining closed-eye based on iris occlusion rate is switched. This solution avoids glare problems at the source, rather than fixing them afterward. Infrared dual-modal eye-closed discrimination: An infrared camera is added as an auxiliary input. In the infrared band, the glasses lenses do not reflect visible light, allowing direct acquisition of eye images without reflection interference for eye-closed discrimination. When the visible light and infrared discrimination results are inconsistent, the infrared result prevails. This scheme eliminates reflection interference at the physical level, offering the highest reliability.

[0042] Please see Figure 2 As shown, it is a logical flowchart of the process of determining the adaptability to complex lighting environments in an embodiment of the present invention. In this embodiment, the adaptability to complex lighting environments is quantified based on the number of key points lost in the behavioral image.

[0043] Specifically, the data acquisition module is used to determine whether the adaptability to complex lighting environments meets the requirements based on whether the number of lost key points in the behavioral image is less than or equal to a preset number of lost points. The data acquisition module is used to determine that the ability to adapt to complex lighting environments does not meet the requirements if the number of lost key points in the behavioral image is greater than the preset number of lost points.

[0044] It is understood that the preset number of lost points is a value determined by those skilled in the art based on the statistical distribution of historical statistical samples. Preferably, the number of lost key points in the behavioral image is negatively correlated with the adaptability to complex lighting environments, wherein, The preset number of lost keypoints is based on the number of historical statistical samples. When the number of lost keypoints in the behavioral image is less than or equal to this value, it indicates that the overexposed areas of the face in the current frame image caused by complex lighting conditions such as strong light from car windows and direct sunlight at noon are small. The keypoint detection model can still extract sufficient effective texture information in key areas such as the eyes and mouth, and the ability to adapt to complex lighting environments is deemed to meet the requirements. When the number of lost keypoints in the behavioral image is greater than this value, it indicates that the brightness of local areas of the facial image is oversaturated due to strong light overexposure in the current frame image. The keypoint detection model cannot extract effective facial texture information in these areas, resulting in a large number of lost keypoints in the eyes and mouth. The ability to adapt to complex lighting environments is deemed to not meet the requirements.

[0045] Regarding the size of the preset number of lost items, those skilled in the art can set a base number based on the number of historical statistical samples, and can also set it according to the actual situation, which will not be elaborated here; Among them, the statistical distribution of historical statistical samples is a standardized reference distribution pre-defined based on the minimum performance requirements of the system's adaptability to complex lighting environments under stable operating conditions. This distribution is derived from the key point positioning accuracy test results of the key point detection model under various standard lighting environments by those skilled in the art, as well as the statistical distribution law of the key point loss rate in different lighting intensity ranges in real road condition data.

[0046] In this embodiment, the system described in this invention determines its adaptability to complex lighting environments by setting a preset number of lost points. Strong light from car windows and direct midday sunlight can cause facial overexposure, leading to oversaturation in localized areas of the facial image. In these areas, the key point detection model cannot extract effective facial texture information, resulting in the loss of key points around the eyes and mouth. By determining the adaptability to complex lighting environments, the system's stability in detecting key facial areas under complex lighting conditions such as strong light and backlight can be quantified. This avoids abnormal extraction of driving state features due to missing key points, improves the accuracy of fatigue driving behavior recognition, and further enhances the accurate recognition of driver facial images during driving.

[0047] Please see Figure 3 As shown, it is a logical flowchart of the cross-frame identity association caching process for face targets in an embodiment of the present invention. In this embodiment, the stable tracking capability of face bounding boxes is quantified based on the number of times the face bounding boxes are lost.

[0048] Specifically, the data processing module is used to determine that the stable tracking capability of the face bounding box meets the requirements if the number of times the face bounding box is lost is less than or equal to a preset first number of times. The data processing module is used to determine that the stable tracking capability of the face bounding box does not meet the requirements if the number of times the face bounding box is lost is greater than the preset first number of times. The data processing module is used to perform cross-frame identity association caching of the face target based on the fact that the number of times the face bounding box is lost is greater than the preset first number of times of loss and less than the preset second number of times of loss. The data processing module is used to preliminarily determine that the fatigue state accurate detection capability does not meet the requirements based on the number of times the face bounding box is lost being greater than or equal to the preset second number of times it is lost, and to determine whether the fatigue state accurate detection capability meets the requirements based on the number of times the fatigue state signal is missed.

[0049] In this embodiment, the preset first number of data loss attempts is less than the preset second number of data loss attempts, wherein... If the number of times the face bounding box is lost is less than or equal to the preset first number of times, the corresponding situation is that the stable tracking capability of the face bounding box meets the requirements. If the number of times the face bounding box is lost is greater than the first preset number of lost boxes but less than the second preset number of lost boxes, the corresponding situation is that due to the bumps and vibrations of the vehicle, the face continuously shifts in the image, making it impossible to maintain continuous and stable face tracking, resulting in frequent loss and re-examination of the face bounding box and frequent clearing of the state cache. If the number of missing face bounding boxes is greater than or equal to the preset second number of missing boxes, the corresponding situation is that due to insufficient effective frontal face samples collected in the early stage of system startup, it is impossible to obtain enough data on eye and mouth opening and closing under frontal face posture to support statistical learning. The adaptive threshold converges slowly, resulting in fluctuations in the EAR and MAR judgment boundaries, leading to missed detection of fatigue driving. The fatigue state signal cannot truly reflect the driver's actual fatigue state.

[0050] It is understood that the preset first loss count and preset second loss count are values ​​determined by those skilled in the art based on the statistical distribution of the number of face bounding box loss counts in historical records of adaptability to complex lighting environments, and in conjunction with the stable tracking capability of face bounding boxes corresponding to different loss counts. Preferably, the number of face bounding box loss counts is negatively correlated with the stable tracking capability of face bounding boxes, wherein, The first loss count is the upper limit of the number of key point loss in the behavior image in the historical statistical distribution, which corresponds to the number of face bounding boxes lost within the normal fluctuation range. When the number of face bounding box loss counts is less than or equal to this value, it indicates that the current driving scenario is stable. The key point detection model can stably output facial feature points of key areas such as eyes and mouth. The face detection model continuously and stably outputs face bounding boxes in continuous video frames. The sliding window and fatigue timer in the closed-eye state accumulate continuously without interruption. The stable tracking capability of the face bounding box meets the requirements of normal system operation. The normal fluctuation range is determined based on the statistical distribution of the number of key points lost per unit time under historical stable driving scenarios. By calculating the average level and standard deviation of the number of lost points under stable scenarios, the average value plus N times the standard deviation is used as the upper limit of normal fluctuation. Among them, the criterion for judging the smooth driving scenario is a standard reference value pre-set by those skilled in the art based on vehicle motion state parameters such as vehicle acceleration sensor data, steering wheel angle change rate, vehicle speed fluctuation amplitude, and the average number of times the system lost track under stable tracking conditions in historical operation.

[0051] The second loss count is preset to be the number of times the face bounding box is lost, corresponding to the deterioration saturation point of the number of lost key points in the behavioral image in the historical statistical distribution. When the number of lost face bounding boxes is greater than or equal to this value, it indicates that the loss of key points has exceeded the normal fluctuation range. Due to the continuous loss of key points, EAR / MAR cannot be calculated or the calculation is unreliable. The adaptive threshold cannot converge due to the lack of effective samples. The false negative rate of fatigue state signal increases significantly. A second judgment is required based on the number of false negatives of fatigue state signal to determine whether the accurate detection capability of fatigue state meets the requirements. The deterioration saturation point is determined based on the paired data of the number of lost face bounding boxes and the number of missed detections of fatigue state signals recorded in history. It is determined by statistically analyzing the upward trend of the number of missed detections of fatigue state signals with the number of lost face bounding boxes and identifying the critical node where the rate of increase of the number of missed detections first significantly increases. Specifically, the process of determining the deterioration saturation point based on the pairing data of the number of lost face bounding boxes and the number of missed detections of fatigue state signals recorded in historical synchronous data is as follows: the historical data is sorted from small to large according to the number of lost face bounding boxes, and the increase rate of missed detections of fatigue state signals is counted segment by segment. When the rate of increase of missed detections after a certain number of lost boxes significantly increases and exceeds the preset rate of increase of missed detections benchmark, then the node is determined as the deterioration saturation point.

[0052] The preset first number of loss counts and the preset second number of loss counts depend on the number of historical statistical samples and the baseline set by those skilled in the art for adapting to complex lighting environments. They can be set according to the actual situation and will not be elaborated here.

[0053] Specifically, the data processing module is used to assign a unique identifier to the face target detected in consecutive video frames, and bind and store the identifier with the corresponding driving status cache; The data processing module is used to keep the identity identifier alive when the face bounding box is lost for a short time, and to restore the binding status data through the identity identifier when the same target is detected again.

[0054] Specifically, a unique identifier is assigned to each face target detected in consecutive video frames. The identifier is then bound and stored with the corresponding closed-eye state sliding window and fatigue timer to form a cross-frame identity association cache. When the face bounding box is lost for a short period, the identifier remains in the cache for a preset survival time, and the bound closed-eye state window data and fatigue timer value remain unchanged. When the same driver target is detected again within the preset survival time, a same-source matching is completed through the identifier, and the closed-eye state window and fatigue timer value accumulated before the loss are directly inherited.

[0055] Specifically, the cross-frame identity association cache is a cross-frame data association storage space composed of an identity identifier and a driving status cache bound to the identity identifier.

[0056] Specifically, the identity identifier is a unique identification code used to identify the same driver target in consecutive video frames.

[0057] Specifically, the driving state cache includes a sliding window for closed-eye state and a fatigue timer.

[0058] Specifically, the closed-eye state window is a state cache queue used to store the closed-eye detection results of several consecutive frames, and the fatigue timer is a timing parameter used to record the cumulative duration of the fatigue state.

[0059] Specifically, the preset survival time is the maximum cache retention time for the identity identifier to remain alive and maintain the bound state data during the period when the face bounding box is lost.

[0060] Specifically, the same-source matching is a process of matching the currently detected face target with the identity identifiers that remain alive in the cache when the face bounding box is detected again, in order to determine whether the current target is the same driver target as before it was lost.

[0061] In this embodiment, the system described in this invention performs cross-frame identity association caching of facial targets by setting a preset first loss count and a preset second loss count. Due to vehicle bumps and vibrations, faces continuously shift in the image, making it impossible to maintain continuous and stable face tracking. This results in frequent loss and re-detection of face bounding boxes, and frequent clearing of the state cache. By performing cross-frame identity association caching of facial targets, a unique identity identifier can be assigned to the same driver target, and the closed-eye state sliding window and fatigue timer are bound to the identity identifier. When the face bounding box is briefly lost, the identity identifier remains alive and maintains the bound state data. When the same target is detected again, the identity identifier is used to complete the same-source matching, directly inheriting the closed-eye state window and fatigue timer value before the loss, achieving seamless continuation of state data. This avoids interruption of driving state cache and repeated reset of fatigue timer caused by bumps, thereby ensuring the continuity of fatigue detection state under bumpy scenarios and further improving the accurate recognition capability of driver face images during driving.

[0062] Please see Figure 4 As shown, it is a logical flowchart of the sample augmentation process of pseudo-labels constrained by the temporal motion of the five senses in an embodiment of the present invention.

[0063] In this embodiment, the accuracy of fatigue state detection is quantified based on the number of missed detections of the fatigue state signal.

[0064] Specifically, the behavior analysis module is used to determine whether the fatigue state accurate detection capability meets the requirements based on whether the number of missed detections of the fatigue state signal is less than or equal to the preset number of missed detections. The behavior analysis module is used to determine that the fatigue state accuracy detection capability does not meet the requirements if the number of missed detections of the fatigue state signal is greater than the preset number of missed detections, and to perform sample augmentation on the pseudo-labels of the temporal motion constraints of the facial features.

[0065] It is understandable that the preset number of missed detections is a value determined by those skilled in the art based on the statistical distribution of the number of missed detections of fatigue state signals in the stable tracking data of historical face bounding boxes, and in combination with the fatigue state accuracy detection capability corresponding to the number of missed detections of different fatigue state signals. Preferably, the number of missed detections of fatigue state signals is negatively correlated with the fatigue state accuracy detection capability. The preset number of missed detections is based on the inflection point where the number of missing face bounding boxes increases significantly with the number of missed detections of fatigue state signals in historical statistical distribution. When the number of missed detections of fatigue state signals is less than or equal to this value, it indicates that the system's adaptive threshold convergence is good, the EAR and MAR judgment boundaries are stable, it can accurately identify the driver's fatigue state, there are few fatigue missed detections, and the fatigue state accurate detection capability meets the recognition requirements. When the number of missed detections of fatigue state signals is greater than this value, it indicates that there are few effective frontal face samples in the early stage of system startup, the adaptive threshold convergence is slow, the EAR and MAR judgment boundaries fluctuate greatly, fatigue state is easily missed, it cannot truly reflect the driver's actual fatigue state, and the fatigue state accurate detection capability does not meet the requirements.

[0066] The preset number of missed detections depends on the number of historically collected samples and the benchmark set by those skilled in the art for the stable tracking capability of face bounding boxes. It can be set according to the actual situation and will not be elaborated here.

[0067] Specifically, the behavior analysis module is used to establish temporal constraint relationships by utilizing the motion trends of key facial features within continuous video frames, and to generate pseudo-label extended samples for the collected frontal face samples. The behavior analysis module is used to combine the pseudo-label extended samples with real-time frontal face samples to participate in the extreme value statistical update of the adaptive threshold.

[0068] Specifically, a temporal constraint relationship is established by utilizing the motion change trend of key points of the eyes and mouth in the time dimension within continuous video frames. Based on the continuous change pattern in the process of eye opening and closing and mouth opening and closing, pseudo-label extended samples that conform to facial motion features are generated for the limited number of frontal face samples that have been collected. The pseudo-label extended samples and the real-time collected frontal face samples are used together to participate in the statistical update of EAR and MAR features.

[0069] Specifically, the temporal constraints include continuous transition constraints for eye opening and closing and inertial constraints for mouth opening and closing motions.

[0070] Specifically, the continuous transition constraint for eye opening and closing requires that there be several continuous transition frames between the open and closed eye states, and the EAR value of each transition frame changes continuously along the time axis.

[0071] Specifically, the inertial constraint of the mouth opening and closing motion is a constraint condition determined based on the physiological characteristic that the mouth opening and closing process is limited by the upper limit of the speed of facial muscle movement and cannot be completed instantaneously.

[0072] Specifically, the pseudo-label extended samples are extended sample data generated based on the collected valid frontal face samples, in accordance with the temporal constraint relationship, and conforming to the continuous transition law of eye opening and closing and the inertial law of mouth opening and closing motion.

[0073] The EAR and MAR feature statistics update is a process of jointly incorporating pseudo-label extended samples and real-time frontal face samples into the extreme value statistics of the adaptive threshold for moving average update, in order to adjust the minimum opening and closing values ​​of the left and right eyes, the maximum opening and closing values ​​of the left and right eyes, the minimum opening and closing value of the mouth, and the maximum opening and closing value of the mouth corresponding to the current driver.

[0074] In this embodiment, the system described in this invention enhances the pseudo-labels of the temporal motion constraints of the facial features by setting a preset number of missed detections. Due to the insufficient number of effective frontal face samples collected in the early stage of system startup, it is impossible to obtain enough data on the opening and closing of the eyes and mouth under the frontal face posture to support statistical learning. The adaptive threshold convergence is slow, resulting in fluctuations in the EAR and MAR judgment boundaries, leading to missed detections of fatigue driving. The fatigue state signal cannot truly reflect the actual fatigue state of the driver. By enhancing the pseudo-labels of the temporal motion constraints of the facial features, the temporal constraint relationship can be established by utilizing the motion trend of key points of the facial features within continuous video frames. Pseudo-label extended samples that conform to the motion law are generated for the collected frontal face samples. Low-weight pseudo-label extended samples are generated for small-angle candidate samples after temporal interpolation and posture compensation to assist in updating. The pseudo-label extended samples are jointly used with the high-quality frontal face samples collected in real time to participate in the extreme value statistics update of the adaptive threshold, expanding the available sample size in the startup stage, suppressing the disordered oscillation of the EAR and MAR judgment boundaries, accelerating the convergence of the adaptive threshold, improving the accuracy of fatigue state detection in the system startup stage, and further improving the accurate recognition capability of driver face images during driving.

[0075] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A driver behavior detection system based on facial recognition, characterized in that, include: The data acquisition module is used to collect images of the driver's behavior during driving, and to determine whether the adaptability to complex lighting environments meets the requirements based on the number of key points lost in the behavior images. The data processing module is used to process the behavior image to output the face bounding box and several key point coordinates, so as to perform cross-frame identity association caching of the face target based on the number of times the face bounding box is lost. The behavior analysis module is used to extract facial features from standardized face images and perform behavior recognition to obtain recognition results. It then uses the results to enhance the pseudo-labels of the temporal motion constraints of the facial features based on the number of missed detections of fatigue state signals. The decision confirmation module is used to verify the time continuity of the fatigue state signal and the attention deviation signal to confirm abnormal behavior. The early warning control module is used to generate early warning information based on the confirmation result of the abnormal behavior and the identity authentication result, so as to execute the corresponding early warning strategy.

2. The driver behavior detection system based on face recognition according to claim 1, characterized in that, The data processing module is used to input the behavior image into the main detection model and output the face bounding box and the coordinates of several key points; The data processing module is used to correct the original face region into a standardized face image based on the coordinates of the key points using an affine transformation algorithm.

3. The driver behavior detection system based on face recognition according to claim 1, characterized in that, The behavior analysis module is used to input the standardized face image into the key point detection model and output the coordinates of several facial key points and three-dimensional head pose parameters. The behavior analysis module is used to input the standardized face image into the face feature extraction model, output the face feature vector, and compare it with the driver face database to confirm the driver's identity authentication result.

4. The driver behavior detection system based on face recognition according to claim 3, characterized in that, The behavior analysis module is used to dynamically update the current driver's eye-closing and mouth-opening judgment thresholds based on historical EAR / MAR data when the head posture is within the frontal range. The behavior analysis module is used to compare the real-time EAR / MAR with a fixed threshold and an adaptive threshold respectively. When any of the eye-closing or mouth-opening judgment conditions are met, the corresponding fatigue state signal is output. The behavior analysis module is used to directly output an attention deviation signal when the head posture angle exceeds the range of the frontal face, and skips the eye and mouth state discrimination.

5. The driver behavior detection system based on face recognition according to claim 1, characterized in that, The decision confirmation module is used to record the corresponding timestamp when abnormal behavior is detected for the first time, and to determine whether the duration of the abnormal behavior exceeds a preset time threshold during the process of the abnormal behavior. The corresponding abnormal behavior is confirmed only when the threshold is exceeded. The decision confirmation module is used to collect multiple frames of images at preset intervals during the continuous process of abnormal behavior to form a sequence of evidence of abnormal behavior. The decision confirmation module is used to establish a sliding window for the closed-eye state and cache the closed-eye detection results for several consecutive frames. The decision confirmation module is used to calculate the deviation between the current EAR value and the adaptive threshold when the historical frame determines that the eyes are closed and the current frame detects that the eyes are open. If the deviation is less than the preset range, it is determined to be eye glare interference, the closed eye state is maintained and threshold self-repair is performed.

6. The driver behavior detection system based on face recognition according to claim 1, characterized in that, The data acquisition module is used to determine whether the adaptability to complex lighting environments meets the requirements based on whether the number of lost key points in the behavioral image is less than or equal to a preset number of lost points. The data acquisition module is used to determine that the ability to adapt to complex lighting environments does not meet the requirements if the number of lost key points in the behavioral image is greater than the preset number of lost points.

7. The driver behavior detection system based on face recognition according to claim 6, characterized in that, The data processing module is used to determine that the stable tracking capability of the face bounding box does not meet the requirements if the number of times the face bounding box is lost is greater than a preset first number of times. The data processing module is used to perform cross-frame identity association caching of the face target based on the fact that the number of times the face bounding box is lost is greater than the preset first number of times of loss and less than the preset second number of times of loss. The data processing module is used to preliminarily determine that the fatigue state accurate detection capability does not meet the requirements based on the number of times the face bounding box is lost being greater than or equal to the preset second number of times it is lost, and to determine whether the fatigue state accurate detection capability meets the requirements based on the number of times the fatigue state signal is missed.

8. The driver behavior detection system based on face recognition according to claim 7, characterized in that, The data processing module is used to assign a unique identifier to the face target detected in consecutive video frames, and bind and store the identifier with the corresponding driving status cache. The data processing module is used to keep the identity identifier alive when the face bounding box is lost for a short time, and to restore the binding status data through the identity identifier when the same target is detected again.

9. The driver behavior detection system based on face recognition according to claim 8, characterized in that, The behavior analysis module is used to determine whether the fatigue state accurate detection capability meets the requirements based on the number of missed detections of the fatigue state signal being less than or equal to the preset number of missed detections. The behavior analysis module is used to determine that the fatigue state accuracy detection capability does not meet the requirements if the number of missed detections of the fatigue state signal is greater than the preset number of missed detections, and to perform sample augmentation on the pseudo-labels of the temporal motion constraints of the facial features.

10. The driver behavior detection system based on face recognition according to claim 9, characterized in that, The behavior analysis module is used to establish temporal constraint relationships by utilizing the motion trends of key facial features within continuous video frames, and to generate pseudo-label extended samples for the collected frontal face samples. The behavior analysis module is used to combine the pseudo-label extended samples with real-time frontal face samples to participate in the extreme value statistical update of the adaptive threshold.

Images