Driver fatigue behavior detection method and system based on video pose invariance

Abstract

The application discloses a driver fatigue behavior detection method and system based on video posture invariance, and relates to the technical field of computer vision. The application proposes a key frame selection model based on facial geometric information and a head and face action information fusion space-time network. First, the driver video captured by the vehicle-mounted camera is subjected to sequential processing, and image preprocessing is performed. Then, the key frame selection model based on facial geometric information is constructed based on the geometric features of the facial key points and a two-stage decision mechanism, and the key frames in the video sequence are extracted. Finally, the facial action modalities under any posture are extracted based on the facial forward processing, and the head posture attributes obtained based on the head posture estimation are combined to construct the head and face action information fusion space-time network, which is used for detecting the yawning, speaking, normal and other driver states. The application fully considers the head posture attributes, has high posture robustness, and can effectively distinguish the yawning and other fatigue behaviors from other driver states.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for detecting driver fatigue behavior based on video pose invariance. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Vision-based Advanced Driver Assistance Systems (ADAS) primarily provide functions such as environmental perception, driver monitoring, warnings, and assisted vehicle control. In recent years, ADAS has become a hot research topic for improving road safety and traffic efficiency. Fatigue driving is a dangerous driving state; drivers in a fatigued state often experience physiological and psychological dysfunction, resulting in a decline in driving skills. Fatigue driving is one of the leading causes of serious traffic accidents worldwide. Vision-based fatigue driving detection methods focus on behavioral characteristics, relying on cameras to capture video images of the driver. These images are typically non-contact, easy to use, and can be applied to monitor the driver's condition and provide timely warnings, demonstrating high practical value in reducing traffic accidents.

[0004] Over the past few decades, numerous researchers have proposed various methods for detecting driver fatigue to help drivers drive safely and improve traffic safety. Behavioral characteristics of fatigued driving include blinking, nodding, closing eyes, and yawning. Among these, yawning is one of the main manifestations of fatigue. In real-world driving environments, due to high real-time requirements, complex facial expressions, and variable head postures, existing methods struggle to accurately and reliably detect fatigue behaviors such as yawning. Therefore, how to efficiently and accurately detect fatigue behavior in real-world driving environments has become an urgent problem to be solved. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a driver fatigue behavior detection method and system based on video pose invariance, solving the problems of excessive video frame redundancy, inability to effectively detect in arbitrary poses, and inability to accurately distinguish between yawning and similar behaviors in existing technologies, thereby improving the accuracy and robustness of fatigue behavior detection.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] The first aspect of this invention provides a method for detecting driver fatigue behavior based on video pose invariance, comprising the following steps:

[0008] The acquired driver video is serialized, and the video image data is preprocessed.

[0009] Design a keyframe selection model based on facial geometry information and implement keyframe selection;

[0010] Based on the selected keyframes, facial motion modalities under arbitrary poses are extracted using facial forward processing, and a spatiotemporal network for head-face motion information fusion is constructed in conjunction with head pose attributes obtained based on head pose estimation to detect fatigue behavior.

[0011] Furthermore, the video image data is preprocessed, including image denoising, histogram normalization, facial landmark detection, and face segmentation.

[0012] Furthermore, a face detection algorithm is used to detect the driver's face region and segment the driver's head and face action region; then, image denoising is performed based on a fast median filtering algorithm, and illumination normalization is performed using contrast-limited adaptive histogram equalization (CLAHE); finally, facial landmark detection is performed using a cascaded regression tree (ETR) based on the dlib library.

[0013] Furthermore, the design of a keyframe selection model based on facial geometric information and the specific steps for implementing keyframe selection are as follows: extracting facial geometric features based on facial key points and designing a two-stage decision mechanism, thereby constructing a keyframe selection model based on facial geometric information and extracting keyframes from the video sequence.

[0014] Furthermore, the specific steps of the first stage decision mechanism in the two-stage decision mechanism are as follows: First, based on the set of facial key points, calculate the distance ratio and angle relationship between points in each video key frame to construct facial geometric features; then, use Euclidean distance to calculate the similarity between consecutive frames to obtain a set of similarity between consecutive frames; finally, determine the similarity threshold to select candidate key frames.

[0015] Furthermore, the specific steps of the second-stage decision mechanism in the two-stage decision mechanism are as follows: remove video frames with outlier characteristics from the candidate keyframe queue to obtain a keyframe set; in this stage, frame selection is performed based on two similarity metrics and outlier frame detection; among them, the similarity metrics are Euclidean distance and root mean square error, and the outlier frame detection uses median absolute deviation.

[0016] Furthermore, the specific steps for extracting facial action modalities under arbitrary poses based on facial forward processing are as follows: an encoder-decoder backbone network is used for representation learning, and on this basis, two auxiliary mechanisms, illumination preservation and attention, are introduced to generate realistic facial images with illumination preservation.

[0017] Furthermore, the specific steps for obtaining head pose attributes based on head pose estimation are as follows: design a head pose estimation method based on Squeeze-Net, and normalize the Euler angle representation of the head pose using sine and cosine functions to obtain the head pose attributes.

[0018] Furthermore, a spatiotemporal network for fusing head and face motion information is constructed based on a 3D convolutional network. This network integrates head pose attributes and facial motion modalities into a dual-channel classifier to achieve pose invariance.

[0019] Furthermore, the spatiotemporal network for fusion of head and face motion information obtains facial motion information through 3D convolutional operations. The feature maps in the convolutional layers are generated from multiple adjacent frames in the previous layer. The 3D convolutional network fuses the head pose vector representation and facial motion information into a dual-channel classifier and uses a fully connected layer for feature mapping.

[0020] A second aspect of the present invention provides a driver fatigue behavior detection system based on video pose invariance, comprising:

[0021] The preprocessing module is configured to serialize the acquired driver video and preprocess the video image data.

[0022] The keyframe module is configured to design a keyframe selection model based on facial geometry information and to implement keyframe selection.

[0023] The fatigue behavior detection module is configured to extract facial action modalities in arbitrary poses based on selected keyframes and facial forward processing, and to construct a spatiotemporal network for head-face action information fusion in conjunction with head pose attributes obtained based on head pose estimation, in order to detect fatigue behavior.

[0024] The above one or more technical solutions have the following beneficial effects:

[0025] (1) This invention proposes a key frame selection algorithm based on facial geometric information. This algorithm has the advantage of fast calculation speed, which can efficiently select key frames and improve the processing efficiency of network structure.

[0026] (2) This invention introduces a face forward processing operation for the first time, which can weaken the influence of head posture and improve the posture robustness of the network structure.

[0027] (3) This invention proposes a spatiotemporal network for head and face motion information fusion based on a three-dimensional convolutional network, which is used to extract spatiotemporal features and detect various driving states, and can effectively improve the accuracy and robustness of yawn detection.

[0028] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0029] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0030] Figure 1 This is a flowchart of the driver yawning detection method based on video pose invariance designed in Embodiment 1 of the present invention;

[0031] Figure 2 This is a network structure diagram of the driver yawning detection method based on video pose invariance designed in Embodiment 1 of the present invention;

[0032] Figure 3 This is a structural diagram of the keyframe selection method based on facial geometric information designed in Embodiment 1 of the present invention;

[0033] Figure 4 This is a demonstration diagram of the keyframe selection results in Embodiment 1 of the present invention;

[0034] Figure 5 This is a diagram of the facial forward processing network structure designed in Embodiment 1 of the present invention;

[0035] Figure 6 This is a demonstration image of the face forward processing result based on the LFW dataset in Embodiment 1 of the present invention;

[0036] Figure 7 The data consists of three types of facial action sequences from the YawDDE dataset preprocessed in Embodiment 1 of this invention. Detailed Implementation

[0037] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0038] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0039] Example 1:

[0040] Pose invariance refers to the ability to forward-process driver video images in any pose under the viewpoint captured by an in-vehicle camera, thereby enhancing the model's ability to handle pose changes and strengthening the representation of facial spatial features. Embodiment 1 of this invention provides a driver fatigue behavior detection method based on video pose invariance. Taking yawning as an example, it proposes a keyframe selection model based on facial geometric information and a spatiotemporal network fusing head and face motion information, aiming to efficiently and accurately detect driver states such as yawning, talking, and normal behavior. The overall flowchart is as follows: Figure 1 As shown in Figure 2, the specific network structure includes the following steps:

[0041] Step 1: Serialize the acquired driver video and preprocess the video image data.

[0042] As a further technical solution, video image data is preprocessed, including image denoising, histogram normalization, facial landmark detection, and face segmentation.

[0043] Video serialization and image preprocessing are crucial steps in this research. Since this invention identifies driver fatigue based on yawning behavior, the recognition network requires high timeliness. First, the original video containing driving behavior is serialized. Second, because the target is the driver's driving state, redundant information such as background should be removed as much as possible, focusing on the driver's facial region. This invention uses MTCNN and LBP face detection algorithms to detect the driver's face region and segment the driver's head and face movement areas. Then, due to the influence of vehicle mechanical vibration and lighting changes in the real driving environment, noise and interference from the onboard camera during recording should be minimized. This invention uses a fast median filtering algorithm for image denoising and CLAHE for illumination normalization. Furthermore, since data features should have the same metric scale, this invention performs data normalization and unifies the image size to 112×112. Finally, based on the dlib library, ETR is used for facial keypoint detection (68 points), preparing for subsequent keyframe selection algorithms based on facial geometry information.

[0044] Step 2: Design a keyframe selection model based on facial geometry information and implement keyframe selection.

[0045] As a further technical solution, facial geometric features are extracted based on facial key points and a two-stage decision mechanism is designed. Then, a keyframe selection model based on facial geometric information is constructed, and keyframes in the video sequence are extracted.

[0046] Because many adjacent frames in the original video frame sequence have very little difference in information, there are a large number of redundant frames, which can lead to a decrease in the accuracy of action classification. This invention designs a keyframe selection model based on facial geometric information and implements keyframe selection. Figure 3 A structural diagram of a keyframe selection method based on facial geometry information is shown. This method can select keyframes from a series of raw video frames F = {F...} i Extract a series of keyframes K = K from the set {i = 1, ..., N}. j ,j=1,…,M}; where M represents the number of keyframes selected, and N represents the number of frames in the original video frame sequence. This model mainly includes extracting facial geometric features based on facial keypoints and designing a two-stage decision mechanism, which consists of a first stage and a second stage (STAGE-1 and STAGE-2):

[0047] In STAGE-1, the distance ratio between points in each video keyframe is calculated based on the set of facial landmarks. ratio ), angular relationship (Dist) ratio First, facial geometric features g are constructed; then, the similarity D between consecutive frames is calculated using Euclidean distance. i This allows us to obtain the set of inter-frame similarity values ​​for consecutive frames, D = {D}. i Let the integers be i = 1, ..., N-1. Then, a similarity threshold T needs to be determined. D To select candidate keyframes, where T D The mean (mean value) represents the average level of similarity between consecutive frames. The relevant calculation formula is as follows:

[0048]

[0049] Angle ralat =θ(v a,b ,v b,c (2)

[0050] g i = Dist ratio,i ⊕ Angle ralat,i (3)

[0051] g i+1 = Dist ratio,i+1 ⊕ Angle ralat,i+1 (4)

[0052]

[0053] T D =Mean(D) (6)

[0054] Where, v a,and v b,c Let θ represent the geometric vectors defined by keypoints a, b and b, c, respectively; θ represents v a,b and v b,c The included angle between them; ⊕ indicates feature splicing. When D is satisfied... i >T D At that time, it was believed that F i With F i+1 The similarity between them is small, so F i Add to the candidate keyframe queue.

[0055] In STAGE-2, outlier video frames are removed from the candidate keyframe queue, resulting in a keyframe set K. This stage selects frames based on two similarity metrics and outlier detection. The similarity metrics used are Euclidean distance (ED) and root mean square error (RMSE), while outlier detection employs median absolute deviation (MAD). First, for any two consecutive candidate keyframes K... j and K j+1 Calculate the RMSE and ED values, and then base the result on RMSE(K). j ,K j+1 ), ED(K j K j+1 From α and β, the frame selection metrics α and β for this stage can be obtained. The relevant calculation formulas are as follows:

[0056]

[0057]

[0058]

[0059] α=Mean(ED)-MAD(ED),β=Mean(RMSE)-MAD(RMSE)(10)

[0060] Where m represents the set of candidate keyframes C K The size of K. When candidate keyframes simultaneously satisfy ED>α and RMSE>β, they are considered to be relatively different. j These will be retained as keyframes, ultimately resulting in a keyframe set K. Figure 4 A demonstration image showing the keyframe selection results demonstrates the effectiveness of the algorithm.

[0061] Step 3: Based on the selected keyframes, extract facial action modalities in any pose using facial forward processing, and construct a spatiotemporal network for head-face action information fusion by combining the head pose attributes obtained based on head pose estimation, in order to detect fatigue behavior.

[0062] As a further technical solution, this invention introduces facial forward processing into yawn detection, which can be used to extract facial action modalities in any posture; then, it combines head posture attributes to construct a spatiotemporal network for head-face action information fusion to complete yawn behavior detection.

[0063] First, this invention introduces facial forward-facing processing into yawn detection for the first time; this processing can weaken the influence of head posture on facial appearance and introduce head posture attributes to improve the posture robustness of the network structure. Figure 5 The network architecture for facial feedforward processing is shown. This network uses a flow-based U-Net as the encoder-decoder backbone for representation learning, and incorporates illumination preservation (G) on top of this. ip ) and attention (G att Two auxiliary mechanisms are used to generate realistic face images that retain illumination. Figure 6 The demonstration image shows the results of face forward processing based on the LFW dataset, which fully demonstrates the superior performance and multi-pose applicability of the face forward processing network.

[0064] Secondly, this invention proposes a spatiotemporal network for fusing head and face motion information based on a 3D convolutional network. This network fuses head pose attributes and facial motion modalities into a dual-channel classifier to achieve pose invariance and improve classification accuracy. On one hand, head pose can be effectively detected using CNN branches or a general 3D head model, with the detection results represented by Euler angles (Yaw, Pitch, Roll). This invention designs a head pose estimation method based on Squeeze-Net, which still has pose estimation capabilities even if facial keypoint detection methods are ineffective. To achieve head pose feature encoding, the Euler angles (Yaw, y; Pitch, p; Roll, r) of the head pose are normalized to a head pose vector representation P (where P = y) using sine and cosine functions. s ,y c ,p s, p c ,r s ,r cOn the other hand, in recent years, researchers have introduced various action recognition networks into yawn detection, achieving significant progress in both accuracy and speed. Among them, two-stream fusion networks and 3D convolutional networks are widely used frameworks in action recognition. Compared with two-stream fusion networks, 3D convolutional networks use multiple consecutive video frames as input, offering advantages in spatiotemporal feature extraction such as faster computation speed and higher accuracy. Experiments have shown that a 3×3×3 3D convolutional kernel can be used to extract the most representative spatiotemporal features. This invention obtains facial action information through 3D convolutional operations, with feature maps in the convolutional layers generated from multiple adjacent frames in the previous layer. Based on this, the network fuses head pose vector representation and facial action information into a dual-channel classifier, employing a fully connected layer for feature mapping. Ultimately, the optimal yawn detection network recognition performance is obtained.

[0065] Finally, regarding the training part of the spatiotemporal network for head and face information fusion, this invention extracts 16 frames from each video as input, and the 3D convolutional network uses pre-trained parameters; the deep neural network is trained using the backpropagation algorithm, and the network parameters are iteratively optimized using the standard multivariate cross-entropy loss function, as shown in Equation 11:

[0066]

[0067] Where the number of samples is K, and the value of sample i is 1≤i≤K; y i p represents the label value. i These are predicted values.

[0068] The experiment selected SGD with momentum as the network optimizer, with a training period of 100 epochs, an initial learning rate of 0.01, and a momentum of 0.5.

[0069] The effects of this invention can be further illustrated by the following experiments:

[0070] Experimental conditions

[0071] This invention was tested on a GTX 2080Ti GPU to complete model training. The proposed method is based on pytorch=1.8.0, python=3.6, and torchvision=0.9.0.

[0072] Experimental content and results

[0073] The experiments of this invention were conducted using a standard public yawn detection dataset—YawDD—to train and test the system and verify the effectiveness of the method. This dataset contains 351 video clips recorded by in-vehicle cameras, collecting a series of driving action videos from volunteers of different genders, ages, nationalities, and ethnicities. The vehicles were parked in an outdoor environment with natural and variable lighting conditions. Glasses status included: no glasses, glasses, and sunglasses. Based on the camera's installation location, the videos were divided into two groups: one group had cameras mounted above the dashboard; the other group had cameras mounted below the rearview mirror. The YawDD dataset contains three or four behaviors, including normal, speaking or singing, yawning, and speaking while yawning; most video clips in the dataset lasted longer than 1 minute and contained multiple facial movements. To construct the YawDDE dataset for evaluation, the videos were segmented into video clips containing only one action. Facial action categories were divided into normal (N), speaking (T), and yawning (Y), with the rare occurrence of singing or laughing being classified as speaking (T). Table 1 records the 1029 image sequences collected by YawDDE and the number of each type of action sequence.

[0074] Table 1 shows the number of action sequences of each type in the YawDDE dataset.

[0075]

[0076] Example images of three types of facial action sequences ((a) normal, (b) speaking, and (c) yawning) after data preprocessing are shown below. Figure 7 middle.

[0077] This invention conducted the following two experiments based on the YawDDE dataset:

[0078] Experiment 1: Experiments were conducted on the YawDDE dataset to demonstrate that the keyframe selection algorithm based on facial geometry information can effectively select video keyframes. The experiment involved three processing methods: WITHOUT, indicating that no keyframe selection algorithm was used, and the video was simply segmented into a sequence of video frames at 10fps; STAGE-1, indicating that only the first stage of the keyframe selection algorithm based on facial geometry information was used, i.e., selecting a sequence of candidate keyframes; and STAGE-2, indicating that the keyframe selection algorithm based on facial geometry information was used, i.e., the sequence of video keyframes.

[0079] This invention was tested using three processing methods, demonstrating that the proposed algorithm can effectively select key frames in driving videos and improve the overall processing efficiency of the network structure. The detection accuracy results for different key frame selection processing methods are shown in Table 2.

[0080] Table 2. Experimental results for different keyframe selection processes (unit: %)

[0081] category Ave (%) Y(％) T(％) N(％) WITHOUT 75.3 85.2 72.6 68.1 STAGE-1 80.4 87.4 80.3 73.3 STAGE-2 81.9 90.4 81.3 73.9

[0082] The table presents the selection results for WITHOUT, validating the algorithm's ability to effectively improve the accuracy of various facial movements. Algorithm STAGE-1, used to remove video frames with minimal differences to obtain candidate keyframes, outperforms WITHOUT, improving the accuracy of various facial movements by 2.2%–7.7%. Building upon STAGE-1, algorithm STAGE-2 uses a similarity metric and outlier detection to eliminate abnormal values ​​in candidate keyframes, obtaining the final video keyframes. It can be seen that, compared to WITHOUT and STAGE-1, the keyframe selection algorithm based on facial geometry information achieves the best recognition performance, validating the algorithm's effectiveness.

[0083] Experiment 2: In this experiment, the method proposed in this invention was compared with some other existing methods. Table 3 shows the detection results (unit: %) of the method proposed in this invention and four advanced methods on the YAWDDE dataset.

[0084] Table 3 shows the detection results (in %) of the proposed method and four advanced methods on the YAWDDE dataset.

[0085] method Ave (%) Y(％) T(％) YT(%) N(％) TFES N / A 83.3 78.2 N / A N / A 2DCNN+RT N / A 86.8 77.3 N / A N / A 3DCNN 75.8 82.5 72.3 64.1 N / A 3D-LTS 83.4 92.1 81.2 77.3 N / A OURS 85.5 93.0 88.3 N / A 74.5

[0086] As shown in Table 3, in the validation based on the YawDDE dataset, most image-based and deep learning-based models performed poorly due to a lack of spatiotemporal information, facing considerable challenges in detecting large pose changes and facial movements similar to yawning distortions. The algorithm published in this invention outperforms existing methods in terms of overall recognition rate and overall performance, showing improvements of over 2.1%, 0.9%, and 7.1% in average accuracy, yawning accuracy, and speaking accuracy, respectively. This algorithm demonstrates superior performance in recognizing various facial movements and can effectively reduce false detections; further, it illustrates that video-based methods can effectively extract sufficient spatiotemporal motion features and achieve dynamic yawning detection.

[0087] Example 2:

[0088] Embodiment 2 of the present invention provides a driver fatigue behavior detection system based on video pose invariance, comprising:

[0089] The preprocessing module is configured to serialize the acquired driver video and preprocess the video image data.

[0090] The keyframe module is configured to design a keyframe selection model based on facial geometry information and to implement keyframe selection.

[0091] The fatigue behavior detection module is configured to extract facial action modalities in arbitrary poses based on selected keyframes and facial forward processing, and to construct a spatiotemporal network for head-face action information fusion in conjunction with head pose attributes obtained based on head pose estimation, in order to detect fatigue behavior.

[0092] The steps and methods involved in the above embodiment two correspond to those in embodiment one. For specific implementation details, please refer to the relevant description section of embodiment one.

[0093] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0094] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for detecting driver fatigue behavior based on video pose invariance, characterized in that, Includes the following steps: The acquired driver video is serialized, and the video image data is preprocessed. Design a keyframe selection model based on facial geometric information and implement keyframe selection. The specific steps are as follows: extract facial geometric features based on facial key points and design a two-stage decision mechanism, thereby constructing a keyframe selection model based on facial geometric information and extracting keyframes from the video sequence. The specific steps of the first stage of the two-stage decision mechanism are as follows: Based on the set of facial key points, calculate the distance ratio and angle relationship between points in each video key frame to construct facial geometric features; then, use Euclidean distance to calculate the similarity between consecutive frames to obtain a set of similarity between consecutive frames; finally, determine the similarity threshold to select candidate key frames. The specific steps of the second-stage decision mechanism in the two-stage decision mechanism are as follows: remove video frames with outlier characteristics from the candidate keyframe queue to obtain a keyframe set; in this stage, frame selection is based on two similarity metrics and outlier frame detection; among them, the similarity metrics are Euclidean distance and mean squared error, and the outlier frame detection uses median absolute deviation. Based on the selected keyframes, facial action modalities under arbitrary poses are extracted using facial forward processing, and a spatiotemporal network for head-face action information fusion is constructed in conjunction with head pose attributes obtained based on head pose estimation to detect fatigue behavior. The specific steps for extracting facial action modalities under arbitrary poses based on facial forward processing are as follows: an encoder-decoder backbone network is used for representation learning, and on this basis, two auxiliary mechanisms, illumination preservation and attention, are introduced to generate realistic facial images with illumination preservation. A spatiotemporal network for fusing head and face motion information is constructed based on a 3D convolutional network. This network integrates head pose attributes and facial motion modalities into a dual-channel classifier to achieve pose invariance.

2. The driver fatigue behavior detection method based on video pose invariance as described in claim 1, characterized in that, A face detection algorithm is used to detect the driver's face region and segment the driver's head and face action region. Then, a fast median filtering algorithm is used for image denoising, and contrast-limited adaptive histogram equalization is used for illumination normalization. Finally, a cascaded regression tree is used based on the dlib library to detect facial landmarks.

3. The driver fatigue behavior detection method based on video pose invariance as described in claim 1, characterized in that, The specific steps for obtaining head pose attributes based on head pose estimation are as follows: design a head pose estimation method based on Squeeze-Net, and normalize the Euler angle representation of the head pose using sine and cosine functions to obtain the head pose attributes.

4. A driver fatigue behavior detection system based on video pose invariance, characterized in that, include: The preprocessing module is configured to serialize the acquired driver video and preprocess the video image data. The keyframe module is configured to design a keyframe selection model based on facial geometric information and implement keyframe selection. The specific steps are: extracting facial geometric features based on facial key points and designing a two-stage decision mechanism, thereby constructing a keyframe selection model based on facial geometric information and extracting keyframes from the video sequence. The specific steps of the first stage of the two-stage decision mechanism are as follows: Based on the set of facial key points, calculate the distance ratio and angle relationship between points in each video key frame to construct facial geometric features; then, use Euclidean distance to calculate the similarity between consecutive frames to obtain a set of similarity between consecutive frames; finally, determine the similarity threshold to select candidate key frames. The specific steps of the second-stage decision mechanism in the two-stage decision mechanism are as follows: remove video frames with outlier characteristics from the candidate keyframe queue to obtain a keyframe set; in this stage, frame selection is based on two similarity metrics and outlier frame detection; among them, the similarity metrics are Euclidean distance and mean squared error, and the outlier frame detection uses median absolute deviation. The fatigue behavior detection module is configured to extract facial action modalities in any pose based on selected keyframes and facial forward processing, and to construct a spatiotemporal network for head-face action information fusion in conjunction with head pose attributes obtained based on head pose estimation, so as to detect fatigue behavior. The specific steps for extracting facial action modalities under arbitrary poses based on facial forward processing are as follows: an encoder-decoder backbone network is used for representation learning, and on this basis, two auxiliary mechanisms, illumination preservation and attention, are introduced to generate realistic facial images with illumination preservation. A spatiotemporal network for fusing head and face motion information is constructed based on a 3D convolutional network. This network integrates head pose attributes and facial motion modalities into a dual-channel classifier to achieve pose invariance.

Images