Method for assessing driver drowsiness based on view angle correction and improved vivit

Abstract

Method for assessing driver drowsiness, comprising: S1: constructing an FGR-ViViT model (a Video Vision Transformer (ViViT) model with a part selection module incorporated in the temporal transformer

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of facial image recognition, and in particular, to a method for assessing driver drowsiness based on view angle correction and improved video vision transformation (ViViT). BACKGROUND

[0002] Fatigue is a biologically dangerous behavior, which usually refers to the accumulation of sleep deprivation, excessive wakefulness, and disruption of circadian rhythms, which may result from persistent fatigue. According to the World Health Organization (WHO), one person dies in a traffic accident every 24 seconds, and fatigue is the leading cause of driving accidents. Fatigue driving accounts for about 20% of all traffic accidents, 40% of major traffic accidents and 83% of traffic fatalities, and is a key factor affecting traffic safety. A detection and early warning system for driver fatigue will reduce the chance of accidents, and fatigue detection has become an important part of automotive safety research.

[0003] At present, machine vision-based detection can divide behavioral carriers into two classes: image and video. The mainstream idea of the first type of method is to capture a current behavior of a driver and directly classify the current behavioral state. Phan et al. analyze the fatigue state of a person in an image in two ways: the first is to compute the eye aspect ratio (EAR) and mouth aspect ratio (MAR) of the driver's face in each video frame, and the second is to classify the captured pictures using MobileNet-V2 and ResNet-50V2 models. However, fatigue is a continuous process, and pictures often only provide static transient information and cannot capture the dynamic changes in the driver. Besides, external behaviors under fatigue may vary from person to person, which may lead to misjudgment. The mainstream idea of the second type of method is to analyze the video behavior of driver during the current period of time and determine the fatigue state of the driver by using image frame sequences. Image sequences can capture dynamic changes in the driver state, including transient minor changes, which can help to identify the fatigue state more accurately. By analyzing video or image sequences, multiple features can be considered at the same time, which improves the accuracy of determining the fatigue state. Chandiwala et al. quantified drowsiness by locally calculating a sequence of changes in EAR and MAR values, which is relatively simple, requiring only the locations of key points of the eyes and mouth. Specific formulas are applied to derive the results through threshold-based judgment, thus providing timely monitoring of the fatigue state. However, different shapes and characteristics of eyes and mouths of different people may lead to different degrees of changes in EAR and MAR values for different people under the same fatigue state. Therefore, it is difficult to determine universal thresholds and misjudgment is highly possible. Experiments have shown that this type of method has better performance.

[0004] In 2020, researchers introduced the Vision Transformer (ViT) model, and many subsequent models based on ViT have demonstrated advanced performance in various image recognition tasks, making it an effective alternative to Convolutional Neural Networks (CNN). ViViT builds upon ViT by incorporating spatial and temporal processing, exploring the application of ViT in video classification. It extracts spatio-temporal markers from an input video using spatio-temporal features of a frame sequence, decomposing the input spatial and temporal dimensions, and encoding them through a series of transformer layers. Arnab et al. conducted a comprehensive ablation study on ViViT, achieving state-of-the-art results on multiple video classification benchmarks, including Kinetics 400 and 600, Epic Kitchens, Something-Something v2, and Moments in Time. The experimental results surpassed previous methods based on deep 3D convolutional networks.

[0005] While the current use of ViViT models shows relatively good recognition of driver fatigue states, most cameras used for collecting driver head information are positioned at fixed angles in practical operations. The datasets also consist of driver facial information captured from a specific fixed angle. This makes the fatigue detection model based on deep learning either achieve the best performance in the corresponding viewing angle, which limits the placement of the camera and the head posture of the driver, or the fatigue detection model based on deep learning needs to be trained with data from various viewing angles to achieve the best performance, which means that the training set includes various head postures and angles of the driver. However, due to the individual differences and personal habits, such an approach would require a huge amount of training data to get a better result. Additionally, factors such as glasses occlusion and lighting conditions can also impact fatigue recognition. In summary, in the prior art, driver fatigue detection imposes high requirements on the camera placement and driver head angles, and the detection accuracy will also be affected by factors such as glasses occlusion and lighting. As a result, the detection accuracy and stability in practical applications are unsatisfactory.

[0006] Therefore, during driver fatigue detection, how to reduce the dependence on the camera placement and driver head angles and reduce the influence of factors such as glasses occlusion and lighting conditions to ensure the detection accuracy and stability in practical applications becomes the problem that needs to be solved urgently. SUMMARY

[0007] To address the shortcomings of the prior art, the present disclosure provides a method for assessing driver drowsiness based on view angle correction and improved ViViT, which can reduce the dependence on the camera placement and driver head angles and reduce the influence of factors such as glasses occlusion and lighting conditions to ensure the detection accuracy and stability in practical applications.

[0008] To solve the above technical problems, the present disclosure adopts the following technical solutions:

[0009] A method for assessing driver drowsiness based on view angle correction and improved ViViT is provided, including the following steps:

[0010] SI: constructing an FGR-ViViT model based on a ViViT model, for fatigue recognition based on image sequences of eye-brow-mouth shapes from a frontal view;

[0011] S2: acquiring facial image sequences of drivers and preprocessing each image to obtain training image sequences;

[0012] S3: training the FGR-ViViT model using the training image sequences; and

[0013] S4: collecting a real-time facial image sequence of a driver, performing preprocessing as in S2 to obtain a real-time image sequence, and performing fatigue recognition on the driver using the trained FGR-ViViT model.

[0014] In S2, the preprocessing includes:

[0015] S21: extracting feature points on each facial image and performing 3D positioning and frontal pose correction to obtain a mesh of facial feature points from the frontal view; and

[0016] S22: orthographically projecting the mesh of the facial feature points after the correction, extracting shapes of eyes, eyebrows, and mouth, constructing eye-brow-mouth line drawings based on corrected coordinates, and using image sequences of the obtained eye-brow-mouth line drawings as the training image sequences.

[0017] Compared with the prior art, the present disclosure has the following beneficial effects:

[0018] 1. With this method, the acquired facial image sequence of the driver is preprocessed. The preprocessing specifically includes two aspects: first, locating facial 3D feature points and correcting them to a frontal view; second, using the image sequences of the eye-brow-mouth line drawings of the frontal face as a training set for the FGR-ViViT model. Subsequently, the FGR-ViViT model is trained. In practical use, real-time facial image sequences of the driver are collected and preprocessed as described, and driver fatigue recognition is performed using the FGR-ViViT model.

[0019] Due to the preprocessing involving the localization of facial 3D feature points and their correction to a frontal view, this method can reduce the dependence of the ViViT model on datasets from different view angles, making camera placement flexible, with lower constraints on the head pose of the driver. Drivers are allowed to operate according to personal preferences. Additionally, this setup enables model training with a smaller amount of data. On the other hand, the method only extracts the eye-brow-mouth line drawings relevant to fatigue, reducing interference from other objects in the images. This eliminates redundant information from the data, emphasizing only the eye and mouth states, reducing the impact of factors such as head pose, glasses occlusion, and lighting conditions, thus strengthening the feature information learned by the model. This ensures the detection accuracy and stability in practical applications.

[0020] In summary, compared with the prior art, during driver fatigue detection, this method can reduce the dependence on the camera placement and driver head angles and reduce the influence of factors such as glasses occlusion and lighting conditions to ensure the detection accuracy and stability in practical applications.

[0021] 2 Since the method uses contour maps that eliminate redundant information and flexible detection view angles, the detector can be placed in any position where most faces can be detected, enabling the detection of fatigue states of different individuals during different operational actions. Therefore, this method can be extended to other industries, such as healthcare, factory assembly lines, hazardous operations, and security patrols, to reduce safety accidents. Compared with the prior art, this method has a broader scope of application.

[0022] Preferably, in S21, the 3D positioning and frontal pose correction include:

[0023] S211: detecting the feature points on the facial image by using a preset facial feature point detection model and performing 3D positioning;

[0024] S212: selecting a feature point at a chin tip as a new origin of a coordinate system, and translating an origin of an original coordinate system to the new origin, where the original coordinate system is a 3D coordinate system; and

[0025] S213: performing face rotation to obtain the mesh of the facial feature points from the frontal view.

[0026] In this way, key feature points can be associated with the original coordinate system and processed, ensuring a stable frontal pose correction of facial images. Selecting the feature point at the chin tip as the new origin ensures accurate identification under various head poses and prevents the corrected face from exceeding an image display range.

[0027] Preferably, in S211, the preset facial feature point detection model is MediaPipe Face Mesh; 468 key feature points on the facial image are detected using MediaPipe Face Mesh, and 3D positioning is performed.

[0028] MediaPipe Face Mesh is a mature facial geometric solution, which can estimate 468 3D facial landmarks in real-time even on mobile devices (surpassing the 68 points detected by dlib). It employs machine learning (ML) to infer 3D surface geometry, requiring only input from a single camera, without the need for dedicated depth sensors. This setup ensures comprehensive and accurate detection of facial feature points while reducing hardware requirements.

[0029] Preferably, in S212, when the origin of the original coordinate system is translated to the new origin, a translation vector is given by the following formula:

[0030] jy Y %

[0031] where 152 , 152 , and 152 are coordinates of the feature point at the chin tip in the original coordinate system.

[0032] This setup enables accurate and fast facial translation.

[0033] Preferably, in S212, after the origin of the original coordinate system is translated to the new origin, a new coordinate system is established using a left-hand screw rule, where an X -axis of the new coordinate system is horizontally to the right, a F -axis is vertically downward, and a Z-axis is vertically outward.

[0034] This setup provides a reference coordinate system for the subsequent facial rotation, ensuring the effectiveness of the subsequent frontal pose correction.

[0035] Preferably, in S213, during the face rotation, an inverse matrix R of a pose matrix of a face pose is used as a rotation matrix; the facial feature points are transformed from coordinates p / y 2 A a n / jn t|1c originai coordinate system to coordinates PC(XC,YC, Zc) in the new coordinate system using the following formula:

[0036]

[0037] where is a 3x1 translation vector, 3 is a 3x3 identity matrix, and R is a 3x3 rotation matrix.

[0038] This setup, using the new coordinate system and the original coordinate system as a link for the rotation, ensures the effectiveness and accuracy of the rotation. Moreover, since the new coordinate system uses the chin tip as the origin and the chin tip is located at the bottom of the face, this setup makes the rotation simpler and efficient compared to the approach using a feature point such as a nose tip.

[0039] Preferably, in S213, a process of generating the inverse matrix R of the pose matrix includes:

[0040] selecting a feature point at a forehead vertex detected by MediaPipe Face Mesh, unitizing a vector connecting the selected feature point to the origin O(Xo,Yo, Zo) of the new coordinate system to serve as a D vector unit_y of the face pose; and unitizing a vector connecting a center point of a left eye to a center point of a right eye in the facial image to serve as a T vector unit_x of the face pose; and

[0041] cross-multiplying unit_x and unit_y based on the left-hand screw rule to obtain a normal vector unit_z of the face pose, and generating the pose matrix thereby , . . , . . R = unit xy.unit yx.unit z' „ . obtaining the inverse matrix L J of the pose matrix.

[0042] In this way, the pose matrix is constructed using the feature points of the forehead vertex and the chin tip (the origin of the new coordinate system), as well as the feature points from the center point of the left eye to the center point of the right eye. Since all these feature points obvious key feature points, the pose matrix can be obtained quickly, which ensures the efficiency and accuracy of the frontal pose correction.

[0043] Preferably, in SI, a process of constructing the FGR-ViViT model includes: adding a part selection module to a temporal transformer encoder of the ViViT model.

[0044] The temporal transformer encoder includes L temporal transformer layers, and the part selection module is added before the last temporal transformer layer. Processing of the part selection module includes:

[0045] first, recursively applying matrix multiplication to original attention weights of all layers before the last temporal transformer layer to combine the original attention weights of each layer: =n y 7=0 , where Y| represents self-attention head weights of an ‘ -th layer, and C represents the number of self-attention heads in the FGR-ViViT model;

[0046] next, selecting indices of maximum values ” 2’ ’ c of different attention heads in Ylast, for extracting corresponding tokens from Pl.! , where is hidden features input before the last temporal transformer layer; and

[0047] then, splicing the selected tokens with class tokens to form a new input sequence ; and inputting the new sequence Pnew into the last temporal transformer layer.

[0048] For fatigue detection, facial changes of drivers, especially eye movements, are often very subtle and even difficult for humans to recognize, while local changes usually play an important role in distinguishing driving states. As the detected object may exhibit only minor visual differences, a fine-grained classification algorithm is introduced to enhance the detection accuracy and sensitivity. Fine-grained classification aims to distinguish classes that are visually and semantically similar. For example, a person with slightly closed or frequently blinking eyes may be fatigue, but these features closely resemble normal facial expressions. Therefore, fine-grained classification can distinguish these subtle differences more accurately and is particularly useful for content analysis.

[0049] Most fine-grained classification is applied to images and cannot be applied to time-based state assessment. Thus, in the present disclosure, a part selection module is added to the ViViT model. Moreover, since this method focuses on subtle image differences over time, the part selection module is added to the temporal transformer encoder of FGR-ViViT. This setup allows for fine-grained classification of image sequences, efficiently analyzing and extracting differentiating features in the time dimension while eliminating redundant information. The model focuses more on subtle changes in the image sequence, thereby better addressing dynamic features and enhancing the accuracy and robustness of classification tasks.

[0050] Preferably, the hidden features input before the last temporal transformer layer are: r cis 1 2 V “|

[0051] 1 ~ LPa 1 ’ P'-1 ’ P,r-1 ’''' ’ PA1J •

[0052] where PlA represents a learnable class token output by an (L-l)-th temporal transformer 1 2 A? layer, and represent N feature vectors output by the (LM)-th temporal transformer layer.

[0053] The new input sequence Pnew obtained by splicing is:

[0054] Pnew ~ I^Ph’PA^Pl?1 v ’ ’>Pzfi] . Xj X, xc

[0055] where P / >’P / -P- 1 represent tokens selected based on the indices of the maximum YY.. Y values 15 2 ’' ” ’ c of the different attention heads.

[0056] The self-attention head weights of the -th layer are:

[0057] ^z = [Tz ,Tz, Jz’’"’Tz ] / g 1,2, •••,£ —1 .

[0058] y'1 = ’ ’' ’ ’ ] z' G 0,1, • • •, C M .

[0059] where y{ represents a weight of an ’ -th self-attention head in the / -th layer; and

[0060] yi represents a weight of a learnable class token in each attention head of the / -th layer, and Az ,Tz , ^1 represent weights of N feature vectors in each attention head of the / -th layer.

[0061] Preferably, in SI, during construction of the FGR-ViViT model, an improved loss function is introduced, where the introduced improved loss function is as follows: t _ t(RD} 1 r(CON)

[0062] • • ’ ■ / (RD} t(CE) , n r(KL) L* ' = Lt + p ■ p = -logPe(1;(zl v)-logpP(T,l xd KW’W *3)

[0063] where ^))]

[0064] L(CK)

[0065] where ' is a cross-entropy loss between a predicted label and a true label of a sample; R I^KL) P is a weight coefficient; ' represents an average KL divergence after two dropouts; D, (P® || P^S rn (2) KL\ 0 0 / represents a KL divergence between two distributions Pg } and Pg } ; p(D / y | y ) e ' ' and Pg (yi I Xi) are two model-predicted distributions after a sample passes through p ( pl y \ I y | y ) a same model "■ f with dropout twice, and training data is denoted by ' 1 Zi';and m z z

[0066] ' is a contrastive loss function; A represents a batch size; ' and J represent I Sim(z z) feature vectors normalized through 2; ' n J' represents cosine similarity; a is a preset constant; V represents a true label; and '■ represent true labels for different classes.

[0067] In ViViT models, where parameters are generally numerous, overfitting can occur easily with insufficient training samples. Dropout, during forward propagation, probabilistically deactivates specific neurons, reducing complex co-adaptations between neurons, and effectively lowering dependence of the model on local features, thus helping prevent overfitting.

[0068] In this method, the cross-entropy loss function in the traditional ViViT model is replaced with a new loss function that combines two dropouts and an improved contrastive loss function. This not only prevents overfitting but also encourages the model to learn more discriminative feature representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to make the objectives, technical solutions, and advantages of the present disclosure more clear, the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0070] FIG. lisa flowchart of a method according to the present disclosure;

[0071] FIG. 2 is a schematic diagram illustrating positioning of facial 3D coordinates using MediaPipe Face Mesh from multiple view angles in an embodiment;

[0072] FIG. 3 is a schematic diagram showing a facial coordinate system and pose vectors in the embodiment;

[0073] FIG. 4 is a schematic diagram showing positions of key points before and after face pose correction in the embodiment;

[0074] FIG. 5 is a schematic diagram of facial correction results under various view angles or poses in the embodiment;

[0075] FIG. 6 is a schematic structural diagram of an FGR-ViViT model;

[0076] FIG. 7 is an overall architecture diagram of a loss function for FGR-ViViT in the embodiment;

[0077] FIG. 8 is a schematic diagram showing partial samples from a YawDD dataset and a DROZY dataset in an experimental analysis of the embodiment;

[0078] FIG. 9 is a schematic diagram of variation curves of Fl-score and loss during a training process on the YawDD dataset in the experimental analysis of the embodiment;

[0079] FIG. 10 is a schematic diagram of variation curves of Fl-score and loss during a training process on the DROZY dataset in the experimental analysis of the embodiment; and

[0080] FIG. 11 is a schematic diagram of some uncorrected facial line drawings from the YawDD dataset in the experimental analysis of the embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0081] The present disclosure is described in more detail below with reference to the specific embodiments

[0082] Embodiment

[0083] As shown in FIG. 1, this embodiment discloses a method for assessing driver drowsiness based on view angle correction and improved ViViT, including the following steps:

[0084] SI: Construct an FGR-ViViT model based on a ViViT model, for fatigue recognition based on image sequences of eye-brow-mouth shapes from a frontal view.

[0085] S2: Acquire facial image sequences of drivers and preprocess each image to obtain training image sequences.

[0086] The preprocessing includes:

[0087] S21: Extract feature points on each facial image and perform 3D positioning and frontal pose correction to obtain a mesh of facial feature points from the frontal view. Preferably, the 3D positioning and frontal pose correction include:

[0088] S211: Detect the feature points on the facial image by using a preset facial feature point detection model and perform 3D positioning. During specific implementation, the preset facial feature point detection model is MediaPipe Face Mesh; 468 key feature points on the facial image are detected using MediaPipe Face Mesh, and 3D positioning is performed.

[0089] Parameters to be considered in the field of fatigue detection are eye features and mouth features of the driver, which have been proved to be directly related to the driver state. In order to eliminate geometric spatial variations due to pose changes, face feature points need to be corrected. MediaPipe Face Mesh is a detector that can estimate 468 3D facial coordinate points in real time, and provides a reliable and accurate facial coordinate positioning scheme for drivers with larger head angles in the case of multiple view angles, and the positioning effect is as shown in FIG. 2.

[0090] S212: Select a feature point at a chin tip as a new origin of a coordinate system, and translate an origin of an original coordinate system to the new origin, where the original coordinate system is a 3D coordinate system.

[0091] When the origin of the original coordinate system is translated to the new origin, a translation vector is given by the following formula:

[0092] X Y Z

[0093] where 152 , 152 , and 152 are coordinates of the feature point at the chin tip in the original coordinate system.

[0094] After the origin of the original coordinate system is translated to the new origin, a new coordinate system is established using a left-hand screw rule, where an A -axis of the new coordinate system is horizontally to the right, a Y -axis is vertically downward, and a Z-axis is vertically outward.

[0095] The correction process of the head pose is a rigid-body transformation, which does not involve facial deformation. Only rotation and translation are needed to correct the face in other view angles or poses to the frontal view or pose, and the face is displayed in the image. In order to prevent the corrected face from exceeding the image display range, a feature point at a chin tip 152 is selected as a new origin of the coordinate system, and face rotation is performed after the origin of the original coordinate system is translated to the new origin.

[0096] S213: Perform face rotation to obtain the mesh of the facial feature points from the frontal view.

[0097] Preferably, during the face rotation, an inverse matrix A of a pose matrix of a face pose is used as a rotation matrix; the facial feature points are transformed from coordinates p (X Y Z } , \ jn t|ie original coordinate system to coordinates PC(XC,YC, Zc) in the new coordinate system using the following formula: v A.

[0098] Yc Zc 1 R 0T4 fl Yw 0 1J 0 1 Zw

[0099] where T is a 3x1 translation vector, ^3 is a 3x3 identity matrix, and is a 3x3 rotation matrix.

[0100] A process of generating the inverse matrix R of the pose matrix includes:

[0101] selecting a feature point at a forehead vertex detected by MediaPipe Face Mesh, unitizing a vector connecting the selected feature point to the origin O(Xo,Yo,Zo) of the new coordinate system to serve as a D vector unit_y of the face pose; and unitizing a vector connecting a center point of a left eye to a center point of a right eye in the facial image to serve as a T vector unit_x of the face pose; and

[0102] cross-multiplying unit_x and unit_y based on the left-hand screw rule to obtain a normal vector unit_z of the face pose, and generating the pose matrix zl^ thereby . R / / fj / 7 xT,umt y1 ,unit zTl obtaining the inverse matrix L - -- ol the pose matrix.

[0103] The new coordinate system established based on the left-hand screw rule is as shown in FIG. 3, where the A _axjs js horizontally to the right, the Y _axis is vertically downward, and the Z-axis is vertically outward.

[0104] After the correction of facial key points, the eyes and mouth in the image are not affected by the head pose, and are only related to the opening degree of the eyes and mouth. A comparison before and after the correction is shown in FIG. 4, where (a) is a captured facial image; (b) is a 3D coordinate face mesh detected in the facial image; (c) is the coordinates of the face mesh in the original coordinate system; and (d) is the coordinates of the face mesh after the face mesh is corrected to the frontal view in the new coordinate system.

[0105] S22: Orthographically project the mesh of the facial feature points after the correction, extract shapes of eyes, eyebrows, and mouth, construct eye-brow-mouth line drawings based on corrected coordinates, and use image sequences of the obtained eye-brow-mouth line drawings as the training image sequences.

[0106] The mesh after the correction is orthographically projected, and shapes of the eyes, eyebrows, and mouth are extracted. The final facial correction effect is as shown in FIG. 5.

[0107] When the camera is directly in front: (a) female, nighttime, eyes and mouth slightly open, head tilted upward; (b) male, nighttime, yawning with eyes open, head tilted upward to the left.

[0108] When the camera is below the rearview mirror: (c) male, daytime, talking with eyes open, looking straight ahead; (d) male, daytime, eyes closed and smiling, tilting head downward.

[0109] When the camera is above the dashboard: (e) female, daytime, talking with eyes open, tilting head to the right; (f) male, daytime, yawning with eyes slightly open, tilting head upward to the right.

[0110] S3: Train the FGR-ViViT model using the training image sequences.

[0111] S4: Collect a real-time facial image sequence of a driver, perform preprocessing as in S2 to obtain a real-time image sequence, and perform fatigue recognition on the driver using the trained FGR-ViViT model.

[0112] During specific implementation, a process of constructing the FGR-ViViT model includes: adding a part selection module to a temporal transformer encoder of the ViViT model.

[0113] FGR-ViViT backbone network

[0114] For fatigue detection, facial changes of drivers, especially eye movements, are often very subtle and even difficult for humans to recognize, while local changes usually play an important role in distinguishing driving states. As the detected object may exhibit only minor visual differences, the present disclosure introduces a fine-grained classification algorithm to enhance the detection accuracy and sensitivity. Fine-grained classification aims to distinguish categories that are visually and semantically similar. For example, a person with slightly closed or frequently blinking eyes may be fatigue, but these features closely resemble normal facial expressions. Therefore, fine-grained classification can distinguish these subtle differences more accurately and is particularly useful for content analysis.

[0115] Most fine-grained classification is applied to images and cannot be applied to time-based state assessment. In this method, the part selection module is added to the ViViT model, such that the fine-grained classification can also be applied to image sequences, and the model focuses more on subtle changes of image sequences. Since this method focuses on subtle image differences over time, the part selection module is added to the temporal transformer encoder of FGR-ViViT. The overall architecture is shown in FIG. 6.

[0116] In FGR-ViViT, the input image sequence enters the embed-to-tokens layer for spatio-temporal pipe sampling to fuse temporal and spatial features and linearly project them into a multidimensional space. Subsequently, a class token is inserted at the beginning of the token sequence for classification, and positional encoding is combined into the input sequence to obtain positional information for each token in the sequence. The spatial transformer encoder outputs new class tokens by modeling tokens indexed in the same space, and the output class tokens are later spliced with representational tokens of the frame dimension and input into the temporal transformer encode.

[0117] The temporal transformer encoder contains multiple temporal transformer layers, and the feature sequence has to enter the part selection module before being input to the last temporal transformer layer. Assuming that the model has C self-attention heads, in the hidden features p = rpds 1 2 L L1’ L1’ L1’ ’ L1J input before the last temporal transformer layer, pf.i and 12 N Pn’P^-i’ ’P^Pt-pPl-v "‘'Pl-i represent a learnable class token and N feature vectors output by an (L-l)-th temporal transformer layer. Attention weights of the previous layers are given by the following formula:

[0118] / g 1,2,1.

[0119] = zg0,1,-,C-1.

[0120] where Y, represents weights of self-attention heads of an ^-th layer, yf represents a ylci* weight of an 1 -th self-attention head in the / -th layer. represents a weight of a learnable class token in the Z-th layer, and ^z ’ ’X represent weights of N feature vectors in each attention head of the Z-th layer.

[0121] Due to the lack of recognizability of the embedded tokens, the original attention weight values do not necessarily correspond to the importance of the input tokens. Therefore, the matrix multiplication is recursively applied to the original attention weights in all the layers, to combine the attention weights of all the layers, as shown in the following formula:

[0122] z=o ;

[0123] Ylast captures the entire information input by the temporal transformer encoder into deeper layers, which is more conducive to selecting recognizable regions compared to the original attention Y X X • • • X weights of a single layer. Indices of maximum values *’ 2’ ’ c of different attention heads are selected in Ylast. These indices are used for extracting corresponding tokens from ppp The selected tokens are spliced with a class token and input to the last temporal transformer layer. The new input sequence is shown as follows:

[0124] Pnew = • Xi x2 xc

[0125] where Pr-i’Pn’ ’P'1 represent tokens selected based on the indices of the maximum values of the different attention heads.

[0126] Replacing the original input sequence with the maximum response tokens selected based on the attention weights not only preserves global information, but also allows the last transformer layer to discard tokens with lower weights and focus on the subtle differences of the features in the temporal dimension. Finally, the class token is extracted from the output obtained by the temporal transformer encoder, and is passed through a multilayer perceptron (MLP) Head to obtain the final classification result.

[0127] Improved loss function

[0128] A cross-entropy loss function is employed in the ViViT model. Assuming that the training (x I y ) P (v| x) data is ' ' if and the model is ■ \ a cross-entropy loss between a predicted label and a true label of each sample is shown as follows:

[0129] } ^log^W *> ).

[0130] However, in ViViT models, where parameters are generally numerous, overfitting can occur easily with insufficient training samples. Dropout, during forward propagation, probabilistically deactivates specific neurons, reducing complex co-adaptations between neurons, and effectively lowering dependence of the model on local features, thus helping prevent overfitting. This method introduces R-Drop, i.e., using two dropouts. Xj is passed forward twice in the model, i.e., the same sample is repeatedly passed through the same model with dropout twice, and two different (y | x ) (2) distributions 6 ' ‘ ‘ ’ and P0 (yj I Xj) predicted by the model can be obtained. Then the KL divergence ' Lj ' is used to regularize the model predictions to make them consistent as much as possible, and the two are weighted to obtain the R-Drop loss function, as shown in the following formula: L{rd} liCE>

[0131] = ^logPf)(yJ xJ^logPp (yj x,) + Da (PJ2>(xl AA’A A]

[0132] where represents a weight coefficient; ' represents an average KL divergence after D (P^ II P^ ) ri') two dropouts; and KL\ e e / represents a KL divergence between two distributions Pg and u

[0133] Meanwhile, since the features between different classes are very similar in the fine-grained domain, this method introduces the combination of the contrastive loss function L^Lon^ and the R-Drop function, which drives the model to learn discriminative feature representations by maximizing the similarity of class tokens corresponding to the same class and minimizing the similarity of class tokens corresponding to different classes, so as to extract essential features of the data. In order to minimize the impact of simple negative samples on the loss, a constant a is introduced in ' to regulate negative samples that contribute to the loss, i.e., only negative samples with similarity greater than a contribute to the loss, as shown in the following formula: 4C0N)=T^£ N 'Lr-y^yj' v N f \ \ E maxll&'mlZpZj^aLO

[0134]

[0135] where represents a batch size; Zj and Z] represents feature vectors normalized I Sim\z ,z \ . . . v y y. through 2; j? represents cosine similarity; y represents a true label; • ' and 3 represent true labels for different classes.

[0136] Finally, the loss function of FGR-ViViT is the sum of ' and the contrastive loss j\CON) function ' , as shown in the following formula:

[0137]

[0138] The overall architecture of the improved loss function is as shown in FIG. 7.

[0139] In order to more fully illustrate the effectiveness of the present method, the following experimental analysis has been conducted in the present disclosure.

[0140] Experimental analysis

[0141] 1. Datasets

[0142] Tests are performed on the YawDD video dataset and the DROZY video dataset respectively.

[0143] YawDD video dataset. This dataset collects a large number of driving videos from 107 volunteers of different races, genders and ages. The shooting scene was inside a car in a bright setting, the on-board cameras were placed on the dashboard and under the rearview mirror, and videos were captured from two view angles. The volunteers performed different actions according to the instructions, such as driving normally, talking and yawning, and some volunteers had coverings on their faces such as glasses and sunglasses. The dataset consists of 351 videos with a frames per second (FPS) rate of 30 and a resolution of 640x480. This dataset is used to test the fatigue classification performance of the model based on mouth movements.

[0144] The DROZY video dataset, also known as ULg multimodal drowsiness database, is a database containing various types of drowsiness-related data (e.g., signals and videos). The dataset contains spectral video recordings of near infrared (NIR) from 14 subjects, each of whom underwent up to three psychomotor vigilance tests. All subjects were not allowed to sleep in 1 day, 5 hours and 30 minutes. Before beginning each test, subjects completed a questionnaire to 15 determine their level of sleepiness. The dataset consists of 36 videos with an FPS rate of 30 and a resolution of 512x424. The dataset is used to test the fatigue classification performance of the model based on eyebrow, eye, and mouth states.

[0145] In the DROZY video dataset, each video is simply labeled with the overall fatigue level of the driver, while specific fatigue actions are not labeled. Therefore, 3,600 sets of samples are extracted from the DROZY dataset in order to extract specific and accurate fatigue features for model training. The extracted DROZY dataset and the entire YawDD dataset are each divided into corresponding training and testing sets through uniform random sampling in a ratio of 4:1, and there is no intersection between the training and testing sets. Some samples of the two datasets are shown in FIG. 8.

[0146] 2. Evaluation indicators

[0147] In order to comprehensively evaluate the proposed method, the present disclosure uses three indicators: precision, recall, and Fl-score, defined by formulas (1) to (3). False negative (FN) is the number of fatigue states that are incorrectly recognized as awake states. False positive (FP) is the number of awake states that are incorrectly recognized as fatigue. True positive (TP) is the number of fatigue states that are correctly recognized as fatigue.

[0148] Precision is a ratio of the number of correctly predicted fatigue samples to the number of samples predicted as fatigue. TP PR =------- (1)

[0149] TP + FP

[0150] Recall: a ratio of the number of correctly predicted fatigue samples to the number of actual fatigue samples. RE = --- (2)

[0151] TP + FN

[0152] Fl-score: weighted average of precision and recall. „ Precision * Recall z x 7) = 2 *------------------ (3)

[0153] Precision + Recall

[0154] Precision reflects the model's ability to differentiate between awake samples; the higher the precision, the stronger the model's ability to differentiate between awake samples. Recall reflects the model's ability to recognize fatigue samples; the higher the recall, the stronger the model's ability to recognize fatigue samples. Fl-score is the reconciled average of the precision and recall. The higher the Fl-score, the stronger the model robustness.

[0155] 3. Model training

[0156] Because the transformer does not have the same inductive bias as CNN, ViViT only shows effectiveness on large-scale datasets. Considering insufficient available fatigue classification video datasets with annotation levels, it is difficult to train the model from scratch. The experiments use a transfer learning approach to initialize the parameters of the ViViT model using a pre-trained ViT-B / 16x2 model that has been trained on the ImageNet21K dataset, in order to be able to fine-tune the model on a relatively small dataset.

[0157] Detection objects of the DROZY dataset include human eyes, and eye closure of the driver during driving is a behavior that can have serious consequences and requires timely warning. Therefore, each sample contains 16 consecutive images with a learning rate of 0.01 and a batch size of 16. YawDD is a dataset that only detects mouth movements, an excessively small number of images per sample will waste the computing power. Therefore, each sample contains 40 consecutive images with a learning rate of 0.03 and a batch size of 8.

[0158] Most of the redundant information has been removed from the feature contour map, so the image size can be reduced to 48x48, and 13-fold cross-validation is chosen for all samples. The experimental environment is shown in Table 1.

[0159] Table 1 Experimental environment configuration

[0160] Configuration name Version parameter Operating system Windows 10 GPU NVIDIA GeForce RTX 3060 CPU Intel(R) Core(TM) i7-10700F CPU @ 2.90GHz Deep learning framework PyTorch, OpenCV

[0161] 4. Model training results and analysis

[0162] In the YawDD video dataset, the main distinction is between yawning and other actions such as talking and laughing, and the variation curves of Fl-score and loss during the training process are shown in FIG. 9. In the DROZY video dataset, the main distinction is between awake and fatigue states, and the variation curves of Fl-score and loss are shown in FIG. 10.

[0163] In order to verify the validity of the face contour map and the corrected face contour map, relevant ablation experiments were conducted. The experimental results are shown in Table 2. The facial image denotes the original facial image extracted directly from the video; the uncorrected facial line drawing denotes the eye-brow-mouth line drawing extracted but not corrected to the frontal view; the corrected facial line drawing denotes the line drawing after the facial features are corrected to the frontal view.

[0164] Table 2 F 1-score results of different input data in FGR-ViViT

[0165] Dataset Method Fl-score YawDD Facial image 91.3 Uncorrected facial line drawing 90.1 Corrected facial line drawing 94.5 DROZY Facial image 87.2 Uncorrected facial line drawing 89.9 Corrected facial line drawing 97.6

[0166] The experimental results of the corrected facial line drawings are better than those of the uncorrected facial line drawings and the original facial images for the samples with multiple view angles, multiple head poses, and various mouth states in the YawDD video dataset and for samples with multiple head poses and various eye-brow-mouth motions in the dark background in the DROZY video dataset. Especially for the monochrome image sequence in DROZY, the small-size 48x48 facial images alone are less effective in identifying the fatigue state, and have an Fl-score 10.4% lower than that of the corrected facial line drawings. The uncorrected facial line drawings present different detection results depending on the pose. In the YawDD dataset, due to the large angle of the driver's head, the extracted facial feature lines are sometimes excessively distorted, as shown in FIG. 11, and the detection result is instead 1.2% lower than that of the original facial images. In the DROZY dataset, the detection objects have smaller movements. The uncorrected facial line drawings emphasizing changes of the eyes, eyebrows and mouth have an Fl-score 2.7% higher than that of the original facial images.

[0167] The above experimental results show that under small poses, the line drawings of the key facial features are more favorable for fatigue feature extraction due to the shielding of interference of changes in face brightness, texture, etc., and the advantage is more obvious especially in the small-size samples. However, as the pose expands, the changes in the size of the facial features with different distances can bring serious interference to the classification, resulting in performance degradation. By reducing this interference through pose correction, the performance can be improved again.

[0168] In order to verify the effectiveness of FGR-ViViT, in the present disclosure, an ablation experiment of the model on corrected facial line drawings was conducted. The Fl-score results of the experiment are shown in Table 3. Under the same parameters, ViViT denotes a baseline model without any modification, R-ViViT denotes a model with an improved loss function, FG-ViViT denotes a model with a part selection module, and FGR-ViViT denotes an improved model of the present disclosure i.e., the model with a part selection module and an improved loss function.

[0169] Table 3 Fl-score results of different models on the test set 18

[0170] Dataset Model Fl-score (%) ViViT 88.4 YawDD R-ViViT 91.5 FG-ViViT 94.3 FGR-ViViT 94.5 ViViT 96.9 DROZY R-ViViT 97.1 FG-ViViT 97.3 FGR-ViViT 97.6

[0171] The results show that the results of the improved model are all better than those of the baseline model and the model with only the part selection module.

[0172] 5. Comparative experiments:

[0173] The comparison results of this method with other methods on the corrected facial line drawings of the YawDD dataset and the DROZY dataset are shown in Table 4. The results show that the method of the present disclosure is significantly better than other methods in each performance index on the DROZY dataset. On the YawDD dataset, although the precision of JHPFA is 2.3% higher than that of the method of the present disclosure, its recall is 3.4% lower than that of FGR-ViViT; similarly, the CNN recall is 2.2% higher than that of the method of the present disclosure, but its precision is 6.4% lower. In contrast, the method of the present disclosure has the optimal comprehensive indicator, i.e., the Fl-score.

[0174] Table 4 Comparison results with recent related work

[0175] Dataset Method PR(%) RE(%) Fl-score (%) 2s-STGCN

[22] 85.5 93.9 89.5 cnn

[21] 89 96 92 YawDD JHPFAl20' 97.7 90.4 93.9 Method of the present disclosure 95.4 93.8 94.5 3DCNN 94.2 93.8 94.0 DROZY ResNet3D

[23] 97 96 97 Method of the present disclosure 97.8 97.5 97.6

[0176] This method analyzes the feasibility and complexity of current methods for studying fatigue driving, and ultimately selects an image sequence-based computer vision detection method for automatic classification of driver states. The proposed FGR-ViViT model assesses driver behaviors mainly based on state changes in an image sequence of corrected eye-brow-mouth lines, and a fine-grained classification method for image sequences is designed. Besides, a new loss function incorporating two dropouts and the improved contrastive loss function is also proposed. Experiments show that the method of the present disclosure outperforms all other methods in terms of the comprehensive evaluation indicator, i.e., the Fl-score, proving the effectiveness of facial coordinate correction and FGR-ViViT framework. In addition, this method has a wide range of application prospects in many fields due to the flexible camera placement that can meet the needs of different application scenarios.

[0177] It should be noted that the above examples are only intended to explain, rather than to limit the technical solutions of the present disclosure. Those of ordinary skill in the art should understand that modifications or equivalent substitutions may be made to the technical solutions of the present disclosure without departing from the spirit and scope of the technical solutions of the present disclosure, and such modifications or equivalent substitutions should be included within the scope of the claims of the present disclosure.

Claims

1. A method for assessing driver drowsiness based on view angle correction and improved video vision transformation (ViViT), comprising the following steps:SI: constructing an FGR-ViViT model based on a ViViT model, for fatigue recognition based on image sequences of eye-brow-mouth shapes from a frontal view;S2: acquiring facial image sequences of drivers and preprocessing each image to obtain training image sequences;S3: training the FGR-ViViT model using the training image sequences; andS4: collecting a real-time facial image sequence of a driver, performing preprocessing as in S2 to obtain a real-time image sequence, and performing fatigue recognition on the driver using the trained FGR-ViViT model;wherein in S2, the preprocessing comprises:S21: extracting feature points on each facial image and performing 3D positioning and frontal pose correction to obtain a mesh of facial feature points from the frontal view; andS22: orthographically projecting the mesh of the facial feature points after the correction, extracting shapes of eyes, eyebrows, and mouth, constructing eye-brow-mouth line drawings based on corrected coordinates, and using image sequences of the obtained eye-brow-mouth line drawings as the training image sequences.

2. The method for assessing driver drowsiness based on view angle correction and improved ViViT according to claim 1, wherein in S21, the 3D positioning and frontal pose correction comprise:S211: detecting the feature points on the facial image by using a preset facial feature point detection model and performing 3D positioning;S212: selecting a feature point at a chin tip as a new origin of a coordinate system, and translating an origin of an original coordinate system to the new origin, wherein the original coordinate system is a 3D coordinate system; andS213: performing face rotation to obtain the mesh of the facial feature points from the frontal view.

3. The method for assessing driver drowsiness based on view angle correction and improved ViViT according to claim 2, wherein in S211, the preset facial feature point detection model is MediaPipe Face Mesh; 468 key feature points on the facial image are detected using MediaPipe Face Mesh, and 3D positioning is performed.

4. The method for assessing driver drowsiness based on view angle correction and improved ViViT according to claim 3, wherein in S212, when the origin of the original coordinate system is translated to the new origin, a translation vector is given by the following formula:y Zwherein 152 , 152 , and 152 are coordinates of the feature point at the chin tip in the original coordinate system.

5. The method for assessing driver drowsiness based on view angle correction and improved ViViT according to claim 4, wherein in S212, after the origin of the original coordinate system is translated to the new origin, a new coordinate system is established using a left-hand screw rule, wherein an X -axis of the new coordinate system is horizontally to the right, a Y -axis is vertically downward, and a Z-axis is vertically outward.

6. The method for assessing driver drowsiness based on view angle correction and improved ViViT according to claim 5, wherein in S213, during the face rotation, an inverse matrix R of a pose matrix of a face pose is used as a rotation matrix; the facial feature points are transformed fromP (X Y Z }coordinates ” 2 in the original coordinate system to coordinates PC(XC,YC, Zc) in thenew coordinate system using the following formula:wherein f is a 3x1 translation vector, ^3 is a 3x3 identity matrix, and R is a 3x3 rotationmatrix.

7. The method for assessing driver drowsiness based on view angle correction and improved ViViT according to claim 6, wherein in S213, a process of generating the inverse matrix R of the pose matrix comprises:selecting a feature point at a forehead vertex detected by MediaPipe Face Mesh, unitizing a vector connecting the selected feature point to the origin of the new coordinate system to serve as aD vector unit^y of the face pose; and unitizing a vector connecting a center point of a left eye to a center point of a right eye in the facial image to serve as a T vector unit_x of the face pose; and cross-multiplying unit_x and unit_y based on the left-hand screw rule to obtain a normal vector unit_z of the face pose, and generating the pose matrix thereby, .... . R = \unit x\unit y{ .unit zTl „ .obtaining the inverse matrix L - - of the pose matrix.

8. The method for assessing driver drowsiness based on view angle correction and improved ViViT according to claim 1, wherein in SI, a process of constructing the FGR-ViViT model comprises: adding a part selection module to a temporal transformer encoder of the ViViT model;the temporal transformer encoder comprises L temporal transformer layers, and the part selection module is added before the last temporal transformer layer; processing of the part selection module comprises:first, recursively applying matrix multiplication to original attention weights of all layers before the last temporal transformer layer to combine the original attention weights of each layer:z=o , wherein Yi represents c self-attention head weights of an ' -th layer, and C represents the number of self-attention heads in the FGR-ViViT model;next, selecting indices of maximum values of different attention heads inY[ast, for extracting corresponding tokens from pL.x, wherein is hidden features input before the last temporal transformer layer; andthen, splicing the selected tokens with class tokens to form a new input sequence ; and inputting the new sequence Pnew into the last temporal transformer layer.

9. The method for assessing driver drowsiness based on view angle correction and improved ViViT according to claim 1, wherein hidden features ' input before the last temporal transformer layer are:Pl-i = [Pl-I’Pl-I’Pl-I’’’’’Pl-i] .wherein p^ represents a learnable class token output by an (L-l)-th temporal transformer layer, and ,PmPl-pPl-prepresent N feature vectors output by the (L-l)-thtemporal transformer layer;the new input sequence P"ew obtained by splicing is:23Pl-i,Pl-i,Pl-iPl-iJ .DX1 DX1 ••• D®wherein ^l-\ represent tokens selected based on the indices of the maximumvalues of the different attention heads;the ( self-attention head weights of the -th layer are: / 61,2,---,1-1.?yi =[j^;j?,j?,---,Xv] zgO,i,---,c.wherein y} represents a weight of an z -th self-attention head in the Z-th layer; andrepresents a weight of a learnable class token in each attention head of the Z-th layer, and J / , , yi represent weights of N feature vectors in each attention head of the Z-th layer.

10. The method for assessing driver drowsiness based on view angle correction and improved ViViT according to claim 1, wherein in SI, during construction of the FGR-ViViT model, an improved loss function is introduced, wherein the introduced improved loss function is as follows:£ _ j\RD) j(CON)\ N N / N / , x£(con) _— £ ll-5zzw(z,z ) + 2 maxIpzWz ,z )-a),0N v 7 1wherein 1 is a cross-entropy loss between a predicted label and a true label of a sample;P is a weight coefficient; 1 represents an average KL divergence after two dropouts;D (P® IIP®) ri') (2}KL\ 3 3 ' represents a KL divergence between two distributions Pg J and Pg 7 ;£(1) / y | y \6 !Z and Pg (yi I Xj) are two model-predicted distributions after a sample passes throughP (v| x'l (x I y )a same model 3~ ' with dropout twice, and training data is denoted by ' ' ''; anditCON) z z1 is a contrastive loss function; represents a batch size; ' and representI Sim I z z ) feature vectors normalized through 2; \ n represents cosine similarity; a is a presetconstant; V represents a true label; and represent true labels for different classes.

Images