Fatigue driving detection method and system based on deep features and graph attention mechanism

Abstract

The application discloses a kind of based on deep feature and graph attention mechanism's fatigue driving detection method and system, the method includes: with multiple original video sequences of whether fatigue driving annotation constructs sample training set;Fatigue driving detection model is constructed, and the fatigue driving detection model is trained using sample training set;The training process includes: based on original video sequence, by face feature extraction and position coding, obtain space-time face feature sequence;Based on the interrelation between graph attention mechanism learning space-time face feature and the importance weight of each feature to final classification, based on interrelation update space-time face feature sequence, based on importance weight and updated space-time face feature sequence are weighted fusion, obtain fusion feature, based on fusion feature carries out two classification training;The original video sequence to be detected is input into the fatigue driving detection model of training completion, completes fatigue driving detection, improves the precision and generalization of fatigue driving detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fatigue driving detection, and particularly relates to a fatigue driving detection method and system based on deep features and a graph attention mechanism. BACKGROUND

[0002] In modern society, highways develop rapidly, and fatigue driving is one of the main causes of traffic accidents. Drivers will appear fatigue driving after a long time of driving. In order to improve the safety of drivers and passengers, it is necessary to detect the fatigue of drivers so as to timely remind the drivers. Therefore, with the improvement of people's safety awareness and scientific technology, fatigue driving detection methods emerge as the times require.

[0003] At present, the commonly used fatigue driving detection methods mainly include a method based on physiological information, a method based on computer vision, and a method based on changes in vehicle driving behavior, etc. Among them, the method based on computer vision has high accuracy in fatigue driving detection through facial features of the driver such as eyes, mouth, head, and expression, has low cost, does not interfere with normal driving, and has high practicability, acceptability, and comfort, and thus is most widely applied. However, the fatigue driving detection method based on computer vision still has deficiencies in the algorithm.

[0004] Firstly, the direct mutual relationship of the facial features of the driver is not fully utilized. The traditional method usually uses 3DCNN and LSTM to study the relationship between the facial features of the driver, such as the method disclosed by J. Lyu et al. in “Long-term multi-granularity deep framework for driver drowsiness detection” and the method disclosed by H. Yang et al. in “Driver Yawning Detection Based on Subtle Facial Action Recognition”. However, 3DCNN is limited by the size of the convolution kernel receptive field, and can only construct the time sequence features in some continuous feature maps, and it is difficult to construct long-time mutual relationship; LSTM processes the features in sequence, and the features located in the rear of the sequence can only obtain the aggregated information of the features located in the front of the sequence, and it is difficult to construct the mutual relationship of the features far apart in the sequence. The direct correlation between the facial features is not discussed in the existing detection method.

[0005] Secondly, the lack of the ability to distinguish between peak frame features and non-peak frame features. Existing driver fatigue detection methods treat each acquired facial feature equally, ignoring the different contributions of different facial features to the final classification. Specifically, for example, a yawning video clip, the driver's facial expression will only change subtly at first, and then develop into a strong expression, i.e. tightly closed eyes and wide open mouth. The former is usually referred to as a non-peak frame, while the latter is referred to as a peak frame. Features from peak frames have greater reference value than features from non-peak frames for the final classification. Existing methods lack the ability to distinguish between peak frame features and non-peak frame features, and cannot highlight the importance of peak frame features.

[0006] In summary, the existing computer vision-based fatigue driving detection method has the above-mentioned defects, and its fatigue driving detection accuracy and generalization are poor. SUMMARY

[0007] To solve the above problems of the prior art, the present application provides a fatigue driving detection method and system based on deep features and graph attention mechanism, which solves the problems of not fully utilizing the direct mutual relationship of facial features and the lack of the ability to distinguish between peak frame features and non-peak frame features in existing detection methods, and improves the accuracy and generalization of fatigue driving detection.

[0008] In a first aspect, the present disclosure provides a fatigue driving detection method based on deep features and graph attention mechanism, comprising:

[0009] Obtaining a plurality of original video sequences containing driver facial images, labeling the original video sequences, and constructing a sample training set with a plurality of labeled fatigue driving original video sequences;

[0010] Constructing a fatigue driving detection model, training the fatigue driving detection model using the sample training set; the training process includes: based on the original video sequence, obtaining a spatio-temporal facial feature sequence through facial feature extraction and position encoding; learning the mutual relationship between the spatio-temporal facial features and the importance weight of each feature to the final classification based on the graph attention mechanism, updating the spatio-temporal facial feature sequence based on the mutual relationship, and performing weighted fusion based on the importance weight and the updated spatio-temporal facial feature sequence to obtain a fusion feature, and performing binary classification training based on the fusion feature;

[0011] Inputting the original video sequence containing the driver's facial image to be detected into the fatigue driving detection model trained, and completing fatigue driving detection.

[0012] Further technical solutions, the fatigue driving detection model includes a spatio-temporal facial feature extraction network, a multi-head graph attention network, and a weighted graph attention feature fusion network.

[0013] Further technical solutions, based on the original video sequence, through face feature extraction and position coding, obtain the space-time face feature sequence, comprising:

[0014] Based on the original video sequence containing the driver's face image, the face image sequence is obtained through the face detection algorithm;

[0015] Based on the face image sequence, the visual representation features of each image in the face image sequence are extracted using a convolutional neural network to form a spatial face feature sequence;

[0016] The spatial face feature sequence is position coded using a multi-frequency cosine position function to inject position information into each face feature in the sequence, and a space-time face feature sequence is obtained.

[0017] Further technical solutions, the spatial face feature sequence is position coded using a multi-frequency cosine position function to inject position information into each face feature in the sequence, and a space-time face feature sequence is obtained, comprising:

[0018] The position coding sequence is obtained by coding using a multi-frequency cosine position function;

[0019] The position coding sequence and the spatial face feature sequence are added, and the space-time face feature sequence is obtained after addition.

[0020] Further technical solutions, based on the graph attention mechanism, the mutual relationship between the space-time face features is learned, and the space-time face feature sequence is updated based on the mutual relationship, comprising:

[0021] Based on the space-time face feature sequence, each space-time face feature is regarded as a node to form a node feature, and a directed complete graph is constructed;

[0022] The input node feature is mapped to a feature subspace, and the attention coefficient for each pair of nodes in the directed graph is calculated through a shared self-attention mechanism to construct an adjacency matrix;

[0023] The adjacency matrix and the node feature matrix are fused to update the node feature, and the updated node features of each attention head are spliced to form new node features, and an updated space-time face feature sequence is obtained.

[0024] Further technical solutions, based on the graph attention mechanism, the importance weight of each feature to the final classification is learned, and the updated space-time face feature sequence is weighted and fused based on the importance weight to obtain a fusion feature, comprising:

[0025] The node feature is projected into a unified linear space, a shared self-attention mechanism is used to calculate the attention coefficient for each pair of nodes in the graph, and an adjacency matrix is constructed based on the attention coefficient;

[0026] The adjacency matrix is added by row to form a weight vector; each element in the weight vector is a corresponding facial feature importance weight;

[0027] The updated spatio-temporal facial feature sequence is weighted and fused with the weight vector to obtain a fused feature.

[0028] In a further technical solution, the loss function is a cross-entropy function, and the formula of the cross-entropy function is:

[0029]

[0030] wherein K is the number of samples per batch, cls represents the class of classification, cls = 0 / 1, 0 represents not tired, and 1 represents tired, is a sample label, and represents that the ith sample belongs to the cls class, represents the probability of detecting the ith sample as the cls class.

[0031] In a second aspect, the present disclosure provides a fatigue driving detection system based on deep features and graph attention mechanisms, comprising:

[0032] A sample training set construction module is configured to obtain a plurality of original video sequences containing driver facial images, label the original video sequences, and construct a sample training set from a plurality of original video sequences labeled with whether fatigue driving occurs or not;

[0033] A detection model construction module is configured to construct a fatigue driving detection model, wherein the model comprises a spatio-temporal facial feature extraction network, a multi-head graph attention network, and a weighted graph attention feature fusion network.

[0034] A detection model training module is configured to train the fatigue driving detection model using the sample training set; the training process comprises: based on the original video sequence, obtaining a spatio-temporal facial feature sequence through facial feature extraction and position encoding; learning the mutual relationship between the spatio-temporal facial features and the importance weight of each feature on the final classification based on the graph attention mechanism, updating the spatio-temporal facial feature sequence based on the mutual relationship, weighting and fusing the updated spatio-temporal facial feature sequence based on the importance weight, obtaining a fused feature, and performing binary classification training based on the fused feature.

[0035] A detection module is configured to input an original video sequence containing a driver's facial image to be detected into the fatigue driving detection model trained, and complete fatigue driving detection.

[0036] In a third aspect, the present disclosure further provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein when the computer instructions are run by the processor, the steps of the method of the first aspect are completed.

[0037] In a fourth aspect, the present disclosure also provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the method of the first aspect.

[0038] The above one or more technical solutions have the following beneficial effects:

[0039] 1. The present application provides a fatigue driving detection method and system based on deep features and graph attention mechanism, based on the graph attention mechanism, the direct mutual relationship of each pair of features in the driver's face feature sequence is learned autonomously, and the face feature information is updated based on the learned mutual relationship; for the updated face feature, the graph attention mechanism is used to autonomously learn the contribution degree of each feature to the final classification, and the importance weight is assigned according to the contribution degree to the final classification, the features are weighted and fused, based on the fused features, the detection model can effectively distinguish the peak frame features and non-peak frame features, and fully utilize the peak frame feature information, further improve the precision and generalization of fatigue driving detection.

[0040] 2. The fatigue driving detection method and system provided by the present application solves the problem of not fully utilizing the direct mutual relationship of face features and lacking the ability to distinguish peak frame features and non-peak frame features in the existing detection method, and improves the precision and generalization of fatigue driving detection. BRIEF DESCRIPTION OF DRAWINGS

[0041] The drawings accompanying the specification of the present application form a part of the present application and serve to provide a further understanding of the present application, the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation of the present application.

[0042] Figure 1 A network structure schematic diagram of the fatigue driving detection model based on deep features and graph attention mechanism in the embodiment one of the present application;

[0043] Figure 2 A schematic diagram of a complete directed graph in the embodiment one of the present application;

[0044] Figure 3 A flowchart of the graph attention learning method in the embodiment one of the present application;

[0045] Figure 4 A visualization column chart of the importance weight of each face feature in each face feature sequence corresponding to the input face image in the embodiment one of the present application. DETAILED DESCRIPTION

[0046] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0047] It is also important to note that the terms "example" and / or "exemplary" as used herein illustrate concepts that are considered to be the most useful examples, but not limiting, of the exemplary embodiments according to the present application. As used in this document, the singular forms "a," "an," and "the" include their plural referents unless the context clearly dictates otherwise. Additionally, it should be understood that the terms "comprises", "comprising", "includes", "including", and the like, when used in this specification, specify the presence of stated features, steps, operations, devices, components, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, steps, operations, devices, components, and / or combinations thereof.

[0048] Embodiment One

[0049] The embodiment provides a fatigue driving detection method based on deep features and graph attention mechanism. The method is based on a driving video segment of a driver and uses a graph attention mechanism to detect fatigue driving. Based on the graph attention mechanism, the direct mutual relationship of each pair of features in the driver's facial feature sequence is autonomously learned, and the facial feature information is updated based on the learned mutual relationship. The graph attention mechanism is used to autonomously learn the contribution degree of each feature to the final classification, and the importance weight is assigned according to the contribution degree to the final classification. The features are weighted and fused, and based on the fused features, the detection model can effectively distinguish between peak frame features and non-peak frame features, and fully utilize the peak frame feature information, thereby further improving the precision and generalization of fatigue driving detection.

[0050] The fatigue driving detection method provided in the embodiment specifically includes the following steps:

[0051] A plurality of original video sequences containing driver facial images are obtained, and the original video sequences are labeled to construct a sample training set with a plurality of labeled fatigue driving original video sequences;

[0052] A fatigue driving detection model is constructed, which includes a spatio-temporal facial feature extraction network, a multi-head graph attention network, and a weighted graph attention feature fusion network;

[0053] The fatigue driving detection model is trained using the sample training set. The training process includes: based on the original video sequence, the spatio-temporal facial feature sequence is obtained through facial feature extraction and position coding; the mutual relationship between the spatio-temporal facial features and the importance weight of each feature to the final classification are learned based on the graph attention mechanism, the spatio-temporal facial feature sequence is updated based on the mutual relationship, the updated spatio-temporal facial feature sequence is weighted and fused based on the importance weight, the fused features are obtained, and the binary classification training is performed based on the fused features;

[0054] The original video sequence containing the driver's facial image is input into the trained fatigue driving detection model to complete the fatigue driving detection.

[0055] In the above detection method, firstly, multiple original video sequences containing driver facial images are acquired through a camera in the driver's cab. Each original video sequence is labeled to indicate whether it represents fatigued driving. A sample training set is constructed using multiple original video sequences labeled as indicating fatigued driving.

[0056] Secondly, a fatigue driving detection model is constructed and trained using a sample training set. The constructed fatigue driving detection model includes a spatiotemporal facial feature extraction network, a multi-head graph attention network, and a weighted graph attention feature fusion network: In the spatiotemporal facial feature extraction network, a spatiotemporal facial feature sequence is obtained based on the original video sequence through facial feature extraction and position encoding; in the multi-head graph attention network, the interrelationships between spatiotemporal facial features are learned based on the graph attention mechanism, and the spatiotemporal facial feature sequence is updated based on the interrelationships; in the weighted graph attention feature fusion network, the importance weight of each feature to the final classification is learned based on the graph attention mechanism, and the importance weight is weighted and fused with the updated spatiotemporal facial feature sequence to obtain the fused feature; finally, binary classification training for fatigue or non-fatigue is performed based on the fused feature to complete the training of the fatigue driving detection model.

[0057] The fatigue driving detection method provided in this embodiment is implemented by constructing a fatigue driving detection model (DCFGA-Net) based on deep features and graph attention mechanism. The network structure of this detection model is as follows: Figure 1 As shown, based on this, the above training process includes the following steps:

[0058] Step 1: In the spatiotemporal facial feature extraction network, based on the original video sequence, a spatiotemporal facial feature sequence is obtained through facial feature extraction and location encoding.

[0059] Step 1.1: Based on the original video sequence containing the driver's facial image, obtain the facial image sequence using a face detection algorithm.

[0060] In step 1.1 above, the MTCNN face detection algorithm is first used to detect the position coordinates of the face bounding boxes appearing in all video frames of the original video sequence, based on the position coordinates of the face bounding boxes, and the face images in the video frame images are cut out to obtain the face image sequence.

[0061] Step 1.2: Based on the facial image sequence, use a convolutional neural network to extract the visual representation features of each image in the facial image sequence, thereby constructing a spatial facial feature sequence.

[0062] In step 1.2 above, a VGG16 convolutional neural network pre-trained on the ImageNet dataset is used to extract the visual representation features (i.e., spatial facial features) of each face image in the facial feature sequence, thereby constructing a spatial facial feature sequence. The obtained spatial facial feature sequence is described as follows: Where N is the length of the input video sequence, i.e., the number of video frames in the video sequence, and d f The dimension of each spatial facial feature.

[0063] Step 1.3: Use the multi-frequency cosine position function to encode the spatial facial feature sequence, inject position information into each facial feature in the sequence, and obtain the spatiotemporal facial feature sequence for subsequent learning by the multi-head image attention network.

[0064] In step 1.3 above, considering that the absolute position of each video frame in a video segment is clear and there is a clear temporal relationship between various facial features, this embodiment constructs a position-encoded feature extractor (PEFE) and uses a multi-frequency cosine position function to inject the position information of facial features in the sequence. First, the multi-frequency cosine position function is expressed as:

[0065]

[0066]

[0067] Here, pos represents the position of the facial feature in the facial feature sequence, and base is a predefined hyperparameter.

[0068] The positional encoding sequence PE = {pe1, pe2, ..., pe...} is obtained by encoding in the above manner. N}, each item in the sequence Since spatial facial features have the same dimension, the above location encoding sequence and spatial facial feature sequence are added together to obtain the spatiotemporal facial feature sequence, which can be expressed as F = {f1, f2, ..., f...}. N},in, Each feature f in n =r n +pe n They all contain spatial feature information of the corresponding facial images and their temporal information in the video sequence.

[0069] Step 2: In the Multi-heads GAT network, the spatiotemporal facial features are learned based on the graph attention mechanism, and the spatiotemporal facial feature sequence is updated based on the interrelationships.

[0070] Step 2.1, based on the spatio-temporal facial feature sequence, each spatio-temporal facial feature is regarded as a node, and a directed complete graph is constructed.

[0071] In step 2.1 above, the embodiment adopts a multi-head graph attention network to construct the direct mutual relationship between the spatio-temporal facial features, and updates the feature information based on the learned mutual relationship. Unlike other application scenarios such as social networks, there is no explicit topological relationship between the features of a video sequence. Therefore, the embodiment makes the following assumptions: 1) there is extensive correlation between the spatio-temporal facial features in the same sequence, that is, each video frame (regarded as a node in the graph) is correlated with other video frames (nodes); 2) the relationship between the spatio-temporal facial features is directional, and the influence of node i on node j is not necessarily the same as the influence of node j on node i. Based on the above assumptions, the spatio-temporal facial feature sequence constitutes a completely connected directed graph.

[0072] The constructed directed graph is a triple (V(G), E(G), ψ(G)), where G is a graph, V(G) is a set of graph nodes, E(G) is a set of directed edges, and ψ(G) is a relationship function, each element in E(G) corresponds to an ordered pair of elements in V(G). The constructed complete directed graph is as shown in Figure 2 The complete directed graph contains three nodes V(G) = {h i , h j , h k} and nine directed edges E(G) = {e ij , e ji , e ik , e ki , e jk , e kj , e ii , e jj , e kk}, each directed edge is defined by the relationship function ψ(G). In the embodiment, V(G) is a spatio-temporal facial feature sequence, E(G) is a set of direct mutual relationships between spatio-temporal facial features, and ψ(G) is a self-attention function.

[0073] Step 2.2, a multi-head graph attention network composed of parallel multi-head self-attention layers is built, and a fully connected feedforward network layer is used before each graph attention head to map the input node features to a feature subspace.

[0074] Specifically, based on the above directed graph structure, a graph attention mechanism is introduced to construct the mutual relationship between the spatio-temporal facial features, and the learning process of the graph attention is as shown in Figure 3 In order to enable the fatigue driving detection model to autonomously learn and jointly focus on information from different feature subspaces, a multi-head graph attention architecture is constructed. In each graph attention head, a shared linear transformation matrix is used to map the spatio-temporal facial features F = {f1, f2,..., f N} into a new feature sequence H = {h1, h2,..., h

[0075] H = F · M = {f1M, f2M,..., f N M} (3)

[0076] Step 2.3, compute the attention coefficients for each pair of nodes in the directed graph through a shared self-attention mechanism, and construct an adjacency matrix therefrom. Specifically, a shared attention mechanism is used to compute H = {h1, h2,..., h N}, where is the attention coefficient for each pair of nodes. This computation method can be represented as:

[0077] e ij = LeakyReLU(a(h i || h j )) (4)

[0078] In the above formula, || represents vector splicing, e ij represents the importance of node j's feature to node i, and is the direct relationship between node i's feature and node j's feature. In order to make the attention coefficients of different nodes easy to compare, the softmax function is used to normalize the attention coefficients of all adjacent nodes j to node i, i.e.:

[0079]

[0080] According to the above empirical complete directed graph, the corresponding adjacency matrix can be obtained:

[0081]

[0082] Step 2.4, fuse the adjacency matrix learned autonomously with the node feature matrix, update the node features so that different node features are complementary and make up for information loss; then, concatenate the features of each attention head after updating to form new node features, i.e. obtain the updated spatio-temporal facial feature sequence.

[0083] Specifically, from the adjacency matrix, the output of each attention head can be obtained:

[0084]

[0085] where σ represents a nonlinear activation function. Then, concatenate the outputs of each attention head together to form the output of the multi-head graph attention network, i.e.

[0086]

[0087] wherein the eigenvector hd=1,2, …, heads represents the vector A n from the hd-th attention head. Each row O n represents an updated node feature, which is constructed based on the original spatio-temporal facial features and the learned inter-feature correlations by the attention mechanism.

[0088] Step 3, in the weighted graph attention feature fusion network (Weighted-GAT Feature Fusion), the importance weight of each feature to the final classification is learned based on the graph attention mechanism, and the updated spatio-temporal facial feature sequence is weighted and fused based on the importance weight to obtain the fused feature.

[0089] After passing through the multi-head graph attention network, the node feature is updated according to the learned correlation between features. The updated feature is more valuable to the final classification than the feature information initially extracted by the convolutional neural network. However, considering that different features have different contributions to the final classification, it is also necessary to distinguish the peak frame features and the non-peak frame features so that the peak frame features have greater contribution to the final classification. In addition, the sequence form output also needs to be further fused for binary classification. Based on the above needs, the embodiment gives a weighted graph attention feature fusion network, which realizes the following steps:

[0090] Step 3.1, a fully connected feedforward network is used to reduce the dimension of the updated node feature, and the node feature is projected into a unified linear space for the next graph learning.

[0091] Specifically, since each row O n contains different feature subspaces, in order to project them into a linear space for further graph learning, the embodiment uses a transformation matrix wherein d o is the dimension of the output vector, the node feature is projected into a unified linear space to complete the linear transformation.

[0092] Step 3.2, a shared self-attention mechanism is used to calculate the attention coefficient for each pair of nodes in the graph, and the adjacency matrix is constructed from the attention coefficient; the adjacency matrix is added by row to form a weight vector, and each element in the vector is the importance weight of the corresponding facial feature.

[0093] Specifically, the same as the multi-head attention network, the weighted graph attention feature fusion network uses a shared attention mechanism to calculate the attention coefficient for each pair of graph nodes:

[0094]

[0095] The output of the attention can be expressed as the product of the adjacency matrix composed of the above attention coefficients and the multi-head graph attention network output matrix after linear transformation, that is:

[0096]

[0097] The above formula (10) more clearly presents the entire calculation process, and the importance weight The acquisition method is clear.

[0098] Then, by row average pooling (RAP), the reconstructed facial features are weighted and fused with the importance weight of each feature to form the final fusion feature, which can be expressed as:

[0099]

[0100]

[0101] Through the above scheme, the final feature can be calculated as the weighted sum of the reconstructed facial feature O n and the importance weight , wherein Mo is the linear transformation matrix adopted by the weighted graph attention feature fusion network.

[0102] Step 4, based on the fusion feature, the two-class training of fatigue or not fatigue is performed to complete the training of the fatigue driving detection model.

[0103] Finally, the obtained fusion feature is input into the full connection layer for two-class training of fatigue or not fatigue, until the loss function converges, and the training of the fatigue driving detection model is completed.

[0104] In the iterative training process, the cross-entropy function is used to optimize the network model parameters, and the cross-entropy function formula is as follows:

[0105]

[0106] Wherein, K is the number of samples per batch, cls represents the class of classification (in this embodiment, it is two-class, cls=0 / 1, 0 represents not fatigue, and 1 represents fatigue), is the sample label, which represents that the i-th sample belongs to the cls class, is the result of the model output after softmax, which represents the probability of detecting the i-th sample as the cls class.

[0107] The back propagation algorithm is used to iteratively train the model by taking the cross-entropy loss function as the cost function. In this embodiment, the AdamW network parameter optimizer is selected, the training period is set to 50 epochs, the batch size is set to 16, the initial learning rate is set to 0.0001, and the learning rate is decayed after every half epoch with a decay rate of 0.05.

[0108] Through the above method, a trained fatigue driving detection model is obtained.

[0109] Finally, the original video sequence containing the driver's face image to be detected is input into the trained fatigue driving detection model to complete the fatigue driving detection.

[0110] Specifically, the fatigue driving detection proposed in this embodiment is further illustrated and verified by the following examples.

[0111] First, the hardware conditions of the verification example are as follows: Ubuntu 16.04L, CPU: intel i9-9900X, RAM 64G, and one TITAN XP graphics card; the software environment used is: python3.7, pytorch = 1.6.0, torchvision = 0.7.0; the data comes from the NTHU-DDD dataset. The NTHU-DDD dataset simulates five driver conditions, including daytime driver bare face, daytime driver wearing glasses, nighttime driver bare face, nighttime driver wearing glasses, and daytime driver wearing sunglasses; each condition has four videos containing four driver states, including non-drowsy state, drowsy state, slow blinking and nodding, and yawning. The dataset is divided into a training set, a test set, and a validation set. The training set contains 18 drivers, the test set contains 4 drivers, and the validation set contains 14 drivers.

[0112] The fatigue driving detection experiment is performed on the NTHU-DDD dataset. In order to make the experimental results more convincing, the k-fold cross-validation method is used. First, the dataset is randomly divided into k non-overlapping sample subsets, and the sample sizes of these subsets are approximately the same. Then, each subset is selected as the evaluation set in turn, and the other k-1 subsets are selected as the training set. The average of k experimental results is taken as the final result. In this example, k = 3 is selected.

[0113] The existing algorithm and the method (DCFGA-Net) proposed in this embodiment are used to detect fatigue driving, and the detection results are shown in Table 1. The evaluation indicators are recognition accuracy and F1 Score, and the larger the value, the better the effect. Obviously, the scheme described in this embodiment is superior to the previous algorithm in performance.

[0114] Table 1 detection results of different fatigue driving detection algorithms

[0115]

[0116]

[0117] In addition, the ablation experiment is also carried out in the embodiment, and the effectiveness of the position encoding, the multi-head graph attention network and the weighted graph attention feature fusion network is proved. Table 3, table 4 and table 5 respectively prove the performance improvement of the multi-head graph attention module, the weighted graph attention feature fusion module and the position encoding.

[0118] Table 2 detection results of fatigue driving detection algorithms using different multi-head attention networks

[0119]

[0120] Table 3 detection results of detection algorithms whether using weighted graph attention feature fusion network

[0121]

[0122] Table 4 detection results of detection algorithms using different position encodings

[0123]

[0124] In addition, Figure 4 The importance weight of each face image given by the weighted graph attention feature fusion network is shown, and it is further illustrated that the method described in the embodiment can distinguish the peak frame and the non-peak frame, so as to improve the detection accuracy.

[0125] Embodiment two

[0126] The embodiment provides a fatigue driving detection system based on deep features and graph attention mechanism, and the system comprises:

[0127] A sample training set construction module is configured to obtain a plurality of original video sequences containing driver face images, label the original video sequences, and construct a sample training set by using a plurality of original video sequences labeled with fatigue driving or not;

[0128] A detection model construction module is configured to construct a fatigue driving detection model, and the model comprises a space-time face feature extraction network, a multi-head graph attention network and a weighted graph attention feature fusion network.

[0129] The detection model training module is configured to train the fatigue driving detection model by using a sample training set. The training process comprises: obtaining a time-space face feature sequence by face feature extraction and position coding based on an original video sequence; learning the mutual relationship between the time-space face features and the importance weight of each feature to the final classification based on a graph attention mechanism, updating the time-space face feature sequence based on the mutual relationship, performing weighted fusion based on the importance weight and the updated time-space face feature sequence to obtain a fusion feature, and performing binary classification training based on the fusion feature.

[0130] The detection module is configured to input an original video sequence containing a driver's face image to be detected into the trained fatigue driving detection model to complete fatigue driving detection.

[0131] Embodiment three

[0132] The embodiment provides an electronic device, which comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the fatigue driving detection method based on deep features and a graph attention mechanism are completed.

[0133] Embodiment four

[0134] The embodiment also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the fatigue driving detection method based on deep features and a graph attention mechanism are completed.

[0135] The steps and methods in the above embodiments two to four correspond to the method embodiment one. The specific implementation can be understood with reference to the related description part of the embodiment one. The term "computer-readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets. It should also be understood as including any medium capable of storing, encoding, or carrying a set of instructions for execution by a processor and causing the processor to perform any method in the present application.

[0136] Those skilled in the art should understand that each module or step of the present application described above can be realized by a general computer device. Alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be respectively manufactured into individual integrated circuit modules, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0137] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0138] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A fatigue driving detection method based on deep features and graph attention mechanism, characterized in that, include: Multiple raw video sequences containing driver facial images are obtained, and the raw video sequences are labeled. A sample training set is constructed using multiple raw video sequences labeled with whether the driver is fatigued. A fatigue driving detection model is constructed, and the fatigue driving detection model is trained using a sample training set. The training process includes: obtaining a spatiotemporal facial feature sequence based on the original video sequence through facial feature extraction and location encoding; learning the interrelationships between spatiotemporal facial features and the importance weight of each feature to the final classification based on a graph attention mechanism; updating the spatiotemporal facial feature sequence based on the interrelationships; performing weighted fusion with the updated spatiotemporal facial feature sequence based on the importance weights to obtain fused features; and performing binary classification training based on the fused features; learning the interrelationships between spatiotemporal facial features based on a graph attention mechanism and updating the spatiotemporal facial feature sequence based on the interrelationships includes: Based on the spatiotemporal facial feature sequence, each spatiotemporal facial feature is regarded as a node to form node features, thereby constructing a directed complete graph; The input node features are mapped to a feature subspace, and attention coefficients are calculated for each pair of nodes in the directed graph through a shared self-attention mechanism to construct the adjacency matrix. The adjacency matrix and the node feature matrix are fused to update the node features. The updated node features of each attention focus are then concatenated to form new node features, thereby obtaining the updated spatiotemporal facial feature sequence. The original video sequence containing the driver's facial image is input into the trained fatigue driving detection model to complete the fatigue driving detection.

2. The fatigue driving detection method based on deep features and graph attention mechanism as described in claim 1, characterized in that, The fatigue driving detection model includes a spatiotemporal facial feature extraction network, a multi-head image attention network, and a weighted image attention feature fusion network.

3. The fatigue driving detection method based on deep features and graph attention mechanism as described in claim 1, characterized in that, The process of obtaining a spatiotemporal facial feature sequence based on the original video sequence through facial feature extraction and location encoding includes: Based on the original video sequence containing the driver's facial images, a facial image sequence is obtained through a face detection algorithm; Based on facial image sequences, visual representation features of each image in the facial image sequence are extracted using a convolutional neural network to construct a spatial facial feature sequence. The spatial facial feature sequence is encoded using a multi-frequency cosine position function, injecting position information into each facial feature in the sequence to obtain a spatiotemporal facial feature sequence.

4. The fatigue driving detection method based on deep features and graph attention mechanism as described in claim 3, characterized in that, The step of using a multi-frequency cosine position function to perform position encoding on the spatial facial feature sequence, injecting position information into each facial feature in the sequence, and obtaining a spatiotemporal facial feature sequence includes: Encoding is performed using a multi-frequency cosine position function to obtain a position-coded sequence; The location encoding sequence and the spatial facial feature sequence are added together to obtain the spatiotemporal facial feature sequence.

5. The fatigue driving detection method based on deep features and graph attention mechanism as described in claim 1, characterized in that, The importance weight of each feature to the final classification is learned based on the graph attention mechanism. This importance weight is then weighted and fused with the updated spatiotemporal facial feature sequence to obtain the fused features, including: Node features are projected into a unified linear space, and a shared self-attention mechanism is used to calculate attention coefficients for each pair of nodes in the graph. An adjacency matrix is ​​then constructed based on these attention coefficients. The adjacency matrix is ​​summed row by row to form a weight vector; each element in the weight vector is the weight of the corresponding facial feature importance. The updated spatiotemporal facial feature sequence is weighted and fused with the weight vector to obtain the fused feature.

6. The fatigue driving detection method based on deep features and graph attention mechanism as described in claim 1, characterized in that, The loss function is the cross-entropy function, and the formula for the cross-entropy function is: in, This refers to the number of samples per batch. The category that represents the classification. 0 indicates no fatigue, 1 indicates fatigue. These are sample labels, indicating the first... Each sample belongs to kind, Indicates the first Each sample was tested for The probability of a class.

7. A fatigue driving detection system based on deep features and graph attention mechanism, characterized in that, include: The sample training set construction module is used to acquire multiple original video sequences containing driver facial images, label the original video sequences, and construct a sample training set using multiple original video sequences labeled as indicating whether the driver is fatigued. The detection model construction module is used to construct a fatigue driving detection model, which includes a spatiotemporal facial feature extraction network, a multi-head image attention network, and a weighted graph attention feature fusion network. The detection model training module is used to train the fatigue driving detection model using a sample training set; The training process includes: obtaining a spatiotemporal facial feature sequence based on the original video sequence through facial feature extraction and location encoding; learning the interrelationships between spatiotemporal facial features and the importance weight of each feature to the final classification based on a graph attention mechanism; updating the spatiotemporal facial feature sequence based on the interrelationships; performing weighted fusion with the updated spatiotemporal facial feature sequence based on the importance weights to obtain fused features; and performing binary classification training based on the fused features; learning the interrelationships between spatiotemporal facial features based on a graph attention mechanism and updating the spatiotemporal facial feature sequence based on the interrelationships includes: Based on the spatiotemporal facial feature sequence, each spatiotemporal facial feature is regarded as a node to form node features, thereby constructing a directed complete graph; The input node features are mapped to a feature subspace, and attention coefficients are calculated for each pair of nodes in the directed graph through a shared self-attention mechanism to construct the adjacency matrix. The adjacency matrix and the node feature matrix are fused to update the node features. The updated node features of each attention focus are then concatenated to form new node features, thereby obtaining the updated spatiotemporal facial feature sequence. The detection module is used to input the original video sequence containing the driver's facial image into the trained fatigue driving detection model to complete the fatigue driving detection.

8. An electronic device, characterized in that: It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, complete the steps of a fatigue driving detection method based on deep features and graph attention mechanism as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of a fatigue driving detection method based on deep features and graph attention mechanism as described in any one of claims 1-6.

Images