A driver fatigue detection method

Abstract

The application discloses a driver fatigue detection method, which comprises the following steps: S1, training an MTCNN model structure by using a Wider_face face dataset to obtain a trained MTCNN model structure; S2, locating a face key feature in a face image by using the trained MTCNN model structure to obtain a positioned face image; S3, extracting an eye feature from the positioned face image to obtain an eye time feature and an eye space feature; S4, fusing the eye time feature and the eye space feature by using a multi-dimensional deep fusion network to obtain a fusion result; and S5, identifying a state of the fusion result to obtain a driver fatigue detection result. The application can convert eye features into time features and space features, deeply excavate local features from different angles, and maximize the fatigue detection effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fatigue detection, in particular to a driver fatigue detection method. BACKGROUND

[0002] It is recognized that the most practical method for judging fatigue state in the current market is to change the facial features. This method is a non-contact detection method, which will not cause discomfort to the driver, and can be easily applied to the actual driving environment. In previous studies, many single features were extracted to detect fatigue. The PERCLOS, which is the proportion of time when the eyes are closed in a certain time, is the most commonly used method in fatigue driving detection, and is also one of the most recognized reasonable methods. However, due to the low time resolution of PERCLOS, the detection effect based on only time features cannot meet the expected effect. At the same time, only based on spatial features can only learn features layer by layer, and lacks connectivity. SUMMARY

[0003] The purpose of the present application is to provide a driver fatigue detection method, which can convert eye features into time features and spatial features, so as to deeply mine local features from different angles, thereby maximizing the effect of fatigue detection.

[0004] The technical solution of the present application to solve the above technical problems is as follows:

[0005] The present application provides a driver fatigue detection method, which comprises:

[0006] S1: training a multi-task cascaded convolutional neural network (MTCNN) model structure by using a Wider_face face dataset, to obtain a trained MTCNN model structure;

[0007] S2: positioning the key features of the face in the face image by using the trained MTCNN model structure, to obtain a positioned face image;

[0008] S3: extracting eye features from the positioned face image, to obtain eye time features and eye spatial features;

[0009] S4: fusing the eye time features and the eye spatial features by using a multi-dimensional deep fusion network, to obtain a fusion result;

[0010] S5: identifying the state of the fusion result, to obtain a driver fatigue detection result.

[0011] Optionally, the MTCNN model structure comprises a P-Net layer, an R-Net layer and an O-Net layer arranged in sequence;

[0012] The P-Net layer is used to perform three convolution operations on input pictures of different scales to obtain a rough face window frame;

[0013] The R-Net layer is used to adjust the size of the rough face window frame, and then perform three convolution operations and full connection to obtain a more accurate face region position image;

[0014] The O-Net layer is used to adjust the size of the more accurate face region position image to 48x48 to obtain the final position of the face and the positions of five key feature points.

[0015] Optionally, in the S3,

[0016] The eye temporal feature is extracted by using PERCLOS.

[0017] Optionally, the eye temporal feature is extracted by using PERCLOS, which includes:

[0018]

[0019] Wherein, PERCLOS represents the percentage of closed eyes, interval represents the total time and interval = blink + CLOS + fixation + saccade, blink represents the eye blinking state, CLOS represents the closed eye state, fixation represents the eye fixation state, and saccade represents the eye scanning state.

[0020] Optionally, in the S3, the eye spatial feature is obtained by:

[0021] The calculation result of the PERCLOS is converted into a one-dimensional face feature signal;

[0022] The one-dimensional face feature signal is reconstructed into a three-dimensional multi-granularity by spatial projection to obtain the eye spatial feature.

[0023] Optionally, the one-dimensional face feature signal is:

[0024]

[0025] Wherein, N represents the number of samples, T Eye represents the final one-dimensional face feature signal, T i Eye (i∈[1,N]) represents the one-dimensional face feature signal of the i-th sample;

[0026] The eye spatial feature is:

[0027]

[0028] Where N represents the number of samples, S Eye S represents the final spatial features of the eye. i Eye (i∈[1,N]) represents the spatial features of the eye of the i-th sample, and R represents a real number.

[0029] Optionally, the Gaussian time-domain subnet includes a two-layer bidirectional long short-term memory network and a Gaussian error linear unit, wherein the bidirectional long short-term memory network includes a plurality of spaced memory unit structures;

[0030] Therefore, the multi-layer attention space subnet includes multiple attention network modules set at intervals. Before each attention network module, the multi-layer attention space subnet also includes a convolutional neural network and a ReLU function to perform dimensionality increase operations and linear transformations between networks.

[0031] Alternatively, the attention network module may be a global attention network module composed of channel attention network and spatial attention network;

[0032] The global attention network is:

[0033] G1 = M channel (F i )×F i

[0034] G2 = M spatial (G1)×G1

[0035] Where G1 represents the output of the first part of the global attention network, G2 represents the output of the second part of the global attention network, and M... channel and M spatial These are the output images of the channel attention network and the spatial attention network, respectively. i This represents the output of the i-th convolutional layer.

[0036] Alternatively, the loss function of the multi-dimensional deep fusion network is:

[0037]

[0038] Where m represents the size of the data, y i Indicates the actual value. This represents the predicted value.

[0039] The present invention has the following beneficial effects:

[0040] 1) This invention trains an MTCNN face detector. After using it to locate key facial features, it segments and extracts key facial features of the driver. Based on this, it can extract effective local and global features while compressing parameters, thereby reducing computational costs.

[0041] 2) This invention transforms eye features into temporal and spatial features to deeply explore local features from different angles, thereby maximizing the effect of fatigue detection;

[0042] 3) This invention was evaluated using a self-made dataset and the NTU-DDD public dataset. The experimental results show that the performance of the method provided by this invention is better than that of existing methods, and this invention has practical application value. Attached Figure Description

[0043] Figure 1 This is a flowchart of the driver fatigue detection method of the present invention;

[0044] Figure 2 The detection performance of the MTCNN model structure in different scenarios;

[0045] Figure 3 This is a schematic diagram of the structure of a multidimensional deep fusion network (MMDFNet);

[0046] Figure 4 This is a schematic diagram showing the results of obtaining the confusion matrix of the two test sets in this invention;

[0047] Figure 5 This is a schematic diagram illustrating the classification performance of MMDFNet on a portion of a self-made dataset.

[0048] Figure 6 This is a schematic diagram illustrating the classification performance of MMDFNet on a subset of NTHU-DDD samples. Detailed Implementation

[0049] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0050] This invention provides a method for detecting driver fatigue, with reference to... Figure 1 As shown, the driver fatigue detection method includes:

[0051] S1: Train the MTCNN model structure using the Wider_face face dataset to obtain the trained MTCNN model structure;

[0052] A well-trained MTCNN model can detect faces and extract ROI regions (regions of interest) even in low-light conditions.

[0053] S2: Use the trained MTCNN model structure to locate key facial features in the face image to obtain the located face image;

[0054] Optionally, the MTCNN model structure includes: a P-Net layer, an R-Net layer, and an O-Net layer arranged sequentially;

[0055] The P-Net layer is used to perform three convolution operations on input images of different scales to obtain an approximate face window box.

[0056] The R-Net layer is used to resize the approximate face window frame, and after three convolution operations, a more accurate face region location image is obtained after full connection.

[0057] The O-Net layer is used to adjust the size of the more accurate face region location image to 48×48 to obtain the final position of the face and the positions of 5 key feature points.

[0058] Specifically, before inputting into the P-Net layer, a pyramid is constructed to obtain images of different scales. These images are then input into the P-Net layer, and after three convolution operations, a rough face window is initially obtained.

[0059] Then, the R-Net layer is used to adjust the size of the face window to 24×24, and after three convolution operations, a more accurate face region location image is obtained after full connection.

[0060] Finally, the O-Net layer is used to adjust the face region image to 48×48 for final confirmation of the face region location, resulting in the final face location and the positions of 5 key feature points. The MTCNN model structure can simultaneously perform different face classifications, face region bounding, and key feature localization, exhibiting good robustness and making it highly suitable for real-world driving environments. Detection performance in various scenarios is as follows: Figure 2 As shown.

[0061] S3: Extract eye features from the located face image to obtain temporal and spatial eye features;

[0062] Alternatively, in S3:

[0063] The ocular temporal features were extracted using PERCLOS.

[0064] Alternatively, the extraction of the ocular temporal features using PERCLOS includes:

[0065]

[0066] Wherein, PERCLOS represents the percentage of time the eyes are closed, interval represents the total time and interval = blink + CLOS + fixation + saccade, blink represents the blinking state, CLOS represents the closed state, fixation represents the fixation state of the eyes, and saccade represents the scanning state of the eyes.

[0067] Furthermore, the ocular spatial features are obtained through the following methods:

[0068] The calculation results of PERCLOS are converted into one-dimensional facial feature signals;

[0069] The one-dimensional facial feature signal is reconstructed in three dimensions using spatial projection to obtain the spatial features of the eye.

[0070] This invention defines a one-dimensional facial feature signal as:

[0071]

[0072] Where N represents the number of samples, T Eye T represents the final one-dimensional facial feature signal. i Eye (i∈[1,N]) represents the one-dimensional facial feature signal of the i-th sample;

[0073] The spatial features of the eye are:

[0074]

[0075] Where N represents the number of samples, S Eye S represents the final spatial features of the eye. i Eye (i∈[1,N]) represents the spatial features of the eye of the i-th sample, and R represents a real number.

[0076] S4: The ocular temporal features and the ocular spatial features are fused using a multi-dimensional deep fusion network to obtain the fusion result;

[0077] This invention inputs the temporal features of the eye and the spatial features of the eye into a multi-scale Gaussian temporal subnet (MGT-SubNet) and a multi-layer attention spatial subnet (MAS-SubNet), respectively, and finally outputs the classification result through a fully connected layer fusion.

[0078] The multi-dimensional deep fusion network includes a multi-scale Gaussian temporal subnetwork and a multi-layer attention space subnetwork, referencing... Figure 3As shown (where MGT-SubNet and MAS-SubNet represent multi-scale Gaussian temporal subnets and multi-layer attention spatial subnets, respectively; LSTM represents long short-term memory network; GeLU represents Gaussian linear error unit; Output represents output; Convolution represents convolution; GAM represents global attention mechanism; FC represents fully connected layer; Alert and Fatigue represent alertness and fatigue, respectively; and Classifier represents classifier), the Gaussian temporal subnet includes a two-layer bidirectional long short-term memory network and Gaussian linear error unit, and the bidirectional long short-term memory network includes multiple memory unit structures with intervals;

[0079] The computation result out for a single-layer bidirectional long short-term memory network is:

[0080] out = GeLU(P) i +P' i )

[0081] P i P' represents the forward output of the LSTM. i Indicates the inverse output result of LSTM

[0082] Finally, the output result T of the multi-scale Gaussian time-domain subnet after processing by Gaussian error linear units is... out for:

[0083]

[0084] This represents the output of the first branch of the Gaussian time-domain subnet. This represents the output of the second branch of the Gaussian time-domain subnet.

[0085] Therefore, the multi-layer attention spatial subnetwork is based on a global attention mechanism composed of channel attention networks and spatial attention networks to extract spatial features of the eye. Specifically, it includes multiple attention network modules with intervals. Before each attention network module, the multi-layer attention spatial subnetwork also includes a convolutional neural network and a ReLU function to perform dimensionality increase operations and linear transformations between networks.

[0086] The calculation process for the first convolutional layer is as follows:

[0087] F1 = ReLU(W1*S) Eye )

[0088] Where F1 represents the output of the first convolutional layer, W1 represents the weights of the convolutional layer, and S... Eye This describes the spatial characteristics of the eye.

[0089] Meanwhile, the calculation process for other convolutional layers involves multiplying the weights of the convolutional layer by the output value of the attention network to obtain the feature weights. The calculation process is as follows:

[0090]

[0091] F i This represents the output of the i-th convolutional layer, where W1 represents the weights passed through each convolutional layer. This represents the output value of each layer of the attention network.

[0092] The attention network module is a global attention network module based on channel attention network and spatial attention network; the global attention network is:

[0093] G1 = M channel (F i )×F i

[0094] G2 = M spatial (G1)×G1

[0095] Where G1 represents the output of the first part of the global attention network, G2 represents the output of the second part of the global attention network, and M... channel and M spatial These are the output images of the channel attention network and the spatial attention network, respectively. i This represents the output of the i-th convolutional layer.

[0096] Alternatively, the loss function of the multi-dimensional deep fusion network is:

[0097]

[0098] Where m represents the size of the data, y i Indicates the actual value. This represents the predicted value.

[0099] S5: Perform state recognition on the fusion result to obtain the driver fatigue detection result.

[0100] Based on the above technical solution, the present invention uses a fully connected layer to reduce the dimensionality of the results calculated by the Gaussian time domain subnet and the results calculated by the multi-layer attention space subnet, and outputs them with Softmax for state recognition and classification.

[0101]

[0102] in, σ represents the final output of the network. softThis indicates that the Sigmoid function is used for the final classification, FC represents a fully connected layer, and T' and S' represent the results calculated after the Gaussian temporal subnet and the results calculated after the multi-layer attention spatial subnet, respectively.

[0103] As one specific implementation, the present invention utilizes the following eight indicators to evaluate the fatigue detection method provided by the present invention, specifically:

[0104]

[0105]

[0106]

[0107]

[0108] Missrate = 1 - Recall

[0109]

[0110]

[0111]

[0112] Where TP represents correctly identifying the alert state, TN represents correctly identifying the fatigued state, FP represents incorrectly identifying the alert state, FN represents incorrectly identifying the fatigued state, Accuracy represents accuracy, Precision represents precision, Recall represents recall, Specificity represents specificity, Missrate represents false positive rate, G-Mean represents G-means, F1_score represents F1 score, Kappa represents Kappa score, and P represents the false positive rate. e Indicates random consistency and N represents the number of samples.

[0113] Tables 3 and 4 show the performance of the multi-scale Gaussian temporal subnet and the multi-layer attention spatial subnet on two datasets. It can be observed that the classification accuracy based on temporal features is 85.95% and 82.12%, respectively, indicating that PERCLOS has low temporal resolution, and extracting only temporal features to detect fatigue cannot achieve the expected results. Furthermore, the multi-scale Gaussian temporal subnet has low recall and G-mean, indicating a deficiency in imbalanced classification. However, the multi-layer attention spatial subnet performs better in learning the spatial features of PERCLOS, achieving accuracies of 95.80% and 89.02%, respectively. The multi-layer attention spatial subnet also shows improvements in recall and G-mean, but still does not reach the best performance. Therefore, Table 5 presents the performance of the model after fusing multi-dimensional features. It can be seen that fusing temporal and spatial features significantly improves all performance metrics of the model. It not only improves the overall detection accuracy but also effectively addresses the imbalanced classification problem. It should be noted that the performance metrics in Tables 3, 4, and 5 are based on... Figure 4 (a) Confusion matrix of a multi-scale Gaussian temporal subnet on self-made data; (b) Confusion matrix of a multi-scale Gaussian temporal subnet on NTHU-DDD; (c) Confusion matrix of a multi-layer attention space subnet on self-made data; (d) Confusion matrix of a multi-layer attention space subnet on NTHU-DDD; (e) Confusion matrix of a multi-stage, multi-dimensional deep fusion network on self-made data; (f) Confusion matrix of a multi-stage, multi-dimensional deep fusion network on NTHU-DDD. The confusion matrices were calculated from the results of these confusion matrices.

[0114] This invention also uses the trained model to test different scenarios in NTHU-DDD, as shown in Table 6. It can be seen that the model's performance in detecting drivers wearing sunglasses is relatively low, which is related to the significant occlusion of the driver's eyes. Furthermore, the model performs well in the other four scenarios (especially during the day without glasses), indicating that the model has strong scene transfer capabilities and practical application value.

[0115] Figure 5 and Figure 6(Where, TP in glasses represents daytime scenes where wearing glasses is correctly classified as alert, TN innoglasses represents daytime scenes where not wearing glasses is correctly classified as fatigued, TP in sunglasses represents scenes where wearing sunglasses is correctly classified as alert, TN in glasses represents daytime scenes where wearing glasses is correctly classified as fatigued, TP in night_noglasses represents nighttime scenes where not wearing glasses is correctly classified as alert, TN in sunglasses represents nighttime scenes where wearing sunglasses is correctly classified as fatigued, FN in night_glasses represents nighttime scenes where wearing glasses is incorrectly classified as alert, FP in night_noglasses represents nighttime scenes where not wearing glasses is incorrectly classified as fatigued, and FP innight_glasses represents nighttime scenes where wearing glasses is incorrectly classified as fatigued.) Samples from two randomly selected datasets are presented. Figure 5 In practice, it was found that people with smaller eyes are more easily mistaken for fatigued individuals even when alert. Furthermore, a driver's smile makes their eyes appear smaller, potentially causing confusion in the model's recognition process. Similarly, in... Figure 6 In (k), the driver's eyes are almost closed when laughing, causing the model to misjudge it as fatigue. However, whether in a normal or fatigued state, it is dangerous to squint or even close the eyes when laughing. This can easily lead to traffic accidents when the driver is distracted.

[0116] Table 3 Performance metrics of multi-scale Gaussian time-domain subnets

[0117] Evaluation metrics Self-made dataset NTHU-DDD Accuracy 0.8595 0.8212 Precision 0.9120 0.8181 Recall 0.8329 0.8397 False detection rate 0.1671 0.1603 F1 score 0.8707 0.8287 Specificity 0.8943 0.8016 G-mean 0.8631 0.8204 Kappa 0.7175 0.6418

[0118] Table 4 Performance metrics of MAS-SubNet

[0119] Evaluation metrics Self-made dataset NTHU-DDD Accuracy 0.9580 0.8902 Precision 0.9704 0.8524 Recall 0.9498 0.9341 False detection rate 0.0502 0.0659 F1 score 0.9600 0.8914 Specificity 0.9673 0.8492 G-mean 0.9585 0.8906 Kappa 0.9158 0.7807

[0120] Table 5 MMDFNet Performance Metrics

[0121] Evaluation metrics Accuracy Precision Recall False detection rate F1 score Specificity G-mean Kappa Self-made dataset 0.9752 0.9739 0.9782 0.0218 0.9760 0.9720 0.9751 0.9503 NTHU-DDD 0.9280 0.9713 0.9004 0.0996 0.9345 0.9647 0.9320 0.8549

[0122] Table 6. Classification performance of multi-stage, multi-dimensional deep fusion networks in different scenarios in NTHU-DDD.

[0123] Evaluation metrics Accuracy Precision F1 score Driver without glasses 0.9538 0.9877 0.9620 Driver with glasses 0.9515 0.9754 0.9546 Driver without glasses at night 0.9329 0.9512 0.9472 Driver with glasses at night 0.9107 0.9568 0.9233 Driver with sunglasses 0.8891 0.9352 0.9067

[0124] As shown in Tables 7 and 8, this invention compares its proposed method with existing state-of-the-art deep learning methods on two separate datasets. It is important to emphasize that NTHU-DDD is a publicly available dataset, so its results are directly used for comparison. For the self-made dataset, this invention reproduces and optimizes the model according to the description in the paper before comparing it with the proposed method. Table 7 shows that our proposed method outperforms other existing methods on the self-made dataset, with average accuracy and F1 score improvements of approximately 2% and 4%, respectively. Table 8 shows that this invention's method outperforms most models except for the ensemble of two InceptionV3 models. This may be due to differences in sample size; these models used extremely complex models and large sample sizes for their studies. Even though they achieved high detection results, they severely lacked practical application capabilities. Furthermore, compared to other methods, this invention achieves good accuracy using only a single feature, thus possessing potential for practical application.

[0125] Table 7 compares the classification performance of the proposed method on the self-made dataset with other methods.

[0126]

[0127] Table 8 compares the classification performance of the proposed method on NTU-DDD with other methods.

[0128]

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting driver fatigue, characterized in that, The driver fatigue detection method includes: S1: Train the MTCNN model structure using the Wider_face face dataset to obtain the trained MTCNN model structure; S2: Use the trained MTCNN model structure to locate key facial features in the face image to obtain the located face image; S3: Extract eye features from the located face image to obtain temporal and spatial eye features; S4: The ocular temporal features and the ocular spatial features are fused using a multi-dimensional deep fusion network to obtain the fusion result; S5: Perform state recognition on the fusion result to obtain the driver fatigue detection result; In S3: The ocular temporal features were extracted using PERCLOS; In step S3, the spatial features of the eye are obtained in the following manner: The calculation results of PERCLOS are converted into one-dimensional facial feature signals; The one-dimensional facial feature signal is reconstructed in three dimensions using spatial projection to obtain the spatial features of the eye. The multi-dimensional deep fusion network includes a Gaussian temporal subnet, a multi-layer attention space subnet, and a fully connected layer. The Gaussian temporal subnet processes eye temporal features, the multi-layer attention space subnet processes eye spatial features, and the fully connected layer fuses and outputs the calculation results of the Gaussian temporal subnet and the multi-layer attention space subnet. The driver fatigue detection results include: using Softmax output on the fusion results for state recognition and classification: in, This represents the final output of the network. This indicates that the Sigmoid function is used for the final classification. Indicates a fully connected layer. and These represent the results calculated using the Gaussian time-domain subnet and the results calculated using the multi-layer attention space subnet, respectively. The Gaussian time-domain subnet includes a two-layer bidirectional long short-term memory network and a Gaussian error linear unit. The bidirectional long short-term memory network includes multiple memory unit structures arranged at intervals. Therefore, the multi-layer attention space subnet includes multiple attention network modules set at intervals. Before each attention network module, the multi-layer attention space subnet also includes a convolutional neural network and a ReLU function to perform dimensionality increase operations and linear transformations between networks.

2. The driver fatigue detection method according to claim 1, characterized in that, The MTCNN model structure includes: a P-Net layer, an R-Net layer, and an O-Net layer arranged sequentially; The P-Net layer is used to perform three convolution operations on input images of different scales to obtain an approximate face window box. The R-Net layer is used to resize the approximate face window frame, and after three convolution operations, a more accurate face region location image is obtained after full connection. The O-Net layer is used to adjust the size of the more accurate face region location image to 48×48 to obtain the final position of the face and the positions of 5 key feature points.

3. The driver fatigue detection method according to claim 1, characterized in that, The extraction of the ocular temporal features using PERCLOS includes: in, Indicates the percentage of eyes closed. Indicates the total time and , Indicates the blinking state. This indicates a closed-eye state. Indicates the state of eye fixation. This indicates the scanning status of the eye.

4. The driver fatigue detection method according to claim 1, characterized in that, The one-dimensional facial feature signal is: in, N Indicates the number of samples. This represents the final one-dimensional facial feature signal. Indicates the first i One-dimensional facial feature signals of a sample; The spatial features of the eye are: in, N Indicates the number of samples. This represents the final spatial features of the eye. Indicates the first i Ocular spatial features of each sample R Represents a real number.

5. The driver fatigue detection method according to claim 1, characterized in that, The attention network module is a global attention network module based on channel attention network and spatial attention network; The global attention network is: in, This represents the output of the first part of the global attention network. This represents the output of the second part of the global attention network. and These are the output images of the channel attention network and the spatial attention network, respectively. This represents the output of the i-th convolutional layer.

6. The driver fatigue detection method according to any one of claims 1-5, characterized in that, The loss function of the multi-dimensional deep fusion network is: in, Indicates the size of the data. Indicates the actual value. This represents the predicted value.

Images