A fatigue driving detection method fusing space-time features

Abstract

The application relates to a fatigue driving detection method fusing space-time features and belongs to the fatigue driving field. In view of the problems that existing fatigue driving detection methods ignore the time features of fatigue, are difficult to detect dynamic and key features of fatigue, and are low in fatigue detection accuracy, a fatigue driving detection method fusing space features and time features is provided. First, a target driving video is subjected to face detection by using a multi-task convolutional neural network, deep fatigue space features of a global face of a driver are extracted, then an LSTM-FCN network is used to study long-term dependence between fatigue video frame sequences, time features of fatigue are extracted, and finally fatigue driving detection is completed. The method explores potential nonlinear relationships between continuous fatigue video frames, is high in fatigue driving detection accuracy, can realize real-time detection, and effectively judges fatigue.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of fatigue driving and relates to a fatigue driving detection method fusing space-time features. BACKGROUND

[0002] Fatigue driving is one of the main causes of traffic accidents. If the driver is reminded in time when driving fatigue, the occurrence of traffic accidents can be greatly reduced or even avoided. Therefore, fatigue driving detection is crucial. At present, fatigue driving detection methods mainly include the following three kinds: driver physiological state detection, vehicle driving state detection and driver facial feature detection. The physiological state detection judges fatigue by detecting electrocardiogram (ECG) and electroencephalogram (EEG) signals of the driver. These signals can directly reflect the physical and mental state of the driver, thereby achieving the most accurate prediction. However, the driver needs to wear related instruments, which is difficult to apply in reality. The vehicle state detection observes the vehicle trajectory, steering wheel angle and acceleration, but is greatly affected by road conditions and weather, and has low accuracy. Therefore, the fatigue driving detection based on facial features has become the mainstream of current research because it does not need to wear related instruments, is non-invasive and has high accuracy. The fatigue driving detection based on facial features mainly judges whether the driver is fatigue by detecting behaviors such as closing eyes, yawning and dozing off. Zheng Weicheng et al. extract eye features and mouth features through a convolutional neural network (CNN), and then use parameters such as PERCLOS (percentage of eye closure) and FOM (yawn ratio) to judge whether the driver is fatigue. However, this method does not consider the time feature of fatigue, and only uses single-frame picture detection, which can not distinguish between normal blinking and fatigue blinking, and between yawning and speaking. Yu et al. use a 3D CNN to convolve the fatigue driving video to extract time features, but the 3D CNN network model is large and has high computational cost, which is difficult to realize real-time fatigue detection, and the accuracy of the model is not high. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a fatigue driving detection method fusing space-time features. The spatial features of the face are extracted by adding a non-local attention module to the Xception network to learn the facial fatigue representation. Then, the LSTM-FCN network is used to extract the time features of fatigue to learn the long-term time dependence of fatigue, thereby completing the fatigue driving detection.

[0004] To achieve the above purpose, the present application provides the following technical solutions.

[0005] A fatigue driving detection method fusing space-time features, the method comprising the following steps:

[0006] S1: input a driving video;

[0007] S2: face detection is performed;

[0008] S3: the Drowsy-CNN network extracts face spatial features;

[0009] S4: the LSTM-FCN extracts temporal features;

[0010] S5: the Softmax classifies whether fatigue or not.

[0011] Optionally, the S1 includes the following steps:

[0012] The input driver face video is segmented, and each video mixes fatigue and non-fatigue states. Each video is cut according to the corresponding fatigue label to ensure that each video has only one state of fatigue or non-fatigue. The segmented video is randomly divided into a training set and a test set, and the training set accounts for 80% and the test set accounts for 20%.

[0013] Optionally, the S2 includes the following steps:

[0014] S2-1: the multi-task convolutional neural network MTCNN is used to detect the face in the segmented video in step S1-1, and the face region is extracted. The detected face region is expanded by 40 pixels, and finally the resolution of the face region video frame sequence image is 160x160;

[0015] S2-2: the extracted face video frame sequence image is converted into a numpy array and saved in an npy file.

[0016] Optionally, the S3 includes the following steps:

[0017] S3-1: the face video frame sequence extracted in step S2 is sent to the Drowsy-CNN network for training, and the driver face spatial fatigue feature is extracted. The Drowsy-CNN network is a network model based on the Xception network and the nonlocal attention mechanism;

[0018] S3-2: the nonlocal self-attention module is calculated, and the calculation formula is:

[0019]

[0020] where x i and x j represent a certain spatiotemporal position in the input video frame, f(x i ,x j ) is a similarity measurement function, and an Embedded Gaussian function is used to calculate the similarity, and the calculation formula is:

[0021]

[0022] where θ and The functions respectively represent linear transformations on x i and x j , i.e. convolution operations, and the normalization function C(x) corresponding to the similarity measure function is:

[0023]

[0024] S3-3: The non-local attention module first performs three 1x1x1 convolution operations on the input feature map to obtain θ, φ, and g features, then performs a point multiplication operation on θ and φ to calculate the autocorrelation in the features, performs a softmax operation on the autocorrelation features to calculate the normalized attention weight coefficient, then applies this coefficient to the feature matrix g, performs a 1x1x1 convolution, and finally performs a residual connection with the original input feature map to output the driver's face spatial feature vector.

[0025] Optionally, the S4 includes the following steps:

[0026] S4-1: The spatial feature vector obtained in step S3-3 is input into the LSTM-FCN network to mine long-term temporal dependencies in the fatigue video.

[0027] S4-2: The LSTM network contains two layers of LSTM, the first layer of LSTM has 1024 hidden units, the input of which is the output of S3-3, and the second layer has 512 hidden units, the input of which is the output of the first layer of LSTM, and a dropout layer is added after each layer to prevent overfitting.

[0028] S4-3: The FCN network contains three one-dimensional time convolution blocks, which usually apply multiple 1xk 1D filters to the input sequence to extract time features. Each time convolution block is accompanied by a batch normalization layer BN and a ReLU activation layer. The BN layer accelerates the training of the network and improves the generalization ability of the network, and the ReLU activation layer enables the network to have nonlinear fitting capability. The formula for each layer of the time convolution block in the FCN is:

[0029]

[0030] s = BN(y)

[0031] h = ReLU(s)

[0032] where Represents a convolution operation, and the filter size in each time convolution block is 64, 128 and 256 respectively, and the step is 1, 2 and 4 respectively.

[0033] Optionally, the S5 comprises the following steps:

[0034] S5-1: Concat feature fusion of the LSTM network and the FCN network to obtain a space-time feature vector;

[0035] S5-2: Obtain the probability distribution of the driver's fatigue driving using the softmax classifier, and obtain the fatigue classification result using the argmax function;

[0036] S5-3: According to the result of fatigue driving detection, it is decided whether to give the driver a fatigue driving warning.

[0037] The beneficial effects of the present application are:

[0038] 1) The Xception network model of the present application fuses non-local attention, learns and trains the convolutional neural network to learn the area that needs to be focused on in each frame of video, improves the ability to extract the spatial features of the driver's face fatigue, and makes the fatigue driving detection more accurate.

[0039] 2) The LSTM-FCN model proposed in the present application can capture the fatigue action process of the input video frame sequence over time, not only can extract effective face spatial features in a single frame of picture, but also can mine time information in the video, better judge fatigue, and distinguish between speaking and yawning.

[0040] Other advantages, objects and features of the present application will be set forth in part in the following specification, and in part will become apparent to those skilled in the art from the following, or can be learned from the practice of the present application. The objects and other advantages of the present application can be realized and obtained by the following description. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to make the purpose, technical scheme and advantages of the present application more clear, the preferred detailed description of the present application will be combined with the drawings as follows, wherein:

[0042] Figure 1 The overall flowchart of the present application;

[0043] Figure 2 The nonlocal attention module of the present application;

[0044] Figure 3 The LSTM-FCN module of the present application. DETAILED DESCRIPTION

[0045] The present application is described herein with reference to particular non-limiting embodiments. Various modifications and changes can be made thereto by those skilled in the art without departing from the spirit and scope of the application as set forth in the claims. The disclosure is not to be limited to the specific embodiments described but only by the claims. The following examples are provided to further illustrate the application.

[0046] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0047] The same or similar components in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only for illustrative purposes, and cannot be understood as a limitation of the present application, for those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0048] Figure 1 The main flow of the present application is shown, and the main implementation process is:

[0049] Step S1, input the driving video, including the following sub-steps:

[0050] Step S1-1, segment the input driver face video, because each video mixes fatigue and non-fatigue state, so each video is cut according to the corresponding fatigue label, so as to ensure that each video has only one state (fatigue or non-fatigue), and the segmented video is randomly divided into a training set and a test set, the training set accounts for 80%, and the test set accounts for 20%.

[0051] Step S2, face detection, including the following sub-steps:

[0052] Step S2-1, the segmented video in step S1-1 is subjected to face detection by using a multi-task convolutional neural network (MTCNN) to extract a face region. In view of the nodding and other fatigue features, the image close to the face region can not enable the network to better learn the features about the drowsy nodding, and the extracted face region is expanded by 40 pixel points, and finally a face region video frame sequence image with a resolution of 160x160 is obtained.

[0053] Step S2-2, the extracted face video frame sequence image is converted into a numpy array and saved in an npy file.

[0054] Step S3, the Drowsy-CNN network extracts face spatial features, including the following sub-steps:

[0055] Step S3-1, the face video frame sequence extracted in step S2 is sent into the Drowsy-CNN network for training to extract the driver face spatial fatigue features. The Drowsy-CNN network is a network model based on an Xception network and fusing a nonlocal attention mechanism.

[0056] Step S3-2, a nonlocal self-attention module is calculated, and the calculation formula is as follows:

[0057]

[0058] wherein x i and x j respectively represent a certain space-time position in the input video frame, f(x i ,x j ) is a similarity measurement function, and the Embedded Gaussian function is used to calculate the similarity, and the calculation formula is as follows:

[0059]

[0060] wherein θ and are linear transformation functions for x i and x j , that is, convolution operation, and the normalization function C(x) corresponding to the similarity measurement function is as follows:

[0061]

[0062] Step S3-3, the non-local attention module is as follows: Figure 2As shown, firstly, three 1x1x1 convolution operations are performed on the input feature map respectively to obtain theta, phi and g features, then point multiplication is performed on theta and phi to calculate the autocorrelation in the features. Next, the autocorrelation features are subjected to a softmax operation to calculate the normalized attention weight coefficient, and then the coefficient is correspondingly applied to the feature matrix g, and a 1x1x1 convolution is performed, and finally the residual connection is performed with the original input feature map, and finally output.

[0063] Step S4, the LSTM-FCN extracts the time feature, including the following sub-steps:

[0064] Step S4-1, the spatial feature vector obtained in step S3-3 is input into the LSTM-FCN network to mine the long-term time dependence in the fatigue video, and the LSTM-FCN network model diagram is as shown in Figure 3 .

[0065] Step S4-2, the LSTM network contains two layers of LSTM, the first layer of LSTM has 1024 hidden units, and the input is the output in S3-3, the second layer has 512 hidden units, and the input is the output of the first layer of LSTM, and a dropout layer is added after each layer to prevent overfitting.

[0066] Step S4-3, the FCN network contains three one-dimensional time convolution blocks, which usually apply multiple 1xk 1D filters on the input sequence to extract time features, and each time convolution block is accompanied by a batch normalization layer (Batch Normalization BN) and a ReLU activation layer. The BN layer can accelerate the training of the network and improve the generalization ability of the network, and the ReLU activation layer makes the network have nonlinear fitting ability. The formula of each layer of the time convolution block in the FCN is:

[0067]

[0068] s=BN(y)

[0069] h=ReLU(s)

[0070] wherein represents a convolution operation, and the filter size in each time convolution block in this paper is 64, 128 and 256 respectively, and the step is 1, 2 and 4 respectively.

[0071] Step S5, Softmax classification determines whether fatigue, including the following sub-steps:

[0072] Step S5-1, the LSTM network and the FCN network are subjected to Concat feature fusion to obtain a spatio-temporal feature vector.

[0073] Step S5-2, using a softmax classifier to obtain the probability distribution of the driver's fatigue driving, and using an argmax function to obtain the result of the fatigue classification.

[0074] Step S5-3, determining whether to give the driver a fatigue driving warning according to the result of the fatigue driving detection.

[0075] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the purpose and scope of the technical solutions, and they should all be covered in the scope of the claims of the present application.

Claims

1. A fatigue driving detection method integrating spatiotemporal features, characterized in that: The method includes the following steps: S1: Input driving video; including the following steps: The input driver facial video is segmented, and each video contains a mixture of fatigue and non-fatigue states. Each video is cut according to the corresponding fatigue label to ensure that each video has only one state of fatigue or non-fatigue. The segmented videos are randomly divided into training set and test set, with the training set accounting for 80% and the test set accounting for 20%. S2: Perform face detection; including the following steps: S2-1: The multi-task convolutional neural network MTCNN is used to perform face detection on the video segmented in step S1 and extract the face region; the detected face region is expanded by 40 pixels to finally obtain a video frame sequence image of the face region with a resolution of 160×160. S2-2: Convert the extracted face video frame sequence images into NumPy arrays and save them in an npy file; S3: The Drowsy-CNN network extracts spatial features of the human face; including the following steps: S3-1: The facial video frame sequence extracted in step S2 is fed into the Drowsy-CNN network for training to extract the driver's facial spatial fatigue features; the Drowsy-CNN network is a network model based on the Xception network and fused with a nonlocal attention mechanism. S3-2: Calculate the nonlocal self-attention module, the formula is as follows: Where x i and x j f(x) represents a spatiotemporal location in the input video frame. i ,x j The similarity measure function is , and the EmbeddedGaussian function is used to calculate the similarity. The formula is as follows: Where θ and The functions respectively represent the expression for x i and x j Performing a linear transformation, i.e., a convolution operation, the normalization function C(x) corresponding to the similarity metric function is: S3-3: The non-local attention module first performs three 1×1×1 convolution operations on the input feature map to obtain features θ, φ, and g. Then, it performs a dot product operation on θ and φ to calculate the autocorrelation in the features. A softmax operation is performed on the autocorrelation features to calculate the normalized attention weight coefficients. Then, these coefficients are applied to the feature matrix g, and another 1×1×1 convolution is performed. Finally, a residual connection is performed with the original input feature map to output the driver's facial spatial feature vector. S4: LSTM-FCN extracts temporal features; including the following steps: S4-1: Input the spatial feature vector obtained in step S3-3 into the LSTM-FCN network to mine the long-term temporal dependencies in fatigue videos; S4-2: The LSTM network consists of two LSTM layers. The first LSTM layer has 1024 hidden units, and its input is the output in S3-3. The second layer has 512 hidden units, and its input is the output of the first LSTM layer. A dropout layer is added after each layer to prevent overfitting. S4-3: The FCN network contains three one-dimensional temporal convolutional blocks. These blocks typically apply multiple 1×k 1D filters to the input sequence to extract temporal features. Each convolutional block is accompanied by a batch normalization (BN) layer and a ReLU activation layer. The BN layer accelerates network training and improves generalization ability, while the ReLU activation layer enables the network to perform non-linear fitting. The formula for each layer in the FCN temporal convolutional block is: s = BN(y) h = ReLU(s) in This indicates a convolution operation, with filter sizes of 64, 128, and 256 in each convolutional block, and strides of 1, 2, and 4, respectively. S5: Softmax classification determines whether fatigue exists.

2. The fatigue driving detection method integrating spatiotemporal features according to claim 1, characterized in that: S5 includes the following steps: S5-1: Concat feature fusion of LSTM network and FCN network to obtain spatiotemporal feature vector; S5-2: Use the softmax classifier to obtain the probability distribution of driver fatigue driving, and use the argmax function to obtain the fatigue classification result; S5-3: Determine whether to issue a fatigue driving warning to the driver based on the results of the fatigue driving test.

Images