A driver fatigue detection method based on a multi-feature fusion state recognition network

Abstract

The application discloses a driver fatigue detection method based on a multi-feature fusion state recognition network, and is applied to the field of fatigue driving detection. In view of the problems of low detection efficiency and false detection of the prior art, the application proposes a multi-feature fusion state recognition network combined with two subnetworks, selects eye and mouth and head posture features in an NTHU-DDD data set respectively, and detects fatigue from video images. First, an MTCNN face detector is trained to position facial features of a driver, then a multi-feature segmentation extraction algorithm is proposed to extract effective facial features while compressing parameters, and then a multi-feature fusion state recognition network based on an Inception module, a residual module and a channel attention mechanism is proposed to detect eyes, a mouth and a head posture to determine whether the driver is fatigued. The method of the application is superior to the prior art and has good stability and practicality.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and particularly relates to a driver fatigue detection technology. BACKGROUND

[0002] Fatigue driving is one of the main causes of traffic accidents, accounting for 30%. In recent years, millions of people have died or been injured due to accidents caused by fatigue driving. The NHTSA report shows that 697 people died in traffic accidents caused by fatigue in 2019. By 2020, the number of deaths has further increased, so accidents caused by fatigue driving have become one of the human death killers. The causes of fatigue are divided into active, passive and sleep. During the day, people are most likely to feel tired in the afternoon (about 1-3 pm) and early morning (about 2-4 am). If driving at this time, the probability of traffic accidents is much higher than at other times.

[0003] Therefore, how to efficiently detect the fatigue state of the driver and timely issue a warning to remind the driver to rest is a hot topic of research, and it is particularly important to design a fatigue driving detection system to reduce the incidence of traffic accidents. In the past 20 years, many scientific research institutions have made many attempts to prevent drivers from being tired. Many methods and evaluation indexes have been used to detect the fatigue state of the driver, and these methods are based on the percentage of closed eyes (PERCLOS), the continuous closing time of the eyes, yawning, head posture and other characteristic indexes. In recent years, these methods have attracted widespread attention from researchers. Features such as closing eyes, yawning, nodding, etc. all imply the fatigue of the driver.

[0004] Although these existing fatigue detection methods have achieved good results in non-complex environments, their detection effect will decrease when the driver's head posture changes continuously and the driver's facial features are not clear due to poor lighting conditions, etc. Moreover, the existing fatigue detection methods perform poorly in distinguishing similar facial expression features of the driver. These problems will all lead to false detection of the fatigue state. Moreover, in previous studies, many methods are based on the state of the eyes and mouth to judge fatigue, and fewer studies use the head posture feature to judge fatigue. In short, the first big challenge in technology is the influence of the change of the driver's head posture on the accurate extraction of facial features.

[0005] The second technical challenge is that the limited feature extraction cannot achieve an optimal effect in detecting fatigue, and judging only the state of the eyes and mouth is not comprehensive, and more features should be used for research. Recent research has identified the state of the eyes and mouth by constructing two InceptionV3 modules and an eye-mouth state recognition network (EM-CNN). Gulbadan Sikander et al. used three-dimensional reconstruction technology to extract eye, mouth, and forehead features. Although these methods have achieved good results in identifying the state of the eyes and mouth, they ignore the problem that when the driver's head posture changes, the eye and mouth features cannot be effectively extracted.

[0006] Fatigue driving has always been one of the important reasons for traffic accidents and has been a concern for people. In actual fatigue driving detection, the large amplitude swing of the driver's head posture greatly increases the difficulty of facial feature extraction. At the same time, previous research has shown that the effect of distinguishing between yawning and smiling, which are similar features, needs to be improved. Moreover, the accuracy of judging fatigue by a single feature is low in a complex environment. SUMMARY

[0007] To solve the above technical problems, the present application provides a driver fatigue detection method based on a multi-feature fusion state recognition network, which effectively improves the accuracy of fatigue driving detection.

[0008] The technical scheme adopted by the present application is as follows: a driver fatigue detection method based on a multi-feature fusion state recognition network, comprising:

[0009] S1, performing face feature point positioning detection on the obtained driver face image;

[0010] S2, performing segmentation and extraction operation according to the positioned face feature points, respectively obtaining the contour of the face, the contour of the nose, the contour of the left and right eyes of the person, and the contour of the mouth of the person;

[0011] S3, according to the contour of the face, the contour of the nose, the contour of the left and right eyes of the person, and the contour of the mouth of the person extracted in step S2, respectively evaluating the state of the eyes, the state of the mouth, and the head posture;

[0012] According to the state of the eyes and the state of the mouth, the face image processed in step S3 is marked to obtain a first input image;

[0013] According to the head posture, the face image processed in step S3 is marked to obtain a second input image;

[0014] S4, the multi-feature fusion state recognition network obtains a fatigue classification result according to the first input image and the second input image.

[0015] The eye state evaluation in step S3 is specifically:

[0016] Whether the driver is closing eyes is determined by calculating the eye aspect ratio, and the fatigue state of the driver is determined according to the time of continuous closing eyes.

[0017] When the average of the eye aspect ratios of the left eye and the right eye is less than 0.15, it is determined that the driver is closing eyes.

[0018] When the driver is determined to be closing eyes in three consecutive frames, it is considered that the driver is in a fatigue state.

[0019] The mouth state evaluation in step S3 is specifically:

[0020] The fatigue state of the driver is determined by calculating the mouth aspect ratio.

[0021] The head posture evaluation in step S3 is specifically:

[0022] The coordinate point of the nose tip is selected in the three-dimensional space, a rotation matrix is obtained, the Euler angle is calculated through the rotation matrix, and thus the head posture feature is obtained.

[0023] The multi-feature fusion state recognition network in step S4 specifically includes: a head posture recognition subnetwork, an eye and mouth state recognition subnetwork, a first splicing module, and a softmax module.

[0024] The head posture recognition subnetwork includes: a three-convolution pooling operation module, an Inception module, and a residual module; the second input image is processed by the three-convolution pooling operation module to obtain a 28x28x64 feature map, the 28x28x64 feature map is processed by the Inception module to obtain features of different scales, the features of different scales are spliced and then processed by the residual module to obtain a head posture recognition result.

[0025] The eye and mouth state recognition subnetwork includes: two convolution layers, a pooling layer, three channel attention modules, a second splicing module, and a convolution module; the first input image is processed by the two convolution layers and the pooling layer in sequence to obtain a 112x112x32 feature map, the 112x112x32 feature map is input into the three channel attention modules to obtain three weighted feature maps, the three weighted feature maps are spliced by the second splicing module, and then are convolved by the convolution module to obtain an eye and mouth state recognition result.

[0026] The head posture recognition result and the eye and mouth state recognition result are spliced by the first splicing module, and then a fatigue classification result is output by the softmax module.

[0027] The beneficial effects of the present application are as follows: firstly, the present application uses a multi-task cascaded detector MTCNN to locate a human face and key features. Secondly, the present application proposes a multi-feature segmentation and extraction algorithm (MFSE) to segment a face through the located feature points and extract an effective feature region. Then, the present application designs a new multi-feature fusion state recognition network (MFF-SRNet). The network is composed of an Inception module, a residual module and a channel attention mechanism network, and can realize the task of fatigue detection from multiple feature angles, jointly model different features, and reduce the influence of a complex driving environment on fatigue detection. The present application uses the time of eye closure, the degree of mouth opening and the degree of head deviation for fatigue detection. The embodiments of the present application select the NTHU-DDD dataset to evaluate the performance of the model. The experimental results show that the model proposed in the present application has an accuracy of 99.61% and an F1_score of 99.57%, which is better than existing methods. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present application;

[0029] Figure 2 is a detection effect diagram of the MTCNN used in the embodiments of the present application under different situations;

[0030] Figure 3 is a schematic diagram of a face key feature point;

[0031] Figure 4 is a diagram showing the implementation effect of the MFSE algorithm;

[0032] Figure 5 is a diagram showing the key points of the left eye of a person;

[0033] Figure 6 is a diagram showing the key points of the mouth of a person;

[0034] Figure 7 is a multi-feature fusion state recognition network;

[0035] Figure 8 is an eye and mouth state and head posture judgment;

[0036] Figure 9 is a head posture recognition subnetwork;

[0037] Figure 10 is an eye and mouth state recognition subnetwork. DETAILED DESCRIPTION

[0038] To facilitate those skilled in the art to understand the technical content of the present application, the content of the present application is further explained below in conjunction with the drawings.

[0039] In order to solve the influence caused by the change of head posture, the interference of similar expression features and the intensity of light, in the application, firstly, the MTCNN face detector is trained, which can detect the face well in the case of low light intensity, then the multi-feature segmentation and extraction algorithm (MFSE) is proposed to segment and extract the key feature parts, and then a new multi-feature fusion state recognition network is proposed. The network contains two subnets: 1) the first subnet is used to identify the head posture using the Inception module and residual connection, 2) the second subnet is used to identify the state of the local features such as eyes and mouth using the channel attention mechanism network. Then the two subnets are spliced and fused to improve the recognition accuracy of the model.

[0040] The application firstly uses the trained MTCNN detector to locate the key features of the face, and then uses the multi-feature segmentation and extraction algorithm (MFSE) to segment and extract the key feature regions, and extracts the very effective detection region while compressing the parameters, and then proposes a head posture recognition subnet composed of the Inception module and the residual module connection and an eye and mouth state recognition subnet composed of the channel attention mechanism to form a multi-feature fusion state recognition network. And three indexes EAR, MAR and HeadAngle are designed to detect the driver fatigue. Finally, the method proposed in the application is compared with other existing methods. The details of the experimental process framework are shown in the figure Figure 1 Figure 1 MTCNN represents multi-task cascaded detector, Facedetection represents face detection, Feature point location represents feature point positioning, MFENet represents multi-feature extraction algorithm, ROI extraction represents key region extraction, Head pose represents head posture, eyes represents eyes, mouth represents mouth, Head pose recognition Subnet represents head state recognition subnet, Eye and mouth state recognition Subnet represents eye and mouth state recognition subnet, MFFNet represents multi-feature fusion state recognition network, and Driver fatigue represents driver fatigue.

[0041] 1、Face feature point positioning detection

[0042] ​Due to environmental factors and the changeable driving posture of the driver, there are many difficulties in face detection in the actual driving environment. The most common face detector at present is MTCNN, dlib and Haar. Although the dlib detector is faster than MTCNN in detection speed, the accuracy is far inferior to MTCNN. Although the Haar detector can detect the faces of multiple people at the same time, it can only be used to detect the front face, and when the head changes slightly, the Haar cannot detect the face. Therefore, the MTCNN face detector is adopted in the application. The detector is currently the best detector in detection effect. It has good robustness in light and driver head posture change. By using a deep cascade multi-task framework, the face detection and alignment effect can be well completed to improve the performance. Thus, the key feature positions, including eyes, nose tip, mouth corner and whole face, are located. The Wider_face face dataset is used to train the MTCNN model structure. The model includes three subnets, namely P-Net, R-Net and O-Net.

[0043] P-Net: Before inputting into the P-Net layer, a pyramid is first constructed to obtain pictures of different scales, and then the pictures are input into the P-Net layer. After three convolution operations, a rough face window frame is obtained.

[0044] R-Net: After obtaining a rough face window area through the P-Net layer, the size of the output image is adjusted to 24x24, and then three convolution operations and full connection are performed to obtain a more accurate face area position.

[0045] O-Net: After the R-Net layer, the size of the output image is adjusted to 48x48, the face area position is finally confirmed, and the final position of the face and the positions of five key feature points are obtained. The five key feature points specifically refer to the left eye, the right eye, the nose, the left mouth corner and the right mouth corner.

[0046] The MTCNN face detector can complete the classification of different faces, face region framing and key feature positioning at the same time, has good robustness, and is very suitable for use in the actual driving environment. The detection effect in various situations is as shown in Figure 2

[0047] The face data in the embodiment adopts the public dataset NTHU-DDD. The website link is: NTHU CVlab-Driver Drowsiness Detection Dataset.

[0048] ​2. Multi-feature segmentation and extraction algorithm (MFSE)

[0049] After the important feature parts of the face are located, the next step is to segment and extract the located facial features to facilitate subsequent fatigue detection. The present application proposes a multi-feature segmentation and extraction algorithm to segment and extract the entire face, left and right eyes, and mouth. This has the advantage of reducing parameters while extracting effective features. The algorithm first finds the feature area range based on a 68-feature point detector, and then performs width and height cropping of the image. As shown in FIG. 2, feature points 1-27 are the contours of the face, feature points 28-36 are the contours of the nose, feature points 37-42 and 43-48 are the contours of the left and right eyes, respectively, and feature points 49-68 are the contours of the mouth. In order to avoid excessively small feature sizes, the present application performs a little expansion processing. The extraction effect of the method proposed by the present application is shown in FIG. 3. Figure 3 Figure 4

[0050] 3. Fatigue detection and judgment index

[0051] (1) Eye state evaluation index

[0052] In terms of eye fatigue state evaluation, the number of blinks per unit time, the percentage of eye closure (PERCLOS), and the continuous eye closure time can be used to judge, among which PERCLOS and continuous eye closure time are more commonly used. The present application selects the continuous eye closure time obtained by calculating the eye aspect ratio to judge the fatigue state of the driver. According to the contours of the left and right eyes extracted by the face detector, the present application extracts and enlarges the left eye, and numbers the left eye features to correspond to [P1, P6], as shown in FIG. 4. Figure 5

[0053] Then the Euclidean distance between the two groups of vertical eye coordinates and the horizontal eye coordinates is calculated, and the eye aspect ratio is calculated, as shown in formula (1).

[0054]

[0055] In a normal state, the EAR of the driver remains relatively stable. In a fatigue state, the EAR of the driver tends to be zero. The present application synchronizes the left and right eyes, averages the two EARs of the left and right eyes, and obtains the result. The calculation formula is shown in formula (2).

[0056]

[0057] ​​​In order to avoid the influence of the small eyes of some people as much as possible, when ear < 0.15, it can be determined that the driver closes his eyes. When the ear of the driver in the video image is less than 0.15 for three consecutive frames, it can be determined that the driver is in a fatigue state.

[0058] (2) Mouth state evaluation index

[0059] The evaluation method of the mouth state is similar to the evaluation method of the eyes. The contour of the mouth of the person extracted by the face detector is numbered as [M1, M6] by the application, which is used to calculate the key points of the mouth aspect ratio, as shown in the formula (2). Figure 6

[0060] Then the Euclidean distance between the two groups of vertical mouth coordinates and the horizontal mouth coordinates is calculated respectively, and the mouth aspect ratio is calculated, and the calculation formula is shown in formula (3).

[0061]

[0062] According to research, the value of MAR of normal people when speaking and smiling generally does not reach 0.8, so the application takes MAR >= 0.8 as one of the fatigue evaluation standards of the driver.

[0063] (3) Head posture evaluation index

[0064] In the 3D space, the rotation of an object can be represented by three Euler angles (Euler Angle): pitch (rotation around the X axis), yaw (rotation around the Y axis) and roll (rotation around the Z axis), that is, the pitch angle, the yaw angle and the roll angle, which are commonly known as lifting the head, shaking the head and turning the head.

[0065] The evaluation method of the head posture is different from the previous two. First, the model coordinate points (including the nose tip, the cheek, the left and right eye corners, and the mouth corner) are selected in the three-dimensional space, the rotation matrix is obtained by following the three-dimensional coordinates of each feature point, and the Euler angle is calculated through the rotation matrix. The calculation formula is as follows:

[0066] x = a tan2(R[2, 1], R[2, 2]) (4)

[0067] y = a tan2(-R[2, 0], sy) (5)

[0068] z = a tan2(R[1, 0], R[0, 0]) (6) Wherein, x represents the x axis of the Euler angle, y represents the y axis of the Euler angle, and z represents the z axis of the Euler angle, ​R represents the obtained rotation matrix, rows and columns in R are arranged from 0, R[0, 0] represents an element in the 0th row and the 0th column of R, and correspondingly, R[1, 0] represents an element in the 1st row and the 0th column of R, R[2, 0] represents an element in the 2nd row and the 0th column of R, R[2, 1] represents an element in the 2nd row and the 1st column of R, and R[2, 2] represents an element in the 2nd row and the 2nd column of R. Then a line is extended from the nose tip perpendicular to the face image plane, the radian of the rotation matrix is converted into an angle value to calculate the head tilt angle in degrees, so as to obtain the head pose feature.

[0069] 4. Multi-feature fusion state recognition network

[0070] After the data is processed through the above process, it is necessary to fuse and detect. The fusion detection can well reduce the influence of the environment and the posture of the driver on the fatigue detection. The multi-feature fusion state recognition network structure proposed in the application is as shown in the figure. Figure 7 Figure 7 In the figure, Input1 represents input 1, Input2 represents input 2, Head pose recognition Subnet represents a head state recognition subnet, Eye and mouth state recognition Subnet represents an eye and mouth state recognition subnet, Based on Residual Block represents a residual block, Based on Channel attention represents a channel attention, MFFNet represents a multi-feature fusion state recognition network, Concentrate represents connection, Eye and mouth fatigue represents an eye and mouth fatigue state, Eye and mouth normal represents an eye and mouth normal state, fatigue represents fatigue, normal represents normal, Head fatigue represents head fatigue, Head normal represents head normal, and Classifier represents a classifier.

[0071] In a real driving environment, the sizes of the faces, eyes and mouths of the drivers obtained are different. Therefore, the size of the input picture is set to 224x224, and two subnets are created to input the picture into the network. The input picture is as shown in the figure. Figure 8

[0072] The first subnet is an improvement on the EM-CNN network. In the existing EM-CNN network, the eye and mouth picture is input into the network, and the last four classifications output the open and closed eye and mouth states. The application makes some improvements based on this eye and mouth state recognition network and proposes a head pose recognition subnet. The specific structure of the branch is as shown in the figure. Figure 9 ​​The face image of the labeled driver's head posture feature is input and a 28x28x64 feature map is obtained after three convolutional pooling operations. The size of the convolution kernel is set to 3x3 and the step is set to 1. The size of the convolution kernel of the pooling layer is set to 2x2. Then, an Inception module is used, which obtains different scales of receptive fields by copying the image through different convolution and pooling operations, and then splices and fuses different scale features. The module can automatically determine the required filters, automatically determine the network parameters, and connect the outputs. Then, a residual module is used, which has three convolutional layers, a pooling layer, and a convolution kernel size of 3x3, and then outputs a 14x14x32 feature map. Then, the feature map is flattened and prevented from overfitting or underfitting. The detailed structure of the subnet is as shown in Figure 9 . Figure 9 Inception block represents an Inception module, Residual connection represents a residual connection, and 1*1Conv represents a 1*1 convolution block.

[0073] The second subnet is based on a channel attention mechanism to identify eye and mouth states. The application proposes an eye and mouth state identification subnet. A LeakyReLu function is used as an activation function to complete the nonlinear transformation between each layer of the network. The operation of each layer of the channel attention is to multiply the output of the convolution layer and the attention value to obtain the feature weight. The calculation formula is shown in formula (7).

[0074] F i =f i ×A i (7)

[0075] where f i represents the output after the convolution layer, and A i represents the channel attention value.

[0076] After inputting the face image of the labeled driver's eye and mouth state, a 112x112x32 feature map is obtained after two convolution and one pooling operations, and 1x1, 3x3 convolution layers and 2x2 pooling layers are used to facilitate the network of the application to adapt to the size, and then the weighted eye and mouth feature maps are convolved. The detailed structure of the subnet is as shown in Figure 10 . Figure 10Input, Convolution, BN, MP, LeakyReLU, ReLU, Avg_pool, Max_pool, Downsampling, Upsampling, Concat, Channel attention block.

[0077] Finally, the two subnets are spliced and fused, and the fatigue classification result is output through softmax.

[0078] The technical effects of the method of the present application are verified by the following experiments:

[0079] The comparison of the experimental results of the present application on the NTHU-DDD dataset and the experimental results of other most advanced methods on the dataset is shown in Table 1. The detection effect of the MFF-SRNet model of the present application and the latest model in the past two years on the NTHU-DDD dataset is compared. Table 1 shows the comparison of the model proposed in the present application and the better research CNN-TSCLSTM, CNN-based Deep Learning model, Multi-Granularity Deep Convolutional Model, 3DcGAN+TLABiLSTM+Refinement, 2s-STGCN and MTCNN with Ensemble of 2InceptionV3 models in the past two years. From Table 1, it can be seen that the MFF-SRNet model proposed in the present application is better than the existing method in terms of accuracy and F1_score.

[0080] The comparison of the experimental results of the present application and the results of other feature extraction detection is shown in Table 2. The number of features selected by the MFF-SRNet proposed in the present application and the existing research 3D deep learning network, Adrowsiness detection system based on driver's real time video capture, DDDN, Deep model, Personalised full model are compared. From Table 2, it can be seen that when the number of selected features is more, the detection effect is also better.

[0081] Table 1 Detection effect of different methods on NTHU-DDD dataset

[0082]

[0083] Table 2 compares the detection effects of extracting different features and methods

[0084]

[0085] Those skilled in the art will appreciate that the embodiments described herein are presented for purposes of illustration and understanding of the principles of the application and should not be construed as limiting the scope of the application to such specifically enumerated embodiments. Various modifications and changes can be made thereto by those skilled in the art without departing from the spirit and principles of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application should be included in the scope of the claims of the application.

Claims

1.A driver fatigue detection method based on a multi-feature fusion state recognition network, characterized in that, The method comprises the following steps: S1, facial feature point positioning detection is performed on the obtained driver face image; S2, segmentation and extraction are performed according to the positioned facial feature points, and the contour of the face, the contour of the nose, the contour of the left and right eyes of the person, and the contour of the mouth of the person are obtained respectively; S3, the contour of the face, the contour of the nose, the contour of the left and right eyes of the person, and the contour of the mouth of the person extracted in step S2 are used to respectively evaluate the eye state, the mouth state, and the head posture; The face image processed in step S3 is marked according to the eye state and the mouth state, and a first input image is obtained; The face image processed in step S3 is marked according to the head posture, and a second input image is obtained; S4, a multi-feature fusion state recognition network is used to obtain a fatigue classification result according to the first input image and the second input image; The multi-feature fusion state recognition network specifically comprises a head posture recognition subnetwork, an eye and mouth state recognition subnetwork, a first splicing module, and a softmax module; The head pose recognition subnetwork includes: a triple convolutional pooling module, an Inception module, and a residual module; the second input image is obtained after passing through the triple convolutional pooling module. Feature map, The feature map is processed by the Inception module to obtain features of different scales. The features of different scales are then concatenated and processed by the residual module to obtain the head pose recognition result. The eye and mouth state recognition subnetwork includes: two convolutional layers, one pooling layer, three-channel attention modules, a second stitching module, and a convolutional module; the first input image is processed sequentially through two convolutional layers and one pooling layer to obtain... Feature map, The feature map is input into the three-channel attention module to obtain three weighted feature maps. The three weighted feature maps are then concatenated by the second concatenation module and then convolved by the convolution module to obtain the eye and mouth state recognition result. The head posture recognition result and the eye and mouth state recognition result are spliced by the first splicing module, and the fatigue classification result is output by the softmax module. 2.The driver fatigue detection method based on the multi-feature fusion state recognition network according to claim 1, characterized in that, The eye state evaluation in step S3 is specifically: The time of whether closing eyes is obtained by calculating the eye aspect ratio, and the fatigue state of the driver is judged according to the continuous closing eye time. 3.The driver fatigue detection method based on the multi-feature fusion state recognition network of claim 2, characterized in that, When the average of the left eye and the right eye aspect ratio is less than 0.15, it is determined that the driver closes eyes. 4.The driver fatigue detection method based on the multi-feature fusion state recognition network according to claim 3, characterized in that, When the driver is determined to close eyes for 3 consecutive frames, it is considered that the driver is in a fatigue state. 5.The driver fatigue detection method based on the multi-feature fusion state recognition network according to claim 4, characterized in that, The mouth state evaluation in step S3 is specifically: The fatigue state of the driver is judged by calculating the mouth aspect ratio. 6.The driver fatigue detection method based on the multi-feature fusion state recognition network according to claim 5, characterized in that, The head posture evaluation in step S3 is specifically: The coordinate point of the nose tip is selected in the three-dimensional space to obtain a rotation matrix, and the Euler angle is calculated through the rotation matrix to obtain the head posture feature.

Images