A multi-angle driver fatigue degree evaluation method

Abstract

The present application relates to the technical field of vehicle auxiliary driving system, in particular to a multi-angle driver fatigue degree evaluation method, comprising a training step and a test step; the training step comprises collecting multi-angle driver fatigue face images, and marking and classifying the collected images to form a multi-angle driver fatigue dataset; a network structure of a fatigue detection model is established; the fatigue detection model is trained using transfer learning according to the multi-angle driver fatigue dataset; the test step comprises the following specific steps: a single camera is used to acquire a driver's face image; the acquired face image is input into the driver fatigue detection model to output an image fatigue classification result; the number of fatigue frames is accumulated and calculated to evaluate the driver fatigue degree. The method of the present application can realize multi-angle, accurate and rapid detection of the driver fatigue degree using a single RGB camera.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle auxiliary driving system, and particularly relates to a multi-angle driver fatigue degree evaluation method. BACKGROUND

[0002] Fatigue can significantly reduce the attention and reaction ability of a driver, thereby increasing the probability of accidents. According to a research report of the Traffic Safety Foundation of the American Automobile Association, about 328,000 driving accidents are related to driver fatigue every year. In addition, fatigue detection is particularly important in human-machine co-driving, because fatigue can significantly reduce the performance of a driver in taking over, and a driver in human-machine co-driving is more likely to be fatigued than a driver in traditional pure human driving. Therefore, accurately detecting the fatigue state of a driver is crucial to ensuring traffic safety.

[0003] At present, automatic detection of driver fatigue mainly relies on three kinds of information, namely machine vision information, physiological signals and vehicle state information. The vehicle state information includes data such as steering angle, lane deviation and acceleration. Since the vehicle state information is indirectly related to the fatigue state of a driver, the robustness and accuracy of the method relying on such information are low. Physiological signals include electrocardiogram, electroencephalogram, electromyogram and heart rate variability data. Physiological signals can effectively reflect the fatigue state of a driver, but the measuring device is sensitive to signal noise, and the medical-grade detection equipment is expensive. Machine vision signals are mainly facial image information of a driver. With the development of machine vision, fatigue detection methods based on machine vision signals have become the main trend in recent years, which have the advantages of low cost, high precision and robustness.

[0004] Driver fatigue detection methods based on machine vision can be further divided into two processes. One is to realize the detection of fatigue based on two-step detection. The first step is to extract manually designed fatigue-related features such as eyelid closure time, blink frequency and yawning from a video. Then, the fatigue state of a driver is inferred according to these features. This method requires less fatigue data, but the operation is complex and the robustness is low. The other is an end-to-end data-driven method, in which a model automatically extracts features from a video to judge the fatigue of a driver. This kind of method is simple and has high accuracy, but the end-to-end method based on deep learning needs a large amount of training data to avoid overfitting.

[0005] Most of the current machine vision-based fatigue detection methods only target the front face image of the driver, without considering the influence of the relative pose change between the driver and the camera on fatigue detection. However, in the real world, the relative pose between the camera and the driver's face is always changing, and the pose change is usually large. In addition, on an autonomous vehicle, the driver can perform non-driving related activities such as playing a mobile phone, and the relative pose between the driver and the camera has high uncertainty. The relative pose change between the driver and the camera will cause inconsistency in obtaining face information, making it difficult to achieve accurate and robust driver fatigue detection. SUMMARY

[0006] The purpose of the present application is to provide a method for detecting driver fatigue at multiple angles under a single camera, which is based on deep learning, innovatively uses a multi-angle image training set to train a deep learning driver fatigue detection model, and uses transfer learning to solve the problem of insufficient training samples when detecting driver fatigue using deep learning, uses the knowledge in the fields of rich data face detection and face alignment to improve the performance of the driver fatigue detection model, so that the trained model can detect driver fatigue at multiple angles.

[0007] The technical scheme of the present application is a multi-angle driver fatigue degree evaluation method, which includes a training step and a testing step.

[0008] The training step includes the following specific steps:

[0009] Collect multi-angle driver fatigue face images, and label and classify the collected images to form a multi-angle driver fatigue dataset;

[0010] Establish the network structure of the fatigue detection model;

[0011] According to the multi-angle driver fatigue dataset, use transfer learning to train the fatigue detection model;

[0012] The testing step includes the following specific steps:

[0013] Use a single camera to obtain the face image of the driver;

[0014] Input the obtained face image into the driver fatigue detection model to output the image fatigue classification result;

[0015] Cumulatively calculate the number of fatigue frames to evaluate the degree of driver fatigue.

[0016] Compared with the prior art, the present application has the following beneficial technical effects:

[0017] (1) The application uses a multi-angle driver fatigue facial image training model, uses deep learning to adaptively learn fatigue features in the multi-angle driver fatigue facial image, greatly improves the robustness of the model to changes in the driver's facial position, and thus realizes multi-angle driver fatigue detection.

[0018] (2) The application uses transfer learning to solve the problem of insufficient fatigue data. Transfer learning does not require complex large-scale fatigue data collection, and uses existing data in other fields to improve the performance of the fatigue detection model. This method is simple and effective.

[0019] (3) The application assesses the degree of fatigue of the driver according to the length of time taken to detect a certain number of fatigue frames. The image classification result is not directly used as a basis for judging the degree of fatigue of the driver, and the fatigue classification of the driver can be flexibly adjusted according to the application needs without retraining the model. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 a schematic view of the relative position of the camera and the driver during the collection of driver fatigue data;

[0021] Figure 2 a schematic view of the fatigue detection framework;

[0022] Figure 3 a structural schematic view of the fatigue detection network. DETAILED DESCRIPTION

[0023] The multi-angle driver fatigue detection method described in the application comprises a training step and a testing step. In the training step, multi-angle driver fatigue facial images are collected, and the collected images are labeled and classified to generate a multi-angle driver fatigue detection training set. Based on the generated training set, a designed fatigue detection model is trained using the transfer learning method. The specific steps are as follows:

[0024] 1. Collect multi-angle driver fatigue data. Multi-angle driver fatigue data can be collected through a driver-in-the-loop driving simulator. The fatigue state of the driver can be caused by sleep deprivation, i.e. the driver can only sleep for 5 hours on the day before data collection, and the experiment should be conducted during 2 pm to 4 pm, because humans are more likely to be fatigued during this time period. In order to ensure the fatigue state of the driver, the driving scene is designed as a monotonous continuous curve driving condition. In addition, there are no obstacles in the traffic flow.

[0025] 1.1 The pose change of the driver and the camera during the data collection process is realized by simultaneously collecting the facial images of the driver through multiple cameras at different positions, and the head pose of the driver does not change greatly during the data collection process. The relative position of the camera and the driver during data collection is as follows Figure 1As shown, a total of 5 cameras are used, one in the middle and two on the left and right, with an included angle of 35 degrees between the cameras.

[0026] 1.2, generating a multi-angle fatigue detection training set. First, the front face image collected by the middle camera is labeled. The Dlib algorithm is used for face detection and face alignment of the front face image. Based on the face alignment result, the eye closure degree S of each front face image is calculated, and the average value of S is calculated. The front face image with S value less than 0.5 times the average S value is marked as a candidate front face fatigue sample, and the front face image with S value greater than the average S value is marked as a candidate front face non-fatigue sample. Artificial inspection removes abnormal samples from the candidate front face fatigue sample and the candidate front face non-fatigue sample to generate a front face fatigue training set.

[0027] Then, according to the front face fatigue training set, the corresponding images obtained by other cameras are labeled. Random sampling is performed on the labeled data of all cameras to generate a multi-angle fatigue detection training set. In order to maintain the class balance of the training set, the multi-angle fatigue detection training set generated by random sampling needs to meet the following requirements: (1) the data quantity from each camera is consistent; (2) the number of fatigue data and non-fatigue data is consistent.

[0028] 2, establishing a fatigue detection framework and a fatigue detection network. The fatigue detection framework is as shown in Figure 2 , which includes a face detection network and a fatigue detection network.

[0029] Fatigue detection mainly uses face information, and the input image data always contains a large amount of irrelevant information. Therefore, the face should be extracted from each input image. The MTCNN network combines a cascade structure and a multi-task learning method, which is one of the most effective face detection methods at present, so in this embodiment, the MTCNN network is selected as the face detection network.

[0030] The face image output by the face detection network is first uniformly scaled to 48x48 pixels, and then the image is normalized. The normalized image is input into the fatigue detection network.

[0031] The MTCNN network is trained by a face detection and face alignment dataset, and the cascade structure of the MTCNN network is composed of P-Net, R-Net and O-Net. The O-Net model is the last level of the three-level cascade structure of the MTCNN network, and is the deepest and most accurate level. Therefore, in this embodiment, the fatigue detection network is obtained by transfer learning of the O-Net model of the MTCNN network. The structure of the fatigue detection network is as shown in Figure 3As shown, the fatigue detection network replaces the original fully connected layer of the O-Net model with a new fully connected layer, and replicates other layers of the O-Net model. The new fully connected layer includes two layers, of which the first layer has 256 neurons and the second layer has 2 neurons. The activation function of the fatigue detection network uses the ReLU function, but the last layer of the network uses the Softmax function to obtain the classification result of the driver fatigue state.

[0032] The fatigue detection network can output the classification result of the image, i.e., fatigue and non-fatigue.

[0033] 3. Training the fatigue detection model by transfer learning. During the training process, the data input into the fatigue detection framework is the multi-angle fatigue detection training set. The weights of the fully connected layer of the fatigue detection network are initialized using the Glorot uniform distribution initializer, and the other layers share the same weights as the corresponding layers of the O-Net model in the MTCNN, and only the weights of the fully connected layer of the fatigue detection network are updated during the training process. During the training process, the stochastic gradient descent (SGD) method is selected as the optimizer, with the learning rate set to 0.005 and the momentum set to 0.9. The batch size is set to 30. The loss function used during the training process is the cross-entropy loss function.

[0034] In the testing step, the camera captures the driver image, and uses the trained driver fatigue detection model to classify the image, and judges the driver fatigue state according to the classification result. The specific steps are as follows:

[0035] 4. Use a single camera and install the camera in a suitable position;

[0036] 5. Input the image obtained by the camera into the fatigue detection framework composed of the trained driver fatigue detection model to output the image classification result;

[0037] 5. Cumulative calculation of the number of fatigue frames; according to the length of time used to detect a certain number of fatigue frames, the fatigue degree of the driver is evaluated. In this embodiment, the fatigue degree of the driver is set to two levels, and when the time for detecting 10 fatigue frames is less than 5 seconds, the driver is in a fatigue state, otherwise the driver is in a non-fatigue state.

[0038] The above is only an example of the concept of the present application. The present application can also be optimized and improved for specific scenarios, devices, data acquisition methods, and network topologies. Any non-substantial modification of the present application shall be deemed to infringe the protection scope of the present application.

Claims

1. A multi-angle driver fatigue level evaluation method, characterized in that, The method comprises a training step and a testing step; The training step comprises the following specific steps: S1, collecting multi-angle driver fatigue facial images, and marking and classifying the collected images to form a multi-angle driver fatigue dataset; S2, establishing a network structure of the fatigue detection model; the network structure of the fatigue detection model in S2 comprises a face detection network and a fatigue detection network; The face detection network selects an MTCNN network combining a cascade structure and a multi-task learning method; The face image output by the face detection network is first uniformly scaled to 48x48 pixels, and then normalized; The normalized image is input into the fatigue detection network; The cascade structure of the MTCNN network comprises a P-Net, an R-Net and an O-Net; the O-Net model is the last stage of the three-stage cascade structure of the MTCNN network; the fatigue detection network is obtained by transfer learning of the O-Net model of the MTCNN network; specifically, the fatigue detection network replaces the original fully connected layer of the O-Net model with a new fully connected layer, and copies other layers of the O-Net model; the fatigue detection network outputs two results of fatigue and non-fatigue; S3, training the fatigue detection model using transfer learning according to the multi-angle driver fatigue dataset; in S3, the transfer learning trains the fatigue detection model, and in the training process, the data input into the fatigue detection framework is the multi-angle fatigue detection training set; The weights of the fully connected layers of the fatigue detection network are initialized using a Glorot uniform distribution initializer, and the other layers share the same weights as the corresponding layers of the O-Net model in the MTCNN; only the weights of the fully connected layers of the fatigue detection network are updated during the training process; The testing step comprises the following specific steps: S4, acquiring a driver's facial image using a single camera; S5, inputting the acquired facial image into the driver fatigue detection model to output an image fatigue classification result; S6, cumulatively calculating the number of fatigue frames to evaluate the fatigue degree of the driver.

2. The multi-angle driver fatigue level evaluation method according to claim 1, characterized in that, In S1, a plurality of cameras are arranged on a ring-shaped driving simulator, and multi-angle driver fatigue data is collected by the plurality of cameras at different angles.

3. The multi-angle driver fatigue level assessment method of claim 2, wherein Five groups of cameras are arranged, one group in the middle of the ring-shaped driving simulator and two groups on the left and right sides; the included angle between the cameras is 35°.

4. The multi-angle driver fatigue level assessment method of claim 1, wherein The driver's fatigue state is caused by sleep deprivation; before data collection, the driver is only allowed to sleep for 5 hours the night before, and the experiment is conducted from 2 pm to 4 pm.

5. The multi-angle driver fatigue level assessment method of claim 2, wherein In S1, the multi-angle fatigue detection training set is generated; first, the front face images collected by the middle cameras are marked and classified, and then the side face images are marked according to the marked front face images.

6. The multi-angle driver fatigue level assessment method of claim 1, wherein In the testing process, the number of fatigue frames is cumulatively calculated; the fatigue degree of the driver is evaluated according to the length of time taken to detect a certain number of fatigue frames.

Images