Fatigue driving detection method based on train cab scene

Abstract

The present invention belongs to the field of face detection technology, and specifically relates to a fatigue driving detection method based on a train cab scene. The deep learning method is used to first perform face detection, and 68 key points are detected for face key points. The 68 points are used to calculate the distance between the eyes in the eye area, the distance between the lips, and the head turning state. The first step of fatigue state judgment is performed by converting the 68 points into Euler angles to calculate three states. After that, the visual features of the eye area are extracted to establish a visual model for eye opening and closing judgment to improve the recognition ability of visual features, and then the yawning judgment is performed in combination with the opening and closing features of the lip area, and the driver is further judged whether he is tired based on the frequency of eye opening and closing and yawning, so as to reduce the misjudgment of a single feature and reduce the false alarm signal caused by misjudgment. Therefore, this scheme adopts a scheme combining two optimization strategies in terms of detection accuracy, which can accurately detect the fatigue state of the face.

Description

Technical Field

[0001] The present invention belongs to the technical field of face detection, and in particular relates to a fatigue driving detection method based on a train cab scenario. Background Art

[0002] With the rapid economic development and large-scale construction of urban roads, the mileage of railways, high-speed railways and other roads has increased year by year, and the incidence of traffic accidents has also increased. At the same time, the pace of life of modern people is accelerating, the phenomenon of staying up late is serious, lack of exercise, etc., which makes drivers easily fatigued, so fatigue driving has become the main cause of traffic accidents.

[0003] How to effectively prevent fatigue driving? Domestic and foreign scholars have used different technologies to study the existing fatigue driving detection methods. According to fatigue characteristics, they can be divided into two categories: methods based on physiological characteristics and methods based on behavioral characteristics. Physiological characteristics include physiological signal characteristics and physiological response characteristics. The physiological signal-based method proposes a sleep detection model based on EEG signals. The results show that EEG signals can correctly distinguish between the awake state and the sleep state, but EEG signals are easily disturbed by noise and difficult to collect. There are also some methods based on eye contact signals, which are easier to collect than EEG signals and are immune to slight noise, but they still need to install head equipment for collection; the behavior-based method uses a remote video camera to perform real-time image acquisition to monitor eyelid movement and facial The method based on external sensors embeds distributed pressure sensors into the steering wheel to measure the pressure and position of the hand holding the steering wheel, and statistically analyzes the measurement data of each component for fatigue warning. This sensor has a simple structure and low cost, and has achieved preliminary results in practical applications. The disadvantage is that it will be interfered by the driver's personal habits. The lane line detection method uses a camera to capture the road conditions ahead, and refers to the white lines on the road to determine whether the vehicle is driving in a standardized manner and thus determine whether the vehicle is out of control. This state-based method does not require the driver to directly contact the detection device and has low equipment requirements based on the existing devices of the car. It is very practical and easy to promote, but it will be limited by road conditions and vehicle models. The existing technology has problems in safety, real-time, and interference, such as low real-time performance, easy interference to the driver, and high detection error rate.

[0004] With the success of deep learning in target detection and other fields in recent years, deep learning technology has also promoted the research of fatigue driving detection. For example, CN201810368035.9 proposed a method based on fatigue detection regression model, which uses convolutional neural network for unsupervised learning feature expression, replacing the process of manually designed feature extraction, and uses LDS (Linear Dynamic System) method in post-processing to reduce adverse interference; for example, CN2017215451667 uses convolutional neural network to extract the discriminant information of hand position to unsupervisedly learn and predict four safe and unsafe driving postures; for example, CN108309331A proposed an eye state recognition method based on convolutional neural network to calculate blinking frequency, but the fatigue parameter is too single. Studies have shown that using deep network to directly learn visual feature representation from images has better robustness to changes in conditions such as lighting and posture than manually designed features, and the prediction accuracy is significantly improved.

[0005] The above fatigue driving detection methods have achieved certain results and progress, but there are still great limitations. Although the detection method based on physiological signal characteristics has high accuracy, it requires the driver to have direct physical contact with the detection equipment to collect signals, which may interfere with the driver's driving operation and the equipment cost is too high. It is more suitable for laboratory environment and not suitable for practical application; the detection method based on behavioral characteristics does not require the driver to directly contact the detection device and has low equipment requirements based on the existing equipment of the car. It is very practical and easy to promote, but it will be limited by the driver's personal habits, road conditions and vehicle models. In rainy and snowy weather or unsatisfactory road conditions, the detection accuracy is low; while ensuring a certain accuracy and good real-time performance, the method based on computer vision processing physiological response characteristics is easier to promote, but the artificially designed feature resolution ability is poor and relying on a single visual feature may encounter difficulties. For example, when the driver wears glasses, the reflection of the glasses may cause the camera to be unable to capture eye movements. In addition, the degree of eye opening and closing may vary from person to person, and irregular head movements may produce false alarms, etc. Summary of the invention

[0006] In view of the above-mentioned shortcomings, the present invention provides a method for detecting fatigue driving in a train cab scenario.

[0007] The present invention is achieved through the following technical solutions:

[0008] A method for detecting fatigue driving in a train cab scenario comprises the following steps:

[0009] S0: Obtaining a face image, using an image acquisition device to collect a face image in a train cab scene;

[0010] S1: face detection;

[0011] S2: facial key point detection, using a convolutional neural network to obtain the key points of the facial image;

[0012] S3: judging the face status of the current frame;

[0013] S4: Based on the judgment result obtained in step S3, performing multi-frame face status statistics;

[0014] S5: Determine whether the set frame number is reached. If so, output the mouth, eyes and head status. If not, return to step S1.

[0015] Furthermore, after step S2, step S2.1 is included: face alignment, using the positions of five key points in the facial features area, calculating the similarity transformation matrix between two points, and using the similarity transformation matrix to perform face alignment to obtain the aligned face.

[0016] Furthermore, the step S3 further includes:

[0017] S3.1: Eye opening and closing detection, using the face area aligned in step S2.1, cropping out the left eye area and the right eye area; using the first classification model to classify the eye area of ​​the current face image into open and closed eyes, and outputting the open or closed state of the left and right eyes;

[0018] S3.2: Mouth opening and closing detection: using the face area aligned in step S2.1, cropping out the mouth area of ​​the face; using the second classification model to judge whether the mouth area of ​​the face is open or closed, and outputting the open or closed state;

[0019] S3.3: head deflection angle analysis, using step S2 to extract 68 facial key points of the current image, select 14 points to calculate the rotation Euler angle of the current face, compare the rotation Euler angle with the set head deflection angle, and determine whether the current face has inattention phenomenon;

[0020] S3.4: Eye opening and closing analysis, using the 68 key points of the current face image extracted in step S2, within the head deflection angle range set in step S3.3, select the center point of the left eye area and the center point of the right eye area, and calculate the ratio of the Euclidean distance between the upper and lower eyelids of the left eye and the center point of the left eye, and the ratio of the Euclidean distance between the upper and lower eyelids of the right eye and the center point of the right eye; compare the above ratios with the set eye opening and closing thresholds to determine the open or closed state of the left and right eyes;

[0021] S3.5: Mouth opening and closing analysis, using the 68 key points of the current face image extracted in step S2, select the center point of the mouth area within the head deflection angle range set in step S3.3, calculate the Euclidean distance ratio between the upper lip and lower lip in the mouth area and the mouth center point, and compare it with the mouth opening and closing set threshold to determine the open or closed state of the mouth.

[0022] Furthermore, in step S1, libfacedetection is used as the face detection model.

[0023] Furthermore, in step S2, mobilenet_v2 is used to extract facial key points.

[0024] Further, in step S3.4, the above ratio is compared with the set eye opening and closing thresholds to determine the open or closed state of the left and right eyes. Specifically, if the ratio is less than the set threshold, it is in the closed state, and if it is greater than the threshold, it is in the open state.

[0025] And combined with the result of the eye-opening and eye-closing detection in step S3.1, the values ​​of the two are combined to perform an AND operation. Only when both are in the eye-closing state at the same time will the final result be the eye-closing state.

[0026] Further, in step S3.5, the ratio is compared with the mouth opening and closing setting threshold to determine whether the mouth is open or closed, specifically: if the ratio is greater than the threshold, it is in the open mouth state, and if it is less than the threshold, it is in the closed mouth state;

[0027] And combined with the result of the mouth opening and closing detection in step S3.2, the values ​​of the two are combined to perform an AND operation. Only when both are in the open mouth state will the mouth be judged to be in the open mouth state.

[0028] Furthermore, after step S2.1, the method further includes:

[0029] S2.2: extracting facial features, according to the images aligned in step S2.1, extracting facial features based on the facial feature extraction model, and obtaining a facial feature extractor;

[0030] S2.3: Face matching, to determine whether the current train driver is the driver himself, and to record the first login time, which is used as the driver's attendance statistics.

[0031] A computer-readable storage medium stores a computer program, wherein when the program is executed by a processor, the steps of a fatigue driving detection method based on a train cab scenario are implemented.

[0032] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a fatigue driving detection method based on a train cab scenario when executing the program.

[0033] The present invention proposes a method based on deep learning to detect single-frame and multi-frame driver facial state fatigue driving. In order to improve the accuracy and real-time performance of the overall detection, different optimization strategies are collected in model training compression and method implementation, which has a significant performance improvement in fatigue driving state judgment. Compared with the prior art, the present invention has at least the following beneficial effects or advantages:

[0034] 1. The present invention comprehensively judges whether a person is in a fatigue state based on the deflection angle of the head, the closure of the eyes and the degree of the mouth opening; in addition, it judges whether there is fatigue driving through statistics of multiple frames of data, thereby improving the accuracy of fatigue driving judgment.

[0035] 2. Mobilenet_v2 is used as the backbone network for key point detection, and the weights of facial features are increased to improve the accuracy of key point detection in the eye area. At the same time, in order to ensure the real-time detection, the key point detection model is cropped and quantized to different degrees to further improve the detection speed.

[0036] 3. In order to improve the accuracy of eye opening and closing detection in fatigue state, the eye area is cropped according to the face key point detection model, and a new eye opening and closing classification model is added to the eye area. The eye key point distance calculation and classification results of the classification model are combined to judge the eye key point closing.

[0037] 4. In order to improve the accuracy of yawn detection in the mouth area, the mouth area is cropped according to the facial key point detection model, and a new mouth area open and closed classification model is added. The mouth area distance calculation and classification model are combined to further improve the accuracy of yawn detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention will be further described in detail below with reference to the accompanying drawings;

[0039] Figure 1 It is a flowchart for detecting facial fatigue status based on multiple frames of video. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] This embodiment discloses a method for detecting fatigue driving based on deep learning. The method first detects and judges the state of a single-frame face image, and then uses the video information collected by the camera to perform multi-frame statistical judgment. It mainly includes the following steps:

[0042] S1: Obtain a face image, and collect a face image in a train cab scene through an image acquisition device;.

[0043] This embodiment can obtain pictures through a camera installed in the train cab, and can use the libfacedetection face detection model to detect faces in the obtained pictures to obtain face images. In addition, this embodiment also uses the Onet network of MTCNN as a confidence filtering model, and passes the libfacedetetion face results through the Onet network to reduce the false detection rate of face detection.

[0044] S2: Face key point detection: The face image obtained in step 1 is subjected to key point detection by a face key point detection model to output face key points. There are 68 face key points in this step. The face key point detection model can use mobilenet_v2, and has different weights according to the facial features and contour areas. Those skilled in the art know that this embodiment uses the mobilenet_v2 model, but other face key point detection models may also be applicable.

[0045] S3: Face alignment: After performing S2 key point detection on the face image, the similarity transformation matrix between two points is calculated using the positions of five key points such as the facial features, and the face alignment is performed using the similarity transformation matrix to obtain the aligned face.

[0046] S4: Eye opening and closing detection: Using the face area aligned in S3, crop the left eye area and the right eye area, without distinguishing between left and right eyes, and sort out the open eye pictures and closed eye pictures as two categories. Here, the closed eye pictures are pictures with all the eye areas closed. The cropped Mobilent_v2 is used as the backbone network to train the face open and closed eye classification model. The classification model is used to classify the eye area of ​​the current face picture into open and closed eyes, and output the open and closed state of the left and right eyes at the same time. Here, 0 represents open eyes and 1 represents closed eyes.

[0047] S5: Mouth opening and closing detection: Using the face area aligned by S3, crop the mouth area of ​​the face, and sort out the two categories of pictures of mouth opening and closing. Here, the closed mouth picture means that the mouth area is all closed. Use the cropped Mobilenet_v2 as the backbone network to train the face mouth opening and closing classification model. Use the current classification model to judge the mouth area of ​​the face, and output the mouth opening and closing state. Here, 0 represents the mouth opening and 1 represents the mouth closing state.

[0048] S6: Head deflection angle analysis: Use S2 to extract 68 facial key points of the current image, and select 14 points to calculate the rotation Euler angle of the current face. And set the threshold for large-angle faces. When the face yaw angle exceeds ±60 degrees, the pitch angle exceeds ±30 degrees, and the roll angle exceeds ±30 degrees, it can be judged that the head deflection angle is too large and the current face has a lack of concentration.

[0049] S7: Eye opening and closing analysis: using the 68 key points of the current face image extracted by S2, within the range of the head deflection angle set in S8, select the left eye area to calculate the center point of the left eye area, and calculate the ratio of the Euclidean distance between the upper and lower eyelids of the left eye and the center point; select the right eye area to calculate the center point of the right eye area, and calculate the ratio of the Euclidean distance between the upper and lower eyelids of the right eye and the center point of the right eye; take out the smallest value of the right eye ratio and the right eye ratio and compare it with the set eye opening and closing threshold (2.08), if it is less than the set threshold, it is the closed eyes state 1, and if it is greater than the threshold, it is the open eyes state 0.

[0050] In order to improve the accuracy of eye-opening and eye-closing judgment, the above threshold value is too small to easily lead to misjudgment. For example, some faces have small eyes. Therefore, it is necessary to make a decision together with the eye state judged by the S4 eye-opening and eye-closing model. The method is as follows: first perform eye-opening and eye-closing analysis and judgment according to the threshold in S6, and then perform an AND operation with the classification result obtained by the S4 eye-opening and eye-closing model. Only when both meet the eye-closing state at the same time, the final eye state can be obtained.

[0051] S8: Mouth opening and closing analysis: Using the 68 key points of the current face image extracted by S2, within the head deflection angle range set by S6, select the center point of the mouth area, calculate the Euclidean distance ratio between the upper lip and the lower lip in the mouth area and the center point of the mouth, and compare it with the mouth opening and closing threshold (2.22). If it is greater than the threshold, it is an open mouth state 1, and if it is less than the threshold, it is a closed mouth state 0. Moreover, the threshold of this mouth area is set relatively large because the mouth opens and closes more when people yawn. In order to improve the accuracy of mouth area judgment here, it is necessary to combine the results of the mouth opening and closing detection model of S5, and combine the values ​​of the two to perform an AND operation. Only when both are 1 will the mouth be judged as an open mouth state.

[0052] Combining the eight steps mentioned above, fatigue driving detection and analysis can be performed on a single-frame face image, and three analysis states can be correctly obtained in real time. These three states can be used to make supervisory reports to back-end managers.

[0053] S9: Multi-frame face state analysis: Combined with S1-S8 for single-frame face state analysis, if the train cab collects a long video sequence, it is necessary to perform statistics on the multi-frame states. The statistical process is as follows Figure 1As shown: First, read each frame of the picture according to the video sequence input, perform single-frame face state analysis on each input frame, obtain the final analysis result of each picture and save it to a buffer queue. The buffer queue size is 8-frame face state storage area. Read eight frames of fatigue state analysis results each time and cache them until the eight-frame queue is full. Start reading and counting the state value of each frame from the head of the queue. If more than five frames of face have open or closed eyes, yawning, or inattention, an alarm will be issued, and the subsequent frame states will be analyzed and saved at the end of the queue. In this way, the queue will output the face state statistics of consecutive frames.

[0054] The method of this embodiment is used on the Rising Micro RK3399 tablet, and the overall fatigue state judgment accuracy is above 96%. The speed can reach 25 to 30 frames per second, which can meet the real-time requirements of the train cab.

[0055] The driving behavior of this embodiment is comprehensively judged to determine whether the person is in a fatigue state based on the deflection angle of the head, the closure of the eyes, and the degree of mouth opening. In addition, statistics of multiple frames of data are used to determine whether there is fatigue driving, thereby improving the accuracy of fatigue driving judgment.

[0056] This embodiment uses a deep learning method to detect the driver's fatigue driving state. The basic steps are to first perform face detection, detect 68 key points of the face, and use the 68 points to calculate the distance between the eyes and the distance between the lips in the eye area, and use the 68 points to convert the Euler angle to calculate the head steering and other three states to perform the first step of fatigue state judgment. After that, the visual features of the eye area are extracted to establish a visual model for eye opening and closing judgment to improve the recognition ability of visual features, and then the yawning judgment is combined with the opening and closing features of the lip area, and the driver is further judged whether he is tired based on the frequency of eye opening and closing and yawning. Compared with the local judgment of the prior art, it reduces the misjudgment of a single feature and reduces the false alarm signal caused by misjudgment. Therefore, this solution adopts a combination of two optimization strategies in terms of detection accuracy, which can accurately detect the fatigue state of the face.

[0057] In terms of real-time performance, compared with the existing technology, this solution has performed different degrees of quantization compression on models such as face detection, key point detection, eye area, and mouth area, which greatly increases the overall detection speed compared to traditional methods and can reach a detection speed of 25-30fps, which can basically meet the system's real-time requirements.

[0058] Preferably, after step S3, the method further includes:

[0059] S31: facial feature extraction. Based on the image aligned by S3, the facial feature extraction model is trained using ArcFace, which compresses the intra-class spacing of facial features and expands the inter-class spacing, so that it can extract more discernible facial features. The ArcFace loss function uses the currently mainstream softmax and its variant losses (A-softmax, arcface, cosface, etc.) for facial feature extraction to obtain a facial feature extractor.

[0060] S32: Face matching, establish a standard test library for driver faces, including as many face sample images as possible in various scenes and at various angles in the cab; use the test library to test the face feature extractor obtained in S31 on the current input face image, using cosine similarity as the evaluation index, and analyze the test results to determine whether the current driver of the train is the driver himself, and record the first login time, which is used as the driver's attendance statistics.

[0061] The present invention also provides a computer-readable storage medium on which a computer program is stored, wherein the program implements the steps of the fatigue driving detection method when executed by a processor.

[0062] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the fatigue driving detection method when executing the program.

[0063] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention also fall within the scope of protection of the present invention.

Claims

1. A method for detecting fatigue driving in a train cab scenario, characterized in that: The method comprises the following steps: S0: Obtaining a face image, using an image acquisition device to collect a face image in a train cab scene; S1: face detection, using a face detection model to perform face detection on the acquired face picture to obtain a face image; S2: facial key point detection, using a convolutional neural network to obtain the key points of the facial image; S3: judging the face status of the current frame; S4: Based on the judgment result obtained in step S3, performing multi-frame face status statistics; S5: Determine whether the set frame number is reached, if so, output the mouth, eyes and head status, if not, return to step S1; After step S2, the method includes step S2.1: face alignment, using the positions of five key points in the facial features area to calculate the similarity transformation matrix between two points, and using the similarity transformation matrix to perform face alignment to obtain the aligned face; The step S3 further includes: S3.1: Eye opening and closing detection, using the face area aligned in step S2.1, cropping out the left eye area and the right eye area; using the first classification model to classify the eye area of ​​the current face image into open and closed eyes, and outputting the open or closed state of the left and right eyes; S3.2: Mouth opening and closing detection: using the face area aligned in step S2.1, cropping out the mouth area of ​​the face; using the second classification model to judge whether the mouth area of ​​the face is open or closed, and outputting the open or closed state; S3.3: head deflection angle analysis, using step S2 to extract 68 facial key points of the current image, select 14 points to calculate the rotation Euler angle of the current face, compare the rotation Euler angle with the set head deflection angle, and determine whether the current face has inattention phenomenon; S3.4: Eye opening and closing analysis, using the 68 key points of the current face image extracted in step S2, within the head deflection angle range set in step S3.3, select the center point of the left eye area and the center point of the right eye area, and calculate the ratio of the Euclidean distance between the upper and lower eyelids of the left eye and the center point of the left eye, and the ratio of the Euclidean distance between the upper and lower eyelids of the right eye and the center point of the right eye; compare the above ratios with the set eye opening and closing thresholds to determine the open or closed state of the left and right eyes; S3.5: Mouth opening and closing analysis, using the 68 key points of the current face image extracted in step S2, select the center point of the mouth area within the head deflection angle range set in step S3.3, calculate the Euclidean distance ratio between the upper lip and the lower lip in the mouth area and the mouth center point, and compare it with the mouth opening and closing set threshold to determine the open or closed state of the mouth; In step S3.4, the above ratio is compared with the set eye opening and closing thresholds to determine the open or closed state of the left and right eyes. Specifically, if the ratio is less than the set threshold, it is in the closed state, and if it is greater than the threshold, it is in the open state. And combined with the result of the eye-opening and eye-closing detection in step S3.1, the two values ​​are combined to perform an AND operation, and only when both are in the eye-closing state at the same time, the final eye-closing state is obtained; In step S3.5, the ratio is compared with the mouth opening and closing threshold to determine whether the mouth is open or closed. Specifically, if the ratio is greater than the threshold, the mouth is open; if it is less than the threshold, the mouth is closed. And combined with the result of the mouth opening and closing detection in step S3.2, the values ​​of the two are combined to perform an AND operation. Only when both are in the open mouth state will the mouth be judged to be in the open mouth state.

2. The method for detecting fatigue driving in a train cab scenario according to claim 1 is characterized in that: In step S1, libfacedetection is used as the face detection model.

3. The method for detecting fatigue driving in a train cab scenario according to claim 1 is characterized in that: In step S2, mobilenet_v2 is used to extract facial key points.

4. The method for detecting fatigue driving in a train cab scenario according to claim 1 is characterized in that: After step S2.1, the method further includes: S2.2: extracting facial features, according to the images aligned in step S2.1, extracting facial features based on the facial feature extraction model, and obtaining a facial feature extractor; S2.3: Face matching, to determine whether the current train driver is the driver himself, and to record the first login time, which is used as the driver's attendance statistics.

5. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the program is executed by a processor, the steps of the fatigue driving detection method based on a train cab scenario as described in any one of claims 1-4 are implemented.

6. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the fatigue driving detection method based on a train cab scenario as described in any one of claims 1 to 4 when executing the program.

Images