A fatigue detection method based on deep learning face posture estimation

[0025] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0026] Reference Figure 1-2 , The fatigue detection method based on deep learning face pose estimation includes the following steps:

[0027] S1. Obtain the video stream of the camera in the cab: First, collect the driving state video of the driver through the on-board camera in the cab, detect the face area based on the HOG extraction algorithm, and frame it with a rectangular frame, and use the face recognition module, including the face recognition module Use the on-board camera installed in front of the driver to collect the driver's driving status video, use the HOG-based face detector to detect each frame of the video stream, identify the area where the face is located, and mark the face with a rectangular frame;

[0028] S2. Positioning of key points of the face: Use the landmark facial feature point extraction technology based on ERT integrated regression tree to determine the key points of the face, where the key points of the face include the positions of eyes, mouth, jaw and nose; where ERT integrated regression tree The core formula is as follows:

[0029]

[0030] Train each r using the regression tree of gradient enhancement learning t , Use the least square method to minimize the error. t represents the cascade serial number, r t (·,·) represents the current level of regressor. The input parameters of the regressor are image I and the updated shape of the previous regressor, and the features used can be gray values ​​or other. Each regressor is composed of many trees, and the parameters of each tree are trained based on the coordinate difference between the current shape and the ground truth and randomly selected pixel pairs.

[0031] ERT is to directly store the updated value ΔS of shape in the leaf node leaf node in the process of learning Tree. The initial position S is after passing through all the learned trees, and the meanshape plus the ΔS of all the leaf nodes that have passed, you can The final position of the key points of the face is obtained, and the key point positioning module of the face is adopted, and the key point positioning module of the face uses the landmark facial feature point extraction technology based on the ERT integrated regression tree to detect the key points of the face.

[0032] S3. Define a 3D face model with 6 key points, and the 3D face model includes left and right corners of the eyes, left and right corners of the mouth, nose tip and chin. The 6 key points corresponding to the driver’s face will be obtained using the solvePnP function in OpenCv To estimate Pose, determine the affine transformation matrix from the 3D model to the driver’s face image, which contains the rotation and translation information, that is, the movement information of tilting and nodding. The rotation vector is converted into Euler angles with the rotation vector, ( Pitch rotates around the X axis, the pitch angle yaw rotates around the Y axis, and the yaw angle roll rotates around the Z axis, which is called the roll angle). Pitch and yaw are used to represent the angle of nodding and tilting the head respectively. Pitch: Nodding, up and down. Yaw: Tilt your head, left and right. Set the threshold. If the pitch and yaw exceed the set threshold within the specified time, it will be judged as a fatigue state and an alarm will be triggered.

[0033] The head posture detection and early warning module and the eye and mouth early warning module are used. The head posture detection and early warning module mainly defines a 3D face model with 6 key points, and then uses the landmark detection technology to detect each face in the video frame The corresponding 6 key points of the face in the picture (lower jaw: 8 nose tip: 30 left eye corner: 36 right eye corner: 45 left mouth corner: 48 right mouth corner: 54), use OpenCV's solvePnP function to solve the rotation vector, and finally convert the rotation vector It is the Euler angle. The rotation angle of this Euler angle is used to determine whether the driver has nodding and tilting his head. If it is determined to be fatigued, an alarm will be triggered to give an alarm; the eye and mouth posture warning module mainly uses the eyes and mouth The key points of the part calculate the Euclidean distance between the key points in the horizontal and vertical directions, and finally calculate the aspect ratio and compare with the set threshold to determine the driver's state. If it is determined to be fatigued, an alarm will be triggered to give an alarm.

[0034] The head posture detection and early warning module and the eye and mouth early warning module are both early warning modules. The early warning module includes a head posture judgment unit and an eye and mouth posture judgment unit; the head posture judgment unit uses the detected driver’s face The 6 key points are compared with the defined 6 key points of the 3D face model. The rotation vector is parsed through the solvePnP function and converted into Euler angles. The angle of the Euler angles determines whether the driver’s head is nodding or tilting his head. If the duration of the nodding and tilting state exceeds the set time threshold, it is determined that the driver is driving fatigued, and then an alarm is triggered. The eye and mouth posture judgment unit mainly locates the key points of the eyes and the mouth through the face key point positioning technology, and then calculates the Euclidean distance between the horizontal and vertical key points of the eyes and the mouth, and finally calculates the level The ratio of the Euclidean distance between the direction and the vertical direction (aspect ratio) is calculated by comparing the aspect ratio with the set threshold for eye opening and mouth opening and closing, and combining the duration of eye closure or mouth opening and The comparison between the set time thresholds comprehensively determines whether the driver is in a state of fatigue, and triggers an alarm if the driver is in a state of fatigue.

[0035] S4. Locate the positions of the eyes and mouth according to the key points of the face in S3, and calculate the horizontal and vertical Euclidean distances between the eyes and the mouth according to the coordinates;

[0036]

[0037]

[0038]

[0039]

[0040] In the formula, d1 is the Euclidean distance in the horizontal direction of the eyes, d2 and d3 are the Euclidean distance in the vertical direction; A is the eye aspect ratio; the mouth aspect ratio is calculated in the same way as the eyes;

[0041] Step 5: According to the cumulative number of nodding of the driver, it is determined that the number of nodding exceeds the set threshold. The number of nods of the driver is lower than the set threshold. The aspect ratio of the mouth and the aspect ratio of the eyes are compared with the set threshold to determine that the mouth is closed The state of the eye exceeds the set threshold alarm.

[0042] The invention is ingenious in design, capable of real-time detection, and has the characteristics of strong anti-interference, comprehensive identification and detection, high accuracy and strong stability, etc., and can fundamentally remind drivers to drive to reduce the probability of accidents.

[0043] Judging whether the driver's head is nodding or tilting the head through the angle of Euler angle, the nodding and tilting state lasts longer than the set time threshold, it is determined that the driver is driving fatigued, and then the alarm is triggered.

[0044] Through the comparison between the aspect ratio and the set threshold of eye opening and mouth opening and closing, and combining the comparison between the duration of eye closure or mouth opening and the set time threshold, comprehensively determine whether the driver is In a state of fatigue, an alarm will be triggered if it is in a state of fatigue.

[0045] The above are only preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited to this. Anyone familiar with the technical field within the technical scope disclosed by the present invention, according to the technical solution of the present invention The equivalent replacement or change of the inventive concept thereof shall be covered by the protection scope of the present invention.