A fatigue detection method based on binary differentiable facial key points

Abstract

This invention discloses a fatigue detection method based on binary differentiable facial key points. The method collects driver facial video data, obtains facial position information and key point coordinates in each frame of the video data, crops left and right eye images based on the key point coordinates, creates a dataset, establishes a binary differentiable deep convolutional segmentation network model, trains the eye segmentation model, and inputs new video data. The method then uses the MTCNN face detection algorithm and eye segmentation model to obtain the segmented regions of the left and right eyes in each frame and calculates the corresponding areas. Based on the calculated areas of the segmented regions, a formula is used to calculate eye closure within a time period t. This invention proposes a differentiable binary eye segmentation algorithm and a method for calculating eye closure, thereby improving the accuracy of driver fatigue detection.

Description

Technical Field

[0001] The present invention relates to a fatigue detection method, in particular to a fatigue detection method based on binary differentiable facial key points, belonging to the technical field of intelligent driving. Background Art

[0002] With my country's socioeconomic development, improved living standards, and the improvement of domestic road infrastructure, the types and number of commuter vehicles in the municipal sanitation sector are increasing, including road sweepers, garbage transfer trucks, and water sprinklers. Simultaneously, artificial intelligence technology has experienced rapid development in recent years, with the emergence of numerous novel and efficient deep learning algorithms. Technological advancement requires practical application scenarios, and motor vehicle safety assisted driving is a representative area for the application of deep learning technology in machine vision. Traffic accidents caused by dangerous driver behavior during operation are a major cause of significant losses to socioeconomic development, the national economy, and people's livelihoods. In particular, sanitation vehicles operate at various times, some even at night when traffic is low. During these times, operators are prone to involuntary fatigue while on duty. Therefore, it is crucial to address driver fatigue during operation. Real-time detection and early warning can effectively reduce the incidence of traffic accidents involving sanitation vehicles.

[0003] Some current research focuses on methods for diagnosing driver fatigue based on facial key points, primarily using deep learning target detection. For example, patent CN108460345 first extracts a video stream containing the driver's entire facial expression while driving within a unit of time. Each frame in the video stream is then processed, using the facial key points in each frame to determine whether the eyes are closed or the mouth is open too wide. Finally, the PECLOS method is used to detect the number of times the eyes are closed or the mouth is open too wide within the video stream within a unit of time, thereby determining whether the driver is fatigued. However, the eyes and mouth do not completely close when fatigued, resulting in inaccurate fatigue detection. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a fatigue detection method based on binary differentiable facial key points to improve the accuracy of driving fatigue detection.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A fatigue detection method based on binary differentiable facial key points, characterized by comprising the following steps:

[0007] Step 1: Collect driver’s facial video data;

[0008] Step 2: Obtain the face position information and key point coordinates in each frame of the video data;

[0009] Step 3: Crop the left-eye and right-eye images according to the key point coordinates to create a dataset;

[0010] Step 4: Establish a binary differentiable deep convolutional segmentation network model and train the human eye segmentation model;

[0011] Step 5: Input new video data, obtain the segmented regions of the left and right eyes in each frame based on the MTCNN face detection algorithm and the eye segmentation model, and calculate the corresponding areas;

[0012] Step 6: Based on the calculated area of ​​the left and right eye segments, use the formula to calculate the eye closure situation during time t.

[0013]

[0014] Among them, close_eye is the eye closing condition, t is the calculation time period, T is the time threshold, area i is the area of ​​the eye region segmented at time i, area1, area2, ..., area t Indicates the area of ​​the eye region segmented at different times.

[0015] Furthermore, the step 1 is specifically as follows: collecting video data of the driver's face through a camera set in the cockpit, and the camera collection area reserves the driver's activity area.

[0016] Furthermore, the step 2 is specifically as follows: decomposing the collected facial video data into image data in frames, and then processing the image data of each frame based on the MTCNN face detection algorithm to obtain the face position information and key point coordinates in each frame of image data.

[0017] Furthermore, the key point coordinates include left eye coordinates, right eye coordinates, nose coordinates and mouth coordinates.

[0018] 5. The method for fatigue detection based on binary differentiable facial key points according to claim 1, wherein the step 3 is specifically:

[0019] 3.1、Use the obtained left eye coordinates (x left ,y left ) and right eye coordinates (x right ,y right ), generate the coordinates of the upper left corner and lower right corner of the rectangular area of ​​the left eye, respectively x left_top =x left -20,yleft_top =,y left -20,x left_bottom =x left +20,y left_bottom =y left +20;x right_top =x right -20,y right_top =y right -20,x right_bottom =x right +20,y right_bottom =y right +20, and then get the corresponding left eye area and right eye area according to the coordinates.

[0020] 3.2. Create probability map labels and threshold labels respectively. In the rectangular area, the left eye area can be described as D1, and the right eye area can be described as D2. Calculate the perimeter L1 and area A1 of D1, calculate the perimeter L2 and area A2 of D2, and calculate the shrinkage ratio of D1 and D2 respectively. Where r = 0.4;

[0021] 3.3. Use the Vatti cropping algorithm to shrink D1 to D1′ and D2 to D2′; fill D1′ and D2′ with 0, that is, use D1′ and D2′ to represent the probability map label;

[0022] 3.4. Use the Vatti clipping algorithm to expand D1 to D1′′, D2 to D2′′, and make the values ​​between D1′′-D1 and D2′′-D2 follow the normal distribution, that is, D1′′, D2′′ represent the threshold labels.

[0023] Furthermore, the step 4 is specifically as follows:

[0024] 4.1. Unify the size of the input image and ensure that the size of the input image is a multiple of 32;

[0025] 4.2. First, the input image is passed through a five-layer feature pyramid. The size of each layer of the feature pyramid will be halved relative to the previous layer, and finally reduced by 32 times, and the features obtained at each feature layer are saved; secondly, the features of each layer of the pyramid are upsampled and the feature size is made the same, and all feature layers are fused; then the fused features are convolved separately, keeping the feature size unchanged, and part of them are used to generate the probability map probability_feature and part of them are used to generate the threshold map threshold_feature; finally, the probability_feature and threshold_feature are used to generate the binary image binary_feature;

[0026] 4.3. The loss function of the binary differentiable deep convolutional segmentation network model is

[0027] Loss = L binary_feature +L probability_feature +L threshold_feature ;

[0028] 4.4. Set other hyperparameters for training.

[0029] Furthermore, the method for producing the binary image binary_feature is specifically as follows:

[0030] The binary image binary_feature is produced using a differentiable binary equation. The differentiable binary equation formula can be expressed as

[0031]

[0032] The k value is set to 30, and i and j represent the positions in the two-dimensional matrix respectively.

[0033] Furthermore, the loss function is specifically:

[0034] The loss of the binary differentiable deep convolutional segmentation network model consists of three parts: L binary_feature , L probability_feature , L threshold_feature ;

[0035] L binary_feature , L probability_feature Using the cross entropy loss function, it can be expressed as

[0036] L binary_feature =L probability_feature =∑ i∈S y i logx i +(1-y i )log(1-x i )

[0037] L threshold_feature Using the L1 loss function, it can be expressed as

[0038] L threshold_feature =∑ i∈S |y i -x i |

[0039] Right now

[0040] Loss = L binary_feature +L probability_feature +L threshold_feature

[0041] Among them, Lbinary_feature represents the binary image loss, L probability_feature represents the probability map loss, L threshold_feature Threshold map loss.

[0042] Furthermore, the step 5 is specifically as follows: input the t time video stream, first use the MTCNN model to detect the coordinates of the left and right eye key points, extract the left and right eye areas, input the left and right eye area information to the binary differentiable deep convolutional segmentation network model to segment the actual range of the eyeball, and calculate the area after segmentation, which is recorded as eye_area=[area1,area2,...,area t ].

[0043] Compared with the prior art, the present invention has the following advantages and effects:

[0044] 1. The present invention segments the human eye region through a differentiable binary segmentation method based on facial key points. First, the present invention can adaptively set the threshold during the training process and can efficiently and accurately segment the human eye, which plays a key role in the subsequent accurate judgment of whether the human eye is closed. It also solves the traditional post-segmentation processing operation and has good robustness compared with traditional methods.

[0045] 2. The present invention proposes a method for adaptively judging whether the human eye is closed based on the time dimension. The threshold T is set to 0.5. First, the left eye and the right eye are segmented within the time t, and the area of ​​each frame after segmentation is recorded. eye_area = [area1, area2, ..., area t ], the formula for calculating eye closure is Among them, one pixel point is one unit area. The ratio of the area of ​​the eye at each moment to the area of ​​the largest eye within time t is calculated, and the ratio is compared with the set threshold T. If the ratio is less than the threshold T, it means that the eyes are closed, which increases the accuracy of eye closure judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of a fatigue detection method based on binary differentiable facial key points of the present invention.

[0047] Figure 2 Schematic diagram of faces and key points detected based on MTCNN of the present invention.

[0048] Figure 3 It is a schematic diagram of producing probability map labels and threshold labels of the present invention.

[0049] Figure 4 It is a schematic diagram of the binary differentiable deep convolutional segmentation network model of the present invention. DETAILED DESCRIPTION

[0050] In order to elaborate on the technical solutions adopted by the present invention to achieve the predetermined technical purpose, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, and the technical means or technical features in the embodiments of the present invention can be replaced without creative work. The present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0051] like Figure 1 As shown, the present invention is a method for fatigue detection based on binary differentiable facial key points, comprising the following steps:

[0052] Step 1: Collect driver's facial video data.

[0053] The camera installed in the cockpit collects video data of the driver's face, and the camera collection area reserves the driver's activity area.

[0054] Step 2: Obtain the face position information and key point coordinates in each frame of the video data.

[0055] The collected facial video data is decomposed into image data in frames, and then the image data of each frame is processed based on the MTCNN face detection algorithm to obtain the face position information and key point coordinates in each frame of image data. Figure 2 As shown, the key point coordinates include the left eye coordinates, the right eye coordinates, the nose coordinates and the two mouth coordinates.

[0056] Step 3: Crop the left-eye and right-eye images according to the key point coordinates to create a dataset.

[0057] 3.1、Use the obtained left eye coordinates (x left ,y left ) and right eye coordinates (x right ,y right ), generate the coordinates of the upper left corner and lower right corner of the rectangular area of ​​the left eye, respectively x left_top =x left -20,y left_top =,y left -20,x left_bottom =x left +20,y left_bottom =y left +20;x right_top =x right -20,y right_top =y right -20,x right_bottom =x right +20,y right_bottom =yright +20, and then get the corresponding left eye area and right eye area according to the coordinates.

[0058] 3.2. Create probability map labels and threshold labels respectively, such as Figure 3 As shown, in the rectangular area, the left eye area can be described as D1, the right eye area can be described as D2, calculate the perimeter L1 and area A1 of D1, calculate the perimeter L2 and area A2 of D2, and calculate the shrinkage ratio of D1 and D2 respectively Where r = 0.4;

[0059] 3.3. Use the Vatti cropping algorithm to shrink D1 to D1′ and D2 to D2′; fill D1′ and D2′ with 0, that is, use D1′ and D2′ to represent the probability map label;

[0060] 3.4. Use the Vatti clipping algorithm to expand D1 to D1′′, D2 to D2′′, and make the values ​​between D1′′-D1 and D2′′-D2 follow the normal distribution, that is, D1′′ and D2′′ represent the threshold labels.

[0061] Step 4: Establish a binary differentiable deep convolutional segmentation network model and train the human eye segmentation model.

[0062] 4.1. Unify the size of the input image and ensure that the size of the input image is a multiple of 32;

[0063] 4.2, such as Figure 4 As shown, first, the input image passes through the five-layer feature pyramid. The size of each layer of the feature pyramid will be halved relative to the previous layer, and finally reduced by 32 times, and the features obtained in each feature layer are saved; secondly, the features of each layer of the pyramid are upsampled, and the feature size is made the same size, and all feature layers are fused; then the fused features are convolved separately, keeping the feature size unchanged, and one part generates the probability map probability_feature, and the other part generates the threshold map threshold_feature; finally, the probability_feature and threshold_feature are used to generate the binary image binary_feature;

[0064] In the previous state-of-the-art methods, binary_feature is generated by comparing the size of the elements in probability_feature under the action of the hyperparameter threshold t, using the formula described as

[0065]

[0066] In this method, the hyperparameter t cannot be learned during training. If the model is trained in a large number of complex scenarios, the model will not be robust.

[0067] The specific method for producing binary images binary_feature is:

[0068] The present invention proposes a differentiable binary equation, which is used to produce a binary image binary_feature. The differentiable binary equation formula can be expressed as

[0069]

[0070] Here, the k value is set to 30, and i and j represent the positions in the two-dimensional matrix. The threshold_feature can be adjusted by learning from a large amount of data to learn the most appropriate parameters and finally complete the segmentation of the model.

[0071] 4.3. The loss function of the binary differentiable deep convolutional segmentation network model is

[0072] Loss = L binary_feature +L probabiliry_feature +L threshold_feature ;

[0073] The loss function is specifically:

[0074] The loss of the binary differentiable deep convolutional segmentation network model consists of three parts: L binary_feature , L probability_feature , L threshold_feature ;

[0075] L binsry_feature , L probability_feature Using the cross entropy loss function, it can be expressed as

[0076] L binary_feature =L probability_feature =∑ i∈S y i logx i +(1-y i )log(1-x i )

[0077] L threshold_feature Using the L1 loss function, it can be expressed as

[0078] L threshold_feature =∑ i∈s |y i -x i |

[0079] Right now

[0080] Loss = Lbinary_feature +L probabiliry_feature +L threshold_feature

[0081] Among them, L binery_festure represents the binary image loss, L probability_feature represents the probability map loss, L threshold_feature Threshold map loss.

[0082] 4.4. Set other training hyperparameters, such as the initial value of the learning rate lr to 0.01 and the Xavier initialization weight parameters, and train the segmentation network.

[0083] Step 5: Input new video data, obtain the segmentation areas of the left and right eyes in each frame image based on the MTCNN face detection algorithm and the eye segmentation model, and calculate the corresponding areas.

[0084] Input the t time video stream, first use the MTCNN model to detect the coordinates of the left and right eye key points, extract the left and right eye areas, input the left and right eye area information to the binary differentiable deep convolutional segmentation network model to segment the actual range of the eyeball, and calculate the area after segmentation, recorded as eye_area = [area1, area2, ..., area t ].

[0085] Step 6: Based on the calculated area of ​​the left and right eye segments, use the formula to calculate the eye closure situation during time t.

[0086]

[0087] Among them, close_eye is the eye closing condition, t is the calculation time period, T is the time threshold, area i is the area of ​​the eye region segmented at time i, area1, area2, ..., area t Indicates the area of ​​the eye region segmented at different times.

[0088] Based on the key points of the face, the present invention proposes a differentiable binarized human eye segmentation algorithm and a method for calculating whether the human eyes are closed, in order to solve the problem of driver fatigue driving. The differentiable binarized human eye segmentation algorithm proposed in the present invention is used to adaptively segment the human eyes, making the segmentation more accurate and more robust in complex environments, such as strong light or dark light. At the same time, it can solve the post-processing operations of the segmentation completed in the current most advanced technology to a certain extent. The present invention proposes a method for adaptively judging whether the human eyes are closed based on the time dimension, and proposes the formula

[0089] Taking into account the state changes in continuous time, the driver's state in the time period can be better expressed, and the judgment is more reasonable and accurate.

[0090] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principles of the present invention, without departing from the content of the technical solution of the present invention, shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A fatigue detection method based on binary differentiable facial key points, characterized by The following steps are involved: Step 1: Collect driver’s facial video data; Step 2: Obtain the face position information and key point coordinates in each frame of the video data; Step 3: Crop the left-eye and right-eye images according to the key point coordinates to create a dataset; The step three is specifically as follows: 3.

1. Using the acquired left eye coordinates and right eye coordinates , generate the coordinates of the upper left corner and lower right corner of the rectangular area of ​​the left eye, respectively ， , Then obtain the corresponding left eye area and right eye area according to the coordinates; 3.

2. Create probability map labels and threshold labels respectively. In the rectangular area, the left eye area can be described as D1, and the right eye area can be described as D2. Calculate the perimeter L1 and area A1 of D1, calculate the perimeter L2 and area A2 of D2, and calculate the shrinkage ratio of D1 and D2 respectively. , where r=0.4; 3.3, Use Vatti clipping algorithm to shrink D1 to , D2 to ; , Internally filled with value 0, that is, represents the probability map label; 3.4, use the vatti clipping algorithm to expand D1 to ', D2 to ', and make '-D1, The values ​​between '-D2 follow the normal distribution, that is, ', ' represents the threshold label; Step 4: Establish a binary differentiable deep convolutional segmentation network model and train the human eye segmentation model; Step 5: Input new video data, obtain the segmented regions of the left and right eyes in each frame based on the MTCNN face detection algorithm and the eye segmentation model, and calculate the corresponding areas; Step 6: Based on the calculated area of ​​the left and right eye segments, use the formula to calculate the eye closure situation during time t. close_eye= / max( )<T Among them, close_eye is the eye closing condition, t is the calculation time period, T is the time threshold, is the area of ​​the eye region segmented at time i, Indicates the area of ​​the eye region segmented at different times.

2. The method for fatigue detection based on binary differentiable facial key points according to claim 1, characterized in that: The step 1 specifically includes: collecting video data of the driver's face through a camera set in the cockpit, and the camera collection area reserves the driver's activity area.

3. The method for fatigue detection based on binary differentiable facial key points according to claim 1, characterized in that: The step 2 is specifically as follows: the collected facial video data is decomposed into image data in frames, and then the image data of each frame is processed based on the MTCNN face detection algorithm to obtain the face position information and key point coordinates in each frame of image data.

4. The method for fatigue detection based on binary differentiable facial key points according to claim 3, characterized in that: The key point coordinates include left eye coordinates, right eye coordinates, nose coordinates and mouth coordinates.

5. The method for fatigue detection based on binary differentiable facial key points according to claim 1, characterized in that: The step 4 is specifically as follows: 4.

1. Unify the size of the input image and ensure that the size of the input image is a multiple of 32; 4.

2. First, the input image is passed through a five-layer feature pyramid. The size of each layer of the feature pyramid will be halved relative to the previous layer, and finally reduced by 32 times, and the features obtained at each feature layer are saved; secondly, the features of each layer of the pyramid are upsampled and the feature size is made the same, and all feature layers are fused; then the fused features are convolved separately, keeping the feature size unchanged, and part of them are used to generate the probability map probability_feature and part of them are used to generate the threshold map threshold_feature; finally, the probability_feature and threshold_feature are used to generate the binary image binary_feature; 4.

3. The loss function of the binary differentiable deep convolutional segmentation network model is Loss = ; 4.

4. Set other hyperparameters for training.

6. The method for fatigue detection based on binary differentiable facial key points according to claim 5, characterized in that: The method for producing the binary image binary_feature is specifically as follows: The binary image binary_feature is produced using a differentiable binary equation. The differentiable binary equation formula can be expressed as The k value is set to 30, and i and j represent the positions in the two-dimensional matrix respectively.

7. The method for fatigue detection based on binary differentiable facial key points according to claim 6, characterized in that: The loss function is specifically: The loss of the binary differentiable deep convolutional segmentation network model consists of three parts: , , ; , Using the cross entropy loss function, it can be expressed as = = Using the L1 loss function, it can be expressed as = Right now Loss = in, represents the binary image loss, represents the probability map loss, Threshold map loss.

8. The method for fatigue detection based on binary differentiable facial key points according to claim 6, characterized in that: The step 5 is as follows: input the t time video stream, first use the MTCNN model to detect the coordinates of the left and right eye key points, extract the left and right eye areas, input the left and right eye area information into the binary differentiable deep convolutional segmentation network model to segment the actual range of the eyeball, and calculate the segmented area, which is recorded as .

Images