A driver fatigue monitoring method based on Gaussian distribution weakly supervised learning

Abstract

The present invention belongs to the field of traffic safety assistance systems, and specifically relates to a driver fatigue monitoring method based on Gaussian distribution weak supervision learning. The method of the present invention first uses a target detection model to obtain an image of the driver's eye area; then obtains the eye aspect ratio from a facial feature point model. Then, based on a Gaussian weak supervision learning method, a Gaussian weak supervision learning classifier is provided according to changes in the individual driver's eye features, and a flexibly and dynamically formulated eye closure threshold is used to improve the accuracy of fatigue monitoring and reduce the model algorithm's dependence on the amount of data.

Description

Technical Field

[0001] The present invention belongs to the field of traffic safety auxiliary systems, and in particular relates to a driver fatigue monitoring method based on Gaussian distribution weak supervision learning. Background Art

[0002] Fatigue monitoring is to detect fatigue status and give early warning signals in time by monitoring various fatigue characteristics of the human body, involving multiple fields such as physiology, psychology, image processing, motion tracking, pattern recognition, etc. During a long period of driving, the driver's fatigue level gradually accumulates from shallow to deep. If technical means can be used to detect the driver's mental state in real time, and once there are signs of fatigue, an early warning will be issued immediately, then the safe driving coefficient will be effectively improved.

[0003] There are many research methods for detecting driver fatigue status. According to the detection category, they can be roughly divided into detection based on driver physiological response, detection based on driver operation behavior, detection based on vehicle status information and other methods. Among them, the detection method based on driver physiological response is a non-contact detection, which uses machine vision to judge fatigue. The measurement process will not interfere with the driver's normal driving behavior, and has great development potential. However, the existing methods of analyzing the state of human eyes to detect driver fatigue status are not accurate enough. The changes in the eye aspect ratio when the driver is fatigued and in normal state are often collected by statistical data to set a fixed eye closure threshold. For example, when the eye aspect ratio is less than 0.2, it is judged as fatigue state. References: Fang Haojie, Dong Hongzhao, Lin Shaoxuan, et al. Driver fatigue state detection method based on multi-feature fusion [J]. Journal of Zhejiang University: Engineering Edition, 2023, 57(7): 1287-1296; Wang Han, Qiu Liankui. Driver fatigue detection based on optimized probabilistic neural network [J]. Electronic Measurement Technology, 2023, 46(12): 105-110; Wang Peng, Shen Helong, Yin Yong, et al. Ship driver fatigue detection algorithm based on deep learning [J]. Traffic Information and Safety, 2022, 40(1): 63-71. The main problem of this method is that the setting of the eye closure threshold is based on statistics, which leads to its high dependence on data. When there is a deviation between the statistical data set and the target data to be detected, the accuracy of fatigue judgment is greatly reduced. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a driver fatigue monitoring method based on Gaussian distribution weak supervision learning with high accuracy.

[0005] The driver fatigue monitoring method based on Gaussian distribution weak supervised learning of the present invention will solve the problem of decreased fatigue monitoring accuracy due to the diversification of driver eye features. The specific technical route is: first, a target detection model is used to obtain an image of the driver's eye area; then the eye aspect ratio is obtained by the facial feature point model. The next step is the main innovation. In traditional methods, a fixed eye closure threshold is often set through statistical data. When there is a deviation between the data set used for statistics and the target data of the individual to be detected, the fatigue judgment accuracy is greatly reduced. Based on the Gaussian weak supervised learning method, the present invention provides a Gaussian weak supervised learning classifier according to the changes in the eye features of individual drivers, and flexibly and dynamically formulates the eye closure threshold, thereby improving the accuracy of fatigue monitoring and reducing the model algorithm's dependence on the amount of data.

[0006] The frequency and amplitude of a driver's eye opening and closing are often used as an important indicator to measure their fatigue level. In current research, a threshold of eyelid height-to-width ratio when closing eyes is usually set, such as Figure 1 As shown, the left eye aspect ratio is Wl / Ll, and the right eye aspect ratio is Wr / Lr. Through this fixed threshold, the eye opening and closing state is judged, and then the proportion of closed eye frames to the total number of frames is counted to quantitatively judge the driver's fatigue level, see formula 1-1.

[0007]

[0008] A driver fatigue monitoring method based on Gaussian distribution weak supervision learning of the present invention comprises the following steps:

[0009] Step 1: Use the target detection model to obtain the driver's eye area image;

[0010] Step 2: Obtain the left eye aspect ratio Wl / Ll and the right eye aspect ratio Wr / Lr, that is, the left eye opening and closing ratio and the right eye opening and closing ratio, from the facial feature point model;

[0011] Step 3: Count the discrete change curve of the left-right eye aspect ratio in a video of the driver in a non-fatigue state in real time and continuously; then perform Gaussian normalization distribution calculation on the discrete change curve to obtain the mean and variance of the left-right eye opening and closing ratio, and set the opening and closing ratio corresponding to -2σ as the eye closing threshold of the driver;

[0012] Step 4: After the eye-closing threshold is set, the value is saved in the vehicle computer system, and then compared with the threshold according to the subsequent real-time input eye aspect ratio. When the aspect ratio of any eye is greater than the threshold, it is judged that the driver has his eyes open; when the aspect ratios of both eyes are less than the threshold, it is judged that the driver has his eyes closed;

[0013] Step 5: Finally, count the ratio of the number of closed-eye frames to the total number of frames in the continuous n-frame images, thereby obtaining the driver's fatigue level.

[0014] When the ratio of closed frames of the two eyes reaches 30%, 50%, and 80% respectively, it is set as mild, moderate, and severe fatigue respectively.

[0015] The above step 1 includes the following steps:

[0016] 1. Use the in-car camera to capture the driver's facial area image and transmit the RGB image to the on-board controller;

[0017] 2. In the vehicle controller, the driver's facial region image is first scaled and input into the target detection model in the controller frame by frame. The target detection model adopts the Faster RCNN model. The target detection model loss function calculation formula is shown in formula (1), where L cls is the classification loss function, L bbox is the bounding box regression loss function, L is the total loss, and λ is a hyperparameter used to balance the weight of classification loss and regression loss, which is set to 0.5 in the present invention;

[0018] L=L cls +λL bbox Formula (1)

[0019] The classification loss function uses the cross entropy loss, as shown in formula (2), where N is the number of positive samples, y i is the true label (0 or 1), p i is the probability predicted by the model.

[0020]

[0021] The bounding box regression loss function is adopted as formula (3) and formula (4), where b i is the predicted bounding box of the i-th image, is the true bounding box of the i-th image. The dimensions of each bounding box are the horizontal and vertical coordinates of the upper left corner and the lower right corner, and N is the number of positive samples.

[0022]

[0023] The backbone of the FasterRCNN model is finally connected to the fully connected layer to output the predicted bounding box: the horizontal and vertical coordinates of the upper left and lower right corners, the category label, and the confidence level.

[0024] The confidence is obtained by the Softmax function, the formula is shown in formula (5), Z i is the output value of the i-th node, σ(z i ) is the confidence of category i, is an exponential function that ensures that the output is a positive number. is the sum of the exponential functions of all categories and is used for normalization.

[0025]

[0026] Only two categories, face and non-face, are trained in this model. When the confidence level is greater than 0.8, the category is face. The corresponding face frame position coordinates are output, and the image of the area is cropped for the next model step.

[0027] When the confidence level is less than 0.8, for example, when the head rotates too much or the face is blocked, the confidence level is reduced, and the algorithm discards the frame image and does not pass it to the next step.

[0028] Step 2 includes the following steps:

[0029] The facial region images with a confidence level higher than 0.8 are cropped and scaled, and then fed into the feature point model, which locates the key points of the left and right eyelid contours.

[0030] The feature point model uses the HRNet model and outputs a heat map for each predicted key point. The heat map loss function is constructed based on the mean square error, and the formula is (6), where Loss heatmap is the heat map loss function, N is the number of key points, H i is the real heat map, For the prediction heatmap:

[0031]

[0032] The heat map is in three dimensions (H, W, K), which represent the height, width and number of sheets (the number of key points) of the heat map. The probability value of each pixel will be output in the heat map. The Softmax function is also used to complete the normalization operation, and the probability value is controlled between 0 and 1. The coordinates of the point with the highest probability are the coordinates of the predicted key points. The key point model will output a total of 8 point coordinates for the left and right eyes, including the upper, lower, left, and right sides of the left eyelid, and the upper, lower, left, and right sides of the right eyelid.

[0033] 2. Calculate the distance between features, including the left eye aspect ratio Wl / Ll and the right eye aspect ratio Wr / Lr.

[0034] The overall process of the method of the present invention is as follows: Figure 3Description. In the method of using machine vision to judge driver fatigue, the accuracy of the eye closure threshold setting directly affects the correctness of the eye opening and closing judgment, thereby affecting the judgment of the degree of fatigue. In the traditional method, a large amount of face data is often collected, and the fatigue degree of the face and the corresponding eye height-to-width ratio in various graphics are marked to statistically calculate within which threshold range the probability of human eye fatigue is the highest. When there is a deviation between the data set used for statistics and the target individual data to be detected, the accuracy of fatigue judgment is greatly reduced. Therefore, the design of the present invention is based on the Gaussian weak supervised learning method. According to the changes in the eye features of individual drivers, a Gaussian weak supervised learning classifier is designed to flexibly and dynamically formulate the eye closure threshold, so as to improve the accuracy of fatigue monitoring and reduce the dependence of the model algorithm on the amount of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 : Schematic diagram of the eye opening and closing ratio.

[0036] Figure 2 .The eye opening degree discrete curve is transformed into a Gaussian distribution curve.

[0037] Figure 3 . Schematic diagram of the process of the present invention.

[0038] Figure 4 ,Confusion matrix of eye opening and closing monitoring: (a) Gaussian clustering dynamic calibration; (b) fixed threshold.

[0039] Figure 5 .Real car recording pictures. DETAILED DESCRIPTION

[0040] A driver fatigue monitoring method based on Gaussian distribution weak supervision learning of the present invention comprises the following steps:

[0041] The in-vehicle camera is used to capture the driver's facial area image and the RGB image is transmitted to the in-vehicle controller;

[0042] In the vehicle controller, the driver's facial area image is first scaled and input into the target detection model in the controller frame by frame.

[0043] The target detection model adopts the FasterRCNN model. The loss function calculation formula of the target detection model is shown in formula (1), where L cls is the classification loss function, L bbox is the bounding box regression loss function, L is the total loss, and λ is a hyperparameter used to balance the weight of classification loss and regression loss, which is set to 0.5 in the present invention;

[0044] L=L cls +λL bbox Formula (1)

[0045] The classification loss function uses the cross entropy loss, as shown in formula (2), where N is the number of positive samples, y i is the true label (0 or 1), p i is the probability predicted by the model.

[0046]

[0047] The bounding box regression loss function is adopted as formula (3) and formula (4), where b i is the predicted bounding box of the i-th image, is the true bounding box of the i-th image. The dimensions of each bounding box are the horizontal and vertical coordinates of the upper left corner and the lower right corner, and N is the number of positive samples.

[0048]

[0049] The backbone of the FasterRCNN model is finally connected to the fully connected layer to output the predicted bounding box: the horizontal and vertical coordinates of the upper left and lower right corners, the category label, and the confidence level.

[0050] The confidence is obtained by the Softmax function, the formula is shown in formula (5), Z i is the output value of the i-th node, σ(z i ) is the confidence of category i, is an exponential function that ensures that the output is a positive number. is the sum of the exponential functions of all categories and is used for normalization.

[0051]

[0052] Only two categories, face and non-face, are trained in this model. When the confidence level is greater than 0.8, the category is face. The corresponding face frame position coordinates are output, and the image of the area is cropped for the next model step.

[0053] When the confidence level is less than 0.8, for example, when the head rotates too much or the face is blocked, the confidence level is reduced, and the algorithm discards the frame image and does not pass it to the next step.

[0054] The face area images with confidence higher than 0.8 are cropped and scaled, and then sent to the feature point model, which locates the key points of the left and right eyelid contours. Figure 1 shown.

[0055] The feature point model adopts the HRNet model and uses the heat map prediction method to replace the traditional key point position regression method. A heat map is output for each predicted key point (the initial key point is randomly selected by the computer). The heat map loss function is constructed based on the mean square error, and the formula is (6), where Lossheatmap is the heat map loss function, N is the number of key points, H i is the real heat map, For the prediction heatmap:

[0056]

[0057] The heat map is in three dimensions (H, W, K), which represent the height, width and number of sheets (the number of key points) of the heat map. The probability value of each pixel will be output in the heat map. The Softmax function is also used to complete the normalization operation, and the probability value is controlled between 0 and 1. The coordinates of the point with the highest probability are the coordinates of the predicted key points. The key point model will output a total of 8 point coordinates for the left and right eyes, including the upper, lower, left, and right sides of the left eyelid, and the upper, lower, left, and right sides of the right eyelid.

[0058] Calculate the distance between features, including the left eye aspect ratio Wl / Ll and the right eye aspect ratio Wr / Lr, that is, the left eye opening and closing ratio and the right eye opening and closing ratio.

[0059] The discrete change curve of the left-right eye aspect ratio in a video of the driver in a non-fatigue state is continuously counted in real time. Then the Gaussian normalization distribution calculation is performed on the discrete change curve to obtain the mean and variance of the left-right eye opening and closing ratio, and the opening and closing ratio corresponding to -2σ is set as the left-right eye closing threshold of the driver, as follows: Figure 2 shown.

[0060] The driver is required to drive the vehicle normally without fatigue, and then start to count the Gaussian distribution of the left and right eye opening and closing ratios. Each statistical cycle is 30 seconds, and 10 rounds of statistics are conducted.

[0061] During the statistical period, the driver is required to be in a fatigue-free state and will only close his eyes when blinking, so closing his eyes is a low-probability event. In addition, in the Gaussian distribution, within the standard deviations of 1σ, 2σ, and 3σ, the probability density of the event is 68%, 95%, and 99.7%, respectively. Values ​​less than -2σ can be regarded as low-probability events. Therefore, the opening and closing ratio corresponding to -2σ is set as the eye-closing threshold of the driver. This method can effectively replace the setting method based on a fixed eye-closing threshold, so it can perform refined recognition based on the driver's personal facial features (compared to the current algorithm).

[0062] After the eye closing threshold is set, the value is saved in the vehicle computer system. Then the eye aspect ratio input in real time is compared with the threshold. When the aspect ratio of any eye is greater than the threshold, the driver is judged to have opened his eyes; when the aspect ratios of both eyes are less than the threshold, the driver is judged to have closed his eyes.

[0063] Finally, the ratio of closed eye frames to total frames in n consecutive frames (n can generally be set to 60, 80, 100) is counted to get the driver's fatigue level. When the ratio of closed eye frames of both eyes reaches 30%, 50%, and 80%, it is set as mild, moderate, and severe fatigue (both eyes must be closed at the same time, one eye may be affected by the shooting angle, so both eyes need to be closed to reduce misjudgment).

[0064] In order to verify the effect of using Gaussian distribution weak supervision learning algorithm, the present invention designs experiments. Different people have different eye sizes, and the difference in eye size is the key, so the eyes of different size areas should be classified, and these interval classes should be compared respectively. That is, whether the model can meet the open and closed eye detection of people with small eyes and large eyes. In the experiment, 32 volunteers were invited to participate in the input. The eye sizes of the volunteers must be significantly different. These videos were divided by the video frame method, and 29420 available images with clear pictures and no residual images were screened. Among them, 80% of the pictures entered by each driver were used as the Gaussian calibration stage, and 20% were used for testing. In the test set, each picture was labeled with closed eyes and open eyes. Table 1 shows the form of the test set data, where TS is the closed eye threshold range and the corresponding number is the amount of data for open eyes and closed eyes.

[0065] Table 1 Test set data format

[0066]

[0067] Through this experiment, Figure 4 To compare the confusion matrices for determining the eye opening and closing state using the test set, sub-image (a) uses the weakly supervised learning algorithm with Gaussian distribution, and sub-image (b) uses the traditional fixed threshold method with the threshold set to 0.2. Table 2 shows the comparison of the accuracy and recall rate of eye opening and eye closing state recognition.

[0068] Table 2 Comparison of eye opening and closing state recognition performance (for machine learning accuracy and recall formulas, see https: / / cloud.tencent.com / developer / article / 2084688)

[0069]

[0070] The following conclusions were drawn from the analysis of experimental data: (1) For drivers whose average eye opening degree is less than 0.16, the detection accuracy of the fixed threshold fatigue detection algorithm drops sharply, and this method can easily misidentify the open eyes as closed eyes. (2) When the calibration program is not turned on, the average detection accuracy is 85.73% and the average recall is 81.55%; after adding the Gaussian clustering calibration method, the average detection accuracy is 93.25% and the average recall is 91.83%. In comparison, the accuracy has increased by 7.52% and the recall has increased by 10.28%, indicating that the weakly supervised learning method based on Gaussian distribution has obvious effects.

[0071] The method of the present invention can monitor the driver's fatigue state in real time and give early warning prompts. The experimental vehicle bench used in the present invention includes a driver camera, a display, a control keyboard, a controller and a wiring harness, wherein the controller is based on NVIDIA XAVIER and NXP's MPC57XX series, adopts a heterogeneous mode of GPU+CPU+MCU, and has a computing power of 32TOPS.

[0072] Table 3 Comparison of fatigue detection based on Gaussian weakly supervised learning algorithm (see https: / / cloud.tencent.com / developer / article / 2084688 for the machine learning accuracy and recall formula)

[0073]

[0074]

[0075] Figure 5 The pictures of the test personnel are taken in real scenes. Table 3 shows the comparison of the accuracy and recall rate of fatigue degree. It can be seen that in practical applications, this method not only improves the accuracy and recall rate, but also has no obvious reduction in running speed, and can still meet the basic requirements of real-time monitoring.

Claims

1. A driver fatigue monitoring method based on Gaussian distribution weak supervision learning, characterized in that: The steps include: Step 1: Use the target detection model to obtain the driver's eye area image; Step 2: Obtain the left eye aspect ratio Wl / Ll and the right eye aspect ratio Wr / Lr, that is, the left eye opening and closing ratio and the right eye opening and closing ratio, from the facial feature point model; Step 3: Count the discrete change curve of the left-right eye aspect ratio in the video of the driver in a non-fatigue state in real time and continuously; then perform Gaussian normalization distribution calculation on the discrete change curve to obtain the mean and variance of the left-right eye opening and closing ratio, and set the opening and closing ratio corresponding to -2σ as the eye closing threshold of the driver; Step 4: After the eye-closing threshold is set, the value is saved in the vehicle computer system, and then compared with the threshold according to the subsequent real-time input eye aspect ratio. When the aspect ratio of any eye is greater than the threshold, it is judged that the driver has his eyes open; when the aspect ratios of both eyes are less than the threshold, it is judged that the driver has his eyes closed; Step 5: Count the ratio of the number of closed-eye frames to the total number of frames in n consecutive image frames, thereby obtaining the driver's fatigue level.

2. The driver fatigue monitoring method according to claim 1, characterized in that: In step five, when the proportion of closed-eye frames reaches 30%, 50%, and 80%, respectively, it is set as mild, moderate, and severe fatigue, respectively.

3. The driver fatigue monitoring method according to claim 1, characterized in that: The above step 1 includes the following steps: 1) Use the in-vehicle camera to capture the driver's facial area image and transmit the RGB image to the vehicle controller; 2) In the vehicle controller, the driver's facial region image is first scaled and input into the target detection model in the controller frame by frame. The target detection model adopts the FasterRCNN model. The loss function calculation formula of the target detection model is shown in formula (1), where L cls is the classification loss function, L bbox is the bounding box regression loss function, L is the total loss, and λ is a hyperparameter used to balance the weight of classification loss and regression loss, which is set to 0.5; L = L cls + λL bbox Equation (1) The classification loss function uses the cross entropy loss, as shown in formula (2), where N is the number of positive samples, y i is the true label, p i is the probability predicted by the model; The bounding box regression loss function uses formula (3) and formula (4), where b i is the predicted bounding box of the i-th image, b i * is the true bounding box of the i-th image. The dimensions of each bounding box are the horizontal and vertical coordinates of the upper left corner and the lower right corner, and N is the number of positive samples; The backbone of the Faster RCNN model is finally connected to the fully connected layer to output the predicted bounding box: the horizontal and vertical coordinates of the upper left and lower right corners, the category label, and the confidence level. The confidence is obtained by the Softmax function, the formula is shown in formula (5), Z i is the output value of the i-th node, σ(z i ) is the confidence of category i, is an exponential function that ensures that the output is positive. is the sum of exponential functions of all categories, used for normalization: Only two categories, face and non-face, are trained in this model. When the confidence is greater than 0.8, the category is face, the corresponding face frame position coordinates are output, and the image of the area is cropped for the next model step; when the confidence is less than 0.8, the algorithm discards the frame image and does not pass it to the next step.

4. The driver fatigue monitoring method according to claim 1, characterized in that: Step 2 includes the following steps: 1) After cropping and scaling, the face region image with a confidence level higher than 0.8 is sent to the feature point model, and the feature point model locates the key points of the left and right eyelid contours; The feature point model uses the HRNet model and outputs a heat map for each predicted key point. The heat map loss function is constructed based on the mean square error, and the formula is (6), where Loss heatmap is the heat map loss function, N is the number of key points, H i is the real heat map, For the prediction heatmap: The form of the heat map is three-dimensional (H, W, K), which represent the height, width and number of sheets of the heat map respectively. The probability value of each pixel point will be output in the heat map. The Softmax function is also used to complete the normalization operation to control the probability value between 0 and 1. The coordinates of the point with the highest probability are taken as the coordinates of the predicted key points. The key point model will output a total of 8 point coordinates for the left and right eyes, including the upper, lower, left and right of the left eyelid, and the upper, lower, left and right of the right eyelid. Calculate the distance between features, including the left eye aspect ratio Wl / Ll and the right eye aspect ratio Wr / Lr.

5. The driver fatigue monitoring method according to claim 1, characterized in that: In step three, the driver is required to drive the vehicle without fatigue, and the Gaussian distribution of the left and right eye opening and closing ratios is counted. Each statistical cycle is 30 seconds, and 10 rounds are counted.

6. The driver fatigue monitoring method according to claim 1, characterized in that: In step 5, the ratio of the number of closed-eye frames to the total number of frames in n consecutive image frames is counted, where n is 60-100.

Images