A multi-method fusion fatigue driving detection method

Abstract

The application provides a multi-method fusion fatigue driving detection method, which fuses a feature extraction method of key point positioning and a feature extraction method based on MobileNetV3_small. Firstly, in order to improve the accuracy of the feature extraction based on the key point positioning method, a personalized threshold is used instead of a traditional average threshold. In order to solve the problem of fewer types and quantities of fatigue driving detection public data sets, when training the eye and mouth state classifier, the method of transfer learning is used to make the model quickly converge and achieve a higher accuracy. Finally, a multi-feature fusion method is used to detect the fatigue driving of the driver through the method of multi-parameter joint discrimination. Compared with the feature extraction method based on key point positioning or the feature extraction method based on MobileNetV3_small, the F1 score is increased by 23.12% and 2.56%, respectively. The final fatigue detection F1 score reaches 92.21%, and the FPS is 29. The method has the advantages of light weight, high accuracy and good real-time performance, and is convenient to use and promote.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of fatigue driving detection method, and particularly relates to a fatigue driving detection method based on multi-method fusion. BACKGROUND

[0002] At present, traffic accidents have become one of the main factors threatening life safety. Lack of road safety driving awareness, drunk driving and fatigue driving are the main factors endangering traffic safety. Among them, traffic accidents caused by fatigue driving account for 14%-20%, and other traffic accidents caused by improper operation and carelessness are also likely to be related to fatigue driving to some extent. The existing methods for detecting fatigue driving of drivers can be divided into three categories: detection methods based on physiological characteristics of drivers, detection methods based on driving information of vehicles of drivers and detection methods based on facial features of drivers. Among them, the detection method based on physiological characteristics of drivers generally needs hardware devices, and the use method is contact type, which is not friendly to drivers; the detection method based on driving information of vehicles of drivers is greatly related to road conditions, traffic conditions and driving habits of drivers, and the detection result of the method will have a large error; the detection method based on facial features of drivers has become a research hotspot due to its advantages such as friendliness to drivers, high accuracy and strong practicability. The key step of the detection method based on facial features of drivers is the extraction of fatigue features of the faces of drivers, and the extraction method of fatigue features of the faces of drivers can be divided into three categories, which are feature extraction method based on appearance, feature extraction method based on artificial and feature extraction method based on deep learning.

[0003] Firstly, most researchers only use one of the three types of feature extraction methods, i.e. feature extraction method based on appearance, feature extraction method based on artificial and feature extraction method based on deep learning, when extracting facial features of drivers. Since fatigue driving detection needs to consider real-time performance and accuracy, according to the advantages and disadvantages of the three types of feature extraction methods, a multi-method fusion method can be considered for extracting fatigue features of the faces of drivers. In addition, a large number of studies have shown that the accuracy of the fatigue feature detection method based on deep learning is better than that of the other two methods. The feature extraction method based on appearance, i.e. the method based on key point positioning, is to extract fatigue features through formula calculation, so the detection speed of this method is much better than that of the feature extraction method based on artificial and the feature extraction method based on deep learning. Secondly, when the driver is fatigue driving, the eyes and mouth are the most direct reflection of the fatigue state of the driver. However, many researchers only consider one of the two states, but in actual application, when one feature detection is blocked, the universality of the fatigue driving detection method will be reduced, and the accuracy of the fatigue driving detection will be affected.

[0004] In addition, when using the feature extraction method based on key point detection, most of them do not consider the problem that the driver's eyes and mouth have different sizes. When setting the threshold value to distinguish between open eyes (open mouth) and closed eyes (closed mouth), the average threshold value is mostly used. But the same threshold value will produce different detection results for different people. If the average threshold value is used, different drivers will produce different detection results, which will affect the detection of the driver's fatigue driving. Therefore, using the average threshold value cannot solve the problem of individual differences of drivers. Although some scholars propose that a classification library can be trained for each driver to solve this problem. But this method requires higher storage capacity of the computer in practical application, and if a new driver is detected before each driving, a new model needs to be trained for a period of time, which is not friendly to the driver.

[0005] Based on this, a multi-method fusion fatigue driving detection method is proposed. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a multi-method fusion fatigue driving detection method to solve the problems in the background art.

[0007] To solve the above technical problems, the technical scheme adopted by the present application is: a multi-method fusion fatigue driving detection method, comprising the following steps:

[0008] First, the face detection and key point positioning part is performed to realize driver face detection on each frame of image in the video, and face region labeling and face key point positioning are performed when the face is detected;

[0009] Then, feature extraction is performed, specifically, the pre-trained classifier is used to extract the driver's eye and mouth features of each frame of image in the video; on the basis of face key point positioning, the driver's eye and mouth features are extracted by calculating the eyelid aspect ratio and mouth aspect ratio, and the results of the two feature extraction methods are fused as the final driver's eye and mouth feature extraction results;

[0010] Finally, fatigue driving detection is performed, on the basis of feature extraction, the number of slow blinking, PERCLOS value and number of yawning of the driver in a certain period of time are counted to detect the driver's fatigue driving, and the detection is completed.

[0011] Further, the HOG+SVM method is used for face detection of the driver, and then the ERT algorithm is used for face key point positioning and labeling based on the detected face. Specifically, the HOG features in the picture are extracted, and then the extracted HOG features are sent to the SVM classifier for face detection. If there is a face region in the picture, the face position information is recorded, and the face frame is labeled according to the face position information, and the ERT algorithm is used for face key point labeling.

[0012] Further, the main steps of extracting the image HOG features are as follows:

[0013] First, the target image is grayed, and then in order to adjust the contrast of the image and reduce the influence of local image shadow and light change, the Gamma correction method is used for color and space normalization of the image. In order to capture the image edge and contour information and further weaken the interference of light, the gradient of each pixel point in the image is calculated.

[0014] Then, the image is divided into multiple small cells, the gradient histogram of each cell is counted, and the features of each cell are obtained. Several cells form a block, and the features of the cells in a block are concatenated to obtain the HOG features of a block.

[0015] Then, the HOG features of all blocks in the image are concatenated to obtain the HOG features of the target image, and then the obtained HOG features are used as the final feature vector for classification and sent to the SVM classifier for face detection.

[0016] When the face is detected, the face information is recorded immediately to realize face frame labeling, and the ERT algorithm is used for face key point positioning.

[0017] Further, the pre-trained classifier is MobileNetV3_small eye and mouth state classifier.

[0018] Further, based on the face key point positioning, the eye and mouth features of the driver are extracted by calculating the eye and mouth aspect ratios. The probability is used to judge the state of the driver's eyes and mouth. The eye and mouth aspect ratios EAR and MAR of the driver are calculated by obtaining the pictures of the driver's open eyes or mouth, closed eyes or mouth, and the specific formula is as follows:

[0019]

[0020]

[0021] Where: E iand M i , i∈[1,6] is the eye and mouth coordinates;

[0022] After obtaining the EAR and MAR when the eyes are open or yawning and the eyes are closed or the mouth is closed, the EAR or MAR when the eyes or mouth is half closed is calculated as the personalized threshold of the current driver, and the calculation method is as shown in the following formula:

[0023]

[0024]

[0025] When the eyes are closed and the mouth is half open, it is considered that there is a 50% chance of being an eye-closed and yawning frame, and as the blinking and yawning actions proceed, the greater the degree of eye closure and mouth opening, the greater the probability of being an eye-closed and yawning frame at this time, and when the eyes are completely closed and the mouth is completely open, it is considered that the probability of being an eye-closed and yawning frame is 100%;

[0026] The eye-closed probability calculation formula is as shown in the following formula:

[0027]

[0028]

[0029] P iFormEye is the probability of the ith frame being an eye-closed frame, PC eye is the threshold change amount when the eye-closed probability increases by 10% each time;

[0030] The yawning probability calculation formula is as shown in the following formula:

[0031]

[0032]

[0033] P iFormMouth is the probability of the ith frame being a yawning frame, PC mouth is the threshold change amount when the yawning probability increases by 10% each time.

[0034] Further, the result fusion is to use a multi-method fusion feature extraction model to jointly extract the driver's eye and mouth features, and when a multi-method fusion method is used to judge the eye and mouth state, the probability calculation method when the ith frame is an eye-closed frame and a yawning frame is as follows:

[0035] P ieye = λ1×P iFormEye + λ2×P iCnnEye (9)

[0036] P imouth = λ1×PiFormMouth + λ2×P iCnnMouth (10)

[0037] Slow blinking, yawn state probability calculation formula as follows:

[0038] P eye = P 1eye + P 2eye +…+ P i-1eye + P ieye (11)

[0039] P mouth = P 1mouth + P 2mouth +…+ P i-1mouth + P imouth (12)

[0040] λ1 and λ2 in formula (9) and formula (10) are 0.5; when P ieye , P imouth probability is greater than 0.5, consider that the frame is closed eyes, yawn frame; at this time, slow blinking probability P eye and yawn probability P mouth add up, as shown in formula (11) and formula (12); when the video reads to the last or a frame probability is less than or equal to 0.5, if i is greater than the pre-set threshold and P eye , P mouth is greater than 0.8, consider that the driver has a slow blinking, yawn action, finally, the model saves the slow blinking, yawn times and closed eye frame number, facilitate the next step of fatigue driving detection.

[0041] Further, using multi-feature fusion method, through the method of multi-parameter common discrimination to detect fatigue driving of the driver;

[0042] The percentage of eyelid closure covering the pupil area in a certain period of time is the most authoritative fatigue discrimination parameter, which means the time occupied by the eye closure in a certain proportion in unit time, the calculation formula is as follows:

[0043]

[0044] Where N CloseFrame is the number of closed eye frames in unit time, N TotalFrame is the total number of frames in unit time.

[0045] The percentage of blinking times in unit time can also reflect the fatigue state of the driver, and the formula of this fatigue parameter is as follows.

[0046]

[0047] Where NEyeBlink is the number of yawns, and T is the total time.

[0048] Similarly, the number of yawns per unit time is taken as the mouth state fatigue parameter, and the mouth state fatigue parameter formula is as follows.

[0049]

[0050] where N Yawn is the number of yawns, and T is the total time.

[0051] Compared with the prior art, the present application has the following advantages:

[0052] The present application fuses the feature extraction method of key point positioning and the feature extraction method based on MobileNetV3_small. Firstly, in order to improve the accuracy of the key point positioning method in feature extraction, the personalized threshold is used instead of the traditional average threshold. Compared with the traditional average threshold method, the F1 score of the method in detecting the driver's eye and mouth features is increased by 9.63% and 8.81% respectively. Secondly, in order to solve the problem of fewer types and quantities of public data sets in fatigue driving detection, the method of transfer learning is used when training the eye and mouth state classifier, so that the model can quickly converge and achieve high accuracy. Finally, in order to solve the universality and accuracy of the fatigue driving detection method when a single facial feature detection is blocked in practical application, a multi-feature fusion method is used to detect the fatigue driving of the driver through the method of multi-parameter joint discrimination. Compared with the feature extraction method based on key point positioning or MobileNetV3_small, the F1 score of the multi-method fusion feature extraction method is increased by 23.12% and 2.56% respectively; the final fatigue detection F1 score reaches 92.21%, and the FPS is 29. The method proposed in this paper has the advantages of light weight, high accuracy and good real-time performance, and is easy to popularize and use. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is the overall method flowchart of the present application;

[0054] Figure 2 is the face detection and key point positioning flowchart in the embodiment of the present application;

[0055] Figure 3 is the HOG feature extraction flowchart in the embodiment of the present application;

[0056] Figure 4 is the MobileNetV3_small two-class network structure diagram in the embodiment of the present application;

[0057] Figure 5is a bneck structure diagram in an embodiment of the present application;

[0058] Figure 6 is a channel separable convolution structure diagram in an embodiment of the present application;

[0059] Figure 7 is an SE module structure diagram in an embodiment of the present application;

[0060] Figure 8 is an example diagram of a data set for training an eye state classifier in an embodiment of the present application;

[0061] Figure 9 is an example diagram of a data set for training a mouth state classifier in an embodiment of the present application;

[0062] Figure 10 is an accuracy and loss value curve diagram under a training set and a validation set in an embodiment of the present application;

[0063] Figure 11 is a feature extraction method flowchart based on key point positioning in an embodiment of the present application;

[0064] Figure 12 is an eye aspect ratio solving schematic diagram in an embodiment of the present application;

[0065] Figure 13 is a mouth aspect ratio solving schematic diagram in an embodiment of the present application;

[0066] Figure 14 is an eye state judging schematic diagram based on multi-method fusion in an embodiment of the present application;

[0067] Figure 15 is a mouth state judging schematic diagram based on multi-method fusion in an embodiment of the present application;

[0068] Figure 16 is a fatigue driving detection flowchart in an embodiment of the present application;

[0069] Figure 17 is a comparison experiment result diagram of four scenes under individualized threshold slow blinking detection in an experimental example of the present application;

[0070] Figure 18 is a comparison experiment result diagram of four scenes under multi-method fusion slow blinking detection in an experimental example of the present application;

[0071] Figure 19 is a comparison experiment result diagram of four scenes under multi-method fusion yawning detection in an experimental example of the present application;

[0072] Figure 20 is a comparison experiment result diagram of four scenes under multi-method fusion slow blinking detection in an experimental example of the present application;

[0073] Figure 21 is a visualization diagram of the fatigue driving detection experiment in the experimental example of the present application. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0075] As shown in Figure 1 , the present application provides a technical solution: a multi-method fusion fatigue driving detection method, comprising the following steps:

[0076] First, the face detection and key point positioning part is performed to realize the driver face detection on each frame of image in the video, and the face region labeling and face key point positioning are performed when the face is detected;

[0077] Then, the feature extraction is performed, specifically, the pre-trained classifier is used to extract the driver eye and mouth features of each frame of image in the video; on the basis of the face key point positioning, the driver eye and mouth features are extracted by calculating the eyelid aspect ratio and mouth aspect ratio, and the results of the two feature extraction methods are fused as the final driver eye and mouth feature extraction results;

[0078] Finally, the fatigue driving detection is performed, on the basis of the feature extraction, the driver slow blinking times, PERCLOS value, and yawning times in a time period are counted to detect the driver fatigue driving, and the detection is completed.

[0079] When the face detection is performed, the HOG+SVM method is used to detect the driver face, and then on the basis of the detected face, the ERT algorithm is used to position and label the face key points, and the face detection and key point positioning process is as shown in Figure 2 ;

[0080] The HOG feature in the picture is first extracted, and then the extracted HOG feature is sent to the SVM classifier for face detection. If there is a face region in the picture, the face position information is recorded, the face frame is labeled according to the face position information, and the ERT algorithm is used to label the face key points.

[0081] The HOG feature extraction process is as shown in Figure 3 , and the main steps of the image HOG feature extraction are as follows:

[0082] First, the target image is grayed, and then in order to adjust the contrast of the image, reduce the influence of local image shadow and light change, the image is normalized in color and space by using Gamma correction method, in order to capture the image edge and contour information and further weaken the interference of light, the gradient of each pixel point in the image is calculated;

[0083] Then the image is divided into a plurality of small cells, the gradient histogram of each cell is counted, the features of each cell are obtained, several cells are combined into a block, and the features of the cells in a block are concatenated to obtain the HOG features of the block;

[0084] Then the HOG features of all blocks in the image are concatenated to obtain the HOG features of the target image, and then the obtained HOG features are sent to the SVM classifier as the final feature vector for classification to perform face detection;

[0085] When the face is detected, the face information is recorded immediately, the face frame is labeled, and the ERT algorithm is used to locate the face key points.

[0086] The pre-trained classifier is an eye and mouth state classifier trained by MobileNetV3_small. MobileNetV3_small is a lightweight network with small parameters, short inference time and suitable for mobile embedded devices and other edge computing devices. Since the real-time requirement of fatigue driving detection is high, the eye and mouth state classifier is trained by MobileNetV3_small, and the network structure used is as shown in Figure 4 .

[0087] Among them, bneck is the basic module of the network, which is also the core module, which mainly realizes channel separable convolution, SE channel attention mechanism and residual link, and the structure is as shown in Figure 5 .

[0088] The channel separable convolution in the bneck module is the main factor to realize the lightweight of the model. When the standard convolution is used for convolution operation, each convolution kernel is convolved with all channels of the picture, while the channel separable convolution is to split a standard convolution into depth convolution and pointwise convolution. As shown in Figure 6 .

[0089] The depth convolution corresponds to a convolution kernel, and the pointwise convolution is a normal convolution with a 1*1 convolution kernel, which is used to realize the fusion of features of different channels and the dimensionality reduction and dimensionality increase in the channel direction. The operation amount of channel separable convolution is much smaller than that of standard convolution.

[0090] In the bneck module, the SE channel attention mechanism is used, the importance of each channel of the feature map is obtained, and the weight is assigned, so that the neural network pays more attention to the channel with high weight, so that the channel with greater feature extraction effect is paid attention to, and the performance of network feature extraction is improved. SENet, Squeeze-and-Excittion Networks is divided into compression and excitation two parts, the compression is mainly to compress the global spatial information, then learn in the channel dimension and get the importance of each channel. The excitation part is mainly to allocate weights to each channel, and the SE module structure is as shown in Figure 7

[0091] In the SENet network, the compression part mainly compresses the picture with the dimension of HxWxC to 1x1xC through global average pooling. In the excitation part, first, the 1x1xC dimension obtained by compression is integrated into the full connection layer to predict the importance of each channel, and then the original image is multiplied by the weight matrix and excited to the corresponding channel of the original feature map for operation.

[0092] Since there are few public real fatigue driving data sets in the field of fatigue driving detection at present, in order to improve the learning efficiency of the model, the network weight parameters of MobileNetV2 are migrated to MobileNetV3_small network through the method of transfer learning when training the eye and mouth state classification model, and then the model is trained using NTHU-DDD data set. 22 different slow blinking videos and yawning videos are used to design a frame taking algorithm, and the label file corresponding to each video is taken every 10 frames. If it is a closed eye frame, the frame picture is stored in the closed eye folder, if it is a normal frame, it is stored in the eye normal driving folder. Similarly, when processing the yawning data set, if it is a yawning frame, the frame picture is stored in the yawning folder, if it is a normal frame, it is stored in the mouth normal driving folder. Then artificial screening is carried out to remove pictures that do not meet the requirements. The number of data sets used is shown in Table 1 and Table 2.

[0093] Table 1 Eye state classifier data set composition

[0094]

[0095] Table 2 Mouth state classifier data set composition

[0096]

[0097] The used data set examples are shown in Figure 8 and Figure 9

[0098] ​​It is found through experiments that the model can quickly converge and has a high accuracy rate when the batch_size is 32, the epoches is 50 and the learning rate lr is 0.0001, as shown in Figure 10

[0099] It can be seen from Figure 10 that the model can quickly converge and has a high accuracy rate after using the transfer learning. The accuracy rate of eye state classification is 95.2%, and the accuracy rate of mouth state classification is 97.6%. The detection accuracy is high, and the feature extraction method based on key point positioning can be used together to extract the features of the driver's eyes and mouth.

[0100] At present, in the research of fatigue driving detection, when using the method based on key point positioning, most researchers generally use the average threshold method to judge the state of the driver's eyes and mouth. However, the experiments in the first chapter of the present application show that because the driver's eyes and mouth have different sizes, the corresponding threshold values are also different, as shown in Tables 3 and 4.

[0101] Table 4 Blinking threshold values of different drivers

[0102]

[0103] Table 4 Yawning threshold values of different drivers

[0104]

[0105] As can be seen from Tables 3 and 4, different drivers have different threshold values. Therefore, if the average threshold value is used, it will lead to misjudgment of the state of the driver's eyes and mouth, and thus affect the accuracy of fatigue driving detection. Therefore, when using the key point positioning method, the present application proposes a personalized threshold to solve the problem of driver personalization, and designs a probability formula to abandon the traditional 0 and 1 discrimination method and uses probability to judge the state of the driver's eyes and mouth. The method flow is as shown in Figure 11

[0106] On the basis of face key point positioning, the driver's eye and mouth features are extracted by calculating the eyelid aspect ratio and mouth aspect ratio. The state of the driver's eyes and mouth is judged by using probability. The eye aspect ratio EAR and the mouth aspect ratio MAR of the driver who is driving are calculated by obtaining the picture of the driver's open eyes or mouth, closed eyes or mouth. The eye aspect ratio solving schematic diagram is as shown in Figure 12 The mouth aspect ratio solving schematic diagram is as shown in Figure 13

[0107] The calculation formula is as shown in the following formula:

[0108] ​​​

[0109]

[0110] wherein: E i and M i , i∈[1,6] are the eye and mouth coordinates;

[0111] After obtaining the EAR and MAR when the eyes are open or yawning and the eyes are closed or the mouth is closed, the EAR or MAR when the eyes or mouth is half closed is calculated as the personalized threshold of the current driver, and the calculation method is as shown in the following formula:

[0112]

[0113]

[0114] When the eyes are half closed and the mouth is fully open, it is considered that there is a 50% probability of being an eye-closed and yawning frame, and as the blinking and yawning actions proceed, the greater the degree of eye closure and mouth opening, the greater the probability of being an eye-closed and yawning frame at this time, until the eyes are fully closed and the mouth is fully open, it is considered that the probability of being an eye-closed and yawning frame is 100%;

[0115] The eye-closed probability calculation formula is as shown in the following formula:

[0116]

[0117]

[0118] P iFormEye is the probability of the i-th frame being an eye-closed frame, PC eye is the threshold change amount when the eye-closed probability increases by 10% each time;

[0119] The yawning probability calculation formula is as shown in the following formula:

[0120]

[0121]

[0122] P iFormMouth is the probability of the i-th frame being a yawning frame, PC mouth is the threshold change amount when the yawning probability increases by 10% each time.

[0123] The feature extraction model based on key point positioning can well solve the problem of driver individualization, but the key point positioning is inaccurate when the driver wears glasses, drives at night and has a large posture, which leads to an unsatisfactory feature extraction result in the case of night driving, driving with glasses and the like. The feature extraction method based on deep learning can obtain more effective features in a complex environment, but the MobilenetV3_small used in the present application meets the requirements of real-time and lightness of fatigue driving, but compared with a model with a deeper network structure, the accuracy of feature extraction is slightly lower. Therefore, the present application uses a multi-method fusion feature extraction model to jointly extract the features of the driver's eyes and mouth. First, the method based on key point positioning not only solves the problem of driver individualization, but also has a faster detection speed. In addition, the MobileNetV3_small model can make up for the shortcomings of the key point positioning method in complex environments and the case of the driver wearing glasses, and will not affect the real-time performance of fatigue driving detection. Therefore, the fusion of the feature extraction model based on key point positioning and the feature extraction model based on MobileNetV3_small can maintain high accuracy while meeting the real-time requirements of fatigue driving detection. The detection process is shown in Figure 14 and Figure 15 .

[0124] The result fusion is to use a multi-method fusion feature extraction model to jointly extract the features of the driver's eyes and mouth. When the multi-method fusion method is used to judge the state of the eyes and mouth, the probability calculation method when the i-th frame is a closed-eye frame and a yawning frame is as follows:

[0125] P ieye =λ1×P iFormEye +λ2×P iCnnEye (9)

[0126] P imouth =λ1×P iFormMouth +λ2×P iCnnMouth (10)

[0127] The probability calculation formula of the slow blinking and yawning state is as follows:

[0128] P eye =P 1eye +P 2eye +…+P i-1eye +P ieye (11)

[0129] P mouth =P 1mouth +P 2mouth +…+P i-1mouth +P imouth (12)

[0130] λ1 and λ2 in formula (9) and formula (10) are 0.5; when P ieye , P imouth When the probability is greater than 0.5, the frame is considered to be a closed eye, yawn frame; at this time, the slow blinking probability P eye and the yawn probability P mouth are accumulated, as shown in formula (11) and formula (12); when the video is read to the end or the probability of a frame is less than or equal to 0.5, if i is greater than a pre-set threshold and P eye , P mouth is greater than 0.8, it is considered that the driver has made a slow blinking, yawning action, finally, the model saves the number of slow blinking, yawning and the number of closed eye frames, which is convenient for the next step of fatigue driving detection.

[0131] When the driver is fatigue driving, the eyes and mouth are the most direct reflection of the driving state of the driver. In order to solve the universality and accuracy of the fatigue driving detection method when a single face feature detection is blocked in practical application, a multi-feature fusion method is used to detect the fatigue driving of the driver through a multi-parameter joint discrimination method, as shown in the flow chart of Figure 16 , a multi-feature fusion method is used to detect the fatigue driving of the driver through a multi-parameter joint discrimination method;

[0132] The percentage of the area of the eyelid covering the pupil in a certain period of time is the most authoritative fatigue discrimination parameter, which means the time occupied by the closing of the eyes in a unit of time, and the calculation formula is as follows:

[0133]

[0134] Where N CloseFrame is the number of closed eye frames in a unit of time, and N TotalFrame is the total number of frames in a unit of time.

[0135] The percentage of the number of blinks in a unit of time can also reflect the fatigue state of the driver, and the formula of this fatigue parameter is as follows.

[0136]

[0137] Where N EyeBlink is the number of blinks, and T is the total time;

[0138] Similarly, the number of yawns in a unit of time is taken as the fatigue parameter of the mouth state, and the formula of the fatigue parameter of the mouth state is as follows.

[0139]

[0140] Where N Yawnwhere y is the number of yawns, T is the total time.

[0141] In the experiment, the NTHU-DDD fatigue video dataset was used, which was collected from 36 different male and female drivers, with and without glasses, simulating five driving scenarios: normal driving, yawning, slow blinking, falling asleep, and laughing. The videos were recorded under daytime and nighttime lighting conditions, with the subjects sitting in a chair simulating driving. The total duration of the entire dataset was nine and a half hours. Each person was recorded in four videos, with about one minute of video recorded for yawning and slow blinking; the combination of fatigue-related actions, yawning, nodding, and slow blinking, and the combination of non-fatigue-related actions, speaking, smiling, and looking left and right, each for about five minutes. Each video corresponds to 4 label files, which record the state of each frame of picture. The label file content and the state of the driver are shown in Table 5.

[0142] Table 5 NTHU-DDD dataset label value represents state

[0143]

[0144] In the experiment, the NTHU-DDD fatigue video dataset was used, which was collected from 36 different male and female drivers, with and without glasses, simulating five driving scenarios: normal driving, yawning, slow blinking, falling asleep, and laughing. The videos were recorded under daytime and nighttime lighting conditions, with the subjects sitting in a chair simulating driving. The total duration of the entire dataset was nine and a half hours. Each person was recorded in four videos, with about one minute of video recorded for yawning and slow blinking; the combination of fatigue-related actions, yawning, nodding, and slow blinking, and the combination of non-fatigue-related actions, speaking, smiling, and looking left and right, each for about five minutes. Each video corresponds to 4 label files, which record the state of each frame of picture. The label file content and the state of the driver are shown in Table 5.

[0145] In the slow blinking detection and yawning detection experiment, a total of 420 slow blinking video datasets and 174 yawning datasets were collected. The slow blinking detection dataset had a video every 5 seconds, of which 177 videos were slow blinking in a fatigue state, and the rest were normal blinking and non-blinking videos; the yawning detection dataset had a video every 10 seconds, of which 55 were yawning videos in a fatigue state, and the rest were normal driving videos, including not speaking, speaking, and laughing, etc.

[0146] In the fatigue driving detection experiment, a total of 164 video datasets were collected, each of which was cut to 30 seconds. Among them, there were 74 fatigue driving videos, each of which at least had one of the behaviors of slow blinking and yawning; there were 90 normal driving videos, each of which at least had one of the behaviors of normal blinking, not speaking, normal speaking, and laughing, etc.

[0147] In the feature extraction experiment, the miss rate, precision, recall, and F1 score were used as evaluation indicators. In the fatigue driving detection experiment, the F1 score was used as the final evaluation indicator.

[0148] Miss Rate (MissRate), which is the proportion of positive class judgment errors, is calculated as follows.

[0149]

[0150] Precision, i.e. the proportion of correct predictions of positive class in all predictions of positive class, is calculated as follows.

[0151]

[0152] Recall, i.e. the proportion of correct predictions of positive class in all actual positive classes, is calculated as follows.

[0153]

[0154] F1 score, i.e. the harmonic mean of precision and recall, is calculated as follows.

[0155]

[0156] Where TP, FP, TN and FN represent the number of correct predictions of positive class, the number of predictions of negative class as positive class, the number of correct predictions of negative class and the number of predictions of positive class as negative class, respectively.

[0157] Experimental design and result analysis:

[0158] Slow blinking detection comparison experiment

[0159] 1) Personalized threshold comparison experiment, by using different threshold methods, record the precision, recall, false detection rate and F1 score of slow blinking detection in each scene, to verify that the personalized threshold is better than the average threshold in slow blinking detection, and the experimental results are shown in Table 5. Figure 17

[0160] It can be seen that in the three scenes of not wearing glasses in the daytime, wearing glasses in the daytime and not wearing glasses at night, the personalized threshold proposed in the application is superior to the commonly used average threshold in the four evaluation indexes of false detection rate, recall rate, precision and F1 score; but in the scene of wearing glasses at night, the personalized threshold is inferior to the average threshold in false detection rate and recall rate, but still superior to the average threshold in precision and F1 score. The main reason is that the method based on key point positioning is accurate in the case of good driving environment and unobstructed face, if in the case of poor night lighting or face obstruction, the key point positioning will deviate, thus affecting the result of driver eye blinking detection. The change amount of the final average value and the personalized threshold compared with the average threshold in the evaluation index in the four scenes is shown in Table 6.

[0161] Table 6 Change amount of slow blinking of personalized threshold

[0162] ​

[0163] As shown in Table 6, compared with the commonly used average threshold, the false negative rate of the personalized threshold proposed in this experimental example is reduced by 6.02%, and the recall rate, precision rate and F1 score are increased by 6.02%, 11.61% and 9.63% respectively. Therefore, from the experimental results, it can be seen that the personalized threshold proposed in the present application is superior to the commonly used average threshold in detecting slow blinking of the driver.

[0164] 2) Multi-method fusion comparison experiment. In this experiment, the accuracy, recall rate, false negative rate and F1 score of the personalized threshold, MobileNetV3_small and multi-method fusion method in detecting slow blinking in each scene are recorded to verify the superiority of the multi-method fusion method proposed in the present application in detecting slow blinking of the driver, and the experimental results are shown in Table 6. Figure 18

[0165] It can be seen that in the three scenes of daytime without glasses, daytime with glasses and nighttime without glasses, the multi-method fusion method proposed in the present application is superior to the other two methods in the four evaluation indexes of false negative rate, recall rate, accuracy and F1 score. But in the scene of wearing glasses at night, the multi-method fusion method is slightly inferior to the MobileNetV3_small method in accuracy and F1 score, but superior to the personalized threshold method. The main reason is that the face key point positioning is not accurate in the night with shielding based on the personalized threshold method, so the result of detecting slow blinking of the driver based on the personalized threshold method is poor, which leads to the detection result of the multi-method fusion method not being ideal in the night with glasses. On the other hand, compared with the driving environment in good condition, the result of slow blinking detection based on MobileNetV3_small method is also affected in the night or with glasses, but the degree of influence is lower than that based on the personalized threshold method. Although the method based on the personalized threshold has an adverse effect on the multi-method fusion method in the driving environment of wearing glasses at night, it can be seen from the experimental results in (a) daytime without glasses in Table 6 that the method based on the personalized threshold is superior to the MobileNetV3_small method in false negative rate and recall rate, and in the three scenes of daytime without glasses, daytime with glasses and nighttime without glasses, the multi-method fusion method proposed in the present application is superior to the other two single methods in the four evaluation indexes of false negative rate, recall rate, accuracy and F1 score. The final average value in the four scenes and the change rate of the evaluation indexes based on the multi-method fusion method compared with the other two single methods are shown in Table 7. Figure 18

[0166] Table 7 Change amount of slow blinking detection of multi-method fusion ​​

[0167]

[0168] As shown in Table 7, compared with the method based on the personalized threshold, the missed detection rate of slow blinking detection is reduced by 22.74%, and the recall rate, precision rate and F1 score are increased by 22.74%, 22.59% and 22.68% respectively; compared with the method based on MobileNetV3_small, the missed detection rate of slow blinking detection is reduced by 3.03%, and the recall rate, precision rate and F1 score are increased by 3.03%, 2.24% and 2.32% respectively.

[0169] Therefore, from the above experimental results, it can be seen that the method based on multi-method fusion proposed in the present application is superior to a single method in slow blinking detection of drivers.

[0170] Yawning detection experiment

[0171] 1) Personalized threshold comparison experiment. By using different threshold methods, the precision, recall, missed detection rate and F1 score of detecting slow blinking in each scene are recorded to verify that the personalized threshold is superior to the average threshold in slow blinking detection, and the experimental results are shown in Table 7. Figure 19

[0172] As can be seen, in the four scenes of daytime without glasses, daytime with glasses, nighttime without glasses and nighttime with glasses, the personalized threshold proposed in the present application is superior to the commonly used average threshold in the four evaluation indexes of missed detection rate, recall rate, precision rate and F1 score. However, the yawning detection result is poor when driving at night or when the face is covered, which is mainly due to the accurate positioning of facial feature points based on key point positioning in the case of good driving environment and no face covering. The change amount of the final average value and the personalized threshold compared with the average threshold in the evaluation indexes in the four scenes is shown in Table 8.

[0173] Table 8 Change amount of slow blinking of personalized threshold

[0174]

[0175] As shown in Table 8, compared with the commonly used average threshold, the missed detection rate of the personalized threshold proposed in the present application is reduced by 12.3%, and the recall rate, precision rate and F1 score are increased by 12.3%, 4.89% and 8.81% respectively. Therefore, from the experimental results, it can be seen that the personalized threshold proposed in the present application is superior to the commonly used average threshold in yawning detection of drivers.

[0176] ​2) Multi-method fusion comparative experiment. In this experiment, the precision, recall, missed detection rate and F1 score of the personalized threshold, MobileNetV3_small and multi-method fusion methods in detecting yawning in each scene are recorded to verify the superiority of the multi-method fusion method proposed in the application in detecting yawning of the driver. The experimental results are shown in Table 9. Figure 20

[0177] 2It can be seen that in the four scenes of not wearing glasses during the day, wearing glasses during the day, not wearing glasses at night, and wearing glasses at night, the multi-method fusion method proposed in the application is superior to the other two single methods in the four evaluation indexes of missed detection rate, recall rate, precision and F1 score.

[0178] However, compared with the daytime driving environment without face occlusion, the performance of yawning detection in the nighttime driving or face occlusion driving environment is slightly worse. The final average value in the four scenes and the change rate of the evaluation index of the multi-method fusion method compared with the other two single methods are shown in Table 9.

[0179] Table 9 Change amount of multi-method fusion slow blinking detection

[0180]

[0181]

[0182] As shown in Table 9, compared with the method based on personalized threshold, the missed detection rate of yawning detection of the multi-method fusion method proposed in the application is reduced by 26.21%, and the recall rate, precision and F1 score are increased by 26.21%, 20.84% and 23.56%, respectively; compared with the MobileNetV3_small method, the missed detection rate and recall rate of slow blinking detection do not change significantly, but the precision and F1 score are increased by 5.10% and 2.80%, respectively. Therefore, overall, the performance of the multi-method fusion method based on the application in detecting yawning of the driver is superior to the single detection method.

[0183] Fatigue driving detection experiment:

[0184] Real-time and accuracy are two important evaluation indexes for evaluating fatigue driving detection. When the driver is fatigue driving, real-time and accurate warning to the driver is the ultimate goal of fatigue driving detection. Therefore, this section experiment comprehensively evaluates the performance of the method proposed in the application from the aspects of real-time and accuracy. The visualization of the fatigue driving detection experiment is shown in FIG. 9. Figure 21

[0185] ​​Fatigue driving detection time-consuming analysis experiment. It is important for the driver to be able to detect the driving state of the driver in real time and give timely warnings. Therefore, real-time is also an important indicator of fatigue driving detection. This experiment tests 6 videos, and the time-consuming of each module in the detection process is counted, as shown in Table 10.

[0186] Table 10 Analysis of time-consuming of each module of the algorithm

[0187]

[0188] As can be seen from Table 10, the face detection and feature extraction parts consume a long time, among which the face detection module consumes 14.77 ms to process a frame of picture, and the feature extraction module consumes 17.55 ms to process a frame of picture. The total time-consuming of the method is 34.51 ms / f, that is, it can process about 29 frames of pictures per second, which can meet the requirements of fatigue driving detection for real-time.

[0189] Fatigue driving detection accuracy experiment. This experiment verifies the method proposed in the present application on the NTHU-DDD data set, and compares it with recent researches, including a feature extraction method using only key point positioning

[31] and a feature extraction method using deep learning [32-35]

[22] , and the experimental results are shown in Table 11.

[0190] Table 11 Comparison experiment of different methods under the NTHU-DDD data set

[0191]

[0192] As can be seen from Table 11, the F1 score of the multi-method fusion feature extraction model proposed in the present application is increased by 21.77% compared with the key point positioning method

[31] ; the F1 score is increased by 19.61%, 11.21%, 7.85%, 7.77% and 2.79% compared with the deep learning model method [32-35]

[22] , respectively. It can be seen that the fatigue driving detection method proposed in the present application has high accuracy and can meet the requirements of fatigue driving accuracy.

[0193] To sum up: the present application proposes a multi-method fusion fatigue driving detection method, using HOG features and ERT algorithm for face detection and key point positioning, using a multi-method fusion method for feature extraction of driver's eyes and mouth state, that is, using a key point positioning method and a MobileNetV3_small model fusion; when using the key point positioning method, a personalized threshold is designed to replace the traditional average threshold, and a probability formula is designed, compared with the traditional average threshold method, the personalized threshold proposed in the present application increases the F1 score by 9.63% and 8.81% respectively when extracting the driver's eye and mouth state; when training the eye and mouth classification model, through the model migration method, the model converges quickly and achieves high accuracy; when the multi-method fusion finally judges the eye and mouth state, the traditional non-0 judgment fatigue frame method is abandoned, and the probability of this frame being a fatigue frame is used to judge whether it is a fatigue frame, so when judging whether the driver has performed a slow blink or a yawn action, not only is it judged whether the continuous fatigue frame exceeds the pre-set threshold, but also whether the continuous frame probability accumulation exceeds the pre-set probability threshold, the experimental results show that, compared with the single key point positioning method and the single MobileNetV3_small method, the F1 score of the multi-method fusion feature extraction method in detecting the slow blink and yawn state of the driver is increased by 23.12% and 2.56% respectively;

[0194] In the fatigue driving detection stage, the eyes and mouth are comprehensively considered, and multiple features are combined to solve the universality when one feature detection is blocked in actual application and improve the accuracy of fatigue driving detection. Through experimental comparison and analysis, the F1 score of the method proposed in the present application can reach 92.21%, and 29 frames of pictures can be processed per second; in addition, under the premise of lightweight overall model, the method of using personalized threshold improves the practicality and robustness of different drivers in actual application; compared with other fatigue driving detection methods, the method of the present application not only meets the requirements of accuracy and real-time of fatigue driving detection, but also has the advantages of non-contact, low cost, lightweight and universality, and can be deployed on mobile terminals, which has practical significance for improving traffic safety.

[0195] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and illustrative figures, it should be apparent that the scope of the present application is not limited to these specific embodiments.

[0196] While the embodiments of the application have been shown and described herein, it will be understood by those skilled in the art that many changes, modifications, substitutions and alterations to these embodiments can be made without departing from the principles and spirits of the application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-method fusion fatigue driving detection method, characterized in that: The method comprises the following steps: First, the face detection and key point positioning part is carried out, so as to realize the driver face detection on each frame of image in the video, and the face region labeling and face key point positioning when the face is detected; Then, the feature extraction is carried out, specifically, the driver eye and mouth features of each frame of image in the video are extracted through the pre-trained classifier; on the basis of the face key point positioning, the driver eye and mouth features are extracted through the calculation of the eyelid aspect ratio and mouth aspect ratio, and the results of the two feature extraction methods are fused as the final driver eye and mouth feature extraction results; Finally, the fatigue driving detection is carried out, on the basis of the feature extraction, the driver slow blinking times, PERCLOS value and yawning times in a time period are counted, the driver fatigue driving detection is carried out, and the detection is completed; On the basis of the face key point positioning, the driver eye and mouth features are extracted through the calculation of the eyelid aspect ratio and mouth aspect ratio, the state of the driver's eyes and mouth is judged by using probability, the open-eye or open-mouth, closed-eye or closed-mouth pictures of the driver driving are acquired to calculate the eye aspect ratio EAR and mouth aspect ratio MAR, and the specific calculation is as follows: （1） （2） wherein: and , are eye and mouth coordinates; (1-6) are eye and mouth six key points, After the EAR and MAR in the open-eye or yawning and closed-eye or closed-mouth are obtained, the MobileNetV or MAR when the eye or mouth is closed by half is calculated as the personalized threshold of the current driver, and the calculation mode is as follows: （3） （4） When the eye is closed and the mouth is opened by half, it is considered that there is a 50% possibility of closed-eye and yawning frame, with the blinking and yawning actions, the greater the degree of eye closure and mouth opening, the greater the probability of closed-eye and yawning frame at this time, and when the eye is completely closed and the mouth is completely opened, it is considered that the probability of closed-eye and yawning frame is 100%; The closed-eye probability calculation formula is as follows: （5） （6） P iFormEye For the first The probability that a frame is a closed-eye frame, PC eye is a threshold change amount when the closed-eye probability is raised by ten percent each time; The yawning probability calculation formula is as follows: （8） P iFormMouth For the first The probability of a frame being a yawn frame, PC mouth The threshold change amount when the yawn probability increases by every ten percent; As a result of fusion, the feature extraction model using multi-method fusion is used to extract the driver's eye and mouth features together, and the multi-method fusion method is used to determine the eye and mouth states, and the probability calculation method when the frame is a closed-eye frame or a yawn frame is as follows: The probability calculation method when the frame is a closed-eye frame or a yawn frame is as follows: P ieye = λ1x P iFormEye + λ2x P iCnnEye (9) P imouth = λ1 x P iFormMouth + λ2 x P iCnnMouth (10) The slow blinking and yawning state probability calculation formula is as follows: P eye = P 1eye + P 2eye +... + P i-1eye + P ieye (11) P mouth = P 1mouth + P 2mouth +... + P i-1mouth + P imouth (12) PslowblinkandP and Pslowblink ieye Pslowblink imouth Pslowblink eye Pslowblink mouth Pslowblink Pslowblink Pslowblink mouth Pslowblink 2. The multi-method fused drowsiness detection method of claim 1, wherein, When the face detection is carried out, the HOG+SVM method is used to detect the driver face, then on the basis of the detected face, the ERT algorithm is used for face key point positioning and labeling, specifically, the HOG features in the picture are extracted, then the extracted HOG features are sent into the SVM classifier for face detection, if there is a face region in the picture, the face position information is recorded, the face frame is labeled according to the face position information, and the ERT algorithm is used for face key point labeling.

3. The multi-method fused drowsiness detection method of claim 2, wherein, The main steps of extracting the image HOG features are as follows: Firstly, the target image is subjected to grayscale processing, then in order to adjust the contrast of the image and reduce the influence of local image shadow and light change, the Gamma correction method is adopted to normalize the color and space of the image, and in order to capture the image edge and contour information and further weaken the interference of light, the gradient of each pixel point in the image is calculated; The image is divided into multiple small cells, the gradient histogram of each cell is counted, the features of each cell are obtained, several cells are combined to form a block, and the HOG features of a block are obtained by concatenating the features of the cells in the block; The HOG features of all blocks in the image are concatenated to obtain the HOG features of the target image, and then the obtained HOG features are sent to the SVM classifier as the final feature vector for classification to perform face detection; When a face is detected, the face information is recorded immediately to realize face frame labeling, and the ERT algorithm is used to locate the face key points.

4. The multi-method fused drowsiness detection method of claim 1, wherein, The pre-trained classifier is MobileNetV3_small, which is used to classify the eye and mouth states.

5. The multi-method fused drowsiness detection method of claim 1, wherein, A multi-feature fusion method is used to detect driver fatigue through multi-parameter joint discrimination; The percentage of the area of the eyelid covering the pupil in a certain period of time is the most authoritative fatigue discrimination parameter, which means the time occupied by the eye closing a certain percentage in a unit of time, and the calculation formula is as follows: where N CloseFrame is the number of closed-eye frames per unit time, N TotalFrame is the total number of frames per unit time; The percentage of the number of blinks in a unit of time can also reflect the fatigue state of the driver, and the formula of this fatigue parameter is as follows: （14） wherein is the number of blinks, is the total time; Similarly, the number of yawns in a unit of time is taken as the fatigue parameter of the mouth state, and the formula of the fatigue parameter of the mouth state is as follows: （15） wherein is the number of yawns, is the total time.

Images