Method for detecting and analyzing fatigue of driver of commercial vehicle
By combining a multi-stage trained neural network model with multiple heat maps, the target coordinates and fatigue characteristics of key points are generated, which solves the problem of insufficient accuracy in facial key point detection in existing technologies and achieves high-precision and all-round detection of driver fatigue.
Patent Information
- Application Number
- CN202510956154.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-11
- Publication Date
- 2025-10-17
AI Technical Summary
The existing CNN-based heatmap regression model has problems with theoretical error lower bound in facial key point detection and poor performance on low-resolution images, resulting in insufficient fatigue detection accuracy.
A neural network model with multi-stage training is used to generate the target coordinates of key points through the first, second and third heat maps, and combined with the expression fatigue value to generate the proportion of closed eyes, yawning and expression fatigue to determine the fatigue state.
The accuracy of facial key point detection and fatigue detection has been improved, achieving all-round detection of the driver's fatigue status.
Smart Images
Figure CN120808318A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field related to road traffic safety, and particularly relates to a fatigue detection and analysis method for drivers of commercial vehicles. BACKGROUND
[0002] With the development of urban road construction, road traffic tools have become the main means of transportation in people's lives, and traffic accidents occur frequently. Traffic accidents are usually caused by the operation errors of drivers, and fatigue driving is an important factor leading to the operation errors of drivers.
[0003] In the fatigue detection of drivers, the face picture is collected through the camera lens, the face position in the image is accurately detected, the face is modeled by using the related technology, the effective feature data is obtained according to the face structure characteristics and the distance characteristics between the local regions of the face, and finally the feature data is matched with the feature template in the database, the fatigue degree is judged according to the matching result, so as to realize the working process of the simulation fatigue driving monitoring system.
[0004] At present, in improving the accuracy of face key point detection, a heat map regression model based on CNN (Convolutional Neural Networks) is usually used. However, the heat map regression model based on CNN outputs a series of key point heat maps, and the heat map cannot be differentiated, and the gradient can only flow from the heat map to the original input image, so that the loss function cannot directly optimize the error between the key point prediction value and the true value, there is a lower bound of theoretical error, and the model performs poorly on low resolution images. SUMMARY
[0005] The purpose of the present application is to provide a fatigue detection and analysis method for drivers of commercial vehicles, which improves the accuracy of face key point detection and realizes accurate detection of whether the user is in a fatigue state.
[0006] To achieve the above purpose, the present application provides the following technical scheme: a fatigue detection and analysis method for drivers of commercial vehicles, specifically comprising the following steps: Step S110: obtaining a first heat map, a second heat map, a third heat map of each key point of a user face in each frame of video image in a preset monitoring video, and an expression fatigue value of the user; wherein the first heat map is an initial heat map of the key point, the second heat map is a heat map after the key point on the first heat map is offset, and the third heat map is a heat map of a key point adjacent to the key point on the first heat map; Step S120: generating target coordinates of each key point in each frame of the video image according to the first heat map, the second heat map and the third heat map; Step S130: generating a closed-eye proportion and a yawning proportion of a user in the monitoring video according to the target coordinates, and generating an expression fatigue proportion of the user in the monitoring video according to the expression fatigue value; Step S140: determining a fatigue state of the user according to the closed-eye proportion, the expression fatigue proportion and the yawning proportion.
[0007] As a further improvement of the present application, the first heat map, the second heat map and the third heat map are probability maps corresponding to each key point in the video image, and each pixel value on the probability map represents the probability of the key point corresponding to the pixel position; and the expression fatigue value is a score value for the facial expression state in each frame of the video image.
[0008] As a further improvement of the present application, the first heat map, the second heat map, the third heat map and the expression fatigue value are trained, and the specific training steps include the following: Step S111: training a preset first neural network model in a first stage by using a first sample set to obtain a first neural network model after first stage training; wherein the first sample set includes first samples labeled with first label information; Step S112: training the first neural network model after first stage training by using a second sample set to obtain a first neural network model after second stage training; wherein the second sample set includes second samples labeled with first label information and second samples not labeled with second label information; Step S113: training a preset second neural network model by using a third sample set to obtain a trained second neural network model; wherein the third sample set includes third samples labeled with second label information; Step S114: obtaining the first heat map, the second heat map and the third heat map of each key point in each frame of the video image in the monitoring video according to the first neural network model after second stage training, and obtaining the expression fatigue value of the user in each frame of the video image in the monitoring video according to the trained second neural network model.
[0009] As a further improvement of the present application, the specific obtaining steps of the first heat map, the second heat map and the third heat map include the following: Step S1141: down-sampling each frame of the video image in the monitoring video according to the down-sampling module to obtain first feature information of each key point in each frame of the video image; Step S1142: input the first feature information into the regression module to obtain a first heat map, a second heat map and a third heat map of each key point of the face in each frame of the video image.
[0010] As a further improvement of the present application, the specific obtaining step of the expression fatigue value comprises the following: Step S1143: performing feature extraction on each frame of the video image in the monitoring video to obtain second feature information of the user's face in each frame of the video image; Step S1144: classifying the second feature information to obtain the expression fatigue value of the user in each frame of the video image in the monitoring video.
[0011] As a further improvement of the present application, the target coordinate obtaining step of the key point comprises the following: Step S121: performing coordinate conversion on the first heat map to obtain initial coordinates of each key point on the video image; Step S122: generating a first intermediate coordinate of each key point on the video image according to the initial coordinates and the second heat map; Step S123: generating a second intermediate coordinate of each key point on the video image according to the initial coordinates and the third heat map; Step S124: generating a target coordinate of each key point on the video image according to the first intermediate coordinate and the second intermediate coordinate.
[0012] As a further improvement of the present application, the obtaining step of the closed-eye proportion and the yawning proportion comprises the following: Step S131: generating an eye horizontal-vertical distance ratio and a mouth horizontal-vertical distance ratio of the user in each frame of the video image according to the target coordinates; wherein the eye horizontal-vertical distance ratio is the ratio between the vertical distance and the horizontal distance of the user's eyes, and the mouth horizontal-vertical distance ratio is the ratio between the vertical distance and the horizontal distance of the user's mouth; Step S132: determining the closed-eye proportion and the yawning proportion of the user in the monitoring video according to the eye horizontal-vertical distance ratio and the mouth horizontal-vertical distance ratio, respectively.
[0013] As a further improvement of the present application, the determination step of the fatigue state comprises the following: Step S141: if the closed-eye proportion is higher than a first threshold or / and the expression fatigue proportion is higher than a second threshold, determining that the user is in a fatigue state; Step S142: if the closed-eye proportion is lower than the first threshold, the expression fatigue proportion is lower than the second threshold, and the yawning proportion is higher than a third threshold, determining that the user is not in a fatigue state.
[0014] As a further improvement of the present application, the first label information is label information of a first sample labeling a key point position, and the second label information is label information of a third sample labeling an expression fatigue value.
[0015] Compared with the prior art, the present application has the beneficial effects that: the technical solution realizes obtaining a first heat map, a second heat map, a third heat map of each key point of a user's face in each frame of video image, and an expression fatigue value of the user; the technical solution generates target coordinates of each key point in the video image according to the first heat map, the second heat map and the third heat map; generates a closed-eye proportion and a yawn proportion of the user in the monitoring video according to the target coordinates, and generates an expression fatigue proportion of the user according to the expression fatigue value; and detects the fatigue of the user according to the closed-eye proportion, the expression fatigue proportion and the yawn proportion. The technical solution not only improves the accuracy of face key point detection, but also can conduct all-round fatigue detection on the user, thereby improving the accuracy of fatigue detection. BRIEF DESCRIPTION OF DRAWINGS
[0016] Figure 1 It is a schematic diagram of the overall process of the method.
[0017] Figure 2 It is a schematic diagram of the training process of the method.
[0018] Figure 3 It is a schematic diagram of the first heat map, the second heat map and the third heat map acquisition process of the method.
[0019] Figure 4 It is a schematic diagram of the expression fatigue value acquisition process of the method.
[0020] Figure 5 It is a schematic diagram of the target coordinate acquisition process of the key point of the method.
[0021] Figure 6 It is a schematic diagram of the closed-eye proportion and yawn proportion acquisition process of the method.
[0022] Figure 7 It is a schematic diagram of the fatigue state determination process of the method. DETAILED DESCRIPTION
[0023] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. The described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0024] Referring to Figures 1 to 7 The present application provides the following technical solutions.
[0025] S110, obtaining a first heat map, a second heat map, a third heat map of each key point of a user face in each frame of video image in a preset monitoring video, and an expression fatigue value of the user; wherein the first heat map is an initial heat map of the key point, the second heat map is a heat map of the key point after offset on the first heat map, and the third heat map is a heat map of a key point adjacent to the key point on the first heat map.
[0026] Specifically, when fatigue detection of a user in a monitoring video is needed, only the monitoring video needs to be framed to obtain each frame of image in the monitoring video, and then each frame of video image is input into a pre-trained neural network model to output a first heat map, a second heat map and a third heat map of each key point of the face in the video image, so that the face key point detection in each frame of video image can be realized, and each frame of video image is input into a pre-trained second neural network model to output an expression fatigue value of the user, so that the facial expression of the user in each frame of video image can be recognized to obtain the expression fatigue value of the user in each frame of video image.
[0027] Wherein, the heat map is a probability map of each key point in the video image, and each pixel value on the probability map represents the probability size of the position corresponding to the pixel point being a key point; the expression fatigue value is a score value for the facial expression state in each frame of video image.
[0028] In this embodiment, the first heat map is a probability map generated under the condition of a conventional heat map regression, that is, an initial heat map of the key point, and the pixel value size of the first heat map is 0 or 1, wherein the pixel value of 1 in the first heat map represents the position of the key point; the second heat map is a heat map of the actual offset of the key point on the first heat map; and the third heat map is a probability map of a key point adjacent to the key point on the first heat map.
[0029] S111, using a first sample set to train a pre-set first neural network model in a first stage to obtain a first neural network model after first stage training; wherein the first sample set includes first samples labeled with first label information; S112, using a second sample set to train the first neural network model after the first stage training to obtain a first neural network model after the second stage training; wherein the second sample set includes second samples labeled with the first label information and second samples without second label information; S113, training the preset second neural network model by using a third sample set to obtain a trained second neural network model; wherein the third sample set comprises third samples labeled with second label information; S114, obtaining, according to the first neural network model trained in the second stage, a first heat map, a second heat map and a third heat map of each key point in each video image in the monitoring video, and simultaneously obtaining, according to the trained second neural network model, an expression fatigue value of a user in each video image in the monitoring video.
[0030] Specifically, the first label information is label information about key point positions labeled by the first samples, the second label information is label information about expression fatigue values labeled by the third samples, and the first samples, the second samples and the third samples are all sample images with facial features.
[0031] In the process of the first stage training of the first neural network model, due to the high cost of feature point labeling data, the first sample set used for the first stage training of the first neural network model is small, and the model trained in this stage has poor generalization on other data of the same domain image or other domain images. After the first stage training of the first neural network model by using the first sample set, a large amount of unannotated data is used to perform cyclic promotion self-training on the first neural network model, that is, the second stage training of the first neural network model by using the second sample set. When the first neural network model is trained in the second stage, two groups of small resolution heat map outputs are additionally added, the first resolution heat map output branch can be named as Task1, the second resolution heat map output branch can be named as Task2, and the third resolution heat map output branch can be named as Task3, wherein the first resolution is lower than the second resolution, the second resolution is lower than the third resolution, and the third resolution heat map output branch is the normal output of the first neural network model.
[0032] Specifically, in the process of training the first neural network model in the second stage, the first neural network model is trained in advance by the second samples in the second sample set labeled with the first label information, and the second samples in the second sample set not labeled with the first label information are input into the first neural network model to output the heat map of the first resolution, the pseudo label is labeled for the second samples not labeled with the first label information according to the heat map, and the training of the second neural network model is continued until the model converges; then the first neural network model is trained again by the second samples in the second sample set labeled with the first label information, and the second samples in the second sample set not labeled with the first label information are input into the first neural network model to output the heat map of the second resolution, the pseudo label is labeled for the second samples not labeled with the first label information according to the heat map, and the training of the second neural network model is continued until the model converges; finally, the first neural network model is trained again by the second samples in the second sample set labeled with the first label information, and the second samples in the second sample set not labeled with the first label information are input into the first neural network model to output the heat map of the third resolution, the pseudo label is labeled for the second samples not labeled with the first label information according to the heat map, and the training of the second neural network model is continued until the model converges, thereby completing the training of the first neural network model.
[0033] Among them, the first heat map adopts the mean square error loss function, and the second heat map and the third heat map both adopt the L1 loss function. Among them, the L1 loss function is a function of all absolute differences between the true value and the predicted value, and the mean square error loss function is a variance function between the predicted value and the true value.
[0034]
[0035] In the process of training the second neural network model by using the third sample set, since the third sample set includes the third samples labeled with the second label information, the second neural network model only needs to be trained by using the third sample set until the second neural network model converges. Among them, the second neural network model can be constructed by using the most common convolution, pooling, BN, activation layer, etc., and the loss function of the second neural network model can adopt the cross-entropy loss function as shown below: Among them, p(x) represents the true distribution of the sample, and q(x) represents the distribution predicted by the model.
[0036] The first neural network model includes a down-sampling module and a regression module, and step S114 includes steps S1141 and S1142.
[0037] S1141, down-sampling each video image in the monitoring video according to the down-sampling module, to obtain first feature information of each key point in each video image; S1142, inputting the first feature information into the regression module, to obtain a first heat map, a second heat map and a third heat map of each key point of the face in each video image.
[0038] In the embodiment, the first neural network model comprises a down-sampling module and a regression module. The down-sampling module can extract features of the video image. The structure of the down-sampling module mainly comprises a convolution layer, a BN (Batch Normalization) layer, an activation layer and a pooling layer. The regression module comprises a first regression module, a second regression module and a third regression module. The first regression module, the second regression module and the third regression module each comprise a convolution layer and an activation layer. The first regression module is used to output the first heat map of each key point of the face in the video image. The second regression module is used to output the second heat map of each key point of the face in the video image. The third regression module is used to output the third heat map of each key point of the face in the video image.
[0039] When the video image is down-sampled, only the size of the video image needs to be reduced, and the number of sampling points of the matrix is reduced, so as to extract features of the key points of the face in the video image. After the video image is down-sampled, the first heat map, the second heat map and the third heat map can be generated by the first regression module, the second regression module and the third regression module, respectively.
[0040] S1143, extracting features of each video image in the monitoring video, to obtain second feature information of a user face in each video image. S1144, classifying the second feature information, to obtain an expression fatigue value of the user in each video image in the monitoring video.
[0041] In the embodiment, the second neural network model can comprise a feature extraction module and a classification module. After each video image in the monitoring video is subjected to convolution and pooling operations in the feature extraction module, the second feature information of the user face in the video image can be obtained. Then, by classification in the classification module, the expression fatigue value of the user in each video image in the monitoring video can be obtained.
[0042] In the embodiment, before each video image in the monitoring video is input into the first neural network model and the second neural network model, the video image needs to be preprocessed due to the size of the video image and noise and other factors, so as to more accurately detect and classify each video image.
[0043] Specifically, the video image can be scaled in advance to adjust the size of the video image to a size that matches the neural network, and then the video image can be data augmented to avoid overfitting of the model. At the same time, the robustness of the model can also be improved. For example, the noise areas in the video image are filled with 0 pixels. After the data augmentation processing, the video image needs to be normalized to map the pixel values of the video image to between 0 and 1.
[0044] S120. Generate target coordinates of each key point in each frame of the video image according to the first heat map, the second heat map, and the third heat map.
[0045] In this embodiment, the first heat map is a probability map generated by the video image under conventional heat map regression conditions, that is, the initial heat map of the key points; the second heat map is a heat map after the key points on the first heat map are offset, that is, the second heat map includes a probability map of the key points offset in the X-axis direction and a probability map of the key points offset in the Y-axis direction; the third heat map is actually a heat map after the key points on the first heat map are offset, but the heat map can be used as a heat map of the key points adjacent to the key points, that is, the third heat map also includes a probability map of the offset in the X-axis direction and a probability map of the offset in the Y-axis direction. The coordinates of the real position of each key point in the video image can be obtained through the first heat map and the second heat map, and the other coordinates of each key point in the video image can be obtained through the third heat map. Finally, the average value is taken to obtain the final coordinate point of each key point in the video image, that is, the target coordinate.
[0046] S121, performing coordinate transformation on the first heat map to obtain initial coordinates of each key point on the video image; S122: Generate first intermediate coordinates of each key point on the video image according to the initial coordinates and the second heat map; S123, generating second intermediate coordinates of each key point on the video image according to the initial coordinates and the third heat map; S124: Generate target coordinates of each key point on the video image according to the first intermediate coordinates and the second intermediate coordinates.
[0047] In the embodiment, the first heat map extracts the peak coordinates of the heat map by using the Argmax function, and maps back to the original image, so as to obtain the initial coordinates of the key points on the video image. The second heat map and the third heat map each include two probability maps offset along the X-axis and the Y-axis, so the probabilities on the X-axis and the Y-axis can be directly used as two offset coordinates. Then, the initial coordinates are subtracted from the two offset coordinates, so as to obtain the first intermediate coordinates and the second intermediate coordinates of the key points on the video image. The two offset coordinates can be understood as the coordinates of the key points after offset on the first heat map, and the offset coordinates corresponding to the third heat map can be understood as an intermediate coordinate of a key point adjacent to the key point. The first intermediate coordinates and the second intermediate coordinates can be understood as the coordinates of the real positions of each key point in the video image. The initial prediction of the coordinates of the key points in the video image is the initial coordinates. In the actual detection process, the key points actually shift, so the initial coordinates need to be corrected by the two offset coordinates to obtain the coordinates of the real positions of the key points in the video image, that is, the first intermediate coordinates and the second intermediate coordinates.
[0048] In addition, in the embodiment, in order to more accurately detect the target coordinates of the key points, after the intermediate coordinates of the key points are generated, the second intermediate coordinates and the first intermediate coordinates of other key points are averaged to obtain the target coordinates of the key points on the video image. For example, the key points of a face include a first key point and a second key point, and the first key point is adjacent to the second key point. At this time, the model outputs three heat maps of the first key point and three heat maps of the second key point. The first heat map of the first key point is converted in coordinates, and the first intermediate coordinates of the first key point are generated by using the initial coordinates of the first key point and the corresponding second heat map. The second intermediate coordinates of the first key point are generated by using the initial coordinates of the second key point and the third heat map of the second key point. Finally, the first intermediate coordinates and the second intermediate coordinates of the first key point are added and averaged, so as to obtain the target coordinates of the first key point. Similarly, the target coordinates of the second key point can also be calculated in the above manner.
[0049] S130, generating the closed-eye proportion and the yawning proportion of the user in the monitoring video according to the target coordinates, and generating the expression fatigue proportion of the user in the monitoring video according to the expression fatigue value.
[0050] In the embodiment, the key points of the face include key points of eyes and key points of a mouth, wherein the key points of the eyes and the key points of the mouth each consist of a plurality of key points. After the target coordinates of the key points of the eyes and the key points of the mouth are obtained, whether the eyes of the face in each frame of video image in the monitoring video are in a closed eye state and whether the mouth is in a yawning state can be determined. The state in each frame of video image can be used to obtain the closed eye proportion and the yawning proportion of the user in the monitoring video.
[0051] When the expression fatigue proportion of the user in the monitoring video is generated by the expression fatigue value, it is only necessary to determine whether the expression fatigue value of the user in each frame of video image exceeds a preset fatigue value. If the preset fatigue value is exceeded, it is determined that the user is in a fatigue state. If the preset fatigue value is not exceeded, it is determined that the user is not in a fatigue state. Thus, the expression fatigue proportion of the user in the monitoring video can be generated.
[0052] S131, generating an eye horizontal-vertical distance ratio and a mouth horizontal-vertical distance ratio of the user in each frame of the video image according to the target coordinates; wherein the eye horizontal-vertical distance ratio is a ratio between a vertical distance and a horizontal distance of the eyes of the user, and the mouth horizontal-vertical distance ratio is a ratio between a vertical distance and a horizontal distance of the mouth of the user. S132, determining a closed eye proportion and a yawning proportion of the user in the monitoring video according to the eye horizontal-vertical distance ratio and the mouth horizontal-vertical distance ratio, respectively.
[0053] In the embodiment, the key points of the eyes include at least two horizontal key points and at least two vertical key points, and the key points of the mouth include at least two horizontal key points and at least two vertical key points. After the eye horizontal-vertical distance ratio and the mouth horizontal-vertical distance ratio of the user are generated, whether the user is in a closed eye state and a yawning state in each frame of video image can be determined by judging whether the eye horizontal-vertical distance ratio is lower than a preset first ratio value and whether the mouth horizontal-vertical distance ratio is lower than a preset second ratio value. Finally, the state of the user in each frame of video image is counted, and the closed eye proportion and the yawning proportion of the user in the monitoring video can be obtained.
[0054] S140, detecting fatigue of the user according to the closed eye proportion, the expression fatigue proportion, and the yawning proportion.
[0055] Specifically, when fatigue of the user in the monitoring video is detected, the technical solution is mainly used to detect whether the user is in a fatigue state within a preset time. By detecting the closed eye proportion, the expression fatigue proportion, and the yawning proportion of the user within the preset time, whether the user is in a fatigue state within the preset time can be accurately detected.
[0056] S141, if the closed-eye proportion is higher than a first threshold or / and the expression fatigue proportion is higher than a second threshold, it is determined that the user is in a fatigue state; S142, if the closed-eye proportion is lower than the first threshold, the expression fatigue proportion is lower than the second threshold, and the yawn proportion is higher than a third threshold, it is determined that the user is not in a fatigue state.
[0057] In the embodiment, the first threshold is a critical value of the closed-eye proportion of the user in the monitoring video within a preset time, the second threshold is a critical value of the expression fatigue proportion of the user in the monitoring video within a preset time, and the third threshold is a critical value of the yawn proportion of the user in the monitoring video within a preset time. When the closed-eye proportion is higher than the first threshold or / and the expression fatigue proportion is higher than the second threshold, it can be determined that the user is in a fatigue state in the monitoring video. When the closed-eye proportion is lower than the first threshold, the expression fatigue proportion is lower than the second threshold, and the yawn proportion is higher than the third threshold, it can be accurately determined that the user is not in a fatigue state, but it can be determined that the user is currently in a daydreaming state at this time. At this time, the user can be prompted by voice, and thus the risk of accidents of the user in the driving process can be reduced.
[0058] The fatigue detection method of the technical solution, by acquiring a first heat map, a second heat map, and a third heat map of each key point of a user's face in each frame of video image in a preset monitoring video, and an expression fatigue value of the user; wherein the first heat map is an initial heat map of the key point, the second heat map is a heat map of the key point after offset on the first heat map, and the third heat map is a heat map of a key point adjacent to the key point on the first heat map; generating target coordinates of each key point in each frame of the video image according to the first heat map, the second heat map, and the third heat map; generating a closed-eye proportion and a yawn proportion of the user in the monitoring video according to the target coordinates, and generating an expression fatigue proportion of the user in the monitoring video according to the expression fatigue value; and performing fatigue detection on the user according to the closed-eye proportion, the expression fatigue proportion, and the yawn proportion. The technical solution combines the first heat map, the second heat map, and the third heat map of each key point of the face in the video image to generate the target coordinates of each key point in the video image, improves the accuracy of face key point detection, and further acquires the expression fatigue value of the user through the video image to realize comprehensive fatigue detection on the user in the monitoring video, and improves the accuracy of fatigue detection.
[0059] The above merely describes the preferred examples of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that modifications can be made to the foregoing embodiments, or some of the technical features thereof can be equivalently replaced, without departing from the spirit and principle of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A method for detecting and analyzing fatigue of a commercial vehicle driver, characterized in that: The specific steps include: Step S110: Obtaining a first heat map, a second heat map, and a third heat map of each key point on the user's face in each frame of a preset surveillance video, as well as the user's facial fatigue value; wherein the first heat map is an initial heat map of the key point, the second heat map is a heat map of the key point on the first heat map after being offset, and the third heat map is a heat map of the key point adjacent to the key point on the first heat map; Step S120: generating target coordinates of each key point in each frame of the video image according to the first heat map, the second heat map, and the third heat map; Step S130: generating a user's eye closing ratio and yawning ratio in the surveillance video according to the target coordinates, and generating a user's facial fatigue ratio in the surveillance video according to the facial fatigue value; Step S140: determining the fatigue state of the user according to the eye closing ratio, the facial fatigue ratio, and the yawning ratio.
2. The method for detecting and analyzing driver fatigue of an operating vehicle according to claim 1, wherein: The first heat map, the second heat map, and the third heat map are probability maps corresponding to each key point in the video image, and each pixel value on the probability map represents the probability that the position corresponding to the pixel point is a key point; the expression fatigue value is the score value for the facial expression state in each frame of the video image.
3. The method for detecting and analyzing driver fatigue of an operating vehicle according to claim 1, wherein: The first heat map, the second heat map, the third heat map and the expression fatigue value are trained, and the specific training steps include the following: Step S111: performing a first phase of training on a preset first neural network model using a first sample set to obtain a first neural network model after the first phase of training; wherein the first sample set includes first samples annotated with first label information; Step S112: using the second sample set to train the first neural network model trained in the first stage to obtain the first neural network model trained in the second stage; wherein the second sample set includes second samples marked with the first label information and second samples not marked with the second label information; Step S113: using the third sample set to train the preset second neural network model to obtain a trained second neural network model; wherein the third sample set includes third samples marked with second label information; Step S114: Obtain the first heat map, the second heat map, and the third heat map of each key point in each frame of the surveillance video according to the first neural network model trained in the second stage, and at the same time obtain the user's expression fatigue value in each frame of the surveillance video according to the trained second neural network model.
4. The method for detecting and analyzing driver fatigue of an operating vehicle according to claim 1, wherein: The specific steps of obtaining the first heat map, the second heat map, and the third heat map include the following: Step S1141: downsampling each frame of the surveillance video according to the downsampling module to obtain first feature information of each key point in each frame of the video image; Step S1142: Input the first feature information into the regression module to obtain a first heat map, a second heat map, and a third heat map of each key point of the face in each frame of the video image.
5. The method for detecting and analyzing driver fatigue of an operating vehicle according to claim 1, wherein: The specific steps of obtaining the expression fatigue value include the following: Step S1143: extracting features from each frame of the surveillance video to obtain second feature information of the user's face in each frame of the video image; Step S1144: classify the second feature information to obtain the user's facial expression fatigue value in each frame of the surveillance video.
6. The method for detecting and analyzing driver fatigue of an operating vehicle according to claim 1, characterized in that: The step of obtaining the target coordinates of the key points includes the following steps: Step S121: performing coordinate transformation on the first heat map to obtain initial coordinates of each key point on the video image; Step S122: generating first intermediate coordinates of each key point on the video image according to the initial coordinates and the second heat map; Step S123: generating second intermediate coordinates of each key point on the video image according to the initial coordinates and the third heat map; Step S124: generating target coordinates of each key point on the video image according to the first intermediate coordinates and the second intermediate coordinates.
7. The method for detecting and analyzing driver fatigue of an operating vehicle according to claim 1, wherein: The steps for obtaining the eye closing ratio and the yawning ratio include the following: Step S131: generating a horizontal-to-vertical distance ratio of the user's eyes and a horizontal-to-vertical distance ratio of the user's mouth in each frame of the video image according to the target coordinates; wherein the horizontal-to-vertical distance ratio of the eyes is the ratio between the vertical distance and the horizontal distance of the user's eyes, and the horizontal-to-vertical distance ratio of the mouth is the ratio between the vertical distance and the horizontal distance of the user's mouth; Step S132: determining the proportion of eyes closed and the proportion of yawns of the user in the surveillance video according to the horizontal-to-vertical distance ratio of the eyes and the horizontal-to-vertical distance ratio of the mouth, respectively.
8. The method for detecting and analyzing driver fatigue of an operating vehicle according to claim 1, wherein: The fatigue state determination step includes the following steps: Step S141: if the eye-closing ratio is higher than a first threshold and / or the facial fatigue ratio is higher than a second threshold, determining that the user is in a fatigue state; Step S142: If the eye closing ratio is lower than the first threshold, the facial expression fatigue ratio is lower than the second threshold, and the yawning ratio is higher than the third threshold, it is determined that the user is not in a fatigued state.
9. The method for detecting and analyzing driver fatigue of an operating vehicle according to claim 3, wherein: The first label information is label information of the first sample marked with key point positions, and the second label information is label information of the third sample marked with expression fatigue values. The first sample, the second sample, and the third sample are all sample images with facial features.