Fatigue driving monitoring method

Abstract

The invention relates to a fatigue driving monitoring method, and relates to the technical field of vehicle safety. The method comprises the following steps: collecting a first face image of a driver in a preset time period in real time after the driver gets on the vehicle, and determining a face behavior feature threshold value of the driver; a second face image of the driver is collected in real time in the driving process; calculating a real-time face behavior feature value corresponding to each frame of second face image; determining a facial behavior characteristic change rate time curve according to a facial behavior characteristic threshold value of the driver and the real-time facial behavior characteristic value corresponding to each frame of second facial image, determining a facial behavior characteristic jump value and jump duration within a preset duration range before and after each moment, and determining a facial behavior characteristic change rate of the driver according to preset facial behavior calibration data. Outputting a facial behavior judgment result of the driver; and the fatigue evaluation grade of the driver is determined according to the face behavior judgment result, the vehicle state collected in real time and the driving duration. The fatigue state of the driver can be recognized and reminded, and the driving safety is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle safety technology, and in particular to a method for monitoring fatigue driving. Background Art

[0002] At present, with the improvement of traffic safety awareness, fatigue driving has become one of the major hidden dangers of road safety. Existing fatigue driving monitoring systems usually rely on single physiological signals or behavioral characteristics for judgment, such as eye tracking, facial expression analysis, etc. Specifically, existing fatigue driving monitoring systems usually rely on facial key point detection models to identify facial key points, and use the key points of the eyes and mouth to calculate the aspect ratio of the eye and mouth contours to judge the driver's fatigue state.

[0003] However, for the dynamic behavior of the driver during driving, the position of the key points of the face detected by the deep learning model is floating. At the same time, the size of the driver's facial features and behaviors such as talking and blinking can also lead to misjudgment of fatigue detection.

[0004] Therefore, there is an urgent need for a fatigue driving monitoring method to improve the recognition rate of fatigue driving. Summary of the invention

[0005] Based on this, it is necessary to provide a fatigue driving monitoring method to address the above technical issues.

[0006] In a first aspect, a method for monitoring driving fatigue is provided, the method comprising:

[0007] Real-time collection of the first facial image of the driver within a preset time after he gets on the vehicle;

[0008] Determining a facial behavior feature threshold of the driver according to the first facial image;

[0009] Acquiring a second facial image of the driver in real time during driving;

[0010] Calculate the real-time facial behavior feature value corresponding to the second facial image in each frame;

[0011] Determining a facial behavior feature change rate time curve according to the facial behavior feature threshold of the driver and the real-time facial behavior feature value corresponding to each frame of the second facial image;

[0012] In the facial behavior feature change rate time curve, determine the facial behavior feature jump value and jump duration within a preset time range before and after each moment;

[0013] Outputting the driver's facial behavior determination result according to the facial behavior feature jump value, the jump duration and preset facial behavior calibration data;

[0014] The driver's fatigue evaluation level is determined by comprehensively considering the facial behavior determination result, the vehicle status and driving time collected in real time.

[0015] As an optional implementation manner, determining the facial behavior feature threshold of the driver according to the first facial image includes:

[0016] For each frame of the first facial image, identifying facial key points in the first facial image of the frame;

[0017] Performing B-spline curve fitting on the facial key points to calculate the degree of facial behavior opening and closing corresponding to the first facial image of the frame;

[0018] The average value of the facial behavior opening and closing degree corresponding to each frame of the first facial image is determined as the facial behavior feature threshold of the driver.

[0019] As an optional implementation manner, the calculating the real-time facial behavior feature value corresponding to each frame of the second facial image includes:

[0020] For each frame of the second facial image, identifying facial key points in the second facial image of the frame;

[0021] B-spline curve fitting is performed on the facial key points to calculate the facial behavior opening and closing degree corresponding to the second facial image of the frame, and the facial behavior opening and closing degree is determined as the real-time facial behavior feature value corresponding to the second facial image of the frame.

[0022] As an optional implementation, the facial key points include eye key points and mouth key points.

[0023] As an optional implementation manner, determining the facial behavior feature change rate time curve according to the facial behavior feature threshold of the driver and the real-time facial behavior feature value corresponding to each frame of the second facial image includes:

[0024] For each frame of the second facial image, a ratio of the real-time facial behavior feature value corresponding to the frame of the second facial image to the facial behavior feature threshold of the driver is determined as a facial behavior feature change rate corresponding to the frame of the second facial image;

[0025] The facial behavior feature change rate corresponding to each frame of the second facial image is fitted according to time to obtain the facial behavior feature change rate time curve.

[0026] As an optional implementation, the method further includes:

[0027] Kalman filtering is performed on the facial behavior feature change rate time curve.

[0028] As an optional implementation manner, in the facial behavior feature change rate time curve, determining the facial behavior feature jump value and jump duration within a preset time range before and after each moment includes:

[0029] In the facial behavior feature change rate time curve, a point where the slope change is greater than a preset first slope change threshold is determined as a jump point;

[0030] Determine the difference in the facial behavior feature change rate between the jump point and the low change point as the facial behavior feature jump value;

[0031] The time difference between the adjacent jump and the low change point is determined as the duration of the jump.

[0032] As an optional implementation, the preset facial behavior calibration data includes standard eyes-open behavior calibration data, standard eyes-closed behavior calibration data, tired eyes-closed behavior calibration data, speaking behavior calibration data and yawning behavior calibration data.

[0033] As an optional implementation manner, outputting the driver's facial behavior determination result according to the facial behavior feature jump value, the jump duration and preset facial behavior calibration data includes:

[0034] For the facial behavior calibration data corresponding to each facial behavior, if the facial behavior feature jump value and the jump duration meet the facial behavior calibration data corresponding to the facial behavior, the facial behavior is determined as the facial behavior judgment result of the driver.

[0035] As an optional implementation, the method further includes:

[0036] If the fatigue evaluation level is a low level, determining that the driver is in a non-fatigue state;

[0037] If the fatigue evaluation level is medium, the driver is determined to be in a non-focused state, and a focus reminder is sent to the driver;

[0038] If the fatigue evaluation level is a high level, it is determined that the driver is in a fatigue state, and a fatigue reminder is given to the driver.

[0039] In a second aspect, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the method steps described in any one of the first aspects are implemented.

[0040] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0041] The present application provides a fatigue driving monitoring method, and the technical solution provided by the embodiment of the present application brings at least the following beneficial effects: the method includes: real-time acquisition of a first facial image of the driver within a preset time after getting on the vehicle; determining the facial behavior feature threshold of the driver according to the first facial image; real-time acquisition of the second facial image of the driver during driving; calculating the real-time facial behavior feature value corresponding to each frame of the second facial image; determining the facial behavior feature change rate time curve according to the facial behavior feature threshold of the driver and the real-time facial behavior feature value corresponding to each frame of the second facial image; determining the facial behavior feature jump value and jump duration within the preset time range before and after each moment in the facial behavior feature change rate time curve; outputting the facial behavior determination result of the driver according to the facial behavior feature jump value, the jump duration and the preset facial behavior calibration data; and determining the fatigue evaluation level of the driver according to the facial behavior determination result, the real-time acquired vehicle state and the driving time. The present application can determine the facial behavior feature threshold based on the individual characteristics of the driver by real-time acquisition of the first facial image of the driver within a preset time after getting on the vehicle. This personalized setting avoids the limitations of unified standards and can more accurately adapt to the differences in facial behavior characteristics of different drivers, thereby improving the accuracy of fatigue detection. During driving, the second facial image of the driver is collected in real time, and the facial behavior feature values ​​corresponding to each frame image are calculated. By monitoring the changes in the driver's facial behavior in real time, dangerous states such as fatigue or distraction can be captured instantly to ensure safety during driving. By determining the time curve of the facial behavior feature change rate, the changes in the driver's facial behavior can be dynamically monitored to provide a time evolution analysis of the driver's state. This dynamic analysis method can better judge the development trend of fatigue or inattention, rather than making a judgment based solely on a single behavior at a certain moment. On the time curve of the facial behavior feature change rate, the jump value and jump duration of the facial behavior feature can be identified and compared with the preset facial behavior calibration data. This method can accurately distinguish different types of facial behavior changes, such as blinking, yawning, etc., thereby providing a more detailed basis for fatigue state determination. In addition, the embodiment of the present application not only takes into account the changes in facial behavior, but also comprehensively analyzes factors such as vehicle status and driving time, and determines the driver's fatigue evaluation level through the combination of multi-dimensional information. The embodiment of the present application makes a more comprehensive and reasonable fatigue assessment based on actual driving conditions, changes in the driver's state, etc., thereby improving the applicability and accuracy of the system. Through the comprehensive assessment results, reminders of different intensities can be provided according to different fatigue evaluation levels. For example, low-level reminders help keep the driver focused, while high-level reminders promptly warn the driver of potential fatigue risks to ensure driving safety.In summary, the fatigue driving monitoring method of the embodiment of the present application can accurately and effectively identify and remind the driver of the fatigue state and improve driving safety through personalized threshold setting, real-time monitoring, dynamic analysis and multi-dimensional comprehensive evaluation.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flowchart of a method for monitoring driving fatigue provided in an embodiment of the present application;

[0045] Figure 2 A flowchart of another method for monitoring driving fatigue provided in an embodiment of the present application;

[0046] Figure 3 A flowchart of another method for monitoring driving fatigue provided in an embodiment of the present application;

[0047] Figure 4 A flowchart of another method for monitoring fatigue driving provided in an embodiment of the present application;

[0048] Figure 5 A flowchart of another method for monitoring driving fatigue provided in an embodiment of the present application;

[0049] Figure 6 A flowchart of an example of a fatigue driving monitoring method provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The following will describe in detail a fatigue driving monitoring method provided by an embodiment of the present application in combination with a specific implementation method. Figure 1A flowchart of a fatigue driving monitoring method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the specific steps are as follows:

[0053] Step 101, collecting in real time the first facial image of the driver within a preset time after he gets on the vehicle.

[0054] In implementation, at the initial stage when the driver enters the vehicle, the driver's first facial image can be captured by an image acquisition device such as an infrared camera. Based on the driver's sobriety level at the stage when he just enters the vehicle, the driver's first facial image in a normal state is relatively acquired, and used as a reference image for subsequent fatigue analysis. In addition, the driver of the same vehicle is not necessarily the same person. Collecting the first facial image of the driver entering the vehicle before each fatigue detection can achieve differentiated analysis of different drivers and improve the accuracy of fatigue monitoring. For example, the image acquisition device can be set on the dashboard or rearview mirror of the vehicle. Within the first few minutes after the driver enters the vehicle, multiple frames of facial images of the driver are captured by a high-definition camera, and these images are stored as the first facial image.

[0055] Step 102: Determine a facial behavior feature threshold of the driver based on the first facial image.

[0056] In implementation, by analyzing the key points in the first facial image, the driver's facial behavior features, such as the degree of eye opening and closing and the degree of mouth opening, etc., are extracted, and based on the facial behavior features of the driver in an awake state, a facial behavior feature threshold is determined as a reference standard for the driver in a normal state. For example, the average degree of eye opening and closing and the average degree of mouth opening and closing of the driver can be determined based on the first facial image and used as the facial behavior feature threshold.

[0057] As an optional implementation, Figure 2 A flowchart of another method for monitoring driving fatigue provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the specific steps of determining the driver's facial behavior feature threshold according to the first facial image in step 102 are as follows:

[0058] Step 201 : for each frame of the first facial image, identify the facial key points in the first facial image of the frame.

[0059] In implementation, the first facial image is analyzed by a preset image processing algorithm to identify key points of the face, such as the coordinates of the corners of the eyes, the corners of the mouth, the tip of the nose, etc. For example, the system uses a convolutional neural network (CNN) or a facial key point detection algorithm based on the Dlib library to accurately identify 68 facial key points in each frame of the image, including the four corner points of the eyes, the upper and lower lip edge points of the mouth, etc. This process may process dozens of frames of images per second to ensure the real-time and accuracy of key point recognition.

[0060] Step 202, performing B-spline curve fitting on facial key points to calculate the degree of facial behavior opening and closing corresponding to the first facial image of the frame.

[0061] In implementation, after the facial key points are identified, the geometric shapes of these points are fitted using B-spline curves to more smoothly and accurately describe the degree of facial behavior, such as the degree of eye opening and closing, mouth opening and closing, nose movement amplitude, and eyebrow movement amplitude. B-spline curves can effectively handle changes in facial contours and provide a continuous curve fitting model to calculate the degree of opening and closing in each frame of the image. For example, the eye contour curve is generated by fitting the coordinates of the four corner points of the eye with a B-spline curve. The longitudinal opening and closing of the eyes is calculated to obtain the degree of eye closure. In the calculation of the mouth opening, the key points of the upper and lower lip edges are fitted, and the distance between the upper and lower lips is measured to calculate the degree of mouth opening and closing.

[0062] Step 203: Determine the average value of the facial behavior opening and closing degrees corresponding to the first facial image of each frame as the facial behavior feature threshold of the driver.

[0063] In implementation, a series of opening and closing values ​​are obtained by calculating the facial behavior opening and closing degree of each frame in the first facial image. These values ​​are then averaged as the baseline facial behavior feature threshold of the driver in a normal state. This threshold is used for subsequent detection of driver fatigue or abnormal behavior. For example, 100 frames of the driver's first facial image are collected, and the eye opening and closing degree and mouth opening and closing degree of each frame are calculated respectively to obtain two feature value sequences. Then the two feature value sequences are averaged, assuming that the average eye opening and closing degree is 20% and the average mouth opening degree is 10%. The system uses these two averages as the driver's facial behavior feature thresholds to evaluate future facial behavior changes.

[0064] As an optional implementation, after performing B-spline curve fitting, the blinking frames may be removed from the first facial image, and the mean of the facial behavior opening and closing degrees may be calculated as the facial behavior feature threshold.

[0065] Step 103: collecting a second facial image of the driver in real time during driving.

[0066] In practice, the driver's face can be continuously monitored throughout the entire process of vehicle driving, and a second facial image can be captured in real time through an image acquisition device for subsequent fatigue detection and facial behavior analysis. For example, 10 frames of facial images can be collected per second to ensure real-time monitoring of the driver's status.

[0067] Step 104, calculating the real-time facial behavior feature value corresponding to each frame of the second facial image.

[0068] In implementation, the facial behavior feature values ​​of each frame of the second facial image, such as the eye opening state, the mouth opening degree, etc., can be extracted through a preset image processing algorithm as an indicator for measuring the driver's facial behavior. For example, through the eye detection algorithm, the system can calculate the degree of eye opening of the driver in each frame of the second facial image, and record these data for further analysis.

[0069] As an optional implementation, Figure 3 A flowchart of another method for monitoring fatigue driving provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the real-time facial behavior feature value corresponding to each frame of the second facial image is calculated in step 104, including:

[0070] Step 301 : for each frame of the second facial image, identify facial key points in the second facial image of the frame.

[0071] In implementation, similar to the processing of the first facial image, during driving, a facial recognition algorithm is applied to each frame of the second facial image to identify the key points of the face. These key points include key positions of the eyes, mouth, nose and other parts, which are used to characterize the driver's current facial behavior. For example, the system uses a real-time facial key point detection algorithm to capture 10 frames of the driver's facial images per second and identify 68 facial key points in each frame. These key points include the inner and outer corners of the eyes, the upper and lower points of the lip edges, etc.

[0072] Step 302, performing B-spline curve fitting on facial key points, calculating the facial behavior opening and closing degree corresponding to the second facial image of the frame, and determining the facial behavior opening and closing degree as the real-time facial behavior feature value corresponding to the second facial image of the frame.

[0073] In implementation, a smooth curve contour is obtained by fitting the identified facial key points with a B-spline curve to capture the facial behavior characteristics of the driver at different time points. The degree of opening and closing reflects the degree of closure of the driver's eyes or the degree of opening of the mouth, and is an important indicator of the real-time facial behavior feature value. For example, the system performs B-spline curve fitting on the eye and mouth key points of each frame image to generate the eye and mouth contour curves for each frame image. By calculating the opening and closing distance in the vertical direction of the eyes, the degree of closure of the driver's eyes can be obtained in real time (such as 20% or 50% closed). Similarly, the system calculates the degree of opening and closing of the mouth (such as 10% or 30% open) by fitting the key points of the mouth. These real-time calculated facial behavior opening and closing degrees will be used as the facial behavior feature values ​​of each frame image, and will be updated in real time for fatigue detection.

[0074] As an optional implementation, the facial key points include eye key points and mouth key points.

[0075] Step 105 , determining a facial behavior feature change rate time curve according to the driver's facial behavior feature threshold and the real-time facial behavior feature value corresponding to each frame of the second facial image.

[0076] In implementation, the real-time facial behavior feature value is compared with the previously set facial behavior feature threshold, the rate of change is calculated, and a time curve is generated to reflect the dynamic changes of the driver's facial behavior features.

[0077] As an optional implementation, Figure 4 A flowchart of another method for monitoring fatigue driving provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, in step 105, the specific steps of determining the facial behavior feature change rate time curve according to the driver's facial behavior feature threshold and the real-time facial behavior feature value corresponding to each frame of the second facial image are as follows:

[0078] Step 401, for each frame of the second facial image, the ratio of the real-time facial behavior feature value corresponding to the frame of the second facial image to the facial behavior feature threshold of the driver is determined as the facial behavior feature change rate corresponding to the frame of the second facial image.

[0079] In implementation, the rate of change is calculated by comparing the real-time facial behavior feature value in each frame of the second facial image with the facial behavior feature threshold calculated from the first facial image. The rate of change can be quantified by the ratio of the real-time feature value to the threshold, and is used to measure the degree of deviation between the driver's current facial behavior and the normal state. The greater the rate of change, the more the driver's facial behavior deviates from the normal state, and fatigue or inattention may occur. For example, suppose the threshold value of eye opening calculated in the first facial image is 20%, and the real-time eye opening in a frame of the second facial image is 40%. The rate of change of the eye behavior feature corresponding to the frame image is 40% / 20%=2, indicating that the degree of eye closure has doubled compared to the normal state. Similarly, if the mouth opening in a frame image is 30% and the threshold is 10%, the rate of change of the mouth corresponding to the frame image is 30% / 10%=3.

[0080] Step 402 , fitting the facial behavior feature change rate corresponding to each frame of the second facial image according to time to obtain a facial behavior feature change rate time curve.

[0081] In implementation, the change rates of the second facial images of each frame are arranged in chronological order, and a time fitting algorithm is applied to generate a time curve of the change rate of facial behavior features. This time curve can intuitively show the changing trend of the driver's facial behavior over time, and is used to identify whether the driver has persistent abnormal behavior or signs of fatigue. During driving, 10 frames of facial images are collected per second, and the change rate of facial behavior features of each frame is calculated in real time. These change rates are then arranged in chronological order, and a time curve of the change rate of facial behavior features is generated using a linear fitting or smoothing filtering method. If the driver's eye opening degree increases continuously over a period of time, the time curve will show a clear upward trend, indicating that the driver may have had his eyes closed for a long time. Similarly, if the change rate of the mouth opening degree continues to increase, it indicates that the driver may have yawned many times and may be in a state of fatigue.

[0082] As an optional implementation, Kalman filtering can also be performed on the time curve of the facial behavior feature change rate.

[0083] Step 106, in the facial behavior feature change rate time curve, determine the facial behavior feature jump value and jump duration within a preset time range before and after each moment.

[0084] In practice, the facial behavior feature change rate time curve can be used to identify the jump points of facial behavior features and the duration of these jumps. The size and duration of the jump value can reflect the degree of driver fatigue. For example, if the degree of eye closure increases sharply over a period of time and lasts for more than 5 seconds, this moment can be identified as a jump point and its duration can be recorded. For example, the user opens and closes his eyes longer when he is tired.

[0085] As an optional implementation, Figure 5 A flowchart of another method for monitoring fatigue driving provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, in step 106, in the facial behavior feature change rate time curve, the specific steps of determining the facial behavior feature jump value and the jump duration within the preset time range before and after each moment are as follows:

[0086] Step 501: In the facial behavior feature change rate time curve, a point where the slope change is greater than a preset first slope change threshold is determined as a jump point.

[0087] In implementation, on the facial behavior feature change rate time curve, the change rate slope is calculated at each moment, and the slope change is compared with a preset first threshold. When the slope change at a certain point exceeds the threshold, it means that the change at that point is more drastic, and the system marks it as a jump point. The jump point is the moment when the driver's facial behavior changes significantly, usually related to fatigue or abnormal conditions. Suppose the slope change of the driver's eye closure change rate time curve at a certain moment is 0.8, and the slope change threshold set by the system is 0.5. Since 0.8 is greater than 0.5, this moment is identified as a jump point, indicating that the driver's facial behavior has changed significantly at this moment, such as sudden eye closure or frequent blinking.

[0088] Step 502: Determine the difference in the facial behavior feature change rate between the jump point and the low change point as the facial behavior feature jump value.

[0089] In implementation, the jump point and the low change point represent the moments when the facial behavior features change drastically and stably, respectively. The low change point can be a local minimum, indicating that at this moment, the driver's facial behavior has not changed significantly, in contrast to the surrounding jump points. A time window (for example, 1 second or 2 seconds) can also be set, and the points in this time window where the change rate is continuously lower than the threshold can be regarded as low change points. By calculating the difference in the change rate of facial behavior features between the jump point and the low change point, the intensity of facial behavior changes can be quantified. The larger the jump value, the more significant the change in the driver's facial behavior, which may mean that the driver's fatigue level is aggravated or the abnormal behavior is more prominent. For example, at the jump point, the change rate of the facial behavior feature is 4, while at the adjacent low change point, the change rate is only 1, then the jump value is 41=3. This indicates that the driver's facial behavior during this period has undergone a more drastic change, such as a rapid transition from a normal state to a fatigued state.

[0090] Step 503: determine the time difference between the adjacent jumps and the low change point as the jump duration.

[0091] In practice, the duration of the jump refers to the time interval from the low change point to the jump point, indicating how long the process of facial behavior characteristics changing from a stable state to a drastic change lasts. The longer the duration, the more likely the driver's abnormal facial behavior is to persist and require more attention. Assuming that the low change point appears at the 3rd second and the jump point appears at the 7th second, the duration of the jump is 7 seconds 3 seconds = 4 seconds. This indicates that the driver's facial behavior characteristics continued to change significantly during this period, possibly because the driver kept his eyes half-closed for a long time or yawned repeatedly during this period, indicating increased fatigue.

[0092] Step 107, outputting the driver's facial behavior determination result according to the facial behavior feature jump value, the jump duration and the preset facial behavior calibration data.

[0093] In practice, based on the facial feature jump value, duration and preset calibration data in the system (such as the classification criteria for normal, fatigue, etc.), the driver's facial behavior is comprehensively analyzed to obtain specific judgment results, and the judgment results such as "normal blinking", "fatigue closing eyes" or "yawning" are output.

[0094] As an optional implementation, the preset facial behavior calibration data includes standard eyes-open behavior calibration data, standard eyes-closed behavior calibration data, tired eyes-closed behavior calibration data, speaking behavior calibration data, and yawning behavior calibration data.

[0095] As an optional implementation, in step 107, the method for outputting the driver's facial behavior determination result according to the facial behavior feature jump value, the jump duration and the preset facial behavior calibration data is:

[0096] For the facial behavior calibration data corresponding to each facial behavior, if the facial behavior feature jump value and the jump duration meet the facial behavior calibration data corresponding to the facial behavior, the facial behavior is determined as the driver's facial behavior determination result.

[0097] In implementation, each facial behavior corresponds to preset facial behavior calibration data, such as different facial behaviors such as blinking, yawning, and eye closure, and their jump value and duration characteristics can be pre-calibrated. When the detected jump value and duration meet these calibration data, it can be determined that the driver is performing a specific facial behavior. Through this method, the driver's fatigue, distraction or other dangerous state can be accurately identified, and appropriate feedback can be made according to different facial behavior types. For example: Assume that the facial behavior calibration data stipulates that the characteristic jump value of blinking is 1.5 and the jump duration is 0.2 seconds. It is detected that the driver's eye behavior feature jump value is 1.6 and the jump duration is 0.25 seconds, so it is determined that the driver has blinked at this time. For example, the facial behavior calibration data stipulates that the jump value of yawning is 2.5 and the jump duration is 2 seconds. During the analysis, it was found that the jump value of the driver's mouth opening and closing feature is 2.6 and the duration is 2.1 seconds, which meets the calibration data of yawning. Therefore, judging this behavior as yawning may indicate that the driver is in a fatigued state. For example, the facial behavior calibration data stipulates that eyes closed for more than 3 seconds and a jump value of more than 2.0 is considered to be fatigue eye-closing behavior. If the jump value of the driver's eye closure feature is detected to be 2.2 and the duration is 4 seconds, it is determined that the driver is in a fatigue eye-closing state.

[0098] Step 108, determining the driver's fatigue evaluation level based on the facial behavior determination result, the real-time collected vehicle status and driving time.

[0099] In practice, the driver's fatigue level is determined and specific warnings or suggestions are given by combining the facial behavior judgment results, vehicle status (such as vehicle speed, number of brakes, etc.) and driving time through preset rules or models. For example, the driver's fatigue level can be judged through a variety of combinations. The following are several typical scenarios and corresponding fatigue level assessments: (1) The facial behavior judgment results show slight fatigue, the vehicle status is normal, and the driving time is short. The system detects that the driver's eye opening and closing frequency increases slightly, but does not remain closed for too long. The mouth opening frequency is also within the normal range. The vehicle speed is stable and the vehicle is running smoothly. There is no abnormal operation, such as sudden braking or acceleration. The driving time is 30 minutes. Since the facial behavior characteristics only show slight fatigue, and the vehicle status and driving time are within the normal range, the driver's fatigue level is assessed as "low", and the driver is advised to continue to pay attention. (2) The facial behavior judgment results show moderate fatigue, the vehicle status is slightly abnormal, and the driving time is moderate. It is detected that the driver's eye closing time interval gradually increases, the blinking frequency is abnormal, and the driver occasionally yawns. During the driving process, the vehicle speed was detected to fluctuate slightly, and there were several sudden speed slowdowns, which seemed to indicate that the driver was distracted or had lost focus. The driver had been driving for 1.5 hours. The facial behavior characteristics and vehicle status both showed that the driver was tired. Combined with the driving time, the system assessed the fatigue level to be "medium", and recommended the driver to take a break at an appropriate time. (3) The facial behavior judgment results showed severe fatigue, the vehicle status was significantly abnormal, and the driving time was too long. It was detected that the driver's eyes were closed for too long, and the closure duration exceeded more than 2 seconds. The blinking frequency increased significantly, the eye opening degree decreased significantly, and the mouth opened widely several times. During the driving process, multiple abnormal braking and acceleration operations were detected, the lane deviation was frequent, and the vehicle speed fluctuated greatly. The driver may have been distracted several times. The driver had been driving for more than 3 hours without taking a break. Comprehensively judging that the facial behavior characteristics and vehicle status were highly abnormal, and the driving time had exceeded the safety range, the driver's fatigue level was assessed to be "high", and it was strongly recommended that the driver stop and rest immediately, and the vehicle alarm system was triggered to remind the driver. (4) The facial behavior judgment results showed normal, the vehicle status was abnormal, and the driving time was short. The driver's facial behavior characteristics are normal, and the eye opening and closing time, blinking frequency and mouth opening are all within the normal range. Multiple sudden acceleration and braking operations were detected, the vehicle speed was unstable, and the vehicle had a tendency to deviate from the lane. The driving time was only 45 minutes. Although the facial behavior judgment result was normal, the vehicle status was abnormal. The system assessed that the driver might be distracted or make operational errors. It is recommended that the driver concentrate or adjust the driving status appropriately. (5) The facial behavior judgment result showed moderate fatigue, the vehicle status was normal, and the driving time was too long. It was detected that the driver's blinking frequency increased, the eye opening and closing time was longer, and there was a slight sign of yawning at the mouth. The vehicle speed remained stable, there was no abnormal operation, and the vehicle status was normal.The driver has been driving for 2.5 hours without any breaks. Although the vehicle is in good condition, the system assesses it as "medium to high fatigue level" considering the signs of fatigue in facial behavior and the long driving time, and recommends the driver to rest as soon as possible. Through these scenarios, the vehicle can make accurate fatigue level assessments based on different combinations of variables (facial behavior, vehicle status, driving time), and give corresponding driving safety recommendations.

[0100] As an optional implementation, if the fatigue evaluation level is low, the driver is determined to be in a non-fatigue state. If the fatigue evaluation level is medium, the driver is determined to be in a non-focused state, and a focus reminder is sent to the driver. If the fatigue evaluation level is high, the driver is determined to be in a fatigue state, and a fatigue reminder is sent to the driver.

[0101] In practice, based on the facial behavior judgment results and vehicle status data, the system will grade the driver's fatigue status, usually divided into low, medium and high levels. Each level corresponds to a different driver state and the corresponding system response. The low level indicates that the driver is in a normal mental state without obvious fatigue or distraction. The medium level means that the driver may be distracted, but has not yet entered a fatigue state. At this time, the system will issue a focus reminder to help the driver regain focus. The high level indicates that the driver has obvious fatigue and may pose a threat to driving safety. Therefore, the system will issue a fatigue reminder and require the driver to take measures such as rest to avoid accidents. For example: If the driver's facial behavior feature jump value and jump duration are small, and there is no long-term eye closure, yawning or other signs of fatigue, the system will set the fatigue evaluation level to a low level and determine that the driver is in a non-fatigue state. If the driver blinks frequently or turns his head many times in a short period of time, but does not close his eyes for a long time or yawn continuously. The system sets this state to a medium level and issues a focus reminder, such as a voice prompt or displaying "Please focus" on the in-vehicle screen. This reminder is intended to help the driver refocus and avoid further distraction or fatigue. For example, if the jump value and duration of the driver's facial behavior characteristics obviously exceed the calibration value, such as yawning continuously, closing the eyes for more than 3 seconds or not concentrating for a long time, the system will set the fatigue evaluation level to a high level and issue a fatigue reminder, which may include a strong voice warning, seat vibration or other emergency reminders, prompting the driver to stop and rest immediately to prevent accidents caused by fatigue driving.

[0102] As an optional implementation, during long-distance driving, the fatigue assessment threshold can be dynamically adjusted, and the driver's status can be comprehensively evaluated based on driving time and driving conditions (such as night driving or high-speed driving). Fatigue reminders may be enhanced or upgraded based on the driver's response. If the driver ignores the medium-level reminder and continues to show signs of fatigue, the system will upgrade to a high-level reminder and may suggest that the driver stop and rest at a nearby rest area.

[0103] As an optional implementation, Figure 6 A flowchart of an example of a fatigue driving monitoring method provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the specific steps are as follows:

[0104] Step 601, facial information recording: After the driver gets on the bus, the driver's facial key points are identified, including B-spline curve fitting for the mouth and eye key points. After removing the blinking frames, the average eye and mouth opening degree is calculated as the driver's facial behavior feature threshold.

[0105] Step 602, facial key point recognition: After the driver triggers the fatigue driving detection function, the facial key point detection model starts to recognize each frame of data from the infrared camera and obtains facial key point data for each frame.

[0106] Step 603, mouth and eye change rate calculation: B-spline curve fitting is performed on the eye and mouth key points in each frame image to calculate the degree of opening and closing. The degree of opening and closing of the current frame is compared with the previously set facial behavior feature threshold to calculate the change rate.

[0107] Step 604, curve filtering correction: the calculated change rate and time are plotted as a change curve, and the curve is corrected by a Kalman filter to eliminate errors.

[0108] Step 605, threshold determination: Determine based on pre-calibrated eye and mouth thresholds (such as normal eye opening, blinking, fatigue eye closing, talking, yawning, etc.) Compare the size and duration of the jump point with the calibrated threshold to determine the current driver's facial behavior.

[0109] Step 606, fatigue determination: Based on the determination result of facial behavior, combined with the driving condition and driving time, the driver's fatigue level is comprehensively evaluated, and corresponding reminders or warnings are given.

[0110] The embodiment of the present application provides a method for monitoring fatigue driving, the method comprising: collecting a first facial image of a driver within a preset time after getting on the vehicle in real time. Determine the threshold of the facial behavior feature of the driver according to the first facial image. Collect the second facial image of the driver in real time during driving. Calculate the real-time facial behavior feature value corresponding to each frame of the second facial image. Determine the time curve of the facial behavior feature change rate according to the facial behavior feature threshold of the driver and the real-time facial behavior feature value corresponding to each frame of the second facial image. In the time curve of the facial behavior feature change rate, determine the facial behavior feature jump value and the jump duration within the preset time range before and after each moment. Output the facial behavior determination result of the driver according to the facial behavior feature jump value, the jump duration and the preset facial behavior calibration data. Determine the fatigue evaluation level of the driver by combining the facial behavior determination result, the real-time collected vehicle state and the driving time. The fatigue driving monitoring method provided by the embodiment of the present application has the following beneficial effects: by collecting the first facial image of the driver within a preset time after getting on the vehicle in real time, the facial behavior feature threshold can be determined based on the individual characteristics of the driver. This personalized setting avoids the limitations of unified standards and can more accurately adapt to the differences in facial behavior characteristics of different drivers, thereby improving the accuracy of fatigue detection. During driving, the second facial image of the driver is collected in real time, and the facial behavior feature values ​​corresponding to each frame image are calculated. By monitoring the changes in the driver's facial behavior in real time, dangerous states such as fatigue or distraction can be captured instantly to ensure safety during driving. By determining the time curve of the facial behavior feature change rate, the changes in the driver's facial behavior can be dynamically monitored to provide a time evolution analysis of the driver's state. This dynamic analysis method can better judge the development trend of fatigue or inattention, rather than making a judgment based solely on a single behavior at a certain moment. On the time curve of the facial behavior feature change rate, the jump value and jump duration of the facial behavior feature can be identified and compared with the preset facial behavior calibration data. This method can accurately distinguish different types of facial behavior changes, such as blinking, yawning, etc., thereby providing a more detailed basis for fatigue state determination. In addition, the embodiment of the present application not only takes into account the changes in facial behavior, but also comprehensively analyzes factors such as vehicle status and driving time, and determines the driver's fatigue evaluation level through the combination of multi-dimensional information. The embodiment of the present application makes a more comprehensive and reasonable fatigue assessment based on actual driving conditions, changes in the driver's state, etc., thereby improving the applicability and accuracy of the system. Through the comprehensive assessment results, reminders of different intensities can be provided according to different fatigue evaluation levels. For example, low-level reminders help keep the driver focused, while high-level reminders promptly warn the driver of potential fatigue risks to ensure driving safety.In summary, the fatigue driving monitoring method of the embodiment of the present application can accurately and effectively identify and remind the driver of the fatigue state and improve driving safety through personalized threshold setting, real-time monitoring, dynamic analysis and multi-dimensional comprehensive evaluation.

[0111] It should be understood that although Figures 1 to 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 1 to 6 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0112] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0113] In one embodiment, a computer device is provided, such as Figure 7 As shown, it includes a memory and a processor, the memory stores a computer program that can be run on the processor, and the processor implements the above-mentioned method steps for fatigue driving monitoring when executing the computer program.

[0114] In one embodiment, a computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned method for fatigue driving monitoring when executed by a processor.

[0115] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0116] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0117] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0118] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0119] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for monitoring fatigue driving, characterized in that: The method comprises: Real-time collection of the first facial image of the driver within a preset time after he gets on the vehicle; Determining a facial behavior feature threshold of the driver according to the first facial image; Acquiring a second facial image of the driver in real time during driving; Calculate the real-time facial behavior feature value corresponding to the second facial image in each frame; Determining a facial behavior feature change rate time curve according to the facial behavior feature threshold of the driver and the real-time facial behavior feature value corresponding to each frame of the second facial image; In the facial behavior feature change rate time curve, determine the facial behavior feature jump value and jump duration within a preset time range before and after each moment; Outputting the driver's facial behavior determination result according to the facial behavior feature jump value, the jump duration and preset facial behavior calibration data; The driver's fatigue evaluation level is determined by comprehensively considering the facial behavior determination result, the vehicle status and driving time collected in real time.

2. The method according to claim 1, characterized in that The step of determining the facial behavior feature threshold of the driver according to the first facial image includes: For each frame of the first facial image, identifying facial key points in the first facial image of the frame; Performing B-spline curve fitting on the facial key points to calculate the degree of facial behavior opening and closing corresponding to the first facial image of the frame; The average value of the facial behavior opening and closing degree corresponding to each frame of the first facial image is determined as the facial behavior feature threshold of the driver.

3. The method according to claim 1, characterized in that The calculating the real-time facial behavior feature value corresponding to the second facial image of each frame includes: For each frame of the second facial image, identifying facial key points in the second facial image of the frame; B-spline curve fitting is performed on the facial key points to calculate the facial behavior opening and closing degree corresponding to the second facial image of the frame, and the facial behavior opening and closing degree is determined as the real-time facial behavior feature value corresponding to the second facial image of the frame.

4. The method according to claim 2 or 3, characterized in that: The facial key points include eye key points and mouth key points.

5. The method according to claim 1, characterized in that The step of determining a facial behavior feature change rate time curve according to the facial behavior feature threshold of the driver and the real-time facial behavior feature value corresponding to each frame of the second facial image comprises: For each frame of the second facial image, a ratio of the real-time facial behavior feature value corresponding to the frame of the second facial image to the facial behavior feature threshold of the driver is determined as a facial behavior feature change rate corresponding to the frame of the second facial image; The facial behavior feature change rate corresponding to each frame of the second facial image is fitted according to time to obtain the facial behavior feature change rate time curve.

6. The method according to claim 5, characterized in that The method further comprises: Kalman filtering is performed on the facial behavior feature change rate time curve.

7. The method according to claim 1, characterized in that Determining the facial behavior feature jump value and jump duration within a preset time range before and after each moment in the facial behavior feature change rate time curve includes: In the facial behavior feature change rate time curve, a point where the slope change is greater than a preset first slope change threshold is determined as a jump point; Determine the difference in the facial behavior feature change rate between the jump point and the low change point as the facial behavior feature jump value; The time difference between the adjacent jump and the low change point is determined as the duration of the jump.

8. The method according to claim 1, characterized in that The preset facial behavior calibration data includes standard eyes-open behavior calibration data, standard eyes-closed behavior calibration data, tired eyes-closed behavior calibration data, speaking behavior calibration data and yawning behavior calibration data.

9. The method according to claim 1, characterized in that: Outputting the driver's facial behavior determination result according to the facial behavior feature jump value, the jump duration and preset facial behavior calibration data includes: For the facial behavior calibration data corresponding to each facial behavior, if the facial behavior feature jump value and the jump duration meet the facial behavior calibration data corresponding to the facial behavior, the facial behavior is determined as the facial behavior judgment result of the driver.

10. The method according to claim 1, characterized in that The method further comprises: If the fatigue evaluation level is a low level, determining that the driver is in a non-fatigue state; If the fatigue evaluation level is medium, the driver is determined to be in a non-focused state, and a focus reminder is sent to the driver; If the fatigue evaluation level is a high level, it is determined that the driver is in a fatigue state, and a fatigue reminder is given to the driver.

Images