Method and device for detecting fatigue of driver

Abstract

The invention provides a method and a device for performing fatigue detection on a driver. The method comprises the following steps: acquiring a first facial feature index of the driver in a normal state; acquiring a second facial feature index and a vehicle driving state index of the driver in a vehicle driving state; and comparing the first facial feature index with the second facial feature index, determining a facial feature detection result, and then determining a final fatigue detection result of the driver based on the facial feature detection result and the vehicle driving state index. Through the method, fatigue detection can be carried out in combination with the facial feature index of the driver and the vehicle driving state index when the vehicle is driven, fatigue detection is carried out in combination with different dimensions, and the accuracy of fatigue detection can be improved. And various characteristic indexes of the driver in a normal state can be collected as a contrast, and fatigue detection is realized from the characteristic indexes of multiple dimensions, so that the accuracy of fatigue detection is further improved.

Description

Technical Field

[0001] The present application relates to the field of safe driving technology, and in particular to a method and device for detecting driver fatigue. Background Art

[0002] With the rapid development of transportation, the number of cars is increasing, and cars have become an important means of transportation for people. While driving a car brings convenience to people, it also causes many traffic safety problems. More and more traffic accidents have attracted people's attention. During driving, driver distraction and fatigue driving are one of the main causes of traffic accidents. Therefore, accurately detecting the driver's fatigue level and giving targeted reminders to the driver are of great significance to reducing the occurrence of traffic accidents and maintaining the safety of life and property.

[0003] At present, the main method for detecting driver fatigue is to analyze the driver's physiological state. For example, the driver's behavior of closing eyes and yawning is analyzed. However, driver fatigue detection from a single physiological dimension is prone to false detection and missed detection, resulting in low accuracy of driver fatigue detection. In addition, the physiological states of different drivers are also different. Using the same detection method for different drivers will also affect the accuracy of fatigue detection. Summary of the invention

[0004] In view of this, the present application is dedicated to providing a method and device for driver fatigue detection, so as to improve the accuracy of driver fatigue detection.

[0005] In a first aspect, the present application provides a method for detecting driver fatigue, the method comprising:

[0006] Acquire a first facial feature index of the driver in a normal state;

[0007] Acquire a second facial feature index of the driver when driving a vehicle and a vehicle driving state index, wherein the first facial feature index and the second facial feature index are indicators of the same type;

[0008] Comparing the first facial feature index and the second facial feature index to determine a facial feature detection result;

[0009] Based on the facial feature detection result and the vehicle driving state index, a fatigue detection result of the driver is determined.

[0010] In a possible implementation, determining the driver's fatigue detection result based on the facial feature detection result and the vehicle driving state indicator specifically includes:

[0011] Acquiring head feature indicators of the driver in the state of driving the vehicle;

[0012] Comparing the head feature index with the fatigue detection condition, determining within a detection time period a first ratio of a target frame image corresponding to the head feature index that meets the fatigue detection condition to all frame images within the detection time period;

[0013] The fatigue detection result is determined based on the first ratio, the facial feature detection result, and the vehicle driving state indicator.

[0014] In a possible implementation, the head feature index includes: at least one of a head posture estimation angle, a facial expression, or an eye occlusion result;

[0015] When the head characteristic index includes the head posture estimation angle, the fatigue detection conditions include: the head yaw angle is less than a first preset angle, the head upward angle is less than a second preset angle, the head downward angle is less than a third preset angle, and the head roll angle is less than a fourth preset angle;

[0016] When the head feature index includes the facial expression, the fatigue detection condition includes that the facial expression is not laughing or crying;

[0017] When the head feature index includes the eye occlusion result, the fatigue detection condition includes that both eyes are not occluded at the same time.

[0018] In a possible implementation, determining the fatigue detection result based on the first ratio, the facial feature detection result, and the vehicle driving state indicator specifically includes:

[0019] Comparing the vehicle driving state index with a vehicle driving state preset condition to determine a vehicle driving state detection result;

[0020] Determine a first weight corresponding to the facial feature detection result based on the first ratio, and determine a second weight corresponding to the vehicle driving state detection result based on the first ratio;

[0021] Based on the first weight and the second weight, a weighted sum is performed on the facial feature detection result and the vehicle driving state detection result to determine the fatigue detection result.

[0022] In one possible implementation, the vehicle driving state index includes a steering wheel angle, and the vehicle driving state preset conditions include: the proportion of zero steering wheel speed in a first preset time period is greater than a first preset value, the standard deviation of the steering wheel speed in a second preset time period is greater than a second preset value, or the ratio of the standard deviation of the steering wheel speed in a third preset time period to the mean of the steering wheel speed in the third preset time period is greater than at least one of a third preset value.

[0023] In a possible implementation, the first facial feature indicator includes at least one of a first eye closure degree, a first single eye closure duration, or a first blinking frequency, and comparing the first facial feature indicator with the second facial feature indicator to determine the facial feature detection result specifically includes:

[0024] Comparing the first facial feature index and the second facial feature index to determine an initial facial feature detection result;

[0025] Acquire a sight line vector of the driver in the state of driving the vehicle, wherein the sight line vector is used to determine whether the driver's eyes are in a dull state;

[0026] Determining a sluggishness detection result of the driver based on a plurality of the sight line vectors within a detection time period;

[0027] The facial feature detection result is determined based on the initial facial feature detection result and the stagnation detection result.

[0028] In a possible implementation, determining the driver's sluggishness detection result based on the plurality of sight line vectors within a detection time period specifically includes:

[0029] Calculating a deviation index between a sight line vector corresponding to a current frame image and sight line vectors corresponding to each frame image within a fourth preset time period before the current frame image;

[0030] When the deviation index is less than a fourth preset value, marking the current frame image;

[0031] Counting a second ratio of all marked images within the detection time period to all frame images within the detection time period;

[0032] The sluggishness detection result is determined based on the second ratio and the sluggishness base score.

[0033] In a possible implementation, the first facial feature indicator includes at least one of a first eye closure degree, a first single eye closure duration, or a first blinking frequency, and comparing the first facial feature indicator with the second facial feature indicator to determine the facial feature detection result specifically includes:

[0034] Comparing the first facial feature index and the second facial feature index to determine an initial facial feature detection result;

[0035] Acquire a first number of yawns of the driver in the driving vehicle state and within a detection time period, and a second number of yawns of the driver in the driving vehicle state and within a preset time period;

[0036] Determining a yawn detection result based on the accumulation result of the first yawn number and the second yawn number;

[0037] The facial feature detection result is determined based on the initial facial feature detection result and the yawn detection result.

[0038] In a possible implementation, comparing the first facial feature indicator and the second facial feature indicator to determine a facial feature detection result includes one or more of the following:

[0039] When the first facial feature indicator includes a first eye closure and the second facial feature indicator includes a second eye closure, determining a third ratio of a first target frame image whose first eye closure is less than a first preset closure within a target time period to all frame images within the target time period, and a fourth ratio of a second target frame image whose second eye closure is less than the first preset closure within a detection time period to all frame images within the detection time period; determining the facial feature detection result based on the third ratio and the fourth ratio; or,

[0040] When the first facial feature index includes a first single eye closure duration and the second facial feature index includes a second single eye closure duration, determining the facial feature detection result based on the first single eye closure duration and the second single eye closure duration; or,

[0041] When the first facial feature index includes a first blinking frequency and the second facial feature index includes a second blinking frequency, the facial feature detection result is determined based on the first blinking frequency and the second blinking frequency.

[0042] In a second aspect, the present application provides a device for detecting driver fatigue, the device comprising:

[0043] A first acquisition unit, used to acquire a first facial feature index of the driver in a normal state;

[0044] A second acquiring unit is used to acquire a second facial feature index of the driver when driving a vehicle and a vehicle driving state index, wherein the first facial feature index and the second facial feature index are indicators of the same type;

[0045] A facial detection unit, configured to compare the first facial feature index with the second facial feature index to determine a facial feature detection result;

[0046] A fatigue detection unit is used to determine the driver's fatigue detection result based on the facial feature detection result and the vehicle driving state index.

[0047] In a third aspect, the present application provides an electronic device, the device comprising: a memory and a processor;

[0048] The memory is used to store relevant program codes;

[0049] The processor is used to call the program code to execute the method for detecting driver fatigue as described in any one of the implementations of the first aspect above.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method for detecting driver fatigue as described in any one of the implementations of the first aspect above.

[0051] In a fifth aspect, the present application provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the method for detecting driver fatigue as described in any one of the implementation methods of the first aspect above is implemented.

[0052] In the above implementation of the present application, in order to perform fatigue detection on the driver, first obtain the first facial feature index of the driver in a normal state. Among them, the normal state can be understood as a non-fatigue state, and the first facial feature index can be obtained by pre-measurement and can be directly obtained during fatigue detection. Obtain the second facial feature index of the driver in the state of driving the vehicle and the vehicle driving state index, wherein the first facial feature index and the second facial feature index belong to the same type of index. Compare the first facial feature index and the second facial feature index to determine the facial feature detection result, that is, the fatigue detection result corresponding to the facial feature index. Then, based on the facial feature detection result and the vehicle driving state index, determine the final fatigue detection result of the driver. Through the method provided by the present application, fatigue detection can be performed in combination with the driver's facial feature index and the vehicle driving state index when driving the vehicle, and fatigue detection can be performed in combination with different dimensions, which can improve the accuracy of fatigue detection. In addition, various feature indicators of the driver in a normal state can be collected as a comparison to avoid feature differences between different drivers, and fatigue detection can be achieved from feature indicators of multiple dimensions, further improving the accuracy of fatigue detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. It is obvious that the drawings described below are only some embodiments provided in the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0054] Figure 1 A flow chart of a method for detecting driver fatigue provided in an embodiment of the present application.

[0055] Figure 2 A schematic diagram of face recognition provided in an embodiment of the present application.

[0056] Figure 3 A schematic diagram of key points of the eye provided in an embodiment of the present application.

[0057] Figure 4 A schematic diagram of a device for detecting driver fatigue provided in an embodiment of the present application.

[0058] Figure 5 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. The described embodiments are only exemplary implementation methods of the present application, not all implementation methods. Those skilled in the art can combine the embodiments of the present application to obtain other embodiments without creative work, and these embodiments are also within the scope of protection of the present application.

[0060] At present, when fatigue detection is performed on the driver's driving state, it is mainly based on the analysis of the driver's physiological state. For example, the driver's closing of eyes, yawning and other behaviors are analyzed. However, driver fatigue detection from a single physiological dimension is prone to false detection and missed detection, resulting in low accuracy of driver fatigue detection. In addition, the physiological states of different drivers are also different. Using the same detection method for different drivers will also affect the accuracy of fatigue detection.

[0061] Based on this, the embodiment of the present application provides a method for fatigue detection of a driver, which can improve the accuracy of fatigue detection of a driver. In specific implementation, a first facial feature index of the driver in a normal state is obtained. Among them, the normal state can be understood as the driver being in a non-fatigue state, and the first facial feature index can be obtained by pre-measurement, and the first facial feature index can be directly obtained during fatigue detection. The second facial feature index of the driver in the state of driving a vehicle and the vehicle driving state index are obtained, wherein the first facial feature index and the second facial feature index belong to the same type of index. The first facial feature index and the second facial feature index are compared to determine the facial feature detection result, that is, the fatigue detection result corresponding to the facial feature index. Then, based on the facial feature detection result and the vehicle driving state index, the final fatigue detection result of the driver is determined. Through the method provided by the present application, fatigue detection can be performed in combination with the driver's facial feature index and the vehicle driving state index when driving a vehicle, and fatigue detection can be performed in combination with different dimensions, which can improve the accuracy of fatigue detection. In addition, various feature indicators of the driver in a normal state can be collected as a reference, and fatigue detection can be realized from feature indicators of multiple dimensions, further improving the accuracy of fatigue detection.

[0062] In order to facilitate the understanding of the technical solutions provided by the embodiments of the present application, a detailed introduction will be given below in conjunction with the drawings in the specification.

[0063] See also Figure 1 As shown, Figure 1 A flow chart of a method for detecting driver fatigue provided in an embodiment of the present application.

[0064] The method may be performed by a processing device, which may be an electronic device or other device. Optionally, when the processing device is an electronic device, it may be a terminal device or a server. For example, the terminal device may be a device with processing function in a vehicle, but the embodiments of the present application do not limit this.

[0065] The method may include the following steps:

[0066] S101: Acquire a first facial feature index of the driver in a normal state.

[0067] The normal state can be understood as the driver being in a non-fatigue state. For example, the state of the driver after having enough sleep, or the state of the driver when driving a vehicle in a short time, etc. There is no limitation on the specific value of driving a vehicle in a short time, for example, it can be the state of driving a vehicle within one hour. Optionally, the first facial feature index can be obtained by combining multiple facial feature indexes in normal states. For example, the first facial feature index can be obtained by averaging multiple facial feature indexes in normal states. Thereby, the first facial feature index of the driver in a normal state is more accurate, and the accuracy of fatigue detection is improved.

[0068] Optionally, the first facial feature index may be a parameter representing a facial feature. For example, the first facial feature index may include at least one of a first eye closure degree, a first single eye closure duration, and a first blinking frequency. Among them, the first eye closure degree may be understood as the aspect ratio of the eye, and the specific calculation method may refer to the subsequent embodiments. Optionally, the degree of eye closure during normal eye opening is expressed as D, then D50 may be understood as the current eye closure degree of 50%, and D100 may be understood as the current eye closure degree of 100%, that is, completely closed eyes. The first single eye closure duration may be understood as the length of time from the start of the current eye closure to the end of the eye closure. The first blinking frequency may be understood as the number of blinks per unit time.

[0069] In a possible implementation, the vehicle's camera can be used to shoot a driver in a normal state, obtain multiple frames of images containing the driver's facial state, and then obtain the driver's first facial feature index from the image. Optionally, the acquisition frequency of the captured image can be pre-set, and the acquisition frequency can be understood as the number of image frames that need to be collected per unit time. Alternatively, multiple frames of images can be generated based on the video of the driver shot during the detection time period, and the first facial feature index corresponding to the driver can be obtained through each frame of the image. That is, the processing device can obtain the multiple frames of images captured by the camera, and obtain the driver's first facial feature index through the image.

[0070] It should be noted that the first facial feature index can be obtained by pre-measurement. That is, the first facial feature index is measured when the driver is in a normal state, and then the first facial feature index is stored. When fatigue detection is performed on the driver's state when driving the vehicle later, the pre-stored first facial feature index can be directly obtained without measuring again. Alternatively, the driver's facial feature index can be measured in a short period of time when the driver is driving the vehicle, as the first facial feature index of the driver in a normal state, and then the measured first facial feature index can be stored, so that the stored first facial feature index can be directly obtained when fatigue detection is performed on the driver later. In this way, there is no need to re-measure the first facial feature index each time, which can simplify the calculation procedure and improve the efficiency of fatigue detection.

[0071] Optionally, before each fatigue detection on the driver, the first facial feature index of the driver in a normal state (such as a state of driving a vehicle in a short period of time) can be remeasured. The normal state during the driving process is used as a reference to improve the accuracy of fatigue detection.

[0072] S102: Acquire a second facial feature index of the driver when driving a vehicle and a vehicle driving state index.

[0073] When the driver is driving the vehicle, the driver's second facial feature index and the vehicle driving state index of the driver can be obtained to detect the driver's fatigue. The vehicle driving state index can represent the driving state parameters of the vehicle when the driver is driving the vehicle. For example, the vehicle driving state index can include the steering wheel angle of the vehicle. The first facial feature index and the second facial feature index belong to the same type of index, and the two facial feature indexes are used as a comparison to obtain the fatigue detection result. The second facial feature index can refer to the introduction of the first facial feature index in the above embodiment.

[0074] When performing fatigue detection, the detection time period for fatigue detection can be pre-set. For example, the detection time period can be set to one minute, that is, the fatigue detection result is obtained based on the collected data within one minute. After starting to perform fatigue detection on the driver, fatigue detection can be performed on the driver continuously. That is, the driver's facial feature indicators are continuously collected, and each minute is used as a detection time period to obtain the driver's fatigue detection result corresponding to each detection time period. Optionally, the frame images collected in each detection time period may contain repeated frame images, or they may not overlap with each other. For example, one frame image can be used as an interval step to determine the next detection time period, that is, the step length of the sliding window is one frame image. It is also possible to use all frame images included in the detection time period as the interval step to determine the next detection time period, that is, the step length of the sliding window is one detection time period.

[0075] In a possible implementation, fatigue detection can be performed on the driver after the driver has driven the vehicle for more than a certain period of time. Since the driver is more likely to get tired when driving the vehicle for a long time, it is more meaningful to perform fatigue detection on the driver after a certain period of time. Among them, the embodiment of the present application does not limit the specific value of the certain period of time, and it can be set in combination with the actual application scenario. For example, it can be set to two hours. That is, after the driver has driven the vehicle for more than two hours, the second facial feature index of the driver and the vehicle driving state index can be obtained to perform fatigue detection.

[0076] S103: Compare the first facial feature index and the second facial feature index to determine a facial feature detection result.

[0077] By comparing the first facial feature index with the second facial feature index, and taking the facial feature index of the driver in a normal state as a reference, the degree of fatigue of the driver's driving state relative to the normal state can be more accurately reflected. And taking the facial feature index of the driver himself in a normal state as a reference avoids the feature differences between different drivers, which can make the fatigue detection of the driver more accurate. Among them, the facial feature detection result represents the fatigue detection result for the facial feature, which can reflect the degree of fatigue. For example, the facial feature detection result can be a fatigue detection score for the facial feature determined after comparing the first facial feature index and the second facial feature index.

[0078] S104: Determine the driver's fatigue detection result based on the facial feature detection result and the vehicle driving state index.

[0079] After obtaining the facial feature detection result, the facial feature detection result and the vehicle driving state index (i.e., the vehicle driving state parameter when the driver is driving the vehicle) are comprehensively considered to obtain the fatigue detection result, that is, the driver's fatigue level. By combining the driver's facial feature index and the vehicle driving state index when the driver is driving the vehicle for fatigue detection, fatigue detection is performed from different dimensions, which can improve the accuracy of fatigue detection.

[0080] Through the method provided in the embodiment of the present application, fatigue detection is performed in combination with the driver's facial feature indicators and the vehicle driving state indicators when driving the vehicle, that is, fatigue detection can be implemented from different dimensions, which can improve the accuracy of fatigue detection. By collecting various characteristic indicators of the driver in a normal state as a reference, the degree of fatigue of the driver's driving state relative to the normal state can be more accurately reflected. And the characteristic differences between different drivers are avoided, further improving the accuracy of fatigue detection.

[0081] In order to more clearly understand the technical solutions of the embodiments of the present application, each step in the above process and the technical effects that can be further produced will be described in detail below in combination with the embodiments.

[0082] The following first introduces step S103 “comparing the first facial feature index and the second facial feature index to determine the facial feature detection result” in detail in conjunction with the embodiment.

[0083] Specifically, the facial feature detection result can be determined after comparing the first facial feature index with the second facial feature index. Since facial feature indexes can include multiple types, they will be introduced below in combination with different facial feature indexes.

[0084] (i) Facial feature indicators include eye closure.

[0085] In a possible implementation, when the first facial feature indicator includes a first eye closure and the second facial feature indicator includes a second eye closure, the facial feature detection result can be determined in the following manner: a third ratio of a first target frame image whose first eye closure is less than a first preset closure in a target time period to all frame images in the target time period, and a fourth ratio of a second target frame image whose second eye closure is less than the first preset closure in a detection time period to all frame images in the detection time period are determined. Then, based on the third ratio and the fourth ratio, the facial feature detection result is determined.

[0086] Wherein, the target time period matches the detection time period, the target time period represents the time period when the driver is in a normal state, the detection time period represents the time period when the driver is driving the vehicle, and the target time period and the detection time period represent the same length of time. Wherein, the first preset closure degree can be used as a reference value for the eyes being in a fatigued state, and the first preset closure degree needs to be determined based on the eye closure degree when the driver opens his eyes normally in a normal state. Normal eye opening can be understood as the eyes being in a naturally open state, without trying to open or close the eyes. When the first eye closure degree is less than the first preset closure degree, it indicates that the current eye opening degree is small and the closing degree is large. That is, the cumulative time when the driver's eyes are closed more when driving the vehicle is greater than or equal to the cumulative time when the driver's eyes are closed more in a normal state.

[0087] In an embodiment of the present application, a method for determining the degree of eye closure of a driver when the driver's eyes are normally open is provided. Specifically, when obtaining facial feature indicators, it is first necessary to collect a facial image of the driver in a normal state, and then perform face recognition based on the facial image. Through face recognition, various key points of the face are extracted to obtain the driver's facial key point information, and then the facial feature indicators are obtained using the facial key point information.

[0088] Generally speaking, the currently commonly used face recognition algorithms (such as the Dlib algorithm) can identify 68 key points of the face. In the embodiment of the present application, the existing face recognition algorithm can be improved to increase the key point information of the face to improve the accuracy of face recognition, thereby improving the accuracy of determining facial feature indicators. In a possible implementation, the embodiment of the present application can perform face recognition on the driver's facial image to obtain 72 key points of the face. For details, please refer to Figure 2 As shown, Figure 2 A schematic diagram of face recognition provided in an embodiment of the present application.

[0089] Optionally, after obtaining the key point information of the face, the key point information of the eyes can also be obtained, so that facial feature indicators such as eye closure, blinking frequency, single blink duration, etc. can be obtained based on the key point information of the eyes.

[0090] See also Figure 3 , which is a schematic diagram of eye key points provided in an embodiment of the present application.

[0091] In this embodiment, the eye includes 8 key points, namely P1, P2, P3, P4, P5, P6, P7, and P8. Taking the eye state in this scene as the driver's eye state in a normal state, the eye closure degree D of the driver when the eyes are normally open can be determined based on the 8 key points of the eye. Specifically, it can be calculated in the following way: the opening height h of the eye is calculated by the height difference between each key point, and the opening width w of the eye is determined based on the maximum width between each key point, then:

[0092]

[0093] Wherein, |P3-P4| represents the height difference between key point P3 and key point P4, that is, the distance between them. That is, the average value of the height difference between each key point is calculated as the eye opening height.

[0094] Then calculate the ratio of the eye opening height to the eye opening width to obtain the eye closure degree D. That is, D = h / w.

[0095] According to the above embodiments, facial feature indicators can be obtained for each frame image of the driver collected. In one possible implementation, in order to improve the accuracy of determining the degree of eye closure of the driver when the eyes are normally open, the degree of eye closure of the driver when the eyes are normally open can be determined based on the multiple eye closure degrees by counting the degree of eye closure of the driver over a period of time under normal conditions.

[0096] In a specific implementation, when the driver is in a normal state, multiple eye closures within a period of time are obtained. Then the final eye closure is determined based on the multiple eye closures. That is, when the driver is in a normal state, multiple frames of images when the driver is normally open are collected within a period of time. Then, face recognition is performed on each frame of the image, and after obtaining the eye key point information, the eye closure corresponding to the current frame image can be obtained, so that the multiple eye closures corresponding to the multiple frames of the image within the period of time can be obtained.

[0097] After obtaining multiple eye closures, optionally, the average of the multiple eye closures can be calculated as the eye closure of the driver when the eyes are open normally. Alternatively, after obtaining the average of the multiple eye closures, the eye closures that are less than the average are removed first, and then the average of the remaining multiple eye closures is calculated as the eye closure of the driver when the eyes are open normally. By calculating by counting the multiple eye closures of the driver in a normal state, the accuracy of determining the eye closure when the eyes are open normally in a normal state can be improved.

[0098] The specific duration of the period of time can be determined by referring to the detection period during fatigue detection, and the embodiment of the present application does not limit the specific value of the period of time. For example, it can be set to one minute. That is, multiple frames of images of the driver in a normal state within one minute can be collected to obtain the eye closure degree of the driver when the eyes are normally open.

[0099] Since the eye closure represents the aspect ratio of the eye, and the width of the eye remains unchanged, the greater the eye closure, the greater the height of the eye opening, that is, the greater the degree of eye opening. The first eye closure is less than the first preset closure, which means that the degree of eye opening of the driver in a normal state is less than the degree of opening corresponding to the first preset closure. In specific implementation, all frame images of the driver in a normal state within a target time period can be obtained, and the first eye closure of the driver in each frame image can be calculated. Then, each frame image that satisfies the first eye closure is less than the first preset closure is determined as the first target frame image. The third ratio of the number of first target frame images to the number of all frame images in the target time period is calculated.

[0100] When the driver is driving the vehicle, all frame images of the driver in the detection time period are obtained, and the second eye closure degree of the driver in each frame image is calculated. Then, each frame image satisfying that the second eye closure degree is less than the first preset closure degree is determined as the second target frame image. A fourth ratio of the number of the second target frame images to the number of all frame images in the detection time period is calculated. Then, based on the third ratio and the fourth ratio, a facial feature detection result is determined.

[0101] It should be noted that the embodiment of the present application does not limit the specific value of the first preset closure degree. For example, the first preset closure degree can be set to half of the eye closure degree D when the eyes are normally open, or the first preset closure degree can be set to be completely closed. That is, the eye closure degree corresponding to the first preset closure degree can be expressed as D50 or D100.

[0102] In a possible implementation, in order to reduce the detection accuracy, when the eye closure is less than 0.3D, the eye can be considered to be in a completely closed state. That is, D100 can be considered to correspond to 0.3D. Wherein, D100 is a symbol mark, and 0.3D represents 0.3 times D. Wherein D can be calculated according to the above embodiment, that is, D=h / w (the ratio of the opening height h of the eye to the opening width w). That is, the threshold corresponding to D100 can be set to 0.3D. Based on this, it can be determined that the value corresponding to the first preset closure D100 is 0.3D.

[0103] In addition, an embodiment of the present application also provides a calculation method for determining D50. For example, the threshold value corresponding to D50 can be determined to be 0.3D+(D-0.3D)*0.5. Based on this, the value corresponding to the first preset closure degree D50 can be determined to be 0.3D+(D-0.3D)*0.5. After determining the first preset closure degree, the third ratio of the number of frame images whose first eye closure degree is less than the first preset closure degree to the number of all frame images in the target time period, and the fourth ratio of the number of frame images whose second eye closure degree is less than the first preset closure degree to the number of all frame images in the detection time period can be counted. Then, based on the third ratio and the fourth ratio, the facial feature detection result is determined.

[0104] In one possible implementation, the fatigue detection score corresponding to the facial feature index can be calculated as the facial feature detection result. Specifically, the difference between the fourth ratio and the third ratio can be calculated, and each percentage point in the difference can increase the preset fatigue detection score. Then, the facial feature detection result is obtained by adding the increased fatigue detection score to the basic score. For example, when the third ratio is less than the fourth ratio, the basic score can be set to 10 points, and 2 points are added for every percentage point difference between the third ratio and the fourth ratio, then the fatigue detection result = 10 + (fourth ratio - third ratio) / 0.01 * 2.

[0105] Optionally, when the fourth ratio is less than or equal to the third ratio, the fatigue detection score determined based on the third ratio and the fourth ratio may be set to zero.

[0106] It should be noted that the implementation method of determining the facial feature detection result based on the eye closure degree provided in the above embodiment is only an exemplary description and is not limited to the above implementation method.

[0107] (ii) Facial feature indicators include the duration of single eye closure.

[0108] The embodiment of the present application can determine the first single eye closure duration of the driver in a normal state and the second single eye closure duration when driving a vehicle. Specifically, the total duration of multiple blinks of the driver's eyes from reaching D100 to the end of D100 within a period of time under a normal state and the corresponding total number of times can be counted, and then the ratio of the total duration to the total number of times can be calculated to obtain the first single eye closure duration. Among them, D100 means that the degree of eye closure is 100%, that is, the eyes are completely closed. Among them, the duration of the time period from reaching D100 to the end of D100 can be determined by the images of adjacent frames.

[0109] Similarly, when the driver is driving the vehicle, the total duration and total number of blinks of the driver's eyes from reaching D100 to ending D100 are counted during the detection period, and then the ratio of the total duration to the total number is calculated to obtain the second single eye closure duration.

[0110] In one possible implementation, when the first facial feature indicator includes a first single eye closure duration and the second facial feature indicator includes a second single eye closure duration, the facial feature detection result can be determined based on the first single eye closure duration and the second single eye closure duration.

[0111] Specifically, the fatigue detection score can be calculated by calculating the difference between the second single eye closure duration and the first single eye closure duration, and the fatigue detection score can be used as the facial feature detection result. Specifically, when the difference between the second single eye closure duration and the first single eye closure duration increases by a preset duration, the preset fatigue detection score can be increased accordingly. That is, facial feature detection result = (second single eye closure duration - first single eye closure duration) / preset duration * preset fatigue detection score. For example, it can be set that when the difference between the second single eye closure duration and the first single eye closure duration increases by 100ms, the fatigue detection score increases by 10 points. Then the facial feature detection result = (second single eye closure duration - first single eye closure duration) ms / 100ms*10.

[0112] Optionally, when the second single eye-closing duration is less than or equal to the first single eye-closing duration, the fatigue detection score determined based on the first single eye-closing duration and the second single eye-closing duration may be set to 0.

[0113] It should be noted that the implementation method of determining the facial feature detection result based on a single eye closure duration provided in the above embodiment is only an exemplary description and is not limited to the above implementation method.

[0114] (iii) Facial feature indicators include blinking frequency.

[0115] The embodiment of the present application can determine the first blink frequency of the driver in a normal state and the second blink frequency when driving a vehicle. That is, the number of blinks of the driver in a unit time can be counted. Specifically, when the driver is in a normal state, the number of blinks in a period of time can be counted, and then the ratio of the number of blinks to the period of time can be calculated to obtain the first blink frequency. While the driver is driving the vehicle, the number of blinks in a detection time period is counted, and then the ratio of the number of blinks to the detection time period is calculated to obtain the second blink frequency. Among them, the embodiment of the present application does not limit the length of the period of time. For example, it can be determined with reference to the length of the detection time period.

[0116] In a possible implementation, when the first facial feature indicator includes a first blinking frequency and the second facial feature indicator includes a second blinking frequency, the facial feature detection result may be determined based on the first blinking frequency and the second blinking frequency.

[0117] Specifically, the fatigue detection score can be obtained by calculating the ratio of the second blinking frequency to the first blinking frequency, and the fatigue detection score can be used as the facial feature detection result. Specifically, the ratio of the second blinking frequency to the first blinking frequency is calculated. When the ratio is greater than the preset ratio, the ratio is multiplied by the preset fatigue detection score, and the product obtained is used as the facial feature detection result. For example, the preset ratio can be set to 1. That is, when the ratio of the second blinking frequency to the first blinking frequency is greater than 1, the driver's blinking frequency is relatively large at this time, and the driver may be in a fatigued state. At this time, the fatigue detection score can be determined based on the first blinking frequency and the second blinking frequency. Among them, the embodiment of the present application does not limit the specific value of the preset fatigue detection score, and can be set in combination with the actual application scenario. For example, the preset fatigue detection score can be set to 30 points.

[0118] Optionally, when the ratio of the second blinking frequency to the first blinking frequency is less than or equal to a preset ratio, the fatigue detection score determined based on the second blinking frequency and the first blinking frequency may be set to 0.

[0119] It should be noted that the implementation method of determining the facial feature detection result based on the blinking frequency provided in the above embodiment is only an exemplary description and is not limited to the above implementation method.

[0120] In the above embodiment, an implementation method for determining facial feature detection results based on three different facial feature indicators is provided, namely, eye closure, single eye closure duration, and blinking frequency. The facial feature detection result can be determined based on any one of the facial feature indicators. In addition, the embodiment of the present application can also arbitrarily combine the above three facial feature indicators to determine the facial feature detection result. Specifically, when the facial feature indicators simultaneously include any two or three of the eye closure, single eye closure duration, and blinking frequency, the corresponding fatigue detection score can be calculated for each facial feature indicator, and then the various fatigue detection scores can be summed up as the facial feature detection result of the driver. Among them, the embodiment of the present application does not limit the specific method for determining the weight of each facial feature indicator.

[0121] In a possible implementation, when the driver is driving the vehicle, if the direction of the driver's line of sight changes very little within a period of time, it indicates that the driver's eyes are likely to be in a dull state, that is, the driver may be in a fatigued state. Therefore, the embodiment of the present application can also perform fatigue detection in combination with the driver's line of sight. By combining indicators of multiple dimensions to determine the fatigue detection result, the accuracy of fatigue detection can be further improved.

[0122] In a possible implementation, determining the facial feature detection result may include the following steps:

[0123] B1: Compare the first facial feature index and the second facial feature index to determine an initial facial feature detection result.

[0124] The process of determining the initial facial feature result may refer to the above embodiment.

[0125] B2: Get the driver's sight line vector when driving the vehicle.

[0126] The sight line vector may be used to determine whether the driver's eyes are in a dull state.

[0127] Optionally, after acquiring the driver's facial image, a convolutional neural network may be used to recognize the image to obtain the driver's sight line vector. The convolutional neural network may be trained with sample images, each sample image including a human face image and annotated with a sight line vector, and used to train the convolutional neural network, so that the trained convolutional neural network can recognize facial images and output a sight line vector.

[0128] B3: Determine the driver's sluggishness detection result based on multiple sight line vectors within the detection time period.

[0129] Multiple frames of images can be collected during the detection period, and a sight vector can be obtained for each frame of image. The driver's sluggishness detection result can be obtained by counting the number of image frames in the sluggish state. The sluggishness detection result can be understood as the fatigue detection result obtained based on the sight vector, and the fatigue degree corresponding to the sluggish state.

[0130] In a possible implementation, for a current frame image within a detection time period, a sight line vector of the current frame image is obtained. And each frame image within a fourth preset time period before the current frame image is obtained to obtain a sight line vector corresponding to each frame image. Among them, the fourth preset time period before the current frame image is within the current detection time period. That is, the current frame image can represent each frame image after the fourth preset time period within the detection time period.

[0131] Then, a deviation index between the sight line vector corresponding to the current frame image and the sight line vectors corresponding to the frame images in the fourth preset time period before is calculated. The deviation index is used to represent the error between the sight line vectors, and the degree of change of the sight line vector (the driver's sight line direction) can be determined by the error. For example, the deviation index can be a variance or a standard deviation. That is, the variance or the standard deviation between the sight line vector corresponding to the current frame image and the sight line vectors corresponding to the frame images in the fourth preset time period before can be calculated. When the error is smaller, it indicates that the degree of change between the sight line vectors is smaller, and the driver's eyes are more likely to be in a sluggish state, that is, the driver is more likely to be in a fatigued state.

[0132] The embodiment of the present application does not limit the specific value of the fourth preset time period. For example, when the detection time period is 1 minute, the fourth preset time period can be set to 10 seconds, and the 10 seconds is 10 seconds within the detection time period. That is, each frame image within 10 seconds before the current frame image can be obtained, and then the deviation index between the current frame image and each frame image within the previous 10 seconds can be calculated.

[0133] In a possible implementation, when the deviation index is less than the fourth preset value, it indicates that the deviation of the sight line vector in each frame image is small, that is, the change in the driver's sight line direction is small, indicating that the driver is likely to be in a fatigued state, and the current frame image can be marked. The second ratio of the number of all marked images in the detection time period to the number of all frame images in the detection time period is counted. Among them, the second ratio can represent the proportion of the driver in a sluggish state during the detection time period. Then, based on the second ratio and the sluggish basic score, the sluggish detection result is determined. For example, the second ratio can be multiplied by the sluggish basic score, and the product obtained is used as the sluggish detection result.

[0134] In a possible implementation, before calculating the deviation index, the degree of eye closure of the driver in the current frame image may be calculated first. Only when the degree of eye closure is greater than the second preset degree of closure, the step of calculating the deviation index between the sight line vector corresponding to the current frame image and the sight line vectors corresponding to each frame image in the fourth preset time period before the current frame image is performed. When the degree of eye closure is greater than the second preset degree of closure, it can be understood that the degree of eye opening is relatively large, and the sight line vector measured at this time is relatively accurate, and the deviation index obtained based on this is also more accurate, thereby improving the accuracy of determining the fatigue detection result.

[0135] The embodiment of the present application does not limit the specific value of the second preset closure degree. For example, the second preset closure degree can be set to 0.8D.

[0136] B4: Determine facial feature detection results based on the initial facial feature detection results and the dullness detection results.

[0137] After obtaining the stagnation detection result, the facial feature detection result can be determined based on the initial facial feature detection result and the stagnation detection result. For example, the initial facial feature detection result and the stagnation detection result can be summed. The embodiment of the present application does not limit the specific method of determining the weights of the initial facial feature detection result and the stagnation detection result.

[0138] In one possible implementation, usually, when a driver yawns frequently, it indicates that the driver may be in a fatigued state. Therefore, the embodiment of the present application can also perform fatigue detection based on the number of times the driver yawns, and determine the fatigue detection result through indicators in multiple dimensions, which can further improve the accuracy of fatigue detection.

[0139] In a possible implementation, determining the facial feature detection result may include the following steps:

[0140] A1: Determine an initial facial feature detection result based on the first facial feature indicator and the second facial feature indicator.

[0141] The process of determining the initial facial feature result may refer to the above embodiment.

[0142] A2: Obtaining a first yawning number of the driver while driving the vehicle and within a detection time period, and a second yawning number of the driver while driving the vehicle and within a preset time period.

[0143] Optionally, even when the driver is in a fatigued state, the number of first yawns in the current detection time period may not be many, so the number of second yawns of the driver in the preset time period can be continuously counted as a reference to improve the accuracy of fatigue detection based on the number of yawns. That is, the length of the preset time period is greater than the length of the detection time period. For example, the detection time period can be set to one minute, and the preset time period can be set to ten minutes. Among them, the detection time period and the preset time period may have overlapping time periods, or may not have overlapping time periods. For example, the preset time period can be the front time period adjacent to the detection time period, or the rear time period adjacent to the detection time period.

[0144] A3: Determine a yawn detection result based on the accumulation result of the first yawn number and the second yawn number.

[0145] In a possible implementation, the first yawning frequency and the second yawning frequency may be multiplied to obtain a cumulative result, and then a yawn detection result is determined based on the cumulative result, wherein the yawn detection result may represent a fatigue detection result determined based on the yawning frequency.

[0146] Optionally, a fatigue detection score can be determined based on the cumulative result of the first yawn number and the second yawn number as the yawn detection result. Specifically, the product of the first yawn number and the second yawn number is first calculated, and then the product is multiplied by the preset fatigue detection score to obtain the yawn detection result. Among them, the embodiment of the present application does not limit the specific value of the preset fatigue detection score. For example, the preset fatigue detection score can be set to 5 points.

[0147] It should be noted that the implementation method for determining the cumulative result of the first yawn number and the second yawn number provided in the above embodiment is only an exemplary description and is not limited to the above possibilities. Other conceivable implementation methods are also within the scope of protection of this application. For example, the sum of the first yawn number and the second yawn number can be calculated as the cumulative result, and the yawn detection result can be determined based on the sum.

[0148] A4: Determine a facial feature detection result based on the initial facial feature detection result and the yawn detection result.

[0149] After obtaining the yawn detection result, the facial feature detection result can be determined based on the initial facial feature detection result and the yawn detection result. For example, the initial facial feature detection result and the yawn detection result can be summed. The embodiment of the present application does not limit the specific method of determining the weights of the initial facial feature detection result and the yawn detection result.

[0150] Based on the above embodiments, it can be known that the embodiments of the present application introduce a method for determining facial feature detection results based on five feature indicators, namely, eye closure, single eye closure duration, blinking frequency, and line of sight vector yawning times. In a possible implementation, the embodiments of the present application can combine any one or more of the above five feature indicators to determine the driver's facial feature detection results. After obtaining any one or more feature indicators, the detection results corresponding to each feature indicator are determined respectively, and then the detection results corresponding to each feature indicator are weighted and summed to obtain the driver's facial feature detection results.

[0151] It should be noted that the characteristic indicators provided in the embodiments of the present application are only exemplary and are not limited to the above-mentioned possibilities. Other achievable methods also fall within the protection scope of the present application. For example, characteristic indicators such as the number of times the driver nods can be obtained for fatigue detection.

[0152] The following will introduce in detail step S104 "determining the driver's fatigue detection result based on the facial feature detection result and the vehicle driving state index" in conjunction with the embodiment.

[0153] When the driver is driving the vehicle, the driver's facial feature detection result and the vehicle driving index can be combined to determine the driver's fatigue detection result. The vehicle driving state index can represent the driving state parameters of the vehicle when the driver is driving the vehicle. For example, the vehicle driving state index can include the steering wheel angle. When the driver is in a fatigued state, there may be a situation where the steering wheel is turned suddenly and significantly, so fatigue detection can be performed in combination with the steering wheel angle.

[0154] In specific implementation, after obtaining the vehicle driving state index, the vehicle driving state index can be compared with the vehicle driving state preset condition to determine the vehicle driving state detection result, that is, the fatigue degree determined based on the vehicle driving state index. Then, based on the facial feature detection result and the vehicle driving state detection result, the fatigue detection result is determined. Among them, the vehicle driving state preset condition is used to determine the fatigue detection result corresponding to the vehicle driving state index.

[0155] In one possible implementation, when the vehicle driving state indicator includes a steering wheel angle, the vehicle driving state preset conditions include: the proportion of zero steering wheel speed in a first preset time period is greater than a first preset value, the standard deviation of the steering wheel speed in a second preset time period is greater than a second preset value, or the ratio of the standard deviation of the steering wheel speed in a third preset time period to the mean of the steering wheel speed in the third preset time period is greater than at least one of the third preset value, wherein the first preset time period, the second preset time period, and the third preset time period are within the detection time period.

[0156] Among them, the steering wheel speed of zero speed indicates that the steering wheel has not turned. When the steering wheel is at zero speed for a long time, it indicates that the driver may be in a fatigued state and therefore did not turn the steering wheel. When the driver is driving the vehicle normally, except for the scene of turning, the steering wheel is rarely turned significantly. Generally, the change of the steering wheel speed is relatively small. Therefore, when the standard deviation of the steering wheel speed is large over a period of time, it indicates that the driver may be in a fatigued state, resulting in a large turn of the steering wheel. The ratio of the standard deviation of the steering wheel speed to the mean of the steering wheel speed is calculated to exclude the scene of turning, because in the turning scene, the mean of the steering wheel speed is relatively large, and the standard deviation of the steering wheel speed is relatively small, so the ratio of the standard deviation of the steering wheel speed to the mean of the steering wheel speed is also relatively small. When the ratio of the standard deviation of the steering wheel speed to the mean of the steering wheel speed is less than the third preset value, it can be considered that the driver is driving the vehicle to turn at this time and is not in a fatigued state. That is, when the ratio of the standard deviation of the steering wheel speed to the mean of the steering wheel speed is greater than the third preset value, it meets the condition that the driver is in a fatigued state.

[0157] It should be noted that the first preset time period, the second preset time period, and the third preset time period may be the same time period or different time periods, which does not affect the implementation of the embodiment of the present application. For example, when the detection time period is one minute, the first preset time period may be set to the first 5 seconds of every 10 seconds, and the second preset time period and the third preset time period may be set to the last 5 seconds of every 10 seconds.

[0158] In a possible implementation, the vehicle driving state detection result can be determined in the following manner: the vehicle driving state index is compared with the vehicle driving state preset condition, and the number of times the vehicle driving state index meets the vehicle driving state preset condition within the detection time period is determined. Then, based on the number and the vehicle driving state basic score, the vehicle driving state detection result is determined. For example, the number and the vehicle driving state basic score can be multiplied, and the product obtained is used as the vehicle driving state detection result.

[0159] Taking the detection time period of one minute as an example, every 10 seconds is regarded as a time period, and the vehicle driving state indicators within the time period are compared to see whether they meet the preset conditions of the vehicle driving state. The vehicle driving state detection result is determined based on the six comparison results within one minute and the basic score of the vehicle driving state.

[0160] After obtaining the facial feature detection results and the vehicle driving state detection results, the facial feature detection results and the vehicle driving state detection results can be weighted to determine the driver's fatigue detection result. The embodiment of the present application provides a method for determining the weight of the facial feature detection results and the weight of the vehicle driving state detection results, which will be specifically introduced in conjunction with the embodiment below.

[0161] In the process of the driver driving the vehicle, the head feature index of the driver in the driving state can be obtained, wherein the head feature index is used to characterize whether the second facial feature index is used for facial fatigue detection accurately. For example, when the head feature index meets the fatigue detection condition, it indicates that the accuracy of facial fatigue detection based on the second facial feature index is high. Thus, the fatigue detection result can be determined with reference to the head feature index. Specifically, the head feature index and the fatigue detection condition can be compared to determine the target frame image corresponding to the head feature index that meets the fatigue detection condition within the detection time period and the first ratio of all frame images within the detection time period. That is, all frame images within the detection time period can be obtained, and the driver's head feature index can be obtained for each frame image, and the head feature index is compared with the fatigue detection condition to obtain the target frame image corresponding to the head feature index that meets the fatigue detection condition, and the first ratio of the number of target frame images to all frame images within the detection time period is calculated. Then, based on the first ratio, the facial feature detection result and the vehicle driving state index, the fatigue detection result is determined. Among them, the fatigue detection condition can be pre-set for the head feature index, indicating the condition satisfied by the head feature index when fatigue detection is performed on the driver.

[0162] In one possible implementation, when the head feature index meets the fatigue detection condition, it indicates that the accuracy of facial fatigue detection based on the second facial feature index is high at this time, and the first weight corresponding to the facial feature detection result can be determined based on the above-mentioned first ratio. When the head feature index does not meet the fatigue detection condition, that is, the accuracy of fatigue detection based on the second facial feature index is low, then the accuracy of fatigue detection based on the vehicle driving state index is higher at this time. Then, the second weight corresponding to the vehicle driving state detection result can be determined based on the first ratio. Then, based on the first weight and the second weight, the facial feature detection result and the vehicle driving state detection result are weighted and summed to determine the fatigue detection result.

[0163] For example, the first ratio can be determined as the first weight corresponding to the facial feature detection result. The second weight corresponding to the vehicle driving status detection result is determined based on the difference between 1 and the first weight. When the first ratio is larger, the first weight of the facial feature detection result determined based on the second facial feature index is also larger. When the first ratio is smaller, the smaller the first weight corresponding to the facial feature detection result is, and the larger the second weight corresponding to the vehicle driving status detection result is. Since the head feature index can characterize whether the second facial feature index is accurate for facial fatigue detection, the accuracy of fatigue detection can be improved by comparing the head feature index with the fatigue detection condition and determining the weights of the facial feature detection result and the vehicle driving status detection result.

[0164] In a possible implementation, the head feature index may include at least one of a head posture estimation angle, a facial expression, or an eye occlusion result. Different head feature indexes have different corresponding fatigue detection conditions, which will be introduced below in different situations.

[0165] In a possible implementation, when the head feature index includes a head posture estimation angle, the head posture estimation angle may include a head yaw angle, a head pitch angle (head looking down angle and head looking up angle) and a head roll angle. Among them, the head posture estimation angle may be the deflection angle of the driver relative to the vehicle's camera. At this time, the fatigue detection conditions include: the head yaw angle is less than a first preset angle, the head looking up angle is less than a second preset angle, the head looking down angle is less than a third preset angle, and the head roll angle is less than a fourth preset angle. When the driver is driving the vehicle, if the angle of the head rotation is too large, the facial features captured will not be comprehensive and accurate enough, affecting the accuracy of fatigue detection. Therefore, it is necessary to set a threshold for head rotation as a fatigue detection condition.

[0166] The embodiment of the present application does not limit the specific values ​​of the first preset angle, the second preset angle, the third preset angle, and the fourth preset angle, and can be set in combination with actual application scenarios. For example, the first preset angle can be set to 50°, the second preset angle to 20°, the third preset angle to 50°, and the fourth preset angle to 20°.

[0167] In a possible implementation, when the head feature index includes facial expression, the fatigue detection condition includes that the facial expression is not laughing or crying. That is, when the driver's facial expression is laughing or crying, it indicates that the fatigue detection condition is not met at this time. Among them, the neural network model can be used to realize the expression detection of the driver, which will not be described in detail in the embodiment of the present application.

[0168] In a possible implementation, when the head feature index includes an eye occlusion result, the fatigue detection condition includes that both eyes are not occluded at the same time. That is, when both eyes of the driver are occluded at the same time, the fatigue detection condition is not met. The eye occlusion result can be determined by key point information of the face (key point information of the eyes).

[0169] Through the method provided in the embodiment of the present application, fatigue detection can be performed by combining the driver's facial feature index and the vehicle driving state index when driving the vehicle. Combining different dimensions for fatigue detection can improve the accuracy of fatigue detection. In addition, the driver's facial feature index in a normal state can be collected as a reference to avoid feature differences between different drivers, further improving the accuracy of fatigue detection.

[0170] Based on the above method embodiment, the present application embodiment also provides a device for detecting driver fatigue. Figure 4 As shown, it is a schematic diagram of a device for detecting driver fatigue provided in an embodiment of the present application.

[0171] The device 400 comprises:

[0172] A first acquisition unit 401 is used to acquire a first facial feature index of the driver in a normal state;

[0173] A second acquisition unit 402 is used to acquire a second facial feature index of the driver when driving a vehicle and a vehicle driving state index, wherein the first facial feature index and the second facial feature index are of the same type;

[0174] A facial detection unit 403, configured to compare the first facial feature index with the second facial feature index to determine a facial feature detection result;

[0175] The fatigue detection unit 404 is used to determine the driver's fatigue detection result based on the facial feature detection result and the vehicle driving state index.

[0176] In one possible implementation, the fatigue detection unit 404 is specifically used to obtain the head feature indicators of the driver in the driving vehicle state; compare the head feature indicators with the fatigue detection conditions, and determine a first ratio of the target frame image corresponding to the head feature indicators that meet the fatigue detection conditions within the detection time period to all frame images within the detection time period; determine the fatigue detection result based on the first ratio, the facial feature detection result and the vehicle driving state indicator.

[0177] In one possible implementation, the head feature indicators include: at least one of: a head posture estimation angle, a facial expression, or an eye occlusion result; when the head feature indicators include the head posture estimation angle, the fatigue detection conditions include: a head yaw angle less than a first preset angle, a head upward angle less than a second preset angle, a head downward angle less than a third preset angle, and a head roll angle less than a fourth preset angle; when the head feature indicators include the facial expression, the fatigue detection conditions include that the facial expression is not laughing or crying; when the head feature indicators include the eye occlusion result, the fatigue detection conditions include that both eyes are not occluded at the same time.

[0178] In one possible implementation, the fatigue detection unit 404 is specifically used to compare the vehicle driving state index with a preset vehicle driving state condition to determine a vehicle driving state detection result; determine a first weight corresponding to the facial feature detection result based on the first ratio, and determine a second weight corresponding to the vehicle driving state detection result based on the first ratio; and perform weighted summation of the facial feature detection result and the vehicle driving state detection result based on the first weight and the second weight to determine the fatigue detection result.

[0179] In one possible implementation, the vehicle driving state index includes a steering wheel angle, and the vehicle driving state preset conditions include: the proportion of zero steering wheel speed in a first preset time period is greater than a first preset value, the standard deviation of the steering wheel speed in a second preset time period is greater than a second preset value, or the ratio of the standard deviation of the steering wheel speed in a third preset time period to the mean of the steering wheel speed in the third preset time period is greater than at least one of a third preset value.

[0180] In a possible implementation, the first facial feature indicator includes: at least one of a first eye closure degree, a first single eye closure duration, or a first blinking frequency, and the facial detection unit 403 is specifically used to compare the first facial feature indicator with the second facial feature indicator to determine an initial facial feature detection result; obtain the sight line vector of the driver in the driving vehicle state, and the sight line vector is used to determine whether the driver's eyes are in a dull state; determine the driver's dullness detection result based on multiple sight line vectors within a detection time period; determine the facial feature detection result based on the initial facial feature detection result and the dullness detection result.

[0181] In one possible implementation, the facial detection unit 403 is specifically used to calculate a deviation index between a sight line vector corresponding to a current frame image and sight line vectors corresponding to frame images within a fourth preset time period before the current frame image; when the deviation index is less than a fourth preset value, mark the current frame image; count a second ratio of all marked images within the detection time period to all frame images within the detection time period; and determine the stagnation detection result based on the second ratio and the stagnation basic score.

[0182] In a possible implementation, the first facial feature indicator includes: at least one of a first eye closure degree, a first single eye closure duration, or a first blinking frequency. The facial detection unit 403 is specifically used to compare the first facial feature indicator with the second facial feature indicator to determine an initial facial feature detection result; obtain the first yawn number of the driver in the driving vehicle state and within the detection time period, and the second yawn number of the driver in the driving vehicle state and within a preset time period; determine a yawn detection result based on the cumulative result of the first yawn number and the second yawn number; determine the facial feature detection result based on the initial facial feature detection result and the yawn detection result.

[0183] In a possible implementation, the facial detection unit 403 is specifically used to determine, when the first facial feature indicator includes a first eye closure degree and the second facial feature indicator includes a second eye closure degree, a third ratio of a first target frame image whose first eye closure degree is less than a first preset closure degree during a target time period to all frame images within the target time period, and a fourth ratio of a second target frame image whose second eye closure degree is less than the first preset closure degree during a detection time period to all frame images within the detection time period; determine the facial feature detection result based on the third ratio and the fourth ratio; or, when the first facial feature indicator includes a first single eye closure duration and the second facial feature indicator includes a second single eye closure duration, determine the facial feature detection result based on the first single eye closure duration and the second single eye closure duration; or, when the first facial feature indicator includes a first blinking frequency and the second facial feature indicator includes a second blinking frequency, determine the facial feature detection result based on the first blinking frequency and the second blinking frequency.

[0184] Based on the above method embodiment and device embodiment, the present application embodiment further provides an electronic device, which will be described below in conjunction with the accompanying drawings.

[0185] See also Figure 5 , Figure 5 A schematic diagram of an electronic device provided in an embodiment of the present application.

[0186] The device 500 includes: a memory 501 and a processor 502;

[0187] The memory 501 is used to store relevant program codes;

[0188] The processor 502 is used to call the program code to execute the method for detecting driver fatigue as described in the above method embodiment.

[0189] In addition, an embodiment of the present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the method for detecting driver fatigue as described in the above method embodiment.

[0190] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the method for detecting driver fatigue as described in the above method embodiment is implemented.

[0191] It should be noted that the technical features of the higher-level means provided in the embodiments of the present application are clear to those skilled in the art, and the problems to be solved by the higher-level means are also clear. The means for obtaining the corresponding features can be selected by those skilled in the art according to specific implementation requirements. The means provided in this application should not be regarded as a limitation on the solution or as the only means of implementation.

[0192] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can refer to each other. In particular, for the system or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only exemplary, in which the units or modules described as separate components may or may not be physically separated, and the components displayed as units or modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units, and some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0193] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the method, device and equipment etc. of various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0194] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0195] It should also be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0196] The steps of the method or algorithm described in conjunction with the embodiments disclosed in this application can be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0197] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined in this application can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown in the present application, but will conform to the widest range consistent with the principles and novel features disclosed in the present application.

Claims

1. A method for detecting driver fatigue, characterized in that: The method comprises: Acquire a first facial feature index of the driver in a normal state; Acquire a second facial feature index of the driver when driving a vehicle and a vehicle driving state index, wherein the first facial feature index and the second facial feature index are indicators of the same type; Comparing the first facial feature index and the second facial feature index to determine a facial feature detection result; Based on the facial feature detection result and the vehicle driving state index, a fatigue detection result of the driver is determined.

2. The method according to claim 1, characterized in that The step of determining the driver's fatigue detection result based on the facial feature detection result and the vehicle driving state indicator includes: Acquiring head feature indicators of the driver in the state of driving the vehicle; Comparing the head feature index with the fatigue detection condition, determining within a detection time period a first ratio of a target frame image corresponding to the head feature index that meets the fatigue detection condition to all frame images within the detection time period; The fatigue detection result is determined based on the first ratio, the facial feature detection result, and the vehicle driving state indicator.

3. The method according to claim 2, characterized in that The head feature index includes: at least one of head posture estimation angle, facial expression or eye occlusion result; When the head characteristic index includes the head posture estimation angle, the fatigue detection conditions include: the head yaw angle is less than a first preset angle, the head upward angle is less than a second preset angle, the head downward angle is less than a third preset angle, and the head roll angle is less than a fourth preset angle; When the head feature index includes the facial expression, the fatigue detection condition includes that the facial expression is not laughing or crying; When the head feature index includes the eye occlusion result, the fatigue detection condition includes that both eyes are not occluded at the same time.

4. The method according to claim 2, characterized in that: The step of determining the fatigue detection result based on the first ratio, the facial feature detection result, and the vehicle driving state indicator includes: Comparing the vehicle driving state index with a vehicle driving state preset condition to determine a vehicle driving state detection result; Determine a first weight corresponding to the facial feature detection result based on the first ratio, and determine a second weight corresponding to the vehicle driving state detection result based on the first ratio; Based on the first weight and the second weight, a weighted sum is performed on the facial feature detection result and the vehicle driving state detection result to determine the fatigue detection result.

5. The method according to claim 4, characterized in that The vehicle driving state index includes a steering wheel angle, and the vehicle driving state preset conditions include: a proportion of zero steering wheel speed in a first preset time period is greater than a first preset value, a standard deviation of the steering wheel speed in a second preset time period is greater than a second preset value, or a ratio of the standard deviation of the steering wheel speed in a third preset time period to the mean of the steering wheel speed in the third preset time period is greater than at least one of a third preset value.

6. The method according to claim 1, characterized in that The first facial feature indicator includes at least one of a first eye closure degree, a first single eye closure duration, or a first blinking frequency, and comparing the first facial feature indicator with the second facial feature indicator to determine a facial feature detection result includes: Comparing the first facial feature index and the second facial feature index to determine an initial facial feature detection result; Acquire a sight line vector of the driver in the state of driving the vehicle, wherein the sight line vector is used to determine whether the driver's eyes are in a dull state; Determining a sluggishness detection result of the driver based on a plurality of the sight line vectors within a detection time period; The facial feature detection result is determined based on the initial facial feature detection result and the stagnation detection result.

7. The method according to claim 6, characterized in that The determining of the driver's sluggishness detection result based on the plurality of sight line vectors within the detection time period includes: Calculating a deviation index between a sight line vector corresponding to a current frame image and sight line vectors corresponding to each frame image within a fourth preset time period before the current frame image; When the deviation index is less than a fourth preset value, marking the current frame image; Counting a second ratio of all marked images within the detection time period to all frame images within the detection time period; The sluggishness detection result is determined based on the second ratio and the sluggishness base score.

8. The method according to claim 1, characterized in that The first facial feature indicator includes at least one of a first eye closure degree, a first single eye closure duration, or a first blinking frequency, and comparing the first facial feature indicator with the second facial feature indicator to determine a facial feature detection result includes: Comparing the first facial feature index and the second facial feature index to determine an initial facial feature detection result; Acquire a first number of yawns of the driver in the driving vehicle state and within a detection time period, and a second number of yawns of the driver in the driving vehicle state and within a preset time period; Determining a yawn detection result based on the accumulation result of the first yawn number and the second yawn number; The facial feature detection result is determined based on the initial facial feature detection result and the yawn detection result.

9. The method according to claim 1, characterized in that: The comparing the first facial feature indicator and the second facial feature indicator to determine a facial feature detection result includes one or more of the following: When the first facial feature indicator includes a first eye closure and the second facial feature indicator includes a second eye closure, determining a third ratio of a first target frame image whose first eye closure is less than a first preset closure in a target time period to all frame images in the target time period, and a fourth ratio of a second target frame image whose second eye closure is less than the first preset closure in a detection time period to all frame images in the detection time period; determining the facial feature detection result based on the third ratio and the fourth ratio; or, When the first facial feature index includes a first single eye closure duration and the second facial feature index includes a second single eye closure duration, determining the facial feature detection result based on the first single eye closure duration and the second single eye closure duration; or, When the first facial feature index includes a first blinking frequency and the second facial feature index includes a second blinking frequency, the facial feature detection result is determined based on the first blinking frequency and the second blinking frequency.

10. A device for detecting driver fatigue, characterized in that: The device comprises: A first acquisition unit, used for acquiring a first facial feature index of the driver in a normal state; A second acquiring unit is used to acquire a second facial feature index of the driver when driving a vehicle and a vehicle driving state index, wherein the first facial feature index and the second facial feature index are indicators of the same type; A facial detection unit, configured to compare the first facial feature index with the second facial feature index to determine a facial feature detection result; A fatigue detection unit is used to determine the driver's fatigue detection result based on the facial feature detection result and the vehicle driving state index.

Images