Fatigued driving detection method and apparatus, electronic device, and readable storage medium

Abstract

The present application provides a fatigued driving detection method and apparatus, an electronic device, and a readable storage medium. The method comprises: during the process of a driver driving a vehicle, acquiring in real time a facial image of the driver; on the basis of a first opening degree of eyes in the facial image, determining the eye state of the driver, wherein the eye state comprises an open‑eye state and a non‑open‑eye state, and the non‑open‑eye state comprises a closed‑eye state and a downward‑looking state; when the eye state of the driver in the facial image is the non‑open‑eye state, identifying a head change trend of the driver on the basis of multiple consecutive first facial images corresponding to the facial image, wherein the head change trend comprises a gaze direction change trend and / or a brow center position change trend; on the basis of the head change trend of the driver, determining the non-open-eye state of the driver to be the closed-eye state or the downward-looking state; and if the driver is in the closed‑eye state, determining the driving state of the driver to be a fatigued driving state. The method improves the accuracy of fatigue state detection.

Description

A fatigue driving detection method, device, electronic equipment, and readable storage medium.

[0001] Cross-reference to related applications

[0002] This disclosure claims priority to Chinese Patent Application No. 202411187251.5, filed on August 28, 2024, entitled "A method, apparatus, electronic device and readable storage medium for detecting fatigue driving", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of vehicle technology, and in particular to a fatigue driving detection method, apparatus, electronic device, and readable storage medium. Background Technology

[0004] Fatigue driving is one of the main causes of traffic accidents. By promptly detecting driver fatigue and issuing warnings, accidents that may be caused by slow reaction or inattention can be effectively prevented, thereby protecting the safety of drivers and passengers and reducing the occurrence of traffic accidents.

[0005] Currently, driver fatigue can be determined by calculating the degree of eye opening. While fatigue detection technology based on eye opening plays an important role in improving driving safety, it still has some limitations. For example, when a driver looks down, the degree of eye opening is smaller, making it difficult to distinguish whether the driver is in a closed-eye state or looking down based solely on eye opening, which can easily lead to misjudgments of driver fatigue.

[0006] Summary of the Invention

[0007] In view of this, the purpose of this disclosure is to provide a fatigue driving detection method, device, electronic device and readable storage medium to improve the accuracy of driver fatigue detection.

[0008] In a first aspect, embodiments of this disclosure provide a method for detecting driver fatigue, including:

[0009] The driver's facial image is acquired in real time while the driver is driving the vehicle;

[0010] Based on the first degree of eye opening in the facial image, the driver's eye state is determined; the eye state includes an open eye state and an closed eye state; the closed eye state includes a closed eye state and a downward looking state;

[0011] When the driver's eyes are in the closed-eye state in the face image, the driver's head change trend is identified based on multiple consecutive first face images corresponding to the face image; wherein, the multiple consecutive first face images corresponding to the face image include: the face image and multiple consecutive face images obtained after the face image; the head change trend includes: gaze direction change trend, and / or, brow position change trend;

[0012] Based on the driver's head movement trend, the driver's non-open-eye state is determined to be either the closed-eye state or the downward-looking state;

[0013] When the driver is in the closed-eye state, the driver's driving state is determined to be fatigued driving state.

[0014] In conjunction with the first aspect, this disclosure provides a first possible implementation of the first aspect, wherein determining the driver's eye state based on a first degree of eye opening in the face image includes:

[0015] The first degree of eye opening of the driver is calculated based on the eye contour feature point information identified from the face image;

[0016] If the first opening degree is less than the first threshold, then the first eye image is segmented from the face image;

[0017] The first eye image is input into a first classification model, and the driver's eye state is identified by the first classification model; wherein, the first classification model is trained using normal eye samples and small eye samples; the opening degree of the eyes in the small eye samples is less than that in the normal eye samples.

[0018] In conjunction with the first aspect, this disclosure provides a second possible implementation of the first aspect, wherein, when the head change trend is the gaze direction change trend, the step of identifying the driver's head change trend based on multiple consecutive first face images corresponding to the face image includes:

[0019] For each of the multiple consecutive first face images corresponding to the face image, a second eye image is cropped from the first face image;

[0020] The second eye image is input into the gaze direction recognition model to determine the driver's gaze direction in the second eye image through the gaze direction recognition model;

[0021] Based on the direction of the gaze, determine the area where the driver's gaze falls in the second eye image;

[0022] Based on the driver's gaze direction and the landing area corresponding to multiple consecutive second eye images, the change angle of the driver's gaze direction and the change direction of the landing area are determined.

[0023] The trend of the change in the line of sight direction is determined based on the change angle and the change direction.

[0024] In conjunction with the first aspect, this disclosure provides a third possible implementation of the first aspect, wherein, when the head transformation trend is the brow position transformation trend, the step of identifying the driver's head transformation trend based on multiple consecutive first face images corresponding to the face image includes:

[0025] For multiple consecutive first face images corresponding to the face image, the position information of the driver's brow center in each first face image is calculated based on the eye contour feature point information identified from each first face image.

[0026] Based on the position information of the driver's brow center in each first face image, the trend of the driver's brow center position change is determined.

[0027] In conjunction with the second possible implementation of the first aspect, this disclosure provides a fourth possible implementation of the first aspect, wherein determining the trend of the line-of-sight direction change based on the change angle and the change direction includes:

[0028] When the change angle is greater than a preset angle threshold and the change direction is downward, the trend of the line of sight change is determined to be downward.

[0029] When the transformation angle is less than or equal to the preset angle threshold, and / or the transformation direction is not downward, the trend of the line of sight transformation is determined to be non-downward.

[0030] The step of determining whether the driver's non-open-eye state is the closed-eye state or the downward-looking state based on the driver's head-changing trend includes:

[0031] If the trend of the change in the direction of the gaze is downward, then the driver's non-open-eye state is determined to be the downward-looking state;

[0032] If the trend of the change in the direction of the gaze is not downward, then the driver's non-open-eye state is determined to be the closed-eye state.

[0033] In conjunction with the third possible implementation of the first aspect, this disclosure provides a fifth possible implementation of the first aspect, wherein determining the trend of the driver's brow position change based on the position information of the driver's brow center in each first face image includes:

[0034] Based on the order of acquisition time of each of the first face images, if the position information of the driver's brow moves downward and the distance of movement is greater than or equal to a preset distance threshold, then the trend of the driver's brow position change is determined to be downward.

[0035] If the driver's brow position information is not moving downwards, or if the driver's brow position information moves downwards and the distance moved is less than the preset distance threshold, then the driver's brow position change trend is determined to be non-downward movement.

[0036] The step of determining whether the driver's non-open-eye state is the closed-eye state or the downward-looking state based on the driver's head-changing trend includes:

[0037] If the trend of the change in the position of the brow is downward, then the driver's non-open-eye state is determined to be the downward-looking state;

[0038] If the trend of the change in the position of the brow is not downward, then the driver's non-open-eye state is determined to be the closed-eye state.

[0039] In conjunction with the third possible implementation of the first aspect, this disclosure provides a sixth possible implementation of the first aspect, wherein, after acquiring the driver's facial image in real time while the driver is driving the vehicle, the method further includes:

[0040] The second degree of mouth opening of the driver is calculated based on the mouth contour feature points identified from the face image.

[0041] If the second opening degree is greater than the second threshold, then the mouth image is segmented from the face image;

[0042] The mouth image is input into a second classification model, which identifies whether the driver is smiling with their mouth open or yawning. The second classification model is trained using samples of smiling with their mouth open and samples of yawning.

[0043] When the driver is in the closed-eye state, the driver's driving state is determined to be a state of fatigued driving, including:

[0044] When the driver's eyes are closed and his mouth is yawning, the driver's driving state is determined to be fatigued driving.

[0045] Secondly, embodiments of this disclosure also provide a fatigue driving detection device, comprising:

[0046] The acquisition module is used to acquire the driver's facial image in real time while the driver is driving the vehicle;

[0047] The judgment module is used to determine the driver's eye state based on the first degree of eye opening in the face image; the eye state includes an open eye state and an closed eye state; the closed eye state includes a closed eye state and a downward looking state;

[0048] The recognition module is used to identify the driver's head change trend based on multiple consecutive first face images corresponding to the face image when the driver's eye state in the face image is the closed-eye state; wherein, the multiple consecutive first face images corresponding to the face image include: the face image and multiple consecutive face images obtained after the face image; the head change trend includes: gaze direction change trend, and / or, brow position change trend;

[0049] The first determining module is used to determine, based on the driver's head change trend, whether the driver's non-open-eye state is the closed-eye state or the downward-looking state.

[0050] The second determining module is used to determine that the driver's driving state is a fatigued driving state when the driver is in the closed-eye state.

[0051] In conjunction with the second aspect, this disclosure provides a first possible implementation of the second aspect, wherein, when the determining module is used to determine the driver's eye state based on a first degree of eye opening in the face image, it is specifically used for:

[0052] The first degree of eye opening of the driver is calculated based on the eye contour feature point information identified from the face image;

[0053] If the first opening degree is less than the first threshold, then the first eye image is segmented from the face image;

[0054] The first eye image is input into a first classification model, and the driver's eye state is identified by the first classification model; wherein, the first classification model is trained using normal eye samples and small eye samples; the opening degree of the eyes in the small eye samples is less than that in the normal eye samples.

[0055] In conjunction with the second aspect, this disclosure provides a second possible implementation of the second aspect, wherein, when the head change trend is the gaze direction change trend, the recognition module, when used to recognize the driver's head change trend based on multiple consecutive first face images corresponding to the face image, is specifically used for:

[0056] For each of the multiple consecutive first face images corresponding to the face image, a second eye image is cropped from the first face image;

[0057] The second eye image is input into the gaze direction recognition model to determine the driver's gaze direction in the second eye image through the gaze direction recognition model;

[0058] Based on the direction of the gaze, determine the area where the driver's gaze falls in the second eye image;

[0059] Based on the driver's gaze direction and the landing area corresponding to multiple consecutive second eye images, the change angle of the driver's gaze direction and the change direction of the landing area are determined.

[0060] The trend of the change in the line of sight direction is determined based on the change angle and the change direction.

[0061] In conjunction with the second aspect, this disclosure provides a third possible implementation of the second aspect, wherein, when the head transformation trend is the brow position transformation trend, the recognition module, when used to recognize the driver's head transformation trend based on multiple consecutive first face images corresponding to the face image, is specifically used for:

[0062] For multiple consecutive first face images corresponding to the face image, the position information of the driver's brow center in each first face image is calculated based on the eye contour feature point information identified from each first face image.

[0063] Based on the position information of the driver's brow center in each first face image, the trend of the driver's brow center position change is determined.

[0064] In conjunction with the second possible implementation of the second aspect, this disclosure provides a fourth possible implementation of the second aspect, wherein the identification module, when determining the trend of the line-of-sight direction change based on the change angle and the change direction, is specifically used for:

[0065] When the change angle is greater than a preset angle threshold and the change direction is downward, the trend of the line of sight change is determined to be downward.

[0066] When the transformation angle is less than or equal to the preset angle threshold, and / or the transformation direction is not downward, the trend of the line of sight transformation is determined to be non-downward.

[0067] When the first determining module determines whether the driver's non-open-eye state is the closed-eye state or the downward-looking state based on the driver's head-changing trend, it is specifically used for:

[0068] If the trend of the change in the direction of the gaze is downward, then the driver's non-open-eye state is determined to be the downward-looking state;

[0069] If the trend of the change in the direction of the gaze is not downward, then the driver's non-open-eye state is determined to be the closed-eye state.

[0070] In conjunction with the third possible implementation of the second aspect, this disclosure provides a fifth possible implementation of the second aspect, wherein, when the recognition module is used to determine the trend of the driver's brow position change based on the position information of the driver's brow in each first face image, it is specifically used for:

[0071] Based on the order of acquisition time of each of the first face images, if the position information of the driver's brow moves downward and the distance of movement is greater than or equal to a preset distance threshold, then the trend of the driver's brow position change is determined to be downward.

[0072] If the driver's brow position information is not moving downwards, or if the driver's brow position information moves downwards and the distance moved is less than the preset distance threshold, then the driver's brow position change trend is determined to be non-downward movement.

[0073] When the first determining module determines whether the driver's non-open-eye state is the closed-eye state or the downward-looking state based on the driver's head-changing trend, it is specifically used for:

[0074] If the trend of the change in the position of the brow is downward, then the driver's non-open-eye state is determined to be the downward-looking state;

[0075] If the trend of the change in the position of the brow is not downward, then the driver's non-open-eye state is determined to be the closed-eye state.

[0076] In conjunction with the third possible implementation of the second aspect, this disclosure provides a sixth possible implementation of the second aspect, wherein the apparatus further includes:

[0077] The calculation module is used to calculate the second opening degree of the driver's mouth based on the mouth contour feature point information identified from the face image after the acquisition module acquires the driver's face image in real time during the driver's driving of the vehicle.

[0078] The segmentation module is used to segment the mouth image from the face image if the second opening degree is greater than the second threshold.

[0079] The input module is used to input the mouth image into the second classification model, and the second classification model is used to identify whether the driver is smiling with his mouth open or yawning; wherein, the second classification model is trained using samples of smiling with his mouth open and samples of yawning.

[0080] When the second determining module determines that the driver's driving state is fatigued when the driver is in the closed-eye state, it is specifically used for:

[0081] When the driver's eyes are closed and his mouth is yawning, the driver's driving state is determined to be fatigued driving.

[0082] Thirdly, embodiments of this disclosure also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps in any of the possible implementations of the first aspect described above are performed.

[0083] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps in any of the possible implementations of the first aspect described above.

[0084] This disclosure provides a fatigue driving detection method, device, electronic device, and readable storage medium. When the driver's eye state is determined to be non-open based on the first degree of eye opening, the method further determines whether the non-open eye state is a closed eye state or a downward-looking state based on the driver's head movement trend. This avoids misjudging a driver's downward-looking state as a closed eye state, thereby improving the accuracy of driver fatigue detection.

[0085] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0086] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this disclosure and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0087] Figure 1 shows a flowchart of a fatigue driving detection method provided in an embodiment of this disclosure;

[0088] Figure 2 shows a flowchart of another fatigue driving detection method provided in an embodiment of this disclosure;

[0089] Figure 3 shows a schematic diagram of the structure of a fatigue driving detection device provided in an embodiment of this disclosure;

[0090] Figure 4 shows a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0092] Considering that judging a driver's fatigue state solely based on the degree of eye opening or closing can easily lead to misjudgment—specifically, when a driver is looking downwards, it can easily be misinterpreted as having closed eyes—this disclosure provides a fatigue driving detection method, apparatus, electronic device, and readable storage medium, which are described below through embodiments.

[0093] To facilitate understanding of this embodiment, a fatigue driving detection method disclosed in this disclosure will first be described in detail. Figure 1 shows a flowchart of a fatigue driving detection method provided in this disclosure embodiment. As shown in Figure 1, it includes the following steps S101-S105:

[0094] S101: Acquire the driver's facial image in real time while the driver is driving the vehicle.

[0095] In this embodiment, a camera is installed in the vehicle's cockpit to capture real-time video streams of the driver driving the vehicle. The real-time video stream captured by the camera is received and decomposed to obtain single-frame images.

[0096] The face detection model detects whether a face exists in a single frame image. When a face is found in a single frame image, that single frame image is taken as a face image.

[0097] S102: Determine the driver's eye state based on the first degree of eye opening in the face image; the eye state includes open eye state and closed eye state; the closed eye state includes closed eye state and looking down state.

[0098] In this embodiment, an eye image is segmented from a face image, and the driver's eye state is determined to be open or closed based on the first degree of eye opening in the eye image.

[0099] Among them, the open-eye state indicates that the driver is not in a state of fatigue driving; the closed-eye state indicates that the driver is in a state of fatigue driving; while the downward-looking state only indicates that the driver's eyes are looking down, and does not indicate that the driver is in a state of fatigue driving. In other words, the downward-looking state is used to indicate that the driver is not in a state of fatigue driving.

[0100] S103: When the driver's eyes are not open in the face image, the driver's head change trend is identified based on multiple consecutive first face images corresponding to the face image; wherein, the multiple consecutive first face images corresponding to the face image include: the face image and multiple consecutive face images obtained after the face image; the head change trend includes: the gaze direction change trend, and / or, the brow position change trend.

[0101] In this embodiment, when the driver's eyes are open, it indicates that the driver's fatigue state is a non-fatigue driving state.

[0102] When the driver's eyes are closed, it is necessary to continue to identify the driver's head movement trend in order to further determine whether the driver's closed eyes state is a closed eyes state or a downward look state.

[0103] In identifying the driver's head movement trend, for example, if the facial image is a single frame captured by the camera at the first moment, then the multiple consecutive first facial images corresponding to this facial image are: n single frame images captured by the camera from the first moment to the nth moment. Here, the first moment to the nth moment are consecutive acquisition moments, and the first moment to the nth moment are acquisition moments arranged in chronological order.

[0104] In this embodiment, the driver's gaze direction change trend and / or brow position change trend can be identified based on multiple consecutive first face images corresponding to the face image.

[0105] S104: Based on the driver's head movement trend, determine whether the driver's non-open-eye state is a closed-eye state or a downward-looking state.

[0106] In this embodiment, when the head changing trend is a changing trend of the gaze direction, if the changing trend of the gaze direction is downward, then the driver's non-open-eye state is determined to be a downward-looking state; if the changing trend of the gaze direction is not downward, then the driver's non-open-eye state is determined to be a closed-eye state.

[0107] When the head changes direction with the direction of the brow position changing, if the brow position changes downward, the driver's non-open-eye state is determined to be looking down; if the brow position changes direction is not downward, the driver's non-open-eye state is determined to be closed-eye.

[0108] S105: When the driver's eyes are closed, the driver's driving state is determined to be fatigued driving state.

[0109] In this embodiment, when the driver is looking down, the driver's driving state is determined to be a non-fatigue driving state.

[0110] In one possible implementation, Figure 2 shows a flowchart of another fatigue driving detection method provided by an embodiment of this disclosure. As shown in Figure 2, considering that different drivers have different eye sizes, the first opening degree of smaller eyes is less than that of normal eyes. Therefore, when the first opening degree is less than a first threshold, it cannot be directly determined that the driver is in a closed-eye state. It is also possible that the driver is actually in a closed-eye state, but the calculated first opening degree is smaller due to the smaller size of the eyes. Based on this, when performing step S102, it can be specifically executed according to the following steps S1021-S1023:

[0111] S1021: Calculate the first degree of eye opening of the driver based on the eye contour feature points identified from the face image.

[0112] In this embodiment, a facial image is input into a facial feature point localization model, which then outputs eye contour feature point information and mouth contour feature point information. Based on the eye contour feature point information, the first degree of eye opening of the driver is calculated.

[0113] S1022: If the first opening degree is less than the first threshold, then the first eye image is segmented from the face image.

[0114] In this embodiment, if the first opening degree is greater than or equal to the first threshold, it indicates that the driver's eye state is open, and at this time, the driver is in a non-driving fatigue state.

[0115] When the first opening degree is less than the first threshold, it is necessary to further identify whether the reason for the first opening degree being less than the first threshold is due to the driver's eye state being in a closed state, or whether the driver's eye state is actually open, but due to the small size of the eyes.

[0116] S1023: Input the first eye image into the first classification model, and identify the driver's eye state through the first classification model; wherein, the first classification model is trained using normal eye samples and small eye samples; the opening and closing degree of the eyes in the small eye samples is less than that in the normal eye samples.

[0117] In this embodiment, to avoid misjudging the driver's eye state due to the driver's small eyes, the eye image is segmented from the face image and then input into the first classification model. The first classification model identifies whether the driver's eye state is actually closed or open.

[0118] Since the first classification model is trained using normal eye samples and small eye samples, it has the ability to distinguish between the open and closed states of small eyes.

[0119] In one possible implementation, as shown in Figure 2, when the head change trend is a gaze direction change trend, step S103 can specifically identify the driver's gaze direction change trend according to the following steps S1031-S1035:

[0120] S1031: For each of the multiple consecutive first face images corresponding to the face image, crop out the second eye image from the first face image.

[0121] S1032: Input the second eye image into the gaze direction recognition model to determine the driver's gaze direction in the second eye image through the gaze direction recognition model.

[0122] S1033: Determine the area where the driver's gaze falls in the second eye image based on the direction of the gaze.

[0123] S1034: Based on the driver's gaze direction and focal point area corresponding to multiple consecutive second eye images, determine the changing angle of the driver's gaze direction and the changing direction of the focal point area.

[0124] In this embodiment, the angle of change of the driver's gaze direction is determined based on the driver's gaze direction corresponding to multiple consecutive second eye images. The direction of change of the driver's landing point area is determined based on the driver's landing point area corresponding to multiple consecutive second eye images.

[0125] S1035: Determine the trend of the changing line of sight based on the changing angle and changing direction.

[0126] In this embodiment, when the change angle is greater than a preset angle threshold and the change direction is downward, the trend of the change in the line of sight direction is determined to be downward.

[0127] When the change angle is less than or equal to a preset angle threshold, and / or the change direction is not downward, the trend of the change in the line of sight direction is determined to be non-downward movement.

[0128] At this point, when executing step S104, the specific steps to determine whether the driver's non-open-eye state is a closed-eye state or a downward-looking state can be followed:

[0129] If the direction of the gaze changes downwards, then the driver's non-open-eye state is determined to be looking downwards.

[0130] If the direction of the gaze changes without moving downwards, then the driver's non-open-eye state is determined to be a closed-eye state.

[0131] In one possible implementation, when the head change trend is a change trend in the position of the brow, when performing step S103, the driver's brow position change trend can be identified according to the following steps S1036-S1037:

[0132] S1036: For multiple consecutive first face images corresponding to the face image, calculate the position information of the driver's brow center in each first face image based on the eye contour feature point information identified from each first face image.

[0133] In this embodiment, the driver's brow is located between the driver's two eyes. Therefore, after identifying the feature information of the driver's eye contour points in the first face image, the position information of the driver's brow in the first face image can be calculated.

[0134] S1037: Determine the trend of the driver's brow position change based on the position information of the driver's brow center in each first face image.

[0135] Based on the position information of the driver's brow center in multiple consecutive first face images, the trend of the driver's brow center position change is determined.

[0136] Specifically, based on the order in which the first face images are acquired, if the position information of the driver's brow moves downward and the distance of the movement is greater than or equal to a preset distance threshold, then the trend of the driver's brow position change is determined to be downward.

[0137] If the driver's brow position information is not moving downwards, or if the driver's brow position information moves downwards and the distance moved is less than the preset distance threshold, then the driver's brow position change trend is determined to be non-downward movement.

[0138] At this point, when executing step S104, the specific steps to determine whether the driver's non-open-eye state is a closed-eye state or a downward-looking state can be followed:

[0139] If the position of the brow changes downwards, then the driver's non-open-eye state is determined to be looking downwards.

[0140] If the trend of the brow position change is not downward, then the driver's non-open-eye state is determined to be a closed-eye state.

[0141] Non-downward movement includes: no movement, leftward movement, rightward movement, and upward movement.

[0142] In one possible implementation, to perform fatigue driving detection more accurately, as shown in Figure 2, after executing step S101, the following steps S1061-S1063 can also be performed:

[0143] S1061: Calculate the second degree of mouth opening of the driver based on the mouth contour feature points identified from the face image.

[0144] S1062: If the second opening degree is greater than the second threshold, then the mouth image is segmented from the face image.

[0145] In this embodiment, when the second opening degree is greater than the second threshold, it indicates that the driver may be yawning or smiling.

[0146] S1063: Input the mouth image into the second classification model, and use the second classification model to identify whether the driver is smiling with his mouth open or yawning; wherein, the second classification model is trained using samples of smiling with his mouth open and samples of yawning.

[0147] In this embodiment, the second classification model trained using samples of open-mouth smiling and yawning states can distinguish whether the driver is in an open-mouth smiling state or a yawning state.

[0148] It is worth noting that the execution order of steps S1061-S1063 and steps S102-S104 is not fixed. Steps S102-S104 can be executed first, followed by steps S1061-S1063; or steps S1061-S1063 can be executed first, followed by steps S102-S104; or steps S1061-S1063 and steps S102-S104 can be executed simultaneously.

[0149] At this point, when executing step S105, specifically: when the driver's eyes are closed and their mouth is yawning, the driver's driving state is determined to be fatigued. This allows for more accurate fatigue detection.

[0150] Based on the same technical concept, this disclosure also provides a fatigue driving detection device. Figure 3 shows a structural schematic diagram of a fatigue driving detection device provided in an embodiment of this disclosure. As shown in Figure 3, the device includes:

[0151] The acquisition module 301 is used to acquire the driver's facial image in real time while the driver is driving the vehicle;

[0152] The judgment module 302 is used to judge the driver's eye state based on the first opening degree of the eyes in the face image; the eye state includes an open eye state and an closed eye state; the closed eye state includes a closed eye state and a downward looking state;

[0153] The recognition module 303 is used to identify the driver's head change trend based on multiple consecutive first face images corresponding to the face image when the driver's eye state in the face image is the non-open eye state; wherein, the multiple consecutive first face images corresponding to the face image include: the face image and multiple consecutive face images obtained after the face image; the head change trend includes: gaze direction change trend, and / or, brow position change trend;

[0154] The first determining module 304 is used to determine, based on the driver's head change trend, whether the driver's non-open-eye state is the closed-eye state or the downward-looking state.

[0155] The second determining module 305 is used to determine that the driver's driving state is a fatigued driving state when the driver is in the closed-eye state.

[0156] Optionally, when the judgment module 302 is used to judge the driver's eye state based on the first degree of eye opening in the face image, it is specifically used for:

[0157] The first degree of eye opening of the driver is calculated based on the eye contour feature point information identified from the face image;

[0158] If the first opening degree is less than the first threshold, then the first eye image is segmented from the face image;

[0159] The first eye image is input into a first classification model, and the driver's eye state is identified by the first classification model; wherein, the first classification model is trained using normal eye samples and small eye samples; the opening degree of the eyes in the small eye samples is less than that in the normal eye samples.

[0160] Optionally, when the head change trend is the same as the gaze direction change trend, the recognition module 303, when recognizing the driver's head change trend based on multiple consecutive first face images corresponding to the face image, specifically performs the following:

[0161] For each of the multiple consecutive first face images corresponding to the face image, a second eye image is cropped from the first face image;

[0162] The second eye image is input into the gaze direction recognition model to determine the driver's gaze direction in the second eye image through the gaze direction recognition model;

[0163] Based on the direction of the gaze, determine the area where the driver's gaze falls in the second eye image;

[0164] Based on the driver's gaze direction and the landing area corresponding to multiple consecutive second eye images, the change angle of the driver's gaze direction and the change direction of the landing area are determined.

[0165] The trend of the change in the line of sight direction is determined based on the change angle and the change direction.

[0166] Optionally, when the head transformation trend is the same as the brow position transformation trend, the recognition module 303, when used to recognize the driver's head transformation trend based on multiple consecutive first face images corresponding to the face image, specifically uses the following methods:

[0167] For multiple consecutive first face images corresponding to the face image, the position information of the driver's brow center in each first face image is calculated based on the eye contour feature point information identified from each first face image.

[0168] Based on the position information of the driver's brow center in each first face image, the trend of the driver's brow center position change is determined.

[0169] Optionally, when the recognition module 303 determines the trend of the line-of-sight direction change based on the change angle and the change direction, it is specifically used for:

[0170] When the change angle is greater than a preset angle threshold and the change direction is downward, the trend of the line of sight change is determined to be downward.

[0171] When the transformation angle is less than or equal to the preset angle threshold, and / or the transformation direction is not downward, the trend of the line of sight transformation is determined to be non-downward.

[0172] When the first determining module 304 determines whether the driver's non-open-eye state is the closed-eye state or the downward-looking state based on the driver's head-changing trend, it is specifically used for:

[0173] If the trend of the change in the direction of the gaze is downward, then the driver's non-open-eye state is determined to be the downward-looking state;

[0174] If the trend of the change in the direction of the gaze is not downward, then the driver's non-open-eye state is determined to be the closed-eye state.

[0175] Optionally, when the recognition module 303 determines the trend of the driver's brow position change based on the position information of the driver's brow in each first face image, it is specifically used for:

[0176] Based on the order of acquisition time of each of the first face images, if the position information of the driver's brow moves downward and the distance of movement is greater than or equal to a preset distance threshold, then the trend of the driver's brow position change is determined to be downward.

[0177] If the driver's brow position information is not moving downwards, or if the driver's brow position information moves downwards and the distance moved is less than the preset distance threshold, then the driver's brow position change trend is determined to be non-downward movement.

[0178] When the first determining module 304 determines whether the driver's non-open-eye state is the closed-eye state or the downward-looking state based on the driver's head-changing trend, it is specifically used for:

[0179] If the trend of the change in the position of the brow is downward, then the driver's non-open-eye state is determined to be the downward-looking state;

[0180] If the trend of the change in the position of the brow is not downward, then the driver's non-open-eye state is determined to be the closed-eye state.

[0181] Optionally, the device further includes:

[0182] The calculation module is used to calculate the second opening degree of the driver's mouth based on the mouth contour feature point information identified from the face image after the acquisition module 301 acquires the driver's face image in real time during the driver's driving of the vehicle.

[0183] The segmentation module is used to segment the mouth image from the face image if the second opening degree is greater than the second threshold.

[0184] The input module is used to input the mouth image into the second classification model, and the second classification model is used to identify whether the driver is smiling with his mouth open or yawning; wherein, the second classification model is trained using samples of smiling with his mouth open and samples of yawning.

[0185] The second determining module 305, when determining that the driver's driving state is fatigued when the driver is in the closed-eye state, is specifically used for:

[0186] When the driver's eyes are closed and his mouth is yawning, the driver's driving state is determined to be fatigued driving.

[0187] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure, including: a processor 401, a memory 402 and a bus 403. The memory 402 stores machine-readable instructions that can be executed by the processor 401. When the electronic device runs the above-described information processing method, the processor 401 and the memory 402 communicate through the bus 403. The processor 401 executes the machine-readable instructions to perform the steps of the method described in Embodiment 1.

[0188] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps described in Embodiment 1.

[0189] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, electronic devices, and computer-readable storage media described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0190] In the several embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0191] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0193] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims. Industrial applicability

[0195] This disclosure provides a fatigue driving detection method, device, electronic device, and readable storage medium. When the driver's eye state is determined to be non-open based on the first degree of eye opening, the method further determines whether the non-open eye state is a closed eye state or a downward-looking state based on the driver's head movement trend. This avoids misjudging a driver's downward-looking state as a closed eye state, thereby improving the accuracy of driver fatigue detection.

[0196] Furthermore, it is understood that the fatigue driving detection method, apparatus, electronic device, and readable storage medium provided in the embodiments of this disclosure are reproducible and can be used in various industrial applications. For example, the fatigue driving detection method, apparatus, electronic device, and readable storage medium provided in the embodiments of this disclosure can be used in the field of vehicle technology.

Claims

1. A method for detecting driver fatigue, characterized in that, include: The driver's facial image is acquired in real time while the driver is driving the vehicle; Based on the first degree of eye opening in the facial image, the driver's eye state is determined; the eye state includes an open eye state and an closed eye state; the closed eye state includes a closed eye state and a downward looking state; When the driver's eyes are in the closed-eye state in the face image, the driver's head change trend is identified based on multiple consecutive first face images corresponding to the face image; wherein, the multiple consecutive first face images corresponding to the face image include: the face image and multiple consecutive face images obtained after the face image; the head change trend includes: gaze direction change trend, and / or, brow position change trend; Based on the driver's head movement trend, the driver's non-open-eye state is determined to be either the closed-eye state or the downward-looking state; When the driver is in the closed-eye state, the driver's driving state is determined to be fatigued driving state.

2. The method according to claim 1, characterized in that, The step of determining the driver's eye state based on the first degree of eye opening in the facial image includes: The first degree of eye opening of the driver is calculated based on the eye contour feature point information identified from the face image; If the first opening degree is less than the first threshold, then the first eye image is segmented from the face image; The first eye image is input into a first classification model, and the driver's eye state is identified by the first classification model; wherein, the first classification model is trained using normal eye samples and small eye samples; the opening degree of the eyes in the small eye samples is less than that in the normal eye samples.

3. The method according to claim 1, characterized in that, When the head change trend is the same as the gaze direction change trend, the step of identifying the driver's head change trend based on multiple consecutive first face images corresponding to the face image includes: For each of the multiple consecutive first face images corresponding to the face image, a second eye image is cropped from the first face image; The second eye image is input into the gaze direction recognition model to determine the driver's gaze direction in the second eye image through the gaze direction recognition model; Based on the direction of the gaze, determine the area where the driver's gaze falls in the second eye image; Based on the driver's gaze direction and the landing area corresponding to multiple consecutive second eye images, the change angle of the driver's gaze direction and the change direction of the landing area are determined. The trend of the change in the line of sight direction is determined based on the change angle and the change direction.

4. The method according to claim 1, characterized in that, When the head transformation trend is the same as the brow position transformation trend, the step of identifying the driver's head transformation trend based on multiple consecutive first face images corresponding to the face image includes: For multiple consecutive first face images corresponding to the face image, the position information of the driver's brow center in each first face image is calculated based on the eye contour feature point information identified from each first face image. Based on the position information of the driver's brow center in each first face image, the trend of the driver's brow center position change is determined.

5. The method according to claim 3, characterized in that, Determining the trend of the line of sight change based on the change angle and the change direction includes: When the change angle is greater than a preset angle threshold and the change direction is downward, the trend of the line of sight change is determined to be downward. When the transformation angle is less than or equal to the preset angle threshold, and / or the transformation direction is not downward, the trend of the line of sight transformation is determined to be non-downward. The step of determining whether the driver's non-open-eye state is the closed-eye state or the downward-looking state based on the driver's head-changing trend includes: If the trend of the change in the direction of the gaze is downward, then the driver's non-open-eye state is determined to be the downward-looking state; If the trend of the change in the direction of the gaze is not downward, then the driver's non-open-eye state is determined to be the closed-eye state.

6. The method according to claim 4, characterized in that, The step of determining the trend of the driver's brow position change based on the position information of the driver's brow center in each first face image includes: Based on the order of acquisition time of each of the first face images, if the position information of the driver's brow moves downward and the distance of movement is greater than or equal to a preset distance threshold, then the trend of the driver's brow position change is determined to be downward. If the driver's brow position information is not moving downwards, or if the driver's brow position information moves downwards and the distance moved is less than the preset distance threshold, then the driver's brow position change trend is determined to be non-downward movement. The step of determining whether the driver's non-open-eye state is the closed-eye state or the downward-looking state based on the driver's head-changing trend includes: If the trend of the change in the position of the brow is downward, then the driver's non-open-eye state is determined to be the downward-looking state; If the trend of the change in the position of the brow is not downward, then the driver's non-open-eye state is determined to be the closed-eye state.

7. The method according to claim 4, characterized in that, After acquiring the driver's facial image in real time while the driver is driving the vehicle, the method further includes: The second degree of mouth opening of the driver is calculated based on the mouth contour feature points identified from the face image. If the second opening degree is greater than the second threshold, then the mouth image is segmented from the face image; The mouth image is input into a second classification model, which identifies whether the driver is smiling with their mouth open or yawning. The second classification model is trained using samples of smiling with their mouth open and samples of yawning. When the driver is in the closed-eye state, the driver's driving state is determined to be a state of fatigued driving, including: When the driver's eyes are closed and his mouth is yawning, the driver's driving state is determined to be fatigued driving.

8. A fatigue driving detection device, characterized in that, include: The acquisition module is used to acquire the driver's facial image in real time while the driver is driving the vehicle; The judgment module is used to determine the driver's eye state based on the first degree of eye opening in the face image; the eye state includes an open eye state and an closed eye state; the closed eye state includes a closed eye state and a downward looking state; The recognition module is used to identify the driver's head change trend based on multiple consecutive first face images corresponding to the face image when the driver's eye state in the face image is the closed-eye state; wherein, the multiple consecutive first face images corresponding to the face image include: the face image and multiple consecutive face images obtained after the face image; the head change trend includes: gaze direction change trend, and / or, brow position change trend; The first determining module is used to determine, based on the driver's head change trend, whether the driver's non-open-eye state is the closed-eye state or the downward-looking state. The second determining module is used to determine that the driver's driving state is a fatigued driving state when the driver is in the closed-eye state.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the memory via the bus, and the machine-readable instructions, when executed by the processor, perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 7.

Images