Method and device for detecting driver's state, and storage medium

Abstract

The present application relates to the field of computer vision technology, and in particular to a method and device for detecting a driver's state and a storage medium. The method includes: acquiring a first image frame and a second image frame, the first image frame and the second image frame are both frame images including a driver's face, and the first image frame is an image frame captured before the second image frame; acquiring a first detection information of the first image frame and a second detection information of the second image frame, the first detection information and the second detection information indicate the driver's eye state and head posture; and when the second detection information indicates that the first eye state corresponding to the second image frame is a closed eye state, determining the second eye state corresponding to the second image frame based on the first detection information and the second detection information. According to the present application, three scenarios that have little difference when analyzed for an image, namely, a normal eyes closed scenario, a downward facing scenario, and a dozing scenario, can be accurately distinguished. This ensures an open / closed eye state identification effect with high accuracy and high robustness.

Description

[Technical Field]

[0001] The present application relates to the field of computer vision technology, and in particular to a method and apparatus for detecting a driver's state, and a storage medium. [Background technology]

[0002] Human eye detection technology includes performing image detection on multiple image frames containing human eyes to estimate whether the eyes are currently in a closed or open state. With the popularity and development of video processing and video surveillance technologies, human eye detection technology has become an essential and important part of the eye image analysis process.

[0003] In the related art, an in-car scenario is used as an example. A camera is positioned facing the driver's face. The system acquires multiple image frames including human eyes through the camera, and based on the multiple image frames, identifies whether the driver's eyes are open or closed, and then, in combination with the head posture, can accurately monitor the user's fatigue state.

[0004] However, with the above method, when the driver's head is tilted downward, the system cannot accurately determine the driver's eye-open / eye-closed state based on the images captured by the camera. For example, three image frames captured by the camera are shown in FIG. 1. Image frame 12 corresponds to a normal eye-closed state, image frame 14 corresponds to a head-down eye-open state, and image frame 16 corresponds to a head-down eye-drowsy state. However, from the camera's perspective, the driver's eyes appear closed in these three cases, which has obvious drawbacks in the accuracy and robustness of identifying the eye-open / eye-closed state directly based on the images. This significantly reduces the accuracy of the user's fatigue state subsequently determined based on the eye-open / eye-closed state. Summary of the Invention

[0005] In view of this, a method and apparatus for detecting a driver's state, and a storage medium are provided, which can ensure high accuracy and high robustness of eyes-open / eyes-closed discrimination effect, and subsequently help to obtain a stable and accurate fatigue state of the user.

[0006] According to a first aspect, an embodiment of the present application provides a method for detecting a driver's state, the method comprising: acquiring a first image frame and a second image frame, both of the first image frame and the second image frame including a face of the driver; Image Frame and the first image frame is an image frame captured before the second image frame; obtaining first detection information of a first image frame and second detection information of a second image frame, the first detection information and the second detection information indicating an eye state and a head posture of a driver; determining a second eye state corresponding to the second image frame based on the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is a closed eye state; Includes.

[0007] In this implementation, first detection information of a first image frame and second detection information of a second image frame (e.g., the driver's continuous eye state and corresponding continuous head posture) are obtained. For different scenarios, such as a head-down-looking-down scenario and a head-down-dozing scenario, a second eye state corresponding to the second image frame can be determined based on the first detection information and the second detection information if the first eye state corresponding to the second image frame is determined to be a closed-eye state. That is, the eye state corresponding to the second image frame is corrected to obtain an accurate eye state. This solves the problem in the related art that when the driver lowers his / her head, the system cannot accurately determine the open / closed-eye state by directly using the image, and ensures a highly accurate and robust open / closed-eye discrimination effect, which is helpful in subsequently obtaining a stable and accurate fatigue state of the user.

[0008] In a possible implementation, determining a second eye state corresponding to a second image frame based on the first detection information and the second detection information includes: and determining that the second eye state corresponding to the second image frame is an open eye state when the first detection information and the second detection information satisfy a first preset condition.

[0009] The first preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an open eye state.

[0010] In this implementation, when the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction and the first detection information indicates that the eye state corresponding to the first image frame is in the eyes-open state, the second eye state corresponding to the second image frame is determined to be the eyes-open state, that is, the false detection of the eyes-closed state caused by looking down is accurately restored to the eyes-open state based on the head pose sequence and the eye state sequence, and the eyes-closed state in the dozing scenario is not falsely corrected, and the eye state identification effect is further improved.

[0011] In another possible implementation, determining a second eye state corresponding to a second image frame based on the first detection information and the second detection information includes: The method further includes determining that the second eye state corresponding to the second image frame is a closed eye state when the first detection information and the second detection information satisfy a second predetermined condition.

[0012] The second preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is a closed eye state.

[0013] In this implementation, if the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction and the first detection information indicates that the eye state corresponding to the first image frame is a closed-eye state, the second eye state corresponding to the second image frame is determined to be a closed-eye state. That is, the head-down dozing scenario is accurately determined based on the head pose sequence and the eye state sequence, and the eye state identification effect is further improved.

[0014] In another possible implementation, the method comprises: The method further includes determining that the second eye state corresponding to the second image frame is a closed eye state when the second detection information indicates that the head posture corresponding to the second image frame does not jump in the pitch angle direction.

[0015] In this implementation, the second detection information determines that the second eye state corresponding to the second image frame is an eye-closed state if the head pose corresponding to the second image frame does not jump in the pitch angle direction, that is, the normal eye-closed scenario is accurately determined by using the head pose sequence, and the eye state identification effect is further improved.

[0016] In another possible implementation, the method comprises: The method further includes determining a fatigue state detection result based on the head pose and the second eye state corresponding to the second image frame.

[0017] In another possible implementation, the method comprises: The method further includes outputting alarm information when the fatigue state detection result satisfies a preset alarm condition.

[0018] In this implementation, when the fatigue state detection result meets the preset alarm condition, alarm information is output, so that dangerous driving behavior is monitored, the driver is warned in time, and the possibility of the driver getting into a traffic accident is prevented.

[0019] According to a second aspect, an embodiment of the present application provides an apparatus for detecting a driver's state, the apparatus comprising: a first capture unit configured to capture a first image frame and a second image frame, both of the first image frame and the second image frame including a face of the driver; Image Frame a first acquisition unit, wherein the first image frame is an image frame captured before the second image frame; a second acquisition unit configured to acquire first detected information of a first image frame and second detected information of a second image frame, the first detected information and the second detected information indicating an eye state and a head posture of a driver; a determining unit configured to determine a second eye state corresponding to the second image frame based on the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is an eye-closed state; Includes.

[0020] In another possible implementation, the decision unit The device is further configured to determine that the second eye state corresponding to the second image frame is an open eye state when the first detection information and the second detection information satisfy a first preset condition.

[0021] The first preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an open eye state.

[0022] In another possible implementation, the decision unit The device is further configured to determine that the second eye state corresponding to the second image frame is a closed eye state when the first detection information and the second detection information satisfy a second preset condition.

[0023] The second preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is a closed eye state.

[0024] In another possible implementation, the decision unit The system is further configured to determine that the second eye state corresponding to the second image frame is a closed eye state when the second detection information indicates that the head pose corresponding to the second image frame does not jump in the pitch angle direction.

[0025] In another possible implementation, the device further comprises a detection module.

[0026] The detection module is configured to determine a fatigue state detection result based on the head pose and the second eye state corresponding to the second image frame.

[0027] In another possible implementation, the device further comprises an alarm module.

[0028] The alarm module is configured to output alarm information when the fatigue state detection result meets a preset alarm condition.

[0029] According to a third aspect, an embodiment of the present application provides an apparatus for detecting a driver's state, the apparatus comprising: a processor; and a memory configured to store processor-executable instructions.

[0030] The processor is configured to perform the method provided in the first aspect or any one of the possible implementations of the first aspect when executing the instructions.

[0031] According to a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, perform the method provided in the first aspect or any one of the possible implementations of the first aspect.

[0032] According to a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product including computer readable code or a non-volatile computer readable storage medium carrying the computer readable code, which, when executed on an electronic device, causes a processor in the electronic device to perform the method provided in the first aspect or any one of the possible implementations of the first aspect.

[0033] According to a sixth aspect, an embodiment of the present application provides a vehicle, the vehicle including the apparatus provided in the second aspect or any one of the possible implementations of the second aspect.

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, and together with this specification, illustrate exemplary embodiments, features, and aspects of the present application and are intended to explain the principles of the present application. [Brief explanation of the drawings]

[0035] [Figure 1] FIG. 2 is a schematic diagram of three image frames captured by a camera. [Figure 2] FIG. 1 is a schematic diagram of the architecture of a DMS according to an exemplary embodiment of the present application. [Figure 3] 1 is a schematic diagram of the architecture of a device for detecting a driver's state according to an exemplary embodiment of the present application; [Figure 4] 1 is a schematic flowchart of a method for detecting a driver's state, according to an exemplary embodiment of the present application; [Figure 5]4 is a schematic flowchart of a method for detecting a driver's state according to another exemplary embodiment of the present application; [Figure 6] 1 is a schematic diagram of the principle of an eye condition detection method according to an exemplary embodiment of the present application; [Figure 7] 1 is a schematic diagram of the principle of a head pose detection method according to an exemplary embodiment of the present application; [Figure 8] FIG. 1 is a schematic diagram of a state sequence correction process according to an exemplary embodiment of the present application; [Figure 9] FIG. 10 is a schematic diagram of a state sequence correction process according to another exemplary embodiment of the present application; [Figure 10] FIG. 10 is a schematic diagram of a state sequence correction process according to another exemplary embodiment of the present application; [Figure 11] 1 is a schematic diagram of three scenarios in a method for detecting a driver's state and the principle of a correction process according to an exemplary embodiment of the present application; [Figure 12] 1 is a block diagram of an apparatus for detecting a driver's state according to an exemplary embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0036] Various exemplary embodiments, features, and aspects of the present application are described in detail below with reference to the accompanying drawings, in which like reference numerals represent elements having the same or similar functions. While various aspects of the embodiments are illustrated in the accompanying drawings, the accompanying drawings are not necessarily drawn to scale unless specifically specified otherwise.

[0037] The specific word "exemplary" herein means "serving as an example, embodiment, or illustration." Any embodiment described herein as an "exemplary" is not necessarily described as superior or better than other embodiments.

[0038] Furthermore, in order to better describe the present application, numerous specific details are provided in the following specific implementations. Those skilled in the art will understand that the present application can be practiced without some of the specific details. In some instances, methods, means, elements, and circuits that are well known to those skilled in the art are not described in detail so as to emphasize the subject matter of the present application.

[0039] The fatigue driving warning system accurately identifies the driver's eye state, yawning behavior, head posture, and risky behavior using the driver monitoring system (DMS) in the driver's seat. Using this information, the system accurately determines the driver's fatigue state and provides alarm information to ensure the user's driving safety. Eye state identification technology is a key module of the fatigue driving warning system. The driver's fatigue level can be analyzed based on the frequency of continuous eye closure and blinking, and whether the driver's vision is obstructed can be analyzed based on the duration of continuous eye closure. This allows for accurate identification of the user's eye state and accurate fatigue warning. High accuracy and robust eye state identification helps obtain a stable and accurate user fatigue state. In particular, in-car scenarios require not only high accuracy, but also the algorithm needs to adapt to various ambient light and camera installation scenarios. The camera is used by the DMS to monitor the driver. When the camera is mounted on the vehicle's pillar A or rearview mirror and the driver's head is facing downward, the eye open / closed state cannot be accurately determined based on the image acquired from the camera. In the related art, the accuracy and robustness of analyzing the eye open / closed state based on the image alone are poor. This significantly reduces the accuracy of the user's fatigue state that is subsequently determined based on the eye open / closed state.

[0040] The present embodiment provides a method and device for detecting a driver's state, as well as a storage medium. If the first eye state corresponding to the second image frame is a closed-eye state, the system determines the second eye state corresponding to the second image frame based on the first detection information of the first image frame (i.e., the image frame captured before the second image frame) and the second detection information of the second image frame, and corrects the eye state corresponding to the second image frame to obtain the correct eye state. This solves the problem in the related art that when the driver lowers his / her head, the system cannot correctly determine the open / closed-eye state by directly using the image, ensures high-precision and high-robustness eye-open / closed-eye discrimination effect, and helps to subsequently obtain the user's stable and accurate fatigue state. It should be noted that the definitions of the first image frame and the second image frame should refer to the relevant descriptions in the following embodiments.

[0041] First, the application scenario of this application is described.

[0042] The product in the embodiment of this application is a DMS classified into a front-mounted DMS and a rear-mounted DMS. A pre-assembled DMS is generally designed and developed by an original equipment manufacturer (OEM) and is only compatible with the current vehicle model and current installation location. An aftermarket DMS is primarily designed and developed by relevant software and hardware suppliers and can be conveniently installed on various vehicle models and various locations. This product is mainly used in a cockpit environment to monitor the driver's status. This product promptly warns the driver in dangerous situations for the driver's safety. Furthermore, countries have stipulated that a DMS must be installed in some scenarios to promptly monitor dangerous driving behaviors and promptly prevent traffic accidents that may be caused by the driver.

[0043] 2 is a schematic diagram of the architecture of a DMS according to an exemplary embodiment of the present application. As shown in FIG. 2, the DMS may include a vehicle 21. The vehicle 21 may be a vehicle equipped with a wireless communication function. The wireless communication function may be set in an on-board terminal, an on-board module, an on-board unit, a chip (system), or other components or assemblies of the vehicle 21. The vehicle 21 in the embodiment of the present application is configured to monitor the driver's state and send attention reminder information when it is determined that the driver is fatigued and distracted. The attention reminder information instructs the driver to pay attention to safe driving.

[0044] At least one sensor 22, such as an onboard radar (e.g., microwave radar, lidar, or ultrasonic radar), a rain sensor, a camera, a vehicle attitude sensor (e.g., gyroscope), an inertial measurement unit (IMU), or a global navigation satellite system (GNSS), may be disposed on the vehicle 21. Other sensors may also be disposed on the vehicle 21.

[0045] The camera is configured to capture still images or videos. Generally, the camera may include a light-sensitive element such as a lens group and an image sensor. The lens group includes multiple lenses (convex or concave lenses) that collect light signals reflected by the object and transfer the collected light signals to the image sensor. The image sensor generates an original image of the object based on the light signals. For example, the camera is a DMS infrared camera. The image frame of the driver may be captured by at least one camera disposed in the vehicle 21. The image frame of the driver is an image frame that includes the driver's face.

[0046] Data such as point cloud data of the road surface and vertical acceleration of the vehicle 21 (i.e., acceleration data of the vehicle 21 in a direction perpendicular to the road surface) may further be collected by at least one on-board radar disposed on the vehicle 21.

[0047] Data such as point cloud data of the road surface and vertical acceleration of the vehicle 21 (i.e., acceleration data of the vehicle 21 in a direction perpendicular to the road surface) may further be collected by at least one sensor 22 disposed on the vehicle 21.

[0048] A driver monitoring system 23 may be further disposed in the vehicle 21, and the driver monitoring system 23 is configured to monitor the state of the driver. An automated driving system 24 may also be disposed in the vehicle 21. The driver monitoring system 23 assists the automated driving system 24. If the driver monitoring system 23 determines that the driver is fatigued or distracted, the automated driving system 24 can take control of the vehicle. The automated driving system 24 may be configured to generate an automated driving policy to be used to respond to road conditions based on data collected by sensors, and to perform automated driving of the vehicle 21 in accordance with the generated policy.

[0049] A human-machine interface (HMI) 25 may further be disposed in the vehicle 21. The human-machine interface 25 may be configured to broadcast, by visual icons or audio broadcast, the current road conditions and the policies being used by the automated driving system 24 for the vehicle 21 to alert the relevant driver and passengers.

[0050] A processor 26 may also be disposed in the vehicle 21. For example, the processor 26 is a high-performance computing processor. The processor 26 is configured to acquire a first image frame and a second image frame by a camera, where the first image frame and the second image frame are both image frames including the driver's face, and the first image frame is an image frame captured before the second image frame. The processor 26 is configured to acquire first detection information of the first image frame and second detection information of the second image frame by the driver monitoring system 23, and, when the first detection information and the second detection information both indicate the driver's eye state and head posture, and the second detection information indicates that the first eye state corresponding to the second image frame is a closed eye state, determine a second eye state corresponding to the second image frame based on the first detection information and the second detection information.

[0051] In a possible implementation, the DMS in the embodiment of the present application may further include a server. The server may be located in the vehicle 21 as an on-board computing unit or may be located on the cloud. The server may be a physical device or a virtual device such as a virtual machine or a container. The server has a wireless communication function. The wireless communication function may be built into a chip (system) or other component or assembly of the server. The server and the vehicle 21 may communicate with each other via a wireless connection method. For example, the server and the vehicle 21 may communicate with each other via a wireless communication method such as Wi-Fi, Bluetooth, frequency modulation (FM), radio modem, or satellite communication by using mobile communication technologies such as 2G / 3G / 4G / 5G. For example, in a test, the server may be carried in the vehicle 21 and communicate with the vehicle 21 via a wireless connection method. The server may collect data collected by sensors on one or more vehicles 21 or sensors located on roads or other locations for calculation through communication between the server and the vehicle 21, and send the calculation results back to the corresponding vehicle 21.

[0052] 3 is a schematic diagram of the architecture of a device for detecting a driver's state according to an exemplary embodiment of the present application. The device for detecting a driver's state may be implemented as a whole or part of the DMS of FIG. 2 by using a dedicated hardware circuit or a combination of software and hardware. The device for detecting a driver's state includes an image capture module 310, an eye state detection module 320, and a head pose detection module 330.

[0053] The image capture module 310 is configured to capture an image frame of the driver, the eye state detection module 320 is configured to detect the eye state of the driver within the image frame of the driver, and the head pose detection module 330 is configured to detect the pose of the driver's head within the image frame of the driver.

[0054] The device for detecting a driver's state may further include a separate fatigue-dependent detection module 340 and a fatigue state detection module 350. The separate fatigue-dependent detection module 340 is configured to detect another specified state of the driver in the image frame of the driver. The other specified state is a biological function state related to the driver's fatigue state. For example, the other specified state is a yawning state.

[0055] The eye state detection module 320 is further configured to input the detected eye state of the driver to the fatigue state detection module 350. The head posture detection module 330 is further configured to input the detected head posture of the driver to the fatigue state detection module 350. The other fatigue-dependent detection module 340 is further configured to input other detected specified states of the driver to the fatigue state detection module 350. Accordingly, the fatigue state detection module 350 is configured to comprehensively determine the fatigue state of the driver based on the input states (e.g., eye state, head posture, and other specified states).

[0056] The following uses the architecture of the DMS (hereinafter referred to as the system for short) provided in Figure 2 or Figure 3 as an example to describe the procedure of the method for detecting the driver's state provided in the embodiment of the present application.

[0057] 4 is a schematic flowchart of a method for detecting a driver's state according to an exemplary embodiment of the present application. As shown in FIG. 4, the procedure of the method for detecting a driver's state includes the following steps:

[0058] Step 401: Obtain a first image frame and a second image frame, where the first image frame and the second image frame are both image frames including a driver's face, and the first image frame is an image frame captured before the second image frame.

[0059] Optionally, the system captures a sequence of image frames by the camera, the sequence of image frames comprising at least two image frames, namely at least one first image frame and one second image frame.

[0060] The first image frame and the second image frame are both image frames that include the driver's face, ie, the first image frame and the second image frame both include the driver's full facial features.

[0061] The first image frame is an image frame captured before the second image frame. Optionally, the first image frame is an image frame captured at least two image frames before the second image frame.

[0062] Optionally, the at least one first image frame and the second image frame are a plurality of consecutive image frames.

[0063] Step 402: Obtain first detection information of a first image frame and second detection information of a second image frame, where the first detection information and the second detection information both indicate the driver's eye state and head posture.

[0064] Optionally, the system performs eye state detection and head pose detection on the first image frame to obtain first detection information. The first detection information indicates the driver's eye state and head pose corresponding to the first image frame. For example, the first detection information includes first eye detection information and first head detection information. The first eye detection information indicates the driver's eye state corresponding to the first image frame. The first head detection information indicates the driver's head pose corresponding to the first image frame. The eye state includes an eye closed state or an eye open state.

[0065] Optionally, the system performs eye state detection and head pose detection on the second image frame to obtain second detection information. The second detection information indicates the driver's eye state and head pose corresponding to the second image frame. For example, the second detection information includes second eye detection information and second head detection information. The second eye detection information indicates the driver's eye state corresponding to the second image frame. The second head detection information indicates the driver's head pose corresponding to the second image frame.

[0066] It should be noted that the eye state detection and head pose detection may be performed in parallel or sequentially, which is not limited in this embodiment of the present application. For details about the eye state detection and head pose detection, please refer to the related descriptions in the following embodiments, and the details will not be described here.

[0067] Step 403: If the second detection information indicates that the first eye state corresponding to the second image frame is a closed eye state, determine the second eye state corresponding to the second image frame based on the first detection information and the second detection information.

[0068] Optionally, the system determines whether the second detection information indicates that the first eye state corresponding to the second image frame is a closed-eye state. If the first eye state corresponding to the second image frame is a closed-eye state, the system determines a second eye state corresponding to the second image frame based on the first detection information and the second detection information. If the first eye state corresponding to the second image frame is an open-eye state, the process ends.

[0069] Optionally, if the first detection information and the second detection information satisfy a first preset condition, the system determines that the second eye state corresponding to the second image frame is an open-eye state, where the first preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an open-eye state.

[0070] For example, the first image frame includes a plurality of first image frames, and the meaning of "the first detection information indicates that the eye state corresponding to the first image frame is an open eye state" includes the first detection information indicating that the ratio of the number of first image frames in which the eye state is an open eye state to the total number of first image frames is greater than a first preset threshold.

[0071] The first preset threshold value may be set by default or may be set in a customized manner, for example, the first preset threshold value may be 0.9, which is not limited in this embodiment of the present application.

[0072] Optionally, if the first detection information and the second detection information satisfy a second preset condition, the system determines that the second eye state corresponding to the second image frame is an eye-closed state, where the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an eye-closed state.

[0073] For example, the first image frame includes a plurality of first image frames, and the meaning of "the first detection information indicates that the eye state corresponding to the first image frame is a closed-eye state" includes the first detection information indicating that the ratio of the number of first image frames in which the eye state is a closed-eye state to the total number of first image frames is greater than a second preset threshold.

[0074] The second preset threshold may be set by default or may be set in a customized manner, for example, the second preset threshold may be 0.95, which is not limited in this embodiment of the present application.

[0075] Optionally, if the second detection information indicates that the head pose corresponding to the second image frame does not jump in the pitch angle direction, the system determines that the second eye state corresponding to the second image frame is a closed eye state.

[0076] Optionally, the system determines a fatigue state detection result based on the head posture and the second eye state corresponding to the second image frame, and outputs alarm information if the fatigue state detection result satisfies a preset alarm condition.

[0077] Optionally, the system obtains a preset fatigue detection model, and outputs the fatigue detection model based on the head pose and the second eye state corresponding to the second image frame to obtain a fatigue state detection result. The fatigue detection model indicates a correlation between the head pose and the fatigue state and a correlation between the eye state and the fatigue state, and the fatigue detection model is a pre-trained model based on the sample image frame. For example, the fatigue detection model is a model obtained by fusion based on the eye state and the head pose. Alternatively, the fatigue detection model is a model obtained by fusion based on the eye state, the head pose, and the yawning state. Alternatively, the fatigue detection model is a model obtained by fusion based on the eye state, the head pose, the yawning state, and other information. This is not limited to this embodiment of the present application.

[0078] In a possible implementation, the fatigue state detection result includes one of a first detection result and a second detection result, the first detection result indicating that the driver is in a fatigued state and the second detection result indicating that the driver is not fatigued.

[0079] In this implementation, outputting alarm information when the fatigue state detection result satisfies the preset alarm condition includes outputting alarm information when the fatigue state detection result is the first detection result.

[0080] In another possible implementation, the fatigue state detection result includes a fatigue state level, and the fatigue state level is related to the driver's predicted fatigue intensity. For example, there is a positive correlation between the fatigue state level and the driver's predicted fatigue intensity, that is, the higher the fatigue state level, the greater the driver's predicted fatigue intensity.

[0081] In this implementation, outputting alarm information when the fatigue state detection result satisfies a preset alarm condition includes outputting alarm information when the fatigue state level is greater than a preset level threshold.

[0082] The preset level threshold may be set by default or in a customized manner, which is not limited in this embodiment of the present application.

[0083] Optionally, the system outputs the alarm information based on a preset prompt format, which includes at least one of a voice format, a text format, an image format, and an animation format. The output method and output content of the alarm information are not limited in this embodiment of the present application.

[0084] In summary, in this embodiment of the present application, first detection information of a first image frame and second detection information of a second image frame (e.g., a driver's continuous eye state and corresponding continuous head posture) are obtained. For different scenarios, such as a head-down-looking-down scenario and a head-down-dozing scenario, a second eye state corresponding to the second image frame can be determined based on the first detection information and the second detection information if the first eye state corresponding to the second image frame is determined to be a closed-eye state. That is, the eye state corresponding to the second image frame is corrected to obtain an accurate eye state. This solves the problem in the related art that when the driver lowers his / her head, the system cannot accurately determine the open / closed-eye state by directly using the image, and ensures a highly accurate and robust open / closed-eye discrimination effect, which is helpful in subsequently obtaining a stable and accurate fatigue state of the user.

[0085] 5 is a schematic flowchart of a method for detecting a driver's state according to another exemplary embodiment of the present application. As shown in FIG. 5, the procedure of the method for detecting a driver's state includes the following steps:

[0086] Step 501: Perform image capture by a camera to obtain a sequence of image frames.

[0087] Optionally, the system performs image capture by a camera located at at least one location within the vehicle to obtain a sequence of image frames, the camera may be an infrared camera.

[0088] Optionally, the at least one position includes one or more of the following positions: a position on or near the steering column and dashboard, a position on or near the center console, a position on or close to pillar A, and a position on or close to the rearview mirror. If the camera is placed on the steering column in a position that is optimal for identifying the eye state, it is unlikely that the eye state will be accurately identified when the head is lowered. However, in this position, the steering wheel is likely to block the face, and as a result, the eye state may not be identified. If the camera is placed at pillar A or the rearview mirror, the steering wheel will not block the face during image capture, but a scenario will inevitably occur in which the eye state will not be accurately identified when the head is lowered.

[0089] Optionally, performing image capture by the camera to obtain the image frame sequence includes performing image capture by the camera to obtain the image frame sequence when the vehicle is in a driving state, and / or performing image capture by the camera to obtain the image frame sequence when the vehicle's traveling speed exceeds a preset vehicle speed, and / or performing image capture by the camera to obtain the image frame sequence after it is detected that the vehicle is ignited, and / or performing image capture by the camera to obtain the image frame sequence when a vehicle start command is detected, and / or performing image capture by the camera to obtain the image frame sequence when a control command for the vehicle or a component or system within the vehicle is detected. It should be noted that the trigger conditions and capture manner for image capture are not limited in this embodiment of the present application.

[0090] Step 502: Perform face detection on the captured image frame sequence.

[0091] Face detection is the basis for other face applications. The face-based eye state detection algorithm and head pose algorithm are affected by the frontal face detection algorithm. If the positioning of the face detection algorithm is jittery or inaccurate, resulting in false positives or false negatives, the eye state will be misidentified or there will be large jitter in the head pose. As a result, the system will mistakenly identify a normal state as a head-down event. Only when the presence or absence of a face and accurate face position information are obtained can the system obtain stable and accurate eye state results.

[0092] Optionally, the system performs face detection on the captured image frame sequence by using a preset face detection algorithm.

[0093] In a possible implementation, the preset face detection algorithm is a dedicated face detection algorithm, such as a multi-tasking face detection algorithm. The multi-tasking face detection algorithm is used to detect a face and output facial keypoint information, so that a subsequent system can correct the face by using the facial keypoint information, provide better input for the back-end eye state detection algorithm, and improve the accuracy of the single-frame eye state identification algorithm. The dedicated face detection algorithm has high accuracy. However, since it takes time to deploy the algorithm inside the vehicle, the algorithm is not suitable for deployment inside the vehicle.

[0094] In another possible implementation, the preset face detection algorithm is a lightweight face detection algorithm, which is suitable for being placed inside the vehicle.

[0095] In another possible implementation, the preset face detection algorithm is a general target detection algorithm, such as a single-phase target detection algorithm, which is fast and accurate inside the vehicle and suitable for deployment in the vehicle infotainment system as a face detection algorithm. It should be noted that the preset face detection algorithm is not limited to this embodiment of the present application.

[0096] Step 503: Determine whether a face is detected in the image frame sequence.

[0097] If the system does not detect a face in the image frame sequence, execution continues with step 501. If the system detects a face in the image frame sequence, step 504 is executed.

[0098] Step 504: If a face is detected in the sequence of image frames, perform eye state detection to obtain a sequence of eye states, and perform face pose detection to obtain a sequence of face poses.

[0099] Optionally, for each image frame in the image frame sequence, the system obtains eye state detection data and face pose detection data corresponding to the image frame, and performs eye state detection and head pose detection to obtain an eye state sequence and a head pose sequence corresponding to the image frame sequence, i.e., the eye state sequence includes eye state detection data corresponding to each of the multiple image frames in the image frame sequence, and the head pose sequence includes head pose detection data corresponding to each of the multiple image frames in the image frame sequence.

[0100] In the case of single-frame eye state detection, in a possible implementation, the system may perform eye state detection by using a target detection algorithm. Eye state identification performed based on the target detection algorithm has high robustness and is unlikely to be obstructed, so scenarios such as wearing a mask, having the face obscured by hands, and wearing makeup are easily supported. Eye state identification is only affected in scenarios where the eyes are obscured or obstructed. However, the target detection algorithm cannot distinguish between a scenario where the eyes are squinting and a scenario where the eyes are squinting due to fatigue, resulting in the two states being mistaken for each other. The eye state identification algorithm can be applied to scenarios where the amount of face data is small and cannot cover all face samples in the driving environment for all scenarios.

[0101] In another possible implementation, since the size of the face and the size of the eyes vary slightly, the system may perform the detection of the eye state by using a general target detection algorithm after pruning, and predict the eye state by using only some network branches, which can greatly improve the detection speed while ensuring accuracy, and has a greater advantage inside the vehicle.

[0102] In another possible implementation, the system may perform eye state detection by using a preset classification algorithm. Eye state detection performed based on the preset classification algorithm can be classified into two cases. In one case, the eye state is determined based on an image frame containing a face. Because the ratio of the area covered by the eyes to the area of ​​the face is very small, interference from other parts of the face can easily cause misidentification. For example, wearing a mask or makeup can both cause misidentification. Furthermore, this method cannot distinguish between invalid states caused by whether the eyes are occluded or not. In the other case, an eye detection model is added, and then an image frame containing the eyes is input to the eye detection model and output to obtain the eye state. The eye detection model is a pre-trained neural network model used to identify the eye state in the image frame. Compared to the above case, this method is not obstructed by other parts of the eyes and has higher robustness and accuracy. This method cannot distinguish between scenarios where the eyes are squinting and scenarios where the eyes are fatigued. Furthermore, since the eye detection model is added, resource consumption increases and performance inside the vehicle decreases.

[0103] In another possible implementation, the system may perform eye condition detection by using a keypoint algorithm. Optionally, as shown in FIG. 6, the system may use six keypoint locations "p" on the upper eyelid, lower eyelid, and corner of the eye. 43 , p 44 , p 45 , p 46 , p 47 , and p 48 ” and then calculate the normalized distance EAR by the following formula:

number

[0104] The normalized distance is used as an eye-open indicator. If the eye-open indicator is smaller than a preset eye-open threshold, the eye is determined to be in a closed state; if the eye-open indicator is equal to or greater than the preset eye-open threshold, the eye is determined to be in an open state. The preset eye-open threshold can be set by default or in a customized manner. For example, the preset eye-open threshold is 20% of the normal eye-open value. This is not limited to this embodiment of the present application. Based on the keypoint algorithm and head pose calibration, the eye-slanted scenario and the fatigued eye-squinting scenario can be distinguished, so the accuracy of identifying the eye state in the two scenarios is slightly improved. The eye-open / eye-closed status based on the eye-open indicator is highly suitable for use as a basis for subsequent determination of the driver's fatigue state. However, this method has high requirements for the stability of the keypoint algorithm, requiring that large jitter does not occur at the keypoint position, that is, the corresponding eye-opening must be stable. In order to avoid large jitter in the eye opening, a filtering method may be added, that is, the system performs a filtering process on the output eye opening indicator to obtain a more appropriate eye opening indicator.

[0105] For single-frame head pose detection, the system maps two-dimensional image frames to three-dimensional image frames to obtain head pose detection data. The head pose detection data indicates the orientation of the driver's face. Optionally, the head pose detection data is expressed using three degrees of freedom (dof): pitch angle, roll angle, and yaw angle. As shown in FIG. 7, the head pose detection data includes the pitch angle between the driver's head and the horizontal axis of the device coordinate system, the yaw angle between the head and the longitudinal axis of the device coordinate system, and the roll angle between the head and the vertical axis of the device coordinate system.

[0106] The accuracy and stability of head pose detection are crucial for the whole eye condition identification system. Furthermore, due to the camera mounting position and head-down reasons, the head pose detection algorithm needs to support large angle head poses.

[0107] In a possible implementation, the system performs head pose detection based on facial keypoints. This method can be trained based on a large amount of existing keypoint data and can obtain keypoint information for each part of the face. Since the disturbance of points has little effect on the angle, the head pose is more stable.

[0108] In another possible implementation, the system performs head pose detection based on a regression algorithm. This method can directly regress Euler angles, which is simple and convenient. Model design is straightforward, and no post-processing or quadratic algorithm optimization is required for the solution. The difficulty of a regression-based head pose detection algorithm lies in collecting head pose data. Head pose data collection methods include a data generation method and a direct collection method. The direct collection method involves directly collecting head pose tags in the actual camera coordinate system using an optical tracker and camera array, allowing 360° head pose data coverage. However, the device is expensive, the deployment is complicated, the collection periodicity is long, and the human investment is large. The data generation method involves performing a 3D morphable model (3DMM) fitting to keypoint data in a 2D image frame, and then performing head pose enhancement to obtain head pose taps at large angles. The method can support head pose data at ±90°. This method relies only on 2D keypoint markings, making it both feasible and cost-effective.

[0109] In another possible implementation, the system performs head pose detection based on face keypoints and a regression algorithm. Optionally, the system inputs image frames into a head pose detection model and outputs them to obtain head pose data. The head pose detection model is a multi-task neural network model based on face keypoints and regression methods, that is, tasks such as Euler angle regression and face keypoints are performed within one network model. Through a shared skeleton network, multiple head networks can be used to perform different tasks, and the prediction error of Euler angles can be reduced to a certain extent using this method.

[0110] It should be noted that the methods of the eye state detection algorithm and head pose detection algorithm are not limited in this embodiment of the present application.

[0111] Step 505: Determine whether the first eye state corresponding to the second image frame is an eye-closed state.

[0112] Eye state detection and head pose detection are performed for each frame in the image frame sequence, the system obtains eye state cues and head pose cues corresponding to the image frame sequence, and the system determines whether the first eye state in the latest frame, i.e., the second image frame, in the image frame sequence is a closed eye state.

[0113] If the first eye state corresponding to the second image frame is an eyes-open state, the process ends. If the first eye state corresponding to the second image frame is an eyes-closed state, the system continues to execute step 506. The following three scenarios are distinguished by checking the eye state cues and head pose cues corresponding to the image frame sequence: normal eyes-closed scenario, head-down, eyes-open, looking down scenario, and head-down and dozing scenario.

[0114] Step 506: If the first eye state corresponding to the second image frame is an eye-closed state, determine whether the head pose corresponding to the second image frame jumps in the pitch angle direction.

[0115] If the first eye state corresponding to the second image frame is a closed eye state, the system determines whether the head pose corresponding to the second image frame jumps in the pitch angle direction. If the head pose corresponding to the second image frame does not jump in the pitch angle direction, step 507 is executed. If the head pose corresponding to the second image frame jumps in the pitch angle direction, step 508 is executed.

[0116] Step 507: If the head posture corresponding to the second image frame does not jump in the pitch angle direction, output that the second eye state corresponding to the second image frame is the eye-closed state.

[0117] If the head pose corresponding to the second image frame does not jump in the pitch angle direction, the scenario is determined to be a normal eyes-closed scenario, and the second eye state corresponding to the second image frame is output as an eyes-closed state.

[0118] In an illustrative example, as shown in Figure 8, the head pose does not jump in pitch angle direction when the eyes change from an open (open) state to a closed (occluded) state, the system determines that the scenario is a normal eyes-closed scenario, and the eye state is maintained in the original closed state to obtain a corrected eye state.

[0119] Step 508: If the head pose corresponding to the second image frame jumps in the pitch angle direction, determine whether the eye state corresponding to the first image frame is a closed eye state.

[0120] If the head pose corresponding to the second image frame jumps in the pitch angle direction, the system determines whether the eye state corresponding to the first image frame is a closed-eye state. If the eye state corresponding to the first image frame is an open-eye state, step 509 is executed. If the eye state corresponding to the first image frame is a closed-eye state, step 510 is executed.

[0121] Step 509: If the eye state corresponding to the first image frame is the eye open state, output that the second eye state corresponding to the second image frame is the eye open state.

[0122] If the eye state corresponding to the first image frame is an open-eye state, the scenario is determined to be a head-down, eyes-open, looking-down scenario, and a second eye state corresponding to the second image frame is output as an open-eye state.

[0123] In an illustrative example, the head pose jumps in the pitch angle direction when the eyes change from an open (open) state to a closed (occluded) state, as shown in Figure 9. The system determines that the scenario is a head-down, eyes-open, looking-down scenario, and the eye state is corrected to the open (open) state to obtain a corrected eye state.

[0124] Step 510: If the eye state corresponding to the second image frame is the eye-closed state, output that the second eye state corresponding to the second image frame is the eye-closed state.

[0125] If the eye state corresponding to the first image frame is the eyes closed state, it is determined that the scenario is a scenario of dozing with the head down, and it is output that the second eye state corresponding to the second image frame is the eyes closed state.

[0126] In an illustrative example, the head pose jumps in the pitch angle direction when the eyes change from an open (open) state to a closed (occluded) state, as shown in Figure 10. The system determines that the scenario is a head-down dozing scenario, and the eye state is maintained as the original closed state to obtain a corrected eye state.

[0127] Optionally, if the first eye state corresponding to the second image frame is a closed-eye state, the system can distinguish between three scenarios based on the eye state sequence and the head pose sequence: a normal eyes-closed scenario, a head-down-looking-down scenario, and a head-down-dozing scenario. If only the eye state sequence is used as an input parameter for the fatigue state, false alarms may occur in these three scenarios, which affects the effectiveness of the alarm system. A head pose detection module is added, which can easily distinguish between two scenarios: a normal eyes-closed scenario and a head-down-eyes-open scenario. When a closed-eye state is detected, a check is made to see if the head pose jumps. If the head pose does not jump, the scenario is determined to be a normal eyes-closed scenario, and the result is not corrected, and the original eye state is maintained. If the head pose jumps, the scenario is determined to be a head-down-looking-down scenario, and the original closed-eye state is corrected to an open-eye state. However, in this case, the head-down-dozing event is incorrectly determined to be a head-down-looking-down scenario. As a result, the eye state is incorrectly corrected. To eliminate the case where a head-down dozing event is erroneously corrected to an eyes-open state, a stable eye state cue is added. Two scenarios, i.e., a head-down, eyes-open, and looking-down scenario, and a head-down dozing event scenario, are distinguished by checking the eye state of the previous consecutive frame, i.e., the eye state corresponding to the first image frame. If the eye state corresponding to the first image frame is the eyes-closed state, the scenario is determined to be a head-down dozing scenario, and the result is not corrected, and the original eyes-closed state is maintained. If the eye state corresponding to the first image frame is the eyes-open state, the scenario is determined to be a head-down, eyes-open, and looking-down scenario, and the original eyes-closed state is corrected to the eyes-open state.

[0128] In an illustrative example, the process of distinguishing between the three scenarios and the correction process is shown in FIG. 11. In FIG. 11, the first row represents the head pose sequence of the head pose in the pitch angle direction, and the second row represents the corresponding eye state sequence without post-processing. The third row represents the corrected eye state sequence obtained after the head pose detection module is added. As can be seen from FIG. 11, the eye state in the head-down dozing scenario is incorrectly corrected to the eyes-open state. The fourth row represents the accurate eye state obtained after the eye stable state cue is further added to further correct the eye state.

[0129] In summary, the method for detecting driver states provided in this embodiment of the present application can accurately distinguish three scenarios that have little difference when analyzed on images: a normal eye-closed scenario, a downward-looking scenario, and a dozing-off scenario. Furthermore, the false recognition of eye-closed caused by looking down can be accurately restored to an open-eye state by using stable state cues, and the closed-eye state in the dozing-off scenario will not be falsely corrected. Furthermore, distinguishing different scenarios based on head pose sequences and eye state sequences has high fault tolerance and greatly improves the accuracy of identifying eye states in driving scenarios.

[0130] Apparatus embodiments of the present application are provided below and can be configured to perform method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0131] 12 is a block diagram of an apparatus for detecting a driver's state according to an exemplary embodiment of the present application. The apparatus may be implemented as a whole or part of the DMS provided in FIG. 2 or the apparatus for detecting a driver's status provided in FIG. 3 by using software, hardware, or a combination of software and hardware. The apparatus may include a first obtaining unit 1210, a second obtaining unit 1220, and a determining unit 1230.

[0132] The first acquisition unit 1210 is configured to acquire a first image frame and a second image frame, both of which include a face of a driver. Image Frame where the first image frame is the image frame captured before the second image frame.

[0133] The second acquisition unit 1220 is configured to acquire first detection information of a first image frame and second detection information of a second image frame, where both the first detection information and the second detection information indicate the driver's eye state and head posture.

[0134] The determination unit 1230 is configured to determine a second eye state corresponding to the second image frame based on the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is a closed-eye state.

[0135] In a possible implementation, the decision unit 1230 is The device is further configured to determine that the second eye state corresponding to the second image frame is an open eye state when the first detection information and the second detection information satisfy a first preset condition.

[0136] The first preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an open eye state.

[0137] In another possible implementation, the decision unit 1230 The device is further configured to determine that the second eye state corresponding to the second image frame is a closed eye state when the first detection information and the second detection information satisfy a second preset condition.

[0138] The second preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is a closed eye state.

[0139] In another possible implementation, the decision unit 1230 The system is further configured to determine that the second eye state corresponding to the second image frame is a closed eye state when the second detection information indicates that the head pose corresponding to the second image frame does not jump in the pitch angle direction.

[0140] In another possible implementation, the device further comprises a detection module.

[0141] The detection module is configured to determine a fatigue state detection result based on the head pose and the second eye state corresponding to the second image frame.

[0142] In another possible implementation, the device further comprises an alarm module.

[0143] The alarm module is configured to output alarm information when the fatigue state detection result meets a preset alarm condition.

[0144] It should be noted that when the device provided in the above embodiments implements the functions of the device, the division of the above functional modules is merely used as an example for description. In actual applications, the above functions may be assigned to different functional modules according to the required implementation, that is, the internal structure of the device is divided into different functional modules to implement all or part of the above functions. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept. For the specific implementation process of the device, please refer to the method embodiments. The details will not be described again here.

[0145] An embodiment of the present application provides an apparatus for detecting a driver's state, the apparatus including a processor and a memory configured to store processor-executable instructions, and when the processor is configured to execute the instructions, the method performed by the DMS in the above embodiment is implemented.

[0146] An embodiment of the present application provides a computer program product including computer-readable code or a non-volatile computer-readable storage medium carrying the computer-readable code, which, when executed by a processor, causes the processor to perform the method performed by the DMS in the above embodiments.

[0147] An embodiment of the present application provides a non-volatile computer-readable storage medium, which stores computer program instructions, which, when executed by a processor, implement the method performed by the DMS in the above embodiment.

[0148] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital video disc (DVD), memory stick, floppy disk, machine-encoded devices such as punch cards or slot-projection structures for storing instructions, and any suitable combination thereof.

[0149] The computer-readable program instructions or code described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in each computing / processing device for storage.

[0150] Computer program instructions used to carry out operations herein may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or source or target code written in any combination of one or more programming languages. Programming languages ​​include object-oriented programming languages ​​such as Smalltalk and C++, as well as traditional procedural programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. When a remote computer is involved, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., over the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as a programmable logic circuit, field-programmable gate array (FPGA), or programmable logic array (PLA), is customized by using the status information in the computer-readable program instructions. The electronic circuitry may execute the computer-readable program instructions to implement various aspects of the present application.

[0151] Various aspects of the present application are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented by computer-readable program instructions.

[0152] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to generate a machine that, when executed by the processor of the computer or other programmable data processing apparatus, produces an apparatus that implements the functions / acts specified in one or more blocks in the flowcharts and / or block diagrams. These computer-readable program instructions may alternatively be stored on a computer-readable storage medium. These instructions cause the computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, a computer-readable medium storing instructions includes an artifact including instructions for implementing various aspects of the functions / acts specified in one or more blocks in the flowcharts and / or block diagrams.

[0153] The computer-readable program instructions may alternatively be loaded into a computer, other programmable data processing apparatus, or other device, such that the example operational steps are executed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process. Thus, the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more blocks in the flowcharts and / or block diagrams.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the system architecture, functionality, and operation of possible implementations of apparatuses, systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or portion of an instruction, which includes one or more executable instructions for implementing a particular logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, in practice, two consecutive blocks may be executed substantially in parallel, or may even be executed in the reverse order, depending on the functionality involved.

[0155] It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by hardware (e.g., a circuit or ASIC (application-specific integrated circuit)) that performs the corresponding function or operation, or may be implemented by a combination of hardware and software, e.g., firmware.

[0156] Although the present application has been described herein with reference to embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments in the course of practicing the claimed application, by studying the accompanying drawings, the disclosed content, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "one" does not exclude "plurality." A single processor or other unit may implement several functions recited in the claims. The fact that certain measures are recited in mutually different dependent claims does not indicate that these measures cannot be combined to advantageous effect.

[0157] The above has described embodiments of the present application. The above description is not exhaustive or limited to the disclosed embodiments. Many modifications and variations are possible within the scope of the described embodiments. Encircled The choice of terminology used herein is intended to best explain the principles, practical applications, or improvements to the art in the marketplace of the embodiments, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method for detecting a driver's state, comprising: acquiring a first image frame and a second image frame, the first image frame and the second image frame both including a face of the driver, the first image frame being an image frame captured before the second image frame; obtaining first detection information of the first image frame and second detection information of the second image frame, the first detection information and the second detection information indicating an eye state and a head posture of the driver; when the second detection information indicates that the first eye state corresponding to the second image frame is an eye-closed state, determining whether the head posture corresponding to the second image frame has changed in a pitch angle direction compared to the head posture corresponding to the first image frame based on the first detection information and the second detection information; determining whether the state of the eyes corresponding to the first image frame is an open eye state or a closed eye state based on the first detection information when the head posture corresponding to the second image frame has changed in a pitch angle direction compared to the head posture corresponding to the first image frame; determining that a second eye state corresponding to the second image frame is an open-eye state when the eye state corresponding to the first image frame is an open-eye state, and determining that a second eye state corresponding to the second image frame is an closed-eye state when the eye state corresponding to the first image frame is a closed-eye state; A method having the following.

2. The method comprises: determining that the second eye state corresponding to the second image frame is the closed eye state when the head pose corresponding to the second image frame has not changed in a pitch angle direction compared to the head pose corresponding to the first image frame. The method of claim 1.

3. The method comprises: determining a fatigue state detection result based on the head pose corresponding to the second image frame and the second eye state; 3. The method according to claim 1 or 2.

4. The method comprises: The method further includes outputting alarm information when the fatigue state detection result satisfies a preset alarm condition. The method of claim 3.

5. A device for detecting a driver's state, a first acquisition unit configured to acquire a first image frame and a second image frame, the first image frame and the second image frame both including a face of the driver, the first image frame being an image frame captured before the second image frame; a second acquisition unit configured to acquire first detection information of the first image frame and second detection information of the second image frame, the first detection information and the second detection information indicating an eye state and a head posture of the driver; a determining unit configured to determine a second eye state corresponding to the second image frame based on the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is an eye-closed state; and and The decision unit: when the second detection information indicates that the first eye state corresponding to the second image frame is a closed eye state, determining, based on the first detection information and the second detection information, whether a head posture corresponding to the second image frame has changed in a pitch angle direction compared to a head posture corresponding to the first image frame; determining whether the state of the eyes corresponding to the first image frame is an open eye state or a closed eye state based on the first detection information when the head posture corresponding to the second image frame has changed in a pitch angle direction compared to the head posture corresponding to the first image frame; If the eye state corresponding to the first image frame is an open eye state, the second eye state corresponding to the second image frame is determined to be an open eye state, and if the eye state corresponding to the first image frame is a closed eye state, the second eye state corresponding to the second image frame is determined to be a closed eye state. It is configured as follows: Device.

6. The decision unit: and determining that the second eye state corresponding to the second image frame is the closed eye state when the head pose corresponding to the second image frame indicates no change in pitch angle direction compared to the head pose corresponding to the first image frame.

6. The apparatus of claim 5.

7. The device further comprises a detection module. the detection module is configured to determine a fatigue state detection result based on the head pose corresponding to the second image frame and the second eye state; 7. Apparatus according to claim 5 or 6.

8. the device further comprises an alarm module; the alarm module is configured to output alarm information when the fatigue state detection result satisfies a preset alarm condition; 8. The apparatus of claim 7.

9. A non-volatile computer-readable storage medium storing computer program instructions, comprising: The computer program instructions, when executed by a processor, perform the method of any one of claims 1 to 4. A non-volatile computer-readable storage medium.

Images