Method and device for detecting driver's state, and storage medium

Abstract

The present application relates to the field of computer vision technology, and in particular to a method and device for detecting a driver's state and a storage medium. The method includes: acquiring a first image frame and a second image frame, the first image frame and the second image frame are both frame images including a driver's face, and the first image frame is an image frame captured before the second image frame; acquiring a first detection information of the first image frame and a second detection information of the second image frame, the first detection information and the second detection information indicate the driver's eye state and head posture; and when the second detection information indicates that the first eye state corresponding to the second image frame is a closed eye state, determining the second eye state corresponding to the second image frame based on the first detection information and the second detection information. According to the present application, three scenarios that have little difference when analyzed for an image, namely, a normal eyes closed scenario, a downward facing scenario, and a dozing scenario, can be accurately distinguished. This ensures an open / closed eye state identification effect with high accuracy and high robustness.

Description

[Technical field]

[0001] The present application relates to the field of computer vision technology, and in particular to a method and apparatus for detecting a driver's state, and a storage medium. [Background technology]

[0002] The human eye detection technology includes performing image detection on multiple image frames containing human eyes to estimate whether the eyes are currently in a closed or open state. With the popularity and development of video processing technology and video surveillance technology, the human eye detection technology has become an essential and important part of the eye image analysis process.

[0003] In the related art, an in-car scenario is used as an example. A camera is positioned relative to the driver's face. The system obtains multiple image frames including human eyes through the camera, and based on the multiple image frames, identifies whether the driver's eyes are open / closed, and then, in combination with the head posture, can accurately monitor the user's fatigue state.

[0004] However, in the above method, when the driver's head is turned down, the system cannot accurately determine the driver's eyes open / closed state based on the image acquired by the camera. As an example, three image frames acquired by the camera are shown in FIG. 1. Image frame 12 corresponds to the case of normally closing the eyes, image frame 14 corresponds to the case of head down and eyes open, and image frame 16 corresponds to the case of head down and dozing. However, from the camera's perspective, the driver's eyes appear to be closed in these three cases, and there is an obvious drawback in the accuracy and robustness of identifying the eyes open / closed state based directly on the image. This greatly reduces the accuracy of the user's fatigue state that is subsequently determined based on the eyes open / closed state. Summary of the Invention

[0005] In view of this, a method and apparatus for detecting a driver's state, and a storage medium are provided, which can ensure an eyes-open / closed discrimination effect with high accuracy and high robustness, and subsequently help to obtain a stable and accurate fatigue state of the user.

[0006] According to a first aspect, an embodiment of the present application provides a method for detecting a driver's state, the method comprising: acquiring a first image frame and a second image frame, the first image frame and the second image frame both being frame images including a face of a driver, the first image frame being an image frame captured prior to the second image frame; Obtaining first detection information of a first image frame and second detection information of a second image frame, the first detection information and the second detection information indicating an eye state and a head posture of a driver; determining a second eye state corresponding to the second image frame based on the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is an eye-closed state; Includes.

[0007] In this implementation, first detection information of a first image frame and second detection information of a second image frame (e.g., the driver's continuous eye state and corresponding continuous head posture) are obtained. For different scenarios, such as a head-down looking down scenario and a head-down dozing scenario, a second eye state corresponding to the second image frame can be determined based on the first detection information and the second detection information when the first eye state corresponding to the second image frame is determined to be a closed eye state. That is, the eye state corresponding to the second image frame is corrected to obtain an accurate eye state. This solves the problem in the related art that when the driver lowers his head, the system cannot accurately determine the open / closed eye state by directly using the image, and ensures a high-precision and high-robust open / closed eye discrimination effect, which is helpful to obtain the user's stable and accurate fatigue state thereafter.

[0008] In a possible implementation, determining a second eye state corresponding to a second image frame based on the first detection information and the second detection information includes: determining that a second eye state corresponding to the second image frame is an open eye state when the first detection information and the second detection information satisfy a first preset condition.

[0009] The first preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an open eye state.

[0010] In this implementation, when the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is in an open-eye state, the second eye state corresponding to the second image frame is determined to be an open-eye state, that is, the false detection of closed eyes caused by looking down is accurately restored to the open-eye state based on the head pose sequence and the eye state sequence, and the closed-eye state in the dozing scenario is not falsely corrected, and the eye state identification effect is further improved.

[0011] In another possible implementation, determining a second eye state corresponding to a second image frame based on the first detection information and the second detection information includes: The method further includes determining that a second eye state corresponding to the second image frame is a closed eye state when the first detection information and the second detection information satisfy a second preset condition.

[0012] The second preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is a closed eye state.

[0013] In this implementation, when the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is a closed-eye state, the second eye state corresponding to the second image frame is determined to be a closed-eye state. That is, the head-down dozing scenario is accurately determined based on the head pose sequence and the eye state sequence, and the eye state identification effect is further improved.

[0014] In another possible implementation, the method comprises: The method further includes determining that a second eye state corresponding to the second image frame is a closed eye state when the second detection information indicates that the head pose corresponding to the second image frame does not jump in the pitch angle direction.

[0015] In this implementation, the second detection information determines that the second eye state corresponding to the second image frame is an eyes-closed state when the head pose corresponding to the second image frame does not jump in the pitch angle direction, that is, the normal eyes-closed scenario is accurately determined by using the head pose sequence, and the eye state identification effect is further improved.

[0016] In another possible implementation, the method comprises: The method further includes determining a fatigue state detection result based on a head pose and a second eye state corresponding to the second image frame.

[0017] In another possible implementation, the method comprises: The method further includes outputting alarm information when the fatigue state detection result satisfies a preset alarm condition.

[0018] In this implementation, when the fatigue state detection result meets the preset alarm condition, alarm information is output, so that dangerous driving behavior is monitored, the driver is warned in a timely manner, and the driver's possible traffic accident is prevented.

[0019] According to a second aspect, an embodiment of the present application provides an apparatus for detecting a driver's state, the apparatus comprising: a first capture unit configured to capture a first image frame and a second image frame, the first image frame and the second image frame both being frame images including a face of a driver, the first image frame being an image frame captured prior to the second image frame; a second acquisition unit configured to acquire first detection information of a first image frame and second detection information of a second image frame, the first detection information and the second detection information being indicative of an eye state and a head posture of a driver; a determining unit configured to determine a second eye state corresponding to the second image frame based on the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is an eye-closed state; Includes.

[0020] In another possible implementation, the decision unit The device is further configured to determine that a second eye state corresponding to the second image frame is an open eye state when the first detection information and the second detection information satisfy a first preset condition.

[0021] The first preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an open eye state.

[0022] In another possible implementation, the decision unit The device is further configured to determine that a second eye state corresponding to the second image frame is a closed eye state when the first detection information and the second detection information satisfy a second preset condition.

[0023] The second preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is a closed eye state.

[0024] In another possible implementation, the decision unit The system is further configured to determine that a second eye state corresponding to the second image frame is a closed eye state when the second detection information indicates that the head pose corresponding to the second image frame does not jump in the pitch angle direction.

[0025] In another possible implementation, the device further comprises a detection module.

[0026] The detection module is configured to determine a fatigue state detection result based on the head pose and the second eye state corresponding to the second image frame.

[0027] In another possible implementation, the device further comprises an alarm module.

[0028] The alarm module is configured to output alarm information when the fatigue state detection result meets a preset alarm condition.

[0029] According to a third aspect, an embodiment of the present application provides an apparatus for detecting a driver's state, the apparatus comprising: A processor; and a memory configured to store processor-executable instructions.

[0030] The processor is configured to perform the method provided in the first aspect or any one of the possible implementations of the first aspect when executing the instructions.

[0031] According to a fourth aspect, an embodiment of the present application provides a non-volatile computer readable storage medium, the non-volatile computer readable storage medium storing computer program instructions, which, when executed by a processor, perform the method provided in the first aspect or any one of the possible implementations of the first aspect.

[0032] According to a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising computer readable code or a non-volatile computer readable storage medium carrying the computer readable code, when the computer program code is executed on an electronic device, a processor in the electronic device performs the method provided in the first aspect or any one of the possible implementations of the first aspect.

[0033] According to a sixth aspect, an embodiment of the present application provides a vehicle, the vehicle comprising an apparatus provided in the second aspect or any one of the possible implementations of the second aspect.

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, and the specification together illustrate exemplary embodiments, features, and aspects of the present application and are intended to explain the principles of the present application. [Brief description of the drawings]

[0035] [Figure 1] FIG. 2 is a schematic diagram of three image frames captured by a camera. [Diagram 2] FIG. 1 is a schematic diagram of the architecture of a DMS according to an exemplary embodiment of the present application. [Diagram 3] FIG. 2 is a schematic diagram of the architecture of a device for detecting a driver's state according to an exemplary embodiment of the present application; [Figure 4] 3 is a schematic flow chart of a method for detecting a driver's state according to an exemplary embodiment of the present application; [Diagram 5]4 is a schematic flow chart of a method for detecting a driver's state according to another exemplary embodiment of the present application; [Figure 6] FIG. 2 is a schematic diagram of the principle of an eye condition detection method according to an exemplary embodiment of the present application; [Figure 7] FIG. 2 is a schematic diagram of the principle of the head pose detection method according to an exemplary embodiment of the present application; [Figure 8] FIG. 2 is a schematic diagram of a state sequence correction process according to an exemplary embodiment of the present application; [Figure 9] FIG. 13 is a schematic diagram of a state sequence correction process according to another exemplary embodiment of the present application; [Figure 10] FIG. 13 is a schematic diagram of a state sequence correction process according to another exemplary embodiment of the present application; [Figure 11] FIG. 2 is a schematic diagram of three scenarios in a method for detecting a driver's state and the principle of a correction process according to an exemplary embodiment of the present application; [Figure 12] FIG. 1 is a block diagram of an apparatus for detecting a driver's state according to an exemplary embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Various exemplary embodiments, features, and aspects of the present application are described in detail below with reference to the accompanying drawings, in which the same reference numerals represent elements having the same or similar functions. Although various aspects of the embodiments are illustrated in the accompanying drawings, the accompanying drawings are not necessarily drawn to scale unless specifically specified otherwise.

[0037] The specific word "exemplary" herein means "serving as an example, embodiment, or illustration." Any embodiment described herein as an "exemplary" is not necessarily being described as superior or better than other embodiments.

[0038] Furthermore, in order to better describe the present application, numerous specific details are given in the following specific implementations. It should be understood by those skilled in the art that the present application can be practiced without some of the specific details. In some instances, methods, means, elements, and circuits that are well known to those skilled in the art are not described in detail so as to emphasize the subject matter of the present application.

[0039] The fatigue driving warning system accurately identifies the driver's eye state, yawning behavior, head posture, and risky behavior through the driver monitoring system (DMS) in the driver's seat, and accurately determines the driver's fatigue state by using these information, and provides alarm information to ensure the user's driving safety. The eye state identification technology is an important module of the fatigue driving warning system. The driver's fatigue degree can be analyzed based on the frequency of continuous eye closure and blinking, and whether the driver's vision is obstructed can be analyzed based on the continuous eye closure duration, so that the user's eye state is accurately identified and accurate fatigue warning is realized. The eye state identification with high accuracy and high robustness is helpful to obtain the stable and accurate fatigue state of the user. Especially in the in-car scenario, not only high accuracy is required, but also the algorithm needs to adapt to various ambient light and various mounting position scenarios of the camera. The camera is a camera used to monitor the driver in the DMS. When the camera is mounted on the vehicle pillar A or rearview mirror position and the driver's head is facing down, the eyes open / closed state cannot be accurately determined based on the image acquired from the camera. In the related art, the accuracy and robustness of analyzing the eyes open / closed state based on the image alone have various shortcomings. This significantly reduces the accuracy of the user's fatigue state that is subsequently determined based on the eyes open / closed state.

[0040] The embodiment of the present application provides a method and device for detecting a driver's state, and a storage medium. When the first eye state corresponding to the second image frame is a closed eye state, the system determines the second eye state corresponding to the second image frame based on the first detection information of the first image frame (i.e., the image frame captured before the second image frame) and the second detection information of the second image frame, corrects the eye state corresponding to the second image frame, and obtains the correct eye state. This solves the problem in the related art that the system cannot correctly determine the open / closed eye state by directly using the image when the driver lowers his head, ensures the open / closed eye discrimination effect with high accuracy and high robustness, and helps to obtain the stable and accurate fatigue state of the user thereafter. It should be noted that the definitions of the first image frame and the second image frame should refer to the relevant descriptions in the following embodiments.

[0041] First, the application scenario of this application is described.

[0042] The product in the embodiment of this application is a DMS classified into front-mounted DMS and rear-mounted DMS. The pre-assembled DMS is generally designed and developed by the original equipment manufacturer OEM, and only fits the current vehicle model and current installation position. The retrofit DMS is mainly designed and developed by the related software and hardware supplier, and can be conveniently installed in various vehicle models and various positions. This product is mainly used in the cockpit environment to monitor the driver's state. This product will promptly warn the driver in dangerous situations for the driver's safety. Furthermore, the country specifies that the DMS needs to be installed in some scenarios, so as to promptly monitor the dangerous driving behavior and promptly prevent the traffic accidents that may be caused by the driver.

[0043] FIG. 2 is a schematic diagram of the architecture of a DMS according to an exemplary embodiment of the present application. As shown in FIG. 2, the DMS may include a vehicle 21. The vehicle 21 may be a vehicle equipped with a wireless communication function. The wireless communication function may be set in an on-board terminal, an on-board module, an on-board unit, a chip (system), or other components or assemblies of the vehicle 21. The vehicle 21 in the embodiment of the present application is configured to monitor the driver's state and send attention reminder information when it is determined that the driver is in a fatigued and distracted state. The attention reminder information instructs the driver to pay attention to safe driving.

[0044] At least one sensor 22, such as an on-board radar (e.g., microwave radar, Lidar, or ultrasonic radar), a rain sensor, a camera, a vehicle attitude sensor (e.g., a gyroscope), an inertial measurement unit (IMU), or a global navigation satellite system (GNSS), may be disposed on the vehicle 21. Other sensors may also be disposed on the vehicle 21.

[0045] The camera is configured to capture still images or videos. In general, the camera may include light-sensitive elements such as a lens group and an image sensor. The lens group includes multiple lenses (convex or concave lenses) to collect light signals reflected by the object and transfer the collected light signals to the image sensor. The image sensor generates an original image of the object based on the light signals. For example, the camera is a DMS infrared camera. The image frame of the driver may be captured by at least one camera disposed in the vehicle 21. The image frame of the driver is an image frame that includes the driver's face.

[0046] Data such as point cloud data of the road surface and vertical acceleration of the vehicle 21 (i.e., acceleration data of the vehicle 21 in a direction perpendicular to the road surface) may further be collected by at least one on-board radar disposed on the vehicle 21.

[0047] Data such as point cloud data of the road surface and vertical acceleration of the vehicle 21 (i.e. acceleration data of the vehicle 21 in a direction perpendicular to the road surface) may further be collected by at least one sensor 22 disposed on the vehicle 21.

[0048] A driver monitoring system 23 may be further arranged in the vehicle 21, and the driver monitoring system 23 is configured to monitor the state of the driver. An automatic driving system 24 may be further arranged in the vehicle 21. The driver monitoring system 23 assists the automatic driving system 24. When the driver monitoring system 23 determines that the driver is fatigued and distracted, the automatic driving system 24 can take control of the vehicle. The automatic driving system 24 may be configured to generate an automatic driving policy used to respond to road conditions based on data collected by the sensors, and to perform automatic driving of the vehicle 21 according to the generated policy.

[0049] A human-machine interface (HMI) 25 may further be disposed in the vehicle 21. The human-machine interface 25 may be configured to broadcast, by visual icons or audio broadcast, the current road conditions and the policies being used by the automated driving system 24 for the vehicle 21 to alert the relevant driver and passengers.

[0050] A processor 26 may be further disposed in the vehicle 21. For example, the processor 26 is a high performance computing processor. The processor 26 is configured to acquire a first image frame and a second image frame by a camera, where both the first image frame and the second image frame are image frames including a face of a driver, and the first image frame is an image frame captured before the second image frame. The processor 26 is configured to acquire first detection information of the first image frame and second detection information of the second image frame by the driver monitoring system 23, and determine a second eye state corresponding to the second image frame based on the first detection information and the second detection information when the first detection information and the second detection information both indicate an eye state and a head posture of the driver, and the second detection information indicates that the first eye state corresponding to the second image frame is an eye closed state.

[0051] In a possible implementation, the DMS in the embodiment of the present application may further include a server. The server may be located in the vehicle 21 as an on-board computing unit, or may be located on the cloud. The server may be a physical device, or may be a virtual device such as a virtual machine or a container. The server has a wireless communication function. The wireless communication function may be set in the chip (system) or other components or assemblies of the server. The server and the vehicle 21 may communicate with each other in a wireless connection manner. For example, the server and the vehicle 21 may communicate with each other in a wireless communication manner such as Wi-Fi, Bluetooth, frequency modulation (FM), radio modem, or satellite communication by using a mobile communication technology such as 2G / 3G / 4G / 5G. For example, in a test, the server may be carried in the vehicle 21 and communicate with the vehicle 21 in a wireless connection manner. The server may collect data collected by sensors on one or more vehicles 21 or sensors placed on the road or other places for calculation, and send the calculation result back to the corresponding vehicle 21 through the communication between the server and the vehicle 21.

[0052] 3 is a schematic diagram of the architecture of a device for detecting a driver's state according to an exemplary embodiment of the present application. The device for detecting a driver's state can be implemented as a whole or part of the DMS of FIG. 2 by using a dedicated hardware circuit or a combination of software and hardware. The device for detecting a driver's state includes an image capture module 310, an eye state detection module 320, and a head pose detection module 330.

[0053] The image capture module 310 is configured to capture an image frame of the driver. The eye state detection module 320 is configured to detect the driver's eye state within the driver's image frame. The head pose detection module 330 is configured to detect the driver's head pose within the driver's image frame.

[0054] The device for detecting the driver's state may further include an other fatigue-dependent detection module 340 and a fatigue state detection module 350. The other fatigue-dependent detection module 340 is configured to detect another specified state of the driver in the image frame of the driver. The other specified state is a biofunctional state related to the fatigue state of the driver. For example, the other specified state is a yawning state.

[0055] The eye state detection module 320 is further configured to input the detected eye state of the driver to the fatigue state detection module 350. The head posture detection module 330 is further configured to input the detected head posture of the driver to the fatigue state detection module 350. The other fatigue-dependent detection module 340 is further configured to input other detected specified states of the driver to the fatigue state detection module 350. Accordingly, the fatigue state detection module 350 is configured to comprehensively determine the fatigue state of the driver based on the input states (e.g., eye state, head posture, and other specified states).

[0056] The following uses the architecture of the DMS (hereinafter referred to as the system for short) provided in FIG. 2 or FIG. 3 as an example to describe the procedure of the method for detecting the driver's state provided in the embodiment of the present application.

[0057] 4 is a schematic flow chart of a method for detecting a driver's state according to an exemplary embodiment of the present application. As shown in FIG. 4, the procedure of the method for detecting a driver's state includes the following steps:

[0058] Step 401: Obtain a first image frame and a second image frame, where the first image frame and the second image frame are both image frames including a driver's face, and the first image frame is an image frame captured before the second image frame.

[0059] Optionally, the system captures a sequence of image frames by the camera, the sequence of image frames including at least two image frames, namely at least one first image frame and one second image frame.

[0060] The first image frame and the second image frame are both image frames including the driver's face, ie, the first image frame and the second image frame both include the driver's full facial features.

[0061] The first image frame is an image frame captured prior to the second image frame. Optionally, the first image frame is at least two image frames captured prior to the second image frame.

[0062] Optionally, the at least one first image frame and the second image frame are a plurality of consecutive image frames.

[0063] Step 402: Obtain first detection information of a first image frame and second detection information of a second image frame, where the first detection information and the second detection information both indicate a driver's eye state and head posture.

[0064] Optionally, the system performs eye state detection and head pose detection on the first image frame to obtain first detection information. The first detection information is indicative of an eye state and a head pose of the driver corresponding to the first image frame. For example, the first detection information includes first eye detection information and first head detection information. The first eye detection information is indicative of an eye state of the driver corresponding to the first image frame. The first head detection information is indicative of a head pose of the driver corresponding to the first image frame. The eye state includes an eye closed state or an eye open state.

[0065] Optionally, the system performs eye state detection and head pose detection on the second image frame to obtain second detection information. The second detection information is indicative of an eye state and a head pose of the driver corresponding to the second image frame. For example, the second detection information includes second eye detection information and second head detection information. The second eye detection information is indicative of an eye state of the driver corresponding to the second image frame. The second head detection information is indicative of a head pose of the driver corresponding to the second image frame.

[0066] It should be noted that the eye state detection and head pose detection can be performed in parallel or sequentially, which is not limited in this embodiment of the present application. For details about the eye state detection and head pose detection, please refer to the related descriptions in the following embodiments, and the details will not be described here.

[0067] Step 403: When the second detection information indicates that the first eye state corresponding to the second image frame is a closed eye state, determine a second eye state corresponding to the second image frame based on the first detection information and the second detection information.

[0068] Optionally, the system determines whether the second detection information indicates that the first eye state corresponding to the second image frame is a closed-eye state. If the first eye state corresponding to the second image frame is a closed-eye state, the system determines a second eye state corresponding to the second image frame based on the first detection information and the second detection information. If the first eye state corresponding to the second image frame is an open-eye state, the process ends.

[0069] Optionally, when the first detection information and the second detection information satisfy a first preset condition, the system determines that the second eye state corresponding to the second image frame is an open-eye state, where the first preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an open-eye state.

[0070] For example, the first image frame includes a plurality of first image frames, and the meaning of "the first detection information indicates that the eye state corresponding to the first image frames is an open eye state" includes the first detection information indicating that a ratio of the number of first image frames in which the eye state is an open eye state to the total number of first image frames is greater than a first preset threshold.

[0071] The first preset threshold value is set by default or in a customized manner, for example, the first preset threshold value is 0.9, which is not limited in this embodiment of the present application.

[0072] Optionally, when the first detection information and the second detection information satisfy a second preset condition, the system determines that the second eye state corresponding to the second image frame is an eye-closed state, where the second preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an eye-closed state.

[0073] For example, the first image frame includes a plurality of first image frames, and the meaning of "the first detection information indicates that the eye state corresponding to the first image frames is a closed-eye state" includes the first detection information indicating that a ratio of the number of first image frames in which the eye state is a closed-eye state to the total number of first image frames is greater than a second preset threshold.

[0074] The second preset threshold value may be set by default or may be set in a customized manner, for example, the second preset threshold value is 0.95, which is not limited in this embodiment of the present application.

[0075] Optionally, if the second detection information indicates that the head pose corresponding to the second image frame does not jump in the pitch angle direction, the system determines that the second eye state corresponding to the second image frame is a closed eye state.

[0076] Optionally, the system determines a fatigue state detection result based on the head pose and the second eye state corresponding to the second image frame, and outputs alarm information when the fatigue state detection result meets a preset alarm condition.

[0077] Optionally, the system obtains a preset fatigue detection model, and outputs to call the fatigue detection model based on the head posture and the second eye state corresponding to the second image frame, and obtain a fatigue state detection result. The fatigue detection model indicates the correlation between the head posture and the fatigue state, and the correlation between the eye state and the fatigue state, and the fatigue detection model is a model pre-trained based on the sample image frame. For example, the fatigue detection model is a model obtained by fusion based on the eye state and the head posture. Alternatively, the fatigue detection model is a model obtained by fusion based on the eye state, the head posture, and the yawning state. Alternatively, the fatigue detection model is a model obtained by fusion based on the eye state, the head posture, the yawning state, and other information. This is not limited in this embodiment of the present application.

[0078] In a possible implementation, the fatigue state detection result includes one of a first detection result and a second detection result, the first detection result indicating that the driver is in a fatigued state and the second detection result indicating that the driver is not fatigued.

[0079] In this implementation, outputting the alarm information when the fatigue state detection result satisfies the preset alarm condition includes outputting the alarm information when the fatigue state detection result is the first detection result.

[0080] In another possible implementation, the fatigue state detection result includes a fatigue state level, and the fatigue state level is related to the predicted fatigue intensity of the driver. For example, there is a positive correlation between the fatigue state level and the predicted fatigue intensity of the driver, that is, the higher the fatigue state level, the greater the predicted fatigue intensity of the driver.

[0081] In this implementation, outputting alarm information when the fatigue state detection result satisfies a preset alarm condition includes outputting alarm information when the fatigue state level is greater than a preset level threshold.

[0082] The preset level thresholds may be set by default or in a customized manner, which is not limited in this embodiment of the present application.

[0083] Optionally, the system outputs the alarm information according to a preset prompt format, where the preset prompt format includes at least one of a voice format, a text format, an image format, and an animation format. The output manner and output contents of the alarm information are not limited in this embodiment of the present application.

[0084] In summary, in this embodiment of the present application, the first detection information of the first image frame and the second detection information of the second image frame (e.g., the driver's continuous eye state and corresponding continuous head posture) are obtained. For different scenarios, such as a head-down looking down scenario and a head-down dozing scenario, the second eye state corresponding to the second image frame can be determined based on the first detection information and the second detection information when the first eye state corresponding to the second image frame is determined to be a closed eye state. That is, the eye state corresponding to the second image frame is corrected to obtain an accurate eye state. This solves the problem in the related art that the system cannot accurately determine the open / closed eye state by directly using the image when the driver lowers his head, and ensures an open / closed eye discrimination effect with high accuracy and high robustness, which is helpful to obtain the user's stable and accurate fatigue state thereafter.

[0085] 5 is a schematic flow chart of a method for detecting a driver's state according to another exemplary embodiment of the present application. As shown in FIG. 5, the procedure of the method for detecting a driver's state includes the following steps:

[0086] Step 501: Perform image capture by a camera to obtain a sequence of image frames.

[0087] Optionally, the system performs image capture by a camera located at at least one location within the vehicle to obtain a sequence of image frames. The camera may be an infrared camera.

[0088] Optionally, the at least one position includes one or more of the following positions: a position on or near the steering column and dashboard, a position on or near the center console, a position on or near pillar A, and a position on or near the rearview mirror. If the camera is placed on the steering column in a position that is optimal for identifying the eye state, it does not easily occur that the eye state cannot be accurately identified when the head is down. However, in this position, the steering wheel is likely to block the face, and as a result, the eye state may not be identified. If the camera is placed at the pillar A or rearview mirror, the steering wheel does not block the face during image capture, but a scenario does occur in which the eye state cannot be accurately identified when the head is down.

[0089] Optionally, performing image capture by the camera to obtain an image frame sequence includes performing image capture by the camera to obtain an image frame sequence when the vehicle is in a driving state, and / or performing image capture by the camera to obtain an image frame sequence when the vehicle's running speed exceeds a preset vehicle speed, and / or performing image capture by the camera to obtain an image frame sequence after it is detected that the vehicle is ignited, and / or performing image capture by the camera to obtain an image frame sequence when a vehicle start command is detected, and / or performing image capture by the camera to obtain an image frame sequence when a control command for the vehicle or a component or system in the vehicle is detected. It should be noted that the trigger condition and capture manner of image capture are not limited in this embodiment of the present application.

[0090] Step 502: Perform face detection on the captured image frame sequence.

[0091] Face detection is the basis of other face applications. The face-based eye state detection algorithm and head pose algorithm are influenced by the frontal face detection algorithm. If the face detection algorithm has jitter in its positioning or is inaccurate in its positioning, which results in false positives and false negatives, the eye state will be misidentified or there will be large jitter in the head pose. As a result, the system will erroneously determine a normal state as a head down event. Only when the presence or absence of a face and accurate position information of the face are obtained can the system obtain stable and accurate eye state results.

[0092] Optionally, the system performs face detection on the captured image frame sequence by using a pre-set face detection algorithm.

[0093] In a possible implementation, the preset face detection algorithm is a dedicated face detection algorithm, for example a multi-task face detection algorithm. The multi-task face detection algorithm is used to detect a face and output face keypoint information, so that the subsequent system can correct the face by using the face keypoint information, provide a better input to the back-end eye condition detection algorithm, and improve the accuracy of the single-frame eye condition identification algorithm. The dedicated face detection algorithm has high accuracy. However, it takes time to deploy the algorithm inside the vehicle, so the algorithm is not suitable for being deployed inside the vehicle.

[0094] In another possible implementation, the preset face detection algorithm is a lightweight face detection algorithm, which is suitable for being deployed inside the vehicle.

[0095] In another possible implementation, the preset face detection algorithm is a general target detection algorithm, for example, a single-phase target detection algorithm, which is fast and highly accurate inside the vehicle and is suitable for being deployed in the in-vehicle infotainment system as a face detection algorithm. It should be noted that the preset face detection algorithm is not limited in this embodiment of the present application.

[0096] Step 503: Determine whether a face is detected in the sequence of image frames.

[0097] If the system does not detect a face in the sequence of image frames, execution continues with step 501. If the system detects a face in the sequence of image frames, step 504 is executed.

[0098] Step 504: If a face is detected in the sequence of image frames, perform eye state detection to obtain a sequence of eye states, and perform face pose detection to obtain a sequence of face poses.

[0099] Optionally, for each image frame in the sequence of image frames, the system performs eye state detection and head pose detection to obtain eye state detection data and face pose detection data corresponding to the image frame to obtain a sequence of eye states and a sequence of head poses corresponding to the sequence of image frames, i.e. the sequence of eye states includes eye state detection data corresponding to each of a plurality of image frames in the sequence of image frames, and the sequence of head poses includes head pose detection data corresponding to each of a plurality of image frames in the sequence of image frames.

[0100] In the case of single-frame eye state detection, in a possible implementation, the system may perform eye state detection by using a target detection algorithm. The eye state identification performed based on the target detection algorithm has high robustness and is unlikely to be hindered from eye state identification, so that scenarios of wearing a mask, scenarios of the face being covered by hands, and scenarios of makeup are easily supported. The eye state identification is only affected in scenarios where the eye part is covered or interfered with. However, there are cases where the target detection algorithm cannot distinguish between the scenario of slanted eyes and the scenario of squinting due to fatigue, resulting in the two states being mistaken for each other. The eye state identification algorithm can be applied to scenarios where the amount of face data is small and cannot cover all the face samples of the driving environment in all scenarios.

[0101] In another possible implementation, since face size and eye size change slightly, the system can perform eye state detection by using a general target detection algorithm after pruning, and predict eye state by using only some network branches, which can greatly improve detection speed while ensuring accuracy, and has greater advantages inside the vehicle.

[0102] In another possible implementation, the system may perform eye state detection by using a preset classification algorithm. The eye state detection performed based on the preset classification algorithm may be classified into two cases. In one case, the eye state is determined based on an image frame including a face. Since the ratio of the area covered by the eyes to the area of ​​the face is very small, interference from other parts of the face can easily cause misidentification, for example, wearing a mask or makeup can both cause misidentification. Furthermore, this method cannot distinguish invalid states caused by whether the eyes are occluded or not. In another case, an eye detection model is added, and then an image frame including the eyes is input to the eye detection model and output to obtain the eye state. The eye detection model is a pre-trained neural network model that is used to identify the eye state in the image frame. This method is not obstructed by other parts of the eyes, has higher robustness, and has higher accuracy compared to the above case. This method cannot distinguish between scenarios where the eyes are slanted and scenarios where the eyes are squinted due to fatigue. Furthermore, this method increases resource consumption and reduces performance inside the vehicle because the eye detection model is added.

[0103] In another possible implementation, the system may perform eye condition detection by using a keypoint algorithm. Optionally, as shown in FIG. 6, the system may detect six keypoint locations of the upper eyelid, lower eyelid, and corner of the eye, “p 43 , p 44 , p 45 , p 46 , p 47 , and p 48 " and then calculate the normalized distance EAR by the following formula:

number

[0104] The normalized distance is used as an eye-open indicator. If the eye-open indicator is smaller than the preset open threshold, the eye is determined to be in the closed eye state, and if the eye-open indicator is equal to or greater than the preset open threshold, the eye is determined to be in the open eye state. The preset open threshold is set by default or in a customized manner. For example, the preset open threshold is 20% of the normal open eye value. This is not limited in this embodiment of the present application. Based on the keypoint algorithm and head pose calibration, the eye-opening scenario and the fatigued eye-squinting scenario can be distinguished, so that the accuracy of identifying the eye state in the two scenarios is slightly improved. The eye-open / closed state based on the eye-open indicator is very suitable to be used as the basis for the subsequent determination of the driver's fatigue state. However, this method has high requirements for the stability of the keypoint algorithm, requiring that no large jitter occurs at the keypoint position, that is, the corresponding eye-opening needs to be stable. In order to avoid large jitter in eye opening, a filtering method may be added, that is, the system performs a filtering process on the outputted eye opening indicator to obtain a more appropriate eye opening indicator.

[0105] For single-frame head pose detection, the system maps the two-dimensional image frame to a three-dimensional image frame to obtain head pose detection data. The head pose detection data indicates the orientation of the driver's face pose. Optionally, the head pose detection data is expressed by using three degrees of freedom (dof): pitch angle, roll angle, and yaw angle. As shown in FIG. 7, the head pose detection data includes the pitch angle between the driver's head and the horizontal axis of the device coordinate system, the yaw angle between the head and the longitudinal axis of the device coordinate system, and the roll angle between the head and the vertical axis of the device coordinate system.

[0106] The accuracy and stability of head pose detection are crucial for the whole eye condition identification system. Furthermore, due to the camera mounting position and head-down reasons, the head pose detection algorithm needs to support large angle head poses.

[0107] In a possible implementation, the system performs head pose detection based on face keypoints. This method can perform training based on a large amount of existing keypoint data, and can obtain keypoint information for each part of the face. The head pose is more stable because the disturbance of the points has little effect on the angle.

[0108] In another possible implementation, the system performs head pose detection based on a regression algorithm. The method can directly regress Euler angles, which is simple and convenient. The model design is simple, and there is no need to perform post-processing or optimization of secondary algorithms for the solution. The difficulty of the regression-based head pose detection algorithm lies in the collection of head pose data. The head pose data collection methods include data generation method and direct collection method. The direct collection method includes that the head pose tag in the real camera coordinate system is directly collected by the optical tracker and the camera array, and 360° head pose data can be covered. However, the device is expensive, the arrangement is complicated, the collection periodicity is long, and the human investment is large. The data generation method includes performing 3D morphable model (3DMM) fitting to the keypoint data in the 2D image frame, and then performing head pose enhancement to obtain head pose taps at large angles. The method can support head pose data at ±90°. This method only relies on 2D keypoint marking, and both feasibility and cost are reasonable.

[0109] In another possible implementation, the system performs head pose detection based on face keypoints and regression algorithm. Optionally, the system inputs the image frame into the head pose detection model and outputs to obtain head pose data. The head pose detection model is a multi-task neural network model based on face keypoints and regression method, that is, tasks such as Euler angle regression and face keypoints are performed in one network model. Through a shared skeleton network, multiple head networks can be used to perform different tasks, and the prediction error of Euler angles can be reduced to some extent by using this method.

[0110] It should be noted that the methods of the eye state detection algorithm and the head pose detection algorithm are not limited in this embodiment of the present application.

[0111] Step 505: Determine whether the first eye-state corresponding to the second image frame is an eye-closed state.

[0112] Eye state detection and head pose detection are performed for each frame in the image frame sequence, and the system obtains eye state cues and head pose cues corresponding to the image frame sequence, and the system determines whether the first eye state in the latest frame in the image frame sequence, i.e., the second image frame, is a closed eye state.

[0113] If the first eye state corresponding to the second image frame is an eyes open state, the process ends. If the first eye state corresponding to the second image frame is an eyes closed state, the system continues to execute step 506. The following three scenarios are distinguished by checking the eye state cues and head pose cues corresponding to the image frame sequence: normal eyes closed scenario, head down, eyes open, looking down scenario, and head down and dozing scenario.

[0114] Step 506: If the first eye state corresponding to the second image frame is an eye-closed state, determine whether the head pose corresponding to the second image frame jumps in the pitch angle direction.

[0115] If the first eye state corresponding to the second image frame is a closed eye state, the system determines whether the head pose corresponding to the second image frame jumps in the pitch angle direction. If the head pose corresponding to the second image frame does not jump in the pitch angle direction, step 507 is performed. If the head pose corresponding to the second image frame jumps in the pitch angle direction, step 508 is performed.

[0116] Step 507: If the head pose corresponding to the second image frame does not jump in the pitch angle direction, output that the second eye state corresponding to the second image frame is an eye-closed state.

[0117] If the head pose corresponding to the second image frame does not jump in the pitch angle direction, the scenario is determined to be a normal eyes closed scenario, and a second eye state corresponding to the second image frame is output as an eyes closed state.

[0118] In an illustrated example, the head pose does not jump in the pitch angle direction when the eyes change from an open-eyes state to a closed-eyes state, as shown in Fig. 8. The system determines that the scenario is a normal eyes-closed scenario, and the eye state is maintained in the original closed-eyes state to obtain a corrected eye state.

[0119] Step 508: If the head pose corresponding to the second image frame jumps in the pitch angle direction, determine whether the eye state corresponding to the first image frame is a closed eye state.

[0120] If the head pose corresponding to the second image frame jumps in the pitch angle direction, the system determines whether the eye state corresponding to the first image frame is a closed-eye state. If the eye state corresponding to the first image frame is an open-eye state, step 509 is executed. If the eye state corresponding to the first image frame is a closed-eye state, step 510 is executed.

[0121] Step 509: If the eye state corresponding to the first image frame is an open-eye state, output that the second eye state corresponding to the second image frame is an open-eye state.

[0122] If the eye state corresponding to the first image frame is an open-eyes state, the scenario is determined to be a head-down, eyes-open, looking-down scenario, and a second eye state corresponding to the second image frame is output as an open-eyes state.

[0123] In an illustrated example, the head pose jumps in the pitch angle direction when the eyes change from an open (open) state to a closed (occluded) state, as shown in Fig. 9. The system determines that the scenario is head down, eyes open, looking down scenario, and the eye state is corrected to the open (open) state to obtain a corrected eye state.

[0124] Step 510: If the eye state corresponding to the second image frame is an eye-closed state, output that the second eye state corresponding to the second image frame is an eye-closed state.

[0125] If the eye state corresponding to the first image frame is a closed-eye state, it is determined that the scenario is a head-down dozing scenario, and a second eye state corresponding to the second image frame is a closed-eye state is output.

[0126] In an illustrated example, the head pose jumps in the pitch angle direction when the eyes change from open (open) to closed (occluded) as shown in Fig. 10. The system determines that the scenario is a head-down dozing scenario, and the eye state is kept as the original closed to obtain a corrected eye state.

[0127] Optionally, if the first eye state corresponding to the second image frame is a closed-eye state, the system may distinguish three scenarios based on the eye state sequence and the head pose sequence, namely, eyes closed normally, head down looking down, and head down dozing. If only the eye state sequence is used as the input parameter of the fatigue state, false alarms may occur in these three states, which will affect the effectiveness of the alarm system. A head pose detection module is added, so that the two scenarios, eyes closed normally and head down eyes open, can be easily distinguished. If it is checked whether the head pose jumps when the closed-eye state is detected, if the head pose does not jump, the scenario is determined to be eyes closed normally, the result is not corrected, and the original eye state is maintained; if the head pose jumps, the scenario is determined to be head down looking down, and the original closed-eye state is corrected to eyes open. However, in this case, the head down dozing event is erroneously determined to be head down eyes open. As a result, the eye state is erroneously corrected. To eliminate the case where the head-down dozing event is erroneously corrected to the eyes-open state, an eye stable state cue is added. The two scenarios, i.e., head-down, eyes-open, looking down and head-down dozing event scenarios, are distinguished by checking the eye state of the past consecutive frames, i.e., the eye state corresponding to the first image frame. If the eye state corresponding to the first image frame is the eyes-closed state, the scenario is determined to be the head-down dozing scenario, the result is not corrected and the original eyes-closed state is maintained. If the eye state corresponding to the first image frame is the eyes-open state, the scenario is determined to be the head-down, eyes-open, looking down scenario and the original eyes-closed state is corrected to the eyes-open state.

[0128] In an illustrated example, the process of distinguishing the three scenarios and the correction process are shown in FIG. 11. In FIG. 11, the first row represents the head pose sequence of the head pose in the pitch angle direction, and the second row represents the corresponding eye state sequence without post-processing. The third row represents the corrected eye state sequence obtained after the head pose detection module is added. As is evident from FIG. 11, the eye state in the head-down dozing scenario is erroneously corrected to the eyes-open state. The fourth row represents the accurate eye state obtained after the eye stable state cue is further added to further correct the eye state.

[0129] In summary, the method for detecting driver's state provided in this embodiment of the present application can accurately distinguish three scenarios that have little difference when analyzed on images, namely, normal eyes closed scenario, looking down scenario, and dozing off scenario. Moreover, the false recognition of eyes closed caused by looking down is accurately restored to eyes open state by using stable state cues, and the eyes closed state in dozing off scenario is not erroneously corrected. Moreover, distinguishing different scenarios based on head pose sequence and eye state sequence is highly fault-tolerant, which greatly improves the accuracy of identifying eye state in driving scenario.

[0130] Apparatus embodiments of the present application are provided below and may be configured to perform the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0131] 12 is a block diagram of an apparatus for detecting a driver's state according to an exemplary embodiment of the present application. The apparatus can be implemented as a whole or part of the DMS provided in FIG. 2 or the apparatus for detecting a driver's status provided in FIG. 3 by using software, hardware, or a combination of software and hardware. The apparatus can include a first obtaining unit 1210, a second obtaining unit 1220, and a determining unit 1230.

[0132] The first acquisition unit 1210 is configured to acquire a first image frame and a second image frame, both of which are frame images including the driver's face, and the first image frame is an image frame captured before the second image frame.

[0133] The second acquisition unit 1220 is configured to acquire first detection information of a first image frame and second detection information of a second image frame, where both the first detection information and the second detection information indicate the driver's eye state and head posture.

[0134] The determination unit 1230 is configured to determine a second eye state corresponding to the second image frame based on the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is a closed-eye state.

[0135] In a possible implementation, the decision unit 1230 is The device is further configured to determine that a second eye state corresponding to the second image frame is an open eye state when the first detection information and the second detection information satisfy a first preset condition.

[0136] The first preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an open eye state.

[0137] In another possible implementation, the decision unit 1230 is The device is further configured to determine that a second eye state corresponding to the second image frame is a closed eye state when the first detection information and the second detection information satisfy a second preset condition.

[0138] The second preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is a closed eye state.

[0139] In another possible implementation, the decision unit 1230 is The system is further configured to determine that a second eye state corresponding to the second image frame is a closed eye state when the second detection information indicates that the head pose corresponding to the second image frame does not jump in the pitch angle direction.

[0140] In another possible implementation, the device further comprises a detection module.

[0141] The detection module is configured to determine a fatigue state detection result based on the head pose and the second eye state corresponding to the second image frame.

[0142] In another possible implementation, the device further comprises an alarm module.

[0143] The alarm module is configured to output alarm information when the fatigue state detection result meets a preset alarm condition.

[0144] It should be noted that when the device provided in the above embodiment implements the functions of the device, the division of the above functional modules is merely used as an example for description. In practical application, the above functions may be assigned to different functional modules for required implementation, that is, the internal structure of the device is divided into different functional modules to implement all or part of the above functions. Furthermore, the device and method embodiments provided in the above embodiment belong to the same concept. For the specific implementation process of the device, please refer to the method embodiment. Details will not be described again here.

[0145] An embodiment of the present application provides an apparatus for detecting a driver's state. The apparatus for detecting a driver's state includes a processor and a memory configured to store processor-executable instructions. When the processor is configured to execute the instructions, the method performed by the DMS in the above embodiment is implemented.

[0146] An embodiment of the present application provides a computer program product including computer readable code or a non-volatile computer readable storage medium carrying the computer readable code, which when executed by a processor causes the processor to perform the method performed by the DMS in the above embodiments.

[0147] An embodiment of the present application provides a non-volatile computer-readable storage medium, which stores computer program instructions, which, when executed by a processor, implement the method performed by the DMS in the above embodiment.

[0148] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital video disc (DVD), memory stick, floppy disk, machine-encoded devices such as punch cards or slot-projection structures that store instructions, and any suitable combination thereof.

[0149] The computer readable program instructions or code described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium in each computing / processing device for storage.

[0150] The computer program instructions used to perform the operations herein may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or source or target code written in any combination of one programming language or more programming languages. Programming languages ​​include object-oriented programming languages ​​such as Smalltalk and C++, as well as traditional procedural programming languages ​​such as "C" or similar programming languages. The computer readable program instructions may be executed entirely on the user computer, partially on the user computer, as a separate software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. When a remote computer is included, the remote computer may be connected to the user computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., over the Internet by using an Internet Service Provider). In some embodiments, electronic circuitry, such as a programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA), is customized by using status information in computer readable program instructions. The electronic circuitry may execute the computer readable program instructions to implement various aspects of the present application.

[0151] Various aspects of the present application are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented by computer readable program instructions.

[0152] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to generate a machine that, when executed by a processor of the computer or other programmable data processing apparatus, produces an apparatus that implements the functions / operations specified in one or more blocks in the flowcharts and / or block diagrams. These computer-readable program instructions may alternatively be stored on a computer-readable storage medium. These instructions enable a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, a computer-readable medium storing instructions includes an artifact that includes instructions for implementing various aspects of the functions / operations specified in one or more blocks in the flowcharts and / or block diagrams.

[0153] The computer-readable program instructions may alternatively be loaded into a computer, other programmable data processing apparatus, or other device, such that the example operational steps are executed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process. Thus, the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more blocks in the flowcharts and / or block diagrams.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the system architecture, functionality, and operation of possible implementations of the apparatus, system, method, and computer program product according to several embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of an instruction, which includes one or more executable instructions for implementing a particular logical function. In some alternative implementations, the functions marked in the blocks may also be performed in a different order than marked in the accompanying drawings. For example, in practice, two consecutive blocks may be executed substantially in parallel, or may be executed in a reverse order depending on the functions involved.

[0155] It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by hardware (e.g., a circuit or ASIC (application-specific integrated circuit)) that performs the corresponding function or operation, or may be implemented by a combination of hardware and software, e.g., firmware.

[0156] Although the present application is described herein with reference to embodiments, in the course of implementing the present application for which protection is claimed, those skilled in the art may understand and implement other variations of the disclosed embodiments by studying the accompanying drawings, the disclosed content, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "one" does not exclude "multiple". A single processor or other unit may implement several functions recited in the claims. Although several measures are recited in mutually different dependent claims, this does not mean that these measures cannot be combined to produce advantageous effects.

[0157] The above has described the embodiments of the present application. The above description is not exhaustive and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical applications, or improvements of the technology in the marketplace of the embodiments, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for detecting a driver's state, comprising: acquiring a first image frame and a second image frame, the first image frame and the second image frame both being frame images including a driver's face, the first image frame being an image frame captured prior to the second image frame; obtaining first detection information of the first image frame and second detection information of the second image frame, the first detection information and the second detection information indicating an eye state and a head posture of the driver; determining a second eye state corresponding to the second image frame based on the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is an eye-closed state; The method according to claim 1,

2. Determining a second eye state corresponding to the second image frame based on the first detection information and the second detection information includes: determining that the second eye state corresponding to the second image frame is an open eye state when the first detection information and the second detection information satisfy a first preset condition; the first preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in a pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is the open eye state; The method of claim 1.

3. Determining a second eye state corresponding to the second image frame based on the first detection information and the second detection information includes: determining that the second eye state corresponding to the second image frame is the eye-closed state when the first detection information and the second detection information satisfy a second preset condition; the second preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is the closed eye state; The method according to claim 1 or 2.

4. The method comprises: determining that the second eye state corresponding to the second image frame is the closed eye state when the second detection information indicates that the head pose corresponding to the second image frame does not jump in the pitch angle direction.

4. The method according to any one of claims 1 to 3.

5. The method comprises: determining a fatigue state detection result based on the head pose and the second eye state corresponding to the second image frame.

5. The method according to any one of claims 1 to 4.

6. The method comprises: and outputting alarm information when the fatigue state detection result satisfies a preset alarm condition. The method according to claim 5.

7. A device for detecting a driver's state, comprising: a first capture unit configured to capture a first image frame and a second image frame, the first image frame and the second image frame both being frame images including a driver's face, the first image frame being an image frame captured prior to the second image frame; a second acquisition unit configured to acquire first detection information of the first image frame and second detection information of the second image frame, the first detection information and the second detection information being indicative of an eye state and a head posture of the driver; a determining unit configured to determine a second eye state corresponding to the second image frame based on the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is an eye-closed state; An apparatus having the above configuration.

8. The determination unit is and determining that the second eye state corresponding to the second image frame is an open eye state when the first detection information and the second detection information satisfy a first preset condition; the first preset condition is that the second detection information indicates that the head pose corresponding to the second image frame jumps in a pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is the open eye state; 8. The apparatus of claim 7.

9. The determination unit is and determining that the second eye state corresponding to the second image frame is the eye-closed state when the first detection information and the second detection information satisfy a second preset condition. the second preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is the closed eye state; 9. Apparatus according to claim 7 or 8.

10. The decision unit is and determining that the second eye state corresponding to the second image frame is the eye-closed state when the second detection information indicates that the head pose corresponding to the second image frame does not jump in the pitch angle direction.

10. Apparatus according to any one of claims 7 to 9.

11. The apparatus further comprises a detection module. the detection module is configured to determine a fatigue state detection result based on the head pose and the second eye state corresponding to the second image frame; Apparatus according to any one of claims 7 to 10.

12. The apparatus further comprises an alarm module; The alarm module is configured to output alarm information when the fatigue state detection result meets a preset alarm condition.

12. The apparatus of claim 11.

13. A device for detecting a driver's state, comprising: A processor; a memory configured to store processor-executable instructions; The processor is configured to perform a method according to any one of claims 1 to 6 when executing the instructions. Device.

14. A non-volatile computer readable storage medium having computer program instructions stored thereon, comprising: The computer program instructions, when executed by a processor, perform the method according to any one of claims 1 to 6. A non-volatile computer-readable storage medium.

15. 1. A computer program product comprising: The computer program product, when executed on a computer, causes the computer to carry out the method according to any one of claims 1 to 6. Computer program products.

16. A vehicle comprising a device according to any one of claims 7 to 12.

Images