Driver behavior monitoring method and device and storage medium

Abstract

The invention discloses a driver behavior monitoring method and device and a storage medium, and belongs to the technical field of vehicle control. The method comprises the steps of obtaining a driver detection result in a video image of a vehicle driving position in response to a vehicle in a starting state, a forward gear and a steering angle of a steering wheel greater than a first angle; responding to the obtained detection result indicating that the driver exists in the video image, and obtaining a fatigue driving detection result, a distraction detection result, a dangerous driving detection result and an abnormal emotion detection result according to the expression and behavior of the driver; and in response to at least one of a detection result indicating that the driver is in a fatigue driving state, a detection result indicating that the driver is in a distraction state, a detection result indicating that the driver is in a dangerous driving state or a detection result indicating that the driver is abnormal in emotion, sending prompt information based on the priority level. The driver is prompted according to the order of the danger degrees, and the driving safety is guaranteed.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of vehicle control technology, and in particular to a method, device and storage medium for monitoring driver behavior. Background Art

[0002] During driving, drivers are prone to fatigue driving, distraction, dangerous driving and abnormal emotions, which endanger driving safety. How to monitor the driver's behavior during driving to avoid fatigue driving, distraction, dangerous driving and driving with abnormal emotions is very important to ensure driving safety and improve the driver's driving experience. Summary of the invention

[0003] The embodiment of the present application provides a method, device and storage medium for driver behavior monitoring, which can be used to ensure driving safety and improve driving experience. The technical solution is as follows:

[0004] On the one hand, an embodiment of the present application provides a method for driver behavior monitoring, the method comprising:

[0005] In response to the vehicle being in a start state, the gear position being a forward gear, and the steering angle of the steering wheel being greater than a first angle, acquiring a video image of a driving position of the vehicle;

[0006] Acquire a driver detection result in a video image, wherein the driver detection result in the video image is used to indicate whether there is a driver in the video image;

[0007] In response to obtaining a detection result indicating that the driver exists in the video image, obtaining a fatigue driving detection result, a distraction detection result, a dangerous driving detection result, and an abnormal emotion detection result according to the driver's expression and behavior, wherein the fatigue driving detection result is used to indicate whether the driver is in a fatigue driving state, the distraction detection result is used to indicate whether the driver is in a distracted state, the dangerous driving detection result is used to indicate whether the driver is in a dangerous driving state, and the abnormal emotion detection result is used to indicate whether the driver's emotions are abnormal;

[0008] In response to obtaining at least one of a detection result indicating that the driver is in the fatigue driving state, a detection result indicating that the driver is in the distracted state, a detection result indicating that the driver is in the dangerous driving state, or a detection result indicating that the driver is emotionally abnormal, a prompt message is issued based on a priority level, and the priority level decreases in the order of dangerous driving prompt, distracted state prompt, fatigue driving prompt, and abnormal emotion prompt.

[0009] In another aspect, a device for driver behavior monitoring is provided, the device comprising:

[0010] A first acquisition module, configured to acquire a video image of a driving position of the vehicle in response to the vehicle being in a start state, the gear position being a forward gear, and the steering angle of the steering wheel being greater than a first angle;

[0011] A second acquisition module is used to acquire a driver detection result in the video image, wherein the driver detection result in the video image is used to indicate whether there is a driver in the video image;

[0012] a third acquisition module, configured to, in response to acquiring a detection result indicating that the driver exists in the video image, acquire a fatigue driving detection result, a distraction detection result, a dangerous driving detection result, and an abnormal emotion detection result according to the driver's expression and behavior, wherein the fatigue driving detection result is used to indicate whether the driver is in a fatigue driving state, the distraction detection result is used to indicate whether the driver is in a distracted state, the dangerous driving detection result is used to indicate whether the driver is in a dangerous driving state, and the abnormal emotion detection result is used to indicate whether the driver's emotions are abnormal;

[0013] A prompt module is used to respond to at least one of the detection results indicating that the driver is in the fatigue driving state, the detection results indicating that the driver is in the distracted state, the detection results indicating that the driver is in the dangerous driving state, or the detection results indicating that the driver is emotionally abnormal, and issue a prompt message based on a priority level, wherein the priority level decreases in the order of dangerous driving prompt, distracted state prompt, fatigue driving prompt and abnormal emotion prompt.

[0014] On the other hand, a non-temporary computer-readable storage medium is also provided, characterized in that a computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement any of the above-mentioned driver behavior monitoring methods.

[0015] On the other hand, a computer program product is also provided, the computer program product comprising computer instructions, which implement the steps of any of the above-mentioned methods for driver behavior monitoring when executed by a processor.

[0016] The technical solution provided by this application brings at least the following beneficial effects:

[0017] The present application obtains a driver detection result in a video image of a vehicle driving seat when the vehicle is in a starting state, the gear position is a forward gear, and the steering angle of the steering wheel is greater than a first angle; if there is a driver in the video image, a fatigue driving detection result, a distraction detection result, a dangerous driving detection result, and an abnormal emotion detection result are obtained according to the driver's expression and behavior to determine whether the driver is in a fatigue driving state, a distracted state, a dangerous driving state, or has abnormal emotions; in response to obtaining at least one of a detection result indicating that the driver is in a fatigue driving state, a detection result indicating that the driver is in a distracted state, a detection result indicating that the driver is in a dangerous driving state, or a detection result indicating that the driver has abnormal emotions, a prompt message is issued based on a priority level to prompt the driver in order of the degree of danger, thereby avoiding problems such as fatigue driving, distraction, dangerous driving, and driving with abnormal emotions, and ensuring driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0020] Figure 2 is a flow chart of a method for driver behavior monitoring provided by an embodiment of the present application;

[0021] Figure 3 It is a structural schematic diagram of a driver behavior monitoring device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0023] This application embodiment provides a method for monitoring driver behavior. Figure 1, which shows a schematic diagram of the implementation environment of the method provided in the embodiment of the present application. The implementation environment may include: DMS (Driver Monitoring System) 11, ECU (Electronic Control Unit) 12, VCU (Vehicle Control Unit) 13, EPS (Electric Power Steering) 14, vehicle-mounted camera 15, center console large screen 16, air conditioning control module 17, window control module 18 and audio 19.

[0024] Optionally, ECU 12 is used to obtain the start / stop state of the vehicle and send it to DMS 11, VCU 13 is used to obtain the gear position of the vehicle and send it to DMS 11, and EPS 14 is used to obtain the steering angle of the steering wheel and send it to DMS 11. The vehicle camera 15 is used to capture a video image of the driving position of the vehicle.

[0025] In a possible implementation, the vehicle's central console screen 16 includes a driver behavior monitoring switch and switches corresponding to fatigue driving detection, distraction detection, dangerous driving detection, and abnormal emotion detection, which are used to control the driver behavior monitoring function, as well as the opening and closing of the fatigue driving detection sub-function, distraction detection sub-function, dangerous driving detection sub-function, and abnormal emotion detection sub-function. It is also used to send a prompt text message based on the detection results obtained by the DMS 11 indicating that the driver is in a fatigue driving state, the detection results indicating that the driver is in a distracted state, the detection results indicating that the driver is in a dangerous driving state, or the detection results indicating that the driver is in an abnormal mood.

[0026] Exemplarily, the audio system 19 is used to issue a prompt voice message based on the detection result obtained by the DMS 11 indicating that the driver is in a fatigued driving state, the detection result indicating that the driver is in a distracted state, the detection result indicating that the driver is in a dangerous driving state, or the detection result indicating that the driver is in an abnormal mood.

[0027] Optionally, the air conditioning control module 17 is used to control the vehicle's air conditioning to start external circulation and negative ion purification functions when the mode of only caring for dangerous driving without prompting is turned on, and the driver is identified as dangerous driving because of smoking. The window control module 18 is used to control the vehicle's windows to open appropriately when the mode of only caring for dangerous driving without prompting is turned on, and the driver is identified as dangerous driving because of smoking. Among them, DMS11, ECU12, VCU13, EPS14, vehicle-mounted camera 15, center console large screen 16, air conditioning control module 17, window control module 18 and audio 19 establish communication connections through wired or wireless networks.

[0028] Based on the above Figure 1 In the implementation environment shown, the present application embodiment provides a method for monitoring driver behavior. Figure 2 As shown, taking the method applied to DMS as an example, the method includes steps 201 to 204.

[0029] In step 201 , in response to the vehicle being in a start state, the gear position being a forward gear, and the steering angle of the steering wheel being greater than a first angle, the DMS acquires a video image of the vehicle's driving position.

[0030] In one possible implementation, the DMS obtains the on / off status of the driver behavior monitoring switch, wherein the driver behavior monitoring switch is located on the large screen of the vehicle's center console. Optionally, when the driver behavior monitoring switch is turned on, the DMS obtains the vehicle's start / stop status through the ECU, wherein the vehicle's start / stop status includes a start state or a stop state. The DMS obtains the vehicle's gear position through the VCU, wherein the vehicle's gear position includes a forward gear, a parking gear, a reverse gear, a neutral gear, and a sport gear. And the DMS obtains the steering angle of the steering wheel through the EPS, and then compares the steering angle of the steering wheel with the first angle.

[0031] Exemplarily, if the vehicle is in a start state, the gear is in a forward gear and the steering angle of the steering wheel is greater than a first angle, the DMS obtains a video image of the vehicle's driving position, including: the DMS collects a video image of the vehicle's driving position through a vehicle-mounted camera. Optionally, the vehicle-mounted camera is installed in front of the vehicle's driving position to shoot the area where the vehicle's driving position is located. If the camera fails, the DMS prompts the camera abnormality through the vehicle's center console large screen.

[0032] In step 202, the DMS obtains a driver detection result in the video image, where the driver detection result in the video image is used to indicate whether a driver exists in the video image.

[0033] Optionally, after acquiring the video image of the vehicle driving position, the DMS acquires the driver detection result in the video image, wherein the driver detection result in the video image is used to indicate whether there is a driver in the video image. In a possible implementation, the DMS acquires the driver detection result in the video image, including: recognizing the video image through a driver recognition model; in response to recognizing that the expression and behavior of at least one character image meet the preset driver characteristics, acquiring a detection result indicating that there is a driver in the video image.

[0034] In a possible implementation, an initial driver recognition model may be pre-established, and the initial driver recognition model may be trained based on a training image of a driver containing annotated driver characteristics to obtain a driver recognition model. Driver characteristics include common expressions and behaviors of drivers. The DMS recognizes the video image through the driver recognition model, and if it recognizes that the expression and behavior of at least one character image meets the preset driver characteristics, a detection result indicating that a driver exists in the video image is obtained, and the expression and behavior of the character image are used as the expression and behavior of the current driver.

[0035] For example, if the expression and behavior of any person image is not recognized to meet the preset driver characteristics, a detection result indicating that there is no driver in the video image is obtained. If it is recognized that the camera is blocked, or the face of a person in the video image is blocked, the DMS prompts the vehicle's center console screen that the video image is blocked.

[0036] In step 203, in response to obtaining a detection result indicating the presence of a driver in the video image, the DMS obtains a fatigue driving detection result, a distraction detection result, a dangerous driving detection result, and an abnormal emotion detection result based on the driver's expression and behavior. The fatigue driving detection result is used to indicate whether the driver is in a fatigue driving state, the distraction detection result is used to indicate whether the driver is in a distracted state, the dangerous driving detection result is used to indicate whether the driver is in a dangerous driving state, and the abnormal emotion detection result is used to indicate whether the driver's emotions are abnormal.

[0037] In a possible implementation, after obtaining the detection result indicating the presence of a driver in the video image, the DMS obtains the on / off status of the switches corresponding to fatigue driving detection, distraction detection, dangerous driving detection, and abnormal emotion detection. The switches corresponding to fatigue driving detection, distraction detection, dangerous driving detection, and abnormal emotion detection are all located on the large screen of the vehicle's center console and are sub-switches of the driver behavior monitoring switch. The user can turn on the fatigue driving detection sub-function, distraction detection sub-function, dangerous driving detection sub-function, and abnormal emotion detection sub-function by clicking the switches corresponding to fatigue driving detection, distraction detection, dangerous driving detection, and abnormal emotion detection.

[0038] Exemplarily, after identifying a signal that at least one switch is turned on, the DMS obtains the vehicle's driving speed and compares the vehicle's driving speed with the speed range. If the vehicle's driving speed is within the speed range, the function corresponding to the switch is successfully activated; if the vehicle's driving speed is not within the speed range, the function corresponding to the switch fails to be activated. The driver is prompted through the vehicle's center console screen that the function has failed to be activated, and is asked to adjust the vehicle speed to within the speed range. Optionally, the speed range can be set based on experience.

[0039] Optionally, taking the example that the fatigue driving detection function, distraction detection function, dangerous driving detection function and abnormal emotion detection function are all turned on, the DMS obtains fatigue driving detection results, distraction detection results, dangerous driving detection results and abnormal emotion detection results according to the driver's expression and behavior, among which the fatigue driving detection result is used to indicate whether the driver is in a fatigue driving state, the distraction detection result is used to indicate whether the driver is in a distracted state, the dangerous driving detection result is used to indicate whether the driver is in a dangerous driving state, and the abnormal emotion detection result is used to indicate whether the driver's emotions are abnormal.

[0040] Next, examples are given to illustrate the method of obtaining fatigue driving detection results, distraction detection results, dangerous driving detection results and abnormal emotion detection results based on the driver's expressions and behaviors.

[0041] (1) Obtaining fatigue driving detection results

[0042] Exemplarily, the DMS obtains fatigue driving detection results based on the driver's expressions and behaviors, including: in response to identifying that the driver yawns more than a first threshold, closes his eyes more than a second threshold, or lowers his head more than a third threshold within a first duration, obtaining a detection result indicating that the driver is in a fatigue driving state and the fatigue level is mild fatigue; in response to identifying that the driver closes his eyes or lowers his head for more than a second duration, obtaining a detection result indicating that the driver is in a fatigue driving state and the fatigue level is severe fatigue.

[0043] Optionally, the DMS analyzes the driver's expressions and behaviors identified within the first time period, counts the number of times the driver yawns, the number and duration of closing eyes, and the number and duration of lowering the head, and then compares the number of yawns with the first threshold, the number of times the eyes are closed with the second threshold, the number of times the head is lowered with the third threshold, and the duration of closing eyes and lowering the head with the second time period.

[0044] Exemplarily, if the driver yawns more than the first threshold, closes his eyes more than the second threshold, or lowers his head more than the third threshold within the first duration, a detection result indicating that the driver is in a fatigued driving state is obtained, and the fatigue level is mild fatigue. If the driver closes his eyes or lowers his head for longer than the second duration, the DMS obtains a detection result indicating that the driver is in a fatigued driving state, and the fatigue level is severe fatigue. Optionally, the first duration can be set based on experience, and the first threshold, the second threshold, the third threshold, and the second duration can be set based on the situation in the driver's fatigue state.

[0045] (2) Obtaining distraction detection results

[0046] In one possible implementation, a distraction detection result is obtained based on the driver's expression and behavior, including: in response to at least one of the driver leaning sideways for more than a second angle, leaning over for more than a third angle, or not looking forward for more than a third period of time, and not meeting the expression and behavior states corresponding to checking the rearview mirror and blind spot, a detection result indicating that the driver is in a distracted state is obtained.

[0047] For example, the DMS analyzes the driver's facial expressions and behaviors identified in the first duration, and counts the driver's sideways angle, leaning angle, and the direction of his / her gaze and the duration of the direction. The sideways angle is then compared with the second angle, the leaning angle is compared with the third angle, and the duration of the eyes not looking forward is compared with the third duration.

[0048] Optionally, if the driver has at least one of leaning sideways beyond the second angle, leaning forward beyond the third angle, or not looking forward for more than the third duration, and does not meet the expression and behavior states corresponding to checking the rearview mirror and blind spot, the DMS obtains a detection result indicating that the driver is in a distracted state. Exemplarily, the second angle, the third angle, and the third duration can be set according to the situation of the driver's distracted state, and the specific expression and behavior of the driver when checking the rearview mirror and blind spot can also be determined according to experiments to avoid being identified as a distracted state due to checking the rearview mirror and blind spot.

[0049] (3) Obtaining dangerous driving detection results

[0050] Exemplarily, obtaining the dangerous driving detection result according to the driver's expression and behavior includes: in response to at least one of the driver drinking water, smoking, and making phone calls, obtaining the detection result indicating that the driver is in dangerous driving. Exemplarily, the DMS analyzes the driver's behavior identified within the first time period to determine whether the driver has the behavior of drinking water, smoking, and making phone calls. If the driver has at least one of the behavior of drinking water, smoking, and making phone calls, the DMS obtains the detection result indicating that the driver is in dangerous driving.

[0051] (4) Obtaining abnormal emotion detection results

[0052] In a possible implementation, obtaining abnormal emotion detection results based on the driver's expression and behavior includes: determining the driver's emotion type based on the driver's expression and behavior; in response to the emotion type being excitement, anger, impatience or sadness, and the duration of the emotion being more than a fourth duration, obtaining a detection result indicating that the driver's emotion is abnormal. Exemplarily, the DMS analyzes the driver's expression identified within the first duration, determines whether the driver is in an excited, angry, impatient or sad emotional state, and counts the duration of the driver being in each emotional state.

[0053] Optionally, the duration of each emotional state of the driver being in excitement, anger, impatience or sadness is compared with the fourth duration. If the emotional type is excitement, anger, impatience or sadness, and the duration exceeds the fourth duration, the DMS obtains a detection result indicating that the driver's emotion is abnormal.

[0054] In step 204, in response to obtaining at least one of a detection result indicating that the driver is in a fatigued driving state, a detection result indicating that the driver is in a distracted state, a detection result indicating that the driver is in a dangerous driving state, or a detection result indicating that the driver is emotionally abnormal, the DMS issues a prompt message based on a priority level, with the priority level decreasing in the order of dangerous driving prompt, distracted state prompt, fatigue driving prompt, and abnormal emotion prompt.

[0055] In one possible implementation, after determining the fatigue driving detection results, distraction detection results, dangerous driving detection results, and abnormal emotion detection results, if at least one of a detection result indicating that the driver is in a fatigue driving state, a detection result indicating that the driver is in a distracted state, a detection result indicating that the driver is in a dangerous driving state, or a detection result indicating that the driver is emotionally abnormal is obtained, the DMS issues a prompt message based on a priority level, wherein the priority level decreases in the order of dangerous driving prompt, distraction state prompt, fatigue driving prompt, and abnormal emotion prompt.

[0056] Exemplarily, issuing prompt information based on priority levels includes: in response to obtaining detection results indicating that the driver is in a fatigued driving state, a distracted state, a dangerous driving state, and a detection result indicating that the driver is emotionally abnormal, the DMS prioritizes issuing prompt information to avoid dangerous driving; in response to obtaining detection results indicating that the driver is in a fatigued driving state, a distracted state, and a detection result indicating that the driver is emotionally abnormal, the DMS prioritizes issuing prompt information to avoid distraction; in response to obtaining detection results indicating that the driver is in a fatigued driving state, and a detection result indicating that the driver is emotionally abnormal, the DMS prioritizes issuing prompt information to avoid fatigued driving; in response to obtaining detection results indicating that the driver is emotionally abnormal, the DMS prioritizes issuing prompt information for abnormal emotions.

[0057] Optionally, if a detection result indicating that the driver is in a fatigued driving state, a detection result indicating that the driver is in a distracted state, a detection result indicating that the driver is in a dangerous driving state, and a detection result indicating that the driver is in an abnormal mood is obtained, the DMS will give priority to issuing a prompt message to avoid dangerous driving, including: the DMS issues a voice message through the vehicle's speakers to prompt the driver to avoid dangerous driving. Among them, the voice prompt lasts for the fifth time period. After the voice prompt is issued, the driver's behavior and expression are continuously monitored. If the driver is still in a fatigued driving state after the sixth time period, the DMS again issues a voice message through the vehicle's speakers to prompt the driver to avoid dangerous driving, and no prompt is issued within the sixth time period. Optionally, the fifth and sixth time periods can be set based on experience and memorized after power-off. For example, the fifth time period is set to 5 seconds and the sixth time period is set to 30 seconds.

[0058] For example, the driver can also use the large screen on the center console to set a mode of only caring about dangerous driving without prompting. Optionally, if the driver is identified as dangerous driving because of smoking, the DMS controls the vehicle's air conditioning to turn on the external circulation and negative ion purification functions through the air conditioning control module, and controls the windows to open appropriately through the window control module.

[0059] In a possible implementation, if the driver is identified as dangerous driving because of making or receiving phone calls, the DMS controls the volume of the vehicle's audio system to be reduced to a preset volume and the air conditioning system to be reduced to a preset speed during the driver's phone call. After the driver completes the phone call, the volume of the audio system and the air conditioning system are restored. If the driver's mobile phone is connected to the vehicle's audio system via Bluetooth, the volume of the incoming call follows the control strategy of the vehicle's audio system.

[0060] Optionally, if detection results indicating that the driver is in a fatigued driving state, detection results indicating that the driver is in a distracted state, and detection results indicating that the driver is in an abnormal emotional state are obtained, the DMS will prioritize issuing a reminder message to avoid distraction, including: DMS issuing a voice message through the vehicle's speakers to remind the driver to "pay attention and avoid distractions"; and displaying a text message "pay attention and avoid distractions" on the large screen of the vehicle's center console.

[0061] Exemplarily, the voice prompt continues for the fifth duration. After the voice prompt is issued, the driver's behavior and expression are continuously monitored. If the driver is still distracted after the sixth duration, the DMS prompts the driver again through the vehicle's audio and large screen until the driver is no longer distracted. No prompt is given during the sixth duration.

[0062] In one possible implementation, if a detection result indicating that the driver is in a fatigue driving state and a detection result indicating that the driver is emotionally abnormal are obtained, the DMS prioritizes issuing a prompt message to avoid fatigue driving, including: in response to the fatigue level being mild fatigue, the DMS issues a voice message of "You are mildly fatigued" through the speakers on the vehicle, and displays a text message of "You are mildly fatigued" on the large screen of the vehicle's center console; in response to the fatigue level being severe fatigue, the DMS issues a voice message of "You are severely fatigued" through the speakers on the vehicle, and displays a text message of "You are severely fatigued" on the large screen of the vehicle's center console.

[0063] Exemplarily, the voice prompt continues for the fifth time period. After the voice prompt is issued, the driver's behavior and expression are continuously monitored. If the driver is still in a fatigued driving state after the sixth time period, the DMS will again prompt the driver through the audio and large screen on the vehicle until the driver is no longer in a fatigued state or the driver has stopped to rest. No prompt will be given during the sixth time period.

[0064] Optionally, if a detection result indicating that the driver is in an abnormal mood is obtained, the DMS gives priority to sending out a reminder message of abnormal mood, including: the DMS sends out a voice message through the vehicle's speakers to remind the driver to pay attention to regulating his mood; and displays a text message on the large screen of the vehicle's center console to remind the driver to pay attention to regulating his mood. In a possible implementation, corresponding music to regulate the mood can be played according to the user's abnormal mood, and the atmosphere light inside the vehicle can also be adjusted according to the user's abnormal mood, thereby regulating the user's mood. Exemplarily, the user can control the music and the atmosphere light through voice.

[0065] The embodiment of the present application obtains the driver detection result in the video image of the driving seat of the vehicle when the vehicle is in the starting state, the gear position is the forward gear and the steering angle of the steering wheel is greater than the first angle; if there is a driver in the video image, the fatigue driving detection result, the distraction detection result, the dangerous driving detection result and the abnormal emotion detection result are obtained according to the driver's expression and behavior to determine whether the driver is in a fatigue driving state, a distracted state, a dangerous driving state or has abnormal emotions; in response to obtaining at least one of the detection result indicating that the driver is in a fatigue driving state, the detection result indicating that the driver is in a distracted state, the detection result indicating that the driver is in a dangerous driving state, or the detection result indicating that the driver has abnormal emotions, a prompt message is issued based on the priority level, so as to prompt the driver in the order of the degree of danger, avoid the driver's fatigue driving, distraction, dangerous driving and driving with abnormal emotions, and ensure driving safety.

[0066] See also Figure 3, an embodiment of the present application provides a device for monitoring driver behavior, the device comprising:

[0067] The first acquisition module 301 is used to acquire a video image of a driving position of the vehicle in response to the vehicle being in a starting state, the gear position being a forward gear, and the steering angle of the steering wheel being greater than a first angle;

[0068] A second acquisition module 302 is used to acquire a driver detection result in the video image, where the driver detection result in the video image is used to indicate whether there is a driver in the video image;

[0069] The third acquisition module 303 is used for obtaining a fatigue driving detection result, a distraction detection result, a dangerous driving detection result and an abnormal emotion detection result according to the driver's expression and behavior in response to obtaining a detection result indicating the presence of a driver in the video image, wherein the fatigue driving detection result is used to indicate whether the driver is in a fatigue driving state, the distraction detection result is used to indicate whether the driver is in a distracted state, the dangerous driving detection result is used to indicate whether the driver is in a dangerous driving state, and the abnormal emotion detection result is used to indicate whether the driver's emotion is abnormal;

[0070] The prompt module 304 is used to respond to at least one of the detection results indicating that the driver is in a fatigued driving state, the detection results indicating that the driver is in a distracted state, the detection results indicating that the driver is in a dangerous driving state, or the detection results indicating that the driver is in an abnormal emotion, and issue a prompt message based on a priority level, wherein the priority level decreases in the order of dangerous driving prompt, distracted state prompt, fatigue driving prompt and abnormal emotion prompt.

[0071] In one possible implementation, the second acquisition module 302 is used to identify the video image through a driver recognition model; in response to recognizing that the expression and behavior of at least one character image meet the preset driver characteristics, a detection result indicating the presence of a driver in the video image is obtained.

[0072] In one possible implementation, the third acquisition module 303 is used to obtain a detection result indicating that the driver is in a fatigue driving state and the fatigue level is mild fatigue in response to identifying that the driver yawns more than a first threshold, closes his eyes more than a second threshold, or lowers his head more than a third threshold within a first time period. In response to identifying that the driver closes his eyes or lowers his head for more than a second time period, obtain a detection result indicating that the driver is in a fatigue driving state and the fatigue level is severe fatigue.

[0073] In one possible implementation, the third acquisition module 303 is used to obtain a detection result indicating that the driver is in a distracted state in response to at least one of the driver leaning sideways more than a second angle, leaning over more than a third angle, or not looking forward for more than a third period of time, and not meeting the facial expression and behavioral state corresponding to checking the rearview mirror and blind spot.

[0074] In a possible implementation, the third acquisition module 303 is used to acquire a detection result indicating that the driver is in dangerous driving in response to at least one of the driver drinking water, smoking, and making a phone call.

[0075] In one possible implementation, the third acquisition module 303 is used to determine the driver's emotion type based on the driver's expression and behavior; in response to the emotion type being excitement, anger, impatience or sadness, and lasting for more than a fourth duration, a detection result indicating that the driver's emotions are abnormal is obtained.

[0076] In one possible implementation, the prompt module 304 is used to, in response to obtaining a detection result indicating that the driver is in a fatigued driving state, a detection result indicating that the driver is in a distracted state, a detection result indicating that the driver is in a dangerous driving state, and a detection result indicating that the driver is emotionally abnormal, give priority to issuing a prompt message to avoid dangerous driving; in response to obtaining a detection result indicating that the driver is in a fatigued driving state, a detection result indicating that the driver is in a distracted state, and a detection result indicating that the driver is emotionally abnormal, give priority to issuing a prompt message to avoid distraction; in response to obtaining a detection result indicating that the driver is in a fatigued driving state, and a detection result indicating that the driver is emotionally abnormal, give priority to issuing a prompt message to avoid fatigued driving; in response to obtaining a detection result indicating that the driver is emotionally abnormal, give priority to issuing a prompt message of abnormal emotions.

[0077] The device obtains the driver detection result in the video image of the driving seat of the vehicle when the vehicle is in the starting state, the gear position is the forward gear and the steering angle of the steering wheel is greater than the first angle; if there is a driver in the video image, the fatigue driving detection result, the distraction detection result, the dangerous driving detection result and the abnormal emotion detection result are obtained according to the driver's expression and behavior to determine whether the driver is in a fatigue driving state, a distracted state, a dangerous driving state or has abnormal emotions; in response to obtaining at least one of the detection result indicating that the driver is in a fatigue driving state, the detection result indicating that the driver is in a distracted state, the detection result indicating that the driver is in a dangerous driving state, or the detection result indicating that the driver has abnormal emotions, a prompt message is issued based on the priority level, so as to prompt the driver in the order of the degree of danger, avoid the driver's fatigue driving, distraction, dangerous driving and driving with abnormal emotions, and ensure driving safety.

[0078] It should be noted that the device provided in the above embodiment only uses the division of the above functional modules as an example to implement its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0079] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor of a computer device to enable the computer to implement any of the above-mentioned driver behavior monitoring methods.

[0080] In one possible implementation, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0081] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the above-mentioned methods for monitoring driver behavior.

[0082] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the start and stop status of the vehicle, the gear position of the vehicle, the steering angle of the steering wheel and the video image of the vehicle driving seat involved in this application are all obtained with full authorization.

[0083] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0084] It should be noted that the terms "first", "second", etc. (if any) in the specification and claims of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0085] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for driver behavior monitoring, characterized in that: The method comprises: In response to the vehicle being in a start state, the gear position being a forward gear, and the steering angle of the steering wheel being greater than a first angle, acquiring a video image of a driving position of the vehicle; Acquire a driver detection result in a video image, wherein the driver detection result in the video image is used to indicate whether there is a driver in the video image; In response to obtaining a detection result indicating that the driver exists in the video image, obtaining a fatigue driving detection result, a distraction detection result, a dangerous driving detection result, and an abnormal emotion detection result according to the driver's expression and behavior, wherein the fatigue driving detection result is used to indicate whether the driver is in a fatigue driving state, the distraction detection result is used to indicate whether the driver is in a distracted state, the dangerous driving detection result is used to indicate whether the driver is in a dangerous driving state, and the abnormal emotion detection result is used to indicate whether the driver's emotions are abnormal; In response to obtaining at least one of a detection result indicating that the driver is in the fatigue driving state, a detection result indicating that the driver is in the distracted state, a detection result indicating that the driver is in the dangerous driving state, or a detection result indicating that the driver is emotionally abnormal, a prompt message is issued based on a priority level, and the priority level decreases in the order of dangerous driving prompt, distracted state prompt, fatigue driving prompt, and abnormal emotion prompt.

2. The method according to claim 1, characterized in that The obtaining of the driver detection result in the video image includes: Recognize the video image by using a driver recognition model; In response to recognizing that the expression and behavior of at least one person image conform to preset driver characteristics, a detection result indicating that the driver exists in the video image is obtained.

3. The method according to claim 1, characterized in that Obtaining fatigue driving detection results according to the driver's expression and behavior, including: In response to recognizing that the driver yawns more than a first threshold, closes eyes more than a second threshold, or lowers his head more than a third threshold within a first time period, obtaining a detection result indicating that the driver is in the fatigue driving state and the fatigue level is mild fatigue; In response to identifying that the driver closes his eyes or lowers his head for a period greater than a second period, a detection result is obtained indicating that the driver is in the fatigue driving state and the fatigue level is severe fatigue.

4. The method according to claim 1, characterized in that: Obtaining distraction detection results based on the driver's expression and behavior, including: In response to the driver at least one of leaning sideways more than a second angle, leaning forward more than a third angle, or not looking forward for more than a third period of time, and not having facial expressions and behavioral states corresponding to checking the rearview mirror and blind spots, a detection result is obtained indicating that the driver is in the distracted state.

5. The method according to claim 1, characterized in that Obtaining dangerous driving detection results based on the driver's expression and behavior, including: In response to the driver drinking water, smoking and making phone calls, a detection result indicating that the driver is in the dangerous driving is obtained.

6. The method according to claim 1, characterized in that Obtaining abnormal emotion detection results based on the driver's expression and behavior includes: Determining the driver's emotion type based on the driver's expression and behavior; In response to the emotion type being excitement, anger, impatience or sadness, and lasting for more than a fourth duration, a detection result indicating that the driver's emotion is abnormal is obtained.

7. The method according to claim 1, characterized in that The sending of prompt information based on the priority level includes: In response to obtaining a detection result indicating that the driver is in the fatigue driving state, a detection result indicating that the driver is in the distracted state, a detection result indicating that the driver is in the dangerous driving state, and a detection result indicating that the driver is emotionally abnormal, giving priority to issuing a prompt message to avoid dangerous driving; In response to obtaining a detection result indicating that the driver is in the fatigue driving state, a detection result indicating that the driver is in the distracted state, and a detection result indicating that the driver is emotionally abnormal, giving priority to issuing a prompt message to avoid distraction; In response to obtaining a detection result indicating that the driver is in the fatigue driving state and a detection result indicating that the driver is emotionally abnormal, giving priority to issuing a prompt message to avoid fatigue driving; In response to obtaining a detection result indicating that the driver is in an abnormal mood, prompt information of abnormal mood is issued preferentially.

8. A driver behavior monitoring device, characterized in that: The device comprises: A first acquisition module, configured to acquire a video image of a driving position of the vehicle in response to the vehicle being in a start state, the gear position being a forward gear, and the steering angle of the steering wheel being greater than a first angle; A second acquisition module is used to acquire a driver detection result in the video image, wherein the driver detection result in the video image is used to indicate whether there is a driver in the video image; a third acquisition module, configured to, in response to acquiring a detection result indicating that the driver exists in the video image, acquire a fatigue driving detection result, a distraction detection result, a dangerous driving detection result, and an abnormal emotion detection result according to the driver's expression and behavior, wherein the fatigue driving detection result is used to indicate whether the driver is in a fatigue driving state, the distraction detection result is used to indicate whether the driver is in a distracted state, the dangerous driving detection result is used to indicate whether the driver is in a dangerous driving state, and the abnormal emotion detection result is used to indicate whether the driver's emotions are abnormal; A prompt module is used to respond to at least one of the detection results indicating that the driver is in the fatigue driving state, the detection results indicating that the driver is in the distracted state, the detection results indicating that the driver is in the dangerous driving state, or the detection results indicating that the driver is emotionally abnormal, and issue a prompt message based on a priority level, wherein the priority level decreases in the order of dangerous driving prompt, distracted state prompt, fatigue driving prompt and abnormal emotion prompt.

9. A computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method for driver behavior monitoring as claimed in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the driver behavior monitoring method as described in any one of claims 1 to 8.

Images