Driver state monitoring method and device and storage medium

Abstract

The invention discloses a driver state monitoring method and device and a storage medium, and relates to the technical field of vehicle safety, the driver state monitoring method comprises the steps that mobile device sensing information is acquired, and the mobile device sensing information is determined according to information of a terminal device in a forecabin space of a vehicle; determining a dangerous driving level of a driver of the vehicle at least according to the mobile device sensing information; and according to a preset alarm strategy corresponding to the dangerous driving level, carrying out danger reminding on the driver. According to the scheme, the dangerous driving level of the driver of the vehicle can be determined according to the sensing information of the mobile equipment, the sensing ability of the vehicle to dangerous behaviors of the driver can be effectively improved, and then the driving safety of the vehicle is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle safety, in particular to a driver state monitoring method, device and storage medium. BACKGROUND

[0002] The existing driver monitoring system of the vehicle only relies on the vehicle-mounted camera to collect visual data, and perceives the driver state according to the visual data. However, the vehicle-mounted camera usually has many visual angle blind areas, which leads to insufficient perception ability of the existing driver monitoring system for the dangerous behavior of the driver, and there is a high missed detection rate.

[0003] Based on the above, how to improve the perception ability of the vehicle for the dangerous behavior of the driver to improve the accuracy of the driver state monitoring is a problem to be solved.

[0004] The above content is only used to assist in understanding the technical solutions of the present application, and does not mean that the above content is prior art. SUMMARY

[0005] The main purpose of the present application is to provide a driver state monitoring method, device and storage medium, which aims to solve the technical problem of how to improve the accuracy of driver state monitoring.

[0006] To achieve the above purpose, the present application provides a driver state monitoring method, which comprises: obtaining mobile device perception information, the mobile device perception information being determined for the information of a terminal device in the front cabin space of a vehicle; determining a dangerous driving level of a driver of the vehicle according to at least the mobile device perception information; warning the driver according to a preset warning strategy corresponding to the dangerous driving level.

[0007] In an embodiment, the step of determining the dangerous driving level of the driver of the vehicle according to at least the mobile device perception information comprises: determining the dangerous driving level of the driver of the vehicle based on the mobile device perception information and driving evaluation auxiliary information; the driving evaluation auxiliary information comprises device state and / or driver monitoring system detection information of the terminal device, and the driver monitoring system detection information is collected for at least one of the posture, face, expression and line of sight of the driver.

[0008] In an embodiment, the mobile device perception information comprises posture information of the terminal device, and the step of determining the dangerous driving level of the driver of the vehicle based on the mobile device perception information and driving evaluation auxiliary information comprises: when the driving evaluation auxiliary information comprises the device state of the terminal device, determining a dangerous driving level of the driver based on the posture information and the device state of the terminal device; when the driving evaluation auxiliary information comprises the driver monitoring system detection information, determining face detection information of the driver according to the driver monitoring system detection information, and determining a dangerous driving level of the driver based on the posture information and the face detection information.

[0009] In an embodiment, the step of determining the dangerous driving level of the driver based on the posture information and the device state of the terminal device comprises: in response to determining, based on the posture information, that the inclination angle of the terminal device is greater than a preset angle threshold and that the device state of the terminal device is a use state, determining that the dangerous driving level of the driver of the vehicle is a first dangerous level.

[0010] In an embodiment, the step of determining the dangerous driving level of the driver based on the posture information and the face detection information comprises: determining an abnormal state of the driver according to the posture information and the face detection information; determining a dangerous driving level of the driver according to the duration of the abnormal state.

[0011] In an embodiment, before the step of determining the dangerous driving level of the driver based on the posture information and the face detection information, the method further comprises: determining that the device state of the terminal device is a use state.

[0012] In an embodiment, the posture information comprises an inclination angle of the terminal device, and the step of determining an abnormal state of the driver according to the posture information and the face detection information comprises: when the inclination angle is greater than a preset angle threshold and the face detection information is that a face is not lost, determining that the driver is in a first abnormal state; when the inclination angle is greater than a preset angle threshold and the face detection information is that a face is lost, determining that the driver is in a second abnormal state.

[0013] In an embodiment, the step of determining a dangerous driving level of the driver according to the duration of the abnormal state comprises: when the duration of the first abnormal state is greater than a first preset duration, determining that the dangerous driving level of the driver is a second dangerous driving level; when the duration of the second abnormal state is greater than a second preset duration, determining that the dangerous driving level of the driver is a third dangerous driving level; determining that the dangerous driving level of the driver is a dangerous driving fourth level when the duration of the second abnormal state is greater than a third preset duration, the third preset duration being greater than the second preset duration; wherein the dangerous driving second level, the dangerous driving third level, and the dangerous driving fourth level represent dangerous degrees that increase in turn.

[0014] In an embodiment, the preset warning strategy is implemented through at least one of the following sub-warning operations: controlling the steering wheel of the vehicle to vibrate; controlling a preset light of the instrument panel of the vehicle to flash; controlling the voice module of the vehicle to broadcast preset voice content; controlling the car machine display screen of the vehicle to display preset text or preset video.

[0015] In an embodiment, the mobile device perception information is collected for the terminal device according to a detection frequency matched with the vehicle speed of the vehicle.

[0016] In an embodiment, the obtaining of the mobile device perception information comprises: when the vehicle speed of the vehicle is greater than a first preset speed, collecting mobile device perception information for the terminal device based on a first detection frequency; when the vehicle speed of the vehicle is greater than a second preset speed and is not greater than the first preset speed, collecting mobile device perception information for the terminal device based on a second detection frequency; the second preset speed is less than the first preset speed, and the second detection frequency is less than the first detection frequency; when the vehicle speed of the vehicle is not greater than the second preset speed, collecting mobile device perception information for the terminal device based on a third detection frequency; the third detection frequency is less than the second detection frequency.

[0017] In an embodiment, the mobile device perception information comprises at least one of wireless signal data, temperature data, image data, video data, audio data, and attitude data.

[0018] In an embodiment, the driver monitoring system detection information comprises at least one of image data, video data, thermal imaging data, point cloud data, attitude data, and audio data.

[0019] In an embodiment, before the step of performing dangerous reminding on the driver according to the preset warning strategy corresponding to the dangerous driving level, the method further comprises: detecting a spatial position of a terminal device corresponding to the mobile device perception information in the vehicle; determining that the spatial position is a main driving position.

[0020] In addition, to achieve the above object, the application further provides a driver state monitoring device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the driver state monitoring method.

[0021] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the driver state monitoring method.

[0022] The one or more technical solutions provided by the application have at least the following technical effects: The driver state monitoring method, device and storage medium provided by the embodiments of the application specifically determine the mobile device perception information, determine the dangerous driving level of the driver of the vehicle according to at least the mobile device perception information, and remind the driver of danger according to the preset warning strategy corresponding to the dangerous driving level.

[0023] The application determines the dangerous driving level of the driver according to at least the mobile device perception information, and then reminds the driver of danger according to the corresponding warning strategy, realizes the driver state monitoring and early warning scheme combined with the mobile device perception information according to the above scheme, can effectively improve the perception ability of the vehicle to the dangerous behavior of the driver, and then improves the driving safety of the vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the application and, together with the specification, serve to explain the principles of the application.

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0026] Figure 1 The flowchart provided for the driver state monitoring method embodiment one of the application; Figure 2 The flowchart provided for the driver state monitoring method embodiment two of the application; Figure 3The flowchart provided for Embodiment Three of the driver state monitoring method of the present application; Figure 4 The flowchart provided for Embodiment Five of the driver state monitoring method of the present application; Figure 5 The driver state detection process involved in the embodiments of the present application; Figure 6 The module structure diagram of the driver state monitoring device of the embodiments of the present application; Figure 7 The device structure diagram of the hardware operating environment involved in the driver state monitoring method of the embodiments of the present application.

[0027] The purposes, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0028] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and do not limit the present application.

[0029] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the drawings of the specification and specific embodiments.

[0030] The main solution of the embodiments of the present application is: obtaining mobile device perception information, the mobile device perception information is determined for the information of the terminal device in the front cabin space of the vehicle; at least according to the mobile device perception information, determining the dangerous driving level of the driver of the vehicle; according to the preset warning strategy corresponding to the dangerous driving level, the driver is reminded of danger.

[0031] Technical terms involved in the embodiments of the present application: Driver Monitoring System (DMS): Driver Monitoring System is a technology that monitors the state of the driver in real time through sensors and algorithms, aiming to improve driving safety. DMS usually includes vehicle-mounted cameras, infrared sensors or biometric identification devices to detect the driver's attention, fatigue level, line of sight direction and even emotional state.

[0032] The existing driver monitoring system of the vehicle only relies on the vehicle-mounted camera to collect visual data and perceives the driver's state according to the visual data. However, the vehicle-mounted camera usually has many visual angle blind areas, resulting in insufficient perception ability of the existing driver monitoring system for the dangerous behavior of the driver, and a high rate of missed detection.

[0033] Even though existing technologies can detect driver distraction through steering wheel torque, this method cannot identify non-driving behaviors such as looking down at a mobile phone, and still suffers from insufficient ability to perceive dangerous behaviors by drivers.

[0034] Therefore, existing technologies face the challenge of improving the accuracy of driver condition monitoring.

[0035] This application provides a solution that obtains mobile device perception information from a terminal device in the front cabin of a vehicle, determines the driver's dangerous driving level based on the mobile device perception information, and then provides a danger warning to the driver according to the corresponding warning strategy based on the different levels of dangerous driving behavior.

[0036] Furthermore, the mobile device sensing information in this application embodiment can be the posture information of the aforementioned terminal device. In this way, it can be determined whether the driver or assistant driver in the front cabin of the vehicle is using the terminal device based on the posture information of the terminal device, and the driver's dangerous driving level can be determined based on the usage of the terminal device, so as to improve the accuracy of dangerous driving identification.

[0037] Furthermore, in this embodiment, the perceived information of the mobile device and at least one related auxiliary judgment information can be used to comprehensively assess the driver's dangerous driving level. For example, driving assessment auxiliary information can be determined based on the device status of the terminal device and / or the detection information of the driver monitoring system. Then, the dangerous driving level of the vehicle's driver can be determined by combining the perceived information of the mobile device and the driving assessment auxiliary information. This can further effectively improve the vehicle's ability to perceive the driver's dangerous behavior, thereby improving the vehicle's driving safety. The detection information of the driver monitoring system can be, but is not limited to, information collected from at least one of the driver's posture, face, expression, and gaze direction.

[0038] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as at least one of a tablet computer, personal computer, mobile phone, vehicle, and server communicating with the vehicle, or a driver status monitoring device, vehicle networking device, Internet of Things device, etc., capable of realizing the above functions. The following description uses a driver status monitoring device as an example to illustrate this embodiment and the subsequent embodiments.

[0039] Based on this, embodiments of this application provide a driver status monitoring method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the driver status monitoring method of this application.

[0040] In this embodiment, the driver status monitoring method includes steps S110~S130: Step S110: Obtain mobile device perception information, wherein the mobile device perception information is determined based on the information of the terminal device in the front cabin space of the vehicle. It should be noted that mobile device sensing information refers to the sensing information collected by terminal devices within the vehicle's front cabin space. Specifically, this refers to the sensing information collected by terminal devices that maintain a communication connection with the vehicle while it is in motion. The vehicle's front cabin space refers to the space corresponding to the front seats.

[0041] In this application embodiment, the type of terminal device in the front cabin space of the vehicle is not limited in much. Those skilled in the art can set it according to actual needs. For example, the terminal device may include, but is not limited to, one or any combination of smartphones, tablets, speakers, smartwatches, smart bracelets, augmented reality glasses, etc.

[0042] Optionally, the mobile device sensing information may include, but is not limited to, at least one of: wireless signal data, temperature data, image data, video data, audio data, and posture data. Specifically, the mobile device sensing information includes wireless signal data from terminal devices communicating with the vehicle, such as WIFI (Wireless Fidelity) data and Bluetooth signal data, which can be used to determine the precise location of the terminal devices; the mobile device sensing information may also be data obtained by at least one type of sensor, such as temperature data, image data, video data, audio data, and posture data, to reflect multi-dimensional information such as the driver's skin temperature, limb movements, head posture, and breathing rate.

[0043] First, it is preferable to establish a communication connection between the terminal device and the vehicle via Bluetooth to maintain data sharing between the vehicle and the terminal device.

[0044] When the driver status monitoring device determines that the vehicle is currently in a driving state, it obtains the mobile device perception information from the terminal device in the front cabin space through the communication connection, so as to determine the driver's dangerous driving level based on the mobile device perception information.

[0045] Understandably, when the Bluetooth connection between the vehicle and the terminal device is lost or cannot be established via Bluetooth, near-field communication can be achieved through in-vehicle Wi-Fi (Wireless Fidelity) on the local area network.

[0046] Step S120: Determine the dangerous driving level of the driver of the vehicle based at least on the sensing information of the mobile device; Specifically, to improve the accuracy of driver status monitoring, in this embodiment, the driver status monitoring device can identify dangerous behaviors of the driver based on the information perceived by the mobile device. These behaviors include, but are not limited to, non-driving behaviors such as using a mobile phone, making phone calls, chatting with passengers, or drinking water inside the vehicle. The device then grades and assesses the identified dangerous behaviors to determine the driver's dangerous driving level, so that a dangerous driving warning can be issued to the driver based on the dangerous driving level. Dangerous behaviors refer to driving and non-driving behaviors that threaten vehicle safety during vehicle operation.

[0047] In one feasible implementation, in step S120, the dangerous driving level of the vehicle's driver can be determined based on the mobile device's perception information and driving assessment assistance information; wherein, the driving assessment assistance information includes the device status of the terminal device and / or the driver monitoring system detection information, wherein the driver monitoring system detection information is collected based on at least one of the driver's posture, face, expression, and gaze direction; the device status is used to indicate the access status of the terminal device, such as including, but not limited to, a usage status and a non-use status, where the usage status can indicate that the terminal device is currently being used / accessed by the user, and the non-use status can indicate that the terminal device is currently not being used / accessed by the user, etc.

[0048] In one feasible implementation, when the mobile device sensing information includes the posture information of the terminal device and the driving assessment assistance information includes the device status of the terminal device, the step of determining the dangerous driving level of the driver of the vehicle based on the mobile device sensing information and the driving assessment assistance information may include step A1: Step A1: Determine the driver's dangerous driving level based on the posture information and the device status of the terminal device.

[0049] It should be noted that the attitude information of the terminal device may refer to, but is not limited to, the position and / or orientation information of the terminal device in three-dimensional space.

[0050] Specifically, the driver status monitoring device uses the position and / or orientation information of the terminal device in three-dimensional space, as well as the current device status, to determine whether the terminal device near the driver's seat is being used, in order to determine whether the current driver is using the mobile phone for entertainment, making calls, or other non-driving behaviors, and thus determine the driver's dangerous driving level.

[0051] Optionally, in response to determining that the tilt angle of the terminal device is greater than a preset angle threshold based on the posture information, and that the device status of the terminal device is in use, the dangerous driving level of the driver of the vehicle is determined to be the first dangerous level.

[0052] The preset angle threshold is used to detect whether the terminal device is being used. It is usually set by relevant personnel according to actual needs. In this embodiment, it can be selected as 45 degrees or 50 degrees, etc.

[0053] Specifically, when the driver status monitoring device determines, through the attitude information of the terminal device, that the tilt angle between the terminal device and the horizontal plane of the vehicle is greater than a preset angle threshold, and based on the perception information of the mobile device, it determines that the terminal device is in use, the driver's dangerous driving level can be determined to be the first dangerous level.

[0054] In another feasible implementation, when the driving assessment assistance information includes driver monitoring system detection information, the step of determining the driver's dangerous driving level based on mobile device perception information and driving assessment assistance information may include step B1: Step B1: Determine the driver's dangerous driving level based on the mobile device's sensing information and the driver monitoring system's detection information.

[0055] It should be noted that the information detected by the driver monitoring system refers to at least one state information of the driver, including posture, face, expression, and gaze direction, detected by the DMS (Driver Monitoring System) system during driving based on monitoring algorithms. This includes, but is not limited to, missing face, duration of continuous mouth opening, head movement frequency, gaze direction, fatigue level, and emotional state. The monitoring algorithm can be a neural network algorithm or a machine learning model, and no specific restrictions are imposed here.

[0056] Optionally, the driver monitoring system may detect information including, but not limited to, at least one of: image data, video data, thermal imaging data, point cloud data, attitude data, and audio data. For example, the driver monitoring system may collect thermal imaging data of the vehicle's front cabin space through thermal imaging, collect driver attitude data through inertial sensors and / or pressure sensors in the seat, and collect audio data through an onboard microphone.

[0057] Specifically, the driver status monitoring equipment achieves collaborative perception of the driver's status by integrating information perceived by mobile devices and information detected by the driver monitoring system. It not only perceives the driver's status through information detected by the driver monitoring system, but also uses information perceived by mobile devices to determine the status information of terminal devices in the vehicle's front cabin. This comprehensive assessment determines whether the driver is engaging in dangerous behavior, and further, by determining the dangerous consequences of such behavior on vehicle driving, establishes the driver's dangerous driving level.

[0058] Step S130: Based on the preset alarm strategy corresponding to the dangerous driving level, provide a danger warning to the driver.

[0059] It should be noted that the preset alarm strategy refers to a set of risk-response mapping rules that are set in advance for different levels of dangerous driving, including vehicle alarm schemes under different levels of dangerous driving.

[0060] A vehicle alarm scheme refers to an alarm method that transmits risk information to the driver and / or other road users around the vehicle through at least one of the perceptible channels (such as vision, hearing, touch, smell, or a combination of multiple modes). A vehicle alarm scheme includes, but is not limited to, at least one of the following: alarm intensity, alarm timing, alarm method, alarm content, alarm duration, repetition cycle, and emergency notification recipients.

[0061] Specifically, in this embodiment, the driver status monitoring device needs to determine the alarm scheme for the current scenario from the preset alarm strategy based on the degree of danger of the current vehicle driving reflected by the dangerous driving level, and realize the danger reminder to the driver by executing the alarm scheme.

[0062] This application, through the aforementioned scheme, specifically by acquiring mobile device perception information, determines at least the driver's dangerous driving level based on the mobile device perception information, and then provides a danger warning to the driver according to the corresponding warning strategy. Based on the above scheme, a driver status monitoring and early warning scheme combining mobile device perception information is realized, which can effectively improve the vehicle's ability to perceive dangerous driver behavior, thereby improving vehicle driving safety.

[0063] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 When the mobile device perception information includes the attitude information of the terminal device and the driving assessment assistance information includes the detection information of the driver monitoring system, the step of determining the dangerous driving level of the vehicle's driver based on the mobile device perception information and the driving assessment assistance information may include steps S210~S220: Step S210: Determine the driver's face detection information based on the detection information from the driver monitoring system; Specifically, the driver status monitoring device first obtains the posture information of the terminal device based on the mobile device's perception information, that is, it obtains the position and orientation information of the terminal device in three-dimensional space through the mobile device's perception information; at the same time, it determines the driver's face detection information based on the driver monitoring system's detection information, which may include, but is not limited to, at least two dimensions of facial features and / or head features.

[0064] Step S220: Based on the posture information and the face detection information, determine the driver's dangerous driving level.

[0065] Specifically, the driver status monitoring equipment can determine whether the terminal device is in "use" state by using the posture information of the terminal device. Then, based on information such as face loss, duration of continuous mouth opening, and head swaying frequency in the face detection information, it can determine whether the driver is yawning, losing face, dozing off, looking down, or tilting their head. It can comprehensively judge whether the driver has dangerous behaviors of distracted driving, and then determine the dangerous driving level of the driver by determining the dangerous consequences of the dangerous behaviors on the vehicle.

[0066] Furthermore, since the driver monitoring system also collects information on the driver's gaze direction and facial expressions, before determining the driver's dangerous driving level, the driver's gaze direction detection information and facial expression detection information can be used to determine whether the driver has a distracted gaze or lack of concentration, or a sudden health abnormality, in order to help determine whether to increase the driver's dangerous driving level.

[0067] Optionally, the step of determining the driver's dangerous driving level based on the posture information and the face detection information includes: Based on the posture information and the face detection information, the abnormal state of the driver is determined; The driver's dangerous driving level is determined based on the duration of the abnormal state.

[0068] Specifically, the driver's abnormal state is first determined by combining the posture information of the terminal device with the driver's face detection information.

[0069] For example, the device's posture information can be used to determine if it is in use. Then, facial detection information can be used to determine if the driver is looking down, tilting their head, or losing their face. This can prevent the front passenger or rear passenger from accidentally triggering alarms when using the device, accurately identify the driver's non-driving behavior when using the device, and determine the driver's current abnormal state.

[0070] Finally, based on the duration of the driver's abnormal state, the degree of dangerous driving by the current driver is assessed, and the driver's dangerous driving level is determined.

[0071] Before the step of determining the driver's dangerous driving level based on the posture information and the face detection information, the method further includes: The device status of the terminal device is determined to be in use.

[0072] In order to accurately determine whether the terminal device is in use, the driver monitoring system needs to obtain the device status of the terminal device based on the device's posture information and the mobile device's perception information, in addition to the device's posture information. If the device status is determined to be in use, the driver's dangerous driving level is further determined based on posture information and face detection information. If the device status is detected to be in non-use, the step of determining the driver's dangerous driving level based on posture information and face detection information is not necessary.

[0073] Understandably, terminal devices can integrate information about whether they are in use into mobile device sensing information and report it. The application usage status, system operation status, and power consumption status of terminal devices can all be used to infer whether the terminal device is in use.

[0074] Furthermore, before the step of providing a hazard warning to the driver based on the preset warning strategy corresponding to the dangerous driving level, steps C1 to C2 are also included: Step C1: Detect the spatial position of the terminal device corresponding to the mobile device's sensing information in the vehicle. Step C2: When the spatial location is detected to be the driver's seat of the vehicle, the driver is then alerted to danger according to the preset warning strategy corresponding to the dangerous driving level.

[0075] In this embodiment of the application, to avoid accidental alarm triggering when the front passenger or rear passenger uses the device, when at least two terminal devices are connected to the vehicle and it is impossible to distinguish whether the mobile device's sensing information corresponds to the terminal device in the driver's seat, the driver status monitoring device can achieve centimeter-level positioning of the terminal device based on UWB (Ultra Wide Band) technology. By using UWB technology to detect the specific spatial location of the terminal device corresponding to the mobile device's sensing information in the vehicle at the centimeter level, it can determine whether the terminal device is being used by the driver or the passenger. If the terminal device is being used by the driver, a danger warning is triggered according to a preset alarm strategy. If the terminal device is being used by the passenger, the danger warning is canceled.

[0076] When the specific spatial location in step C2 is the passenger seat, the warning operation for the driver is turned off, thereby canceling the danger warning to the driver.

[0077] This application, through the aforementioned scheme, specifically determines the driver's face detection information based on the information detected by the driver monitoring system; and comprehensively determines the driver's dangerous driving level by combining the posture information and face detection information of the terminal device. This achieves collaborative perception of the driver's state by combining the perception information of the mobile device and the detection information of the driver monitoring system, thereby constructing a driver state monitoring and early warning scheme. This can effectively improve the vehicle's ability to perceive dangerous behaviors of the driver, thereby improving vehicle driving safety.

[0078] Based on the above embodiments of this application, in the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 The posture information includes the tilt angle of the terminal device, and the abnormal state includes a first abnormal state and a second abnormal state. The step of determining the driver's abnormal state based on the posture information and the face detection information includes steps S310 to S320: Step S310: When the tilt angle is greater than a preset angle threshold and the face detection information indicates that the face has not been lost, it is determined that the driver is in a first abnormal state. Step S320: When the tilt angle is greater than a preset angle threshold and the face detection information indicates a lost face, it is determined that the driver is in a second abnormal state.

[0079] Specifically, when the driver status monitoring device determines that the tilt angle of the terminal device (such as a mobile phone) is greater than the preset angle threshold, it determines that the terminal device is in use. According to the driver's face detection information, no missing parts of the driver's face or head are found, that is, the face detection information indicates that the face is not lost, thus determining that the driver is in the first abnormal state.

[0080] When the driver status monitoring device determines that the tilt angle of the terminal device is greater than the preset angle threshold and the driver's face detection information is that the face is missing, it means that the driver is engaging in non-driving behavior by looking down or tilting his head to use the terminal device (such as a mobile phone). Among these, the missing face indicates that the driver is looking down or tilting his head, thus determining that the driver is in the second abnormal state.

[0081] Furthermore, the dangerous driving level includes dangerous driving level two, dangerous driving level three, and dangerous driving level four. The step of determining the driver's dangerous driving level based on the duration of the abnormal state includes steps D1 to D3: Step D1: When the duration of the first abnormal state is longer than the first preset duration, the driver's dangerous driving level is determined to be dangerous driving level 2. Step D2: When the duration of the second abnormal state is longer than the second preset duration, the driver's dangerous driving level is determined to be dangerous driving level three. Step D3: When the duration of the second abnormal state is greater than the third preset duration, the driver's dangerous driving level is determined to be dangerous driving level four, where the third preset duration is greater than the second preset duration. The degree of danger represented by the second level of dangerous driving, the third level of dangerous driving, and the fourth level of dangerous driving increases sequentially.

[0082] In this embodiment of the application, when it is determined that the tilt angle of the terminal device in the current driving scenario is greater than a preset threshold, the driver's dangerous driving level is further classified and assessed by quantitatively evaluating the duration of the aforementioned two abnormal states of the driver, including dangerous driving level 2, dangerous driving level 3, and dangerous driving level 4.

[0083] The degree of danger corresponding to the second, third, and fourth levels of dangerous driving is as follows: dangerous driving level 2 < dangerous driving level 3 < dangerous driving level 4.

[0084] It should be understood that driver condition monitoring equipment can, according to actual needs, further increase the number of preset time periods within a certain assessment time range to enhance the risk assessment scheme for dangerous driving levels, thereby improving the sensitivity of driver condition monitoring.

[0085] Specifically, when the driver is detected to be in a first abnormal state or a second abnormal state, a timer is started to count the duration of the first abnormal state and the duration of the second abnormal state.

[0086] When the duration of the first abnormal state exceeds a first preset duration, the driver's dangerous driving level is determined to be Level 2; when the duration of the second abnormal state exceeds a second preset duration, the driver's dangerous driving level is determined to be Level 3; when the duration of the second abnormal state exceeds a third preset duration, the driver's dangerous driving level is determined to be Level 4. The first preset duration is no greater than the second preset duration, and the second preset duration is less than the third preset duration.

[0087] This embodiment determines the driver to be in a first abnormal state when the tilt angle is greater than a preset angle threshold and the face detection information indicates that the face is not lost; and determines the driver to be in a second abnormal state when the tilt angle is greater than the preset angle threshold and the face detection information indicates that the face is lost. This scheme identifies two abnormal driver states based on the terminal device's tilt angle and face detection information. This allows for subsequent quantification of the dangerous driving degree of the abnormal state to determine the driver's dangerous driving level. Based on a preset warning strategy, the driver is then given a hazard warning. This driver state monitoring and early warning scheme, combining mobile device sensing information, effectively improves the vehicle's ability to perceive dangerous driver behavior, thereby enhancing vehicle driving safety.

[0088] Based on the above embodiments of this application, in the fourth embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. The preset alarm strategy is implemented through at least one of the following sub-alarm operations: Control the vibration of the vehicle's steering wheel; Control the preset lights on the vehicle's dashboard to flash; Control the vehicle's voice module to broadcast preset voice content; Control the vehicle's infotainment display to show preset text or preset videos.

[0089] It should be understood that since the alarm schemes in the preset alarm strategy can alert the driver through multiple perceptible channels, the alarm schemes can include visual alarms, auditory alarms, tactile alarms, and olfactory alarms. Among them, visual alarms refer to alarms presented on the visual screen or vehicle dashboard through changes in icons, text, color, and brightness; auditory alarms refer to alarms played through the vehicle's speakers in the form of beeps, voice announcements, and warning sounds; tactile alarms refer to alarms through vibrations of the steering wheel or seat, seat belt pretensioning, etc.; and olfactory alarms refer to alarms that stimulate the driver's alertness by releasing mint or lemon scents through an aromatherapy system.

[0090] Specifically, since the degree of danger represented by the second level of dangerous driving, the third level of dangerous driving, and the fourth level of dangerous driving increases sequentially, the driver state monitoring device can determine that the preset alarm strategy corresponding to the second level of dangerous driving is a tactile alarm, which can be achieved by controlling the vibration of the vehicle's steering wheel.

[0091] The driver state monitoring equipment can also determine the preset warning strategy corresponding to the third level of dangerous driving, which is a visual warning. This visual warning can be triggered by controlling preset lights on the vehicle's dashboard or by displaying preset text or video on the vehicle's infotainment display screen. The driver state monitoring equipment can also determine the preset warning strategy corresponding to the fourth level of dangerous driving, which is an auditory warning. This auditory warning can be triggered by controlling the vehicle's voice module to broadcast preset voice content.

[0092] By implementing a pre-set alarm scheme based on different levels of dangerous driving, the driver status monitoring device can alert the driver to dangerous behaviors during driving, thereby ensuring vehicle safety during driving.

[0093] Understandably, due to the different levels of dangerous driving, compared to low-level dangerous driving scenarios, high-level dangerous driving scenarios can adopt a combination of multiple alarm schemes, such as combining visual reminders and auditory alarms to increase the alarm intensity for the driver.

[0094] In this embodiment, the relationship between the dangerous driving level and the corresponding alarm scheme can be referred to as follows: By tilting the phone at an angle θ>45° for 10 seconds, the dangerous driving level is determined to be Level 2, and the corresponding warning scheme is to control slight vibration of the steering wheel.

[0095] Based on the phone's tilt angle θ>45° and the loss of facial recognition for 20 seconds, the dangerous driving level was determined to be Level 3, and the corresponding alarm scheme was to control the instrument panel to flash a red warning light.

[0096] Based on the phone's tilt angle θ>45° and the loss of facial recognition for 30 seconds, the dangerous driving level was determined to be Level 4, and the corresponding alarm scheme was to control the vehicle's speaker to broadcast the message "Please focus on driving".

[0097] This embodiment, through the above-described scheme, sets up three alarm schemes corresponding to three dangerous driving levels, and realizes the detection of driver status and graded alarm scheme by combining the sensor information of mobile devices. This can effectively improve the vehicle's ability to perceive dangerous driver behavior, thereby improving vehicle driving safety.

[0098] Based on the above embodiments of this application, in order to avoid the influence of vehicle speed on the judgment of dangerous driving level during driving, this application proposes a method for adjusting the collection frequency of mobile device sensing information according to the vehicle speed. That is, the mobile device sensing information in the embodiments of this application can be collected for the terminal device according to the detection frequency matched to the vehicle speed. In other words, when the vehicle is traveling at different speeds, mobile device sensing information can be collected for each terminal device in the front cabin space according to different detection frequencies.

[0099] As one embodiment, the detection frequency of mobile device sensing information and the vehicle speed can be positively correlated in this application embodiment. That is, the greater the vehicle speed, the higher the detection frequency is used to collect mobile device sensing information.

[0100] For ease of understanding, in the fifth embodiment of this application, content that is the same as or similar to the above embodiments can be referred to the above description and will not be repeated hereafter. Based on this, a specific example of collecting vehicle speed sensing information from a mobile device is given; please refer to... Figure 4 , Step S110 includes steps S410 to S430: Step S410: When the vehicle speed is greater than the first preset speed, collect mobile device sensing information for the terminal device based on the first detection frequency. Step S420: When the vehicle speed is greater than the second preset speed and not greater than the first preset speed, the mobile device sensing information is collected for the terminal device based on the second detection frequency; the second preset speed is less than the first preset speed, and the second detection frequency is less than the first detection frequency. Step S430: When the vehicle speed is not greater than the second preset speed, the mobile device perception information is collected for the terminal device based on the third detection frequency; the third detection frequency is less than the second detection frequency.

[0101] Specifically, when the vehicle speed is greater than a first preset speed (e.g., 60 km / h), the mobile device perception information is collected based on a first detection frequency. The first preset speed can be selected as 60 km / h or 65 km / h, and the detection period corresponding to the first detection frequency can be selected as 100 ms or 150 ms.

[0102] When the vehicle speed is greater than the second preset speed (e.g., 30 km / h) and not greater than the first preset speed (e.g., 60 km / h), the mobile device perception information is collected based on the second detection frequency. The second preset speed can be selected as 30 km / h or 25 km / h, and the detection period corresponding to the second detection frequency can be selected as 200 ms or 250 ms.

[0103] The second preset speed is less than the first preset speed. The relationship between the first detection frequency, the second detection frequency, and the third detection frequency is: first detection frequency > second detection frequency > third detection frequency.

[0104] When the vehicle speed is no greater than the second preset speed (e.g., 30km / h), the mobile device perception information is collected based on the third detection frequency. The detection period corresponding to the third detection frequency can be selected as 800ms or 1000ms, so as to realize the driver's state perception with low power loss.

[0105] Reference Figure 5 The present application describes a driver state detection process and provides a detailed description of the driver state monitoring method.

[0106] Driver status monitoring methods require the integration of data from both the mobile phone and the vehicle to comprehensively determine the driver's status. Specifically, the vehicle needs to transmit vehicle status data to the mobile phone, which then uses this data to determine that the vehicle's speed is greater than 0, indicating that the vehicle is in motion, and then establishes a real-time communication connection with the vehicle.

[0107] After establishing a communication connection, the phone only sends its tilt angle information to the vehicle when it detects a tilt angle greater than 45 degrees while unlocked, thus triggering facial recognition detection on the vehicle side. Simultaneously, the phone adjusts the frequency of its tilt angle information collection based on vehicle speed data.

[0108] When the vehicle detects a missing face, a driver distraction alert is triggered. The specific alert scheme is determined by analyzing the phone's tilt angle and the duration of the missing face. The driver status detection process stops once the driver's face is detected again. If the vehicle does not detect a missing face, the driver status detection process stops.

[0109] This embodiment provides a driver status monitoring method. By adjusting the collection frequency of mobile device sensing information according to the vehicle speed, the sensitivity of driver status monitoring can be adjusted, improving the monitoring accuracy and response speed in high-risk scenarios, reducing resource consumption when the vehicle is at low speed or stationary, and realizing a driver status monitoring and early warning scheme that combines mobile device sensing information. This can effectively improve the vehicle's ability to perceive dangerous driver behaviors, thereby improving vehicle driving safety.

[0110] This application also provides a driver condition monitoring device, please refer to... Figure 6 The driver status monitoring device includes: The acquisition module 10 acquires mobile device perception information, which is determined based on the information of the terminal device in the front cabin space of the vehicle. The determining module 20 determines the dangerous driving level of the driver of the vehicle, at least based on the sensing information of the mobile device. The reminder module 30 provides a danger reminder to the driver based on a preset alarm strategy corresponding to the dangerous driving level.

[0111] The driver state monitoring device provided in this application, employing the driver state monitoring method in the above embodiments, can solve the technical problem of how to improve the accuracy of driver state monitoring. Compared with the prior art, the beneficial effects of the driver state monitoring device provided in this application are the same as those of the driver state monitoring method provided in the above embodiments, and other technical features in the driver state monitoring device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0112] This application provides a driver state monitoring device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the driver state monitoring method in the above embodiment.

[0113] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing the driver state monitoring device in the embodiments of this application. The driver state monitoring device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The driver condition monitoring device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0114] like Figure 7As shown, the driver state monitoring device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the driver state monitoring device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the driver condition monitoring device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows driver condition monitoring devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0115] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0116] The driver state monitoring device provided in this application, employing the driver state monitoring method in the above embodiments, can solve the technical problem of how to improve the accuracy of driver state monitoring. Compared with the prior art, the beneficial effects of the driver state monitoring device provided in this application are the same as those of the driver state monitoring method provided in the above embodiments, and other technical features of the driver state monitoring device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0117] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0118] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0119] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the driver state monitoring method in the above embodiments.

[0120] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0121] The aforementioned computer-readable storage medium may be included in the driver condition monitoring device; or it may exist independently and not be assembled into the driver condition monitoring device.

[0122] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the driver state monitoring device, cause the driver state monitoring device to: acquire mobile device perception information, the mobile device perception information being determined based on information from a terminal device in the vehicle's front cabin space; determine the dangerous driving level of the vehicle's driver based at least on the mobile device perception information; and provide a danger warning to the driver according to a preset warning strategy corresponding to the dangerous driving level.

[0123] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0126] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described driver state monitoring method, thereby solving the technical problem of how to improve the accuracy of driver state monitoring. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the driver state monitoring method provided in the above embodiments, and will not be repeated here.

[0127] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A method for monitoring driver status, characterized in that, The driver status monitoring method includes: Acquire mobile device sensing information, wherein the mobile device sensing information is determined based on the information of terminal devices in the front cabin space of the vehicle; At least based on the information perceived by the mobile device, determine the level of dangerous driving by the driver of the vehicle; The driver is given a hazard warning based on the preset warning strategy corresponding to the dangerous driving level.

2. The driver status monitoring method as described in claim 1, characterized in that, The step of determining the dangerous driving level of the vehicle's driver, at least based on the information perceived by the mobile device, includes: Based on the mobile device's perception information and driving assessment assistance information, the dangerous driving level of the vehicle's driver is determined; the driving assessment assistance information includes the device status of the terminal device and / or the driver monitoring system's detection information, which is collected based on at least one of the driver's posture, face, expression, and gaze direction.

3. The driver status monitoring method as described in claim 2, characterized in that, The mobile device sensing information includes the posture information of the terminal device. The step of determining the dangerous driving level of the driver of the vehicle based on the mobile device sensing information and driving assessment assistance information includes: When the driving assessment assistance information includes the device status of the terminal device, the driver's dangerous driving level is determined based on the posture information and the device status of the terminal device; When the driving assessment assistance information includes the driver monitoring system detection information, the driver's face detection information is determined based on the driver monitoring system detection information, and the driver's dangerous driving level is determined based on the posture information and the face detection information.

4. The driver status monitoring method as described in claim 3, characterized in that, The step of determining the driver's dangerous driving level based on the posture information and the device status of the terminal device includes: In response to determining, based on the posture information, that the tilt angle of the terminal device is greater than a preset angle threshold, and that the device status of the terminal device is in use, the dangerous driving level of the vehicle driver is determined to be the first dangerous level.

5. The driver status monitoring method as described in claim 3, characterized in that, The step of determining the driver's dangerous driving level based on the posture information and the face detection information includes: Based on the posture information and the face detection information, the abnormal state of the driver is determined; The driver's dangerous driving level is determined based on the duration of the abnormal state.

6. The driver status monitoring method as described in claim 3, characterized in that, Before the step of determining the driver's dangerous driving level based on the posture information and the face detection information, the method further includes: The device status of the terminal device is determined to be in use.

7. The driver status monitoring method as described in claim 5, characterized in that, The posture information includes the tilt angle of the terminal device, and the step of determining the driver's abnormal state based on the posture information and the face detection information includes: When the tilt angle is greater than a preset angle threshold and the face detection information indicates that the face has not been lost, the driver is determined to be in a first abnormal state. When the tilt angle is greater than a preset angle threshold and the face detection information indicates that the face is missing, the driver is determined to be in a second abnormal state.

8. The driver status monitoring method as described in claim 7, characterized in that, The step of determining the driver's dangerous driving level based on the duration of the abnormal state includes: When the duration of the first abnormal state exceeds the first preset duration, the driver's dangerous driving level is determined to be dangerous driving level two. When the duration of the second abnormal state exceeds the second preset duration, the driver's dangerous driving level is determined to be dangerous driving level three. When the duration of the second abnormal state is greater than the third preset duration, the driver's dangerous driving level is determined to be dangerous driving level four, where the third preset duration is greater than the second preset duration; The degree of danger represented by the second level of dangerous driving, the third level of dangerous driving, and the fourth level of dangerous driving increases sequentially.

9. The driver status monitoring method as described in claim 1, characterized in that, The preset alarm strategy is implemented through at least one of the following sub-alarm operations: Control the vibration of the vehicle's steering wheel; Control the preset lights on the vehicle's dashboard to flash; Control the vehicle's voice module to broadcast preset voice content; Control the vehicle's infotainment display to show preset text or preset videos.

10. The driver status monitoring method as described in claim 1, characterized in that, The mobile device sensing information is collected for the terminal device at a detection frequency matched to the vehicle speed of the vehicle. and / or The mobile device sensing information includes at least one of the following: wireless signal data, temperature data, image data, video data, audio data, and posture data.

11. The driver status monitoring method as described in claim 1, characterized in that, Before the step of providing a hazard warning to the driver based on a preset warning strategy corresponding to the dangerous driving level, the method further includes: The spatial location of the terminal device corresponding to the sensing information of the mobile device in the vehicle is detected. The spatial location is determined to be the primary driving position.

12. A driver status monitoring device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the driver state monitoring method as described in any one of claims 1 to 11.

13. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the driver state monitoring method as described in any one of claims 1 to 11.

Images