Driver attention detection

Abstract

A system for monitoring a driver in a vehicle includes a driver monitoring system adapted to monitor a driver of the vehicle; and a system controller in communication with the driver monitoring system and adapted to collect data from the driver monitoring system relating to a driver gaze behavior, classify the driver as one of a plurality of driver attention states according to the data of the driver monitoring system, where the driver attention states include safe, critical and unsafe, and when the driver state is unsafe, proposing a problem to test driver alertness to the driver through a human-machine interface (HMI), receiving a response of the driver to the problem through the HMI, analyzing the response, and initiating a response action according to the response of the driver.

Description

Technical Field

[0001] The present disclosure relates to a system for detecting driver attention. Driver monitoring systems typically use a driver-facing camera equipped with an infrared light emitting diode (LED) or laser to "see" the driver's face even at night and to see the driver's eyes even if the driver is wearing dark sunglasses. Advanced in-vehicle software collects driver data points and creates an initial baseline of the driver's normal, focused state. The software can then determine if the driver is blinking more than usual, if their eyes are squinting or closed, and if their head is tilted at an odd angle. It can also determine if the driver is looking at the road ahead and whether the driver is actually paying attention or just staring ahead absentmindedly. Background Art

[0002] If the system determines that the driver is distracted or drowsy, it can get the driver's attention by sounding an audio alert, illuminating a visual indicator on the instrument panel, or vibrating the seat. If the interior sensors indicate that the driver is distracted and the vehicle's exterior sensors determine that a collision is imminent, the system can automatically brake using information fused from the interior and exterior sensors. However, current systems do not collect feedback from the driver to verify the driver's attention state.

[0003] Therefore, while current driver monitoring systems achieve their intended purpose, a need exists for a new and improved system and method for detecting driver attention and verifying driver status by collecting driver feedback. Summary of the invention

[0004] According to several aspects of the present disclosure, a method for monitoring a driver in a vehicle includes monitoring the driver of the vehicle using a driver monitoring system; collecting data related to the driver's gaze behavior from the driver monitoring system using a system controller that communicates with the driver monitoring system; classifying the driver into one of a plurality of driver attention states based on the data from the driver monitoring system using the system controller, the driver attention states including safe, critical, and unsafe; and when the driver state is unsafe: asking the driver questions to test the driver's alertness through a human-machine interface (HMI) using the system controller; receiving the driver's responses to the questions through the HMI using the system controller; analyzing the responses using the system controller; and initiating a response action based on the driver's responses using the system controller.

[0005] According to another aspect, using the system controller to present questions to the driver through a human machine interface (HMI) to test driver alertness further includes presenting the driver questions related to at least one of the driver's situational awareness and a topic of interest to the driver.

[0006] According to another aspect, using a system controller to classify a driver into one of a plurality of driver attention states based on data from a driver monitoring system further includes calculating a probability that the driver attention state is unsafe, and classifying the driver attention state as unsafe when the probability that the driver attention state is unsafe exceeds a predetermined threshold.

[0007] According to another aspect, calculating the probability that the driver's attention state is unsafe also includes collecting data related to aspects of the current driving session from an onboard system within the vehicle; accessing a historical data database related to classifications of the driver's attention state during past driving sessions; and using a machine learning algorithm to predict the probability that the driver's attention state is unsafe based on the historical data.

[0008] According to another aspect, analyzing the response using the system controller also includes evaluating whether the response is correct; evaluating the driver's understanding of the question; evaluating the speed of the driver's response to the question; and scoring the driver as "pass" or "fail" based on the evaluation of the driver's response to the question.

[0009] According to another aspect, when the system controller scores the driver as a "pass," initiating, using the system controller, a response action based on the driver's response further includes providing a reward to the driver.

[0010] According to another aspect, when the system controller scores the driver as "failed," initiating, using the system controller, a response action based on the driver's response further includes penalizing the driver.

[0011] According to another aspect, punishing the driver includes at least one of the following: using the system controller to shut down the advanced driving assistance function for a period of time through its communication with the onboard system in the vehicle; using the system controller to prohibit the driver from using the advanced driving assistance function for a period of time through its communication with the onboard system in the vehicle; sending an alert to an emergency service provider through the wireless communication module; sending an alert to an emergency contact through the wireless communication module; sending an alert to a parent / guardian through the wireless communication module; and using the system controller to enable the autonomous driving mode through its communication with the vehicle control module.

[0012] According to another aspect, initiating a response action based on a driver's response using a system controller also includes, when the system controller scores the driver as "failed," accessing a historical data database associated with classifications of the driver's attention state during a current driving session and identifying an instance pattern in which the driver was classified as unsafe during the current driving session; accessing a historical data database associated with classifications of the driver's attention state during past driving sessions and identifying an instance pattern in which the driver was classified as unsafe during past driving sessions; and escalating a penalty on the driver based on the identified instance pattern in which the driver was classified as unsafe during at least one of the current driving session and the past driving session.

[0013] According to several aspects of the present disclosure, a system for monitoring a driver in a vehicle includes a driver monitoring system, which is suitable for monitoring the driver of the vehicle; and a system controller, which communicates with the driver monitoring system and is suitable for collecting data related to the driver's gaze behavior from the driver monitoring system, classifying the driver into one of a plurality of driver attention states based on the data from the driver monitoring system, wherein the driver attention states include safe, critical, and unsafe, and when the driver state is unsafe, asking the driver questions to test the driver's alertness through a human-machine interface (HMI), receiving the driver's responses to the questions through the HMI, analyzing the responses, and initiating response actions based on the driver's responses.

[0014] According to another aspect, when presenting questions to the driver through a human machine interface (HMI) to test driver alertness, the system controller is further adapted to present questions to the driver related to at least one of the driver's situational awareness and topics of interest to the driver.

[0015] According to another aspect, when a driver is classified into one of a plurality of driver attention states based on data from a driver monitoring system, the system controller is further adapted to calculate a probability that the driver attention state is unsafe, and to classify the driver attention state as unsafe when the probability that the driver attention state is unsafe exceeds a predetermined threshold.

[0016] According to another aspect, when a driver is classified into one of a plurality of driver states based on data from a driver monitoring system, the system controller is further adapted to collect data related to aspects of a current driving session from an onboard system within the vehicle; access a historical data database related to classifications of the driver's attention states during past driving sessions; predict a probability of the driver's attention state being unsafe based on the historical data using a machine learning algorithm; and classify the driver's attention state as unsafe when the probability of the driver's attention state being unsafe exceeds a predetermined threshold.

[0017] According to another aspect, when analyzing the responses, the system controller is further adapted to evaluate whether the responses are correct; evaluate the driver's understanding of the questions; evaluate the speed of the driver's response to the questions; and score the driver as "pass" or "fail" based on the evaluation of the driver's responses to the questions.

[0018] According to another aspect, when the system controller scores the driver as "passing", the system controller is further adapted to initiate a responsive action including providing a reward to the driver.

[0019] According to another aspect, when the system controller scores the driver as "failed," the system controller is further adapted to initiate a responsive action including penalizing the driver.

[0020] According to another aspect, when a driver is penalized, the system controller is adapted to at least one of: shut down advanced driver assistance functions for a period of time through its communication with an in-vehicle system in the vehicle; prohibit the driver from using advanced driver assistance functions for a period of time through its communication with an in-vehicle system in the vehicle; send an alert to an emergency service provider through a wireless communication module; send an alert to an emergency contact through a wireless communication module; send an alert to a parent / guardian through a wireless communication module; and enable an autonomous driving mode through its communication with a vehicle control module.

[0021] According to another aspect, when penalizing a driver, the system controller is further adapted to access a historical data database associated with classifications of the driver's attention state during a current driving session and identify an instance pattern in which the driver was classified as unsafe during the current driving session; access a historical data database associated with classifications of the driver's attention state during past driving sessions and identify an instance pattern in which the driver was classified as unsafe during past driving sessions; and escalate the penalty against the driver based on the identified instance pattern in which the driver was classified as unsafe during at least one of the current driving session and the past driving session.

[0022] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0024] Figure 1 is a schematic diagram of a system for monitoring a driver of an autonomous vehicle according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram of a system according to an exemplary embodiment;

[0026] Figure 3 is a schematic diagram showing a driver in a vehicle having the disclosed system; and

[0027] Figure 4 is a flow chart illustrating a method according to an exemplary embodiment of the present disclosure.

[0028] The drawings are not necessarily drawn to scale, and some features may be exaggerated or minimized, for example, to show details of a particular component. In some cases, well-known components, systems, materials, or methods are not described in detail to avoid obscuring the present disclosure. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to adopt the present disclosure in different ways. DETAILED DESCRIPTION

[0029] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application or use. In addition, it is not intended to be constrained by any express or implied theory presented in the aforementioned technical field, background technology, brief overview or the following detailed description. It should be understood that throughout the drawings, corresponding reference numerals represent similar or corresponding components and features. As used herein, the term module or controller refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device, individually or in any combination, including but not limited to: application specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or groups) and memories that execute one or more software or firmware programs, combinational logic circuits and / or other suitable components that provide the described functions. Although the drawings shown herein depict examples with certain element arrangements, additional intermediate elements, devices, features or components may be present in actual embodiments. It should also be understood that the drawings are merely illustrative and may not be drawn to scale.

[0030] As used herein, the term "vehicle" is not limited to automobiles. Although the present technology is described herein primarily in conjunction with automobiles, the present technology is not limited to automobiles. These concepts can be used in a variety of applications, such as aircraft, ships, other vehicles, as well as hospitals, healthcare and medical facilities, and consumer electronics components.

[0031] According to an exemplary embodiment, Figure 1A vehicle 10 is shown with an associated system 11 for monitoring driver attention according to various embodiments. Generally speaking, the system 11 works with other systems within the vehicle. The vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and substantially surrounds the components of the vehicle 10. The body 14 and the chassis 12 may together form a frame. The front wheels 16 and the rear wheels 18 are each rotatably connected to the chassis 12 near a respective corner of the body 14.

[0032] In various embodiments, the vehicle 10 is an autonomous vehicle and the system 11 is incorporated into the autonomous vehicle 10. The autonomous vehicle 10 is, for example, a vehicle 10 that is automatically controlled to carry passengers from one location to another. The vehicle 10 is depicted as a passenger car in the illustrated embodiment, but it should be understood that any other vehicle may also be used, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), etc. In an exemplary embodiment, the vehicle 10 is equipped with a so-called level 4 or level 5 automation system. The level 4 system represents "high automation" and refers to the driving mode-specific performance of the autonomous driving system in all aspects of dynamic driving tasks, even if the human driver does not respond appropriately to the intervention request. The level 5 system represents "full automation" and refers to the full-time performance of the autonomous driving system in all aspects of dynamic driving tasks under all road and environmental conditions that the human driver can manage. The novel aspects of the present disclosure are also applicable to non-autonomous vehicles, in which the vehicle will not have the ability to autonomously initiate vehicle actions.

[0033] As shown, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, a vehicle controller 34, and a wireless communication module 36. In an embodiment where the vehicle 10 is an electric vehicle, the transmission system 22 may not be present. In different embodiments, the propulsion system 20 may include an internal combustion engine, a motor such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transmit power from the propulsion system 20 to the front wheels 16 and the rear wheels 18 of the vehicle according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a stepped automatic transmission, a continuously variable transmission, or other suitable transmissions. The braking system 26 is configured to provide braking torque to the front wheels 16 and the rear wheels 18 of the vehicle. In various embodiments, the braking system 26 may include friction brakes, wire control brakes, regenerative braking systems (e.g., motors), and / or other suitable braking systems. The steering system 24 affects the position of the front wheels 16 and the rear wheels 18. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of the present disclosure, the steering system 24 may not include a steering wheel.

[0034] The sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external environment and / or internal environment of the autonomous vehicle 10. The sensing devices 40a-40n may include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal camera, ultrasonic sensor, and / or other sensors. The camera may include two or more digital cameras spaced at a selected distance from each other, wherein the two or more digital cameras are used to obtain stereo images of the surrounding environment in order to obtain a three-dimensional image or map. The plurality of sensing devices 40a-40n are used to determine information about the surrounding environment of the vehicle 10. In an exemplary embodiment, the plurality of sensing devices 40a-40n include a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor. In another exemplary embodiment, the plurality of sensing devices 40a-40n also include sensors for determining information about the surrounding environment of the vehicle 10, such as an ambient air temperature sensor, an atmospheric pressure sensor, and / or a camera and / or video camera for observing the environment in front of the vehicle 10. In another exemplary embodiment, at least one of the plurality of sensing devices 40 a - 40 n is capable of measuring distances around the vehicle 10 .

[0035] In a non-limiting example where the plurality of sensing devices 40a-40n include cameras, the plurality of sensing devices 40a-40n measure distance using an image processing algorithm configured to process images from the cameras and determine distances between objects. In another non-limiting example, the plurality of vehicle sensors 40a-40n include a stereo camera with distance measurement capabilities. In one example, at least one of the plurality of sensing devices 40a-40n is fixed to the interior of the vehicle 10, for example, fixed in the roof lining of the vehicle 10 and visible through the windshield of the vehicle 10. In another example, at least one of the plurality of sensing devices 40a-40n is fixed to the exterior of the vehicle 10, for example, fixed on the roof of the vehicle 10, with a view of the environment surrounding the vehicle 10. It should be understood that various additional types of sensing devices, such as LiDAR sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors are within the scope of the present disclosure. The actuator system 30 includes one or more actuator devices 42 a - 42 n that control one or more functions of the vehicle 10 , such as, but not limited to, the propulsion system 20 , the transmission system 22 , the steering system 24 , and the braking system 26 .

[0036] The vehicle controller 34 includes at least one processor 44 and a computer readable storage device or medium 46. The at least one processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor of multiple processors associated with the vehicle controller 34, a semiconductor-based processor, a microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or any device generally used to execute instructions. The computer readable storage device or medium 46 can include volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the at least one data processor 44 is powered off. The computer readable storage device or medium 46 can be implemented using any of a variety of known storage devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combination storage device capable of storing data used by the controller 34 in controlling the vehicle 10, some of which represents executable instructions.

[0037] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by at least one processor 44, the instructions receive and process signals from the sensor system 28, execute logic, calculations, methods and / or algorithms for automatically controlling components of the vehicle 10, and generate control signals to the actuator system 30 to automatically control components of the vehicle 10 based on the logic, calculations, methods and / or algorithms. Although Figure 1 Only one controller 34 is shown, but embodiments of the vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication mediums and collaborate to process sensor signals, execute logic, calculations, methods and / or algorithms, and generate control signals 10 for automatically controlling autonomous vehicle functions.

[0038] In various embodiments, one or more instructions of the vehicle controller 34 are embodied in the trajectory planning system and, when executed by at least one data processor 44, generate trajectory outputs that address the kinematic and dynamic constraints of the environment. For example, the instructions receive process sensor and map data as input. The instructions implement a graph-based approach and customized cost functions to handle different road scenarios of cities and highways.

[0039] The wireless communication module 36 is configured to wirelessly communicate information with other remote entities 48, such as, but not limited to, other vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems, remote servers, cloud computers, and / or personal devices. In an exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communications. However, additional or alternative communication methods such as dedicated short-range communication (DSRC) channels are also considered to be within the scope of the present disclosure. A DSRC channel refers to a one-way or two-way short-range to medium-range wireless communication channel designed specifically for automotive use and a set of corresponding protocols and standards.

[0040] The vehicle controller 34 is a non-general purpose electronic control device having a pre-programmed digital computer or processor, memory or non-transitory computer readable medium, etc., for storing data such as control logic, software applications, instructions, computer code, data, look-up tables, and the like, and a transceiver [or input / output port]. Computer readable media include any type of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard drive, compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable media excludes wired, wireless, optical or other communication links that transmit temporary electrical or other signals. Non-transitory computer readable media include media that can store data permanently and media that can store data and then be rewritten, such as rewritable optical disks or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code.

[0041] refer to Figure 2 , a schematic diagram of the system 11 is shown. The system 11 includes a system controller 34A in communication with a plurality of sensing devices (onboard sensors) 40a-40n. The system controller 34A may be the vehicle controller 34, or the system controller 34A may be a separate controller in communication with the vehicle controller 34. In addition to the plurality of onboard sensors 40a-40n, the system controller 34A also communicates with a human machine interface (HMI) 50, a vehicle control module (VCM) 52, a driver monitoring system 54, and a wireless communication module 36.

[0042] The driver monitoring system 54 is adapted to monitor the driver 56 of the vehicle 10 using a camera 58 directed toward the eyes and face of the driver 56. The system controller 34A is in communication with the driver monitoring system 54 and is adapted to collect data related to the driver 56 gaze behavior from the driver monitoring system 54 and to classify the driver 56 into one of a plurality of driver attention states based on the data from the driver monitoring system 54. In an exemplary embodiment, the driver attention states include "safe," "marginal," and "unsafe."

[0043] When classifying a driver 56 having one of multiple driver attention states based on data from the driver monitoring system 54, the system controller 34A is also suitable for creating a gaze model based on data collected by the driver monitoring system 54; classifying the gaze model as one of "safe", "critical" or "unsafe"; generating a classification probability for the gaze model; and classifying the driver 56 as one of "safe", "critical" or "unsafe".

[0044] The gaze model is a probability distribution of the observed gaze behavior of the driver 56 of the vehicle 10. In an exemplary embodiment, the gaze model is based on the normalized neural network output. In an exemplary embodiment, the "safe" gaze model is a probability distribution of gaze behavior, indicating that the driver 56 of the vehicle 10 is paying close attention.

[0045] The “critical” gaze model is a probability distribution of gaze behavior indicating that the driver 56 of the vehicle 10 is paying attention but not paying close attention, such as when the vehicle 10 is in a lower level of automated driving mode (i.e., Level 1, 2), and when the driver 56 is paying attention, the driver 56 only pays basic attention to the driving characteristics of the vehicle 10 or non-autonomous driving vehicle, but may be distracted by other activities or things that attract the driver's attention.

[0046] The "unsafe" gaze model is a probability distribution of gaze behaviors that indicate that the driver 56 of the vehicle 10 is not paying attention to an unsafe point. It should be appreciated that there may be any suitable number of predefined gaze models, and the exemplary embodiment described herein is intended to illustrate a non-limiting example.

[0047] The system controller 34A is adapted to classify the gaze model as one of "safe", "critical", or "unsafe" by matching the gaze model with one of the "safe", "critical", or "unsafe" gaze models. Using a neural network loss function, the gaze model is compared with the "safe", "critical", and "unsafe" gaze models, and the gaze model is classified as the model that best matches one of the "safe", "critical", and "unsafe" gazes.

[0048] The driver monitoring system 54 also uses physiological signals to classify the driver's attention state. The most accurate driver monitoring methods use physiological signals, such as brain waves and heart rate, pulse rate and breathing rate signals. Although electroencephalograms (EEGs) that monitor brain waves provide the best physiological signals for assessing the driver's attention state, such data is difficult to obtain while driving. Another biosignal is an electrocardiogram (ECG) signal with heart rate and heart rate variability information. Other methods of collecting physiological data include collecting biosignals such as respiration (via sensors located in the seat and / or seat belt), grip strength and photoplethysmography (PPG) (via sensors on the steering wheel), and muscle current (electromyography (EMG)). Such biosignals can also be received from wearable devices such as smart watches worn by the driver. Various techniques are used to extract meaningful features from the sensor readings, including time series analysis, frequency domain analysis, and spatiotemporal patterns. Different types of machine learning algorithms can be used to classify the driver's attention state, including but not limited to long short-term memory (LSTM), gated recurrent unit (GRU), decision tree, random forest, support vector machine (SVM), neural network (NN), K nearest neighbor (KNN), gradient boosting, and recurrent neural network (RNN). Therefore, the system controller 34A uses the machine learning algorithm 60 and machine learning techniques to predict inattention or medical distress, and therefore predict an "unsafe" driving state.

[0049] When the system controller 34A classifies the driver 56 as one of "safe", "critical" or "unsafe" based on the data of the driver monitoring system 54, the system controller 34A is also adapted to calculate the probability that the driver's attention state is "safe", "critical" or "unsafe". When the probability exceeds the threshold of each driver's attention state, the driver's attention state is classified as that state. Specifically, when the probability that the driver's attention state is "unsafe" exceeds a predetermined threshold (e.g., 50%), the system controller 34A classifies the driver's attention state as "unsafe".

[0050] The system controller 34A collects data related to various aspects of the current driving session from various onboard systems within the vehicle 10. This data is collected in real time by the onboard sensors 40a-40n and the driver monitoring system 54 to collect real-time data related to the driver's current state of attention. The driver's attention history during the current driving session is represented as:

[0051] AH c =[session_id,timestamp,driver_id,P(AS1),P(AS2),…,P(AS n ),Reward / Penalty].

[0052] The timestamps are used to determine the frequency, consistency, and severity of different attention states during driving. For example, during a long drive, a driver may not fully pay attention to the (unsafe) right rear end.

[0053] The system controller 34A also accesses a database 62 of historical data related to the driver state classification of the driver 56 during past driving sessions. The database 62 may be located on the vehicle 10, or may be cloud-based. Each time the vehicle 10 is driven, the system controller 34A identifies the driver 56, collects data related to the driving attention state of the driver 56, and stores the data in the database 62. Then, during any current driving session, the system controller 34A may access the historical data during the past driving sessions of the driver 56 to compare how the driver 56 behaved during the current driving session with how the driver 56 behaved during past driving sessions. The attention history of past driving sessions is represented as:

[0054] AH P =[session_id,timestamp,driver_id,P(AS1),P(AS2),…,P(AS n ), Reward / Penalty], where

[0055] The probabilities and reward / penalty values ​​are calculated statistically (averaged) over the entire driving process.

[0056] The system controller 34A then uses the machine learning algorithm 60 to predict the probability of the driver state being "safe", "marginal", or "unsafe" based on the real-time data and historical data, where:

[0057] P(AS safe )+P(AS bordeline )+P(AS unsafe )=1;

[0058] P(AS safe ) is the probability that the driver's 56 attention state is "safe";

[0059] P(AS borderline ) is the probability that the driver's 56 attention state is "critical"; and

[0060] P(AS unsafe ) is the probability that the driver's 56 attention state is "unsafe".

[0061] It will be appreciated that there may be any number of driver attention states, n, where:

[0062] P(AS1)+P(AS2)+…P(AS n)=1, where each of the n attention states is classified as one of “safe”, “critical” or “unsafe”, where:

[0063] P(AS safe )=∑ j k ＝0 P(AS j ),k <n;

[0064] P(AS borderline )=∑ j k ＝0 P(AS j ),k <n;

[0065] P(AS unsafe )=∑ j k ＝0 P(AS j ),k <n。

[0066] The probability that the driver's attention state is unsafe is related to the driver's 56 gaze behavior and physiological signals from the driver 56 monitored by the driver monitoring system 54 and the predicted probability based on past driving sessions. For example, access to past driving processes can reveal that in many cases, the driver 56 exhibits unsafe driving behavior after driving for more than three hours without interruption. Therefore, if the current driving session has lasted for more than three hours, the system controller 34A will use machine learning to predict that the driver's 56 attention will decrease and place a higher probability of the driving attention state as "unsafe".

[0067] When the driver's attention state is unsafe, the predicted probability P(AS unsafe ) exceeds a predetermined threshold, the system controller 34A classifies the driver state as “unsafe”, and when the driver attention state is classified as “unsafe”, the system controller 34A classifies the driver attention state as “unsafe”. The system controller 34A further asks the driver 56 questions to test the alertness of the driver 56 through the HMI 50, receives the driver 56's responses to the questions through the HMI 50, analyzes the responses to the questions, and initiates response actions according to the driver 56's responses.

[0068] In an exemplary embodiment, when questions are asked to the driver 56 through the HMI 50 to test the driver's 56 alertness, the system controller 34A is also adapted to ask the driver 56 questions related to at least one of the driver's 56 situational awareness and the driver's 56 topics of interest. The situational awareness questions may be, for example, "How many lanes are there on the road?", "What color is the adjacent vehicle on your right?", "What is your current speed?". Questions related to topics of interest to the driver 56 may include questions related to movies, music, politics, etc., wherein the driver's interest in these topics is obtained through direct input from the driver 56 or by accessing the driver's 56 personal device and / or social media account using the system controller 34A. These questions may be open-ended questions, specific questions, multiple choice questions, or yes / no questions.

[0069] In an exemplary embodiment, the system controller 34A will audibly ask the driver 56 questions through a speaker 64 connected to the HMI 50, wherein the driver 56 can provide verbal responses to the questions picked up by a microphone 66 connected to the HMI 50. The system controller 34A will process the responses using natural language understanding (NLU) technology. In another exemplary embodiment, the system controller 34A will visually ask the driver 56 questions by displaying the questions on a touch screen display 68 associated with the HMI 50, wherein the driver 56 can provide verbal responses or can enter responses through the touch screen display 68. For example, the system controller 34A can provide questions audibly or through the HMI 50 touch screen display 68, or both verbally and through the touch screen display 68, thereby giving the driver 56 options on how to provide responses. In another example, if the system controller 34A identifies that the driver 56 is hearing impaired based on accessing driver attributes provided in advance, the system controller 34A will provide questions to receive responses through the touch screen display 68.

[0070] In an exemplary embodiment, when analyzing the responses, the system controller 34A is further adapted to evaluate whether the responses are correct, evaluate the driver's understanding of the responses, evaluate the driver's speed of responding to the responses, and score the driver as "pass" or "fail" based on the driver's evaluation of their answers to the responses.

[0071] The evaluation of the answers is not necessarily dominated by correct and incorrect answers, for example, when evaluating the answers to determine whether the driver state is "unsafe", answers that are not closely related to the question are worse than incorrect answers. For example, if the vehicle 10 is traveling on a road with four lanes and the question is "How many lanes are there on the road you are traveling on?", the response "I am traveling on I-65 southbound" will be more indicative of driver 56's inattention than the response "There are three lanes", which, while incorrect, indicates that driver 56 at least understood the question correctly. In addition, a correct response to a question given several minutes after the question was prompted indicates that driver 56 may have been inattentive even if the response was correct, resulting in driver 56 receiving a score of "failed".

[0072] In an exemplary embodiment, when the system controller 34A scores the driver 56 as "passing", the system controller 34A is further adapted to take responsive actions including providing a reward to the driver 56. For example, in response to a "passing" score, the system controller 34A may provide an audible or displayed message providing positive feedback to the driver 56. In another example, in response to a "passing" score, the system controller 34A, by communicating with a remote system 70 supported by a third party (e.g., the manufacturer of the vehicle 10), may provide a points system for rewards to the driver 56, where the points may be applied to vehicle maintenance services, charging credits (in the case of an electric vehicle), or other products / services related to the vehicle 10.

[0073] In another exemplary embodiment, when the system controller 34A scores the driver 56 as "failed", the system controller 34A is further adapted to take responsive actions including penalizing the driver 56. For example, when penalizing the driver 56, the system controller 34A is adapted to at least one of the following: (1) shut down the advanced driver assistance function for a period of time by communicating with the vehicle system in the vehicle 10; (2) prohibit the driver from using the advanced driver assistance function for a period of time by communicating with the vehicle system in the vehicle 10; (3) send an alert to an emergency service provider 72, such as the police or an ambulance (such as in the case of medical distress) through the wireless communication module 36; (4) send an alert to an emergency contact 74 through the wireless communication module 36, and the content of the alert can be transmitted to the emergency contact 74 by the system controller 34A. communications with the driver's 56 personal device, or provided in advance by the driver 56 directly to the system controller 34A; (5) in the case where the driver 56 is a minor, sending an alert to the parent / guardian 76 via the wireless communication module 36, wherein preferences for when and why such alerts are sent are set in advance by the parent / guardian 76 within the system controller 34A; and (6) in the case of an autonomous vehicle 10, enabling an autonomous driving mode through its communication with the vehicle control module 52, wherein the vehicle control module 52 takes over primary control of the vehicle 10 from the driver 56.

[0074] In another exemplary embodiment, when penalizing the driver 56, the system controller 34A is further adapted to access the historical data database 62 associated with the driver state classification of the driver 56 during the current driving session and identify the instance pattern in which the driver 56 was classified as unsafe during the current driving session; and access the historical data database 62 associated with the driver state classification of the driver 56 during the past driving sessions and identify the instance pattern in which the driver 56 was classified as unsafe during the past driving sessions. If the system controller 34A identifies a pattern in which the driver state is consistently and repeatedly classified as "unsafe" during the current and / or past driving sessions, the system controller 34A will escalate the penalty for the driver 56. In another exemplary embodiment, when the driver 56 is using the vehicle 10, the database 62 can be accessed to obtain learned driver behavior models from other vehicles used by the driver 56, or the driver behavior models can be uploaded from the vehicle 10 to the database 62, where the other vehicles used by the driver 56 can access these driver behavior models.

[0075] For example, if the driver state is classified as "unsafe" and the driver 56 is scored as "failed" after responding to the questions posed by the system controller 34A, the system controller 34A may impose a penalty by turning off the advanced driver assistance features or prohibiting the driver from using the advanced driver assistance features for a period of time. However, if the system controller 34A identifies a pattern in which the driver state is consistently and repeatedly classified as "unsafe" during the current and / or past driving sessions, the system controller 34A may escalate the penalty imposed on the driver 56 and, in addition to turning off the advanced driver assistance features or prohibiting the driver from using such advanced driver assistance features for a period of time, the system controller 34A may also send an alert to the emergency service provider 72, the emergency contact 74, or the parent / guardian 76. In extreme cases, the penalty may be escalated to the point where, in the case of an autonomous vehicle, the autonomous driving mode is enabled and control of the vehicle 10 is taken away from the driver 56.

[0076] refer to Figure 4 , a method 100 for monitoring a driver 56 in a vehicle 10 includes, beginning at box 102, monitoring the driver 56 of the vehicle 10 using a driver monitoring system 54; moving to box 104, collecting data related to the gaze behavior of the driver 56 from the driver monitoring system 54 using a system controller 34A in communication with the driver monitoring system 54; moving to box 106, using the system controller 34A to classify the driver 56 into one of a plurality of driver attention states based on the data from the driver monitoring system 54, the driver attention states including safe, critical, and unsafe; and at box 108, when the driver attention state is unsafe, moving to box 110, using the system controller to ask the driver 56 questions to test the alertness of the driver 56 through a human-machine interface (HMI) 50; moving to box 112, using the system controller 34A to receive a response to the question from the driver 56 through the HMI 50; moving to box 114, analyzing the response with the system controller 34A; and moving to box 116, using the system controller 34A to initiate a response action based on the response of the driver 56. At box 108 , if the driver attention state is not unsafe, the method returns to box 102 as shown at line 118 .

[0077] In an exemplary embodiment, asking the driver 56 questions to test the alertness of the driver 56 via the human machine interface (HMI) 50 using the system controller 34A at box 110 also includes asking the driver 56 questions related to at least one of the situational awareness of the driver 56 and topics of interest to the driver 56.

[0078] In another exemplary embodiment, using the system controller 34A at box 106 to classify the driver 56 into one of multiple driver attention states based on data from the driver monitoring system 54 also includes moving to box 120 to calculate the probability that the driver attention state is unsafe; and at box 122, when the probability that the driver state is unsafe exceeds a predetermined threshold, moving to box 124 to classify the driver attention state as unsafe.

[0079] In another exemplary embodiment, calculating the probability that the driver attention state is unsafe using the system controller at box 120 also includes moving to box 126, collecting data related to aspects of the current driving session from an onboard system within the vehicle 10; moving to box 128, accessing a historical data database 62 related to classifications of the driver attention state of the driver 56 during past driving sessions; and moving to box 130, using the machine learning algorithm 60 to predict the probability that the driver attention state is unsafe based on the historical data.

[0080] In another exemplary embodiment, analyzing the response using the system controller 34A at box 114 also includes moving to box 132 to evaluate whether the response is correct; moving to box 134 to evaluate the driver's understanding of the question; moving to box 136 to evaluate the driver's response speed to the question; and moving to box 138 to score the driver 56 as "pass" or "fail" based on the evaluation of the driver's response to the question.

[0081] In another exemplary embodiment, moving to block 140 , if the system controller 34A scores the driver 56 as “passing,” initiating a response action based on the driver 56 response using the system controller 34A at block 116 also includes moving to block 142 and providing a reward to the driver 56 .

[0082] In yet another exemplary embodiment, again moving to box 140 , if the system controller 34A scores the driver 56 as “failed,” initiating a response action based on the driver 56 response using the system controller 34A at box 116 also includes moving to box 144 to penalize the driver 56 .

[0083] In another exemplary embodiment, penalizing the driver 56 at box 144 includes at least one of the following: moving to box 146, using the system controller 34A, through its communication with the onboard system in the vehicle 10, to turn off the advanced driver assistance function for a period of time; moving to box 148, using the system controller, through its communication with the onboard system in the vehicle, to prohibit the driver from using the advanced driver assistance function for a period of time; moving to box 150, sending an alert to the emergency service provider 72 through the wireless communication module 36; moving to box 152, sending an alert to the emergency contact 74 through the wireless communication module 36; moving to box 154, sending an alert to the parent / guardian 76 through the wireless communication module; and moving to box 156, using the system controller 34A, through its communication with the vehicle control module 52, to enable the autonomous driving mode.

[0084] In another exemplary embodiment, initiating a response action based on a response of the driver 56 using the system controller 34A at box 116 further includes moving to box 140 again, when the system controller 34A scores the driver as "failed", moving to box 158, accessing the historical data database 62 associated with the driver state classification of the driver 56 during the current driving session, and identifying an instance model in which the driver 56 was classified as unsafe during the current driving session; moving to box 160, accessing the historical data database 62 associated with the driver attention state classification of the driver 56 during past driving sessions, and identifying an instance model in which the driver 56 was classified as unsafe during the past driving session; and moving to box 162, when based on the identified instance model in which the driver 56 was classified as unsafe during at least one of the current driving session and the past driving session, upgrading the penalty for the driver 56.

[0085] The system 11 and method 100 of the present disclosure have the advantage of verifying the inferred driver state by prompting the driver with questions and analyzing the responses. This provides a more detailed, accurate and customized way to monitor and predict the driver state.

[0086] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the scope and scope of the present disclosure.

Claims

1. A method for monitoring a driver in a vehicle, comprising: monitoring the driver of a vehicle using a driver monitoring system; collecting data related to the driver's gaze behavior from the driver monitoring system using a system controller in communication with the driver monitoring system; using the system controller to classify the driver into one of a plurality of driver attention states based on the data from the driver monitoring system, the driver attention states including safe, marginal, and unsafe; and When the driver status is unsafe: using the system controller to ask the driver questions through a human machine interface (HMI) to test driver alertness; receiving, using the system controller, via the HMI, a response from the driver to the question; analyzing the response using the system controller; as well as A response action is initiated using the system controller based on the driver's response.

2. The method according to claim 1, wherein: Proposing questions to the driver to test driver alertness using the system controller via a human machine interface (HMI) further includes proposing questions to the driver related to at least one of the driver's situational awareness and a topic of interest to the driver.

3. The method according to claim 2, wherein: Using the system controller to classify the driver into one of a plurality of driver attention states based on the data from the driver monitoring system further includes calculating a probability that the driver attention state is unsafe, and classifying the driver attention state as unsafe when the probability that the driver attention state is unsafe exceeds a predetermined threshold.

4. The method according to claim 3, wherein: Calculating the probability that the driver's attention state is unsafe also includes: collecting data related to aspects of a current driving session from an onboard system within the vehicle; accessing a database of historical data relating to driver attention state classifications of the driver during past driving sessions; and A machine learning algorithm is used to predict the probability that the driver's attention state is unsafe based on the historical data.

5. The method according to claim 4, wherein: Analyzing the response using the system controller further comprises: evaluating whether the response is correct; Assessing the driver's understanding of the problem; evaluating the driver's response speed to the question; and Based on the evaluation of the driver's responses to the questions, the driver is scored as "pass" or "fail".

6. The method according to claim 5, wherein: When the system controller scores the driver as "pass", initiating a response action based on the driver's response using the system controller further includes providing a reward to the driver.

7. The method according to claim 6, wherein: When the system controller scores the driver as "failed", initiating a response action based on the driver's response using the system controller also includes penalizing the driver.

8. The method according to claim 7, wherein: The penalty imposed on the driver includes at least one of the following: Using the system controller, through its communication with an onboard system in the vehicle, to disable an advanced driver assistance function for a period of time; Using the system controller, through communication with an onboard system in the vehicle, to prohibit the driver from using an advanced driver assistance function for a period of time; sending an alert to an emergency service provider via a wireless communications module; sending an alert to an emergency contact via the wireless communication module; sending an alert to a parent / guardian via the wireless communication module; and The system controller is used to enable the autonomous driving mode through its communication with the vehicle control module.

9. The method according to claim 8, wherein: Initiating a response action based on the driver's response using the system controller also includes, when the system controller scores the driver as "failed": accessing the historical data database associated with the driver attention state classification of the driver during a current driving session and identifying an instance pattern in which the driver was classified as unsafe during the current driving session; accessing the historical data database associated with the driver attention state classifications of the driver during past driving sessions and identifying instance patterns in which the driver was classified as unsafe during the past driving sessions; as well as A penalty for the driver is escalated based on the identified instance pattern of the driver being classified as unsafe during at least one of a current driving session and a past driving session.

10. A system for monitoring a driver in a vehicle, comprising: a driver monitoring system adapted to monitor the driver of the vehicle; as well as a system controller in communication with the driver monitoring system and adapted to: collecting data related to the driver's gaze behavior from the driver monitoring system; classifying the driver into one of a plurality of driver attention states based on the data from the driver monitoring system, the driver attention states including safe, marginal, and unsafe; as well as When the driver status is unsafe: asking the driver questions to test driver alertness via a human machine interface (HMI); receiving, via the HMI, a response from the driver to the question; analyzing the response; as well as A response action is initiated based on the driver's response.

Images