Predicting driver state using gaze behavior

Abstract

A method of monitoring a driver in an autonomous vehicle includes monitoring a driver of a vehicle with a driver monitoring system, collecting, with a data processor, data related to gaze behavior of the driver from the driver monitoring system, classifying, with the data processor, the driver into one of a plurality of driver states based on data from the driver monitoring system, and sending, with the data processor, an instruction to at least one vehicle system based on the classification of the driver.

Description

Predicting driver status using gaze behavior Technical Field

[0001] This disclosure relates to a system for predicting the driver's state in a vehicle based on the driver's gaze behavior. Background Technology

[0002] Driver monitoring systems typically use driver-facing cameras equipped with infrared LEDs or lasers, allowing the camera to "see" the driver's face even at night and to see the driver's eyes even when the driver is wearing dark sunglasses. Advanced in-vehicle software collects data points from the driver and creates an initial baseline of what a normal, attentive driver looks like. The software can then determine if the driver is blinking more than usual, squinting or closing their eyes, and if their head is tilted at an unusual angle. The software can also determine if the driver is looking at the road ahead and whether they are truly focused or just staring absentmindedly.

[0003] If the system determines that the driver is distracted or drowsy, it can regain the driver's attention by issuing an audio alert, illuminating a visual indicator on the dashboard, or vibrating the seat. If internal sensors indicate driver distraction and external sensors determine that a collision is imminent, the system can automatically apply the brakes using information fused from both internal and external sensors.

[0004] In autonomous vehicles, one goal of the vehicle control module is to minimize the number of driver takeover events. A driver takeover event occurs when the driver within the autonomous vehicle takes over manual control of the vehicle while it is operating in autonomous mode. A takeover event can occur when the driver simply decides to take over manual control. It can also occur during autonomous operation when the driver becomes uncomfortable and feels the need to take over manual control. This could be due to the vehicle following too closely behind another vehicle, or the vehicle potentially traveling at a speed that the driver perceives as too fast under current weather conditions.

[0005] The vehicle control module determines how to control the vehicle based on a behavioral model, which tells the module how to control the vehicle in various situations and conditions. Since each individual driver has different preferences, it's impossible for the behavioral model to match the preferences of every possible driver.

[0006] Therefore, while current driver monitoring systems achieve their intended purpose, there is a need for new and improved systems and methods to predict driver / passenger discomfort within autonomous vehicles based on gaze behavior observed by the driver monitoring system. This would allow the vehicle control module to modify the vehicle's driving behavior, thereby reducing the number of takeover events due to driver discomfort. Furthermore, there is a need for new and improved systems and methods that update the vehicle control module's behavioral model whenever a takeover event occurs, allowing the system to learn the tendencies of a particular driver. Summary of the Invention

[0007] According to various aspects of this disclosure, a method for monitoring a driver in an autonomous vehicle includes: monitoring the driver of the vehicle using a driver monitoring system; collecting data related to the driver's gaze behavior from the driver monitoring system using a data processor; classifying the driver into one of a plurality of driver states using the data processor based on the data from the driver monitoring system; and sending instructions to at least one vehicle system using the data processor based on the driver's classification.

[0008] According to another aspect, classifying a driver into one of multiple driver states based on data from a driver monitoring system also includes: creating an actual gaze model based on data collected by the driver monitoring system, classifying the actual gaze model into one of multiple predefined gaze models, and classifying the driver into one of multiple driver states based on the classification of the actual gaze model and the vehicle's operating status.

[0009] According to another aspect, multiple driver states include a first driver state, a second driver state, and a third driver state, and multiple predefined gaze models include a first gaze model, a second gaze model, and a third gaze model.

[0010] According to another aspect, classifying the driver into one of multiple driver states based on the classification of the actual gaze model and the vehicle's operating state further includes: classifying the driver into a first driver state when the actual gaze model is classified as a first predefined gaze model and the vehicle's operating state is manual mode; and

[0011] When the actual gaze model is classified as the first predefined gaze model and the vehicle is in autonomous mode, the driver is classified as the second driver state.

[0012] According to another aspect, classifying a driver into one of multiple driver states based on the classification of the actual gaze model and the vehicle's operating state further includes: classifying the driver into a third driver state when the actual gaze model is classified as a second predefined gaze model and the vehicle operating state is manual mode; classifying the driver into a first driver state when the actual gaze model is classified as a second predefined gaze model, the vehicle operating state is autonomous mode, and the vehicle is one of Level 1 autonomous vehicles or Level 2 autonomous vehicles; and classifying the driver into a second driver state when the actual gaze model is classified as a second predefined gaze model, the vehicle operating state is autonomous mode, and the vehicle is one of Level 3 autonomous vehicles, Level 4 autonomous vehicles, or Level 5 autonomous vehicles.

[0013] According to another aspect, classifying the driver into one of multiple driver states based on the classification of the actual gaze model and the vehicle's operating state further includes: classifying the driver into a third driver state when the actual gaze model is classified as a third predefined gaze model and the vehicle's operating state is manual mode; classifying the driver into a third driver state when the actual gaze model is classified as a third predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is one of a Level 1 autonomous vehicle or a Level 2 autonomous vehicle; and classifying the driver into a first driver state when the actual gaze model is classified as a third predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is one of a Level 3 autonomous vehicle, a Level 4 autonomous vehicle, or a Level 5 autonomous vehicle.

[0014] According to another aspect, the method of sending instructions to at least one vehicle system based on driver classification using a data processor also includes: when the driver of the vehicle is classified as a first driver state, sending instructions to the vehicle control module using the data processor to maintain the current operating parameters of the vehicle.

[0015] According to another aspect, the method of sending instructions to at least one vehicle system based on driver classification using a data processor also includes: when the driver of the vehicle is classified into one of a second driver state and a third driver state, sending instructions to a vehicle control module using the data processor to change the current operating parameters of the vehicle.

[0016] According to another aspect, the second driver state indicates that the driver of the vehicle is uncomfortable and may take over the manual control of the vehicle, and the instructions sent to the vehicle control module are adapted to change the current operating parameters of the vehicle in order to make the driver of the vehicle comfortable and reduce the possibility that the driver of the vehicle will take over the manual control.

[0017] According to another aspect, the third driver state indicates the driver's distraction and the instructions sent to the vehicle control module are adapted to change the priority of warning prompts that may be provided to the driver.

[0018] According to another aspect, the method also includes updating the plurality of predefined gaze models.

[0019] According to another aspect, classifying the actual gaze model into one of a plurality of predefined gaze models further includes: generating a confidence score for the classification of the actual gaze model using the data processor.

[0020] According to various aspects of this disclosure, a system for monitoring a driver in an autonomous vehicle includes: a driver monitoring system adapted to collect data related to the driver's gaze behavior; a data processor adapted to receive data from the driver monitoring system and classify the driver into one of a first driver state, a second driver state, and a third driver state based on the data from the driver monitoring system; and at least one vehicle system, wherein the data processor is further adapted to send instructions to the at least one vehicle system based on the driver classification.

[0021] Other application areas will become apparent from the description provided herein. It should be understood that the description and specific examples are intended to illustrate, and not to limit, the scope of this disclosure.

[0022] Option 1. A method for monitoring a driver in an autonomous vehicle, the method comprising:

[0023] Monitor vehicle drivers using a driver monitoring system;

[0024] Data related to the driver's gaze behavior is collected from the driver monitoring system using a data processor;

[0025] The data processor uses data from the driver monitoring system to classify the driver into one of multiple driver states; and

[0026] The data processor sends instructions to at least one vehicle system based on the driver's classification.

[0027] Option 2. The method according to Option 1, wherein classifying the driver into one of multiple driver states based on data from the driver monitoring system further includes:

[0028] A realistic gaze model is created based on data collected by the driver monitoring system.

[0029] The actual gaze model is classified into one of a number of predefined gaze models;

[0030] Based on the classification of the actual gaze model and the vehicle's operating state, the driver is classified into one of multiple driver states.

[0031] Option 3. According to the method in Option 2, wherein the plurality of driver states include a first driver state, a second driver state, and a third driver state, and the plurality of predefined gaze models include a first gaze model, a second gaze model, and a third gaze model.

[0032] Solution 4. The method according to Solution 3, wherein classifying the driver into one of the multiple driver states based on the classification of the actual gaze model and the vehicle's operating state further includes:

[0033] When the actual gaze model is classified as the first predefined gaze model and the vehicle operating state is manual mode, the driver is classified as the first driver state; and

[0034] When the actual gaze model is classified as the first predefined gaze model and the vehicle operating state is autonomous mode, the driver is classified as the second driver state.

[0035] Solution 5. The method according to Solution 4, wherein classifying the driver into one of the multiple driver states based on the classification of the actual gaze model and the vehicle's operating state further includes:

[0036] When the actual gaze model is classified as the second predefined gaze model and the vehicle operating state is manual mode, the driver is classified as the third driver state.

[0037] When the actual gaze model is classified as the second predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is either a Level 1 autonomous vehicle or a Level 2 autonomous vehicle, the driver is classified as the first driver state; and

[0038] When the actual gaze model is classified as a second predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is one of the following autonomous vehicles: Level 3 autonomous vehicle, Level 4 autonomous vehicle, and Level 5 autonomous vehicle, the driver is classified as the second driver state.

[0039] Solution 6. The method according to Solution 5, wherein classifying the driver into one of the multiple driver states based on the classification of the actual gaze model and the vehicle's operating state further includes:

[0040] When the actual gaze model is classified as the third predefined gaze model and the vehicle operating state is manual mode, the driver is classified as the third driver state.

[0041] When the actual gaze model is classified as the third predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is either a Level 1 autonomous vehicle or a Level 2 autonomous vehicle, the driver is classified as the third driver state; and

[0042] When the actual gaze model is classified as a third predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is one of the following autonomous vehicles: Level 3 autonomous vehicle, Level 4 autonomous vehicle, and Level 5 autonomous vehicle, the driver is classified as the first driver state.

[0043] Option 7. The method according to Option 6, wherein sending instructions to at least one vehicle system based on the driver classification using the data processor further includes:

[0044] When the driver of the vehicle is classified as the first driver state, the data processor sends instructions to the vehicle control module to maintain the current operating parameters of the vehicle.

[0045] Option 8. The method according to Option 7, wherein sending instructions to at least one vehicle system based on the driver classification using the data processor further includes:

[0046] When the driver of the vehicle is classified into one of the second driver state and the third driver state, the data processor sends an instruction to the vehicle control module to change the current operating parameters of the vehicle.

[0047] Option 9. The method according to Option 8, wherein the second driver state indicates that the driver of the vehicle is uncomfortable and may take over the manual control of the vehicle, and the instruction sent to the vehicle control module is adapted to change the current operating parameters of the vehicle in order to make the driver of the vehicle comfortable and reduce the likelihood that the driver of the vehicle will take over the manual control.

[0048] Option 10. The method according to Option 8, wherein the third driver state indicates that the driver of the vehicle is distracted, and the instruction sent to the vehicle control module is adapted to change the priority of warning prompts that may be provided to the driver.

[0049] Option 11. The method according to Option 8, wherein the method further comprises: updating the plurality of predefined gaze models whenever the driver of the vehicle takes over manual control of the vehicle.

[0050] Option 12. The method according to Option 11, wherein classifying the actual gaze model into one of a plurality of predefined gaze models further includes: generating a confidence score for the classification of the actual gaze model using the data processor.

[0051] Option 13. A system for monitoring a driver in an autonomous vehicle, the system comprising:

[0052] A driver monitoring system, the driver monitoring system being adapted to collect data related to the driver's gaze behavior;

[0053] A data processor adapted to receive data from the driver monitoring system and classify the driver into one of a plurality of driver states based on the data from the driver monitoring system; and

[0054] At least one vehicle system, the data processor is further adapted to send instructions to the at least one vehicle system based on the driver's classification.

[0055] Option 14. The system according to Option 13, wherein when the driver is classified into one of a plurality of driver states based on data from the driver monitoring system, the data processor is further adapted to create an actual gaze model based on data collected by the driver monitoring system, classify the actual gaze model into one of a first predefined gaze model, a second predefined gaze model, and a third predefined gaze model, generate a confidence score for the classification of the actual gaze model, and classify the driver into one of the first driver state, the second driver state, and the third driver state based on the classification of the actual gaze model and the vehicle's operating state.

[0056] Option 15. The system according to Option 14, wherein when the driver is classified into one of the plurality of driver states based on the classification of the actual gaze model and the vehicle's operating state, the data processor is further adapted to:

[0057] When the actual gaze model is classified as the first predefined gaze model and the vehicle operating state is manual mode, the driver is classified as the first driver state; and

[0058] When the actual gaze model is classified as the first predefined gaze model and the vehicle operating state is autonomous mode, the driver is classified as the second driver state.

[0059] Option 16. The method according to Option 15, wherein when the driver is classified into one of the plurality of driver states based on the classification of the actual gaze model and the vehicle's operating state, the data processor is further adapted to:

[0060] When the actual gaze model is classified as the second predefined gaze model and the vehicle operating state is manual mode, the driver is classified as the third driver state.

[0061] When the actual gaze model is classified as the second predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is either a Level 1 autonomous vehicle or a Level 2 autonomous vehicle, the driver is classified as the first driver state; and

[0062] When the actual gaze model is classified as a second predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is one of the following autonomous vehicles: Level 3 autonomous vehicle, Level 4 autonomous vehicle, and Level 5 autonomous vehicle, the driver is classified as the second driver state.

[0063] Solution 17. The system according to Solution 16, wherein when the driver is classified into one of the plurality of driver states based on the classification of the actual gaze model and the vehicle's operating state, the data processor is further adapted to:

[0064] When the actual gaze model is classified as the third predefined gaze model and the vehicle operating state is manual mode, the driver is classified as the third driver state.

[0065] When the actual gaze model is classified as the third predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is either a Level 1 autonomous vehicle or a Level 2 autonomous vehicle, the driver is classified as the third driver state; and

[0066] When the actual gaze model is classified as a third predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is one of the following autonomous vehicles: Level 3 autonomous vehicle, Level 4 autonomous vehicle, and Level 5 autonomous vehicle, the driver is classified as the first driver state.

[0067] Option 18. The system according to Option 17, wherein the system further includes a vehicle control module, and the data processor is further adapted to:

[0068] When the driver of the vehicle is classified as the first driver state, an instruction is sent to the vehicle control module to maintain the current operating parameters of the vehicle; and

[0069] When the driver of the vehicle is classified into one of the second driver state and the third driver state, an instruction is sent to the vehicle control module to change the current operating parameters of the vehicle.

[0070] Option 19. The system according to Option 18, wherein the second driver state indicates that the driver of the vehicle is uncomfortable and may take over manual control of the vehicle, and the data processor is adapted to send instructions to the vehicle control module to change the current operating parameters of the vehicle in order to make the driver of the vehicle comfortable and reduce the likelihood that the driver of the vehicle will take over manual control, and the third driver state indicates that the driver of the vehicle is distracted, and the data processor is adapted to send instructions to the vehicle control module to change the priority of warning prompts that may be provided to the driver.

[0071] Option 20. The system according to Option 19, wherein the data processor is further adapted to update the first predefined gaze model, the second predefined gaze model, and the third predefined gaze model whenever the driver of the vehicle takes over manual control of the vehicle. Attached Figure Description

[0072] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0073] Figure 1 is a schematic diagram of a system for monitoring the driver of an autonomous vehicle according to an exemplary embodiment of the present disclosure;

[0074] Figure 2 is a schematic diagram of a two-dimensional representation of the actual gaze distribution model compared with several predefined gaze distribution models;

[0075] Figure 3 is a schematic diagram of an alternative two-dimensional representation of the actual gaze distribution model compared with several predefined gaze distribution models;

[0076] Figure 4 is a flowchart illustrating a method according to an exemplary embodiment of the present disclosure; and

[0077] Figure 5 is a flowchart showing the details of box 106 in Figure 4.

[0078] The accompanying drawings are not necessarily to scale, and some features may be enlarged or reduced, for example, to show details of specific components. In some cases, well-known components, systems, materials, or methods are not described in detail so as not to affect the clarity of this disclosure. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art how to utilize this disclosure in various ways. Detailed Implementation

[0079] The following description is exemplary in nature and is not intended to limit this disclosure, application, or use. Furthermore, it is not intended to be bound by any express or implied theory set forth in the foregoing technical field, background art, summary of the invention, or the following detailed description. It should be understood that in all the drawings, corresponding reference numerals denote similar or corresponding parts and features. As used herein, the terms “module” or “controller” refer to any hardware, software, firmware, electronic control components, processing logic, and / or processor device, individually or in any combination including, but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped), and memory executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components providing the described functionality. Although the drawings shown herein depict examples with certain element arrangements, additional intermediate elements, devices, features, or components may be present in actual implementations. It should also be understood that the drawings are merely illustrative and may not be drawn to scale.

[0080] As used herein, the term "vehicle" is not limited to automobiles. Although this document primarily describes the technology in conjunction with automobiles, the technology is not limited to automobiles. The concepts described can be used in a variety of applications, such as in conjunction with aircraft, marine vessels, other vehicles, and consumer electronics components.

[0081] Referring to Figure 1, a system 10 for monitoring a driver in an autonomous vehicle includes a driver monitoring system 12 and a data processor 14. The driver monitoring system is adapted to collect data related to the driver's gaze behavior, and the data processor communicates with and is adapted to receive data from the driver monitoring system 12. The data processor 14 is also adapted to classify the driver into one of a plurality of driver states based on the data received from the driver monitoring system 12. The system 10 also includes at least one vehicle system 16, such as a vehicle control module, and the data processor 14 is further adapted to send instructions to the at least one vehicle system 16 based on the driver classification.

[0082] Data processor 14 is a non-general-purpose electronic control device that includes a pre-programmed digital computer or processor, memory or non-transitory computer-readable medium for storing data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and transceivers or input / output ports. Computer-readable medium includes any type of media that a computer can access, such as read-only memory (ROM), random access memory (RAM), hard disk drive, optical disc (CD), digital video disc (DVD), or any other type of memory. "Non-transitory" computer-readable medium does not include wired communication links, wireless communication links, optical communication links, or other communication links that transmit transient electrical or other signals. Non-transitory computer-readable medium includes media that permanently store data and media that can store data and subsequently be rewritten, such as rewritable optical discs or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code.

[0083] When classifying a driver into one of multiple driver states based on data from the driver monitoring system 12, the data processor 14 is also adapted to create an actual gaze model 18 based on the data collected by the driver monitoring system 12, classify the actual gaze model 18 into one of a first predefined gaze model 20, a second predefined gaze model 22, and a third predefined gaze model 24, generate a confidence score for the classification of the actual gaze model 18, and classify the driver into one of the first, second, and third driver states based on the classification of the actual gaze model 18 and the vehicle's operating state. Those skilled in the art should understand that the system 10 can classify the actual gaze model 18 into any suitable number of predefined gaze models.

[0084] Actual gaze model 18 is a probability distribution of gaze behavior observed by the driver of the vehicle. In one exemplary embodiment, actual gaze model 18 is based on a normalized neural network output. In one exemplary embodiment, a first predefined gaze model 20 is a probability distribution of gaze behavior indicating that the driver of the vehicle is paying close attention in a manner similar to when the driver is manually driving the vehicle. The first predefined gaze model 20 is referred to as the "driving" gaze model.

[0085] The second predefined gaze model 22 is a probability distribution of gaze behavior that indicates the driver is paying attention, but not closely, for example, when the vehicle is in a lower level of autonomous mode (i.e., level 1, 2) and the driver is only giving basic attention to the vehicle's driving characteristics. The second predefined gaze model 22 is referred to as the "supervised" gaze model.

[0086] The third predefined gaze model 24 is a probability distribution of gaze behavior that indicates the vehicle driver is not paying attention at all. The third predefined gaze model 24 is referred to as a “wandering” gaze model. It should be understood that any suitable number of predefined gaze models may be used, and the exemplary embodiments described herein are intended to illustrate non-limiting examples.

[0087] The data processor 14 is adapted to classify the actual gaze model 18 as one of the predefined gaze models 20, 22, and 24 by matching the actual gaze model 18 with one of the predefined gaze models 20, 22, and 24. A neural network loss function is used to compare the actual gaze model 18 with the first, second, and third predefined gaze models 20, 22, and 24, and to classify the actual gaze model 18 as the most matching predefined gaze model among the first, second, and third predefined gaze models 20, 22, and 24. Figure 2 is a two-dimensional representation of the actual gaze model distribution 18, the driving gaze model distribution 20, the supervised gaze model distribution 22, and the wandering gaze model distribution 24.

[0088] Data processor 14 generates a classification confidence score for actual gaze model 18 by comparing its match with one of the first, second, and third predefined gaze models 20, 22, and 24 to the other predefined gaze models. In the example shown in Figure 2, actual gaze model 18 matches either the first predefined gaze model 20 or the driving gaze model distribution 20. This is because the two-dimensional distance 26 between actual gaze model 18 and the driving gaze model distribution 20 is smaller than the distance 28 between actual gaze model 18 and the supervised gaze model distribution 22, and smaller than the distance 30 between actual gaze model 18 and the wandering gaze model distribution 24, indicating that actual gaze model 18 is the best match for either the first predefined or the driving gaze model distribution. The two-dimensional distance 26 between actual gaze model 18 and the driving gaze model distribution 20 is significantly smaller than the distance 28 between actual gaze model 18 and the supervised gaze model distribution 22, and significantly smaller than the distance 30 between actual gaze model 18 and the wandering gaze model distribution 24. Because the differences at distances 26, 28, and 30 are significant, making a final decision regarding the gaze model classification is relatively easy. Referring to Figure 3, the two-dimensional distance 26' between the actual gaze model 18' and the driving gaze model distribution 20' is smaller than the distance 28' between the actual gaze model 18' and the supervised gaze model distribution 22', and smaller than the distance 30' between the actual gaze model 18' and the wandering gaze model distribution 24'. However, in this example, the differences between distances 26, 28, and 30 are not significant; therefore, making a final decision regarding the gaze model classification is very difficult.

[0089] The data processor is adapted to classify the driver into one of a first driver state, a second driver state, and a third driver state. In one exemplary embodiment, the first driver state indicates that the driver is comfortable, attentive, and unlikely to take over manual control of the vehicle. The second driver state indicates that the driver is uncomfortable and more likely to take over manual control of the vehicle. The third driver state indicates that the driver is inattentive or distracted.

[0090] In one exemplary embodiment, the multiple driver states include a first driver state, a second driver state, and a third driver state. It should be understood that system 10 may include any number of driver states without departing from the scope of this disclosure. When a driver is classified into one of the first, second, and third driver states based on the classification of the actual gaze model 18 and the vehicle's operating state, data processor 14 is also adapted to classify the driver into the first driver state when the actual gaze model 18 is classified as a first predefined gaze model 20 and the vehicle's operating state is manual mode.

[0091] Data processor 14 communicates with the vehicle's CAN bus 32, thereby receiving information about the vehicle's operating status. First, the data processor determines whether the vehicle is in automatic or manual mode. When the actual gaze model 18 is classified as the first predefined gaze model 20, the driver is paying close attention, and since the vehicle is in manual mode, this is how the driver should behave. Therefore, the data processor is adapted to send instructions to the vehicle control module 16 to maintain the vehicle's current operating parameters.

[0092] When the driver is classified into one of the first driver state, second driver state, and third driver state based on the classification of the actual gaze model 18 and the vehicle's operating state, the data processor 14 is also adapted to classify the driver into the second driver state when the actual gaze model 18 is classified into the first predefined gaze model 20 and the vehicle's operating state is autonomous mode.

[0093] When the actual gaze model 18 is classified as the first predefined gaze model 20, the driver is paying close attention; however, since the vehicle is in autonomous mode, this indicates that the driver is uncomfortable and may take over the manual control of the vehicle.

[0094] For example, if the vehicle is operating in autonomous mode and the vehicle control module 16 is following a vehicle immediately ahead at a distance of one second, and the driver exhibits gaze behavior that would cause the system to classify the driver as a second driver state, the data processor will send a command to the vehicle control module 16 to increase the distance to two seconds. This may result in the driver behaving differently, and the system 10 will classify the driver as a first driver state, in which case the vehicle control module 16 will not take further action. Alternatively, the driver may still exhibit behavior that keeps the driver classified as a second driver state, in which case the data processor 14 will send additional commands to the vehicle control module 16 to change the current operating parameters, such as by decelerating, changing lanes, etc.

[0095] When the driver is classified into one of the first driver state, second driver state, and third driver state based on the classification of the actual gaze model 18 and the vehicle's operating state, the data processor 14 is also adapted to classify the driver into the third driver state when the actual gaze model 18 is classified into the second predefined gaze model and the vehicle's operating state is manual mode.

[0096] When the actual gaze model 18 is classified as the second predefined gaze model 22, the driver is paying less attention, and since the vehicle is in manual mode, this indicates that the driver is distracted or inattentive. Therefore, when the vehicle's driver is classified as a third driver state, the data processor 14 is adapted to send instructions to the vehicle control module 16 to change the priority of the warning prompts (e.g., issue an attention alert when appropriate, providing an enhanced takeover alert).

[0097] When the driver is classified into one of the first driver state, second driver state, and third driver state based on the classification of the actual gaze model 18 and the vehicle's operating state, the data processor 14 is also adapted to classify the driver into the first driver state when the actual gaze model is classified into the second predefined gaze model, the vehicle operating state is autonomous mode, and the vehicle is one of the autonomous vehicles of level one and level two.

[0098] When the actual gaze model 18 is classified as the second predefined gaze model 22, the driver is paying less attention; however, this behavior is permissible because the vehicle is in autonomous mode and is one of the autonomous vehicles, either Level 1 or Level 2. Level 1 or Level 2 autonomous vehicles require this level of driver attention; therefore, the data processor 14 is adapted to send instructions to the vehicle control module 16 to maintain the vehicle's current operating parameters.

[0099] When the driver is classified into one of the first driver state, second driver state, and third driver state based on the classification of the actual gaze model 18 and the vehicle's operating state, the data processor 14 is also adapted to classify the driver into the second driver state when the actual gaze model is classified into a second predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is an autonomous vehicle of one of the three levels: level three, level four, and level five.

[0100] When the actual gaze model 18 is classified as the second predefined gaze model 22, the driver is not paying close attention, and since the vehicle is in autonomous mode and is one of the Level 3, 4, or 5 autonomous vehicles, this indicates that the driver is uncomfortable and may take over manual control of the vehicle. Level 3, 4, or 5 autonomous vehicles typically do not require that level of driver attention. Therefore, the data processor 14 is adapted to send instructions to the vehicle control module 16 to change the vehicle's current operating parameters, thereby making the driver more comfortable and reducing the likelihood that the driver will take over manual control of the vehicle.

[0101] When the driver is classified into one of the first, second, and third driver states based on the classification of the actual gaze model and the vehicle's operating state, the data processor is also adapted to classify the driver into the third driver state when the actual gaze model is classified as the third predefined gaze model and the vehicle's operating state is manual mode.

[0102] When the actual gaze model 18 is classified as the third predefined gaze model 24, the driver is not paying attention, and since the vehicle is in manual mode, this indicates that the driver is distracted or inattentive. Therefore, when the vehicle's driver is classified as the third driver state, the data processor 14 is adapted to send instructions to the vehicle control module 16 to change the vehicle's current operating parameters in order to alter the driver's ability to take over manual control of the vehicle by preventing the driver from taking over manual control when distracted.

[0103] When the driver is classified into one of the first driver state, second driver state, and third driver state based on the classification of the actual gaze model and the vehicle's operating state, the data processor is also adapted to classify the driver into the third driver state when the actual gaze model is classified into a third predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is an autonomous vehicle of either level one or level two.

[0104] When the actual gaze model 18 is classified as the third predefined gaze model 24, the driver is not paying attention, and since the vehicle is in autonomous mode and is a Level 1 or Level 2 autonomous vehicle, this indicates that the driver is distracted or inattentive. Level 1 or Level 2 autonomous vehicles require more attention from the driver than this. Therefore, when the vehicle's driver is classified as a third driver state, the data processor 14 is adapted to send instructions to the vehicle control module 16 to change the vehicle's current operating parameters in order to alter the driver's ability to take over manual control of the vehicle by preventing the driver from taking over manual control when distracted.

[0105] When the driver is classified into one of the first driver state, second driver state, and third driver state based on the classification of the actual gaze model and the vehicle's operating state, the data processor is also adapted to classify the driver into the first driver state when the actual gaze model is classified into a third predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is an autonomous vehicle of one of the three levels: Level 3, Level 4, and Level 5.

[0106] When the actual gaze model 18 is classified as the third predefined gaze model 24, the driver does not pay attention. However, since the vehicle is in autonomous mode and is a Level 3, 4, or 5 autonomous vehicle, this level of driver attention is acceptable. Level 3, 4, or 5 autonomous vehicles do not require driver attention. Therefore, the data processor 14 is adapted to send instructions to the vehicle control module 16 to maintain the vehicle's current operating parameters.

[0107] Finally, the data processor is adapted to update the first predefined gaze model 20, the second predefined gaze model 22, and the third predefined gaze model 24. In this way, the system 10 continuously learns the driver's behavior and preferences to better predict future driver states.

[0108] Referring to Figure 4, a method 100 for monitoring a driver in an autonomous vehicle includes, starting at box 102, monitoring the driver of the vehicle using a driver monitoring system; moving to box 104, collecting data related to the driver's gaze behavior from the driver monitoring system using a data processor; moving to box 106, the method further includes classifying the driver into one of a plurality of driver states using the data processor based on the data from the driver monitoring system; and moving to box 108, sending instructions to at least one vehicle system based on the driver classification using the data processor.

[0109] Referring to Figure 5, in one exemplary embodiment, the "classifying the driver into one of multiple driver states based on data from the driver monitoring system 12" at box 106 further includes moving to box 110 to create an actual gaze model 18 based on data collected by the driver monitoring system 12, moving to box 112 to classify the actual gaze model into one of multiple predefined gaze models 20, 22, 24, and moving to box 114 to classify the driver into one of multiple driver states based on the classification of the actual gaze model 18 and the vehicle's operating state.

[0110] In one exemplary embodiment, “classifying the actual gaze model into one of a plurality of predefined gaze models 20, 22, 24” at box 112 also includes generating a confidence score for the classification of the actual gaze model 18 using the data processor 14.

[0111] The multiple driver states include a first driver state, a second driver state, and a third driver state, and the multiple predefined gaze models include a first gaze model, a second gaze model, and a third gaze model. In box 112, "classifying the actual gaze model 18 into one of the multiple predefined gaze models 20, 22, and 24" also includes one of the following: moving to box 116 to classify the actual gaze model 18 into the first predefined gaze model 20, or moving to box 118 to classify the actual gaze model 18 into the second predefined gaze model 22, or moving to box 120 to classify the actual gaze model 18 into the third predefined gaze model 24.

[0112] In one exemplary embodiment, the "classifying the driver into one of multiple driver states based on the classification of the actual gaze model 18 and the vehicle's operating state" at box 114 further includes: when the actual gaze model 18 is classified as a first predefined gaze model at box 116 and the vehicle's operating state is manual mode (as indicated by arrow 126 at box 124), moving to box 122 to classify the driver into a first driver state; and when the actual gaze model 18 is classified as a first predefined gaze model at box 116 and the vehicle's operating state is autonomous mode (as indicated by arrow 130 at box 124), moving to box 128 to classify the driver into a second driver state.

[0113] Furthermore, the phrase “classifying the driver into one of multiple driver states based on the classification of the actual gaze model 18 and the vehicle’s operating state” at box 114 also includes: when the actual gaze model 18 is classified as the second predefined gaze model at box 118 and the vehicle’s operating state is manual mode (as indicated by arrow 136 at box 134), moving to box 132 to classify the driver into the third driver state.

[0114] The phrase “classifying the driver into one of multiple driver states based on the classification of the actual gaze model 18 and the vehicle’s operating state” at box 114 further includes: when the actual gaze model 18 is classified as the second predefined gaze model at box 118, the vehicle operating state is autonomous mode (as shown by arrow 138 at box 134), and the vehicle is one of the autonomous vehicles, namely a Level 1 autonomous vehicle and a Level 2 autonomous vehicle (as shown by arrow 142 at box 140), the driver is moved to box 122 and classified as the first driver state.

[0115] The phrase “classifying the driver into one of multiple driver states based on the classification of the actual gaze model 18 and the vehicle’s operating state” at box 114 further includes: when the actual gaze model 18 is classified as the second predefined gaze model at box 118 and the vehicle operating state is autonomous mode (as shown by arrow 138 at box 134) and the vehicle is one of the autonomous vehicles of level 3, level 4 and level 5 (as shown by arrow 144 at box 140), move to box 128 and classify the driver into the second driver state.

[0116] Finally, the "classifying the driver into one of multiple driver states based on the actual gaze model and the vehicle's operating state" at box 114 also includes: when the actual gaze model 18 is classified as the third predefined gaze model at box 120 and the vehicle operating state is manual mode (as indicated by arrow 148 at box 146), move to box 132 to classify the driver into the third driver state.

[0117] The phrase “classifying the driver into one of multiple driver states based on the actual gaze model and the vehicle’s operating state” at box 114 further includes: when the actual gaze model 18 is classified as the third predefined gaze model at box 120, the vehicle operating state is autonomous (as shown by arrow 150 at box 146), and the vehicle is one of the autonomous vehicles, either a Level 1 autonomous vehicle or a Level 2 autonomous vehicle (as shown by arrow 154 at box 152), the driver is moved to box 132 and classified into the third driver state.

[0118] The phrase “classifying the driver into one of multiple driver states based on the classification of the actual gaze model and the vehicle’s operating state” at box 114 further includes: when the actual gaze model 18 is classified as the third predefined gaze model at box 120, the vehicle operating state is autonomous mode (as shown by arrow 150 at box 146), and the vehicle is one of the autonomous vehicles of level 3, level 4, and level 5 (as shown by arrow 156 at box 152), move to box 122 and classify the driver into the first driver state.

[0119] In one exemplary embodiment, the “sending instructions to at least one vehicle system 16 based on driver classification via data processor 14” at block 108 further includes sending instructions to vehicle control module block 16 via data processor 14 to maintain the current operating parameters of the vehicle when the driver of the vehicle is classified as a first driver state.

[0120] In another exemplary embodiment, the “sending instructions to at least one vehicle system 16 based on driver classification via data processor 14” at block 108 further includes sending instructions to vehicle control module block 16 via data processor 14 to change the current operating parameters of the vehicle when the driver of the vehicle is classified into one of the second driver state and the third driver state.

[0121] The second driver status indicates that the driver of the vehicle is uncomfortable and may take over the manual control of the vehicle, and the instructions sent to the vehicle control module 16 are adapted to change the current operating parameters of the vehicle in order to make the driver of the vehicle comfortable and reduce the possibility that the driver of the vehicle will take over the manual control.

[0122] The third driver status indicates that the driver of the vehicle is distracted, and the instructions sent to the vehicle control module 16 are adapted to change the driver's ability to take over manual control of the vehicle by preventing the driver from taking over manual control when distracted.

[0123] In another exemplary embodiment, method 100 further includes updating a plurality of predefined gaze models 20, 22, 24 whenever the driver of the vehicle takes over manual control of the vehicle.

[0124] The system 10 and method 100 of this disclosure provide the following advantages: anticipating driver takeover events and allowing the vehicle control module 16 to change the vehicle's operating parameters or change the priority of warning prompts to make the driver more comfortable and avoid takeover events.

[0125] The description in this disclosure is exemplary in nature only, and variations thereof without departing from the spirit and scope of this disclosure are intended to fall within its scope. Such variations should not be considered as departing from the spirit and scope of this disclosure.

Claims

1. A method for monitoring a driver in an autonomous vehicle, the method comprising: Monitor vehicle drivers using a driver monitoring system; Data related to the driver's gaze behavior is collected from the driver monitoring system using a data processor; The data processor uses data from the driver monitoring system to classify the driver into one of multiple driver states. The system uses the data processor to send instructions to at least one vehicle system based on the driver's classification; wherein classifying the driver into one of multiple driver states based on data from the driver monitoring system further includes: creating an actual gaze model based on data collected by the driver monitoring system; classifying the actual gaze model into one of multiple predefined gaze models; classifying the driver into one of multiple driver states based on the classification of the actual gaze model and the vehicle's operating state; wherein the multiple driver states include a first driver state, a second driver state, and a third driver state, and the multiple predefined gaze models include a first predefined gaze model, a second predefined gaze model, and a third predefined gaze model; wherein classifying the driver into one of the multiple driver states based on the classification of the actual gaze model and the vehicle's operating state further includes: when the actual gaze model is classified... When the actual gaze model is the first predefined gaze model and the vehicle operating state is manual mode, the driver is classified as the first driver state; when the actual gaze model is classified as the first predefined gaze model and the vehicle operating state is autonomous mode, the driver is classified as the second driver state; when the actual gaze model is classified as the second predefined gaze model and the vehicle operating state is manual mode, the driver is classified as the third driver state; when the actual gaze model is classified as the second predefined gaze model, the vehicle operating state is autonomous mode, and the vehicle is one of Level 1 autonomous vehicle and Level 2 autonomous vehicle, the driver is classified as the first driver state; and when the actual gaze model is classified as the second predefined gaze model, the vehicle operating state is autonomous mode, and the vehicle is one of Level 3 autonomous vehicle, Level 4 autonomous vehicle, and Level 5 autonomous vehicle, the driver is classified as the second driver state.

2. The method according to claim 1, wherein, The classification of the driver into one of the multiple driver states based on the actual gaze model classification and the vehicle's operating state further includes: classifying the driver into the third driver state when the actual gaze model is classified as the third predefined gaze model and the vehicle operating state is manual mode; classifying the driver into the third driver state when the actual gaze model is classified as the third predefined gaze model, the vehicle operating state is autonomous mode, and the vehicle is one of Level 1 autonomous vehicle and Level 2 autonomous vehicle; and classifying the driver into the first driver state when the actual gaze model is classified as the third predefined gaze model, the vehicle operating state is autonomous mode, and the vehicle is one of Level 3 autonomous vehicle, Level 4 autonomous vehicle, and Level 5 autonomous vehicle.

3. The method according to claim 2, wherein, The method of sending instructions to at least one vehicle system based on the driver's classification using the data processor further includes: when the driver of the vehicle is classified as the first driver state, sending instructions to the vehicle control module using the data processor to maintain the current operating parameters of the vehicle.

4. The method according to claim 3, wherein, The method of sending instructions to at least one vehicle system based on the driver's classification using the data processor further includes: when the driver of the vehicle is classified into one of the second driver state and the third driver state, sending instructions to the vehicle control module using the data processor to change the current operating parameters of the vehicle.

5. The method according to claim 4, wherein, The second driver state indicates that the driver of the vehicle is uncomfortable and may take over manual control of the vehicle, and the instructions sent to the vehicle control module are adapted to change the current operating parameters of the vehicle in order to make the driver of the vehicle comfortable and reduce the likelihood that the driver of the vehicle will take over manual control.

6. The method according to claim 4, wherein, The third driver state indicates that the driver of the vehicle is distracted, and the instructions sent to the vehicle control module are adapted to change the priority of warning prompts that may be provided to the driver.

7. The method according to claim 4, wherein, The method further includes updating the plurality of predefined gaze models whenever the driver of the vehicle takes over manual control of the vehicle.

8. The method according to claim 7, wherein, Classifying the actual gaze model into one of a plurality of predefined gaze models further includes: generating a confidence score for the classification of the actual gaze model using the data processor.

9. A system for monitoring a driver in an autonomous vehicle, the system comprising: A driver monitoring system, the driver monitoring system being adapted to collect data related to the driver's gaze behavior; A data processor adapted to receive data from the driver monitoring system and classify the driver into one of a plurality of driver states based on the data from the driver monitoring system; The data processor is further adapted to send instructions to the at least one vehicle system based on the driver's classification; wherein, when the driver is classified into one of a plurality of driver states based on data from the driver monitoring system, the data processor is further adapted to create an actual gaze model based on data collected by the driver monitoring system, classify the actual gaze model into one of a first predefined gaze model, a second predefined gaze model, and a third predefined gaze model, generate a confidence score for the classification of the actual gaze model, and classify the driver into one of a first driver state, a second driver state, and a third driver state based on the classification of the actual gaze model and the vehicle's operating state; wherein, when the driver is classified into one of the plurality of driver states based on the classification of the actual gaze model and the vehicle's operating state, the data processor is further adapted to: when the actual gaze model When the actual gaze model is classified as the first predefined gaze model and the vehicle operating state is manual mode, the driver is classified as a first driver state; when the actual gaze model is classified as the first predefined gaze model and the vehicle operating state is autonomous mode, the driver is classified as a second driver state; when the actual gaze model is classified as the second predefined gaze model and the vehicle operating state is manual mode, the driver is classified as a third driver state; when the actual gaze model is classified as the second predefined gaze model, the vehicle operating state is autonomous mode, and the vehicle is one of Level 1 autonomous vehicles and Level 2 autonomous vehicles, the driver is classified as a first driver state; and when the actual gaze model is classified as the second predefined gaze model, the vehicle operating state is autonomous mode, and the vehicle is one of Level 3 autonomous vehicles, Level 4 autonomous vehicles, and Level 5 autonomous vehicles, the driver is classified as a second driver state.

10. The system according to claim 9, wherein, When the driver is classified into one of the multiple driver states based on the classification of the actual gaze model and the vehicle's operating state, the data processor is further adapted to: classify the driver into the third driver state when the actual gaze model is classified into the third predefined gaze model and the vehicle's operating state is manual mode; and classify the driver into the third driver state when the actual gaze model is classified into the third predefined gaze model, the vehicle's operating state is autonomous mode, and the vehicle is an autonomous vehicle of either level one or level two. When the actual gaze model is classified as a third predefined gaze model, the vehicle operating state is autonomous mode, and the vehicle is one of the three autonomous vehicles (Level 3, Level 4, and Level 5), the driver is classified as the first driver state.

11. The system according to claim 10, wherein, The system further includes a vehicle control module, and the data processor is also adapted to: send instructions to the vehicle control module to maintain the current operating parameters of the vehicle when the driver of the vehicle is classified as a first driver state; and send instructions to the vehicle control module to change the current operating parameters of the vehicle when the driver of the vehicle is classified as a second driver state or a third driver state.

12. The system according to claim 11, wherein, The second driver state indicates that the driver of the vehicle is uncomfortable and may take over manual control of the vehicle, and the data processor is adapted to send instructions to the vehicle control module to change the current operating parameters of the vehicle in order to make the driver of the vehicle comfortable and reduce the likelihood that the driver of the vehicle will take over manual control. The third driver state indicates that the driver of the vehicle is distracted, and the data processor is adapted to send instructions to the vehicle control module to change the priority of warning prompts that may be provided to the driver.

13. The system according to claim 12, wherein, The data processor is also adapted to update the first predefined gaze model, the second predefined gaze model, and the third predefined gaze model whenever the driver of the vehicle takes over manual control of the vehicle.

Images