Driver monitoring system

Abstract

The technology of the present disclosure relates to a driver monitoring system. The system includes a controller circuit that receives monitoring data from a driver monitoring sensor configured for monitoring a driver of the vehicle while the vehicle is operating in an autonomous driving mode. The controller circuit determines a score for one or more driver supervision metrics based on the monitoring data, each of the driver supervision metrics indicating whether the driver is supervising operation of the vehicle. The controller circuit determines a supervision score based on the score of the driver supervision metric. The supervision score indicates whether the driver is ready to resume control of the vehicle. The controller circuit indicates a driver perceived state on a display in the driver's field of view based on the supervision score. The system may improve vehicle safety by alerting a driver when the driver exhibits reduced driver-perceived behavior.

Description

This application is a divisional application filed for application number 202110013494.7, filing date January 6, 2021, and titled “Driver Monitoring System”. CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 62 / 957,426, filed on January 6, 2020, the disclosure of which is hereby incorporated by reference in its entirety. Background Art

[0002] The Society of Automotive Engineers (SAE) Level 2 automated driving system includes driver assistance features that provide steering, braking, and acceleration assistance (e.g., lane centering and adaptive cruise control). Vehicles equipped with Level 2 automated driving systems require a human driver to be ready to take over control of the vehicle in the event that the automated driving system relinquishes control. When the vehicle is operating under Level 2 automated control, the driver is responsible for supervising the automated driving functions and the environment around the vehicle. When an inattentive driver is required to take back control from the automated driving system, the driver may not be able to react in time to avoid an accident or collision. Some automated driving systems warn inattentive drivers when the driver removes their hands from the steering wheel or their eyes from the road within a predefined period of time. In some cases, the driver is able to overcome these monitoring systems by placing objects on the steering wheel to simulate torque input from the driver's hands or placing a photo in the driver's seat to make the driver appear to be paying attention to the road. Summary of the invention

[0003] This document describes one or more aspects of a driver monitoring system. In one example, the system includes a controller circuit configured to receive monitoring data from a driver monitoring sensor, the driver monitoring sensor being configured to monitor a driver of the vehicle when the vehicle is operating in an autonomous driving mode. The controller circuit is also configured to determine a score of one or more driver supervision metrics based on the monitoring data, each of the driver supervision metrics at least partially indicating whether the driver is supervising the operation of the vehicle. The controller circuit is also configured to determine a supervision score based on the score of the one or more driver supervision metrics. The supervision score indicates whether the driver is ready to resume control of the vehicle. The controller circuit is also configured to indicate a driver perception state on a display in the driver's field of view based on the supervision score.

[0004] In another example, a method includes receiving, with a controller circuit, monitoring data from a driver monitoring sensor configured to monitor a driver of the vehicle while the vehicle is operating in an autonomous driving mode. The method also includes determining, with the controller circuit, a score of one or more driver supervision metrics based on the monitoring data. The method also includes determining, with the controller circuit, a supervision score based on the score of the one or more driver supervision metrics. The method also includes indicating, with the controller circuit, a driver perception state on a display in the driver's field of view based on the supervision score.

[0005] This Summary is provided to introduce various aspects of a driver monitoring system that is further described in the following Detailed Description and Figures. For ease of description, this disclosure focuses on vehicle-based or automobile-based systems, such as those integrated on a vehicle traveling on a road. However, the techniques and systems described herein are not limited to vehicle or automobile contexts, but are also applicable to other environments in which cameras can be used to detect objects. This Summary is not intended to identify necessary features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] This document describes details of one or more aspects of a driver monitoring system with reference to the following drawings. The same numbers are used throughout the drawings to reference similar features and components: Figure 1 An example of a driver monitoring system installed on a vehicle is shown; Figure 2 Shown with Figure 1 an example isolated example driver monitoring sensor of a driver monitoring system; Figure 3 Shows Figure 1 Examples of driver supervision metrics for driver monitoring systems; Figure 4 An example of an attention buffer value graph for a situational awareness driver supervision metric is shown; Figure 5 An example of a supervision score based on a driver supervision metric score is shown; Figure 6 Another example of a supervision score based on a driver supervision metric score is shown; Figure 7 Yet another example of a supervision score based on a driver supervision metric score is shown; Figure 8 Shows Figure 1 An example of a display of a driver monitoring system extending along a dashboard of a vehicle; Fig. 9 The supervision scores and Figure 8 An example of the associated colors lighting up on a display; Fig.10 Shows Figure 1 An example of a driver monitoring system having a display extending along a door of a vehicle; Fig.11 Is Figure 1 An example flowchart of an example logic flow executed by controller circuitry of a system; and Fig.12 Yes Operation Figure 1 An example method of an example driver monitoring system. DETAILED DESCRIPTION Overview

[0007] The technology disclosed herein relates to a driver monitoring system. A controller circuit receives data from a sensor in the cockpit that detects whether the driver is supervising an automated vehicle operating in an automated driving mode. The sensor in the cockpit detects whether the driver is sitting properly in the driver's seat and whether the driver is paying attention to the vehicle's surroundings. The system determines scores of several driver supervision metrics that indicate whether the driver is supervising the operation of the vehicle. The system determines a supervision score based on the scores of the driver supervision metrics that indicates whether the driver is ready to resume control of the vehicle. The system indicates the driver's perception state on a lighted display based on the supervision score. The lighted display is located in the driver's field of view, and the color lit on the display corresponds to the driver's perception state. When it is determined that the driver is not paying attention, the controller circuit warns the driver by changing the color of the display and pulsating the light on the display. When the driver continues to be inattentive over time, the controller circuit increases the frequency of the pulsation. The driver monitoring system can improve vehicle safety by warning the driver of inattentive behavior when the vehicle is operating in an autonomous driving mode, and smoothly handing over control to the driver, thereby improving the user experience. Example System

[0008] Figure 1 An example of a driver monitoring system 100 (hereinafter referred to as system 100) is shown. System 100 includes a controller circuit 102 configured to receive monitoring data 104 from a driver monitoring sensor 106 installed on a vehicle 108. Driver monitoring sensor 106 may be a component of an occupant classification system 110 (OCS 110) installed on vehicle 108, which will be explained in more detail below. Controller circuit 102 is configured to determine a supervision score 112 (see FIG. 1 ) of a driver. Figure 5 ) and indicates the driver perception status 114 on a display 116 visible in the driver's field of view.

[0009] Although the vehicle 108 may be any vehicle, for ease of description, the vehicle 108 is primarily described as an autonomous vehicle that is configured to operate in an autonomous driving mode for assisting a driver riding in the vehicle 108. The vehicle 108 may be capable of Society of Automotive Engineers (SAE) Level 2 autonomous operation, which assists the driver with steering, braking, and acceleration while the driver is always monitoring the operation of the vehicle 108 in the driver's seat.

[0010] exist Figure 1 In the example shown, the controller circuit 102 is mounted on the vehicle 108 and is communicatively coupled to the driver monitoring sensor 106 and the display 116 via a transmission link. The transmission link may be a wired or wireless interface, for example, Near Field Communication (NFC), Universal Serial Bus (USB), Universal Asynchronous Receiver / Transmitter (UART), or Controller Area Network (CAN). In some examples, controller circuit 112 receives data from other vehicle systems via a CAN bus (not shown), such as ignition status, vehicle speed, vehicle relative motion, and transmission gear selection. Controller Circuit

[0011] The controller circuit 102 may be implemented as a microprocessor or other control circuit system (such as an analog and / or digital control circuit system). The control circuit system may include one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) programmed to perform the technology, or one or more general purpose hardware processors programmed to perform the technology according to program instructions in firmware, memory, other storage, or a combination. The controller circuit 102 may also combine customized hard-wired logic, ASICs, or FPGAs with customized programming to perform the technology. The controller circuit 102 may include a memory or storage medium (not shown), including a non-volatile memory, such as an electrically erasable programmable read-only memory (EEPROM) for storing one or more routines, thresholds, and captured data. The EEPROM stores data and allows a single byte to be erased and reprogrammed by applying a programming signal. The controller circuit 102 may include other examples of non-volatile memory, such as flash memory, read-only memory (ROM), programmable read-only memory (PROM), and erasable programmable read-only memory (EPROM). The controller circuit 102 may include volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)). The controller circuit 102 may include one or more clocks or timers for synchronizing control circuitry or determining the elapsed time of an event. One or more routines may be executed by the processor to perform steps for determining the supervision score 112 based on signals received by the controller circuit 102 from the driver monitoring sensor 106 as described herein. Driver Monitoring Sensors

[0012] Figure 2 An example of a driver monitoring sensor 106 positioned away from the system 100 is shown. As will be described in detail below, the driver monitoring sensor 106 is configured to monitor the driver of the vehicle 108. The driver monitoring sensor 106 may include one or more sensors that detect various aspects of the driver and may be a component of the OCS 110 mounted on the vehicle 108. The driver monitoring sensor 106 may include a camera that captures an image of the driver, and the OSC 110 determines whether the driver's seat is occupied by a person based on the image. The software executed by the OCS 110 may use known image analysis techniques to distinguish people from animals and objects. Objects in the image are detected in an area of ​​interest corresponding to a seating position in the cabin (e.g., the driver's seat position), and the objects are classified into humans and other classifications by the software. The processing blocks or models in the software are pre-trained to recognize the shape of human forms or other objects (e.g., shopping bags, boxes, or animals). The camera may be a two-dimensional (2D) camera or a 3D time-of-flight camera that measures the time when a light pulse leaves the camera and reflects back to the imaging array of the camera.

[0013] The OCS 110 may use the images captured by the camera to determine whether the driver is correctly positioned in the driver's seat. The OCS 110 may use known positioning techniques to locate the driver's head or upper body and determine whether the driver is within the driver's seat area. The OCS 110 may include a head model and a body model that have known rotations and dynamics within the driver's seat that correspond to normal driving behavior. The OCS 110 may compare the position of the head and upper body in the image with the head and body model to determine whether the driver is correctly positioned in the driver's seat.

[0014] OCS110 can use the image captured by the camera to determine the driver's gaze direction, for example, whether the driver is looking at the road in front of the vehicle 108 or looking at other directions. OCS110 can use known eye tracking technology to determine the gaze direction based on the positioning of the pupil of the eye and the determined head posture. OCS110 can track the time when the driver's gaze is fixed in a specific direction (e.g., fixation), and can count the frequency of rapid movement of the eyes between different fixed positions (e.g., rapid eye saccades). As will be explained in more detail below, OCS110 can use the driver's gaze information to indicate the driver's situational awareness. In some examples, when the driver wears glasses or in night conditions, an infrared (IR) light source is used to illuminate the eyes to achieve pupil positioning.

[0015] The OCS 110 can use the images captured by the camera to track the amount of time the driver's eyes are closed to determine the level of drowsiness and sleep. The OCS 110 can use an eye model to determine whether the eyelid is covering the iris of the eye based on the images captured by the camera. The OCS 110 can determine the amount of eyelid opening and the duration of eye closure as an indication of drowsiness or sleep.

[0016] The driver monitoring sensor 106 may also include a seat pressure sensor that detects the pressure or pressure distribution applied to the seat. The OCS 110 may determine whether the driver is sitting in the driver's seat based on a pressure threshold that indicates the driver's weight. For example, if the occupant weighs more than 30 kg, the OCS 110 may determine that the driver is considered an adult. The pressure distribution may indicate whether the object occupying the driver's seat is a person or an object other than a person. The pressure distribution may also indicate whether the driver is in the correct position in the driver's seat; for example, when the driver leans to one side of the seat, the pressure is concentrated on one side of the seat.

[0017] The driver monitoring sensors 106 may also include a steering wheel torque sensor that detects torque applied to the steering wheel. Torque may be detected when the driver has their hands on the steering wheel, even when the autonomous control system is steering the vehicle 108. The steering wheel torque sensor may be an electromechanical device integrated into the power steering system of the vehicle 108 that determines the torsion bar angle required for steering movement. The steering wheel torque sensor may also output the steering angle and the rate of change of the steering wheel angular position.

[0018] The driver monitoring sensor 106 may also include a capacitive steering wheel sensor that detects the touch of the driver's hand on the steering wheel. The capacitive steering wheel sensor may be located at the edge of the steering wheel and may detect the point of contact of the hand with the steering wheel. In some examples, touching the steering wheel with a hand distorts the electric field generated by the sensor and changes the capacitance of the sensor, thereby indicating the presence of the driver's hand. The capacitive steering wheel sensor may detect whether the driver has one hand on the steering wheel or both hands on the steering wheel.

[0019] The driver monitoring sensor 106 may also include a radar sensor that detects the presence of an object in the vehicle cabin, and the OCS 110 may determine whether the driver's seat is occupied by the driver or the object based on the point cloud data received from the radar sensor. The OCS 110 compares the point cloud data with a model in the software to determine whether the seat is occupied by a person or an object. In some examples, the radar sensor may detect relatively small movements (e.g., movement of the chest wall of a breathing driver).

[0020] OCS 110 may periodically update monitoring data 104 from OCS 110 to ensure that controller circuit 102 can accurately determine supervision score 112. For example, OCS 110 may update monitoring data 104 at one-second intervals to account for brief changes in seat occupancy or attention of the driver.

[0021] OCS110 can use machine learning to detect various driver aspects and behaviors. Machine learning is a data analysis technique that teaches computers to learn from experience. Machine learning routines or algorithms use computational methods to learn information from data without relying on predetermined equations as models. These routines improve their performance as the number of samples available for learning increases. Machine learning uses two types of techniques: supervised learning, which trains a model on known input and output data so that it can predict future outputs; and unsupervised learning, which discovers hidden patterns or intrinsic structures in the input data. Supervised learning uses classification and regression techniques to develop predictive models. Common algorithms used to perform classification include support vector machines (SVM), boosted and bagged decision trees, k-nearest neighbors, naive Bayes ( Bayes), discriminant analysis, logistic regression, and neural networks. Common regression algorithms include linear models, nonlinear models, regularization, stepwise regression, boosted and bagged decision trees, neural networks, and adaptive neuro-fuzzy learning. Unsupervised learning discovers hidden patterns or intrinsic structures in data and is used to draw inferences from data sets consisting of input data without labeled responses. Clustering is a commonly used unsupervised learning technique. Common algorithms for performing clustering include k-means and K-center point methods, hierarchical clustering, Gaussian mixture models, hidden Markov models, self-organizing maps, fuzzy c-means clustering, and subtractive clustering. In the context of an autonomous vehicle, OCS110 can use machine learning to specifically detect the driver's attention based on the driver monitoring sensor 106 or other aspects of the driving behavior that feed the driver's supervision metric 118 to ensure that the controller circuit 102 can accurately determine the supervision score 112. Driver Supervision Metrics

[0022] Figure 3 is a table illustrating an example of a driver supervision metric 118 that indicates whether a driver is supervising the operation of a vehicle. Figure 3 This is not intended to be an exhaustive list of driver supervision metrics 118, as other metrics may be included as defined by the user. The controller circuit 102 may determine the score of the driver supervision metric 118 based on the monitoring data 104 received from the OCS 110. The score of the driver supervision metric 118 may be user defined and may be used to determine the score of the driver supervision metric 118. Figure 3In the example shown in , the range is from zero to 100, where zero indicates the highest score for the corresponding metric and 100 indicates the lowest score for the corresponding metric. For example, a score of zero for hands-off time indicates that the driver's hands were in contact with the steering wheel most of the time, while a score of 100 indicates that the driver's hands were not on the steering wheel for a time greater than a threshold, as will be explained below. Some of the driver supervision metrics 118 (e.g., driver sleep state) are binary (e.g., zero or 100), where zero indicates that the driver is awake and 100 indicates that the driver is asleep.

[0023] The driver supervision metric 118 includes the driver's situational awareness, which is based in part on the driver's glance direction, the driver's glance duration, and the driver's glance count as detected by the camera and determined by the OCS 110. The score of the situational awareness metric is calculated based on the buffer algorithm developed by the Advanced Human Factors Evaluators for Automotive Demand (AHEAD) Alliance (including members from the automotive and insurance industries and consumer rights entities) sponsored by the Massachusetts Institute of Technology (MIT) in Cambridge, Massachusetts, USA. The buffer algorithm uses an attention buffer design, where the buffer is initially full (e.g., a value of 2.0) and decreases to zero when the driver glances off the road. Situational awareness is calculated using the equation (2-buffer)*50, where a buffer value of 2 indicates high situational awareness, resulting in a driver supervision metric score of zero; a buffer value of 1 indicates cognitive overload, resulting in a driver supervision metric score of 50; and a buffer value of zero indicates low situational awareness, resulting in a driver supervision metric score of 100. For example, when the driver looks in the same direction for a long time, the buffer algorithm also considers gaze fixation or vision tunneling. Gaze fixation may be related to cognitive load or mental distraction, and although the driver appears to be watching the road, the attention buffer can be reduced because the driver is not scanning the surroundings for changing traffic conditions.

[0024] Figure 4 8. An example of an attention buffer changing over time as the driver glances in different directions for different periods of time, as determined by OCS 110. The buffer starts at a value of 2 at 10 seconds and decreases to a value of 1 when the driver glances away from the forward direction between 9 seconds and 8 seconds. The driver looks back forward between 8 seconds and 7 seconds, at which time the buffer increases to approximately 1.25. The driver continues to glance away from the forward direction for a longer period of time, and at 5 seconds the buffer value is zero, indicating a low level of situational awareness. From approximately 4.5 seconds to 3 seconds, the driver's gaze returns to the forward direction, and the buffer increases back to a maximum value of 2. As shown in FIG. Figure 4As shown in , at approximately 2 seconds, the buffer zone algorithm does not penalize the driver for glancing in directions related to the surrounding environment or related to the operation of the vehicle 108, where the driver glances at the rearview mirror or speedometer, and the buffer zone remains at a maximum value of 2.

[0025] Return to reference Figure 3 , the driver supervision measure 118 includes the driver's distraction level, which is scored from zero (when the driver is not considered distracted) to a maximum of 100 (when the driver is considered distracted). The distraction level can be determined by the controller circuit 102 based on the time when the driver is determined to be looking at the road. For example, OCS110 can define an area of ​​concern on the road corresponding to the driver's side of the windshield. When OCS 110 determines that the driver has not looked at the area of ​​concern on the road for a period of time, OCS110 can determine that the driver is distracted. The controller circuit 102 can use the equation MIN (100, 100 * (eyes off the road time / 2000)) to determine a score between zero and 100, where the eyes off the road time is the time in milliseconds (ms) that the driver does not look at the area of ​​concern on the road. The controller circuit 102 can monitor the driver's gaze direction for a maximum duration of 2000ms (i.e., 2 seconds), whereby the score becomes a maximum of 100. The difference between the distraction level and situational awareness is that it does not consider the scanning behavior, but only considers whether the driver is looking in the forward direction of the windshield.

[0026] The driver supervision metric 118 includes detection of objects in the driver's hand, which may be detected by the camera and determined by the OCS 110 using known object recognition and classification techniques. For example, the camera may capture an image of a driver holding a mobile phone or a book, and the OCS 110 may determine the identity of the object held in the driver's hand. The controller circuit 102 may include an image library of known objects to compare to the object detected in the driver's hand and determine whether to assign a score of zero or 100. For example, a mobile phone or a book may be assigned a score of 100, while a bottle of water that the driver lifts to drink may be assigned a score of zero.

[0027] The driver supervision metric 118 may include the driver's hands-off time, which may be determined by the controller circuit 102 using a timer in conjunction with an image from a camera or a capacitive steering wheel sensor. The hands-off time may be scored from zero to 100 by multiplying the hands-off time (in seconds) determined by a factor of 10. For example, a value of zero may be assigned when the hands-off time is detected to be less than a threshold (e.g., less than 1 second), a value of 50 may be assigned when the hands-off time is determined to exceed another threshold (e.g., greater than 5 seconds), and a value of 100 may be assigned when the hands-off time is determined to exceed yet another threshold (e.g., greater than 10 seconds). The OCS 110 may determine whether the driver has removed their hands from the steering wheel, and the controller circuit 102 may start a timer until the OCS 110 determines that the driver's hands have returned to the steering wheel.

[0028] The driver supervision metric 118 may include driver presence. As described above, the presence of the driver may be determined by the OCS 110 using a camera or a seat pressure sensor. The driver presence metric is scored as a binary value, i.e., zero when the driver is sitting in the driver's seat, or 100 when it is determined that the driver is not in the driver's seat.

[0029] The driver supervision metric 118 may include the driver's body position of the driver. As described above, the driver's body position may be determined by the OCS 110 using a camera or a seat pressure sensor. The driver's body position is scored as a binary value, i.e., zero when the driver is seated in the driver's seat, or 100 when it is determined that the driver is not seated in the driver's seat.

[0030] The driver supervision metric 118 may include the driver sleep state. The driver sleep state is scored as a binary value, i.e., zero when the driver is awake, or 100 when the driver is determined to be asleep. The driver sleep state may include microsleep events, which are brief sleep periods that may last from a few microseconds to tens of seconds. Microsleep events may be manifested as drooping eyelids, slow eyelid closure, and nodding. As described above, the OCS 110 may detect that the driver is exhibiting microsleep events by monitoring the driver's eyes and head posture using a camera. The controller circuit 102 may track the time of eyelid closure and determine whether the duration of eyelid closure is significantly longer than a blink that may indicate sleep or a microsleep event.

[0031] The driver supervision metric 118 may include a driver drowsiness level. The driver drowsiness level is related to the driver's sleep state because the driver's drowsiness level is a fraction of a continuous sleep. For example, when the driver's sleep state is zero (e.g., fully awake), the driver has no signs of drowsiness, and when the driver's sleep state is 100 (e.g., sleeping), the driver has transitioned from drowsiness to sleep. Therefore, Figure 3 The driver drowsiness scores shown in do not include zero and 100. As described above, OCS 110 can determine the driver's drowsiness by monitoring the driver's eyes and head posture. The driver's drowsiness score increases as the count of eyelid closures and noddings occurs over a given time period increases. For example, OCS 110 can determine the percentage of eyelid closure over the pupil over time (PERCLOS), which distinguishes slow eyelid closure or drooping from blinking. As PERCLOS increases, the driver's drowsiness score can be scaled from 10 to 90 to correspond to the percentage of time that the eyes were occluded during the sampled time period, such as Figure 3 shown. Supervision score

[0032] Figure 5 An example of a supervision score 112 determined by the controller circuit 102 based on scores of the driver supervision metrics 118 is shown. The supervision score 112 indicates whether the driver is ready to resume control of the vehicle 108 if the vehicle system exits the autonomous driving mode or responds to changing traffic conditions that the vehicle system may not anticipate. The controller circuit 102 may determine the supervision score 112 by determining the maximum value of any of the driver supervision metrics 118. Figure 5 In the example shown in , it is determined that the driver has high situational awareness, which has a score of zero, the hands-off time is less than a first threshold (e.g., less than one second), which has a score of zero, the driver is present and in the correct position in the driver's seat, which has a score of zero, and a score of zero for the driver's sleep state indicates a high level of driver alertness. The controller circuit 102 selects the driver supervision metric 118 with the maximum score, which is zero in this example.

[0033] Figure 6 Another example of a supervision score 112 determined by the controller circuit 102 is shown, wherein it is determined that the driver exhibits cognitive overload and has a situational awareness score of 50, as determined by the buffer algorithm described above. In this example, the buffer value is 1, resulting in a score of (2-1)*50=50. As described above, the OCS 110 determines that the hands-off time is greater than 5 seconds, which has a score of 50, and determines that the driver's driver drowsiness score is 20 based on PERCLOS. The scores of the remaining driver supervision metrics 118 are zero. In this example, the controller circuit 102 selects the driver supervision metric 118 with the maximum score, which is 50.

[0034] Figure 7Another example of a supervision score 112 determined by the controller circuit 102 is shown, where it is determined that the driver exhibits low situational awareness, with a score of 100, as determined by the buffer algorithm described above. In this example, the buffer value is zero, resulting in a score of (2-0)*50=100. As described above, the OCS 110 determines that the hands-off time is greater than 10 seconds, with a score of 100, and determines that the driver is not in the driver's seat or is not in the proper position, with scores of 100 each. The scores of the remaining driver supervision metrics 118 are zero. In this example, the controller circuit 102 selects the driver supervision metric 118 with the maximum score, which is 100.

[0035] In addition to selecting the maximum value of the driver supervision metric 118, the controller circuit 102 may use other techniques to determine the supervision score 112. For example, the controller circuit 102 may determine an average value of all driver supervision metrics 118 and determine the supervision score 112 based on the average value, or may determine a maximum value of the driver supervision metric 118 over a period of time to reduce the variation of the supervision score 112.

[0036] As described below, the controller circuit 102 may indicate a driver perception state 114 based on a supervision score 112 on a display 116 in the driver's field of view. monitor

[0037] Figure 8 An example of a display 116 is shown that is a light bar that runs the length of the dashboard just below the windshield of the vehicle 108. The controller circuit 102 can change the color of the light illuminated on the light bar based on the supervision score 112 to indicate the driver perception state 114 to the driver, as will be described in more detail below. The light bar may include different light emitting diodes (LEDs) that are configured to illuminate different colors of light based on signals received from the controller circuit 102. Figure 8 In the example shown in , the controller circuit 102 indicates the driver perception state 114 by illuminating a green light on the light bar corresponding to a zero supervision score 112. As the supervision score 112 increases from zero to 100, the controller circuit 102 may indicate the driver perception state 114 by changing the color of the light on the light bar from green to red (including other colors indicating scores between zero and 100).

[0038] Fig. 9An example of a supervision score 112 and an associated driver perception state 114 is shown, ranging from green with a wavelength (λ) of 550 nanometers (nm) to red with a wavelength of 700 nm. Other colors of the visible spectrum between green and red are indicated by their respective wavelengths. For example, light with a wavelength of 580 nm is yellow, while light with a wavelength of 620 nm is orange. The light color can be predetermined by the vehicle manufacturer or selected by the vehicle operator based on the operator's preference. A green driver perception state 114 indicates that the driver is ready to take over control of the vehicle in a short period of time, while a red driver perception state 114 indicates a serious vehicle safety issue, where the driver may not be able to react in time to avoid an accident if the autonomous driving mode is unable to respond to the threat.

[0039] When the supervision score 112 exceeds the first threshold, the controller circuit 102 may pulse the light displayed by the light bar, such as Fig. 9 The controller circuit 102 may pulse the light at a first frequency (e.g., at intervals of two seconds or half a Hertz (0.5 Hz)). The first threshold may be user defined and may be set at Fig. 9 In the example shown in , the first threshold is a supervision score 112 of 80, at which point the driver supervision score 112 indicates that the driver exhibits low attention. When the supervision score exceeds the first threshold for a time that exceeds the second threshold, the controller circuit can increase the frequency of the light pulsation. For example, when the supervision score 112 exceeds 80 for a period of 20 seconds, the controller circuit 102 can increase the pulsation frequency of the light from 0.5 Hz to 1 Hz.

[0040] Fig.10 An example of a light bar extending along the length of a door of a vehicle 108 is shown. The light bar may extend along the length of both the left and right doors of the vehicle 108 and may be in the driver's field of view when the driver is looking to the left or right side of the vehicle 108. The controller circuit 102 may illuminate the door mounted light bar with the same color and effect as the light bar located on the instrument panel as described above. The controller circuit 102 may use monitoring data 104 from the OCS 110 to determine whether the driver's gaze is directed toward the left or right door and, based on the gaze direction, cause the light pulses on the light bar to chase 120, thereby directing the driver's gaze toward the front of the vehicle 108, such as Fig.10. Chasing 120 is an electrical application where adjacent LEDs are cycled on and off to create the illusion of light moving in a particular direction along the light bar. The controller circuit 102 can use different chasing patterns to direct the driver's gaze toward the front of the vehicle 108, for example, gradually lighting up larger and larger clusters of LEDs as the chase 120 moves along the light bar, or lighting up larger sections of LEDs as the chase 120 moves along the light bar.

[0041] Fig.11 2 is a flow chart illustrating an example logic flow 200 executed by the controller circuit 102. The logic flow begins at 202 with indicating the driver perception state 114 on the display 116 upon vehicle ignition and ends at 214 with an increase in the pulsation frequency. In this example, at 202, when the driver actuates the vehicle ignition switch within the vehicle 108, the controller circuit 102 determines the supervision score 112 based on the score of the driver supervision metric 118, as described above. Figure 8-Figure 10 As shown in , the controller circuit 102 indicates the driver perception state 114 on the light bar and determines whether the supervision score 112 is greater than a threshold at 204. If the supervision score 112 is greater than the threshold (e.g., greater than 80), then at 206, the controller circuit 102 starts a timer.

[0042] At 208, the controller circuit 102 begins pulsing light on the light bar at a first frequency of 0.5 Hz to warn the driver of the decreased driver attention. At 210, the controller circuit 102 determines whether the supervision score 112 remains greater than the threshold. If the supervision score 112 drops below the threshold, the controller circuit 102 indicates a new driver perception state 114 corresponding to the new supervision score 112 on the light bar. If the supervision score 112 remains above the threshold, then at 212, the controller circuit 102 determines whether the time is greater than a time threshold (e.g., 20 seconds). When the time exceeds the time threshold of 20 seconds, at 214, the controller circuit 102 increases the frequency of the pulsation to 1 Hz to further warn the driver of the decreased driver attention.

[0043] The controller circuit 102 may use other warning methods besides the color and effects of the light, such as voice warnings and tactile warnings or vibration warnings. For example, a voice warning may call the driver to warn the driver to increase their attention. A tactile warning may be applied to the driver's seat to remind the driver of their supervisory responsibilities. These other notification methods may also be upgraded by increasing the intensity, for example, by increasing the volume of the voice notification, increasing the frequency of repeated voice notifications, and vibrating the driver's seat at an increased frequency or causing the seat to vibrate toward the front of the vehicle. Example Method

[0044] Fig.11 An example method 300 performed by the system 100 is shown. For example, the controller circuit 102 configures the system 100 to perform operations 302 to 308 by executing instructions associated with the controller circuit 102. The operations (or steps) 302 to 308 are performed, but are not necessarily limited to the order or combination of operations shown herein. In addition, any of one or more operations may be repeated, combined, or reorganized to provide other operations.

[0045] Step 302 includes "receiving monitoring data". This can include receiving monitoring data 104 from a driver monitoring sensor 106 installed on a vehicle 108 via a transmission link using the controller circuit 102. As described above, the driver monitoring sensor 106 may include multiple sensors that detect various aspects or behaviors of the driver, and may be a component of the OCS110. As described above, the driver monitoring sensor 106 may include a 2D camera and a 3D camera that captures a video image of the driver, and the OSC 110 determines whether the seat is occupied by the driver or by an object based on the image. The driver monitoring sensor 106 may also include a radar sensor that detects the presence of the driver in the driver's seat. As described above, the OCS110 may use a camera to detect the body and head position of the driver, and detect the direction of the driver's eye gaze. The driver monitoring sensor 106 may also include a seat pressure sensor that detects when the driver is sitting on the seat, and may also include a steering torque sensor and a capacitive steering wheel sensor to detect whether the driver's hands are on the steering wheel. Monitoring data 104 may be periodically updated by OCS 110 and transmitted to controller circuitry 102 to ensure that controller circuitry 102 can accurately determine supervision score 112 .

[0046] Step 304 includes "Determine a driver supervision metric score". This may include determining, using the controller circuit 102, a driver supervision metric score based on the monitoring data 104 received from the OCS 110. The driver supervision metrics include situational awareness, distraction level, hand object detection, hands-off steering wheel time, driver presence in the driver's seat, driver body position in the driver's seat, driver sleep state, and driver drowsiness, such as Figure 3 As shown in and described above.

[0047] As described above, the controller circuit 102 determines a situational awareness score based on the driver's glance behavior detected by the camera. As described above, the situational awareness score ranges from zero for high situational awareness to 100 for low situational awareness and is based on the attention buffer. As described above, the controller circuit 102 determines the distraction level based on the camera's detection of the driver's gaze direction toward the area of ​​interest on the road. As described above, the distraction level score is zero for a non-distracted driver and increases to 100 for a driver who looks outside the area of ​​interest for more than 2 seconds.

[0048] The controller circuit 102 determines hand-on object detection based on the camera detecting whether the driver is holding a mobile phone or other object that may be the object of the driver's attention. When no hand-on object is detected, the hand-on object score is zero; and when the object is detected in the hand, the hand-on object score is 100. The controller circuit 102 determines the hands-off steering wheel time based on the timer and the steering torque and capacitive steering wheel sensors. The hands-off steering wheel time score ranges from zero to 100 and is determined by multiplying the time in seconds by 10.

[0049] The controller circuit 102 determines whether the driver is present in the driver's seat and whether the driver is properly positioned in the driver's seat based on the monitoring data 104 from the camera, radar sensor, and seat pressure sensor. The driver presence score is zero for detecting the driver in the driver's seat and 100 for not detecting the driver. The driver body position score is zero for the driver in the correct position and 100 for the driver being detected out of position.

[0050] As described above, the controller circuit 102 determines whether the driver is awake or asleep based on a camera that detects the driver's eye and head posture and the amount of time the eyelids are closed. The driver sleep state score is 0 when fully awake and 100 when asleep. As described above, the controller circuit 102 can determine the driver's sleepiness based on the driver's eye and head posture and PERCLOS. As described above, the driver sleepiness score is between 10 and 90 and reflects the percentage of time the eyes are blocked.

[0051] Step 306 includes “DETERMINE SUPERVISION SCORE.” This includes determining supervision score 112 based on a score of driver supervision metric 118 using controller circuit 102. As described above, controller circuit 102 may determine supervision score 112 based on a maximum value of driver supervision metric 118 or an average value of driver supervision metric 118.

[0052] Step 308 includes "indicating a driver perception state". As described above, this can include indicating the driver perception state 114 on a light bar using the controller circuit 102. The driver perception state 114 is based on the supervision score 112 and can be displayed by lighting the light bar with different colors. The light bar can be located in the driver's field of view, for example, extending along the dashboard and doors of the vehicle 108. As described above, the controller circuit 102 can change the color and lighting effects of the lights to warn the driver when the controller circuit 102 determines that the supervision score is greater than a threshold indicating that the driver's attention has decreased. As described above, the controller circuit 102 can pulse the light on the light bar to warn the driver, and voice warnings and vibration warnings can be used in addition to the lighting effects. As described above, the controller circuit 102 can determine when the driver is looking toward the left and right sides of the vehicle, and cause the light pulses displayed on the light bar to chase 120 to direct the driver's attention toward the front of the vehicle. Example

[0053] In the following sections, examples are provided.

[0054] Example 1. A system comprising: a controller circuit configured to:

[0055] Receiving monitoring data from a driver monitoring sensor configured to monitor a driver of the vehicle while the vehicle is operating in an autonomous driving mode; determining scores of one or more driver supervision metrics based on the monitoring data, each of the driver supervision metrics at least partially indicating whether the driver is supervising operation of the vehicle; determining a supervision score based on the scores of the one or more driver supervision metrics, the supervision score indicating whether the driver is ready to resume control of the vehicle; and indicating a driver perception state on a display in the driver's field of view based on the supervision score.

[0056] Example 2. The system of the preceding example, wherein the driver monitoring sensor comprises one or more of: a two-dimensional camera, a three-dimensional camera, a steering torque sensor, and a capacitive steering wheel sensor.

[0057] Example 3. The system of any of the preceding examples, wherein the driver supervision metric comprises one or more of: situational awareness, distraction level, hand object detection, hands-off time, driver presence, driver body position, driver sleep state, and driver drowsiness.

[0058] Example 4. The system of any of the preceding examples, wherein the controller circuit determines the supervision score by determining a maximum value of any of the driver supervision metrics.

[0059] Example 5. The system of any of the preceding examples, wherein the situational awareness metric is based in part on one or more of: a driver's glance direction relative to a forward roadway on which the vehicle is traveling, a driver's glance duration, and a driver's glance count.

[0060] Example 6. The system of any of the preceding examples, wherein the controller circuit is further configured to determine a first supervision score indicating that the driver has one or more of: high situational awareness, hands-off time less than a first threshold, the driver is in the driver's seat and in a correct position, and a high level of alertness.

[0061] Example 7. The system of any of the preceding examples, wherein the controller circuit is further configured to determine a second supervision score indicating that the driver has one or more of: cognitive overload and hands-off time greater than a second threshold.

[0062] Example 8. The system of any of the preceding examples, wherein the controller circuit is further configured to determine a third supervision score indicating that the driver has one or more of: low situational awareness, hands-off time greater than a third threshold, the driver not in the driver's seat or not in a correct position in the driver's seat, and a low alertness level.

[0063] Example 9. The system of any of the preceding examples, wherein the display includes a light bar, and the controller circuit is further configured to change a color of the displayed light based on the supervision score.

[0064] Example 10. The system of any of the preceding examples, wherein the controller circuit is further configured to pulse the light displayed by the light bar when the supervision score exceeds a first threshold.

[0065] Example 11. The system of any of the preceding examples, wherein the controller circuit is further configured to increase the frequency of the light pulsations when the supervision score exceeds the first threshold for a time exceeding a second threshold.

[0066] Example 12. The system of any of the preceding examples, wherein the controller circuit is further configured to determine a gaze direction of the driver and to chase the light pulses based on the gaze direction to direct the driver's gaze toward the front of the vehicle.

[0067] Example 13. The system of any of the preceding examples, wherein the light bar extends along a length of a dashboard of the vehicle.

[0068] Example 14. The system of any of the preceding examples, wherein the light bar further extends along the length of the left and right doors of the vehicle.

[0069] Example 15. A method includes: receiving, with a controller circuit, monitoring data from a driver monitoring sensor configured to monitor a driver of a vehicle; determining, with the controller circuit, scores of one or more driver supervision metrics based on the monitoring data; determining, with the controller circuit, a supervision score based on the scores of the one or more driver supervision metrics; and indicating, with the controller circuit, a driver perception state on a display in a field of view of the driver based on the supervision score.

[0070] Example 16. The method of the preceding example, comprising determining a supervision score by determining a maximum value of any of the driver supervision metrics.

[0071] Example 17. The method of any of the preceding examples, wherein the driver supervision metric comprises one or more of: situational awareness, distraction level, hand object detection, hands-off time, driver presence, driver body position, driver sleep state, and driver drowsiness.

[0072] Example 18. The method of any of the preceding examples, comprising determining a supervision score by determining a first supervision score indicating that the driver has one or more of: high situational awareness, hands-off time less than a first threshold, the driver is in the driver's seat and in a correct position, and a high level of alertness.

[0073] Example 19. The method of any of the preceding examples, comprising determining a supervision score by determining a second supervision score indicating that the driver has one or more of: cognitive overload and hands-off time greater than a second threshold.

[0074] Example 20. The method of any of the preceding examples, comprising determining a supervision score by determining a third supervision score indicating that the driver has one or more of: low situational awareness, hands-off time greater than a third threshold, the driver not being in the driver's seat or not being in a correct position in the driver's seat, and a low alertness level. in conclusion

[0075] Although various embodiments of the present disclosure have been described in the foregoing description and shown in the accompanying drawings, it should be understood that the present disclosure is not limited thereto, but can be implemented in various ways within the scope of the following claims. From the foregoing description, it will be apparent that various changes can be made without departing from the spirit and scope of the present disclosure as defined by the following claims.

[0076] Unless the context clearly dictates otherwise, the use of "or" and grammatically related terms represents unlimited, non-exclusive alternatives. As used herein, a phrase referring to "at least one of" a list of items refers to any combination of these items, including individual members. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

Claims

1. A system, comprising: A controller circuit configured to: determining whether a supervision score of a driver of the vehicle indicates a decrease in driver attention; Determining that the driver's gaze direction is not toward the travel direction of the vehicle; as well as Based on an indication of decreased driver attention, a chase of light pulsations displayed in a light bar arranged in the driver's gaze direction is caused, the chase of light pulsations comprising adjacent light emitting diodes (LEDs) in the light bar being sequentially illuminated toward the travel direction of the vehicle.

2. The system of claim 1, wherein: The controller circuit is further configured to change the color of the light pulsation based on the supervision score.

3. The system of claim 1, wherein: The controller circuit is further configured to increase the frequency of the light pulsations in response to the supervision score indicating that the driver's decreased attention has persisted for a predefined period of time.

4. The system of claim 1, wherein: The light bar extends toward a dashboard of the vehicle.

5. The system of claim 4, wherein: The light bar extends along the instrument panel.

6. The system of claim 1, wherein: The light bar is arranged along the length of the left door or the right door of the vehicle.

7. The system of claim 6, wherein: Causing the chasing of the light pulsation includes causing one or more lights of the light bar to appear to move toward the front of the vehicle.

8. The system of claim 1, wherein: The tracking of the light pulsations serves to guide the driver's gaze direction along the light strip and toward the driving direction of the vehicle.

9. A method comprising: determining that a supervision score of a driver of the vehicle indicates a decrease in driver attention; Determining that the driver's gaze direction is not toward the travel direction of the vehicle; as well as Based on an indication of decreased driver attention, chasing of light pulsations displayed in a light bar arranged in the driver's gaze direction is caused, the chasing of light pulsations comprising sequentially lighting up adjacent portions of the light bar toward the travel direction of the vehicle.

10. The method of claim 9, further comprising: Based on the supervision score, the color of the light pulsation is changed.

11. The method of claim 9, further comprising: Based on determining that the supervision score indicates that the driver's decreased attention has persisted for a predefined period of time, the frequency of the light pulsing is increased.

12. The method of claim 9, wherein: The light bar extends toward and along an instrument panel of the vehicle.

13. The method of claim 9, wherein: The light bar is arranged along the length of the left door or the right door of the vehicle.

14. The method of claim 9, wherein: The chasing of the light pulsations is effectively used to direct the driver's gaze direction along the light bar and towards the driving direction of the vehicle.

15. The method of claim 14, wherein: Causing the chasing of the light pulsation includes causing one or more lights of the light bar to appear to move toward the front of the vehicle.

16. The method of claim 15, further comprising: The gaze direction is determined based on at least one of a positioning of the driver's pupils or a head posture of the driver.

17. A non-transitory computer-readable storage medium comprising instructions that, when executed, cause at least one processor to: determining whether a supervision score of a driver of the vehicle indicates a decrease in driver attention; Determining that the driver's gaze direction is not toward the travel direction of the vehicle; as well as Based on an indication of decreased driver attention, a chase of light pulsations displayed in a light bar arranged in the driver's gaze direction is caused, the chase of light pulsations comprising adjacent light emitting diodes (LEDs) in the light bar being sequentially illuminated toward the travel direction of the vehicle.

18. The non-transitory computer-readable storage medium of claim 17, wherein: The chasing of the light pulsations is effectively used to direct the driver's gaze direction along the light bar and towards the driving direction of the vehicle.

19. The non-transitory computer-readable storage medium of claim 18, wherein: Causing the chasing of the light pulsation includes causing one or more lights of the light bar to appear to move toward the front of the vehicle.

20. The non-transitory computer-readable storage medium of claim 19, wherein: The instructions cause the processor to determine the gaze direction based on at least one of a pupil location of the driver or a head posture of the driver.

Images