Gaze tracking and object detection in advanced driver assistance systems

Abstract

Disclosed herein are system, method, and device embodiments for determining the mental state of a driver using gaze tracking and object detection. Specifically, an Advanced Driver Assistance System may determine if the driver's gaze is associated with spatial locations of objects of interest in the surrounding environment. The surrounding environment may be represented with a 360-degree, three-dimensional model generated by the external-facing cameras and other sensors. The driver's gaze direction may be represented by a heatmap derived based on the driver's eye movements. Whether or not the driver's gaze is associated with a spatial location of an object may be used as evidence about the driver's mental state. The system may perform appropriate actions based on the mental state including generating alerts, engaging mechanical systems, and adjusting parameters in ancillary safety systems to increase or decrease tolerance. The mental state may be tracked throughout a driving session and stored in a driver profile.

Description

GAZE TRACKING AND OBJECT DETECTION IN ADVANCED DRIVER ASSISTANCE SYSTEMSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No.63 / 617,838, by Osherov, et al., titled “Determining Driver Mental State Using Gaze Tracking and Object Detection,” filed January 5, 2024; U.S. Provisional Application No. 63 / 617,858, by Osherov, et al., titled “Policy Adjustment Based on Driver Attentiveness,” filed January 5, 2024; U.S. Provisional Application No. 63 / 617,973, by Osherov, et al., titled “Detecting Driver Impairment Based on Eye Scanning Movements,” filed January 5, 2024; U.S. Provisional Application No. 63 / 618,051, by Osherov, et al., titled “Using Polarized Light for Driver Monitoring,” filed January 5, 2024; U.S. Provisional Application No. 63 / 618,080, by Osherov, et al., titled “Classifying Traffic Control Objects Based on Gaze Tracking and Object Detection,” filed January 5, 2024; and U.S.Provisional Application No. 63 / 618,098, by Osherov, et al., titled “Rolling Calibration of Gaze Detection Based on Gaze Tracking and Object Detection,” filed January 5, 2024, all of which are hereby incorporated by reference in their entirety.BACKGROUND

[0002] An Advanced Driver Assistance System (“ADAS”) may provide safety features in vehicles. The ADAS may observe environmental conditions and, when circumstances demand, regulate the vehicle’s mechanical systems, cause a driver to take immediate action, or adjust a driver’s behavior. The ADAS may assume control of the vehicle in extreme cases — e.g., when a driver is incapacitated.

[0003] Specific ADAS features include autonomous driving, adaptive cruise control, blind-spot detection, lane departure warnings, collision avoidance, evasive maneuvering, parking assistance, etc. An ADAS may rely on sensors and cameras to gather data about the surrounding environment and object-detection technologies to draw conclusions about that environment. Sensor data may include images captured by cameras and processed using computer vision techniques. Sensor data may also include data captured with remote sensing technologies including radio detection and ranging (“RADAR”) and lightdetection and ranging (“LIDAR”), ultrasonic sensors, weather-reading devices, global positioning system (“GPS”) components, and other suitable sensors. Other ADAS functions rely upon on-board, mechanical systems — e.g., stability and traction control, anti-lock brakes, tire-pressure monitoring, etc.

[0004] Traditionally, a driver operated a vehicle’s mechanical systems — / .< ., a human driver actually drove the car. More recently, autonomous driving systems have been developed that offload driving responsibilities from the driver onto the ADAS, which acts in concert with the vehicle’s mechanical systems (e.g., steering, braking, accelerating, etc.). Autonomous driving systems have moved towards a hybrid approach — in certain operating modes, the driver continues to shoulder many driving responsibilities with assistance from the ADAS in providing blind-spot detection, parking assistance, lane correction, etc. In other operating modes, the ADAS may operate a vehicle at a nearly fully autonomous level and ask the driver to intervene only when an emergency arises.

[0005] Conceptually, automated driving systems may operate across different levels of automation. The level may determine how much control the driver has over the vehicle at any given time. Some levels may require that the driver’s hands remain on the steering wheel in case the need arises to perform an immediate, manual correction — this may be referred to as a “eyes-on, hands-on” mode. Some levels may require that the driver’s eyes remain on the road, but their hands may be removed from the steering wheel — this may be referred to as an “eye-on, hands-off’ mode. A fully autonomous driving mode may obviate the need for the driver to pay attention at all — this may be referred to as an “eyes- off, hands-off’ mode. In the eyes-off, hands-off mode, a driver may be able to read, work on a computer, play cards with other passengers, and otherwise fully disengage from responsibilities in driving the vehicle.

[0006] To further quantify these modes of operation, standards organizations have published formalized automated driving levels. For example, the Society of Automotive Engineers (“SAE”) defines a 6-level taxonomy ranging from Level 0 (No Automation) to Level 5 (Full Automation) in SAE Standard J3016. Under this taxonomy, Levels 0 through 2 include features that support a driver operating a vehicle — e.g., automatic emergency braking, blind spot warning, lane departure warnings, lane centering, and adaptive cruise control. Levels 3-5 include features for automated driving. Under Level 3, a human may need to intervene at any time. Level 4 and Level 5 cover fully automatedscenarios, with Level 4 being geo-fenced e.g., a local driverless taxi) and Level 5 providing for autonomous operation anywhere in all conditions.

[0007] Various other safety standards and guidelines cover ADAS-related technologies. Some standards are promulgated by governing bodies — e.g., the European Union has promulgated Regulation (EU) 2019 / 2144 of the European Parliament and of the Council covering driver drowsiness and attention warning systems. Other private regulatory bodies and certification agencies publish standards, track requirements, give star rankings, etc. Compliance with these standards may be essential in driving vehicle sales.

[0008] Drivers may attempt to operate a vehicle while experiencing a fatigue-based impairment or a substance-based impairment. Such impaired driving may endanger the driver and others on the road. As a result, there is a need for automated driving systems to detect a driver’s impairment level and perform at a corresponding automation level to improve safety.

[0009] In the “eyes-on” modes, driver monitoring systems can track a person’s eye movements to determine the person’s attentiveness, or lack thereof, regarding the outside environment. To detect eye movements, driver monitoring systems can be equipped with interior-facing camera systems that use unpolarized light illuminators. Such interiorfacing camera systems can be used to capture a high-quality image of the person’s pupil by reflecting the unpolarized light off the person’s eye. However, people often wear sunglasses that may interfere with the camera systems. Specifically, polarized sunglasses may impede unpolarized light from the illuminator, thus inhibiting a camera system from discerning a position of the person’s pupil.

[0010] One difficulty of object recognition in driver assistance technology is recognizing which traffic light at an intersection corresponds to the lane of the driver vehicle. Different traffic lights may correspond to different vehicle lanes in that each traffic light may be instructing vehicles in a particular lane. The system may recognize that there are traffic lights, but not confidently determine which traffic light corresponds to a lane.

[0011] Reliably determining the gaze direction of a driver is a challenge in vehicle systems using ADAS and autonomous vehicle (“AV”) systems. Calibrating a system to reliably determine the direction of a user’s gaze allows the system to determine which objects in the field of view of the driver are in line with the gaze direction and determine which object the driver is looking at.SUMMARYDetermining Driver Mental State Using Gaze Tracking and Object Detection

[0012] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for determining the mental state of a driver using gaze tracking and object detection. An ADAS may determine if the driver’s gaze is associated with the spatial location of objects of interest in the surrounding environment and use this information as evidence about the driver’s mental state. The ADAS may perform appropriate actions based on the mental state including generating alerts, engaging mechanical systems, and adjusting parameters in ancillary safety systems to increase or decrease tolerance.Policy Adjustment Based on Driver Attentiveness

[0013] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for adjusting vehicle policies based on a driver’s awareness state determined using gaze tracking and object detection. The awareness state may indicate driver attention level, stress level, trust level, fatigue, intoxication, etc. By monitoring driver engagement and dynamically controlling policies and parameters, the technique maximizes driving performance across a range of systems while enhancing safety.

[0014] In particular, the approach may be applied to supervised-driving scenarios (Level 3) in which a human driver supervises, monitors, and oversees vehicle operation. In such a mode of operation, the ADAS may operate differently and intelligently based on the awareness state of the driver.Detecting Driver Impairment Based on Eye Scanning Movements

[0015] Accordingly, it is desirable to design a vehicle system that can detect a driver’s impairment level based on eye scanning movements over a period of time and can take appropriate corrective action based on the driver’s impairment level.

[0016] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for detecting driver impairment, such as fatigue-based impairment or substance-based impairment, based on eye scanning movements over a period of time. An ADAS may determine howlong the driver’s gaze is associated with the spatial locations of objects of interest in the surrounding environment and use this information as evidence about the driver’s impairment level. The ADAS may perform appropriate actions based on the impairment level including generating alerts, engaging mechanical systems, and adjusting parameters in ancillary safety systems to increase or decrease tolerance.Using Polarized Light for Driver Monitoring

[0017] Accordingly, it is desirable to design a camera system that can discern a position of a person’s pupil even if the person is wearing polarized sunglasses. Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for capturing an image for monitoring at least one person in a vehicle despite the presence of a polarized surface obstructing an eye of the person.

[0018] In a first embodiment, a camera system for monitoring at least one person in a vehicle can include an illuminator, a polarizing optical device, a camera, and a processor coupled to the camera. The illuminator can emit a light beam along an optical path. The polarizing optical device can be disposed in the optical path of the light beam. The polarizing optical device can perform polarizing of the light beam to generate a polarized light beam. The polarized light beam can penetrate a polarized surface. The camera can capture an image of the polarized light beam. The processor can conduct monitoring of at least one person in a vehicle based on the image.

[0019] A camera apparatus and method are also disclosed.

[0020] Further embodiments, features, and advantages of the invention, as well as the structure and operation of the various embodiments, are described in detail below with reference to accompanying drawings.Classifying Traffic Control Objects Based on Gaze Tracking and Object Detection

[0021] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof for classifying detected objects based on gaze tracking of a driver. In an embodiment, a method provides for traffic light identification. In the method, a first image data of an eye of a driver is captured from an internal camera. A gaze direction of the eye of the driver is determined based on the first image data. Substantially simultaneous with capture of the first imagedata, the second image data is also captured from an external camera of a vehicle. A traffic control object is detected in the second image data. Based on the gaze direction, whether the driver is looking at the traffic control object is determined. A lane where the vehicle is located contemporaneous with the capture of the first and second image data is determined. When the driver is determined to be looking at the traffic control object, data indicating that the traffic control object manages traffic on the lane of the vehicle is generated.

[0022] In some embodiments, this process may be repeated at a single lane for a plurality of drivers until a threshold number of data points are generated for the traffic control object and corresponding lane. In this way, once the threshold number of data points have been generated, the system may crowdsource the generated data by compiling the data points into a dataset that can be relied on to indicate the traffic control object manages traffic for the corresponding lane.

[0023] Also disclosed are system, apparatus, device, method and / or combinations and sub-combinations thereof.

[0024] Further embodiments, features, and advantages of the invention, as well as the structure and operation of the various embodiments, are described in detail below with reference to accompanying drawings.Rolling Calibration of Gaze Detection Based on Gaze Tracking and Object Detection

[0025] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof for rolling calibration of a driver’s gaze direction of a driver based on gaze tracking and object detection. In an embodiment, a method provides for calibrating a driver monitoring system and an object system for the driver of a vehicle. In the method, a first image of an eye of a driver is captured from an internal camera. A gaze direction of the eye of the driver is determined based on the first image. Substantially simultaneous with capture of the first image data, the second image data is also captured from an external camera of a vehicle. The external image may capture a known object, such as an object along a route the driver frequents. The system determines that the driver is looking in the direction of the object and adjusts the calibration parameters in response.

[0026] In some embodiments, the objects used for calibration may be known objects routinely in the driver’s field of view (“FOV”) based on location or map data. The image may be captured while the driver is operating the vehicle, to perform rolling calibration of the system. Using known objects routinely in the driver’s FOV allows the system to continuously improve calibration by repeatedly determining the gaze direction of the driver while they are operating the vehicle on a routinely driven route.

[0027] Also disclosed are system, apparatus, device, method and / or combinations and sub-combinations thereof.

[0028] Further embodiments, features, and advantages of the invention, as well as the structure and operation of the various embodiments, are described in detail below with reference to accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS / FIGURES

[0029] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments of the present disclosure and, together with the description, further serve to explain the principles of the disclosure and to enable a person skilled in the arts to make and use the embodiments.

[0030] FIG. l is a block diagram of an ADAS, according to some embodiments.

[0031] FIG. 2A illustrates an environment in which an ADAS operates, according to some embodiments.

[0032] FIG. 2B illustrates an environment in which an ADAS operates from a perspective inside of a vehicle, according to some embodiments.

[0033] FIG. 3 A is a table illustrating exemplary autonomous driving levels and mentalstate based actions performable at each level, according to some embodiments.

[0034] FIG. 3B is a table illustrating exemplary supervised-driving actions that vary based on a driver’s awareness state, according to some embodiments.

[0035] FIG. 3C is a table illustrating exemplary supervised-driving actions that vary based on a driver’s trust level, according to some embodiments.

[0036] FIG. 4A illustrates a scene as captured by outward-facing cameras, according to some embodiments.

[0037] FIG. 4B illustrates a scene as captured by an object detecting system, according to some embodiments.

[0038] FIG. 5 A illustrates a scene captured by outward-facing sensors and a heatmap representing a driver’s gaze direction as captured by inward-facing sensors, according to some embodiments.

[0039] FIG. 5B illustrates a scene captured by outward-facing sensors including a heatmap representing a driver’s gaze direction, according to some embodiments.

[0040] FIG. 6 illustrates a model of a Purkinje image, according to some embodiments.

[0041] FIG. 7 illustrates a method for determining the mental state of a driver using gaze tracking and object detection, according to some embodiments.

[0042] FIG. 8 illustrates a method for tracking a driver alertness value using gaze tracking and object detection, according to some embodiments.

[0043] FIG. 9 illustrates a method for adjusting an ancillary system based on a driver’s rational response or lack thereof to a high danger object, according to some embodiments.

[0044] FIG. 10 illustrates a method for adjusting policy parameters based on a determined driver awareness state, according to some embodiments.

[0045] FIG. 11 illustrates a method for detecting impairment, such as fatigue or intoxication, based on eye scanning movements over a period of time, according to some embodiments.

[0046] FIG. 12 illustrates a method for tracking a driver alertness value using gaze tracking and object detection over a period of time, according to some embodiments.

[0047] FIG. 13 illustrates a photograph of an unpolarized illumination of polarized sunglasses reflecting unpolarized light, according to some embodiments.

[0048] FIG. 14 illustrates a camera system with a polarizing optical device, according to some embodiments.

[0049] FIG. 15 illustrates a camera system with a rotating polarizing optical device, according to some embodiments.

[0050] FIG. 16 illustrates a camera system with a plurality of polarizers rotated relative to each other, according to some embodiments.

[0051] FIG. 17 illustrates a method for capturing an image for monitoring at least one person in a vehicle, according to some embodiments.

[0052] FIG. 18A depicts a vehicle at an intersection with multiple traffic control objects, according to some embodiments.

[0053] FIG. 18B depicts the perspective from inside the vehicle at an intersection with multiple traffic control objects, according to some embodiments.

[0054] FIG. 18C illustrates an image captured at an intersection with multiple traffic controls objects, according to some embodiments.

[0055] FIG. 19 depicts a flowchart illustrating a method for classifying traffic control objects based on gaze direction and object detection, according to some embodiments.

[0056] FIGS. 20A-20C illustrate environments in which an ADAS may perform rolling calibration for a driver of a vehicle, according to some embodiments.

[0057] FIG. 21 A illustrates the field of view of the driver inside of a vehicle, according to some embodiments.

[0058] FIG. 2 IB illustrates a scene that may be captured by an ADAS of a vehicle for object detection and gaze tracking, according to some embodiments.

[0059] FIG. 22 A illustrates a method for calibration of a driver’ s gaze, according to some embodiments.

[0060] FIG. 22B illustrates a method for calibrating a driver’s gaze while the driver is moving their head, according to some embodiments.

[0061] FIG. 23 illustrates a computer system, according to exemplary embodiments of the present disclosure.

[0062] The present disclosure will be described with reference to the accompanying drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Additionally, the left-most digit of a reference number identifies the drawing in which the reference number first appears.DETAILED DESCRIPTION OF THE INVENTION

[0063] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof for determining the mental state of a driver using gaze tracking and object detection.

[0064] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof for adjusting policies within a vehicle based on a driver’s awareness state. The awareness state may be determined using gaze tracking and object detection.

[0065] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof for detectingdriver impairment, such as fatigue, based on eye scanning movements over a period of time.

[0066] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof for capturing an image for monitoring at least one person in a vehicle despite the presence of a polarized surface obstructing an eye of the person.

[0067] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof for classifying traffic control objects based on driver gaze tracking and object detection.

[0068] Provided herein are system, method, and / or computer program product embodiments, and / or combination and sub-combinations thereof for performing rolling calibration gaze detection for a driver based on gaze tracking and object detection. A driver may operate a vehicle using an ADAS and / or operate an AV. Providing assistance to the driver may involve the system determining the gaze direction of the driver for use in ADAS and / or AV systems. For example, in object detection and recognition, especially in scenarios where there may be moving objects and / or several objects near each other in the driver’s FOV. To reliably determine the location of a user’s gaze, ADAS and AV systems must be calibrated for individual drivers. Drivers may be monitored using an internal system that may include inward facing camera(s), e.g., cameras directed to capture images of the driver in a driver monitoring system (“DMS”). Additionally, externally facing cameras may be used to capture external images of the environment surrounding the driver such as in an external sensing system. Once calibrated, ADAS and AV systems may use the DMS in combination with the external sensing system to reliably determine the gaze direction of the driver. To calibrate the systems for a given driver, the system may use gaze direction techniques and known objects in the driver’s FOV to repeatedly determine a driver’s gaze direction. Additionally, the calibration may be performed on a rolling basis, such that the driver may operate the vehicle normally and while the system is being calibrated.

[0069] A driver may be operating a vehicles using an ADAS and / or operating an AV. In these scenarios, the ADAS provides the driver with instruction based on objects detected. Similarly, the AV may operate the vehicle according to the objects detected. In order to allow the systems to properly provide instruction and / or operate the vehicle, the computer system must determine which detected object may have an affect or manage traffic for thevehicle the driver is occupying. Specifically, when interacting with traffic control objects, such as traffic lights, traffic signs, lane markings, etc., the system may be able to reliably determine which traffic control object is managing traffic for the lane the vehicle is operating.

[0070] An ADAS may provide autonomous driving, adaptive cruise control, blind-spot detection, lane departure warnings, collision avoidance systems, stability and traction control, anti-lock brakes, tire-pressure monitoring, and other partially or fully autonomous driving features. To operate autonomously, an ADAS may identify objects and interpret the surrounding environment using object-detection techniques. Object detectors may identify relevant objects, including other vehicles, lanes, traffic lights, signs, pedestrians, animals, traffic barriers, debris, etc. The object detection system may determine a spatial location of an object including a position, a rate of movement (e.g., speed, velocity, acceleration), a direction of movement, and other suitable characteristics.

[0071] In one embodiment, an object detection system may fuse multiple camera images to build a contemporaneous, 360-degree, three-dimensional image of the surrounding environment. Additionally or alternatively, RADAR and / or LIDAR sensors may also be used to locate and identify objects in the surrounding environment to build a contemporaneous, 360-degree, three-dimensional model of the surrounding environment. The object detection system may employ multiple cameras, such as forward-facing, rearfacing, and side-view cameras to build the 360-degree understanding of the surrounding environment. By using multiple types of sensor data, the system may be resilient to failure of any one system — e.g., due to mechanical failures, adverse weather conditions, etc. The ADAS may use this real-time knowledge about the environment to drive decision-making across the range of safety features.

[0072] In addition to camera images, the ADAS may use the data from multiple sensor systems (e.g., RADAR and LIDAR) to build independent and robust models of the environment based on each sensor type. This use of multiple, independent sensor systems ( / .< ., camera and RADAR) provides true redundancy within the ADAS while ensuring that each system has a capable backup. The ADAS may use this real-time knowledge about the environment to drive decision-making across the range of safety features. In some embodiments, the model of the surrounding environment may cover a narrower field of view, e.g., a 180-degree forward-facing model or other suitable model of the environment may be used to represent the ambient environment.

[0073] An ADAS may employ a DMS to determine if the driver is alert and capable of performing driver-designated duties. For example, if operating in an “eyes-on, hands-on” mode, the DMS may track the driver’s hand position to ensure that the hands remain on the steering wheel. If the hands are not on the steering wheel, the system may generate an alert to request that the driver place their hands on the steering wheel. If the driver is not paying attention and the ADAS detects the possibility of an imminent, dangerous event, the ADAS may act in concert with a collision detection system to apply the brakes, generate a driver alert, or perform other suitable corrective action.

[0074] Some legacy tools may monitor a driver’s facial expressions to determine if a driver is asleep, incapacitated, or otherwise incapable of performing their responsibilities. A DMS may track driver’s eyes using a gaze tracker to ensure that the driver’s eyes are open and directed generally at the road. When the driver exhibits signs of drowsiness, e.g., when the driver’s eyes are closed for a particular length of time (e.g., 4 seconds), a DMS may generate a driver alert. In some circumstances where a driver is non- responsive, the ADAS may take over control of the vehicle, turn on the warning lights, slow the vehicle, and pull over into the shoulder. Thus, it is important for the ADAS to be able to capture images of a person’s eyes, even if the person’s eyes are obstructed by a polarized surface (e.g., polarized sunglasses).

[0075] The inventors recognize that the intersection of these technologies — / .< ., (1) detection of objects located outside of the vehicle and (2) tracking of the driver’s gaze direction — provides an opportunity to improve upon legacy technologies. Specifically, by coordinating gaze-direction tracking technologies inside the vehicle with object detection technologies that track objects outside of the vehicle, the ADAS may draw conclusions about the driver’s mental / awareness state. No legacy technique or system provides a solution to ascertain a driver’s awareness state by combining object detection and gaze tracking.

[0076] To train a reliable system, it may be advantageous to classify objects the driver and / or other drivers on the road frequently interact with while driving. For example, a detected object that the system may need to reliably detect and associate with the vehicle may be traffic control objects, e.g., traffic lights and traffic signs. In many instances, the driver may have more than one traffic control object in their FOV and be capable of determining which of the traffic control objects is managing traffic for the lane the vehicle is occupying. The ADAS via the DMS may track the driver’s eyes using a gazetracker to determine which object the object detection system the driver may be looking at when stopped at the intersection. Based on the object detection and gaze generation, the computer system may generate data associating the traffic control object (e.g., traffic light) with the lane of the driver and vehicle.

[0077] This process may be repeated for multiple lanes, traffic control objects, drivers, and / or vehicles. The system may determine a threshold number of repeated classifications before determining that ADAS and AV systems can confidently rely on a traffic control object to manage a corresponding lane in assisted and autonomous driving situations.

[0078] Specifically, by reliably determining the corresponding lane and managing traffic control objects using ADAS and DMS. More specifically, using a 360-degree, three- dimensional model of the surrounding environment generated by external-facing cameras and other sensors, an ADAS may determine if the driver’s gaze intersects with a traffic control object. In this manner, the ADAS may fuse first image data from the interiorfacing cameras or other sensors with second image data from the exterior-facing cameras, with the first and second image data captured simultaneously or substantially simultaneously. In other embodiments, the model of the surrounding environment may cover a narrower field of view, e.g., a 180-degree forward-facing model of the environment may be used or other suitable model or representation of the ambient environment. In one embodiment, the ADAS may determine a gaze intersection by casting a ray from an origin representing the driver’s position in the three-dimensional environment. The direction of the ray may be represented using a camera pose of the driver determined using gaze detection. In various embodiments, the pose reflecting the eye direction may be calculated with or without reference to a head position. When the ray or vector intersects with a bounding box surrounding an identified object for a threshold amount of time, the object may be considered “seen” by the driver. A bounding box may be a two- or three-dimensional rectangle or other polygon surrounding the object generated by the object detection system. Consider an example where the traffic control object is a STOP sign. As the vehicle approaches a STOP sign, an object detection system may identify the STOP sign in the ambient environment and use three-dimensional geometry to understand the positioning of the STOP sign relative to the vehicle. The object detection system may be able to identify different types of traffic control objects e.g., traffic light compared to a STOP sign or YIELD sign. Additionally, the object detection system may determine that the detected object is a traffic control objectcompared to another object that is not of interest in the particular situation, e.g., a person, other vehicle, bike, and / or other objects detectable by the object detection system. Simultaneously, a DMS may track the driver’s gaze as a pose (having a position and orientation) within the 360-degree, three-dimensional model of the environment. Using the pose, the ADAS may determine if the driver looks in the direction of the STOP sign. When the driver’s gaze intersects with a bounding box surrounding the STOP sign for a threshold amount of time, the ADAS may count the STOP sign as managing traffic for the lane of the driver and vehicle. The determination of the intersection may consider a reasonable tolerance to account for peripheral vision.

[0079] Impaired drivers may have a tendency to stare at the same object, while unimpaired drivers may have a tendency to scan with their eyes to different objects. Thus, the duration of time in which a driver’s gaze is fixated on an object may be indicative of a type of impairment. Therefore, pairing object detection with gaze tracking over a period of time can enable an ADAS to better monitor a driver’s impairment level.

[0080] The term “mental state” as used herein may measure a driver’ s frame of mind, mindset, or mood. Mental state may include characteristics such as attentiveness, alertness, vigilance, awareness, cognizance, watchfulness, perceptiveness, clearheadedness, impairment, fatigue, tiredness, drowsiness, etc. Mental state may also indicate drunkenness, intoxication, inebriation, etc. No legacy technique or system provides a solution to ascertain a mental state by combining object detection and gaze tracking.

[0081] The term “impairment” as used herein may include fatigue-based impairments, such as fatigue, tiredness, drowsiness, etc. The term “impairment” may also indicate substance-based impairments, such as drunkenness, intoxication, inebriation, etc. No legacy technique or system provides a solution to ascertain a mental state and associated impairment level by combining object detection and gaze tracking.

[0082] The ADAS may fuse data from the exterior-facing cameras or other sensors with data from the interior-facing cameras or other sensors. The first and second data may be captured simultaneously or substantially simultaneously. Then, using the fused model, the ADAS may determine if the driver’s gaze is associated with objects of interest.

[0083] In one approach to correlating the driver’s gaze with an object of interest, the gaze direction of the driver may be represented as a heatmap, scatter plot, point cloud, cluster, or other suitable representation of the driver’s gaze. As the vehicle moves through timeand space, the heatmap may change with the driver’s gaze and the movement of the vehicle. In generating, updating, and maintaining such a heatmap, the ADAS may consider both fixated eye movements and saccadic eye movements. Generally speaking, fixation occurs in eye movements when a gaze is directed towards a specific object for a specific period of time, this may also be referred to as dwell time. Saccades occur in eye movement when the gaze moves quickly between objects. Thus, an ADAS may treat erratic eye movements differently from fixed-eye behavior.

[0084] A heatmap may be generated based on both the fixated and saccadic eye movements with fixations being “hotter” (z.e., carrying more weight, greater dwell time, etc.) than regions viewed with saccadic eye movements or in peripheral vision. Some or most regions in three-dimensional space may not be in the view of the driver at a given point in time. In one embodiment, such a heatmap may represent the gaze direction of the driver in the three-dimensional environment in a color gradient. For example, a pinpoint location of vision may be red while areas in periphery may be orange, yellow, green, blue, etc. based on the distance to the primary location of focus, saccadic eye movements, peripheral vision parameters, etc.

[0085] Additionally, the ADAS may segment the three-dimensional environment into particular areas / objects of interest. Such segmentation may be achieved using bounding boxes that identify particular areas of interest within the three-dimensional environment. A bounding box may be a two- or three-dimensional rectangle or other polygon surrounding the object generated by the object detection system.

[0086] The ADAS may determine that the driver’s gaze currently intersects with such a bounding box, intersected with a bounding box in the past, or will intersect with a bounding box in the future. For example, if a heatmap representing the driver’s gaze direction intersects, encompasses, touches, or is otherwise associated with a bounding box representing an object in the three-dimensional environment for a threshold amount of time, the contents of the bounding box may be considered to be “seen” by the driver. Over time, the ADAS may track both a location (z.e., pixels) that the driver currently looks at and the current object (z.e., bounding box) that the driver looks at, z.e., the ADAS may perform both pixel-level segmentation and box-level segmentation. In one embodiment, to determine an intersection of the driver’s gaze and a bounding box, the ADAS may cast a ray from an origin representing the driver’s position in the three- dimensional environment. The direction of the ray may be represented using a camerapose of the driver determined using gaze detection. In various embodiments, the pose reflecting the gaze / eye direction may be calculated with or without reference to a head position.

[0087] Consider an example of a vehicle approaching a STOP sign. An object detection system may identify the STOP sign in the ambient environment and use three- dimensional geometry to understand the positioning of the STOP sign relative to the vehicle. Simultaneously, a DMS may track the driver’s gaze as a heatmap within the 360- degree, three-dimensional model of the environment — e.g., based on a pose of the driver (having a position and orientation). Using the heatmap, the ADAS may determine if the driver looks in the direction of the STOP sign. When the driver’s gaze intersects with a bounding box surrounding the STOP sign for a threshold amount of time, the ADAS may count the STOP sign as a “detected” object. Further factors may be considered to determine whether an object is “comprehended” (e.g., if a driver takes an appropriate corrective action in response to “detecting” the object). The determination of the intersection between gaze and object may consider a reasonable tolerance to account for peripheral vision. An object that remains in the three-dimensional environment without having yet been associated with the driver’s gaze may be considered “undetected.” If the driver’s gaze never intersects with a bounding box surrounding the STOP sign, then the ADAS may consider the STOP sign as a “missed” object. In general, the ADAS may use detected and missed objects as evidence that the driver is appropriately processing stimuli about the surrounding environment (or conversely, that the driver is not paying attention, e.g., due to fatigue). As detailed below, the ADAS may perform appropriate actions based on the detected and missed objects.

[0088] Consider another example in which an object detector identifies a disabled vehicle in a lane of travel on a highway. The object detector may determine that the object is a vehicle, that the vehicle is in the current lane of travel, and that the vehicle is not moving. The object detector may identify such an object as a high danger object, in this case because a collision may be imminent. The gaze detector may determine whether a direction of the driver’s gaze intersects with a bounding box surrounding the disabled vehicle. If the driver is not looking at the disabled vehicle, the ADAS may cause a braking operation to slow the vehicle and avoid the collision. However, if the driver is looking at the disable vehicle, the ADAS may wait slightly longer for a rational responseto be received from the driver, e.g., the driver steps on the brake pedal to slow the vehicle, and avoid automatically taking control of the vehicle unnecessarily.

[0089] These two examples are merely illustrative. Many other scenarios may arise where coordination between the gaze detector and the object detection system, and other systems may be beneficially employed to ascertain the mental / awareness state of the driver.Mental State

[0090] As another action, an ADAS may dynamically adjust parameters used by ancillary systems based on the driver’s mental state to increase or decrease a tolerance of the safety features. Such parameters may control a threshold time, distance, or other value used by the ancillary systems when to intervene. As discussed above, this may apply in a collision detection system to increase the tolerance of the system in avoiding a disabled vehicle. Towards this end, the ADAS may dynamically adjust parameters used by the collision detection system based on the determined gaze direction and / or mental state of the driver. Drivers may generally exhibit a preference to avoid unnecessary lane corrections performed by a lane detection system. Some drivers may even disable such a feature if the lane detection system too frequently corrects course. When the ADAS determines that the driver has a heightened mental state, the ADAS may increase the tolerance of the lane detection system by adjusting appropriate parameters so that the lane detection system is less likely to perform automatic adjustments. Conversely, if the driver has a reduced mental state, the lane detection system may adjust the parameters to perform corrections with increased proactivity.

[0091] The fusion of object detection and gaze tracking technologies creates an additional need to calibrate the external-facing cameras and sensors with the internal-facing gaze trackers. Because unique driver characteristics such as height, distance between eyes, head position, and other physical aspects impact the three-dimensional geometry used to determine gaze intersections with objects, a need arises to calibrate the gaze detector for a particular driver. The ADAS may generate a fundamental correspondence matrix between the internal-facing camera(s) and external-facing camera(s).

[0092] The ADAS may also track the driver’s mental state over time. The ADAS may track and record a missed number of objects and a seen number of objects over a driving session. This data may be used to calculate a driver alertness value that reflects thedriver’s current mental state. For example, at the start of a trip, the driver may be focused — scanning the road for potential dangers and seeing most / all of the objects identified by the object detection system. However, as the trip progresses, a driver may become less alert, scan the environment less, and miss more objects. The ADAS may track this information over time and calculate an alertness score based on the number of objects being seen / missed. In an embodiment, this driver alertness value may be displayed to the driver using an on-board display device. The driver alertness value may further consider a stored driver profile to determine whether the current performance of the driver matches historical averages. This driver alertness score may apply in both manual and autonomous modes of operation.Awareness State

[0093] As discussed above, the gaze detector and the object detection systems may be used to determine a driver’s “awareness state.” In some embodiments, this technique may be supplemented with physiological data (e.g., heart rate, respiratory data, blood pressure, etc.), external data (e.g., weather conditions, road conditions), and other suitable data.

[0094] The “awareness state” may indicate the driver’s “attention level.” The term “attention level” as used herein may measure a driver’s focus, attentiveness, alertness, vigilance, awareness, cognizance, watchfulness, perceptiveness, clear-headedness, fatigue, tiredness, drowsiness, etc. The “awareness state” may include a determination that the driver is operating under the influence of drugs or alcohol. The “awareness state” may indicate further driver characteristics such as the level of stress of the driver (referred to below as “stress level”), the amount of trust that the driver has in the autonomous driving system (referred to below as “trust level”), and other aspects of a driver’s cognitive state that may be determined based on eye tracking, object detection, external data, and other data. Generally speaking, trust level may be classified as over-trust, calibrated (i.e., balanced) trust, or under-trust.

[0095] In one embodiment, the ADAS may use this awareness state to improve autonomous-driving scenarios in which the human driver supervises, monitors, and oversees vehicle operation (e.g., in Level 3). In such scenarios, the human driver may serve as a redundant system in the autonomous driving ecosystem. The ADAS may employ a more nuanced approach to supervised-driving scenarios based on the awareness state. This approach may consider gradients across driver awareness scenarios — e.g., a“fully aware driver,” “a partially aware driver,” and “an unaware driver.” A fully aware driver may have their eyes on the road, may view and identify objects, and react to scenarios on the road. A partially aware driver may scan objects occasionally, view nearby objects when looking - not asleep, maybe reading a book and viewing objects periodically. A driver having no awareness may be entirely engaged in other activities. These different awareness levels may require different actions, reactions, and interactions from the ADAS.

[0096] Thus, the ADAS may operate differently based on the driver awareness state. Generally speaking, an ADAS may control aspects of autonomous driving with “policies.” A policy may control aspects of driving such as speed, acceleration, turning, decision-making, etc. For example, a policy may exist to control: (1) velocity / speed; (2) turning; (3) safe forward and lateral distances; (4) lane-change decisions; (5) a pace of acceleration changes; (6) takeover time, and many other suitable aspects of autonomous driving. For example, for each computational cycle of the autonomous driving, the ADAS may execute a lane-change policy to ensure that the current lane of travel remains appropriate or that a change should be made. If, for example, a disabled vehicle is seen in the current lane of travel, no vehicles are currently in an adjacent lane, the vehicle is travelling at a suitable speed, etc., then the ADAS may coordinate / execute a lane change into the adjacent lane.Impairment Level

[0097] After the ADAS acknowledges an object as “seen” based on the driver’s gaze intersection with the object, the ADAS may track the duration of time in which a driver’s gaze fixates on a seen object. This data may be used to calculate a driver alertness value that reflects the driver’s current mental state and impairment level. For example, at the start of a trip, the driver may be focused — scanning the road for potential dangers and seeing most / all of the objects identified by the object detection system. However, as the trip progresses, a driver may become less alert, scan the environment less, and fixate on an object. The ADAS may track this information over time and calculate an alertness score based on how long the driver looks at the detected objects and the number of objects being seen / missed. In an embodiment, this driver alertness value may be displayed to the driver using an on-board display device. The driver alertness value may further consider a stored driver profile to determine whether the current performance ofthe driver matches historical averages. This driver alertness score may apply in both manual and autonomous modes of operation.

[0098] A further need exists to determine when drivers operate vehicles with a substancebased impairment, e.g., under the influence of drugs or alcohol. In one embodiment, the ADAS may use the number of missed objects as a factor in ascertaining driver impairment level. When a driver frequently misses objects, the ADAS may consider this as evidence that the driver might be operating the vehicle under the influence of drugs or alcohol.

[0099] Once an ADAS determines the driver’s mental / awareness state and / or impairment level, the ADAS may take a wide array of appropriate actions. In some scenarios, the ADAS may alert the driver about a duration of gaze fixation for a seen object(s), a particular missed object, a number of the missed objects, or provide details about their current mental / awareness state and impairment level. For example, the ADAS may generate an alert for the driver using an on-board display device. In other scenarios, the ADAS may switch between autonomous driving levels, either removing responsibilities from the driver or increasing the driver’s responsibilities. In some scenarios, the ADAS may engage a mechanical system to perform a steering operation, a braking operation, an acceleration operation to avoid collisions, correct the trajectory of the vehicle, etc. The ADAS may tighten the driver’s seat belt as a means of physically alerting the driver about their reduced mental state / attention level / impairment level. The ADAS may set limits in terms of the permissible speed or acceleration of the vehicle if the driver’s mental state is diminished or that the driver is impaired.Policies

[0100] Policies may also control driving subsystems such as: (1) an adaptive cruise control system; (2) a collision avoidance system, (3) a blind-spot monitoring system, (4) a lane-change warning system, (5) a gaze-enabled lane change system. (6) a steering-assist system, (7) a lane-keep assist system, (8) a lane departure warning system, (9) an accidental-maneuver detection system, and many other suitable driving subsystems.

[0101] A policy may be represented by a mathematical formula, function, equation, etc. that includes various parameters. Such parameters may be deterministically defined constraints. The ADAS may dynamically control these parameters based on a number of factors to influence the behavior of the vehicle. While for certain policies a minimumlevel of safety may be required by law or regulation (e.g., a trailing vehicle may be required by law to follow at least X meters behind a forward vehicle), gray areas exist where the preferences of the driver, the awareness state of the driver, the nature of the vehicle, the current weather conditions, and a host of other suitable factors allow for dynamic parameter adjustment.

[0102] By monitoring driver engagement in a significant and meaningful way and dynamically adjusting parameters used by the policies, the autonomous experience may be improved. These parameters may fully extendible — parameters may be added, new behaviors / functions / policies may be added, etc. to address a variety of use cases. An ADAS may be pushed to have more autonomy when the driver is disengaged. A policy may take more aggressive approach when the driver is highly engaged.

[0103] In one embodiment, the ADAS may adjust the policies by varying parameters used by the policies based on the driver’s awareness state. For example, a policy may determine the distance at which the vehicle follows behind the vehicle as used by a collision avoidance system or adaptive cruise control system. When the driver is less attentive, a parameter that controls the distance to the forward vehicle may be increased to increase the distance between the vehicle and the forward vehicle.

[0104] Additionally, one or more parameters representing one or more aspects of the driver’s awareness state may be introduced to the various policies. For example, parameters may reflect the driver’s anxiety -level, stress-level, nausea-level, etc. As the DMS tracks the driver’s awareness state, the parameters may be updated, and thus, the behavior of the autonomous driving may be suitably tailored to match the driver’s awareness state.

[0105] Moreover, because the policies allow for a different implementation of the driver awareness parameters for different decisions, an ADAS may consider each policy decision individually with regard to particular objects. That is, parameters may be adjusted based on a driver’s attention level with respect to a specific object. For example, if a driver is very aware of the vehicle in front of the driver, the longitudinal distance requirement component of the safe-distance policy may be relaxed. Thus, the ADAS may configure, adapt, modify, and otherwise control specific components of individual policies based on particular objects that a driver is seeing or missing. For example, driver behaviors may also be used to drive the decision-making of the policies. For example, a lane-change policy may consider whether the driver has looked in the rear-view mirror orin the direction of the adjacent lane recently, and may only perform a lane change if the driver has looked in the mirror or in the direction of the adjacent lane.

[0106] A further technical benefit may be achieved by tuning the policies based on feedback provided by the driver. A model of driver situational awareness may be built based on the data from the driver monitoring system, the outputs of the sensing state, and the policies. The engagement of the driver with a scene may provide feedback to affect the policies in place in real-time. Feedback may include the comfort level, stress level, trust level, etc. of the driver as ascertained by the DMS. The stress level of the driver may be ascertained by the DMS based on the driver’s eye movements, the rate of object detection, other factors considered by the DMS (e.g., heart rate), and external factors. In some embodiments, onboard computer-machine interaction tools (such as voice or text) may be used to explain to the driver what is happening in an effort to reduce stress. Feedback may also include behaviors of the driver such as manual actions taken by the driver to conduct the vehicle (e.g., steering, braking, accelerating), gaze directions of the driver e.g. looking into the rear-view mirror or an adjacent lane), and other suitable behaviors.

[0107] For example, a policy may control the safe distance to the nearest vehicle. A driver may indicate stress or lack of trust in the autonomous systems by constantly looking at the vehicle in front, performing darting eye movements, etc. The driver’s heart rate may rise. Based on this feedback, a parameter that controls the minimum distance to the next vehicle may be increased. Conversely, if a driver indicates high trust / lack of stress, this parameter may be decreased. For another example, a policy may control the lane changes made by the vehicle. A driver may respond negatively to frequent lane changes, and a parameter used by the lane-change policy may be updated to reduce the number of lane changes. In yet another example, the driver may respond negatively to a “jerky” autonomous driver that frequently stops or accelerates. A policy guiding rate-of- acceleration changes may be updated to smooth the travel experience.

[0108] A further technical benefit may be achieved by varying takeover times based on the driver awareness state. “Takeover time” may be the amount of time under a given policy that the driver has to resume manual driving. For instance, the ADAS may identify a threat or condition that requires manual intervention. Because the driver needs time to process the scene and physically position themselves to take control of the vehicle, the ADAS may allot the driver a fixed amount of time to assume control after informing thedriver of the needed change. A regulation may specify a particular amount of takeover time that is required to move from autonomous driving to manual, e.g., 10 seconds. However, by varying the takeover time based on the driver’s awareness level (providing different takeover times for each awareness state), smoother transitions may occur between states. The transitions may be quicker than the default takeover time where the driver is already operating in a fully aware state or slower than the default takeover time where the driver exhibits no awareness.

[0109] A further technical benefit may be achieved by exposing the set of parameters used in the policies for customization by original equipment manufacturers (“OEMs”). This allows different OEM’s to employ different vehicle behaviors without creating a new policy for each manufacturer. For example, a sports car may allow for more aggressive automated driving experience while a sedan may provide a more delicate driving experience. By exposing the set of parameters, the same policies may be used across the OEMs with different parameters adjusted to control the driving behavior. Driver-based parameters and attention-state parameters may exist alongside the OEM- customizable policies, and thus a different implementation of the driver-awareness-based policies may be configured for different OEMs.

[0110] A further technical benefit may be achieved by tuning a warning system based on the driver awareness-state and / or attention level. Constant warnings may be annoying to a user. When the driver is deemed to be highly attentive, warnings may be reduced. Additionally, knowing where the driver is looking allows for more narrowly tailored and adaptive warnings. For example, the ADAS may signal the driver where to look with specificity (at a crossing pedestrian, bicycle, etc.). Challenging situations (e.g., a roundabout, an unprotected left, a right on red, a merge, a pedestrian), may be identified and the driver alerted that additional attention is needed over their current awareness state. This allows the ADAS to ensure that the driver is capable of taking control quickly in a challenging situation without downgrading entirely from an autonomous driving state to a manual driving state. Signals may be provided to the driver either with vocal assist or illumination to focus on the most relevant parts of the scene.[OHl] A further technical benefit may be achieved by pairing derived conclusions about a driver’s mental state and impairment level with data from a vehicle’s mechanical systems. For example, when a driver approaches a STOP sign, does not see the STOP sign, and does not engage the brakes or slow the vehicle, the ADAS may consider this asfurther evidence that the driver is not paying attention or may be impaired by fatigue. Moreover, an ADAS may monitor a driver’s actions for rational responses to stimuli to further derive conclusions about the driver’s mental state and impairment level. For example, when the object detection system identifies a dangerous scenario, the ADAS may recognize that the driver quickly takes corrective actions such as slowing the vehicle, steering around an obstacle, etc. Such a rational response may result in an increased alertness value.

[0112] A further need exists to determine when drivers operate vehicles when impaired, e.g., under the influence of drugs or alcohol. In one embodiment, the ADAS may use the number of missed objects as a factor in ascertaining driver impairment. When a driver frequently misses objects, the ADAS may consider this as evidence that the driver might be operating the vehicle under the influence of drugs or alcohol.

[0113] FIG. 1 is a block diagram of ADAS 100, according to some embodiments. As illustrated in FIG. 1, ADAS 100 may include DMS 110, external sensing system 120, object detection system 130, vehicle controllers 140, alert generator 150, autonomous driving system 155, policy manager 160, state module 165, display 170, driver profile module 175, control unit 180, network interface 190, and storage 195.

[0114] DMS 110 may monitor the driver of the vehicle using inward-facing cameras, gaze detectors, facial detectors, motion detectors, infrared or light-limiting diodes, and other suitable sensors. In an embodiment, DMS 110 may determine if a driver is alert, attentive, and capable of driving. The duties that are designated to a human driver may vary according to a current automation level, and DMS 110 may monitor different driver characteristics based on the current automation level. DMS 110 may include inwardfacing camera(s) 112, gaze detector 114, face tracker 116, motion sensor 118, and eye calibration model 119.

[0115] Inward-facing camera(s) 112 may be one or more cameras oriented towards the driver of a vehicle. A driver’s facial expressions may indicate that a driver is experiencing drowsiness, microsleep, sleep, unresponsiveness, intoxication, etc. Therefore, inwardfacing camera(s)l 12 may capture images of a driver’s facial expressions to determine if a driver is asleep, incapacitated, or otherwise incapable of performing their responsibilities. When the driver exhibits signs of drowsiness, e.g., when the driver’s eyes are closed for a particular length of time (e.g., 4 seconds), DMS 110 may cause ADAS 100 to generate an alert. In an embodiment, inward-facing camera(s) 112 may be may be 8, 12, 24, 50, 96,200, or other high megapixel cameras having a particular field of view. In another embodiment, inward-facing camera(s) 112 may use infrared or other wavelength of light that is invisible to the human high to track a driver’s facial expression.

[0116] Gaze detector 114 may track the eyes of the driver of the vehicle and specifically, estimate, track, or otherwise determine the direction and duration of the driver’s gaze. In addition to gaze direction, gaze detector 114 may monitor blink rate, how open the eyes are, how dilated the pupils are, and other suitable eye characteristics. To ascertain the direction of a driver’s gaze, gaze detector 114 may employ a pupil-corneal reflection technique to generate a Purkinje image with infrared light, as discussed in further detail below with reference to FIG. 6. In addition to gaze direction, gaze detector 114 may monitor various characteristics of the driver, such as blink rate, an eyelid closure duration, an eyelid closure percentage, an eye aspect ratio, a gaze fixation duration, a pupil dilation ratio, and other suitable eye characteristics.

[0117] Face tracker 116 may track the position and angle of a driver’s face and / or head and the duration of time in which the driver’s face and / or head is located at that position and angle. Face tracker 116 may operate independently from gaze detector 114 or in tandem with gaze detector 114 to determine a pose representing the driver’s gaze. Face tracker 116 may also reference the driver’s facial profile to a stored driver profile as a means of recognizing each unique driver of the vehicle.

[0118] Motion sensor 118 may provide an alternative mechanism for tracking motion within the vehicle. Motion sensor 118 may be used to determine if the driver is yawning frequently, asleep, incapacitated, or otherwise not moving over a period of time. Motion sensor 118 may also be able to determine that the driver’s hands are on the wheel.

[0119] Eye calibration model 119 may determine the calibrated eye model for a driver. The calibrated eye model may be based on gaze tracking determined by gaze detector 114 for objects identified for rolling calibration. As described with reference to object detection system 130, ADAS 100 may store a list of objects that may be used for rolling calibration. When the driver interacts with, or looks at, these identified objects, ADAS 100 may perform the calibration method. This may be repeated each time the driver looks at the identified objects. The eye calibration model 119 may be continuously updated to reflect additional calibration data until a threshold number of calibrations have been performed. The threshold may be set based on safety standards of a vehicle manufacturer,ADAS manufacturer, a regulatory body and / or a combination of entities representing stakeholders.

[0120] External sensing system 120 may gather data about the environment surrounding a vehicle using a variety of sensors. Sensors and cameras may be located inside the vehicle, outside of the vehicle, or some combination of the two. External sensing system 120 may fuse multiple camera images with RADAR and LIDAR sensors to build a contemporaneous, 360-degree, three-dimensional model of the surrounding environment. In other embodiments, a field of view of the examined data may be narrower. External sensing system 120 may include forward-facing camera(s) 122, RADAR system 124, LIDAR system 125, GPS unit 126, side-view camera(s) 127, weather system 128, and rear-facing camera(s) 129.

[0121] Forward-facing camera(s) 122 may include one or more cameras that are oriented in a forward direction from the perspective of the vehicle. Forward-facing camera(s) 122 may be located inside the vehicle, outside of the vehicle, or some combination of the two. In an embodiment, forward-facing camera(s) 122 may include two forward-looking cameras placed on the vehicle. Forward-facing camera(s) 122 may have a similar field of view, but forward-facing camera(s) may also have different fields of view. For example, a first camera may provide a narrow view such as a 28-degree field-of-view, and a second camera may provide a wide-view such as a 120-degree field of view. The first camera and the second camera may 8, 12, 24, 50, 96, 200, or other high megapixel cameras or capture another suitable resolution for use in external sensing system 120.

[0122] RADAR system 124 may rely on radiolocation technologies that use radio waves to determine the distances to objects outside of the vehicle. RADAR system 124 may determine the distance, angle, and velocity of objects located outside of the vehicle.

[0123] LIDAR system 125 may provide laser imaging and ranging using pulsed lasers to measure distances. LIDAR system 125 may contribute to the generation of a three- dimensional representation of the surrounding environment of a vehicle. LIDAR system 125 may provide data as point cloud output. Point cloud output may be a collection of three-dimensional points and other associated attributes (e.g., light intensity, surface normal) generated by LIDAR system 125. RADAR system 124 and LIDAR system 125 may be used in tandem because each technology has relative strengths and weaknesses. For example, LIDAR system 125 may have issues with problematic weather conditions such as wind, rain, snow, etc., while RADAR system 124 may operate in such conditions.RADAR system 124 may operate at a reduced resolution as compared to LIDAR system 125. By combining sensor data, the deficiencies of one particular technology are compensated for.

[0124] GPS unit 126 may track a vehicle’s positioning using a global positioning system. GPS unit 126 may also assist in providing driving directions and related features to an onboard display in the vehicle.

[0125] Side-view camera(s) 127 may include one or more cameras facing laterally from a vehicle. In an embodiment, side-view camera(s) 127 may include a forward-looking side camera and a rear-looking side camera situated to capture a comprehensive view of the right and left of the car. Towards this end, side-view camera(s) 127 may include multiple sets of cameras on each side of the vehicle or a single camera on each side of the vehicle. Side-view camera(s) 127 may be located inside the vehicle, outside of the vehicle, or some combination of the two. In this manner, side-view camera(s) 127 may be arranged to capture both the right and left (starboard and port) sides of a vehicle. Side-view camera(s) 127 may be wide-angle, narrow- view, or a combination of the two.

[0126] Weather system 128 may track current weather conditions outside of the vehicle. Weather system 128 may track temperature, barometric pressure, wind direction and speed, rainfall, and other suitable weather characteristics.

[0127] Rear-facing camera(s) 129 may face in a backward-looking direction from the perspective of the vehicle. Rear-facing camera(s) 129 may capture data about what happens behind the vehicle. Rear-facing camera(s) 129 may be wide-angle, narrow-view, or a combination of the two. Rear-facing camera(s) 129 may be located inside the vehicle, outside of the vehicle, or some combination of the two.

[0128] Object detection system 130 may interpret a surrounding environment (i.e., the data gathered by external sensing system 120) and identify relevant objects such as other vehicles, lanes, traffic lights, signs, pedestrians, animals, traffic barriers, debris, etc. Object detection system 130 may determine spatial location of each object. The spatial location may include a position, a rate of movement (e.g., speed, velocity, acceleration), a direction of movement, and other suitable characteristics. Object detection system 130 may generate a bounding box for each identified object. The bounding box may be two- dimensional or three-dimensional. Object detection system 130 may generate a confidence score quantifying the likelihood that the bounding box contains an actual object. Object detection system 130 may also generate a classification for each boundingbox that signifies “what” the object is. In various embodiments, object detection system 130 may include detector 132, localization manager 136, and classification manager 138.

[0129] Object detection system 130 may also include object map manager 134 which utilizes detector 132, localization manager 136, and / or classification manager 138 to generate a global map. In one embodiment, the global map may be stored in storage 195, stored remotely and accessed by network interface 190, and / or a combination of the two such that portions of the global map are stored locally in storage 195 and portions stored remotely and accessed by network interface 190. The map may include traffic objects including lanes markings, road arrows, drivable paths, road edges, traffic control objects, etc. These objects may be classified by classification manager 138. The global map may be updates by object map managers of a plurality of vehicles using ADAS 100. Method 500, described in FIG. 5, may establish and / or update the relationships of objects included in object map manager 134. In some embodiments, object map manager 134 may be accessed by autonomous driving system 155, policy manager 160, and / or other aspects of ADAS 100 to determine and control driving behavior.

[0130] Object detection system 130 may employ detector 132 to detect objects and characteristics. Detector 132 may identify objects using a single-stage or multi-stage algorithm that passes image information to a trained neural network. Detector 132 may determine a bounding box for each object identified in an image. Examples of a single- stage algorithm performed by detector 132 may be Yolo, SSD, SqueezeDet, DetectNet, etc. Detector 132 may also use multiple models to first extract regions of objects and second refine the localization of the object. Examples of such a two-stage algorithm may be R-CNN, FPN, and Mask R-CNN. Detector 132 may also generate a fundamental correspondence matrix between a first virtual frame corresponding to the one or more exterior-facing cameras and a second virtual frame corresponding to the one or more interior-facing cameras based on an origin (i.e., the location of the driver’s eye).

[0131] Localization manager 136 may determine a bounding box of the identified object(s). Localization manager 136 may determine a two- or three-dimensional bounding box. Localization manager 136 may assign appropriate coordinates (four coordinates in two dimensions, eight coordinates in three dimensions) to represent the location of the bounding box within the model of the surrounding environment.

[0132] Classification manager 138 may classify objects identified in the surrounding environment. Classification manager 138 may determine what the object is — e.g.,whether the object is a vehicle, lane, traffic light, sign, pedestrian, animal, traffic barrier, debris, etc. Classification manager 138 may determine a confidence value that indicates the probability that a given classification of a particular object is correct. In an embodiment, classification manager 138 may assign an individual classifier to each bounding box to independently determine a classification for each object in a particular scene. Classification manager 138 may also establish a danger priority list that includes all objects currently identified. Classification manager 138 may assign a danger level to each object in the danger priority list. A high danger object may be an object on a trajectory for collision. A high danger object may also be an object at a high risk of harm to human life, such as a pedestrian. High danger objects may be differentiated for the purposes of collision avoidance, but furthermore, high danger objects may be weighted differently when determining a mental state / attention level / impairment level based on seen / missed objects. Thus, a missed high danger object may cause larger impact on the mental state / attention level / impairment level determination than a missed low danger object. Classification manager 138 may also classify certain objects for purposes of rolling calibration, as described with reference to FIGS. 2A-2C and 3A-3B. Classification manager 138 may establish a list of objects that can be used for rolling calibration. Exemplary objects may include permanent, semi-permanent, and / or fixed objects along a route the driver takes routinely. Additionally, these objects may be located at the beginning of a route so that calibration can also be performed at the beginning of a driver’s route. Classification manager 138 may identify the type of object and / or the object’s location. Classification manager 138 may compare frequent locations of the vehicle, per GPS unit 126, with the locations of objects detected by detector 132. Objects that are frequently near the vehicle may be candidates for rolling calibration.Additionally, external sensing system 120 may determine which objects are in the FOV of the driver. Based on these factors, classification manager 138 may generate a list of objects that are used for rolling calibration.

[0133] Vehicle controllers 140 may control the operation of a vehicle’s mechanical systems. Vehicle controllers 140 may include braking system 142, steering system 144, engine system 146, and seat-belt system 148.

[0134] Braking system 142 may control the operation of the brakes of a vehicle. Braking system 142 may be engaged to automatically slow the vehicle or bring the vehicle to a stop when circumstances demand. Braking system 142 may be controlled by anautonomous driving system when the vehicle operates at an autonomous driving level. Braking system 142 may also be operated by driver 204 when the vehicle operates at a level requiring manual driving.

[0135] Steering system 144 may control the steering system of a vehicle to turn and maneuver a vehicle on the road. Steering system 144 may be controlled by an autonomous driving system when the vehicle is operating at an autonomous driving level. Steering system 144 may be employed by a lane departure system to adjust the travel of the vehicle when the vehicle drifts out of a lane.

[0136] Engine system 146 may control the engine of a vehicle. Engine system 146 may be controlled by autonomous driving system 155 when the vehicle operates at an autonomous driving level to allow the vehicle to accelerate and decelerate to appropriate speeds. Engine system 146 be engaged to automatically slow the vehicle in the event that a driver is incapacitated. ADAS 100 may use engine system 146 to set a maximum permissible speed or acceleration if the driver’s mental state / awareness level is diminished or the driver is highly impaired.

[0137] Seat-belt system 148 may control the seat belts within a vehicle. Generally, seatbelt system 148 may also offer safety features related to the safety belts such as warnings and alerts when a passenger does not have a safety belt buckled. ADAS 100 may cause seat-belt system 148 to tighten a driver’s seat belt to alert the driver when their mental state / awareness level is slipping or if impairment is increasing. This may provide a physical means of interacting with, alerting the driver, and may provide an added benefit of physically waking the driver if they are asleep.

[0138] Alert generator 150 may generate alerts for the driver. Alert generator 150 may prompt a driver to adjust their behavior. For example, alert generator 150 may inform the driver when their mental state / awareness level reduces below a threshold level or their impairment level exceeds a threshold level. Alert generator 150 may alert the driver of an excessively long gaze fixation on an object — e.g., if the driver is staring at a green light without accelerating the vehicle. Alert generator 150 also may alert the driver of a particular missed object — e.g., if the driver does not see a STOP sign, a jogger, or an emergency vehicle. Alert generator 150 may employ audible alerts such as spoken language, beeping noises, and other suitable sounds. For example, a blind spot detector may use alert generator 150 to make a beeping noise if the driver initiates a lane change at a dangerous time. Alert generator 150 may leverage a stereo system installed in thevehicle, special purpose speakers, or other audio devices to generate sound. Alert generator 150 may also generate visual alerts for display in the vehicle on an on-board display, such as display 170.

[0139] Autonomous driving system 155 may be responsible for piloting the vehicle when the vehicle operates autonomously. Autonomous driving system 155 may rely on the data from external sensing system 120 and object detection system 130 to determine appropriate driving strategy and actions at a given point in time. Autonomous driving system 155 may engage vehicle controllers 140 and other mechanical systems in the vehicle to achieve autonomous driving.

[0140] Policy manager 160 may assist autonomous driving system 155 in configuring autonomous levels and storing data related to autonomous operation. Policy manager 160 may include driving strategies that operate within each level. For example, certain autonomous driving features may be disabled at Level 3 but enabled at Level 4 and Level 5, and policy manager 160 may manage such policies. A policy may control aspects of driving such as velocity / speed; (2) turning; (3) safe forward and lateral distances; (4) lane-change decisions; (5) a pace of acceleration changes; (6) takeover time, and many other suitable aspects of autonomous driving.

[0141] Policies may also control driving subsystems such as: (1) an adaptive cruise control system; (2) a collision avoidance system, (3) a blind-spot monitoring system, (4) a lane-change warning system, (5) a gaze-enabled lane change system, (6) a steering-assist system, (7) a lane-keep assist system, (8) a lane departure warning system, (9) an accidental-maneuver detection system, and many other suitable subsystems. A policy may be represented by a mathematical formula that includes various parameters. The parameters leveraged by these subsystems may vary.

[0142] For example, adaptive cruise control system may include policies and parameters that specify: (1) a maximum permissible lateral acceleration, (2) a maximum stopping acceleration, (3) a minimum time gap, (4) a maximum time gap, (5) a minimum clearance, (6) a minimum speed, (7) a maximum speed, and / or (8) a distance to second vehicle. A blind-spot monitoring system, a lane-change warning system, and / or a gaze- enabled lane change system may include policies and parameters that specify: (1) a speed difference tolerance between the vehicle and a target vehicle, (2) a time to collision tolerance, or (3) a warning level. A collision avoidance system may include policies and parameters that specify an alarm tolerance level that controls when a collision avoidancesystem generates a driver alert. A steering-assist system, a lane-keep assist system, or a lane departure system may include policies and parameters that specify may include policies and parameters that specify: (1) a threshold distance to a lane boundary, (2) a warning level, or (3) a lane change tolerance. These parameters are merely exemplary however, and a person of ordinary skill in the arts will understand that the parameters employed by such systems. Moreover, the technique of leveraging dynamic parameters is extensible into other suitable use cases and policies.

[0143] State module 165 may determine the mental state / attention level of driver 204 using the information provided by object detection system 130 and DMS 110. State module 165 may measure a driver’s mental state, z.e., a driver’s frame of mind, mindset, or mood including characteristics such as alertness, vigilance, awareness, cognizance, watchfulness, attentiveness, perceptiveness, clear-headedness, etc. State module 165 may measure a driver’s attention level, / .< ., a driver’s focus, alertness, vigilance, awareness, cognizance, watchfulness, attentiveness, perceptiveness, clear-headedness, etc. State module 165 may track an alertness value that changes over time based on seen / missed objects. Specifically, using a three-dimensional model of the surrounding environment generated by external sensing system 120 and object detection system 130, ADAS 100 may determine if the driver’s gaze is associated with the spatial locations of objects of interest. For example, when a driver’s gaze intersects with a bounding box surrounding an object for a threshold amount of time, State module 165 may consider the object to have been detected. State module 165 may consider missed and detected objects as evidence that the driver is appropriately processing stimuli about the surrounding environment (or conversely, that the driver is not paying attention when they are not seeing a high percentage of objects). When the driver’s gaze is associated with the spatial locations of a bounding box surrounding an object and the driver’s gaze exceeds a gaze fixation threshold amount of time, mental state module 165 may consider the driver to be impaired. Mental state module 165 may consider an excessive gaze fixation duration as evidence that the driver is not appropriately scanning the surrounding environment. In this way, mental state module 165 may measure a driver’s impairment level over a period of time, / .< ., a driver’s fatigue-based impairments (fatigue, tiredness, drowsiness, etc.) or substance-based impairments (drunkenness, intoxication, inebriation, etc.).

[0144] Display 170 may be an in-vehicle display that shows alerts generated by alert generator 150. Display 170 may be integrated into a vehicle’s on-board entertainment andnavigation system. Display 170 may be a standalone display. In one approach, display 170 may display the alertness score calculated by State module 165 as the alertness score changes over time. By displaying the alertness score, driver 204 may be made aware of their performance at the current moment in time and / or as the performance changes over time.

[0145] Driver profile module 175 may store information about one or more individuals that drive a vehicle. For example, driver profile module 175 may store a driver’s calibrated eye model. In another example, driver profile module 175 may store a driver’s cognitive eye scanning pattern when the driver is determined to be alert for the purpose of comparison to data collected during a drive regarding the driver’s impairment level. Driver profile module 175 may also offer a guest profile for use by a guest driver.

[0146] Control unit 180 may be a central processing unit (CPU), graphics processing unit (GPU), or other suitable processor. Control unit 180 may be a specialized electronic circuit designed to process mathematically intensive applications. Control unit 180 may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc. In an embodiment, control unit 180 may be a system-on-chip processor specially designed to support computationally intense computer vision tasks such as those performed in autonomous driving systems.

[0147] Network interface 190 may enable ADAS 100 to communicate and interact with any combination of external devices, external networks, external entities, etc. Such networks may include any network or combination of networks including the Internet, a local area network (LAN), a wide area network (WAN), a wireless network, a cellular network, or various other types of networks as would be appreciated by a person of ordinary skill in the art.

[0148] Storage 195 may be a data storage system used to house information relevant to, used in, and stored by ADAS 100. For instance, storage 195 may include a database management system or relational database tool and associated components. Storage 195 may be housed locally within a vehicle or be cloud-based and accessible using network interface 190. Storage 195 may be a data lake, data silo, semi-structured data system (CSV, logs, xml, etc.), unstructured data system, binary data repository, or other suitable repository.

[0149] FIG. 2 A illustrates environment 200 A in which ADAS 100 may operate, according to some embodiments. As illustrated in FIG. 2A, environment 200A may include vehicle 202, driver 204, object 206A, object 206B, and object 206C. Environment 200A displayed in FIG. 2A is a view from outside of the vehicle, while environment 200B displayed in FIG. 2B reflects a corresponding environment from inside of the vehicle. However, the situation depicted in FIG. 2A and FIG. 2B is merely exemplary and limitless possibilities exist regarding the nature of environments that ADAS 100 may encounter in the real world.

[0150] Vehicle 202 is illustrated in environment 200A as an automobile driving down the left side of the road. In addition to an automobile, vehicle 202 may be any other suitable type of vehicle including a truck, van, motorcycle, all-terrain vehicle, military vehicle, boat, etc.

[0151] Driver 204 may be a human being operating vehicle 202. At any given time, driver 204 may have different responsibilities in driving vehicle 202 ranging from responsibility over all of the vehicles mechanical systems (Level 0) to no responsibilities at all (Level 5). In supervised autonomous driving scenarios (Level 3), driver 204 may be assigned an awareness state (e.g., Fully Aware, Partially Aware, Not Aware).

[0152] In the example of environment 200A, object 206A is a STOP sign, object 206B is a motorcycle, and object 206C is a jogger. These objects may be identified by object detection system 130, with a position, speed, velocity, acceleration, and other suitable physical characteristics tracked for each object.

[0153] FIG. 2B illustrates environment 200B in which ADAS 100 may operate, according to some embodiments. Environment 200B corresponds to environment 200A, only displayed from the perspective inside of the vehicle. As illustrated in FIG. 2B, environment 200 may include gaze direction 203 and display 207.

[0154] Gaze direction 203 may represent the direction in the ambient environment that driver 204 is looking at a given point in time. Gaze direction 203 may be represented as a 6 degrees-of-freedom pose that has a position (x, y, z) and orientation (pitch, yaw, roll) within the three-dimensional environment with an origin at the viewpoint of driver 204. As discussed below with reference to FIGS. 5A and 5B, gaze direction 203 may be considered along with fixated and saccadic eye movements of the driver to maintain a heatmap representing the gaze direction of the driver over time.

[0155] Display 207 may be an on-board display device for presenting information to driver 204. Display 207 may be integrated into on-board display systems. In another embodiment, display 207 may be a standalone display. Display 207 may display a driver alertness value calculated by tracking missed / seen objects and the duration of time in which a driver’s gaze fixates on a seen object over a driving session. Display 207 may also display alerts and other information to adjust the driving behavior of driver 204.

[0156] FIG. 3 A is a table 300A illustrating exemplary autonomous driving levels and mental-state based actions performable at each level, according to some embodiments. Table 300A is merely illustrative but indicates the types of features that are available across autonomous levels. Moreover, table 300A indicates the types of actions that may be performed at each level based on a determined driver mental state and associated impairment level.

[0157] FIG. 3B is a table 300B illustrating exemplary supervised-driving actions that vary based on a driver’s awareness level, according to some embodiments. These actions may control the behavior of the vehicle in operating modes in which the human driver supervises, monitors, and oversees vehicle operation, z.e., in supervised-driving scenarios.

[0158] Table 300B illustrates a more nuanced approach to supervised-driving scenarios based on a determined driver awareness state covering a “fully aware driver,” “a partially aware driver,” and “an unaware driver.” A fully aware driver may have their eyes on the road, may view and identify objects, and react to scenarios on the road. A partially aware driver may scan objects occasionally, view nearby objects when looking - not asleep, maybe reading a book and viewing objects periodically. A driver having no awareness may be entirely engaged in other activities.

[0159] As indicated in FIG. 3B, the actions that ADAS 100 may perform may change based on the driver awareness state. For example, if the driver is partially or unaware, ADAS 100 may be more likely to detect an accidental maneuver when, e.g., the driver bumps the steering wheel. Thus, by providing the nuanced approach to level -three driving that considers driver awareness, the ADAS may better detect accidental wheel maneuvers because a wheel maneuver is more likely to be considered accidental when a driver is not paying attention.

[0160] Additionally, FIG. 3B illustrates that takeover times (z.e., the amount of time under a given policy that the driver has to resume manual driving) may vary based on the driver awareness state. By varying the takeover time based on the driver’s awarenesslevel (providing different takeover times for each awareness state), smoother transitions may occur between states — e.g., quicker transitions where the driver is already operating in a fully aware state and slower transitions where the driver is in an unaware state.

[0161] FIG. 3C is a table 300C illustrating exemplary supervised-driving actions that vary based on a driver’s trust level, according to some embodiments. In an embodiment, the trust level may indicate the stress level, trust level, comfort level, etc. of the driver as ascertained by the DMS. For example, the stress level of the driver may be ascertained by the DMS based on the driver’s eye movements, the rate of object detection, and other factors considered by the DMS (e.g., heart rate). An exemplary policy may control the safe distance to the nearest vehicle. When the driver indicates a high stress level or low trust level, a parameter that controls the minimum distance to the next vehicle may be increased. Conversely, if a driver indicates high trust / lack of stress, the parameter may be decreased. In some embodiments, onboard computer-machine interaction tools (such as voice or text) may be used to explain to the driver what is happening in an effort to reduce stress when the driver is overstressed / under-trusting. When trust is in an appropriately calibrated state, parameter levels may be maintained. When the driver is under-stressed or over-trusting, tolerance parameters may be increased / decreased accordingly to maximized driving performance.

[0162] FIG. 4A illustrates scene 400A as captured by an outward-facing camera such as forward-facing camera(s) 122. Scene 400A may include frame 402 and bounding box 404. Frame 402 may be a single frame of video as captured by DMS 210. Though displayed in scene 400A as a limited field of view for illustration purposes, in an embodiment, scene 400 A may be a 360-degree view of the ambient environment as captured by forward-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129. Frame 402 may represent a single image of the environment that is captured by external sensing system 120 in accordance with a particular frame rate. For example, front-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129 may operate at a particular frame rate that captures a particular number of images per second. In one embodiment, the frame rate may be 60 Hz, though other frame rates are conceivable within the context of this disclosure such that a three-dimensional model of the environment may be rebuilt with a regular periodicity.

[0163] Bounding box 404 may be a two- or three-dimensional rectangle or other polygon surrounding an object detected by object detection system 230. Bounding box 404 mayserve to segment the three-dimensional environment into particular areas / objects of interest. Bounding box 404 may be represented using four coordinates (two-dimensional bounding boxes) or eight coordinates (three-dimensional bounding boxes) representing points in a particular image frame. Bounding box 404 may represent the spatial location of the object as determined by object detection system 230. In this regard, bounding box 404 may also be associated with a position, a rate of movement (e.g., speed, velocity, acceleration), a direction of movement, and other suitable characteristics.

[0164] FIG. 4B illustrates scene 400B as captured by an object detecting system, according to some embodiments. Scene 400B may include map 406 and point cloud 408. Scene 400B may be captured by RADAR system 124 or LIDAR system 125. Scene 400B is a visual representation of the data captured by these object detecting systems for illustrative purpose. Scene 400B corresponds to the scene in 400 A. Object detection system 230 may use this data to build the contemporaneous three-dimensional model of the surrounding environment and determine the spatial locations of the objects of interest.

[0165] FIG. 5A illustrates scene 500A captured by outward-facing sensors (e.g., cameras) with a driver’s current gaze direction represented in a heatmap, according to some embodiments. Scene 500A includes heatmap 501A, head pose 502, face image 503, and eye tracking 504.

[0166] Heatmap 501 A may be calculated continuously over time by ADAS 100 to reflect the gaze direction of driver 204 as vehicle 202 moves through space. ADAS 100 may consider both fixated eye movements (i.e., eye movements when a gaze is directed towards a specific object for a specific period of time) and saccadic eye movements (i.e., eye movement when the gaze moves quickly between objects). Heatmap 501 A may be generated based on both the fixated and saccadic eye movements with fixations carrying more weight than regions viewed with saccadic eye movements or in peripheral vision. In FIG. 5A, heatmap 501 A represents the gaze direction corresponding to head pose 502, face image 503, and eye tracking 504. As discussed above, various sensors may be used to capture these aspects of driver 204.

[0167] In some embodiments, head pose 502 may be used to calculate heatmap 501. Head pose 502 may include a pose of the head having a position and orientation in three dimensional space. For example, head pose 502 may be represented using a 6 degree-of- freedom coordinate frame.

[0168] Face image 503 may reflect the specific orientation and configuration of the face of driver 204. Face image 503 may be used to determine an origin used in tandem with head pose 502 to derive heatmap 501A. Face image 503 may be used to retrieve a driver’s calibrated eye model by driver profile module 175.

[0169] In some embodiments, eye tracking 504 may be used to track the eyes and pupil positions of driver 204 at any given point in time. Eye tracking 504 may be used along with head pose 502 to calculate the driver’s gaze direction and derive heatmap 501A.

[0170] In one embodiment, eye tracking 504 may track both fixated eye movements and saccadic eye movements. Data gathered by eye tracking 504 may be compared to an eye movement profile (e.g., an individualized model in driver profile module 175). When the current data diverges from the eye movement profile, this may be indicative that the driver is fatigued, tired, under the influence, etc.

[0171] FIG. 5B illustrates scene 500B captured by outward-facing cameras with a driver’s current gaze direction intersecting with an object of interest, according to some embodiments. Scene 500B includes heatmap 50 IB, bounding box 505 A, bounding box 505B, and bounding box 505C.

[0172] In the illustrative example provided in FIG. 5B, heatmap 50 IB intersects with bounding box 505A. As such, the motorcycle driver identified by bounding box 505A may be considered to be “detected” by the driver. ADAS 100 may use this happenstance to derive meaningful conclusions about the driver’s mental state, as discussed in further detail below.

[0173] FIG. 6 illustrates a model 600 of a Purkinje image, according to some embodiments. Model 600 may include reflection 602 and ray 604. A Purkinje image may be used by gaze detector 114 to determine the gaze direction of driver 204. Purkinje images are the reflections of external objects from the structure of the eye. A gaze detector may project onto an eye of a drier a light to generate a Purkinje image. Reflection 602 may be of an infrared light shined by gaze detector 114 into the eye of driver 204. Ray 604 may then be determined relative to reflection 602 using three- dimensional geometry, as would be understood by one skilled in the relevant arts.

[0174] FIG. 7 illustrates a method 700 for determining the mental state / attention level of a driver using gaze tracking and object detection, according to some embodiments. Method 700 may be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g.,instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown in FIG. 7, as will be understood by a person of ordinary skill in the art(s).

[0175] In 702, ADAS 100 and / or state module 165 may receive sensor data from external sensing system 120. The sensor data may include images captured by cameras such as forward-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129. The sensor data may also include RADAR and LIDAR data captured by RADAR system 124 and LIDAR system 125. Object detection system 230 may use the data to build a contemporaneous, three-dimensional model of the surrounding environment. In some embodiments, the model may be a 360-degree representation of the surrounding environment. The sensor data may be captured at a particular frame rate, e.g., 60 frames per second, such that the three-dimensional model may be rebuilt with regular periodicity, based on the data received from external sensing system 120 and system capabilities. For example, vehicle 202 may approach object 206 A — a STOP sign, and forward-facing camera(s) 122 and LIDAR system 125 may capture a sequence of images that include the STOP sign.

[0176] In 704, ADAS 100 and / or object detection system 130 may segment the 360- degree, three-dimensional model of the environment built using the sensor data received in 702. Segmentation may involve identifying objects of interest within the three- dimensional model. Such objects may be other vehicles, lanes, traffic lights, signs, pedestrians, animals, traffic barriers, debris, etc. Object detection system 130 may also determine a spatial location of the objects including a position, a rate of movement (e.g., speed, velocity, acceleration), a direction of movement, and other suitable characteristics. Object detection system 130 may employ a single-stage or multiple-stage object-detection algorithm that leverages a trained neural network to both identify objects within an image. Object detection system 130 may also classify each particular identified object as a vehicle, lane, traffic light, sign, pedestrian, etc. Object detection system 130 may determine a bounding box of the identified object(s) and a confidence score that the identified object is in fact an object (as compared to a shadow, discoloration, etc. in the source data. Object detection system 130 may also determine a classification for the object in each bounding box and a probability that the classification is correct. For example, to continue the example of the STOP sign, object detection system 130 mayidentify the STOP sign by placing a bounding box around the STOP sign. A bounding box may be represented using position information such as 4 or 8 coordinates for representing each corner of the rectangle or cuboid. Thus, to continue the STOP sign example, object detection system 130 may classify the object as a STOP sign and determine additional physical characteristics about the STOP sign such as velocity (in this case, the STOP sign is not moving) and position information within the model of the environment.

[0177] In 706, ADAS 100 and / or DMS 110 may determine the gaze direction of driver 204 of vehicle 202. For example, the gaze direction of driver 204 may be represented by a heatmap built by examining fixated and saccadic eye movements of driver 204 over time. DMS 110 may employ gaze detector 114 to track the gaze of driver 204 as a camera pose (having a position and orientation) within the 360-degree, three-dimensional model of the environment as reflected in the sensor data captured in 702. In this manner, the driver’s gaze may be represented using three-dimensional geometry as a ray, vector, line, or other suitable geometric construct. In one embodiment, gaze detector 114 may use the Purkinje images on the eye (discussed above in FIG. 6) to determine gaze direction. DMS 110 may also consider the head position of the driver and other anatomical considerations in determining the driver’s gaze direction and building the heatmap.

[0178] In 708, ADAS 100 may determine if the gaze direction determined in step 706 is associated or otherwise correlated with an object identified in 704. For example, ADAS 100 may determine that the driver’s gaze is currently intersecting with a spatial location of the object of interest, intersected with a spatial location of the object of interest in the past, or will intersect with a spatial location of the object of interest in the future. Step 708 may be iterative because more than one object may be identified in a frame in 704. Thus, ADAS may work through each identified object using an iterative method or construct. In one embodiment, ADAS 100 may cast a ray from an origin at the point where the driver’s eyes are in the three-dimensional environment in accordance with the gaze direction determined in 706. ADAS 100 may determine if the ray intersects with a bounding box associated with the identified object. ADAS 100 may employ a suitable approximation (e.g., a number of degrees of tolerance) or expand the bounding box to accommodate objects that the driver may see using peripheral vision. ADAS 100 may track the number of milliseconds that the gaze intersected with the bounding box and ensure that the intersection exceeds a threshold before considering an object to have beendetected. In this regard, ADAS 100 may consider intersections with respect to a threshold amount of time or a threshold number of samples (given the sampling rate of the cameras). Conversely, if the driver’s gaze never intersects with an object’s bounding box or fails to do so for a threshold amount of time, then ADAS 100 may consider an object to have been missed by driver 204. For example, driver 204 may be looking around the car, up into the sky, etc. and not see the STOP sign identified as an object in 704.

[0179] In 710, ADAS 100 and / or state module 165 may derive a conclusion or conclusions about the driver’s mental state / attention level based on whether the object was seen or missed. In an embodiment, the driver’s mental state / attention level may reflect the driver’s alertness or fatigue. In another embodiment, the mental state / attention level may represent a level of intoxication. For example, ADAS 100 may determine that driver’s mental state / attention level is reducing because the driver did not see a STOP sign. Conversely, ADAS 100 may consider driver 204 to be operating at a heightened mental state / attention level if driver 204 consistently sees an object of importance or high danger or the driver does so with regularity. ADAS 100 may consider missed objects in the aggregate as well — / .< ., a total number of missed objects over a fixed time period, a percentage of missed objects to seen objects, etc. ADAS 100 may also apply a weighted algorithm that considers the importance or danger priority of an object as determined by the object classifier. A disabled vehicle may be given a higher danger priority than a billboard by classification manager 138. Thus, state module 165 may consider a missed disabled vehicle more heavily in calculating the mental state / attention level than a lower weighted object.

[0180] In 712, ADAS 100 and / or vehicle controllers 240 may perform a suitable action based on the mental state / attention level determined in step 710. ADAS 100 may alert driver 204 about a particular missed object. ADAS 100 may generally inform the driver about whether their mental state / attention level is improving or declining. A suitable alert may be displayed on the on-board display device, played as audio, or otherwise conveyed to the driver. ADAS 100 may switch between autonomous driving levels, either removing responsibilities from driver 204 or increasing the driver’s responsibilities. ADAS 100 may engage a mechanical system to perform a steering operation, a braking operation, an acceleration operation to avoid collisions, correct the trajectory of the vehicle, etc. ADAS 100 may tighten the driver’s seat belt as a means of physically alerting the driver about their reduced mental state / attention level. For example, if thedriver’s gaze does not intersect with the STOP sign for a threshold amount of time, state module 165 may generate an alert that the driver missed the STOP sign — e.g., ADAS 100 may inform the driver “You just missed a STOP sign.” ADAS 100 may impose speed or acceleration limits or remove such limits as the driver’s mental state / attention level increases / decreases. ADAS 100 may also engage seat-belt system 148 to tighten the safety belt as a means of physically impacting driver 204 (which may serve to rouse or wake up the driver). As discussed below with reference to FIG. 9, ADAS 100 may dynamically adjust parameters used by ancillary systems (e.g., collision avoidance, lane detection) based on the driver’s mental state / attention level to increase or decrease the tolerance of other safety features. As discussed below with reference to FIG. 8, ADAS 100 may also update a stored value indicative of the mental state / attention level in a profile that tracks driver mental state / attention level over time.

[0181] FIG. 8 illustrates a method 800 for tracking a driver alertness value using gaze tracking and object detection, according to some embodiments. Method 800 may be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown in FIG. 8, as will be understood by a person of ordinary skill in the art(s).

[0182] In 802, ADAS 100 and / or state module 165 may receive sensor data from external sensing system 120. The sensor data may include images captured by cameras such as forward-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129. The sensor data may also include RADAR and LIDAR data captured by RADAR system 124 and LIDAR system 125. Object detection system 230 may use the data to build a contemporaneous, three-dimensional model of the surrounding environment. The sensor data may be captured at a particular frame rate, e.g., 60 frames per second, such that the 360-degree three-dimensional model may be rebuilt with regular periodicity, based on the data received from external sensing system 120 and system capabilities. For example, vehicle 202 may approach object 206C — a jogger, and forward-facing camera(s) 122 and LIDAR system 125 may capture a sequence of images that include the jogger.

[0183] In 804, ADAS 100 and / or object detection system 130 may identify objects within the 360-degree, three-dimensional model of the environment built with the sensor datareceived in 802. Objects may include other vehicles, lanes, traffic lights, signs, pedestrians, animals, traffic barriers, debris, and other objects. Object detection system 130 may also determine a spatial location including a position, speed, velocity, acceleration, direction of movement, and other suitable characteristics for each identified object. Object detection system 130 may employ a single-stage or multiple-stage objectdetection algorithm that leverages a trained neural network to both identify objects within an image. Object detection system 130 may also classify each particular identified object as a vehicle, lane, traffic light, sign, pedestrian, etc. Object detection system 130 may determine a bounding box of the identified object(s) and a confidence score that the identified object is in fact an object (as compared to a shadow, discoloration, etc. in the source data. For example, to continue the example of the jogger, object detection system 130 may identify the jogger by placing a bounding box (e.g., 4 or 8 coordinates representing the corners of the rectangle or cuboid) around the jogger. The bounding box may move with the jogger as time passes. Object detection system 130 may classify the object as a jogger and determine that the jogger is a high danger priority object given the jogger could behave unpredictably and given the potential harm / impact on a human life. Object detection system 130 may calculate a spatial location including a position, velocity, and direction of movement of the jogger.

[0184] In 806, ADAS 100 and / or DMS 110 may determine whether the gaze direction of driver 204 of vehicle 202 intersects with objects identified in 804. DMS 110 may employ gaze detector 114 to track a gaze of driver 204 as a heatmap. The heatmap may be built using both fixated and saccadic eye movements of the driver. The heatmap may further consider an origin and 6 DoF coordinate frame representing the position of the driver’s head within the 360-degree, three-dimensional model of the environment as reflected in the sensor data captured in 802. In one embodiment, gaze detector 114 may use the Purkinje images on the eye (discussed above in FIG. 6) to determine gaze direction. Step 806 may be iterative because more than one object may be identified in a frame in 804, and thus, ADAS 100 may work through each identified object using an iterative method. In one embodiment, ADAS 100 may cast a ray or vector from an origin at the point where the driver’s eyes are in the three-dimensional environment. ADAS 100 may identify bounding boxes associated with objects that the gaze intersects and bounding boxes that the gaze does not intersect. ADAS 100 may employ a suitable approximation (e.g., a number of degrees of tolerance) or expand the bounding boxes to accommodate objectsthat the driver may see using peripheral vision. ADAS 100 may track the number of milliseconds that the gaze intersected with the bounding box and ensure that the intersection exceeds a threshold before considering an object to have been detected. In this regard, ADAS 100 may consider intersections with respect to a threshold amount of time or a threshold number of samples (given the sampling rate of the cameras). If the driver’s gaze never intersects with an object’s bounding box or fails to do so for a threshold amount of time, then ADAS 100 may consider the object “missed.”

[0185] In 808, ADAS 100 may determine that the gaze direction of driver 204 is / was / will not associated with the spatial location of the jogger. For example, driver 204 may not at any point gaze in or near the direction of the jogger identified in 802. In such a scenario, ADAS 100 and / or state module 165 may decrement an alertness value. The alertness value may be stored in a driver profile. In one embodiment, object detection system 130 may identify the jogger as a high danger object with reference to a danger priority list and assign the jogger a higher weight than other objects. In such an embodiment, state module 165 may decrement the alertness value by a larger amount, immediately generate an alert, or perform other suitable action as compared to an object with a reduced danger priority.

[0186] In 810, ADAS 100 may determine that the gaze direction of driver 204 is / was / will be associated with the spatial location of the jogger. For example, driver 204 may gaze at the jogger identified in 802 either directly or in periphery. In such a scenario, ADAS 100 and / or state module 165 may increment an alertness value. Object detection system 130 may also consider a weighted value of the object reflecting its danger or importance in incrementing the alertness value.

[0187] In 812, ADAS 100 may display the alertness value on an on-board display, such as display 170. This provides driver 204 with real-time feedback about their driving performance and / or mental state / attention level. The alertness value may be stored in a driver profile.

[0188] FIG. 9 illustrates a method 900 for adjusting an ancillary system based on a driver’s rational response or lack thereof to a high danger object, according to some embodiments. Method 900 may be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein.Further, some of the steps may be performed simultaneously, or in a different order than shown in FIG. 9, as will be understood by a person of ordinary skill in the art(s).

[0189] In 902, ADAS 100 and / or state module 165 may receive sensor data from external sensing system 120. The sensor data may include images captured by cameras such as forward-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129. The sensor data may also include RADAR and LIDAR data captured by RADAR system 124 and LIDAR system 125. Object detection system 230 may use the data to build a contemporaneous, 360-degree, three-dimensional model of the surrounding environment. The sensor data may be captured at a particular frame rate, e.g., 60 frames per second, such that the 360-degree three-dimensional model may be rebuilt with regular periodicity, based on the data received from external sensing system 120 and system capabilities. For example, vehicle 202 may approach a disabled vehicle in a lane of travel on a highway.

[0190] In 904, ADAS 100 and / or object detection system 130 may identify objects within the 360-degree, three-dimensional model of the environment built with the sensor data received in 902. Objects may include other vehicles, lanes, traffic lights, signs, pedestrians, animals, traffic barriers, debris, and other objects. Object detection system 130 may also determine a spatial location of each object including a position, speed, velocity, acceleration, a direction of movement, and other suitable characteristics for each identified object. Object detection system 130 may employ a single-stage or multiplestage object-detection algorithm that leverages a trained neural network to identify objects within an image. Object detection system 130 may also classify each particular identified object as a vehicle, lane, traffic light, sign, pedestrian, etc. Object detection system 130 may determine a bounding box of the identified object(s) and a confidence score that the identified object is in fact an object (as compared to a shadow, discoloration, etc. in the source data. Object detection system 130 may also determine a classification for the object in each bounding box and a probability that the classification is correct. For example, to continue the example of the disabled vehicle, object detection system 130 may determine that the object is a vehicle, that the vehicle is in the current lane of travel, and that the vehicle is not moving. The object detector may identify such an object as a high danger object using a danger priority list. Gaze detector 114 may determine that a direction of the driver’s gaze either intersects with a bounding box surrounding the disabled vehicle or does not.

[0191] In 906, ADAS 100 may receive a rational response from driver 204. If the driver is not looking at the disabled vehicle, ADAS 100 may immediately proceed to 910. However, if driver 204 is looking at the disable vehicle, ADAS 100 may wait for an amount of time for a rational response to be received from the driver. Generally, a rational response may be an action taken by the driver to engage a mechanical system in the vehicle. For the disabled vehicle, the driver may on the brake pedal to slow the vehicle or perform a steering operation to change lanes.

[0192] In 908, having received a rational response from driver 204 that avoids a possible collision, ADAS 100 may also dynamically adjust parameters used by ancillary systems based on the driver’s mental state / attention level to increase the tolerance of other safety features. Such parameters may control a threshold time, distance, or other value used by the ancillary systems that dictate when the ancillary systems perform an action. Drivers may generally exhibit a preference to avoid unnecessary lane corrections performed by a lane detection system and by exhibiting a heightened mental state / attention level through rational responses, ADAS 100 may reward the driver by applying automatic corrections less frequently. As discussed above, this may apply in a collision detection system to increase the tolerance of the system in avoiding a disabled vehicle. Such ancillary systems may include autonomous driving, adaptive cruise control, blind-spot detection, lane departure warnings, collision avoidance, evasive maneuvering, parking assistance, and others. For example, the collision detection system may use a parameter that specifies a tolerance. Because the driver is paying attention, additional tolerance may be added to the system based on the driver’s heightened mental state / attention level. In an embodiment, ADAS 100 may consider data from weather system 128 in addition to the driver’s mental state / attention level in determining whether to adjust the parameters and increase the tolerance of the system.

[0193] In 910, not having received a rational response, ADAS 100 may cause a braking operation to slow the vehicle and avoid the collision or cause a steering operation to occur to change lanes and avoid the emergency. In 912, ADAS 100 may dynamically adjust parameters used by ancillary systems based on the driver’s mental state to decrease the tolerance of other safety features. Such ancillary systems may include autonomous driving, adaptive cruise control, blind-spot detection, lane departure warnings, collision avoidance, evasive maneuvering, parking assistance, and others. When the driver’smental state is decreasing, less tolerance in these systems may more proactively prevent accidents and increase vehicle safety.

[0194] FIG. 10 illustrates method 1000 for adjusting policy parameters based on a determined driver awareness state, according to some embodiments. Method 1000 may be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown in FIG. 10, as will be understood by a person of ordinary skill in the art(s).

[0195] In 1002, ADAS 100 and / or state module 165 may receive sensor data from external sensing system 120. The sensor data may include images captured by cameras such as forward-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129. The sensor data may also include RADAR and LIDAR data captured by RADAR system 124 and LIDAR system 125. Object detection system 230 may use the data to build a contemporaneous, 360-degree, three-dimensional model of the surrounding environment. The sensor data may be captured at a particular frame rate, e.g., 60 frames per second, such that the 360-degree three-dimensional model may be rebuilt with regular periodicity, based on the data received from external sensing system 120 and system capabilities.

[0196] In 1004, ADAS 100 and / or state module 165 may determine the driver awareness state based, at least in part, on the sensor data received in 1002. An awareness state may indicate the driver’s attention level, stress level, trust level, etc. In an embodiment, the awareness state may measure or indicate a driver’s focus, attentiveness, alertness, vigilance, awareness, cognizance, watchfulness, perceptiveness, clear-headedness, fatigue, tiredness, drowsiness, etc. In an embodiment, the awareness state may be determined relative to specific objects in the three-dimensional environment — z.e., an awareness state might consider whether a particular object was seen, missed, detected, comprehended, etc. In some embodiments, the determination of the awareness state may be supplemented with physiological data (e.g., heart rate, respiratory data, blood pressure, etc), external data (e.g., weather conditions, road conditions), and other suitable data.

[0197] In 1006, ADAS 100 may perform a dynamic parameter adjustment based on the driver awareness state determined in 1004. The range of parameters that may be adjustedto control driving policies is expansive, as will be understood by one skilled in the arts. Additionally, parameters may be included in particular policies that consider / reflect the driver’s anxiety-level, stress-level, nausea-level, etc. In one example, a policy may control the lane changes made by the vehicle. A driver may be responding negatively to frequent lane changes, and a parameter used by the lane-change policy may be updated to reduce the number of lane changes. In another example, the driver may respond negatively to the “jerky” nature of the autonomous driver, and the policy may for rate-of- acceleration changes by be updated to smooth the travel experience. This dynamic policy adjustment may address parameters used by various vehicle subsystems, e.g., (1) an adaptive cruise control system; (2) a collision avoidance system, (3) a blind-spot monitoring system, (4) a lane-change warning system, (5) a gaze-enabled lane change system. (6) a steering-assist system, (7) a lane-keep assist system, (8) a lane departure warning system, (9) an accidental-maneuver detection system, etc.

[0198] The parameters used by various subsystems may vary based on the behavior of the subsystem or policy. For example, adaptive cruise control system may include policies and parameters that specify: (1) a maximum permissible lateral acceleration, (2) a maximum stopping acceleration, (3) a minimum time gap, (4) a maximum time gap, (5) a minimum clearance, (6) a minimum speed, (7) a maximum speed, and / or (8) a distance to second vehicle. A blind-spot monitoring system, a lane-change warning system, and / or a gaze-enabled lane change system may include policies and parameters that specify: (1) a speed difference tolerance between the vehicle and a target vehicle, (2) a time to collision tolerance, or (3) a warning level. A collision avoidance system may include policies and parameters that specify an alarm tolerance level that controls when a collision avoidance system generates a driver alert. A steering-assist system, a lane-keep assist system, or a lane departure system may include policies and parameters that specify may include policies and parameters that specify: (1) a threshold distance to a lane boundary, (2) a warning level, or (3) a lane change tolerance.

[0199] By adjusting the parameters based on the driver’s awareness state, the driving performance may be maximized when the driver is under-stressed or in a high state of awareness. Conversely, when the driver is stressed or unaware, parameters may be increased / decreased accordingly to drive more conservatively.

[0200] FIG. 11 illustrates a method 1100 for detecting driver impairment based on eye scanning movements over a period of time, according to some embodiments. Method1100 may be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown in FIG. 11, as will be understood by a person of ordinary skill in the art(s).

[0201] In step 1102, ADAS 100 and / or state module 165 may receive sensor data from external sensing system 120. The sensor data may include images captured by cameras such as forward-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129. The sensor data may also include RADAR and LIDAR data captured by RADAR system 124 and LIDAR system 125. Object detection system 230 may use the data to build a contemporaneous, three-dimensional model of the surrounding environment. In some embodiments, the model may be a 360-degree representation of the surrounding environment. The sensor data may be captured at a particular frame rate, e.g., 60 frames per second, such that the three-dimensional model may be rebuilt with regular periodicity, based on the data received from external sensing system 120 and system capabilities. For example, vehicle 202 may approach object 206A — a STOP sign, and forward-facing camera(s) 122 and LIDAR system 125 may capture a sequence of images that include the STOP sign.

[0202] In step 1104, ADAS 100 and / or object detection system 130 may segment the 360-degree, three-dimensional model of the environment built using the sensor data received in step 702. Segmentation may involve identifying objects of interest within the three-dimensional model. Such objects may be other vehicles, lanes, traffic lights, signs, pedestrians, animals, traffic barriers, debris, etc. Object detection system 130 may also determine a spatial location of the objects including a position, a rate of movement (e.g., speed, velocity, acceleration), a direction of movement, and other suitable characteristics. Object detection system 130 may employ a single-stage or multiple-stage object-detection algorithm that leverages a trained neural network to both identify objects within an image. Object detection system 130 may also classify each particular identified object as a vehicle, lane, traffic light, sign, pedestrian, etc. Object detection system 130 may determine a bounding box of the identified object(s) and a confidence score that the identified object is in fact an object (as compared to a shadow, discoloration, etc. in the source data). Object detection system 130 may also determine a classification for theobject in each bounding box and a probability that the classification is correct. For example, to continue the example of the STOP sign, object detection system 130 may identify the STOP sign by placing a bounding box around the STOP sign. A bounding box may be represented using position information such as 4 or 8 coordinates for representing each corner of the rectangle or cuboid. Thus, to continue the STOP sign example, object detection system 130 may classify the object as a STOP sign and determine additional physical characteristics about the STOP sign such as velocity (in this case, the STOP sign is not moving) and position information within the model of the environment.

[0203] In step 1106, ADAS 100 and / or DMS 110 may determine the gaze direction of driver 204 of vehicle 202. For example, the gaze direction of driver 204 may be represented by a heatmap built by examining fixated and saccadic eye movements of driver 204 over time. DMS 110 may employ gaze detector 114 to track the gaze of driver 204 as a camera pose (having a position and orientation) within the 360-degree, three- dimensional model of the environment as reflected in the sensor data captured in step 702. In this manner, the driver’s gaze may be represented using three-dimensional geometry as a ray, vector, line, or other suitable geometric construct. In one embodiment, gaze detector 114 may use the Purkinje images on the eye (discussed above in FIG. 6) to determine gaze direction. DMS 110 may also consider the head position of the driver and other anatomical considerations in determining the driver’s gaze direction and building the heatmap.

[0204] In step 1108, ADAS 100 may determine if the gaze direction determined in step 1106 is associated or otherwise correlated with an object identified in step 1104. For example, ADAS 100 may determine that the driver’s gaze is currently intersecting with a spatial location of the object of interest, intersected with a spatial location of the object of interest in the past, or will intersect with a spatial location of the object of interest in the future. Step 1108 may be iterative because more than one object may be identified in a frame in step 1104. Thus, ADAS may work through each identified object using an iterative method or construct. In one embodiment, ADAS 100 may cast a ray from an origin at the point where the driver’s eyes are in the three-dimensional environment in accordance with the gaze direction determined in step 1106. ADAS 100 may determine if the ray intersects with a bounding box associated with the identified object. ADAS 100 may employ a suitable approximation (e.g., a number of degrees of tolerance) or expandthe bounding box to accommodate objects that the driver may see using peripheral vision. ADAS 100 may track the number of milliseconds that the gaze intersected with the bounding box and ensure that the intersection exceeds a threshold before considering an object to have been detected. In this regard, ADAS 100 may consider intersections with respect to a threshold amount of time or a threshold number of samples (given the sampling rate of the cameras). Conversely, if the driver’s gaze never intersects with an object’s bounding box or fails to do so for a threshold amount of time, then ADAS 100 may consider an object to have been missed by driver 204. For example, driver 204 may be looking around the car, up into the sky, etc. and not see the STOP sign identified as an object in step 1104.

[0205] In step 1110, ADAS 100 may determine the period of time that the driver’s gaze intersected with the bounding box (e.g., how long the object was considered “seen”) to confirm whether the gaze intersection exceeds a fixation threshold. This data is important for ADAS 100 to collect because the duration of gaze fixation can indicate whether a driver is impaired or not. For example, a driver experiencing a fatigue-based impairment may stare at an object longer than if the driver were alert e.g., a driver may stare at a particular object during highway hypnosis). In a similar manner, a driver experiencing a substance-based impairment may not be capable of quick eye-scanning movements. Therefore, if the driver’s gaze intersects with an object’s bounding box for too long beyond the fixation threshold, the ADAS 100 may use the collected time period data to consider whether the driver’s gaze fixation indicates an impairment in step 1112 below.

[0206] In step 1112, ADAS 100 and / or state module 165 may derive a conclusion or conclusions about the driver’s impairment level based on at least one of: a period of time of the driver’s gaze intersection with an object (when the object was considered seen), the driver’s saccadic eye movements during the period of time, and / or a characteristic of the driver during the period of time. The driver’s impairment level may comprise data representing at least one of drowsiness, microsleep, sleep, unresponsiveness, or intoxication. For example, ADAS 100 may determine that the driver is highly impaired because the driver’s gaze fixated on another vehicle for an excessively long period of time. Conversely, ADAS 100 may consider driver 204 to be barely impaired, if at all, if the eyes of driver 204 quickly scan the road and driver 204 consistently sees an object of importance or high danger as a result. ADAS 100 may consider excessive gaze fixation on objects in the aggregate as well — i.e., a total number of seen objects over a fixed timeperiod, how long the driver’s gaze fixated on each object, how many gaze intersections with objects exceed a gaze fixation threshold, etc. ADAS 100 may also apply a weighted algorithm that considers the importance or danger priority of an object as determined by the object classifier. A disabled vehicle may be given a higher danger priority than a billboard by classification manager 138. Thus, state module 165 may consider a missed disabled vehicle more heavily in calculating the impairment level than a lower weighted object.

[0207] In an embodiment, the driver’s impairment level may reflect the driver’s alertness or fatigue, such as states of drowsiness, microsleep, or sleep. Drowsiness may be a tired state, in which a driver yawns or blinks frequently, has difficulty remembering stretches of driving time, and misses turns or directions. Microsleep may be a state of short sleep periods lasting less than about 30 seconds, in which a driver nods off and suddenly jerks awake. Sleep may be an unconscious state in which a driver has limited responsiveness to external stimulation. For example, the impairment level may reflect these states with a spectrum based on the Karolinska Sleep Scale (KSS): (1) extremely alert; (2) very alert; (3) alert; (4) fairly alert; (5) neither alert nor sleepy; (6) some signs of sleepiness; (7) sleepy, no effort to keep alert; (8) sleepy, but some effort to keep alert; and (9) very sleepy, great effort to keep alert, fighting sleep.

[0208] In another embodiment, the driver’s impairment level may reflect the driver’s level of intoxication or unresponsiveness. Intoxication may be a state in which the driver is under the influence of drugs or alcohol such that mental and physical control is markedly diminished. Unresponsiveness may be a state in which the driver is unconscious, or even deceased, such that the driver does not respond to any verbal or physical stimuli. For example, the impairment level may reflect these states with a spectrum indicating the impacts of blood alcohol content (BAC): (1) sober and alert; (2) slightly lowered alertness; (3) impaired judgment and difficulty detecting danger; (4) reduced reaction time; (5) confusion; and (6) unresponsive.

[0209] When determining the driver’s impairment level, ADAS 100 may consider a gaze fixation over a fixed time period to evaluate a driver’s saccadic eye movements as a cognitive eye scanning pattern. The length of time in which a driver looks at an object may be indicative of fatigue-based impairment or a substance-based impairment. For example, an alert driver displays an “eye jitter,” in which the driver quickly glances at an object for a short period of time before switching to another object. These “eye jitters” areknown to a person skilled in the art as saccadic eye movements. Saccadic eye movements are quick, simultaneous movements of both eyes between phases of gaze fixation in the same direction as part of an eye scanning pattern. Typical saccadic eye movements can last about 20 ms to about 200 ms, depending on the amplitude of the movement. These saccadic eye movements can help a driver locate objects of interest and construct a mental map of the environment in which they are driving. Therefore, an alert driver relies on the saccadic eye movements to have situational awareness of the environment. Conversely, an impaired driver may not have a sufficient amount of saccadic eye movements to maintain situational awareness of the environment, instead fixating on an object for a prolonged period of time. Accordingly, ADAS 100 may track a driver’s saccadic eye movements over a period of time to determine the driver’s impairment level. ADAS 100 may use inward-facing camera(s) 112 with a high sampling frequency to sample an eye of the driver. For example, inward-facing camera(s) 112 may sample an eye of the driver with a sampling frequency of up to 500 times per second with an eye-tracking sensor (e.g., gaze detector 114).

[0210] To determine the driver’s impairment level based on the detected driver’s saccadic eye movements over a period of time, DMS 110 may build a stored driver profile to include a plurality of normal saccadic eye movements as a cognitive eye scanning pattern. These normal saccadic eye movements are considered to be “normal” when they are collected during routine, non-emergency driving circumstances (“normal” circumstances) when the driver is considered alert. A driver may be considered alert under normal circumstances if their saccadic eye movements last between about 20 ms to about 200 ms. ADAS 100 may store the stored driver profile (which includes the normal saccadic eye movements and a cognitive eye scanning pattern based on the normal saccadic eye movements) in a memory storage device (e.g., storage 195) for later data comparison. ADAS 100 may collect a plurality of saccadic eye movements in real time and compare the information to the stored driver profile to check for deviations. If the collected data of the plurality of saccadic eye movements diverges from the data in the stored driver profile, ADAS 100 may perform an action as described below in step 1114.

[0211] Additionally, when determining the driver’s impairment level, ADAS 100 and / or DMS 110 may determine a characteristic of driver 204 of vehicle 202 during a period of time using image data from inward-facing camera(s) 112. DMS 110 may employ gaze detector 114, face tracker 116, and / or motion sensor 118 to collect data regarding thecharacteristic. The characteristic of the driver may include a blink rate value, an eyelid closure duration value, an eyelid closure percentage value, an eye aspect ratio value, a gaze fixation value, or a yawning frequency value. DMS 110 may track the characteristic for a short-term time period (e.g. milliseconds, seconds, etc.) to monitor microadjustments of the driver’s face. DMS 110 may track the characteristic and / or saccadic eye movements for a long-term time period (e.g., minutes, hours, etc.) to build a data set for behavioral trend comparison to a stored driver profile.

[0212] A blink rate value may be determined based on how often a driver blinks. When a driver experiences an impairment (e.g., a fatigue-based impairment), their eye blinking pattern changes. Accordingly, a drowsy driver is more likely to exhibit longer, more frequent blinks than when alert. However, when a driver begins falling asleep, the blinks become so long that the blink frequency drops below that of an alert driver. For example, an alert driver may blink between about 15 times per minute and about 20 times per minute. Meanwhile, a drowsy driver may blink between about 20 times per minute and about 40 times per minute, and a sleepy driver may blink less than about 10 times per minute.

[0213] An eyelid closure duration value and an eyelid closure percentage value may be determined based on data regarding the length of time in which the driver’s eyelid is closed or mostly closed. The eyelid closure duration value may indicate how long the driver’s eyelid remains closed, indicating that the driver has begun sleeping. For example, eyelid closures lasting for more than about 3 seconds can show that a driver is falling asleep. The eyelid closure percentage value is known to a person skilled in the art as PERCLOS, which indicates the percentage of eyelid closure over a driver’s pupil over time as slow eyelid closures instead of blinks. Specifically, a PERCLOS drowsiness metric assesses the percentage of time in a minute that the driver’s eye is at least 80% closed.

[0214] An eye aspect ratio value may be determined based on the size of the driver’s eye that inward-facing camera(s) 112 may be able to detect. The eye aspect ratio value may also include information about the size of the driver’s pupil. If the driver’s eyelid is mostly closed, then the driver is more likely to be impaired.

[0215] A gaze fixation value may be determined based on the duration of a driver’s gaze fixation on a particular object. A gaze fixation is a stationary state of the driver’s eyes during which the driver is looking at a particular object of interest. The duration of thegaze fixation is the length of time in which the driver’s gaze remains intersected with the object. When a driver is experiencing an impairment, such as a fatigue-based impairment, the driver’s gaze fixation duration tends to last longer than if the driver were alert. For example, an alert driver’s gaze fixation duration may remain under 350 ms, while an impaired driver’s gaze fixation duration may exceed 500 ms.

[0216] A yawning frequency value may be determined based on how often a driver yawns. This data may be collected by face tracker 116 and / or motion sensor 118. Frequent yawning is likely caused by sleep deprivation and therefore may indicate that a driver is experiencing a fatigue-based impairment, such as drowsiness.

[0217] In step 1114, ADAS 100 and / or vehicle controllers 240 may perform a suitable action based on the impairment level determined in step 1112. ADAS 100 may generally inform the driver about whether their impairment level is improving or worsening. ADAS 100 may alert driver 204 about a particular excessively long gaze fixation on an object. A suitable alert may be displayed on the on-board display device, played as audio, or otherwise conveyed to the driver. For example, if the driver’s gaze intersects with another vehicle for longer than a gaze fixation threshold amount of time, mental state module 165 may generate an alert that the driver is fixated on the other vehicle — e.g., ADAS 100 may inform the driver “You have been staring too long and may be drowsy.” ADAS 100 may switch between autonomous driving levels, either removing responsibilities from driver 204 or increasing the driver’s responsibilities. ADAS 100 may engage a mechanical system to perform a steering operation, a braking operation, an acceleration operation to avoid collisions, correct the trajectory of the vehicle, etc. ADAS 100 may tighten the driver’s seat belt as a means of physically alerting the driver about their impairment level. For example, ADAS 100 may engage seat-belt system 148 to tighten the safety belt as a means of physically impacting driver 204 (which may serve to rouse or wake up the driver). AD AS 100 may impose speed or acceleration limits or remove such limits as the driver’s impairment level changes.

[0218] As discussed above with reference to FIG. 9, ADAS 100 may dynamically adjust parameters used by ancillary systems (e.g., collision avoidance, lane detection) based on the driver’s mental state to increase or decrease the tolerance of other safety features. As discussed below with reference to FIG. 12, ADAS 100 may also update a stored value indicative of the impairment level in a profile that tracks driver mental state over time. For example, ADAS 100 may update the stored impairment value when a plurality ofsaccadic eye movements diverge from the plurality of normal saccadic eye movements in the stored driver profile.

[0219] Therefore, in this configuration, ADAS 100 may combine physiological measurements (e.g., gaze tracking over a period of time) with DMS analysis of object detection (e.g., to gauge near-miss and alarming) to get a faster, more accurate response to a driver’s fatigue-based impairments and substance-based impairments.

[0220] FIG. 12 illustrates a method 1200 for tracking a driver alertness value using gaze tracking and object detection over a period of time, according to some embodiments. Method 1200 may be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown in FIG. 12, as will be understood by a person of ordinary skill in the art(s).

[0221] In step 1202, ADAS 100 and / or state module 165 may receive sensor data from external sensing system 120. The sensor data may include images captured by cameras such as forward-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129. The sensor data may also include RADAR and LIDAR data captured by RADAR system 124 and LIDAR system 125. Object detection system 230 may use the data to build a contemporaneous, three-dimensional model of the surrounding environment. The sensor data may be captured at a particular frame rate, e.g., 60 frames per second, such that the 360-degree three-dimensional model may be rebuilt with regular periodicity, based on the data received from external sensing system 120 and system capabilities. For example, vehicle 202 may approach object 206C — a jogger, and forward-facing camera(s) 122 and LIDAR system 125 may capture a sequence of images that include the jogger.

[0222] In step 1204, ADAS 100 and / or object detection system 130 may identify objects within the 360-degree, three-dimensional model of the environment built with the sensor data received in step 1202. Objects may include other vehicles, lanes, traffic lights, signs, pedestrians, animals, traffic barriers, debris, and other objects. Object detection system 130 may also determine a spatial location including a position, speed, velocity, acceleration, direction of movement, and other suitable characteristics for each identified object. Object detection system 130 may employ a single-stage or multiple-stage object-detection algorithm that leverages a trained neural network to both identify objects within an image. Object detection system 130 may also classify each particular identified object as a vehicle, lane, traffic light, sign, pedestrian, etc. Object detection system 130 may determine a bounding box of the identified object(s) and a confidence score that the identified object is in fact an object (as compared to a shadow, discoloration, etc. in the source data). For example, to continue the example of the jogger, object detection system 130 may identify the jogger by placing a bounding box (e.g., 4 or 8 coordinates representing the corners of the rectangle or cuboid) around the jogger. The bounding box may move with the jogger as time passes. Object detection system 130 may classify the object as a jogger and determine that the jogger is a high danger priority object given the jogger could behave unpredictably and given the potential harm / impact on a human life. Object detection system 130 may calculate a spatial location including a position, velocity, and direction of movement of the jogger.

[0223] In step 1206, ADAS 100 and / or DMS 110 may determine the gaze direction of driver 204 of vehicle 202. DMS 110 may employ gaze detector 114 to track the gaze of driver 204 as a heatmap. The heatmap may be built using both fixated and saccadic eye movements of the driver. The heatmap may further consider an origin and 6 DoF coordinate frame representing the position of the driver’s head within the 360-degree, three-dimensional model of the environment as reflected in the sensor data captured in step 1202. In this manner, the driver’s gaze may be represented using three-dimensional geometry as a ray, vector, line, or other suitable geometric construct. In one embodiment, gaze detector 114 may use the Purkinje images on the eye (discussed above in FIG. 6) to determine gaze direction. DMS 110 may also consider the head position of the driver and other anatomical considerations in determining the driver’s gaze direction and building the heatmap.

[0224] In step 1208, ADAS 100 may determine if the gaze direction determined in step 1206 is associated or otherwise correlated with an object identified in step 1204. For example, ADAS 100 may determine that the driver’s gaze is currently intersecting with a spatial location of the object of interest, intersected with a spatial location of the object of interest in the past, or will intersect with a spatial location of the object of interest in the future. Step 1208 may be iterative because more than one object may be identified in a frame in step 1204, and thus, ADAS may work through each identified object using an iterative method or construct. In one embodiment, ADAS 100 may cast a ray or vectorfrom an origin at the point where the driver’s eyes are in the three-dimensional environment in accordance with the gaze direction determined in step 806. ADAS 100 may determine if the ray intersects with a bounding box associated with the identified object. ADAS 100 may employ a suitable approximation (e.g., a number of degrees of tolerance) or expand the bounding boxes to accommodate objects that the driver may see using peripheral vision. ADAS 100 may track the number of milliseconds that the gaze intersected with the bounding box and ensure that the intersection exceeds a threshold before considering an object to have been detected. In this regard, ADAS 100 may consider intersections with respect to a threshold amount of time or a threshold number of samples (given the sampling rate of the cameras). If the driver’s gaze never intersects with an object’s bounding box or fails to do so for a threshold amount of time, then ADAS 100 may consider the object missed.

[0225] In step 1210, ADAS 100 may determine the period of time that the driver’s gaze is associated with the bounding box (e.g., how long the object was considered “seen”) and evaluate whether the gaze intersection exceeds a fixation threshold. This data is important for ADAS 100 to collect because the duration of gaze fixation can indicate whether a driver is impaired or not. For example, a driver experiencing a fatigue-based impairment may stare at an object longer than if the driver were alert (e.g., a driver may stare at a particular object during highway hypnosis). ADAS 100 may use a dynamic fixation threshold that depends on the classified danger priority of the object. For example, it may be permissible for a driver’s gaze to fixate longer on a pedestrian in the middle of the road, while it may not be permissible for a driver’s gaze to fixate on a green light without accelerating the vehicle. Therefore, object detection system 130 may classify each particular identified object with a unique fixation threshold.

[0226] In step 1212, ADAS 100 may determine that the gaze duration of driver 204 exceeds the fixation threshold when looking at an identified object, for example, the jogger identified in step 1202. In such a scenario, ADAS 100 and / or state module 165 may decrement an alertness value. The alertness value may be stored in a driver profile. In one embodiment, object detection system 130 may identify the jogger as a high danger object with reference to a danger priority list and assign the jogger a higher fixation threshold than other objects. In such an embodiment, mental state module 165 may wait longer to decrement the alertness value as compared to an object with a reduced danger priority. Additionally, object detection system 130 may assign a motorcycle (e.g., object206B as shown in and described with regard to FIG. 2A) in the opposite lane a lower fixation threshold compared to the jogger. In such an embodiment, state module 165 may decrement the alertness value if the duration of the driver’s gaze exceeds the fixation threshold for the motorcycle.

[0227] In step 1214, ADAS 100 may determine that the gaze duration of driver 204 is under the fixation threshold when looking at an identified object, for example, the jogger identified in step 1202. For example, driver 204 may gaze at the jogger identified in step 1202 either directly or in periphery during a period of time lasting less than the fixation threshold. In such a scenario, ADAS 100 and / or state module 165 may increment an alertness value. Object detection system 130 may also consider a weighted value of the object reflecting its danger or importance, and associated fixation threshold, in incrementing the alertness value.

[0228] In step 1216, ADAS 100 may display the alertness value on an on-board display, such as display 170. This provides driver 204 with real-time feedback about their driving performance and / or impairment level. The alertness value may be stored in a driver profile.Use of Polarized Light

[0229] As background when a light beam propagates in a specific direction, the light beam has an electric field vector and a magnetic field vector. The electric field vector and the magnetic field vector are perpendicular to each other and to the propagation direction. The orientation of these field vectors denotes the polarization of the light beam. An unpolarized light beam is characterized by an incoherent combination of electric field vectors that can occur at any azimuthal angle around the propagation direction. There are at least two types of polarization for a beam of light — linear polarization and circular polarization. A linearly polarized light beam is characterized by an electric field vector with a fixed azimuthal angle around the propagation direction. Accordingly, the resultant wave lies in a plane. The amount of light incident on a linear polarizing optical device that passes through the optical device depends on the angle between the electric field vector of the light beam and the transmission axis of the optical device. A linear polarizing optical device only allow for transmission of a light having an electric field vector parallel to the transmission axis of the optical device. In contrast, a linearpolarizing optical device can block a light beam having an electric field vector perpendicular to the transmission axis of the optical device.

[0230] A circularly polarized light beam is characterized by an electromagnetic field with a constant magnitude that is rotating at a constant rate in a plane perpendicular to the propagation direction. Circular polarization occurs when the two orthogonal electric field component vectors are of equal magnitude and are out of phase by 90 degrees with each other. Accordingly, the resultant wave lies on a helix oriented along the propagation direction. A circularly polarized wave can rotate in one of two ways: (1) a right-handed circular polarization (RHCP), in which the electric field vector rotates in a right-hand sense with respect to the direction of propagation; and (2) a left-handed circular polarization (LHCP), in which the electric field vector rotates in a left-hand sense. Therefore, a circular polarizing optical device can transmit any photons traveling along the path of the resultant wave while blocking others deviating from that path.

[0231] FIG. 13 illustrates a photograph of unpolarized illumination 1300 of polarized sunglasses 1302 reflecting unpolarized light as glint spots 1304a, 1304b, according to some embodiments. A person’s eyes may be located behind polarized sunglasses 1302.

[0232] Polarized sunglasses 1302 may be linearly or circularly polarized. Polarized sunglasses 1302 may limit, or even block, transmission of a light beam associated with unpolarized illumination 1300. If a light beam associated with unpolarized illumination 1300 is not aligned with a transmission axis of polarized sunglasses 1302, the unpolarized illumination 1300 may be attenuated polarized sunglasses 1302. This misalignment can result in decrease visibility of the person’s eyes through polarized sunglasses 1302, or even complete obscure the person’s eyes from a perspective of a camera system. Accordingly, most of the light beam of unpolarized illumination 1300 may be specularly reflected from polarized sunglasses 1302 as glint spots 1304a, 1304b. As a consequence, eyes of a person behind polarized sunglasses 1302 may be barely discernable by a camera system. Thus, unpolarized illumination 1300 of polarized sunglasses 1302 could impede a driver monitoring system from accurately tracking a person’s eye movements.Example Camera System

[0233] FIG. 14 illustrates a camera system 1400 with a polarizing optical device, according to some embodiments. Camera system 1400 is configured to monitor at least one person in a vehicle and, more particularly, track eye movements of the person. Theperson in the vehicle may be a user, a driver, a passenger, etc. Camera system 1400 may include an illuminator 1402, a polarizing optical device 1406, a camera 1416, and a processor 1418. In some embodiments, camera system 1400 can be a component of DMS 110 described above with reference to FIG. 1.

[0234] Illuminator 1402 is configured to emit an unpolarized light beam 1404 along an optical path. The optical path may align with an eye 1412 of at least one person in the vehicle, such as a user, a driver, a passenger, etc. Eye 1412 may be obstructed by a polarized surface 1410.

[0235] Illuminator 1402 may comprise a plurality of light-emitting diodes (LEDs). For example, illuminator 1402 may include a plurality of LEDs ranging from one LED to four LEDs. The plurality of LEDs may be arranged in at least one of a linear strip, a grid of rows and columns, a rhombus shape, a parallelogram shape, or an irregular cluster. In some embodiments, illuminator 1402 may include a plurality of low-power LEDs with a power output range from about 0 W to about 0.99 W. In some embodiments, illuminator 1402 may include a plurality of high-power LEDs with a power output range from about 1 W to about 10 W.

[0236] Unpolarized light beam 1404 may have an electric field vector that can be at any azimuthal angle around the propagation direction along the optical path (e.g., toward eye 1412). In some embodiments, unpolarized light beam 1404 may have a wavelength just outside a visible wavelength range of about 380 nm to about 750 nm, so as to be invisible to the naked eye. In some embodiments, unpolarized light beam 1404 may be a beam of near-infrared (“near-IR”) light. For example, unpolarized light beam 1404 may have a wavelength within a near-IR wavelength range of about 750 nm to about 1150 nm. Specifically, unpolarized light beam 1404 may have a near-IR wavelength of about 940 nm. The naked eye can be safely exposed to a wavelength of about 940 nm.

[0237] Unpolarized light beam 1404 extends from illuminator 1402 to polarizing optical device 1406. Polarizing optical device 1406 may be a polarizing filter. In an embodiment, polarizing optical device 1406 may be disposed in the optical path of unpolarized light beam 1404. For example, polarizing optical device 1406 may be disposed in the optical path of unpolarized light beam 1404 between illuminator 1402 and eye 1412 of a person in the vehicle. In another example, polarizing optical device 1406 can be mounted adjacent to illuminator 1402 with a predetermined amount of clearance between a surface of polarizing optical device 1406 and a surface of illuminator 1402. In anotherembodiment, polarizing optical device 1406 may be disposed on illuminator 1402. For example, polarizing optical device 1406 can be attached to illuminator 1402 with fasteners such as bolts, adhesive, magnets, clips, rotatable cams, and the like.

[0238] Polarizing optical device 1406 is configured to perform polarizing of unpolarized light beam 1404 (e.g., a near-IR light beam) to generate a polarized light beam 1408. The polarization may be a circular polarization or a linear polarization.

[0239] In an embodiment where polarizing optical device 1406 performs a circular polarization, polarizing optical device 1406 may perform at least one of left-handed circular polarization or right-handed circular polarization on unpolarized light beam 1404. Polarizing optical device 1406 may be aligned with illuminator 1402 to achieve a specific orientation for the circular polarization of unpolarized light beam 1404. In some embodiments, polarizing optical device 1406 may be aligned relative to a transmission axis of polarized surface 1410 such that polarized light beam 1408 may pass through polarized surface 1410 substantially without attenuation (due to the matching circular polarization). As a result, polarizing optical device 1406 may provide a significant improvement over the prior art at reducing unwanted reflections or glare spots (e.g., glint spots 1304a, 1304b as shown in and described with reference to FIG. 13) from polarized surface 1410.

[0240] Alternatively, polarizing optical device 1406 may be configured to perform linear polarization on unpolarized light beam 1404. In some embodiments, polarizing optical device 1406 may be aligned with illuminator 1402 to achieve a specific orientation for the linear polarization of unpolarized light beam 1404. In some embodiments, polarizing optical device 1406 may be aligned relative to a transmission axis of polarized surface 1410 such that polarized light beam 1408 may pass through polarized surface 1410 substantially without attenuation (due to the matching linear polarization).

[0241] Regardless of whether polarized light beam 1408 is polarized in a circular or linear fashion, polarized light beam 1408 may be configured to penetrate polarized surface 1410 as a result of the polarization induced by polarizing optical device 1406. When circularly polarized, polarized light beam 1408 may be circularly polarized to match a transmission axis of a polarized surface 1410 having a circular polarization. For example, polarized light beam 1408 may have an electric field vector that rotates in a helix oriented along the propagation direction toward eye 1412.

[0242] When linearly polarized, polarized light beam 1408 may be linearly polarized to match a transmission axis of a polarized surface 1410 having a linear polarization. For example, polarized light beam 1408 may have an electric field vector with a fixed azimuthal angle around the propagation direction toward eye 1412.

[0243] Polarized light beam 1408, as a modification of unpolarized light beam 1404, may still have a wavelength just outside a visible wavelength range of about 380 nm to about 750 nm, so as to be invisible to the naked eye. In some embodiments, polarized light beam 1408 may be a beam of near-infrared (near-IR) light. For example, polarized light beam 1408 may have a wavelength within a near-IR wavelength range of about 750 nm to about 1150 nm. Specifically, polarized light beam 1408 may have a near-IR wavelength of about 940 nm. The naked eye can be safely exposed to a wavelength of about 940 nm.

[0244] As described above with respect to FIG. 13, polarized surface 1410 may obstruct at least some of unpolarized light beam 1404 from reaching eye 1412. For example, polarized surface 1410 may have a transmission axis configured to block unpolarized light. In some embodiments, polarized surface 1410 may be circularly polarized. In this example, polarized surface 1410 may have a transmission axis that allows polarized light beam 1408, having a circular polarization in this example, to pass through polarized surface 1410. In some embodiments, polarized surface 1410 may be linearly polarized. In this example, polarized surface 1410 may have a transmission axis that allows polarized light beam 1408, having a linear polarization in this example, to pass through polarized surface 1410. In some embodiments, polarized surface 1410 may be a lens of commercially available polarized sunglasses.

[0245] After polarized light beam 1408 passes through polarized surface 1410, polarized light beam 1408 may reflect from eye 1412 to become reflected polarized light beam 1414. Reflected polarized light beam 1414 may propagate from eye 1412 and travel toward camera 1416. Reflected polarized light beam 1414 may possess the same characteristics as those of polarized light beam 1408, specifically regarding polarization and wavelength.

[0246] Camera 1416 may be oriented towards at least one person in a vehicle (e.g., a user, a driver, a passenger, etc.), specifically eye 1412 of the person. In some embodiments, camera 1416 may comprise inward-facing camera(s) 112 (as shown in and described with reference to FIG. 1). For example, camera 1416 may capture images of a person’s facial expressions to determine if a person is asleep, incapacitated, or otherwiseincapable of performing their responsibilities. In one embodiment, camera 1416 may be an 8, 12, 24, 50, 96, 200, or other high megapixel camera having a particular field of view.

[0247] Camera 1416 may be configured to capture infrared or other wavelength of light that is invisible to the human eye to track a person’s facial expression. In some embodiments, camera 1416 may capture wavelengths just outside a visible wavelength range of about 380 nm to about 750 nm. In some embodiments, camera 1416 may be a near-IR camera capable of capturing a near-IR light beam (e.g., reflected polarized light beam 1414). For example, camera 1416 may capture wavelengths within a near-IR wavelength range of about 750 nm to about 1150 nm. Specifically, camera 1416 may capture a near-IR wavelength of about 940 nm.

[0248] In this way, camera 1416 is configured to capture an image of a polarized light beam, such as reflected polarized light beam 1414 (e.g., the polarized light beam reflected from eye 1412 of the person). Camera 1416 may capture a clear, high-quality image of eye 1412 based on reflected polarized light beam 1414. For example, this clear, high- quality image may make polarized surface 1410 appear transparent so that a distinct position of a pupil of eye 1412 can be determined by a driver monitoring system (e.g., DMS 110 as shown in and described with reference to FIG. 1).

[0249] Processor 1418 may be coupled to camera 1416 (e.g., a near-IR camera).Processor 1418 may be configured to conduct monitoring of at least one person in the vehicle based on the image of eye 1412 captured by camera 1416 from reflected polarized light beam 1414. Processor 1418 may be configured to conduct driver monitoring. In some embodiments, processor 1418 may comprise at least one of DMS 110, gaze detector 114, or face tracker 116 (as shown in and described with reference to FIG. 1). For example, processor 1418 may be configured to determine a gaze direction of the person based on the image of eye 1412. To ascertain the direction of a person’s gaze, processor 1418 may employ a pupil-corneal reflection technique to generate a Purkinje image with infrared light, as discussed above with reference to FIG. 6.

[0250] FIG. 15 illustrates a camera system 1500 with a rotating polarizing optical device, according to some embodiments. In some embodiments, camera system 1500 can be an alternate embodiment of camera system 1400 as shown in and described with reference to FIG. 14. It is noted that, the same components that have been described above in connection with FIG. 14, such as illuminator 1402, polarizing optical device 1406,camera 1416, etc., are not repeated herein. In some embodiments, camera system 1500 can include an actuator 1520, an image sensor 1524, and a controller 1526.

[0251] Actuator 1520 may be configured to rotate polarizing optical device 1406 with rotational motion 1522 to perform the polarization on unpolarized light beam 1404. In some embodiments, actuator 1520 can rotate polarizing optical device 1406 to perform circular polarization to produce a polarized light beam 1408 having a circular polarization. In other embodiments, actuator 1520 can rotate polarizing optical device 1406 to perform linear polarization to produce a polarized light beam 1408 having a linear polarization. Actuator 1520 may rotate polarizing optical device 1406 to a desirable rotation angle where transmission of polarized light beam 1408 through a transmission axis of polarized surface 1410 (as shown in and described with reference to FIG. 14) is greatest.

[0252] Actuator 1520 may be a commercially available motor sized and shaped for rotating an optical device, a filter, and / or a lens, such as a stepper motor. Actuator 1520 may be configured to rotate polarizing optical device 1406 in at least one of a clockwise or counter-clockwise manner about a rotation axis located at the optical path where unpolarized light beam 1404 and polarized light beam 1408 travel toward eye 1412 (as shown in and described with reference to FIG. 14).

[0253] Image sensor 1524 may be coupled to camera 1416 to analyze the image captured by camera 1416 based on reflected polarized light beam 1414 (as shown in and described with reference to FIG. 14). Image sensor 1524 may determine an eye visibility value for each image of polarized surface 1410 and eye 1412 detected at each rotation angle of polarizing optical device 1406 provided by actuator 1520. Image sensor 1524 may compare eye visibility values of each captured image against a predetermined eye visibility threshold indicating that polarized surface 1410 appears transparent and a pupil location of eye 1412 is discernable. In this configuration, image sensor 1524 can evaluate whether the rotation angle of polarizing optical device 1406 provided by actuator 1520 is suitable for monitoring the person in the vehicle. As a result, image sensor 1524 can transmit a signal to controller 1526 regarding an optimal rotation angle based on the eye visibility threshold.

[0254] Controller 1526 may be configured to receive signals from image sensor 1524 and transmit rotation commands to actuator 1520. As a result, controller 1526 may enableactuator 1520 to selectively position polarizing optical device 1406 so that camera 1416 perceives an optimal visibility of eye 1412 through polarized surface 1410.

[0255] FIG. 16 illustrates a camera system 1600 with a plurality of polarizers rotated relative to each other, according to some embodiments. In some embodiments, camera system 1600 can be an alternate embodiment of camera systems 1400 and 1500 as shown in and described with reference to FIGS. 14 and 15, respectively. It is noted that, the same components that have been described above in connection with FIGS. 14 and 15, such as illuminator 1402, camera 1416, etc., are not repeated herein. In some embodiments, camera system 1600 can include light sources 1630a-1630d and polarizers 1632a-1632d.

[0256] Illuminator 1402 may comprise a plurality of light sources 1630a-1630d. Each of light sources 1630a-1630d may be configured to emit a plurality of respective unpolarized light beams 1404a-1404d (e.g., light source 1630a emits unpolarized light beam 1404a, etc.) along an optical path. The optical path may align with an eye 1412 (as shown in and described with reference to FIG. 14) of a person in the vehicle. Eye 1412 may be obstructed by a polarized surface 1410 (as shown in and described with reference to FIG. 14). In some embodiments, the plurality of light sources 1630a-1630d can be a plurality of LEDs. In the example shown in FIG. 16, the plurality of light sources 1630a-1630d can be a set of four LEDs. Light sources 1630a-1630d may be arranged in at least one of a linear strip, a grid of rows and columns, a rhombus shape, a parallelogram shape, or an irregular cluster.

[0257] Unpolarized light beams 1404a-1404d may possess the same characteristics as those of unpolarized light beam 1404 (as shown in and described with reference to FIG. 14), specifically regarding polarization and wavelength.

[0258] Polarizing optical device 1406' may be an alternative embodiment of polarizing optical device 1406, as shown in and described with reference to FIGS. 14 and 15. In the example embodiment shown in FIG. 16, polarizing optical device 1406' may comprise a plurality of polarizers 1632a-1632d. Polarizers 1632a-1632d may be configured to perform polarizing of each respective unpolarized light beam 1404a-1404d to generate a plurality of respective polarized light beams 1408a-1408d (e.g., polarizer 1632a may perform polarizing of unpolarized light beam 1404a to generate polarized light beam 1408a, etc.).

[0259] In an embodiment, each of polarizers 1632a-1632d may be disposed in the optical path of a respective one of unpolarized light beams 1404a-1404d (e.g., polarizer 1632amay be disposed in the optical path of unpolarized light beam 1404a, etc.). For example, each of polarizers 1632a-1632d may be disposed in the optical path of a respective one of unpolarized light beams 1404a-1404d between illuminator 1402 and eye 1412 of a person in the vehicle. In another example, each of polarizers 1632a-1632d may be mounted adjacent to light sources 1630a-1630d with a predetermined amount of clearance between a surface of polarizing optical device 1406' and a surface of illuminator 1402. In another embodiment, each of polarizers 1632a-1632d may be disposed on a respective one of light sources 1630a-1630d (e.g., polarizer 1632a may be disposed on light source 1630a, etc.).

[0260] Polarizers 1632a-1632d may be rotated relative to each other by predetermined intervals of degrees. Due to the relative rotation between polarizers 1632a-1632d, each one of polarizers 1632a-1632d may have a unique relative alignment with a transmission axis of polarized surface 1410 such that each one of polarized light beams 1408a-1408d may pass through polarized surface 1410 with a unique amount of attenuation. In the example shown in FIG. 16, the predetermined intervals of degrees comprise 90-degree intervals, as depicted by the orientation of indicator lines 1634a-1634d respectively disposed on each one of polarizers 1632a-1632d (e.g., indicator line 1634a may be disposed on polarizer 1632a, etc.). Indicator lines 1634a-1634d may be for illustrative purposes only and may not be actual markings on polarizers 1632a-1632d.

[0261] Image sensor 1524 may be coupled to camera 1416 (as shown in and described with reference to FIG. 15) to analyze the images captured by camera 1416. Image sensor 1524 may determine an eye visibility value for each image of polarized surface 1410 and eye 1412 detected for each of polarizers 1632a-1632d rotated relative to each other by predetermined intervals of degrees. Image sensor 1524 may compare eye visibility values of each captured image against a predetermined eye visibility threshold indicating that polarized surface 1410 appears transparent and a pupil location of eye 1412 is discernable. In this configuration, image sensor 1524 can evaluate which one of polarized light beams 1408a-1408d, and therefore polarizers 1632a-1632d, is most suitable to use for monitoring a person in the vehicle. As a result, image sensor 1524 can transmit a signal to controller 1526 regarding which one of light sources 1630a-1630d to illuminate for the optimal choice of one of polarizers 1632a-1632d based on the eye visibility threshold.

[0262] Controller 1526 may be configured to receive signals from image sensor 1524 and transmit illumination commands to illuminator 1402. As a result, controller 1526 may enable illuminator 1402 to selectively blink one of light sources 1630a-1630d corresponding to the one of polarizers 1632a-1632d that provides to camera 1416 an optimal visibility of eye 1412 through polarized surface 1410.Example Image Capturing Method

[0263] FIG. 17 illustrates a method 1700 for capturing an image for monitoring at least one person in a vehicle, according to some embodiments. The person in the vehicle may be a user, a driver, a passenger, etc.

[0264] At step 1702, a light beam (e.g., unpolarized light beam 1404) may be emitted with an illuminator (e.g., illuminator 1402) along an optical path. The light beam may be a beam of near-IR light. The light beam may have a near-IR wavelength of about 940 nm. The optical path may align with an eye (e.g., eye 1412) of at least one person in the vehicle. The eye may be obstructed by a polarized surface (e.g., polarized surface 1410).

[0265] At step 1704, polarizing is performed, with a polarizing optical device (e.g., polarizing optical device 1406) disposed in the optical path of the light beam (e.g., unpolarized light beam 1404), of the light beam to generate a polarized light beam (e.g., polarized light beam 1408). In an embodiment, the polarizing optical device may be disposed in the optical path of the light beam between the illuminator (e.g., illuminator 1402) and an eye (e.g., eye 1412) of a person in the vehicle. In another embodiment, the polarizing optical device may be disposed on the illuminator.

[0266] In some embodiments, the polarizing comprises performing circular polarization of the light beam. In some embodiments, the performing circular polarization comprises rotating the polarizing optical device with an actuator (e.g., actuator 1520). In some embodiments, the performing circular polarization comprises transmitting the light beam through a plurality of polarizers (e.g., polarizers 1632a-1632d) rotated relative to each other by predetermined intervals of degrees.

[0267] In some embodiments, the polarizing comprises performing linear polarization of the light beam.

[0268] At step 1706, the polarized surface is penetrated by the polarized light beam.

[0269] At step 1708, a camera captures an image of the polarized light beam. The camera may be a near-IR camera. The camera may capture an image of the polarized light beam after the polarized light beam has reflected from an eye of a person in the vehicle.

[0270] At step 1710, a gaze direction of the person in the vehicle is determined based on the image with a processor (e.g., processor 1418) coupled to the camera.

[0271] The method steps of FIG. 17 can be performed in any conceivable order and it is not required that all steps be performed. Moreover, the method steps of FIG. 17 described above merely reflect an example of steps and are not limiting. That is, further method steps and functions are envisaged based embodiments described in reference to FIGS. 1- 16.

[0272] Embodiments are provided below to allow a driver monitoring system to accurately track a person’s eye movements when wearing polarized sunglasses.Classifying Traffic Control Objects

[0273] FIG. 18A illustrates environment 1800 A in which ADAS 100 may operate, according to some embodiments. As illustrated in FIG. 18, environment 1800A may include vehicle 1802, driver 1804, object 1808 A, object 1808B, and lanes 1810A and 1810B. Environment 1800A displayed in FIG. 18A is a view from outside of the vehicle. FIGS. 18B and 18C reflect corresponding views of environment 1800A: inside the vehicle and the view of the external sensing system, respectively. However, the situation depicted in FIGS. 18A-C is merely exemplary, and limitless possibilities exist regarding the nature of environments that ADAS 100 may encounter in the real world.

[0274] Vehicle 1802 is illustrated in environment 1800A as driving down the right side of the road in the rightmost lane. In addition to an automobile, vehicle 1802 may be any other suitable type of vehicle including a truck, van, motorcycle, all-terrain vehicle, military vehicle, boat, etc. In addition to the roadway example depicted, any suitable transportation infrastructure where the lanes used by vehicles, or transportation devices, may be applicable.

[0275] Driver 1804 may be a human being operating vehicle 1802. At any given time, driver 1804 may have different responsibilities in driving vehicle 1802 ranging from responsibility over all of the vehicle’s mechanical systems (Level 0) to no responsibilities at all (Level 5). In some embodiments, instead of ADAS the vehicle may be an autonomous vehicle utilizing an autonomous driving system, e.g., a system that requiresno driver responsibilities (Level 5) and therefore there may not be a “driver.” The description of FIGS. 18A-C and throughout the detailed description apply to ADAS and AV scenarios alike.

[0276] In the example of environment 1800A, objects 1808A and 1808B are both traffic control objects, e.g., traffic lights. These objects may be identified by object detection system 130. In some embodiments, objects 1808A and 1808B may be other types of traffic control objects, e.g., traffic lights, signs, signals, road marking, barricades, or other traffic control devices meant to manage traffic and provide instruction to a driver and / or vehicle on the road. FIGS. 18A-18C depict traffic control objects, 1808A and 1808B, but some scenarios may incorporate more traffic control objects or fewer.

[0277] The example roadway in environment 1800 A includes lanes 1810A and 1810B. Both lanes 1810A and 1810B are used by vehicles traveling in the same direction, as indicated by vehicle 1802. Environment 1800A depicts lanes 1810A / 1810B at an intersection. The illustrated intersection is a 4-way or four-legged intersection, however in various embodiments the intersection may have greater or fewer intersecting roads or legs. Additionally, the type of intersection may change. For example, a skewed intersection, i.e. the angles of intersection are not 90 degrees, staggered intersection, i.e. secondary roads intersecting a main road at a short distance apart, and / or a round-a-bout or circular intersection.

[0278] The intersection may be managed by traffic control objects and lanes 1810A / B may be managed by traffic control objects 1808 A and 1808B. For example, when approaching the intersection, driver 1804 may look at traffic control object 1808B for instruction on proceeding in the intersection. DMS 110 may track gaze direction 1806 of driver 1804. Additionally, object detection system 130 may detect objects in environment 1800A. For the purposes of this description, classification manager 138 may filter out objects that are not traffic control objects, such as pedestrians, other vehicles, animal, or other object described above. Classification manager 138 may not always filter objects based on this criteria (e.g., traffic control objects versus other objects) but when ADAS 100 is being used to classify traffic control objects, this type of filtering may be performed to efficiently identify objects of interest.

[0279] Using gaze direction 1806 determined by DMS 110, and object detection system 130, ADAS 100 may determine that driver gaze is intersecting a bounding box associated with traffic control object 1808B. This may indicate to ADAS 100 that traffic controlobject 1808B is managing traffic for lane 1810B. In response to determining that gaze direction 1806 is intersecting a bounding box associated with traffic control object 1808B, ADAS 100, via computer system 2300 may generate data indicating that traffic control object 1808B is managing traffic for lane 1810B.

[0280] In some embodiments, if there has been data previously generated indicating which lane traffic control object 1808B is managing traffic for, computer system 2300 may compare the data of the current interaction with stored data from previous interactions. For example, comparing the data indicating traffic control object 1808B is managing traffic for lane 1810B with previous data based on interactions between a driver / vehicle in lane 1810B and / or looking at traffic control object 1808B.

[0281] In some embodiments, this may allow the system to be trained. The process described here, and method 1900 described in FIG. 19, may be repeated until a threshold amount of data has been generated from interactions between drivers / vehicles in lane 1810B with traffic control object 1808B. The threshold may be set based on safety standards of a vehicle manufacturer, ADAS manufacturer, a regulatory body and / or a combination of entities representing stakeholders. For example, a regulatory scheme may require that associating a traffic control object with a lane of traffic may be proven repeatedly by 100 drivers and / or vehicles before it may be implemented. In another example, a driver assistance (e.g., ADAS 100) or automated driver system, may require a different threshold to be met before the manufacturer will include this association in the ruleset for a driver assistance or autonomous driving system, (e.g., autonomous driving system 155.)

[0282] Once the threshold of data representing the interaction in the example has been generated, the computer system may crowdsource the data to confidently determine that traffic control object 1808B manages traffic for lane 1810B. Computer system 2300 may update object map manager 134 to include the data that indicates traffic control object 1808B manages traffic for lane 1810B. This information may also be utilized by other systems within ADAS, for example autonomous driving system 155 and policy manager 160.

[0283] In some embodiments, having a threshold amount of data confirming the association of traffic control objects and corresponding lanes, may allow the system to detect and discard outliers. For example, vehicle 1802 may occupy lane 1810A at the intersection. Instead of looking at traffic control object 1808A such that gaze direction1806 intersects a bounding box associated with traffic control object 1808 A, gaze direction 1806 may instead intersect with traffic control object 1808B or not intersect with a bounding box associated with a traffic control object. Computer system 2300 may compare the data of the interaction with the stored data and determine that the interaction is an outlier and should not be included in the dataset.

[0284] Further, as traffic patterns change and / or traffic control objects are updated, moved, removed, etc., computer system 2300 may be able to recognize that repeated outliers may indicate a change associated with the traffic control object that is managing the lane of traffic. In this instance, the process described above may be repeated until a threshold of data for interactions with a traffic control object has been generated for the lane. Once the threshold is met, the rule may be updated. For example, a traffic control object 1808C may be added to the intersection in FIG. 18 to indicate when the vehicles in lane 1810A may execute a left hand turn. Computer system 2300 may at first classify the interactions between drivers in lane 1810A and traffic control object 1808C as an outlier. However, object map manager 134 may be updated to include traffic control object 1808C and a left hand turn arrow marking lane 1810A. Detector 132 may confirm the presence of traffic control object 1808C and classification manager 138 may classify the object as a traffic control object. The system may then perform process 1900 and determine that traffic control object 1808C manages left hand turns in lane 1810A.

[0285] FIG. 18B illustrates environment 1800B in which ADAS 100 may operate, according to some embodiments. Environment 1800B corresponds with environment 1800A, only displayed from the perspective inside of the vehicle. As illustrated in FIG. 18 A, environment 1800B may include gaze direction 1806, and traffic control objects 1808A and 1808B.

[0286] Gaze direction 1806 may represent the direction in the ambient environment that driver 1804 is looking at while vehicle 1802 is at the intersection and driver 1804 is interacting with traffic control objects 1808 A and 1808B. Gaze direction 1806 may be represented as a 6 degrees-of-freedom pose that has a position (x, y, z) and orientation (pitch, yaw, roll) within the three-dimensional environment. In some embodiments, gaze direction 1806 may represent the gaze direction 1806 captured by inward-facing camera(s) 112 of DMS 110 when vehicle 1802 is stopped at the intersection.

[0287] For example, vehicle 1802 may be in lane 1810B and stopped at the intersection. Driver 1804 may look at traffic control object 1808B for instruction on how to proceedthrough the intersection. While stopped, inward-facing camera(s) 112 may capture an image of the driver 1804. Gaze detector 114 may determine gaze direction 1806. Gaze direction 1806 can be correlated with an image captured by forward-facing camera(s) 122 to determine that the driver is looking at traffic control object 1808B for instruction.

[0288] FIG. 18C illustrates scene 1800C as captured by a forward-facing camera such as forward-facing camera(s) 122 of external sensing system 120. Scene 1800C may include frame 1812 and bounding box 1814. Frame 1812 may be a single frame of video as captured by DMS 110. For example, the frame may be chosen from a time when vehicle 1802 is stopped at the intersection and driver 1804 is determining the instruction indicated by traffic control object 1808B. Though displayed in scene 1800C as a limited field of view for illustration purposes, in an embodiment, scene 1800C may be a 360- degree view of the ambient environment as captured by forward-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129. Frame 1812 may represent a single image of the environment that is captured by external sensing system 120 in accordance with a particular frame rate. For example, front-facing camera(s) 122, sideview camera(s) 127, and rear-facing camera(s) 129 may operate at a particular frame rate that captures a particular number of images per second. In one embodiment, the frame rate may be 60 Hz, though other frame rates are conceivable within the context of this disclosure such that a three-dimensional model of the environment may be rebuilt with regular periodicity.

[0289] Bounding box 1814 may be a two- or three-dimensional rectangle or other polygon surrounding an object detected by object detection system 130. Bounding box 1814 may be represented using four coordinates (two-dimensional bounding boxes) or eight coordinates (three-dimensional bounding boxes) representing points in a particular image frame.

[0290] In some embodiments, scene 1800C may be depicted as a point cloud. Data for the point cloud may be captured using DMS 110, external sensing system 120, and / or object detection system 130. External sensing system 120 and object detection system 130 may use RADAR system 124 and / or LIDAR system 125 to obtain images and corresponding data for the point cloud. The point cloud may be a visual representation of the data captured by these object detecting systems for illustrative purpose. The point cloud may correspond to the captured image represented by 1800C. Object detection system 130 may use this data to build the contemporaneous three-dimensional model of thesurrounding environment. The model may include the traffic control objects identified by classification manager 138. In some embodiments, the object map manager 134 may also indicate which traffic control object(s) manage traffic in which lane(s).

[0291] FIG. 19 depicts a flowchart illustrating method 1900 for classifying traffic control objects based on object detection and gaze tracking. Method 1900 shall be described with reference to FIG. 1 and FIGS. 18A-18C; however, method 1900 is not limited to that example embodiment.

[0292] In an embodiment, a driver 1804 operating vehicle 1802 may interact with a traffic control object. For example, the vehicle 1802 may approach and / or stop at an intersection in the road. Driver 1804 may look at a traffic control object for a specified amount of time which may indicate that the traffic control object is managing traffic in the lane driver 1804 and vehicle 1802 are occupying. Additionally, the intersection may be a 4-way, 3-way, or 2-way intersection. In some embodiments, vehicle 1802 may be at a round-a-bout or circular intersection. For ease of explanation, driver 1804 and vehicle 1802 are described with reference to an intersection. However, the below described method, with reference to FIGS. 1 and 18A-18C may be applicable to any situation on the road where a driver and / or vehicle is interacting with a traffic control object. For example, vehicles may merge lanes and there may be several traffic control objects 1808. ADAS 100 may have to distinguish between traffic control objects 1808 to determine which is managing traffic in the lane vehicle 1802 is occupying.

[0293] At any given time, driver 1804 may have different responsibilities in driving vehicle 1802 ranging from responsibility over all of the vehicle’s mechanical systems (Level 0) to no responsibilities at all (Level 5). In some embodiments, instead of ADAS the vehicle may be an autonomous vehicle utilizing an autonomous driving system, e.g., a system that requires no driver responsibilities (Level 5) and therefore there may not be a “driver.”

[0294] At 1910, DMS 110 of ADAS 100 may capture a first image data of an eye of driver 1804 and external sensing system 120 of ADAS 100 may capture a second image data. For example, inward-facing camera(s) 112 may be used to capture an image of the eye of driver 1804. In some embodiments, the image data may be from a frame in a video sequence captured by inward-facing camera(s) 112.

[0295] Similarly, forward-facing camera(s) 122 may be used to capture an image of traffic control object(s) 1808, e.g., traffic control objects 1808 A and 1808B. The imagedata may be from a frame in a video sequence captured by forward-facing camera(s) 122. In some embodiments, there may be a plurality of traffic control objects 1808 and external sensing system 120 may capture all of them in the image. Additionally, traffic control objects 1808 may be of the same type, e.g., traffic lights, or a combination of several types, e.g., traffic lights, traffic signs, road arrows, lane marking, or another type of object meant to provide instruction and / or traffic management for a lane(s).

[0296] The image / frame capture by either DMS 110 and / or external sensing system 120 may be selected based on several factors, including the gaze direction 1806, a time stamp associated with the frame, quality of the frame / image, whether vehicle 1802 was stopped or moving at the time, and / or other factors that may contribute to the reliability of the frame in determining driver 1804 gaze direction 1806 and / or location of traffic control objects 1808.

[0297] At 1920, DMS 110 of ADAS 100 may determine gaze direction 1806 of the eye of the driver based on the first image data. DMS 110 may use gaze detector 114 and / or face tracker 116 to determine gaze direction 1806 of driver 1804. In some embodiments, driver monitoring system may use the methods described in FIG. 6 to determine gaze direction 206.

[0298] At 1930, object detection system 130 of ADAS 100 may detect a traffic control object in the second image data. Object detection system 130 may identify objects in the ambient environment. In some embodiments, classification manager 138 may allow object detection system 130 to apply a filter to the detected objects and only consider traffic control objects including traffic lights, traffic signs, arrow and landmarks, and similar object intended to provide instruction and / or lane management. Additionally, object map manager may compare the identified traffic control objects, e.g., 1808 A and 1808B, to determine if they exist in the object map or may need to be added. In some embodiments, localization manager 136 may determine a bounding box, e.g., bounding box 1814 in FIG. 18C, for one or all of the identified traffic control objects 1808.

[0299] At 1940, based on the gaze direction 1806 determined at 1920 and / or traffic control objects 1808 detected at 1930, determining whether driver 1804 is looking at the traffic control object. When gaze direction 1806 intersects a bounding box this may indicate that driver 1804 is looking at the object. For example, gaze direction 1806 of driver 1804 may intersect bounding box 1814 associated with traffic control object1808B. The system may then determine that driver 1804 is looking at traffic control object 1808B for instruction on how to proceed through the intersection.

[0300] At 1950, object detection system 130 may determine a lane where the vehicle is located contemporaneous with the capture of the first and second image data. For example, object detection system 130 via detector 132, object map manager 134, and / or classification manager 138 may determine that vehicle 1802 occupies lane 1810B. Specifically, vehicle 1802 occupied lane 1810B at the time of capturing the first and second image data. ADAS 100 may compare time-stamps or other metadata corresponding to the first and second image data to confirm they were taken contemporaneously .

[0301] At 1960, computer system 2300 may generate data indicating that the traffic control object manages traffic on the lane of the vehicle. For example, computer system 2300 may generate data that indicates traffic control object 1808B manages traffic for lane 1810B. This data may be stored in main memory 2308 and / or secondary memory 2310. The data may be collected as method 1900 repeated for a plurality of vehicles in lane 1810B. Once the data has reached a threshold, confirming that traffic control object 1808B manages traffic for vehicles occupying lane 1810B, then computer system 2300 may update the object map manager 134 to indicate the relationship between lane 1810B and traffic control object 1808B.

[0302] This allows the relationship to be based on data that was crowdsourced from a plurality of participating vehicles 1802. The system is able to reliably determine that traffic control object 1808B manages traffic for lane 1810B. The threshold may change depending on safety and reliability concerns. In some embodiments, method 1900 may be repeated 10 times to satisfy the threshold. In others, the threshold may be set to 100, 1,000, or more depending on the requirements of system, manufacture^ s), and / or regulatory body.

[0303] Additionally, method 1900 may be repeated for a plurality of lanes 1810 and plurality of traffic control objects 1808. For example, method 1900 could be repeated for vehicle 1802 occupying lane 1810A to determine that traffic control object 1808B manages traffic for vehicles occupying lane 1810A.

[0304] In some embodiments, once the relationship has been established based on a threshold of crowdsourced data being reached, vehicle 1802 may perform an action in accordance with an instruction from traffic control object 1808B. For example, if vehicle1802 initially stops at the intersection because traffic control object 1808B is red, or another stop instruction is indicated, when traffic control object 1808B instructs vehicle 1802 to move forward based on a corresponding indication, then vehicle 1802 may perform this action. Depending on the level of vehicle autonomy and driver responsibility, ADAS 100 may provide driver 1804 a message or alert through alert generator 150 and / or on a display located in vehicle 1802 to perform the action. In some embodiments, ADAS 100 may provide an alert or message via a display for permission to perform the action. Further, if driver 1804 has no responsibility, ADAS 100 via autonomous driving system 155 and / or policy manager 160 may perform the action autonomously.

[0305] FIGS. 20A-20C illustrates environments with various objects used for calibration of gaze detection for a driver while operating a vehicle. As illustrated in FIGS. 20A-20C, environments 2000A, 2000B, and 2000C may include vehicle 2002 and driver 2004. Additionally, FIGS. 20A-20C include first gaze direction 2006 A, second gaze direction 2006B, and third gaze direction 2006C as well as first object 2008 A, second object 2008B, and third object 2008C, respectively. As examples, first through third objects 2008 A-C are described as three different types of objects, however in some embodiments, they all may be the same type of object or a combination of same and different objects. The objects selected by ADAS 100 rolling calibration may be dependent on the route taken by driver 2004 and the objects in the FOV of driver 2004. Additionally, three different types of objects are described for rolling calibration, however depending on driver 2004, rules and regulations governing ADAS and AV systems, and / or the route being taken, there may be additional or less objects used for rolling calibration. Environments 2000A, 2000B, and 2000C displayed in FIGS. 20A-20C are views from outside of the vehicle. However, the situations depicted in FIGS. 20A-20C are merely exemplary and limitless possibilities that exist regarding the nature of environments that ADAS 100 may encounter in the real world.

[0306] Vehicle 2002 is illustrated in environments 2000A-2000C. In some embodiments, vehicle 2002 may be a car; however, any other suitable types of vehicles are relevant, including a truck, van, motorcycle, all-terrain vehicle, military vehicle, boat, etc. In addition to the roadway example depicted, any suitable transportation infrastructure where vehicle 2002 may be operated with ADAS 100 according to application traffic and / or transportation laws and regulations.

[0307] Driver 2004 may be a human being operating vehicle 2002. At any given time, driver 2004 may have different responsibilities in driving vehicle 2002 ranging from responsibility over all of the vehicle’s mechanical systems (Level 0) to no responsibilities at all (Level 5). In some embodiments, instead of ADAS, the vehicle may be an autonomous vehicle utilizing an autonomous driving system, e.g., a system that requires no driver responsibilities (Level 5) and therefore there may not be a “driver.” The description of FIGS. 20A-20C and throughout the detailed description apply to ADAS and AV scenarios alike.

[0308] In environments 2000A-2000C, driver 2004 and vehicle 2002 are on a route routinely driven, such that driver 2004 may have repeated the route and have the opportunity to view objects 2008A-C repeatedly. For example, driver 2004 and vehicle 2002 may be on a route to work, school, home, a grocery store, or another suitable route with fixed objects that driver 2004 may routinely observe. In some embodiments fixed objects, e.g., semi-permanent or permanently placed objects, may be detected and used for rolling calibration. Fixed objects, unlike unfixed objects or moving objects, e.g., pedestrians, other vehicles, or similar types of objects, may not consistently appear in the same spot along the route routinely taken by driver 2004. For this reason, object detection system 130 may select fixed objects for rolling calibration.

[0309] In the example of environment 2000 A, first object 2008 A may be a traffic control object, such as a traffic sign. Traffic control object may be useful as a fixed object for rolling calibration because driver 2004 may typically interact with traffic control objects and they are semi-permanent, fixed objects. Traffic control objects including traffic signs, traffic lights, markings, barriers, and others devices and / or signage to manage traffic, may be installed to provide consistent instruction to both drivers and pedestrians, therefore they may be objects driver 2004 interacts with (e.g., looks at) frequently. Additionally traffic rules and laws may not change frequently, which makes traffic control objects helpful for rolling calibration of drivers’ gaze directions. In some embodiments, other types of traffic signs or roadway signage may be used as first object 2008 A. For example, first object 2008 A may be an informational sign or billboard. The informational content and / or instruction displayed on first object 2008 A may be changed while first object 2008A is in the same, fixed location.

[0310] Object detection system 130 may be used to determine when vehicle 2002 is near first object 2008 A (and subsequently second object 2008B and / or third object 2008C)such that driver 2004 may be looking at first object 2008 A. Classification manager 138 may have a list of objects that are used for rolling calibration. ADAS 100 may use a GPS Unit 126 and / or external sensing system 120 to determine when vehicle 2002 is near first object 2008 A. When vehicle 2002 is near first object 2008 A, ADAS 100 may capture images of the driver.

[0311] In some embodiments, ADAS 100 may determine first gaze direction 2006 A of driver 2004 looking at first object 2008 A. For example, when vehicle 2002 approaches the location of first object 2008 A, ADAS 100 may capture images of driver 2004 and the external FOV, including first object 2008 A. The images may allow ADAS 100 to determine first gaze direction 2006 A. ADAS 100 may capture image(s) of driver 2004 via inward-facing camera(s) 112 of driving monitoring system 110 and external image(s) including first object 2008 A via forward-facing camera(s) 122 of external sensing system 120. In some embodiments, external sensing system 120 may use forward-facing camera(s) 122, side view camera(s) 127, rear-facing camera(s) 129, and / or a combination of the cameras to capture external images of the FOV of driver 2004.

[0312] In some embodiments, the system may capture several images of driver 2004 while looking at first object 2008 A. While looking at first object 2008 A, driver 2004 may not always be looking at the same exact spot on first object 2008 A. Therefore, first gaze direction 2006 A may intersect first object 2008 A at several different points. The collection of this image data may be used to calibrate gaze detector 114 for driver 2004 while operating vehicle 2002.

[0313] Environments 2000B and 2000C are similar to environment 2000A, but depict a second object 2008B and second gaze direction 2006B in environment 2000B and a third object 2008C and third gaze direction 2006C in environment 2000C. Second object 2008B and third object 2008C provide additional examples of objects that may be used for rolling calibration of the gaze direction of driver 2004. In some embodiments, second object 2008B may be a billboard, or other type of advertisement, and third object 2008C may be a mailbox. Both of these objects are representative of permanent or semipermanent fixed objects. The objects may be used continuously for rolling calibration of the gaze direction for driver 2004. Similar to first object 2008A, second and third objects 2008B-C may also be classified by classification manager 138 as objects that may be used for rolling calibration. In some embodiments, first, second, and third objects 2008 A-C may be located near each other. For example, the objects may be on the same street andcapable of being captured in a single FOV image. In some embodiments, first, second, and third objects 2008 A-C may not be located near each other and may be on different routes that are all routinely taken by driver 2004.

[0314] As shown in FIG. 20B, the objects used for rolling calibration may be located on different sides of vehicle 2002. Additionally, objects 2008 may be located at different distances from the road and / or vehicle 2002. This may allow ADAS 100 to perform calibration for driver 2004 with a variety of gaze directions. In some embodiments, calibration may be performed at the beginning of a route for objects 2008 A-C and / or at the end of a route. ADAS 100 may be able to perform calibration for the same objects while driver 2004 and vehicle 2002 are in different positions.

[0315] FIG. 21 A illustrates environment 2100 A in which ADAS 100 may operate, according to some embodiments. In some embodiments, environment 2100A corresponds to environments 2000A-2000C, including first, second, and third objects 2008A-2008C used for rolling calibration, only displayed from the perspective inside of the vehicle. However, in other embodiments, environment 2100 A may include a single object 2008 used for rolling calibration or another combination of objects used for rolling calibration.

[0316] Gaze direction 2006 may represent the direction in the ambient environment that driver 2004 is looking at while operating vehicle 2002. With reference to FIGS. 20A-20C, in some embodiments gaze direction 2006 may be first gaze direction 2006A while driver 2004 is looking at first object 2008 A, second gaze direction 2006B while driver 2004 is looking at second object 2008B, and / or third gaze direction 2006C while driver is looking at third object 2008C. In some embodiments, rays may be cast from the eye of driver 2004 to objects 2008, based on the image(s) captured. The ADAS 100 may cast rays and determine that the ray intersect an object 2008 detected by object detection system 130. Additionally, object detection system 130 stores the location of objects 2008, so ADAS 100 may determine the depth of gaze direction 2006 when looking at objects 2008 A, 2008B, and / or 2008C. In some embodiments, ADAS 100 may work in the reverse to determine gaze direction. Based on the known location of objects 2008, ADAS 100 may cast rays from objects 2008 to the eye of driver 2004 to determine gaze direction 2006. For example, using the known location of first object 2008 A determined by object detection system 130 and images captured by DMS 110 and external sensing system 120, gaze detector 114 may cast a ray from first object 2008A to the eye of driver 2004. In some embodiments, ADAS 100 may perform ray casting for each of the image(s) and / orframes in the video sequence captured by DMS 110. The rays from each of these images may be used to determine fixations and gaze points, heatmaps, areas of interest (AOI), dwell time, and similar factors that are used to correlate gaze direction 2006 and objects 2008 with the determined ray castings.

[0317] As previously described, rolling calibration may be performed with more or less than three objects. Each object used for rolling calibration may have at least one corresponding gaze direction. Gaze direction 2006 may be represented as a 6 degrees-of- freedom pose that has a position (x, y, z) and orientation (pitch, yaw, roll) within the three-dimensional environment. In some embodiments, gaze direction 2006 may represent the gaze direction 2006 captured by inward-facing camera(s) 112 of DMS 110. In some embodiments, gaze direction 2006 may be captured by inward-facing camera(s) 112 while driver is operating vehicle 2002. For example, this may include while vehicle 2002 is in motion at a high rate of speed, low rate of speed, turning, stopped, and / or a combination of these states so that the calibration can be performed under normal operating procedures. This may allow reliable calibration without disrupting driver 2004 while operating vehicle 2002. In some embodiments, gaze detector 114 may determine gaze direction 2006 from object 2008 to the eye of driver 2004. For example, gaze detector 114 may use the location of second object 2008B and cast rays from second object 2008B to the eye of driver 2004. ADAS 100 may use GPS unit 126 to determine when driver 2004 and vehicle 2002 are near second object 2008B. External sensing system 120 may use a combination of forward facing camera(s) 122, side view camera(s) 127, and / or rear-facing camera(s) 129 to determine when second object 2008B is in the FOV of driver 2004. In some embodiments, ADAS 100 may determine gaze direction 2006 using both methods and compare the positions representing gaze direction 2006 to perform rolling calibration of gaze direction 2006 for driver 2004.

[0318] In some embodiments, calibration metrics corresponding to the rolling calibration of the gaze direction of driver 2004 may be saved to eye calibration model 119. Measurements corresponding to the calibration metrics may include minimum diameters for circular fixations, degrees of a visual angle when looking at objects 2008A-2000C, and / or duration of gaze direction 2006 at a particular location. As additional calibration metric values are determined ADAS 100 may update eye calibration model 119. Eye calibration model 119 may be stored in storage 195 and accessible by different vehicles using ADAS 100. Driver 2004 may have login credentials that allow driver 2004 toaccess driver profile module 175 in any vehicle suing ADAS 100. Therefore calibration metrics corresponding to driver 2004 may be utilized when driver 2004 is operating vehicles other than vehicle 2002.

[0319] FIG. 2 IB illustrates scene 2100B as captured by a forward-facing camera such as forward-facing camera(s) 122 of external sensing system 120. Scene 2100B may include frame 2102 and bounding boxes 2110, 2120, and 2130 corresponding to first object 2008 A, second object 2008B, and third object 2008C, respectively. Frame 2102 may be a single frame of video as captured by DMS 110. For example, the frame may be chosen from a time when vehicle 2002 is near one of objects 2008A-2008C. As described above, using external sensing system 120 and object detection system 130, ADAS 100 may determine when vehicle 2002 is near at least one of objects 2008A-2008C. ADAS 100 can determine that driver 2004 may be looking at one of objects 2008A-2008C because they are known objects in a fixed location on a route routinely taken by driver 2004.

[0320] Though displayed in scene 2100B as a limited field of view for illustration purposes, in an embodiment, scene 2100B may be a 360-degree view of the ambient environment as captured by forward-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129. Frame 2102 may represent a single image of the environment that is captured by external sensing system 120 in accordance with a particular frame rate. For example, front-facing camera(s) 122, side-view camera(s) 127, and rear-facing camera(s) 129 may operate at a particular frame rate that captures a particular number of images per second. In one embodiment, the frame rate may be 60 Hz, though other frame rates are conceivable within the context of this disclosure such that a three-dimensional model of the environment may be rebuilt with regular periodicity.

[0321] Bounding boxes 2110, 2120, and 2130 may be a two- or three-dimensional rectangle or other polygon surrounding an object detected by object detection system 130. Bounding boxes 2110, 2120, and 2130 may be represented using four coordinates (two- dimensional bounding boxes) or eight coordinates (three-dimensional bounding boxes) representing points in a particular image frame.

[0322] In some embodiments, scene 2100B may be depicted as a point cloud. Data for the point cloud may be captured using DMS 110, external sensing system 120, and / or object detection system 130. External sensing system 120 and object detection system 130 may use RADAR system 124 and / or LIDAR system 125 to obtain images and corresponding data for the point cloud. The point cloud may be a visual representation of the datacaptured by these object detecting systems for illustrative purpose. The point cloud may correspond to the captured image represented by 2100B. Object detection system 130 may use this data to build the contemporaneous three-dimensional model of the surrounding environment. The model may include the first object 2008 A, second object 2008B, and / or third object 2008C identified by classification manager 138 for rolling calibration of driver gaze direction.

[0323] FIG. 22A depicts a flowchart illustrating method 2200A for performing rolling calibration of a gaze direction of a driver. Method 2200A shall be described with reference to FIGS. 1, 20A-20C, and 21 A-21B; however, method 2200A is not limited to that example embodiment. The foregoing description will describe an embodiment of the execution of method 2200A with respect to ADAS 100. While method 2200A is described with reference to ADAS 100, method 2200A may be executed on any computing device, such as, for example, the computer system described with reference to FIG. 23 and / or processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof.

[0324] It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown in FIG. 22A, as will be understood by a person of ordinary skill in the art.

[0325] In an embodiment, a driver 2004 may be operating vehicle 2002 on a route taken routinely, as tracked by ADAS 100. For example, driver 2004 may be taking a route from home to work which driver 2004 takes several times a week. Driver 2004 may look at the same fixed objects each time driver 2004 takes the route. These objects may be used for rolling calibration of the gaze direction 2006 of driver 2004. These fixed objects may be ideal candidates because driver 2004 consistently interacts with (e.g., looks at) the objects. ADAS 100, via classification manager 138, may determine objects that can be used for rolling calibration, these may be objects 2008. In some embodiments, several objects may be selected for rolling calibration. For example, and with reference to FIGS. 20A-20C, classification manager 138 may identify first object 2008 A, which may be a traffic sign, second object 2008B, which may be a billboard and / or advertisement, and / or third objects 2008C, which may be a mailbox. In some embodiments, classification manager 138 may identify additional or fewer objects for rolling calibration.Additionally, first through third objects 2008A-2008C are pictured in the same vicinity on the same route such that they are in a single FOV of driver 2004. However, the objects 2008 used for rolling calibration may not be located near each other and / or may be located on different routes that driver 2004 may routinely take with vehicle 2002.

[0326] In some embodiments, driver 2004 may have different responsibilities in driving vehicle 2002 ranging from responsibility over all of the vehicle’s mechanical systems (Level 0) to no responsibilities at all (Level 5). In some embodiments, instead of ADAS the vehicle may be an autonomous vehicle utilizing an autonomous driving system, e.g., a system that requires no driver responsibilities (Level 5) and therefore there may not be a “driver.” Regardless of the level of driver responsibilities, it may be useful for gaze detector 114 to be calibrated for driver 2004.

[0327] At 2210, ADAS 100 may capture one or more images of an eye of driver 2004 of vehicle 2002 by inward-facing camera(s) and respectively corresponding images from external -facing camera(s). ADAS 100 may utilize DMS 110 and / or to capture the one or more images of the eye of driver 2004. External sensing system 120 of ADAS 100 may capture a second image. For example, inward-facing camera(s) 112 may be used to capture one or more images of the eye of driver 2004. In some embodiments, the image may be from a frame in a video sequence captured by inward-facing camera(s) 112.

[0328] Similarly, forward-facing camera(s) 122 may be used to capture an image of object(s) 2008, e.g., objects used for rolling calibration such as objects 2008 A-C. The one or more images may be from a frame in a video sequence captured by forward-facing camera(s) 122. In some embodiments, ADAS 100 may use forward-facing camera(s) 122, side-view camera(s) 127, and / or rear-facing camera(s) 129 to capture one or more external images. In some embodiments, there may be a plurality of objects 2008 and external sensing system 120 may capture all of them in the image. However, ADAS 100 may repeat method 2200A at different locations capturing a plurality of objects 2008 for rolling calibration at various locations.

[0329] In some embodiments, 2220 through 2240 may be repeated for each image of the eye of driver 2004 and respectively corresponding external image. Additionally, ADAS 100 may perform a quality analysis on the one or more images and use only a subset of the captured one or more images for rolling calibration. For example, blurred images may not be useful in determining gaze direction, described with reference to FIG. 6, and may be discarded. There may be other quality factors such as sharpness, exposure, contrast,vignetting, noise, distortion, and similar factors that reduce the quality of the image and / or image frame.

[0330] At 2220, ADAS 100 may determine gaze direction 2006 of the eye of the driver based on the first image and calibration parameters. For example, DMS 110 of ADAS 100 may use gaze detector 114 and / or face tracker 116 to determine gaze direction 2006 of driver 2004. In some embodiments, driver monitoring system may use the methods described in FIGS. 21 A-21B to determine gaze direction 2006. DMS 110 may use calibration parameters to assist in determining gaze direction 2006 of driver 2004. This may include fixations and gaze points, heatmaps, areas of interest (AOI), dwell time, and similar factors that help identify where, with reference to the external FOV of driver 2004, gaze direction 2006 is interaction. With reference to FIG. 21 A and 21B, gaze direction 2006 may be first gaze direction 2006A for the first set of images and move to second gaze direction 2006B and third gaze direction 2006C as diver monitoring system 110 continues to evaluate the gaze direction 2006 of driver 2004.

[0331] At 2230, ADAS 100 may determine one or more objects 2008 in gaze direction 2006 of driver 2004. Object detection system 130 of ADAS 100 may use the external images captured by external sensing system 120 to determine that objects 2008 are in the FOV of view of driver 2004. Particularly, detector 132 may detect objects based on the external image. Object detection system 130 may then use a combination of object map manager 134 and / or classification manager 138 to first determine that the detected objects are objects 2008 used for rolling calibration and the location of the objects. Based on these factors and gaze direction 2006, ADAS 100 may determine that gaze direction 2006 may intersect with objects 2008. For example, for second object 2008B, object detection system 130 has identified the location of second object 2008B. Additionally, ADAS 100 via DMS 110 may know the driver 2004 should be looking at second object 2008B and therefore second gaze direction 2006B is intersecting second object 2008B. In some embodiments, there may be a plurality of objects 2008 in the external image. The gaze direction and known location of the object may allow ADAS 100 to determine that gaze direction 2006 is intersecting object 2008, such as one of objects 2008 A-C.

[0332] At 2240, based on the location of object(s) 2008, ADAS 100 may adjust the set of calibration parameters. As described at 2230, object detection system 130 may determine the location of the object and that gaze direction 2006 of driver 2004 is intersecting object 2008.

[0333] With reference to FIG. 2 IB, bounding boxes may be placed around the objects 2008 for rolling calibration. The calibration parameters may need to be adjusted specifically for driver 2004 to accurately ensure that gaze direction 2006 is intersecting the bounding box. For example, based on the determining at 2220 and / or 2230, ADAS 100 may determine that gaze direction 2006 is intersecting object 2008. However, the calibration parameters may need to be adjusted to indicate this. For example, for first object 2008 A, one calibration parameter may be a heatmap, which may indicate gaze direction 2006 is near object 2008 but a portion of the heatmap may be intersecting outside of bounding box 2130. The calibration parameter may be shifted so that gaze direction 2006A is clearly intersecting bounding box 2130 and therefore first object 2008A.

[0334] Additionally, object detection system 130 has identified the location of object(s) 2008 at the time internal and external images were captured. Therefore, using the known location of object(s) 2008 gaze detector 114 can accurately determine the depth of gaze direction 2006.

[0335] As discussed above, these steps may be repeated for every eye image captured by DMS 110 and respectively corresponding external image. This may allow the system to accurately calibrate the gaze detector 114. Additionally, this method may be performed while driver 2004 is operating vehicle 2002. This may allow the system to perform rolling calibration with little or no interruption to driver 2004 operating vehicle 2002.

[0336] FIG. 22B depicts a flowchart illustrating method 2200B for adjusting calibration when driver head movement is detected. Method 2200B shall be described with reference to FIG. 22A; however, method 2200B is not limited to that example embodiment. The foregoing description will describe an embodiment of the execution of method 2200B with respect to ADAS 100. While method 2200A is described with reference to ADAS 100, method 2200B may be executed on any computing device, such as, for example, the computer system described with reference to FIG. 23 and / or processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof.

[0337] It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown in FIG. 22B, as will be understood by a person of ordinary skill in the art.

[0338] At 2222, ADAS 100 may determine a head movement of driver 2004 during capture of eye images. ADAS 100 may use DMS 110 to capture eye images of driver 2004. Inward-facing camera(s) 112 may capture images of the eye of driver 2004, which may include areas of the head or face of driver 2004 outside of the eye. Face tracker 116 may determine that during capture, driver 2004 was moving their head. This may affect the quality of the eye images and / or gaze direction 2006. Based on the head movement of driver 2004, face tracker 116 may determine that gaze direction 2006 for the corresponding images should be adjusted. For example, the position (x, y, z) and / or the orientation (pitch, yaw, roll) may need to be adjusted.

[0339] At 2224, ADAS 100 may update gaze direction 2006 of driver 2004 to include the adjustment based on the determined head movement at 2224. This may be used at 2230 and / or 2240 in method 2200A to accurately calibrate gaze direction 2006 of driver 2004.

[0340] Various embodiments may be implemented, for example, using one or more well- known computer systems, such as computer system 2300 shown in FIG. 23. One or more computer systems 2300 may be used, for example, to implement any of the embodiments discussed herein, as well as combinations and sub-combinations thereof.

[0341] Computer system 2300 may include one or more processors (also called central processing units, or CPUs), such as a processor 2304. Processor 2304 may be connected to a communication infrastructure or bus 2306.

[0342] Computer system 2300 may also include user input / output device(s) 2303, such as monitors, keyboards, pointing devices, etc., which may communicate with communication infrastructure or bus 2306 through user input / output interface(s) 2302.

[0343] One or more of processors 2304 may be a graphics processing unit (GPU). In an embodiment, a GPU may be a processor that is a specialized electronic circuit designed to process mathematically intensive applications. The GPU may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc.

[0344] Computer system 2300 may also include a main or primary memory 2308, such as random access memory (RAM). Main memory 2308 may include one or more levels of cache. Main memory 2308 may have stored therein control logic (i.e., computer software) and / or data.

[0345] Computer system 2300 may also include one or more secondary storage devices or memory 2310. Secondary memory 2310 may include, for example, a hard disk drive2312 and / or a removable storage device or drive 2314. Removable storage drive 2314 may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, tape backup device, and / or any other storage device / drive.

[0346] Removable storage drive 2314 may interact with a removable storage unit 2318. Removable storage unit 2318 may include a computer usable or readable storage device having stored thereon computer software (control logic) and / or data. Removable storage unit 2318 may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and / any other computer data storage device. Removable storage drive 2314 may read from and / or write to removable storage unit 2318.

[0347] Secondary memory 2310 may include other means, devices, components, instrumentalities or other approaches for allowing computer programs and / or other instructions and / or data to be accessed by computer system 2300. Such means, devices, components, instrumentalities or other approaches may include, for example, a removable storage unit 2322 and an interface 2320. Examples of the removable storage unit 2322 and the interface 2320 may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and / or any other removable storage unit and associated interface.

[0348] Computer system 2300 may further include a communication or network interface 2324. Communication interface 2324 may enable computer system 2300 to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively referenced by reference number 2328). For example, communication interface 2324 may allow computer system 2300 to communicate with external or remote devices 2328 over communications path 2326, which may be wired and / or wireless (or a combination thereof), and which may include any combination of LANs, WANs, the Internet, etc. Control logic and / or data may be transmitted to and from computer system 2300 via communication path 2326.

[0349] Computer system 2300 may also be any of a personal digital assistant (PDA), desktop workstation, laptop or notebook computer, netbook, tablet, smart phone, smart watch or other wearable, appliance, part of the Internet-of-Things, and / or embedded system, to name a few non-limiting examples, or any combination thereof.

[0350] Computer system 2300 may be a client or server, accessing or hosting any applications and / or data through any delivery paradigm, including but not limited to remote or distributed cloud computing solutions; local or on-premises software (“onpremise” cloud-based solutions); “as a service” models (e.g., content as a service (CaaS), digital content as a service (DCaaS), software as a service (SaaS), managed software as a service (MSaaS), platform as a service (PaaS), desktop as a service (DaaS), framework as a service (FaaS), backend as a service (BaaS), mobile backend as a service (MBaaS), infrastructure as a service (laaS), etc.); and / or a hybrid model including any combination of the foregoing examples or other services or delivery paradigms.

[0351] Any applicable data structures, file formats, and schemas in computer system 2300 may be derived from standards including but not limited to JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representations alone or in combination. Alternatively, proprietary data structures, formats or schemas may be used, either exclusively or in combination with known or open standards.

[0352] In some embodiments, a tangible, non-transitory apparatus or article of manufacture comprising a tangible, non-transitory computer useable or readable medium having control logic (software) stored thereon may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 2300, main memory 2308, secondary memory 2310, and removable storage units 2318 and 2322, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system 2300), may cause such data processing devices to operate as described herein.

[0353] Based on the teachings contained in this disclosure, it will be apparent to persons skilled in the relevant art(s) how to make and use embodiments of this disclosure using data processing devices, computer systems and / or computer architectures other than that shown in FIG. 23. In particular, embodiments can operate with software, hardware, and / or operating system implementations other than those described herein.

[0354] It is to be appreciated that the Detailed Description section, and not any other section, is intended to be used to interpret the claims. Other sections can set forth one ormore but not all exemplary embodiments as contemplated by the inventor(s), and thus, are not intended to limit this disclosure or the appended claims in any way.

[0355] While this disclosure describes exemplary embodiments for exemplary fields and applications, it should be understood that the disclosure is not limited thereto. Other embodiments and modifications thereto are possible, and are within the scope and spirit of this disclosure. For example, and without limiting the generality of this paragraph, embodiments are not limited to the software, hardware, firmware, and / or entities illustrated in the figures and / or described herein. Further, embodiments (whether or not explicitly described herein) have significant utility to fields and applications beyond the examples described herein.

[0356] Embodiments have been described herein with the aid of functional building blocks illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined as long as the specified functions and relationships (or equivalents thereof) are appropriately performed. Also, alternative embodiments can perform functional blocks, steps, operations, methods, etc. using orderings different than those described herein.

[0357] References herein to “one embodiment,” “an embodiment,” “an example embodiment,” or similar phrases, indicate that the embodiment described can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it would be within the knowledge of persons skilled in the relevant art(s) to incorporate such feature, structure, or characteristic into other embodiments whether or not explicitly mentioned or described herein. Additionally, some embodiments can be described using the expression “coupled” and “connected” along with their derivatives. These terms are not necessarily intended as synonyms for each other. For example, some embodiments can be described using the terms “connected” and / or “coupled” to indicate that two or more elements are in direct physical or electrical contact with each other. The term “coupled,” however, can also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other.

[0358] The breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.Determining Driver Mental State Using Gaze Tracking and Object Detection

[0359] The techniques disclosed herein may be implemented or realized with the following illustrative examples.

[0360] A computer-implemented method, comprising: receiving sensor data from one or more exterior-facing sensors monitoring an environment outside of a vehicle and image data from one or more interior-facing cameras monitoring a driver of the vehicle; identifying an object in the sensor data; determining a gaze direction of the driver using the image data; determining whether the gaze direction is associated with a spatial location of the object; determining a mental state of the driver based on whether the gaze direction is determined to be associated with the spatial location of the object; and performing an action in an advanced driver assistance system based on the mental state.

[0361] The above method, the determining the gaze direction of the driver further comprising: building a heatmap representing the gaze direction of the driver using the image data, wherein the image data comprises a plurality of fixated movements and a plurality of saccadic eye movements of the driver.

[0362] The above method, further comprising: updating an alertness value that changes over a driving session based on the determined mental state; and displaying the alertness value on a feedback display during the driving session.

[0363] The above method, further comprising: establishing a danger priority list of objects; identifying the object as a high danger object in the danger priority list; receiving a rational response from the driver within a threshold time after identifying the high danger object, wherein the rational response is a braking action, an acceleration action, or a steering action; and increasing the alertness value in response to receiving the rational response.

[0364] The above method, further comprising: establishing a danger priority list of objects; identifying the object as a high danger object in the danger priority list; and decrementing the alertness value in response to determining that the gaze direction does not intersect with the object.

[0365] The above method, further comprising: recording a number of missed objects and a number of seen objects for the driver over a driving session by incrementing the number of missed objects when the gaze direction does not intersect with the object and incrementing the number of seen objects when the gaze direction intersects with the object.

[0366] The above method, further comprising: generating an alert when the number of missed objects over a time frame exceeds a threshold value for the time frame.

[0367] The above method, further comprising: determining a percentage of missed objects between the number of seen objects and the number of missed objects; and when the percentage of missed objects over the driving session differs by a threshold value from a normal percentage of missed objects retrieved from a driver profile, increasing a likelihood of impairment value.

[0368] The above method, further comprising: determining an origin of a viewpoint of the driver; generating a fundamental correspondence matrix between a first virtual frame corresponding to the one or more exterior-facing cameras and a second virtual frame corresponding to the one or more interior-facing cameras based on the origin, the sensor data, and the image data.

[0369] The above method, further comprising: constructing a two-dimensional image of the environment using the sensor data; determining a bounding box associated with the object in the two-dimensional image; casting a ray from the origin into the two- dimensional image using the fundamental correspondence matrix; determining that the driver saw the object by determining that the ray intersects with the bounding box.

[0370] The above method, the identifying the object further comprising: employing an object detector to create a bounding box and a classification of the object, wherein the bounding box is a two-dimensional bounding box or a three-dimensional bounding box.

[0371] The above method, wherein determining the gaze direction further comprises: projecting onto an eye of the driver with a light to generate a Purkinje image; and estimating the gaze direction based on the Purkinje image.

[0372] The above method, wherein the mental state indicates attentiveness, fatigue, or impairment.

[0373] The above method, wherein the sensor data and the image data are captured substantially simultaneously.

[0374] The above method, wherein the object is a traffic light, a traffic sign, a pedestrian, a second vehicle, an animal, or a traffic barrier.

[0375] The above method, wherein the spatial location of the object comprises a position, a rate of movement, and a direction of movement.

[0376] A system, comprising: one or more exterior-facing cameras monitoring an outside environment of a vehicle; one or more interior-facing cameras monitoring a driver of the vehicle; a memory; and at least one processor coupled to the memory and configured to: receive sensor data from the one or more exterior-facing sensors and image data from the one or more interior-facing cameras; identify an object in the sensor data; determine a gaze direction of the driver using the image data; determine whether the gaze direction is associated with a spatial location of the object; determine a mental state of the driver based on whether the gaze direction is determined to be associated with the spatial location of the object; and perform an action in an advanced driver assistance system based on the mental state.

[0377] The above system, wherein to determine the gaze direction of the driver the at least one processor is further configured to: build a heatmap representing the gaze direction of the driver using the image data, wherein the image data comprises a plurality of fixated movements and a plurality of saccadic eye movements of the driver.

[0378] A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising: receiving sensor data from one or more exterior-facing sensors monitoring an environment outside of a vehicle and image data from one or more interior-facing cameras monitoring a driver of the vehicle; identifying an object in the sensor data; determining a gaze direction of the driver using the image data; determining whether the gaze direction is associated with a spatial location of the object; determining a mental state of the driver based on whether the gaze direction is determined to be associated with the spatial location of the object; and performing an action in an advanced driver assistance system based on the mental state.

[0379] The above non-transitory computer-readable device the determining the gaze direction further comprising: building a heatmap representing the gaze direction of the driver using the image data, wherein the image data comprises a plurality of fixated movements and a plurality of saccadic eye movements of the driver.Policy Adjustment Based on Driver Attentiveness

[0380] The techniques disclosed herein may be implemented or realized with the following illustrative examples.

[0381] A computer-implemented method for adjusting a policy that controls operation of a vehicle, comprising: receiving sensor data from one or more exterior-facing sensors monitoring an environment outside of the vehicle and image data from one or more interior-facing cameras monitoring a driver of the vehicle; determining an awareness state of the driver based on the sensor data and the image data; and adjusting a parameter used by the policy based on the awareness state.

[0382] The above method, the determining the awareness state of the driver further comprising: identifying an object in the sensor data; determining a gaze direction of the driver using the image data; and determining whether the gaze direction is associated with a spatial location of the object.

[0383] The above method, wherein the object is a traffic light, a traffic sign, a pedestrian, a second vehicle, an animal, or a traffic barrier.

[0384] The above method, wherein the policy controls at least one of: (1) a velocity of the vehicle, (2) a forward distance, (3) a lateral distance, (4) a lane change decision, and / or (5) a pace of acceleration changes.

[0385] The above method, wherein the policy controls at least one of: (1) an autonomous driving system, (2) an adaptive cruise control system, (3) a collision avoidance system, (4) a blind-spot monitoring system, (5) a lane-change warning system, (6) a gaze-enabled lane change system, (7) a steering-assist system, (8) a lane-keep assist system, and / or (9) a lane departure warning system.

[0386] The above method, wherein the policy specifies an autonomous driving level that is one of a plurality of autonomous driving levels defined by a Society of Automotive Engineers standard.

[0387] The above method, wherein the policy specifies an autonomous driving level that is one of: (1) a level 0 policy of no driving automation, (2) a level 1 policy of driver assistance, (3) a level 2 policy of partial driving automation, (4) a level 3 policy of conditional driving automation, (5) a level 4 policy of high driving automation, or (6) a level 5 policy of full driving automation.

[0388] The above method, the adjusting further comprising: modifying a takeover notice period used by an autonomous driving system based on the awareness state.

[0389] The above method, wherein the policy controls an adaptive cruise control system, and wherein the parameter controls: (1) a maximum permissible lateral acceleration, (2) a maximum stopping acceleration, (3) a minimum time gap, (4) a maximum time gap, (5) a minimum clearance, (6) a minimum speed, (7) a maximum speed, or (8) a distance to a second vehicle.

[0390] The above method, wherein the policy controls a blind-spot monitoring system, a lane-change warning system, or a gaze-enabled lane change system, and wherein the parameter controls: (1) a speed difference tolerance between the vehicle and a target vehicle, (2) a time to collision tolerance, or (3) a warning level.

[0391] The above method, wherein the policy specifies an alarm tolerance level that controls when a collision avoidance system generates a driver alert, the adjusting further comprising: when the gaze direction is associated with the spatial location of the object, increasing the alarm tolerance level; and when the gaze direction is not associated with a spatial location of the object, decreasing the alarm tolerance level.

[0392] The above method, wherein the policy controls a steering-assist system, a lanekeep assist system, or a lane departure warning system, and wherein the parameter controls: (1) a threshold distance to a lane boundary, (2) a warning level, or (3) a lane change tolerance.

[0393] The above method, further comprising: updating a current attention level value that changes over a driving session based on the awareness state; and displaying an indication related to the current attention level value on a feedback display during the driving session.

[0394] The above method, wherein the parameter reflects the driver’s level of anxiety, level of stress, or level of nausea.

[0395] A system for adjusting a policy that controls operation of a vehicle, comprising: one or more exterior-facing sensors monitoring an outside environment of the vehicle; one or more interior-facing cameras monitoring a driver of the vehicle; a memory; and at least one processor coupled to the memory and configured to: receive sensor data from one or more exterior-facing sensors monitoring an environment outside of the vehicle and image data from one or more interior-facing cameras monitoring a driver of the vehicle;determine an awareness state of the driver based on the sensor data and the image data; and adjust a parameter used by the policy based on the awareness state.

[0396] The above system, where to determine the awareness state, the at least one processor is further configured to: identify an object in the sensor data; determine a gaze direction of the driver using the image data; and determine whether the gaze direction is associated with a spatial location of the object.

[0397] The above system, wherein the policy controls: (1) a velocity of the vehicle, (2) a forward distance, (3) a lateral distance, (4) a lane change decision, or (5) a pace of acceleration changes.

[0398] The above system, wherein the policy controls at least one of: (1) an autonomous driving system, (2) an adaptive cruise control system, (3) a collision avoidance system, (4) a blind-spot monitoring system, (5) a lane-change warning system, (6) a gaze-enabled lane change system, (7) a steering-assist system, (8) a lane-keep assist system, and / or (9) a lane departure warning system.

[0399] The above system, wherein the parameter reflects the driver’s level of anxiety, level of stress, or level of nausea.

[0400] A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations for adjusting a policy that controls operation of a vehicle, comprising: receiving sensor data from one or more exterior-facing sensors monitoring an environment outside of the vehicle and image data from one or more interior-facing cameras monitoring a driver of the vehicle; determining an awareness state of the driver based on the sensor data and the image data; and adjusting a parameter used by the policy based on the awareness state.Detecting Driver Impairment Based on Eye Scanning Movements

[0401] The techniques disclosed herein may be implemented or realized with the following illustrative examples.

[0402] A computer-implemented method for detecting driver impairment, comprising: receiving sensor data from one or more exterior-facing sensors monitoring an environment outside of a vehicle and image data from one or more interior-facing cameras monitoring a driver of the vehicle; identifying an object in the sensor data; determining a gaze direction of the driver using the image data; determining whether thegaze direction is associated with a spatial location of the object, such that the driver is determined to be looking at the object; when the driver is determined to be looking at the object, determining a period of time that the driver is looking at the object; determining an impairment level for the driver based on the period of time; and performing an action in an advanced driver assistance system based on the impairment level of the driver.

[0403] The above method, wherein the impairment level comprises data representing at least one of drowsiness, microsleep, sleep, unresponsiveness, or intoxication.

[0404] The above method, wherein the determining an impairment level comprises determining the impairment level based on a characteristic of the driver during the period of time, wherein the characteristic comprises at least one of a blink rate value, an eyelid closure duration value, an eyelid closure percentage value, an eye aspect ratio value, a gaze fixation value, or a yawning frequency value.

[0405] The above method, further comprising: tracking a plurality of saccadic eye movements of the driver; comparing the plurality of saccadic eye movements of the driver to a stored driver profile indicating a plurality of normal saccadic eye movements under normal circumstances; and when the plurality of saccadic eye movements diverge from the plurality of normal saccadic eye movements, updating the impairment level.

[0406] The above method, the tracking the plurality of saccadic eye movements further comprising: sampling an eye of the driver with a sampling frequency of up to 500 times per second with an eye-tracking sensor.

[0407] The above method, further comprising: building the stored driver profile to include a cognitive eye scanning pattern of the driver based on the plurality of normal saccadic eye movements, the cognitive eye scanning pattern gathered during a drive comprising observed environmental conditions; and storing the stored driver profile in a memory storage device.

[0408] The above method, the determining the gaze direction of the driver further comprising: building a heatmap representing the gaze direction of the driver using the image data, wherein the image data comprises a plurality of fixated movements and a plurality of saccadic eye movements of the driver.

[0409] A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising: receiving sensor data from one or more exterior-facing sensors monitoring an environment outside of a vehicle and image data from one or moreinterior-facing cameras monitoring a driver of the vehicle; identifying an object in the sensor data; determining a gaze direction of the driver using the image data; determining whether the gaze direction is associated with a spatial location of the object, such that the driver is determined to be looking at the object; when the driver is determined to be looking at the object, determining a period of time that the driver is looking at the object; determining an impairment level for the driver based on the period of time; and performing an action in an advanced driver assistance system based on the impairment level of the driver.

[0410] The above device, wherein the impairment level comprises data representing at least one of drowsiness, microsleep, sleep, unresponsiveness, or intoxication.

[0411] The above device, wherein the determining an impairment level comprises determining the impairment level based on a characteristic of the driver during the period of time, wherein the characteristic comprises at least one of a blink rate value, an eyelid closure duration value, an eyelid closure percentage value, an eye aspect ratio value, a gaze fixation value, or a yawning frequency value.

[0412] The above device, the operations further comprising: tracking a plurality of saccadic eye movements of the driver; comparing the plurality of saccadic eye movements of the driver to a stored driver profile indicating a plurality of normal saccadic eye movements under normal circumstances; and when the plurality of saccadic eye movements diverge from the plurality of normal saccadic eye movements, updating the impairment level.

[0413] The above device, the operation of tracking the plurality of saccadic eye movements further comprising: sampling an eye of the driver with a sampling frequency of up to 500 times per second with an eye-tracking sensor.

[0414] The above device, the operations further comprising: building the stored driver profile to include a cognitive eye scanning pattern of the driver based on the plurality of normal saccadic eye movements, the cognitive eye scanning pattern gathered during a drive comprising observed environmental conditions; and storing the stored driver profile in a memory storage device.

[0415] The above device, the determining the gaze direction of the driver further comprising: building a heatmap representing the gaze direction of the driver using the image data, wherein the image data comprises a plurality of fixated movements and a plurality of saccadic eye movements of the driver.

[0416] A system, comprising: one or more exterior-facing sensors monitoring an outside environment of a vehicle; one or more interior-facing cameras monitoring a driver of the vehicle; a memory; and at least one processor coupled to the memory and configured to: receive sensor data from the one or more exterior-facing sensors and image data from the one or more interior-facing cameras; identify an object in the sensor data; determine a gaze direction of the driver using the image data; determine whether the gaze direction is associated with a spatial location of the object, such that the driver is determined to be looking at the object; when the driver is determined to be looking at the object, determine a period of time that the driver is looking at the object; determine an impairment level for the driver based on the period of time; and perform an action in an advanced driver assistance system based on the impairment level of the driver.

[0417] The above system, wherein the impairment level comprises data representing at least one of drowsiness, microsleep, sleep, unresponsiveness, or intoxication.

[0418] The above system, wherein the determining an impairment level comprises determining the impairment level based on a characteristic of the driver during the period of time, wherein the characteristic comprises at least one of a blink rate value, an eyelid closure duration value, an eyelid closure percentage value, an eye aspect ratio value, a gaze fixation value, or a yawning frequency value.

[0419] The above system, the at least one processor further configured to: track a plurality of saccadic eye movements of the driver; compare the plurality of saccadic eye movements of the driver to a stored driver profile indicating a plurality of normal saccadic eye movements under normal circumstances; and when the plurality of saccadic eye movements diverge from the plurality of normal saccadic eye movements, update the impairment level.

[0420] The above system, the tracking the plurality of saccadic eye movements further comprising: sampling an eye of the driver with a sampling frequency of up to 500 times per second with an eye-tracking sensor.

[0421] The above system, the at least one processor further configured to: build the stored driver profile to include a cognitive eye scanning pattern of the driver based on the plurality of normal saccadic eye movements, the cognitive eye scanning pattern gathered during a drive comprising observed environmental conditions; and store the stored driver profile in a memory storage device.

[0422] The above system, wherein to determine the gaze direction of the driver with the at least one processor is further configured to: build a heatmap representing the gaze direction of the driver using the image data, wherein the image data comprises a plurality of fixated movements and a plurality of saccadic eye movements of the driver.Using Polarized Light for Driver Monitoring

[0423] The techniques disclosed herein may be implemented or realized with the following illustrative examples.

[0424] A camera system for monitoring at least one person in a vehicle, comprising: an illuminator configured to emit a light beam along an optical path; a polarizing optical device disposed in the optical path of the light beam, the polarizing optical device configured to perform polarizing of the light beam to generate a polarized light beam configured to penetrate a polarized surface; a camera configured to capture an image of the polarized light beam; and a processor coupled to the camera, the processor configured to conduct monitoring of the at least one person in the vehicle based on the image.

[0425] The above camera system, wherein the illuminator comprises a plurality of lightemitting diodes.

[0426] The above camera system, wherein the light beam comprises a near-IR light beam.

[0427] The above camera system, wherein the light beam comprises a wavelength of about 940 nm.

[0428] The above camera system, wherein the optical path is configured to align with an eye of the at least one person in the vehicle.

[0429] The above camera system, wherein the polarizing optical device comprises a polarizing filter.

[0430] The above camera system, wherein the polarizing optical device is configured to perform circular polarization of the light beam.

[0431] The above camera system, further comprising an actuator configured to rotate the polarizing optical device to a rotation angle such that the polarized light beam passes through a transmission axis of the polarized surface within a predetermined eye visibility threshold.

[0432] The above camera system, wherein the polarizing optical device comprises a plurality of polarizers rotated relative to each other by predetermined intervals of degrees such that one of the plurality of polarizers allows the polarized light beam to pass througha transmission axis of the polarized surface within a predetermined eye visibility threshold.

[0433] The above camera system, wherein the predetermined intervals of degrees comprise 90-degree intervals.

[0434] The above camera system, wherein the camera comprises a near-IR camera.

[0435] The above camera system, wherein the processor is configured to determine a gaze direction of the at least one person in the vehicle based on the image.

[0436] A camera apparatus for monitoring at least one person in a vehicle, comprising: an illuminator configured to emit a light beam along an optical path; a polarizing optical device disposed in the optical path of the light beam, the polarizing optical device configured to perform polarizing of the light beam to generate a polarized light beam configured to penetrate a polarized surface; a camera configured to capture an image of the polarized light beam; and a processor coupled to the camera, the processor configured to conduct monitoring of the at least one person in the vehicle based on the image.

[0437] The above camera apparatus, wherein the illuminator comprises a plurality of light-emitting diodes.

[0438] The above camera apparatus, wherein the light beam comprises a near-IR light beam.

[0439] The above camera apparatus, wherein the light beam comprises a wavelength of about 940 nm.

[0440] The above camera apparatus, wherein the optical path is configured to align with an eye of the at least one person in the vehicle.

[0441] The above camera apparatus, wherein the polarizing optical device comprises a polarizing filter.

[0442] The above camera apparatus, wherein the polarizing optical device is configured to perform circular polarization of the light beam.

[0443] The above camera apparatus, further comprising an actuator configured to rotate the polarizing optical device to a rotation angle such that the polarized light beam passes through a transmission axis of the polarized surface within a predetermined eye visibility threshold.

[0444] The above camera apparatus, wherein the polarizing optical device comprises a plurality of polarizers rotated relative to each other by predetermined intervals of degrees such that one of the plurality of polarizers allows the polarized light beam to pass througha transmission axis of the polarized surface within a predetermined eye visibility threshold.

[0445] The above camera apparatus, wherein the predetermined intervals of degrees comprise 90-degree intervals.

[0446] The above camera apparatus, wherein the camera comprises a near-IR camera.

[0447] The above camera apparatus, wherein the processor is configured to determine a gaze direction of the at least one person in the vehicle based on the image.

[0448] A method for capturing an image to monitor at least one person in a vehicle, comprising: emitting a light beam, with an illuminator, along an optical path; polarizing, with a polarizing optical device disposed in the optical path of the light beam, the light beam to generate a polarized light beam; penetrating a polarized surface with the polarized light beam; capturing an image of the polarized light beam with a camera; and determining a gaze direction, with a processor coupled to the camera, of the at least one person in the vehicle based on the image.

[0449] The above method, wherein the polarizing comprises performing circular polarization of the light beam.

[0450] The above method, wherein the polarizing comprises rotating, with an actuator, the polarizing optical device to a rotation angle such that the polarized light beam passes through a transmission axis of the polarized surface within a predetermined eye visibility threshold.

[0451] The above method, wherein the polarizing comprises transmitting the light beam through a plurality of polarizers rotated relative to each other by predetermined intervals of degrees such that one of the plurality of polarizers allows the polarized light beam to pass through a transmission axis of the polarized surface within a predetermined eye visibility threshold.Classifying Traffic Control Objects Based on Gaze Tracking and Object Detection

[0452] The techniques disclosed herein may be implemented or realized with the following illustrative examples.

[0453] A computer-implemented method for classifying traffic control objects based on gaze tracking and object detection, comprising: capturing a first image data of an eye of a driver from an internal camera and a second image data from an external camera of a vehicle; determining a gaze direction of the eye of the driver based on the first imagedata; detecting a traffic control object in the second image data; based on the gaze direction, determining whether the driver is looking at the traffic control object; determining a lane where the vehicle is located contemporaneous with the capture of the first image data and the second image data; and when the driver is determined to be looking at the traffic control object, generating data indicating that the traffic control object manages traffic on the lane of the vehicle.

[0454] The above method, further comprising: detecting a plurality of traffic control objects in the second image data; and determining which of the plurality of traffic control objects the driver is looking at based on the gaze direction and the second image data.

[0455] The above method, wherein the steps of claim 1 are repeated for a plurality of vehicles until a threshold of data is generated for the traffic control object and the lane of the vehicle.

[0456] The above method, further comprising: crowdsourcing the generated data by compiling the data into a dataset indicating that the traffic control object manages the traffic on the lane of the vehicle.

[0457] The above method, further comprising: detecting an instruction from the traffic control object managing the traffic on the lane of the vehicle; and performing an action in accordance with the detected instruction from the traffic control object.

[0458] The above method, wherein the traffic control object is a traffic light.

[0459] The above method, wherein the traffic control object is a sign providing roadway instructions to vehicles.

[0460] The above method wherein the first image data and the second image data are captured when the vehicle is stopped at an intersection with one or more traffic control objects.

[0461] A system, comprising: a memory; and at least one processor coupled to the memory and configured to: capture a first image data of an eye of a driver from an internal camera and a second image data from an external camera of a vehicle; determine a gaze direction of the eye of the driver based on the first image data; detect a traffic control object in the second image data; based on the gaze direction, determine whether the driver is looking at the traffic control object; determine a lane where the vehicle is located contemporaneous with the capture of the first image data and the second image data; and when the driver is determined to be looking at the traffic control object,generate data indicating that the traffic control object manages traffic on the lane of the vehicle.

[0462] The above system, further configured to: detect a plurality of traffic control objects in the second image data; and determine which of the plurality of traffic control objects the driver is looking at based on the gaze direction and the second image data.

[0463] The above system, wherein the steps of claim 9 are repeated for a plurality of vehicles until a threshold of data is generated for the traffic control object and the lane of the vehicle

[0464] The above system, further configured to: crowdsource the generated data by compiling the data into a dataset indicating that the traffic control object manages the traffic on the lane of the vehicle.

[0465] The above system, further configured to: detect an instruction from the traffic control object managing the traffic on the lane of the vehicle; and perform an action in accordance with the detected instruction from the traffic control object.

[0466] The above system, wherein the traffic control object is a traffic light.

[0467] The above system, wherein the traffic control object is a sign providing roadway instructions to vehicles.

[0468] A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising: capturing a first image data of an eye of a driver from an internal camera and a second image data from an external camera of a vehicle; determining a gaze direction of the eye of the driver based on the first image data; detecting a traffic control object in the second image data; based on the gaze direction, determining whether the driver is looking at the traffic control object; determining a lane where the vehicle is located contemporaneous with the capture of the first image data and the second image data; and when the driver is determined to be looking at the traffic control object, generating data indicating that the traffic control object manages traffic on the lane of the vehicle.

[0469] The above non-transitory computer-readable device, further comprising: detecting a plurality of traffic control objects in the second image data; and determining which of the plurality of traffic control objects the driver is looking at based on the gaze direction and the second image data.

[0470] The above non-transitory computer-readable device, wherein the steps of claim 16 are repeated for a plurality of vehicles until a threshold number of data points are met.

[0471] The above non-transitory computer-readable device, further comprising: crowdsourcing the generated data by compiling the data into a dataset indicating that the traffic control object manages the traffic on the lane of the vehicle.

[0472] The above non-transitory computer-readable device, further comprising: detecting an instruction from the traffic control object managing the traffic on the lane of the vehicle; and performing an action in accordance with the detected instruction from the traffic control object.Rolling Calibration of Gaze Detection Based on Gaze Tracking and Object Detection

[0473] The techniques disclosed herein may be implemented or realized with the following illustrative examples.

[0474] A computer-implemented method for classifying traffic control objects based on gaze tracking and object detection, comprising: capturing one or more images of an eye of a driver of a vehicle from an inward-facing camera and respectively corresponding images from an external-facing camera; for each of the one or more images of the eye from the inward-facing camera and the respectively corresponding images from the external-facing camera: determining, based on a set of calibration parameters, a gaze direction of the driver based on stored image data of the respectively corresponding images from the inward-facing camera and the external-facing camera; determining one or more objects intersecting the gaze direction of the driver; and based on a location of the one or more objects, adjusting the set of calibration parameters.

[0475] The above method, further comprising: determining a head movement of the driver during capture at the one or more images of the eye of the driver; and updating the gaze direction based on the determined head movement of the driver.

[0476] The above method, wherein the one or more captured images of the eye and the respectively corresponding images from the external camera are captured on a route commonly taken by the driver and captured each time the driver is taking the route.

[0477] The above method, further comprising: analyzing the gaze direction of the driver to correlate the depth of the gaze direction at the location of the one or more objects in the gaze direction of the driver.

[0478] The above method, further comprising: training a machine learning algorithm using the stored image data correlating the gaze direction and the one or more objects to recognize gaze patterns of the driver.

[0479] The above method, wherein a field of view of the gaze direction, as determined by the stored image data of the eye, and a field of view of the external-facing camera are coincident.

[0480] The above method, wherein the gaze direction is represented as a two dimensional vector forming an epipolar line representing an intersection of the one or more images of the eye and the respectively corresponding images from the external camera.

[0481] A system, comprising: a memory; and at least one processor coupled to the memory and configured to: receive one or more images of an eye of a driver of a vehicle from an inward-facing camera and respectively corresponding images from an externalfacing camera; for each of the one or more images of the eye from the inward-facing camera and the respectively corresponding images from the external-facing camera: determining, based on a set of calibration parameters, a gaze direction of the driver based on stored image data of the respectively corresponding images from the inward-facing camera and the external-facing camera; determining one or more objects intersecting the gaze direction of the driver; and based on a location of the one or more objects, adjusting the set of calibration parameters.

[0482] The above system, where determining the gaze direction is further configured to: determine a head movement of the driver during capture at the one or more images of the eye of the driver; and update the gaze direction based on the determined head movement of the driver.

[0483] The above system, wherein the one or more captured images of the eye and the respectively corresponding images from the external camera are captured on a route commonly taken by the driver and captured each time the driver is taking the route.

[0484] The above system, further configured to: analyzing the gaze direction of the driver to correlate the depth of the gaze direction at the location of the one or more objects in the gaze direction of the driver.

[0485] The above system, further configured to: train a machine learning algorithm using the stored image data correlating the gaze direction and the one or more objects to recognize gaze patterns of the driver.

[0486] The above system, wherein a field of view of the gaze direction, as determined by the one or more captured images of the eye, and a field of view of the external-facing camera are coincident.

[0487] The above system, wherein the gaze direction is represented as a two dimensional vector forming an epipolar line representing an intersection of the one or more images of the eye and the respectively corresponding images from the external camera.

[0488] A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising: capturing one or more images of an eye of a driver of a vehicle from an inward-facing camera and respectively corresponding images from an external-facing camera; for each of the one or more images of the eye from the inwardfacing camera and the respectively corresponding images from the external-facing camera: determining, based on a set of calibration parameters, a gaze direction of the driver based on stored image data of the respectively corresponding images from the inward-facing camera and the external -facing camera; determining one or more objects intersecting the gaze direction of the driver; and based on a location of the one or more objects, adjusting the set of calibration parameters..

[0489] The above non-transitory computer-readable device, where generating the gaze direction further comprises: determining a head movement of the driver during capture at the one or more images of the eye of the driver; and updating the gaze direction based on the determined head movement of the driver.

[0490] The above non-transitory computer-readable device, wherein the one or more captured images of the eye and the respectively corresponding images from the external camera are captured on a route commonly taken by the driver and captured each time the driver is taking the route.

[0491] The above non-transitory computer-readable device, further comprising: analyzing the gaze direction of the driver to correlate the depth of the gaze direction at the location of the one or more objects in the gaze direction of the driver.

[0492] The above non-transitory computer-readable device, further comprising: training a machine learning algorithm using the stored image data correlating the gaze direction and the one or more objects to recognize gaze patterns of the driver.

[0493] The above non-transitory computer-readable device, wherein a field of view of the gaze direction, as determined by the stored image data of the eye, and a field of view of the external-facing camera are coincident.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for adjusting a policy that controls operation of a vehicle, comprising: receiving sensor data from one or more exterior-facing sensors monitoring an environment outside of the vehicle and image data from one or more interior-facing cameras monitoring a driver of the vehicle; determining an awareness state of the driver based on the sensor data and the image data; and adjusting a parameter used by the policy based on the awareness state.

2. The method of claim 1, the determining the awareness state of the driver further comprising: identifying an object in the sensor data; determining a gaze direction of the driver using the image data; and determining whether the gaze direction is associated with a spatial location of the object.

3. The method of claim 2, wherein the object is a traffic light, a traffic sign, a pedestrian, a second vehicle, an animal, or a traffic barrier.

4. The method of claim 1, wherein the policy controls at least one of: (1) a velocity of the vehicle, (2) a forward distance, (3) a lateral distance, (4) a lane change decision, and / or(5) a pace of acceleration changes.

5. The method of claim 1, wherein the policy controls at least one of:(1) an autonomous driving system,(2) an adaptive cruise control system,(3) a collision avoidance system,(4) a blind-spot monitoring system,(5) a lane-change warning system,(6) a gaze-enabled lane change system.(7) a steering-assist system,(8) a lane-keep assist system, and / or(9) a lane departure warning system.

6. The method of claim 1, wherein the policy specifies an autonomous driving level that is one of a plurality of autonomous driving levels defined by a Society of Automotive Engineers standard.

7. The method of claim 1, wherein the policy specifies an autonomous driving level that is one of: (1) a level 0 policy of no driving automation, (2) a level 1 policy of driver assistance, (3) a level 2 policy of partial driving automation, (4) a level 3 policy of conditional driving automation, (5) a level 4 policy of high driving automation, or (6) a level 5 policy of full driving automation.

8. The method of claim 1, the adjusting further comprising: modifying a takeover notice period used by an autonomous driving system based on the awareness state.

9. The method of claim 1, wherein the policy controls an adaptive cruise control system, and wherein the parameter controls: (1) a maximum permissible lateral acceleration, (2) a maximum stopping acceleration, (3) a minimum time gap, (4) a maximum time gap, (5) a minimum clearance, (6) a minimum speed, (7) a maximum speed, or (8) a distance to a second vehicle.

10. The method of claim 1, wherein the policy controls a blind-spot monitoring system, a lane-change warning system, or a gaze-enabled lane change system, and wherein the parameter controls: (1) a speed difference tolerance between the vehicle and a target vehicle, (2) a time to collision tolerance, or (3) a warning level.

11. The method of claim 1, wherein the policy specifies an alarm tolerance level that controls when a collision avoidance system generates a driver alert, the adjusting further comprising: when the gaze direction is associated with the spatial location of the object, increasing the alarm tolerance level; and- I l l - when the gaze direction is not associated with a spatial location of the object, decreasing the alarm tolerance level.

12. The method of claim 1, wherein the policy controls a steering-assist system, a lane-keep assist system, or a lane departure warning system, and wherein the parameter controls: (1) a threshold distance to a lane boundary, (2) a warning level, or (3) a lane change tolerance.

13. The method of claim 1, further comprising: updating a current attention level value that changes over a driving session based on the awareness state; and displaying an indication related to the current attention level value on a feedback display during the driving session.

14. The method of claim 1, wherein the parameter reflects the driver’s level of anxiety, level of stress, or level of nausea.

15. A system for adjusting a policy that controls operation of a vehicle, comprising: one or more exterior-facing sensors monitoring an outside environment of the vehicle; one or more interior-facing cameras monitoring a driver of the vehicle; a memory; and at least one processor coupled to the memory and configured to: receive sensor data from one or more exterior-facing sensors monitoring an environment outside of the vehicle and image data from one or more interior-facing cameras monitoring a driver of the vehicle; determine an awareness state of the driver based on the sensor data and the image data; and adjust a parameter used by the policy based on the awareness state.

16. The system of claim 15, where to determine the awareness state, the at least one processor is further configured to: identify an object in the sensor data;determine a gaze direction of the driver using the image data; and determine whether the gaze direction is associated with a spatial location of the object.

17. The system of claim 15, wherein the policy controls: (1) a velocity of the vehicle, (2) a forward distance, (3) a lateral distance, (4) a lane change decision, or (5) a pace of acceleration changes.

18. The system of claim 15, wherein the policy controls at least one of:(1) an autonomous driving system,(2) an adaptive cruise control system,(3) a collision avoidance system,(4) a blind-spot monitoring system,(5) a lane-change warning system,(6) a gaze-enabled lane change system.(7) a steering-assist system,(8) a lane-keep assist system, and / or(9) a lane departure warning system.

19. The system of claim 14, wherein the parameter reflects the driver’s level of anxiety, level of stress, or level of nausea.

20. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations for adjusting a policy that controls operation of a vehicle, comprising: receiving sensor data from one or more exterior-facing sensors monitoring an environment outside of the vehicle and image data from one or more interior-facing cameras monitoring a driver of the vehicle; determining an awareness state of the driver based on the sensor data and the image data; and adjusting a parameter used by the policy based on the awareness state.

Images