An awareness checker to enhance collaborative driving supervision.

Abstract

The apparatus includes an interface and a control circuit. The interface may be configured to (i) receive sensor-based information from a plurality of sensor signals from a vehicle platform of the vehicle and environmental information regarding the vehicle's environment, and (ii) present one or more control signals to the vehicle platform. The control circuit may be configured to (i) detect whether a driver's attention state is attentive or inattentive in response to one or more of the plurality of sensor signals from the vehicle platform and the driver monitoring system, (ii) evaluate whether the driver is sufficiently attentive by monitoring one or more of the plurality of sensor signals from the vehicle platform and the driver monitoring system, and (iii) maintain a driver's awareness of the vehicle's environment by actively interacting with the driver to determine whether the driver's perception of the vehicle's environment corresponds to the environmental information and the sensor-based information received by the interface.

Description

[Technical field]

[0001] The present invention relates generally to advanced driver assistance systems, and more particularly to a method and / or apparatus for implementing an awareness checker to enhance collaborative driving supervision. [Background technology]

[0002] Supervised advanced driver assistance systems (ADAS) cruising features (SAE L0-L2+) are becoming more prevalent worldwide and are expected to become mainstream in the coming years. A common design-level assumption for the safety case of ADAS cruising features is that the driver is expected to be solely responsible for the safety of the driving task by monitoring the driving environment. However, as can be seen from recently introduced similar features, a large deviation exists between the design-level assumptions and real-world usage for a variety of reasons. Some exemplary reasons include: drivers frequently check out during supervision behaviors; feature capabilities are overestimated (e.g., automation fallacy); features are unintentionally misused and / or intentionally misused; eyes on the road and hands on the steering wheel are not necessarily focused on driving; confusion on "who does what" (e.g., human vs. automation) may occur. As a result, the release of drivers from involvement in driving supervision when using assisted driving features has resulted in accidents and unacceptable risks on roads in the real world. Autopilot features that are considered safe are creating more distracted drivers: Drivers who regularly use cooperative driver assistance systems are nearly twice as likely to drive in a distracted state.

[0003] It is desirable to implement an awareness checker to enhance collaborative driving supervision. Summary of the Invention

[0004] The present invention relates to an apparatus including an interface and a control circuit. The interface may be configured to (i) receive sensor-based information from a plurality of sensor signals from a vehicle platform of a vehicle and environmental information related to the vehicle's environment, and (ii) present one or more control signals to the vehicle platform. The control circuit may be configured to (i) detect whether a driver's attention state is attentive or inattentive in response to one or more of the plurality of sensor signals from the vehicle platform and the driver monitoring system, (ii) evaluate whether the driver is sufficiently attentive by monitoring one or more of the plurality of sensor signals from the vehicle platform and the driver monitoring system, and (iii) maintain the driver's awareness of the vehicle's environment by actively interacting with the driver to determine whether the driver's perception of the vehicle's environment corresponds to the environmental information and the sensor-based information received by the interface.

[0005] Embodiments of the invention will become apparent from the following detailed description and the accompanying claims and drawings. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 illustrates a system according to one embodiment of the present disclosure. [Diagram 2] FIG. 2 illustrates an example implementation of the environment monitoring system blocks of FIG. 1. [Diagram 3] FIG. 1 illustrates an implementation of an Advanced Driver Assistance System (ADAS) feature control including an awareness checker according to an exemplary embodiment of the present invention. [Figure 4] FIG. 2 illustrates an exemplary interaction between a driver and a system, according to an exemplary embodiment of the present invention. [Diagram 5] FIG. 2 illustrates an exemplary interaction between a driver and a system, according to an exemplary embodiment of the present invention. [Figure 6]FIG. 2 illustrates an exemplary interaction between a driver and a system, according to an exemplary embodiment of the present invention. [Figure 7] FIG. 2 illustrates an exemplary interaction between a driver and a system, according to an exemplary embodiment of the present invention. [Figure 8] FIG. 2 illustrates an exemplary interaction between a driver and a system, according to an exemplary embodiment of the present invention. [Figure 9] FIG. 2 illustrates an exemplary interaction between a driver and a system, according to an exemplary embodiment of the present invention. [Figure 10] FIG. 2 illustrates an exemplary interaction between a driver and a system, according to an exemplary embodiment of the present invention. [Figure 11] FIG. 2 illustrates an exemplary interaction between a driver and a system, according to an exemplary embodiment of the present invention. [Figure 12] FIG. 2 illustrates an exemplary interaction between a driver and a system, according to an exemplary embodiment of the present invention. [Figure 13] FIG. 1 illustrates an awareness check process according to an exemplary embodiment of the present disclosure. [Figure 14] FIG. 1 illustrates an electronic control unit implementing an advanced driver assistance system (ADAS) feature control system, according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Embodiments of the present invention include (i) making the task of ADAS supervision more collaborative; (ii) proactively engaging the driver instead of reactively escalating after distractions; (iii) focusing on rewarding good engagement (positive measure) through collaboration rather than punishing bad supervision behavior (negative measure); (iv) reminding the driver to monitor the driving environment when a feature is active, thereby reducing the likelihood of non-driving related task involvement (NDRT); (v) employing an in-vehicle digital assistant as a user interface to explore the idea of ​​interactive safety; (vi) enabling human-autonomous collaboration through dialogue and positive reinforcement; (vii) providing three levels of monitoring for driver engagement detection during assisted driving; and / or (viii) providing an awareness checker for enhancing collaborative driving supervision, which may be implemented as one or more integrated circuits.

[0008] Cooperative driving features are responsible for clarifying supervisory expectations for the driver and ensuring that the driver does not lose situational awareness. In various embodiments, the task of ADAS supervision is made more collaborative by actively engaging the driver instead of reactively escalating responses after driver distraction is detected. In various embodiments, a system may be provided that focuses on rewarding good engagement (positive measure) through collaboration rather than punishing bad supervisory behavior (negative measure). In one example, the feature control system may remind the driver to monitor the driving environment when the feature is active, thereby reducing the likelihood of secondary non-driving related task (NDRT) engagement.

[0009] In one example, a new component (e.g., hardware, software, or a combination thereof) may be implemented. The new component is generally referred to as an awareness checker. In one example, the awareness checker may employ an in-vehicle digital assistant as a user interface to explore the idea of ​​conversational safety (similar to Level 3 in E-Gas3 level monitoring below). The awareness checker generally enables human-autonomous cooperation through dialogue and positive reinforcement.

[0010] In one example, the driver monitoring concept may be implemented at three levels. The first level (or level 1) may be implemented similar to existing systems (e.g., hyper traffic jam assistance (HTJA) attention monitor). The second level (e.g., level 2) may be implemented similar to a human machine interaction monitor (HMIM) disclosed in co-pending U.S. patent application Ser. No. 17 / 493,144, filed Oct. 4, 2021, which is incorporated herein by reference in its entirety. The third level (e.g., level 3) may be implemented as an awareness checker (AC) to detect whether the driver is focused on supervising the autonomous features. In one example, level 1 may be designated as a function level, level 2 may be designated as a function monitoring level, and level 3 may be designated as a controller monitoring level.

[0011] In one example, Level 1 may include engine control functions (e.g., realization of required engine torque, component monitoring, and input / output variable diagnostics to control system reaction if a fault is detected). In one example, Level 2 may detect faulty processes in Level 1 function software (e.g., by monitoring calculated torque values ​​or vehicle acceleration, etc.). If a fault is detected, a system reaction may be triggered. In one example, Level 3 may implement a monitoring module. The monitoring module may be an independent part of the function controller (e.g., ASIC or controller) that tests for correct execution of the program during the question-answer process. Current systems in the market only provide a single level of engagement monitoring and are often susceptible to predictable misuse and abuse. In various embodiments, an industry-first three-level monitoring for driver engagement detection during assisted driving may be provided.

[0012] Referring to FIG. 1, a diagram illustrating a system according to an embodiment of the present invention is shown. In one example, the system (or device) 90 may implement an advanced driver assistance system (ADAS). In various embodiments, the system 90 may include a vehicle platform 92, a driver monitoring system (DMS) 94, a human machine (or vehicle) interface (HMI) 96, an environmental monitoring system (EMS) 98, and a function control module 100. In various embodiments, the vehicle platform 92, the driver monitoring system (DMS) 94, the environmental monitoring system (EMS) 98, and the function control module 100 may be implemented as an Automotive Safety Integrity Level (ASIL), while the human machine interface (HMI) 96 may be implemented as a Quality Management (QM).

[0013] Automotive Safety Integrity Levels (ASILs) are a risk classification scheme defined by the ISO 26262 - Functional Safety for Road Vehicles standard. It is an adaptation of the Safety Integrity Levels (SILs) used in the IEC 61508 for the automotive industry. The ASIL classification helps define the safety requirements that must be met in the ISO 26262 standard to keep the risk at an acceptable level. The ASILs are established by performing a risk analysis of potentially hazardous scenarios in terms of the severity, exposure, and controllability of the vehicle operation scenario. The safety goals for the hazardous scenarios contain the resulting ASIL requirements. The ASILs range from ASIL D, which represents the highest degree of stringency that should be applied to ensure the highest risk of a hazardous scenario escalating into an accident and the resulting safety requirements, to QM, which represents an application where there are no automotive hazardous scenarios with unacceptable risks and therefore no safety requirements to manage under the ISO 26262 safety process. The level QM, which refers to "quality management", means that the risks associated with the hazardous events are not unreasonable and therefore do not require safety measures in accordance with ISO 26262. The intervening levels (ASIL C, ASIL B, and ASIL A) are simply ranges of varying degrees of hazard risk level and degree of assurance and engineering rigor required.

[0014] The standard defines functional safety as "the absence of unreasonable risk due to hazards caused by malfunctioning behavior of electrical or electronic systems." The ASILs define the safety requirements for automotive components to meet ISO 26262 based on the severity, probability, and controllability of hazard scenarios. Systems such as airbags, antilock brakes, and power steering require the highest rigor of safety assurance, which applies with an ASIL D grade, since the risks associated with their failure are the highest. At the other end of the safety spectrum, components such as windshield wiper systems only require an ASIL A grade. Headlights and brake lights should generally be ASIL B, as should rear lights, due to the risk of rear collision, while automatic emergency braking systems should generally be ASIL C due to the risk associated with unintended deceleration.

[0015] In one example, the vehicle platform 92, the DMS 94, the HMI 96, and the EMS 98 may provide input signals to the function control module 100. In one example, the vehicle platform 92 may provide several input signals (e.g., “Vehicle Speed”, “Window and Door Lock Status”, “Tire Pressure”, “HVAC Status”, etc.). The signal “Vehicle Speed” may communicate the vehicle (longitudinal) speed to the function control module 100. The signal “Window and Door Lock Status” may communicate whether the windows are open or closed and the doors are open or closed, locked or unlocked. The signal “Tire Pressure” may communicate the inflation status (e.g., psi) of the tires of the vehicle. The signal “HVAC Status” may communicate whether the air conditioner (AC) or heater (HTR) is running, blower speed, cabin temperature, etc. The DMS 94 may provide input signals conveying information regarding the driver's attention (eg, the driver's eye movements, the driver's hand position, the steering angle, the brake and accelerator pedal positions, etc.).

[0016] In one example, the HMI 96 may provide a first input signal (e.g., “Activate Request”), a second input signal (e.g., “Deactivate Request”), and a third signal (e.g., “Driver Response”) to the feature control module 100. The signal “Activate Request” may communicate a request from the driver to activate an ADAS feature controlled by the feature control module 100. The signal “Deactivate Request” may communicate a request from the driver to deactivate an ADAS feature controlled by the feature control module 100. The signal “Driver Response” may communicate a response from the driver to a query presented to the driver via the HMI 96 (e.g., by an avatar or in-vehicle personal assistant). In some embodiments, the HMI 96 may optionally present an input signal (e.g., “Driver Information”) that conveys information about a particular driver operating the vehicle. In various embodiments, the signal "vehicle speed" may be implemented as ASIL, and the signals "activation request", "deactivation request", and "driver response" may be implemented as QM.

[0017] In one example, the function control module 100 may provide output signals to the vehicle platform 92 and the HMI 96. In one example, the function control module 100 may present an output signal (e.g., “Decelerate Request”) to the vehicle platform 92. The signal “Decelerate Request” may be configured to enable the function control module 100 to safely stop the vehicle. In various embodiments, the signal “Decelerate Request” may be implemented as an ASIL. The function control module 100 may also present several signals (e.g., “Driver Alert”, “Driver Question”, “Driver Attention Status”, etc.) to the HMI 96. The signal “Driver Alert” may communicate information to cause the HMI 96 to present a particular alert to the driver. The signal “Driver Question” may communicate information to cause the HMI 96 to present a particular question to the driver. The signal “Driver Attention Status” may communicate information to cause the HMI 96 to present a particular avatar state to the driver. In various embodiments, the signals "Warn Driver" and "Question Driver" may be implemented as QM. In various embodiments where the signal "Driver Attention to Task Status" is utilized only to actively collaborate with the driver in the task of monitoring the environment and only indirectly supports the safety case, the signal "Driver Attention to Task Status" may be implemented as QM. In embodiments where the signal "Driver Attention to Task Status" is subscribed to by safety-related elements of the function control module 100, the signal "Driver Attention to Task Status" may be implemented as ASIL.

[0018] In one example, the function control module 100 may include a block (or circuit) 102 and a block (or circuit) 104. The block 102 may be implemented as a driver awareness estimator (DAE). The block 104 may be implemented as an ADAS feature mode manager. In one example, the block 104 may be implemented as an autopilot mode manager. However, other autonomous features may be implemented as appropriate. In various embodiments, the blocks 102 and 104 are generally implemented as ASILs. In one example, a signal “vehicle speed” may be presented to a first input of the block 102 and a first input of the block 104. A signal from the DMS 94 may be presented to a second input of the block 102. A signal from the HMI 96 may be presented to a third input of the block 102 and a second input of the block 104. A signal from the EMS 98 may be presented to a fourth input of the block 102. Block 102 may present one signal (e.g., “Awareness Level”) to a third input of block 104 and one signal (e.g., “Fully Attention”) to a fourth input of block 104. The signals “Awareness Level” and “Fully Attention” may be implemented as ASIL. In embodiments in which HMI 96 provides signal “Driver Info” to function control module 100, signal “Driver Info” may be presented to a fifth input of block 102. Signal “Driver Info” may be implemented as QM.

[0019] In various embodiments, the block 102 may be configured as a driver attention estimator (DAE) to systematically detect whether the driver is paying attention and bring about an appropriate safety state when the driver is no longer paying attention. In one example, the driver attention estimator 102 may (i) provide a driver attention level escalation regime that generates warnings at several (e.g., three) different levels via the HMI 96, (ii) detect attention deficits associated with short-term shared visual attention (e.g., from engagement in secondary or non-driving related tasks), (iii) proactively engage the driver instead of reactively escalating after distraction, and (iv) remind the driver to monitor the driving environment when automation features are active, thereby reducing the likelihood of non-driving related task (NDRT) engagement.

[0020] In one example, the driver attention estimator 102 may comprise a block (or circuit) 110, a block (or circuit) 112, and a block (or circuit) 114. The block 110 may be implemented as an attention (or awareness) monitor. The block 112 may be implemented as a human-machine interaction monitor (HMIM). The block 114 may be implemented as an awareness checker (AC). In various embodiments, the blocks 110 and 112 are generally implemented as ASILs. In various embodiments in which the signal “driver attention status” is utilized only to actively cooperate with the driver in the task of environment monitoring and only indirectly assists the safety case, the block 114 may be implemented as a QM. In embodiments in which the signal “driver attention status” is subscribed to by safety-related elements of the function control module 100 (e.g., the blocks 110, 112, etc.), the block 114 may be implemented as an ASIL.

[0021] In one example, the signal "vehicle speed" may be presented to a first input of block 110, a first input of block 112, and a first input of block 114. A signal from DMS 94 may be presented to a second input of block 110, a second input of block 112, and a second input of block 114. A signal from HMI 96 may be presented to a third input of block 114. A signal from EMS 98 may be presented to a fourth input of block 114. Block 110 may present a signal "attention level" to a third input of block 112 and a fifth input of block 114. The signal "attention level" may be implemented as an ASIL. Block 112 may present a signal "fully attentive" to a third input of block 104. The signal "fully attentive" may be implemented as an ASIL. Block 114 may present a signal (e.g., "driver task concentration status") as an output signal. In one example, the signal "Driver Attention to Task Status" may communicate the driver attention level determination by block 114 to the vehicle's external and internal environments. In one example, the signal "Driver Attention to Task Status" may be stored in local memory for use by other modules of function control module 100.

[0022] In various embodiments where the signal "Driver Attention to Task Status" is utilized only to actively cooperate with the driver in the task of monitoring the environment and only indirectly assists the safety case, the signal "Driver Attention to Task Status" may be implemented as a QM. In embodiments where the signal "Driver Attention to Task Status" is subscribed to by block 110 and / or block 112, the signal "Driver Attention to Task Status" may be implemented as an ASIL. In embodiments where the HMI 96 provides the signal "Driver Info" to the function control module 100, the signal "Driver Info" may be presented to a fifth input of block 112 and a sixth input of block 114. The signal "Driver Info" may be implemented as a QM.

[0023] In various embodiments, blocks 110, 112, and 114 may be configured as a driver attention estimator (DAE) to systematically detect whether the driver is paying attention and bring about an appropriate safety state when the driver is no longer paying attention. In one example, the attention monitor 110 may provide a driver attention level escalation regime that generates warnings at several (e.g., three) different levels via the HMI 96 and may detect attention deficits associated with short-term shared visual attention (e.g., from engagement in a secondary task). The attention monitor 110 may be implemented similarly to existing product attention tracking features currently sold in the market that employ algorithms to track and classify the driver's visual attention.

[0024] In various embodiments, the driver attention estimator (DAE) 102 does not rely solely on the attention monitor 110 to keep the driver functionally alert (thereby meeting safety goals) due to the challenges associated with many human factors in keeping the driver engaged in the driving task. In various embodiments, the driver attention estimator (DAE) 102 may account for the types of foreseeable misuse and abuse of similar features documented in the current market. For example, misuse may result in edge cases where the attention monitor 110 labels the driver as "fully aware," thereby preventing the driver from intervening if some hazardous event occurs, while the automation features controlled by the function control module 100 remain active. Therefore, the driver attention estimator (DAE) 102 according to an embodiment of the present invention generally utilizes the HMIM 112 and AC 114 to provide additional functionality to systematically detect drivers who are no longer paying enough attention to the attention monitor 110 to serve as a safety net.

[0025] The HMIM 112 is generally configured to detect lack of attention when the HMI 96 is not operational or there is driver misuse. In one example, the HMIM 112 may examine the driver's off-road gaze distribution pattern by analyzing the switching behavior between the attention levels reported by the attention monitor 110 over a duration longer than the duration (or window) used by the attention monitor 110. In some embodiments, the HMIM 112 may also examine the driver's awareness of the vehicle's surroundings by subscribing to and analyzing a signal "driver on-task attention status" reported by the attention checker 114. In various embodiments, the long-term gaze distribution pattern and the driver on-task attention status may be used to affect the kinesthetic and longitudinal control of the vehicle platform 92.

[0026] In one example, the HMIM 112 may look at a longer term assessment based on switching behavior between attention states reported by the attention monitor 110. By monitoring the driver's attention level in a given time window (e.g., captured by the time distribution of the attention states), an adjustable (or programmable) number of transitions and an acceptable total time in each attention state may be defined. Using assisted driving (e.g., adaptive cruise control (ACC) etc.) gaze behavior as an absolute criterion, the driver engagement may be calculated based on the gaze distribution pattern. In an embodiment where the HMIM 112 also subscribes to the signal "driver's attention to task status", the HMIM 112 may be configured to take into account the transitions between different states of the signal "driver's attention to task status" when calculating (or evaluating) the driver engagement. The HMIM 112 generally assesses the driver's long-term gaze patterns and / or task focus status, and then prevents the driver from repeatedly entering longer duration lower awareness states (which may affect driver controllability) by triggering the transfer of control to the driver and transition of the vehicle to a safe state (e.g., via a signal "request to slow down," etc.). In one example, the HMIM 112 may be implemented similarly to the human-machine interaction monitor (HMIM) disclosed in co-pending U.S. patent application Ser. No. 17 / 493,144, filed Oct. 4, 2021, which is incorporated herein by reference in its entirety.

[0027] The AC 114 may be implemented as part of a driver attention estimator (DAE) 102 for assessing driver attention. In one example, when supervised highway automation is active, the DAE 102 may monitor the driver attention level (e.g., via the attention monitor 110) using road gaze information provided by an in-cabin driver monitoring camera. Based on the road gaze information, the driver attention level may be classified into four different states: aware, temporarily unaware, unaware, and out of the loop. In one example, immediately after the driver is assessed as aware, the AC 114 may be configured to actively maintain the driver in a continuously aware state (e.g., using positive reinforcement or reward techniques).

[0028] In one example, keeping the driver aware may be accomplished by asking questions related to the external world in the vicinity of the vehicle as perceived by environmental sensors and / or the status of the vehicle within the cabin as perceived by in-cabin sensors. The specific content of the questions may change dynamically based on available information at hand of the AC 114 (e.g., from the vehicle platform 92 and the environmental monitoring system 98). For example, the AC 114 may not ask about weather conditions unless the AC 114 is aware of the current weather conditions (e.g., from the Internet or from a camera that provides a view of the exterior of the vehicle, etc.). If the vehicle does not have an Internet connection, the AC 114 may ask questions related to roadside objects or traffic signs, which may be verified against information provided by the perception or location identification module of the feature control module 100. In one example, based on the response by the driver, the driver's on-task concentration status may be classified into three different states: happy (or not worried), worried, and alert.

[0029] Referring to FIG. 2, a diagram illustrating an implementation of an environment monitoring system of an advanced driver assistance system (ADAS) feature control including an awareness checker is shown according to an exemplary embodiment of the present invention. In one example, the function control module 100 may further include a location determination circuit 103, a map interface 105, and a perception circuit 109. In one example, the environment monitoring system 98 may include a block (or circuit) 210, a block (or circuit) 212, a block (or circuit) 214, a block (or circuit) 216, and a block (or circuit) 218. The circuit 210 may implement a high-definition (HD) digital map. The circuit 212 may implement satellite-based positioning. In one example, the circuit 212 may include a global positioning system (GPS) or a global navigation satellite system (GNSS) receiver. The circuit 212 may be configured to determine the location of the vehicle based on received satellite signals. The circuit 214 may implement several cameras (e.g., a forward-looking camera, a rear-looking camera, several corner-view cameras, several side-view cameras, etc.). The circuit 216 may implement several radar sensors (e.g., a forward-looking radar, a corner and / or side-looking radar, a rear-looking radar, etc.). The circuit 218 may implement a wireless communication interface. The circuit 218 may be configured to obtain weather information (e.g., from the Internet).

[0030] The circuit 210 may have an input that may receive raw location information (e.g., latitude, longitude, etc.) from the satellite-based positioning circuit 212. In response to the raw location data, the circuit 210 may be configured to present map horizon data to an input of the location determination circuit 103 and to an input of the map interface circuit 105. The circuit 212 may also be configured to present the raw location data to the location determination circuit 103. The location determination circuit 103 may be configured to present vehicle location information to the map interface circuit 105. The map interface circuit 105 may be configured to generate map-based environmental information (e.g., landmark and traffic sign information) in response to the map horizon data received from the HD map 210 and the vehicle location information received from the location determination circuit 103. The map interface circuit 105 may be configured to present the map-based environmental information to an input of the awareness checker 114.

[0031] The circuit 214 may include a number of on-board cameras, including but not limited to a forward looking camera (FLC) 220. The forward looking camera (FLC) 220 may present one or more signals conveying visual detection (e.g., “visual detection”) to an input of the location determination module (or circuit) 103 and an input of the perception module (or circuit) 109. The circuit 216 may include but is not limited to front corner / side radars (FCR&FSR or FCSR) 222 and a forward looking radar (FLR) 224. The circuit 216 may present one or more signals conveying radar detection (e.g., “radar detection”) to an input of the circuit 103 and an input of the circuit 109. In one example, the front corner / side radar (FCSR) 222 may present a first portion of a signal conveying radar detection "radar detected" to an input of the circuitry 103, and the forward looking radar (FLR) 224 may present a second portion of the signal conveying radar detection "radar detected" to a second input of the perception module 109. The location determination circuitry 103 may be configured to generate vehicle location information presented on the map interface 105 in response to raw position data received from the satellite-based positioning circuitry 212, map horizon data received from the HD map 210, visual detections received from the FLC 220, and radar detections received from the FCSR 222.

[0032] The perception module 109 may be configured to generate signals conveying static and dynamic object reports in response to visual detection from a forward looking camera (FLC) 220 and radar detection from a forward looking radar (FLR) 224. The static and dynamic object report signals generated by the perception module 109 may be presented to an input of the awareness checker 114.

[0033] In one example, the perception module 109 may be implemented as a software component. In one example, the perception module 109 may be utilized in SAE L2+ automation features such as Hyper Traffic Jam Assistance (HTJA). In various embodiments, the perception module 109 may utilize image data from the FLC 220 and point cloud data from several of the FCSRs 222a-222b and FLR 224 to (i) detect the presence of landmarks, traffic signs and traffic lights, pedestrians, and other objects, and (ii) analyze oncoming traffic, which may be further utilized by the awareness checker 114 to determine the driver's awareness state (e.g., as the subject of questions presented to the driver). The perception module 109 generally performs sensor fusion of on-board sensors. The perception module 109 generally fuses image data from the FLC 220 with point cloud data from the FCSRs 222a-222b and the FLR 224 to (i) detect the presence of landmarks, traffic signs and traffic lights, pedestrians, and other objects, (ii) track objects (or targets) in the environment around the vehicle, and (iii) analyze oncoming traffic.

[0034] In one example, the perception module 109 may detect objects in the vehicle's surrounding environment based on the on-board sensor data. In one example, the objects detected by the perception module 109 may be used as a cross-check against objects identified in the map data. For example, the map data may describe roads and their segments, buildings and other items or objects (e.g., lamp posts, crosswalks, curbs, etc.), the location and direction of lanes or lane segments (e.g., the location and direction of parking lanes, turning lanes, bicycle lanes, or other lanes within a particular road), traffic control data (e.g., the location and indication of signs, traffic lights, or other traffic control devices), and / or any other map data that provides information to assist the ADAS system 90 in grasping and perceiving the vehicle's surrounding environment.

[0035] In one example, the perception module 109 may be configured to determine a state for one or more of the objects in the vehicle's surrounding environment. In one example, the state generally describes the current state (or features) of one or more objects. In one example, the state of each object may describe an estimate of each object's current location (or position), each object's current speed (or velocity), each object's current acceleration, each object's current direction of travel, each object's current orientation, each object's size / shape / footprint (e.g., represented by a bounding shape such as a bounding polygon or polyhedron), type / class (e.g., vehicle, pedestrian, bicycle, etc.), each object's yaw rate, each object's distance from the vehicle, each object's minimum path to contact with the vehicle, each object's minimum duration to contact with the vehicle, and / or other state information. In another example, the perception module 109 may also be configured to detect object-free areas (e.g., represented by a bounding shape such as a bounding polygon or polyhedron). In another example, the perception module 109 may be configured to update the state information for each object over time. Thus, the perception module 109 may detect and track objects, such as other vehicles, in the vicinity of the own vehicle over time.

[0036] In one example, the perception module 109 may comprise several modules, including but not limited to an object free area (OFA) module (or circuit), a target tracking (TT) module (or circuit), and a static perception (SP) module (or circuit). In another example, the perception module 109 may also comprise a road estimation and electronic horizon reconstruction module (not shown), which may be used to generate self-generated map information from on-board sensor-based information. In one example, the object free area module may be configured to detect areas without objects. In one example, the object free area module may have a polygon output, which may present a bounding shape, such as a bounding polygon or polyhedron, representing each object free area. In one example, the target tracking module may be configured to detect and track objects, such as other vehicles, in the vicinity of the ego-vehicle over time. The target tracking module may have an output, which may present a target tracking output. In one example, the polygon output of the OFA module and the target tracking output of the target tracking module may be presented to an input of the static perception module. The static perception module may be configured to generate static and dynamic object report signals that are presented to the awareness checker 114 in response to the polygon output received from the OFA module and the target tracking output received from the target tracking module.

[0037] For motion assessment, the static perception module may use object information from the target tracking output of the target tracking module in combination with analysis of the object free area (OFA) polygon output of the OFA module to provide traffic direction and detection confidence. For detection of landmarks, traffic signs and traffic lights, pedestrians, and other objects, the static perception module generally utilizes object information from the target tracking output in combination with analysis of the OFA polygon output to provide intersection type, object type, object location, landmark location, and intersection / object / landmark confidence.

[0038] The attention checker 114 may be configured to generate a signal "driver attention status" based on the static and dynamic object report signals received from the static perception module of the perception module 109, the landmark and traffic sign information received from the map interface 105, the weather information received from the circuit 218, the driver attention level received from the attention monitor 110, and the signal received from the vehicle platform 92. The attention checker 114 may be configured to store the signal "driver attention status" for use by other modules of the function control module 100. In one example, the attention monitor 110 may subscribe to the signal "driver attention status" for use in determining whether to escalate the driver's attention level. In another example, the HMIM 112 may subscribe to the signal "driver attention status" for use in determining whether the driver is paying sufficient attention.

[0039] 3, a diagram illustrating an implementation of a system 100 according to an exemplary embodiment of the present invention is shown. In one example, the apparatus (or system) 100 may be mounted entirely or at least partially within a vehicle 50. In one example, the system (or apparatus) 100 may be implemented as part of an advanced driver assistance system (ADAS) electronic control unit (ECU) 90. In various embodiments, the system 100 implementing a driver attention estimator (DAE) 102 may be implemented within the ADAS ECU 90 of the vehicle 50. The ADAS ECU 90 may be connected to a vehicle platform 92 of the vehicle 50. The vehicle 50 may include a driver monitoring system (DMS) 94, a human machine interface (HMI) 96, a forward looking camera (FLC) 220, a number of corner radar sensors 222a-d, a number of front side radar sensors (not shown), a forward looking radar (FLR) sensor 224, a high definition (HD) map receiver 230, a global navigation satellite system (GNSS) receiver 232, and an inertial measurement unit (IMU) 234. In some embodiments, the vehicle 50 may also include LIDAR and / or sonar sensors (not shown).

[0040] The forward looking camera (FLC) 220 is generally used to detect and identify objects and road features in front of the vehicle 50. In one example, the forward looking camera (FLC) 220 may be configured to provide stereoscopic vision with a 100 degree field of view (FOV). In one example, the forward looking camera (FLC) 220 may be used to detect road markings (e.g., lane markings, etc.), road signs, traffic lights, structures, etc. The corner radar sensors 222a-d and the forward looking radar (FLR) sensor 224 (as well as LIDAR and / or sonar sensors, when present) are generally used to detect and track objects. In one example, each of the corner radar sensors 222a-d may have a 140 degree FOV. In one example, the forward looking radar sensor (FLR) 224 may have two FOVs: an 18 degree FOV for long-range sensing and a 90 degree FOV for short-range sensing. The IMU 234 generally reports the orientation, angular velocity and acceleration, as well as the forces acting on the vehicle 50.

[0041] In one example, the DMS 94, the HD map receiver 230, the GNSS receiver 232, the FLC 220, the FCRs 222a-222b, and the FLR 224 may be connected to the system 90. In one example, the DMS 94, the HD map receiver 230, the GNSS receiver 232, the FLC 220, the FCRs 222a-222b, and the FLR 224 may be connected to the system 90 via one or more vehicle buses of the vehicle 50. In another example, the DMS 94, the HD map receiver 230, the GNSS receiver 232, the FLC 220, the FCRs 222a-222b, and the FLR 224 may be connected to the system 90 via a wireless protocol. In one example, the DMS 94 may communicate driver attention information to the system 90. The FLC 220 may communicate surrounding road information (e.g., lane width, marker type, lane marker crossing indications, and video) to the system 90. The GNSS receiver 232 may communicate position data (e.g., latitude values, longitude values, adjustment information, and reliability information) to the system 90. The HD map receiver 230 may transfer map data to the system 90.

[0042] The FLC 220 may implement an optical sensor. In various embodiments, the FLC 220 may be an optical camera. The FLC 220 generally operates to provide information (or image data) of the surrounding road to the system 90. The road information may include, but is not limited to, lane width data, marker type data, lane change indicators, and video of the road ahead of the vehicle 50 within the field of view of the FLC 220. In various embodiments, the FLC 220 may be a color camera. Color may be useful to distinguish between a solid yellow lane marker (e.g., the left-most lane marker) and a solid white lane marker (e.g., the right-most lane marker). In various embodiments, the FLC 220 may provide an estimated lane width for at least the current lane in the center of the field of view of the FLC 220. In some embodiments, the FLC 220 may provide an estimated lane width for the lane or lanes adjacent to the center lane. In other embodiments, the FLC 220 may provide an estimated lane width for all lanes within the field of view of the FLC 220. Lane width may be determined using standard image recognition and standard analysis methods implemented in the FLC 220. The FLC 220 may also identify all lane markers within the field of view of the FLC 220. When the FLC 220 crosses a lane marker, the FLC 220 may notify the system 90 that a lane change is occurring. Identification of lane markers and lane changes may be determined using standard image recognition and standard analysis methods implemented in the FLC 220. The FLC 220 may transfer road information to the system 90 via a vehicle bus or wireless protocol.

[0043] One or more other types of sensors may be used with the FLC 220. Exemplary sensors may include, but are not limited to, radar sensors, light detection and ranging (LiDAR) sensors, inertial sensors, thermal imaging sensors, and / or acoustic sensors. Some of the sensors may detect objects on the side of the road to provide an estimate of the left and right boundaries of the road. From the left and right boundaries, the width of the road may be calculated. From the calculated width, an estimate of how many lanes likely fit within the width may be made based on standard lane widths. The sensors may then estimate the current lane that the vehicle 50 is occupying based on the relative distance from the sensors on the vehicle 50 to the left and right boundaries of the road and the estimated lane number. Lane crossings may be determined by the sensors based on the estimated lane number and the change in relative distance to the left and / or right boundaries.

[0044] The system 90 may implement control circuitry (e.g., an electronic control unit). The system 90 generally operates to keep track of the current lane occupied by the vehicle 50 and to correct the current position of the vehicle 50 to the center of the current lane. The tracking may be based on satellite position data received at the GNSS receiver 232, map data received from the HD map receiver 230, visual detections from the FLC 220 and road information received at the FCRs 222a-222b and the FLR 224. The satellite position data may include adjustment values ​​and corresponding confidence values.

[0045] The GNSS receiver 232 may implement a satellite navigation device. In various embodiments, the GNSS receiver 232 may include a Global Positioning System (GPS) receiver. Other types of satellite navigation devices may be implemented to meet the design criteria of a particular application. The GNSS receiver 232 generally operates to provide latitude and longitude data for the vehicle 50 based on GNSS signals received from a number of satellites. The GNSS receiver 232 may also operate to adjust the latitude and longitude data based on adjustment values ​​and corresponding confidence values ​​received from the system 90. The confidence values ​​may range from 0 (e.g., unreliable) to 1 (e.g., reliable). If the confidence value is above a high threshold (e.g., greater than 0.7), the GNSS receiver 232 may correct the latitude and longitude data according to the adjustment value. If the confidence value is below a low threshold (e.g., less than 0.3), the GNSS receiver 232 may ignore the adjustment value. If the confidence value is between the high and low thresholds, the GNSS receiver 232 may apply a correction to both the latitude and longitude data that is linearly weighted based on the confidence.

[0046] The HD map receiver 230 may implement a radio frequency receiver. The HD map receiver 230 may operate to receive map data from an antenna (not shown). The map data may be converted to a digital format and presented to the system 90.

[0047] 4-12, diagrams illustrating exemplary interactions between a driver and a driver awareness check system according to an embodiment of the present invention are shown. In one example, the AC 114 may be configured as described above with respect to FIGS. 1 and 2. In one example, the AC 114 may be configured to monitor the output of the attention monitor 110. In one example, the AC 114 may be configured to provide a driver attention-on-task assessment that may be utilized by the attention monitor 110 and / or the HMIM 112 to determine when to transfer control to the driver and cause the vehicle to enter a safe shutdown mode in response to the driver not being sufficiently aware to safely supervise active automation features.

[0048] 4, a diagram is shown illustrating the awareness checker (AC) 114 when an automation feature is activated. A pictorial representation 500a is shown illustrating a view inside the cockpit (or cabin or passenger compartment) of a vehicle from the driver's perspective. In one example, the driver activates an automation feature (e.g., by pressing a button on the steering wheel hub, dashboard, etc.). In response to the automation feature being activated, the AC 114 evaluates the driver's state (e.g., using a signal "awareness level").

[0049] Referring to FIG. 5, a diagram showing the AC 114 when an automation feature is activated is shown. A picture 500b is shown showing a view in the cockpit of the vehicle from the driver's perspective. The driver is driving hands-free on a highway with the ADAS feature still active. When the awareness checker 114 determines that the driver is paying attention, the awareness checker 114 signals the virtual assistant to reflect a happy state. In one example, the reflection of happiness can be implemented as a smiling avatar on the display of the HMI 96. In another example, the happy state can be indicated by the color of the avatar (e.g., green for happy). In another example, the happy state can be displayed throughout the cabin of the vehicle using ambient light, audio, etc.

[0050] Referring to FIG. 6, a diagram is shown showing the AC 114 actively checking the driver's awareness by positive reinforcement. A picture 500c is shown showing a view inside the cockpit of the vehicle from the driver's point of view. In one example, the awareness checker 114 can signal the virtual assistant to provide positive reinforcement (e.g., with a compliment). In response, the virtual assistant can use the vehicle's audio system speaker to give the driver a message such as "I love that you keep your eyes on the road and keep us safe!" or "You're good at keeping a safe distance from other cars." Other messages may be implemented as appropriate.

[0051] Referring to FIG. 7, a diagram showing the AC 114 actively checking the driver's awareness is shown. A picture 500d showing a view inside the vehicle's cockpit from the driver's perspective is shown. In one example, the awareness checker 114 may periodically signal the virtual assistant to ask questions about the vehicle's environment. In response, the virtual assistant may use the vehicle's audio system speakers to ask questions such as "What color is the car in front?"

[0052] Referring to FIG. 8, a diagram is shown illustrating the AC 114 evaluating the driver's awareness based on the driver's response to a question. A picture 500e is shown showing a view inside the vehicle's cockpit from the driver's perspective. In one example, the driver correctly answers a question presented by a virtual assistant. In one example, the virtual assistant may reflect happiness (e.g., by continuing to present a smiling avatar face on the display) for answering the question correctly.

[0053] Referring to FIG. 9, a diagram is shown illustrating the AC 114 evaluating the driver's level of awareness based on the driver's response to a question. A picture 500f is shown showing a view inside the cockpit of the vehicle from the driver's perspective. In one example, the driver may answer a question presented by the virtual assistant incorrectly. In one example, when the driver answers a question incorrectly, the virtual assistant may reflect a state of worry about answering the question incorrectly by showing an alert state. In one example, the alert state may be shown by presenting an avatar with a worried facial expression. In another example, the alert state may be shown by changing the color of the avatar (e.g., to yellow). In one example, the virtual assistant may use the vehicle's audio system speakers to ask the driver follow-up questions such as "Are you sure?" or "Are you looking at the car in front?"

[0054] Referring to FIG. 10, a diagram illustrating the AC 114 detecting that the driver is inattentive is shown. Picture 500g is shown showing a view in the cockpit of the vehicle from the driver's perspective. In one example, the driver may be driving hands-free on a highway with the ADAS features still active. In one example, the driver may check a text message and become inattentive. In picture 500h, a display on the dashboard that is part of the HMI 96 may alert the driver to pay attention to the road. In response to the driver being inattentive, the virtual assistant may reflect an alert state.

[0055] Referring to FIG. 11, a diagram is shown illustrating the AC 114 detecting that the driver is inattentive. Picture 500h is shown showing a view inside the cockpit of the vehicle from the driver's perspective. In picture 500h, a display on the dashboard that is part of the HMI 96 warns the driver to pay attention to the road. In one example, the AC 114 may work with the HMIM 112 to return the driver's attention to the task at hand. In one example, the virtual assistant may reflect the alertness request state presented via the HMI 96 with the display. In one example, the virtual assistant may also give the driver a message to "Watch the road!" using the vehicle's audio system speakers.

[0056] Referring to FIG. 12, a diagram showing the AC 114 after determining that the driver continues to be distracted is shown. A picture 500i is shown showing a view inside the vehicle's cockpit from the driver's perspective. In one example, a display on the dashboard that is part of the HMI 96 can be used by the attention monitor 110 and / or the HMIM 112 to alert the driver to take control of the vehicle. In an embodiment in which the awareness checker 114 works in cooperation with the attention monitor 110 and / or the HMIM 112, the virtual assistant can reflect the deactivation request presented by the HMI 96. In one example, the virtual assistant can reflect concern and use the vehicle's audio system speaker to give the driver the message "Take control!" The deactivation request state can also be indicated by changing the color of the personal assistance avatar (e.g., to red). The vehicle can begin to decelerate to 5 kph.

[0057] In general, AC 114 may provide notification of the driver's task attention status by using the signal "driver task attention status." If AC 114 determines that the driver is paying attention, AC 114 may set the signal "driver task attention status" to reflect the driver's level of awareness as happy. If AC 114 determines that the driver does not appear to be aware of the supervisory task, AC 114 may provide notification by setting the signal "driver task attention status" to a state other than happy. In some embodiments, AC 114 may implement an escalation scheme (e.g., happy (green), worried (yellow), increasing worried (red), etc.) in the levels presented by the signal "driver task attention status."

[0058] In some embodiments, the attention monitor 110 and / or the HMIM 112 may utilize the signal "driver attention status" in assessing the sufficiency of the level of awareness exhibited by the driver. In one example, the attention monitor 110 may be configured to transition between an awareness state and a temporary non-aware state based on the signal "driver attention status". In one example, the HMIM 112 may track transitions in the level of the signal "driver attention status" and make an assessment based on respective thresholds. The attention monitor 110 and / or the HMIM 112 may then initiate a request for the driver to take over operation of the vehicle when the awareness (or attention) level is deemed insufficient for a particular automation feature, and safely abort the automation feature. Since aborting an automation feature activity at high speeds may similarly be unsafe, the HMI 96 may be used to present a request for the driver to take over operation of the vehicle, especially when the driver is known to be non-aware. In one example, the attentiveness monitor 110 and / or HMIM 112 may facilitate a safe transition by slowly ramping down the operation of the automation features to a safe state (e.g., slowing to a safe speed and / or safe stop) before ceasing the coordinated operation entirely. However, other strategies for ceasing coordinated operation may be implemented to meet the design criteria of a particular situation or application.

[0059] Referring to FIG. 13, a flow diagram illustrating an exemplary awareness check process according to an embodiment of the present invention is shown. In one example, the method (or process) 600 may be implemented to proactively maintain the driver in a state of continuous awareness. In one example, the method 600 may be executed by the awareness checker 114. In one example, the method 600 may use positive reinforcement or reward techniques. In one example, the method 600 may include a step (or state) 602, a step (or state) 604, a decision step (or state) 606, a step (or state) 608, a step (or state) 610, a step (or state) 612, a step (or state) 614, a decision step (or state) 616, and a step (or state) 618. The method 600 may start at step 602 and proceed to a decision step 604.

[0060] At decision step 604, method 600 may determine whether an automation feature (e.g., autopilot, adaptive cruise control, hyper traffic jam assistance, etc.) is activated. In one example, method 600 may subscribe to a signal "awareness level" to determine whether the driver is deemed to be in an aware state by attention monitor 110. If an automation feature is not activated or the driver is not deemed to be in an aware state, method 600 may remain at decision step 604. If an automation feature is activated and the driver is deemed to be in an aware state, method 600 may proceed to step 606.

[0061] At step 606, the method 600 may obtain environmental information regarding the vehicle's surroundings and conditions within the vehicle. In one example, the method 600 may utilize information regarding the external world in the vehicle's vicinity as perceived by environmental sensors (e.g., cameras, radar, wireless communication transceivers, inertial measurement units, high-resolution maps, satellite signal receivers, etc.) and / or the status of the vehicle within the cabin as perceived by in-cabin sensors (e.g., cameras, radar, HVAC sensors, etc.). In one example, the environmental information may be stored in a memory of an electronic control unit (ECU) for reference as needed. In one example, obtaining the environmental information may be a continuous process while the automation feature is active.

[0062] In step 608, the method 600 may be configured to keep the driver "aware" by asking the driver questions regarding the external world in the vicinity of the vehicle as perceived by environmental sensors, and / or the status of the vehicle within the cabin as perceived by in-cabin sensors. In one example, the method 600 may determine the specific content of the questions based on information available from various sensors (e.g., from the vehicle platform 92 and the environmental monitoring system 98). In one example, the method 600 may be configured to dynamically change the questions asked based on the available information. For example, the awareness checker 114 may not ask about weather conditions unless the awareness checker 114 has environmental information (e.g., from the Internet, a camera with a view of the exterior of the vehicle, etc.) regarding the current weather conditions. If the vehicle does not have a connection to the Internet, the awareness checker 114 may ask a question related to roadside objects or traffic signs, which may be verified against information provided by a perception or location determination module of the feature control module 100. Once the method 600 selects the question to ask the driver, the method 600 may proceed to step 610 .

[0063] In step 610, the method 600 may present the selected question and wait for a response from the driver. In one example, the awareness checker 114 may be configured to utilize the vehicle's human machine interface (HMI) 96 to communicate the question to the driver and receive the answer to the question from the driver. In one example, the awareness checker 114 may be configured to ask the question using a speaker in the vehicle's radio or infotainment system. In one example, the awareness checker 114 may be configured to receive the answer to the question using a microphone in the vehicle platform or in the vehicle's infotainment system. Once the answer to the question is received, the method 600 may proceed to decision step 612.

[0064] At decision step 612, method 600 may determine whether the answer received from the driver matches the information about the external world in the vicinity of the vehicle as perceived by the environmental sensors and / or the status of the vehicle in the cabin as perceived by the in-cabin sensors. When the answer from the driver approximately matches the information about the external world in the vicinity of the vehicle and / or the status of the vehicle in the cabin, method 600 may return to step 604. When the answer from the driver does not match the information about the external world in the vicinity of the vehicle and / or the status of the vehicle in the cabin, method 600 may proceed to step 614.

[0065] In various embodiments, the awareness checker 114 may be configured to generally use positive reinforcement or reward techniques rather than punishment. In step 614, the method 600 may calmly ask the driver a follow-up question (e.g., to confirm the answer) rather than reacting negatively to an answer that is perceived to be incorrect (e.g., deactivating a feature, correcting the driver's mistake, etc.). Once a response is received from the driver, the method 600 may proceed to decision step 616.

[0066] In decision step 616, method 600 may determine whether the follow-up answer received from the driver is consistent with the information about the outside world in the vicinity of the vehicle as perceived by the environmental sensors and / or the status of the vehicle in the cabin as perceived by the in-cabin sensors. When the answer from the driver is approximately consistent with the information about the outside world in the vicinity of the vehicle and / or the status of the vehicle in the cabin, method 600 may return to step 604. When the answer from the driver is still inconsistent with the information about the outside world in the vicinity of the vehicle and / or the status of the vehicle in the cabin, method 600 may proceed to step 618. In step 618, method 600 may change the value of the task focus status maintained by attention checker 114 according to a defined escalation scheme. Process 600 may then return to step 604.

[0067] Depending on the configuration of the attention monitor 110 and / or the human-machine interaction monitor (HMIM) 112, changes in the value of the task focus status may be used by the attention monitor 110 and / or the HMIM 112 (e.g., according to the respective escalation schemes of the attention monitor 110 and the HMIM 112) and may be used to affect the kinematics and longitudinal control of the vehicle platform 92.

[0068] Referring to FIG. 14, a diagram illustrating an electronic control module implementing an advanced driver assistance system (ADAS) feature control system according to an exemplary embodiment of the present invention is shown. In one example, the device 800 may implement an electronic control unit or module (ECU). In one example, the electronic control module (ECU) 800 may be implemented as a domain controller (DC). In another example, the ECU 800 may be implemented as an active safety domain master (ASDM). In various embodiments, the ECU 800 may be configured to control activation of one or more features (or functions) of an ADAS component of a vehicle. In various embodiments, the function control 100 may be implemented within the ECU 800.

[0069] In one example, the ECU 800 may be connected to the vehicle platform 92, the driver monitoring system (DMS) 94, the human machine interface (HMI) 96, and the environmental monitoring system (EMS) 98. In one example, the environmental monitoring system (EMS) 98 may include an HD map receiver 210, a GNSS receiver 212, a camera 214, a radar sensor 216, a wireless communication transceiver 218, an inertial monitoring unit (IMU) 230, and a vehicle electronic bus 802. In one example, the ECU 800 may be configured to (i) receive signals “vehicle speed”, “deactivation request”, “activation request”, “driver response”, and “driver information” from the vehicle systems, and (ii) communicate signals “deceleration request”, “driver question”, “driver task concentration status”, and “driver alert” to the vehicle systems.

[0070] In one example, the ECU 800 may be connected to a block (or circuit) 802. The circuit 802 may implement an electronic bus of the vehicle. The electronic bus 802 may be configured to transfer data between the ECU 800 and the vehicle platform 92, the DMS 94, the HMI 96, the HD map receiver 210, the GNSS receiver 212, the cameras 214 (e.g., the forward looking camera (FLC) 220 and corner view cameras, side view cameras and / or rear view cameras), the radar 216 (e.g., the corner / side radar sensors 222a-222n, the forward looking radar (FLR) sensor 224, etc.), the wireless communication transceiver 218, and the inertial measurement unit 230. In some embodiments, the electronic bus 802 may be implemented as a vehicle Controller Area Network (CAN) bus. The electronic bus 802 may be implemented as an electronic wired and / or wireless network (e.g., Wi-Fi, BLUETOOTH, ZIGBEE, etc.). In general, the electronic bus 802 may connect one or more components of the vehicle 50 to enable sharing of information in the form of digital signals (e.g., a serial bus, an electronic bus connected by wiring and / or interfaces, a wireless interface, etc.).

[0071] ECU 800 generally comprises block (or circuit) 820, block (or circuit) 822, and block (or circuit) 824. Circuit 820 may implement a processor. Circuit 822 may implement a communication port. Circuit 824 may implement a memory. Other blocks (not shown) may be implemented (e.g., filters, clocks / timers, I / O ports, power connectors, interfaces, etc.). The number and / or types of circuits implemented by module 800 may vary according to the design criteria of a particular implementation.

[0072] The processor 820 may be implemented as a microcontroller, a multi-threaded microprocessor, or any combination thereof. The processor 820 may comprise a block (or circuit) implementing the location determination module 103, a block (or circuit) implementing the mode management 104, a block (or circuit) implementing the perception module 109, a block (or circuit) implementing the attention monitor 110, a block (or circuit) implementing the human-machine interaction monitor 112, a block (or circuit) implementing the awareness checker 114, a block (or circuit) implementing the GNSS module 850, and / or a block (or circuit) implementing the map module 852. In one example, the map module 852 may implement the map interface 105. The processor 820 may comprise other components such as filters and clocks (not shown). In some embodiments, the processor 820 may be a combination (e.g., integrated) chipset that implements the processing functionality.

[0073] In some embodiments, the processor 820 may be comprised of several separate circuits (e.g., a microcontroller, a multi-threaded microprocessor, a digital signal processor (DSP), a graphics processing unit (GPU), etc.). The design of the processor 820 and / or the functionality of the various components of the processor 820 may vary according to the design criteria of a particular implementation. The processor 820 is shown to send and receive data to and from the vehicle platform 92, a communication port 822, and / or a memory 824. However, the processor 820 may be configured to implement additional data and / or control paths to meet the design criteria of a particular application.

[0074] In one example, the memory 824 may include a block (or circuit) 860 and a block (or circuit) 862. The block 860 may store data utilized by the driver awareness (or attention) estimator (DAE) 102 and the mode manager 104. The block 862 may store computer readable instructions (e.g., instructions readable by the processor 820). In one example, the DAE data 860 may store various data sets 870a-870n. In one example, the data set 870a may be utilized by the attention monitor 110 to store awareness states (or levels). For example, the data set 870a may track the progress of transitions to a temporarily unaware state, an unaware state, and an out-of-loop state. In one example, the data set 870b may be utilized by the human-machine interaction monitor (HMIM) 112 to store attention states (or levels), long-term gaze distribution, counts of transitions to a temporarily unaware state, counts of transitions to an unaware state, counts of transitions to an out-of-loop state, etc. In one example, dataset 870c may be utilized by the attention checker 114 to store a driver's task attention status. In one example, dataset 870d may be utilized by the attention checker 114 to store a lookup table or database of questions that may be used to query the driver. In one example, dataset 870e may be utilized to store driver information that may be used by the HMIM 112 and the attention checker 114. The memory 824 may also be configured to store other datasets 870n (e.g., environmental data, etc.).

[0075] In one example, the other data sets 870n may include parameters (e.g., coefficients) and / or calibration data used to convert data received from various sensors (e.g., FLC, FLR, FCR, FCS, IMU, etc.) of the vehicle. In one example, the calibration data 870n may provide many coefficient sets (e.g., one coefficient set for each of the sensors). The calibration data 870n may be updatable. For example, the calibration data 870n may store current values ​​as coefficients for the sensors, and as data from the sensors drift, the module 800 may update the calibration data 870n to maintain accuracy. The format of the calibration data 870n may vary based on the design criteria of a particular implementation.

[0076] Various other types of data may be stored in the data set 870n as part of the DAE data 860. For example, the other data 870n may store gaze distributions of multiple drivers. For example, the other data 870n may store past data values ​​of the calibration data and / or current data values ​​of the calibration data. The past data values ​​and current data values ​​of the calibration data may be compared to determine trends that are used to extrapolate and / or predict potential future values ​​of the calibration data.

[0077] The processor 820 may be configured to execute stored computer-readable instructions (e.g., instructions 862 stored in memory 824). The processor 820 may perform one or more steps based on the stored instructions 862. In an example, the steps of the instructions 862 may be executed / performed by the processor 820 to implement one or more of the attention monitor 110, the human-machine interaction monitor 112, the awareness checker 114, the location determination module 103, the map interface 105, the perception module 109, and the mode manager 104. The order of the instructions executed and / or performed by the processor 820 of the instructions 862 may be varied according to design criteria of a particular implementation.

[0078] The communication port 822 may enable the module 800 to communicate with external devices, such as the vehicle platform 92, the driver monitoring system 94, the human machine interface 96, and the environmental monitoring system 98. For example, the module 800 is shown connected to an external electronic bus 802. In one example, information from the module 800 may be communicated to an infotainment device for display and / or presentation to the driver. In another example, a wireless connection (e.g., Wi-Fi, BLUETOOTH, cellular, etc.) to a portable computing device (e.g., a smartphone, a tablet computer, a notebook computer, a smartwatch, etc.) may enable information from the module 800 to be displayed and / or otherwise communicated (e.g., audibly, etc.) to a user.

[0079] In some embodiments, the ECU 800 may further comprise a filter and a clock. The filter may be configured to perform a linear quadratic estimation. For example, the filter may implement a Kalman filter. In general, the filter may operate recursively on the input data to generate a statistically optimal estimate. For example, the filter may be used to calculate position coordinates and / or estimate the accuracy of the position coordinates utilized by the GNSS module 850 and the map module 852. In some embodiments, the filter may be implemented as a separate module. In some embodiments, the filter may be implemented as part of the memory 824 (e.g., stored instructions 862). The implementation of the filter may vary according to the design criteria of a particular implementation. The clock may be configured to determine and / or track time. The time determined by the clock may be stored in the other data 870n as a timestamp. In some embodiments, the clock may be configured to compare timestamps received from the GNSS receiver 212.

[0080] The module 800 may be configured as a chipset, a system on chip (SoC), and / or a discrete device. For example, the module 800 may be implemented as an electronic control unit (ECU). In some embodiments, the module 800 may be configured to control activation of one or more ADAS features / functions.

[0081] Considering that there is no state-of-the-art ASIL for HMI warning messages, the underlying objective of HMIM 112 and AC 114 according to embodiments of the present invention is to provide monitoring functionality to ensure sufficient controllability of the supervising driver for possible hazardous events. In one example, HMIM 112 and AC 114 may achieve sufficient controllability by ensuring driver engagement. In one example, HMIM 112 and AC 114 may ensure driver engagement by monitoring gaze distribution patterns and the driver's task concentration status as a measure of attention. In various embodiments, several exemplary functionality iterations of HMIM 112 and AC 114 may be implemented to mitigate false positives during decision making.

[0082] In one example, the HMIM 112 may check the delta change in eye gaze shift before and after an HMI alert to determine if the alert is communicated to the driver and prevent HMI messages from being missed. For example, the HMIM 112 may subscribe to a real-time "eyes on the road" signal from a camera of the driver monitoring system 94. The "eyes on the road" signal may be used as feedback to quickly assess whether there is an improvement in gaze distribution after each escalation alert. If there is no improvement, the HMIM 112 may fail-safe appropriately.

[0083] In another example, to avoid false positives from the DMS 94 camera and attention monitor attention level, the HMIM 112 may attempt to intentionally divert the driver's eyes temporarily from the road when the attention monitor 110 reports that the driver is "aware" for a longer duration than expected. When the driver has been reported as "aware" for a longer duration, the HMIM 112 may send a directional prompt to divert the driver's attention from the road (when it is determined that it is safe to do so, e.g., by subscribing to environmental information from on-board sensors, GNSS, HD maps, etc.) and verify whether the front-end signal chain (e.g., DMS 94, attention monitor 110, attention checker 114, etc.) detects that attention has been diverted. If not, the HMIM 112 may fail-safe appropriately.

[0084] In another example, other cabin sensory information may be integrated as inputs into the HMIM 112 and AC 114 to form a holistic driver state estimation. In addition to eyes on the road, the HMIM 112 and AC 114 may subscribe to hands on the steering wheel, pedal information, seat sensors, seat belt status, etc. to form a holistic driver state estimation model. The HMIM 112 and AC 114 may leverage feedback from each of these inputs to detect and mitigate inattention.

[0085] In yet another example, the HMIM 112 and AC 114 may be developed using artificial intelligence / machine learning (AI / ML) based non-deterministic algorithms to baseline a driver attention profile for each individual driver and track the inattention of the particular driver against the baseline. In various embodiments, each vehicle may implement a generic HMIM 112 and AC 114, which may customize itself over time by tracking and learning the inattention profile of the same driver by baselined the driver's inattention profile against the driver's gaze distribution during manual driving. Inattention during supervised driving may subsequently be flagged if the inattention exceeds the threshold noted during previous manual driving.

[0086] The terms "may" and "generally" used in conjunction with the verb "is (are)" herein are intended to convey the intent that the description is illustrative and are considered to be broad enough to encompass both the specific examples presented in this disclosure and alternatives that may be derived based on this disclosure. The terms "may" and "generally" used herein should not be construed as necessarily implying that it is desirable or possible to omit the corresponding element.

[0087] The designation of various components, modules, and / or circuits as "a" through "n," as used herein, discloses either a single component, module, and / or circuit, or a plurality of such components, modules, and / or circuits, with the designation "n" being applied to mean any particular integer. Different components, modules, and / or circuits, each having instances (or occurrences) designated "a" through "n," may indicate that the different components, modules, and / or circuits may have a matching number of instances or a different number of instances. An instance designated "a" may represent a first instance of a plurality of instances, and instance "n" may refer to a last instance of a plurality of instances, but does not imply a particular number of instances.

[0088] Although the present invention has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention.

Claims

1. 1. An apparatus comprising: an interface configured to (i) receive sensor-based information from a plurality of sensor signals from a vehicle platform of a vehicle and environmental information regarding an environment of the vehicle, and (ii) present one or more control signals to the vehicle platform; a control circuit configured to: (i) detect whether a driver's attention state is attentive or inattentive in response to one or more of the plurality of sensor signals from the vehicle platform and the driver monitoring system; (ii) assess whether the driver is sufficiently attentive by monitoring the one or more of the plurality of sensor signals from the vehicle platform and the driver monitoring system; and (iii) maintain the driver's awareness of the vehicle's environment by actively interacting with the driver to determine whether the driver's perception of the vehicle's environment corresponds to the environmental information and the sensor-based information received by the interface or to alert the driver. An apparatus comprising:

2. the sensor-based information includes one or more of radar detection, visual detection, vehicle speed, ignition status, window and door lock status, tire pressure, HVAC status, and line-of-sight status of the driver; the environmental information includes one or more of objects and other vehicles present around the vehicle, types of objects and other vehicles around the vehicle, landmarks, traffic signals, road signs, and weather conditions; 10. The apparatus of claim 1.

3. 2. The apparatus of claim 1, wherein the control circuit comprises a driver attention estimator configured to (i) generate a first control signal that communicates the driver's state of attention, and (ii) generate a second control signal that communicates an assessment of whether the driver is paying sufficient attention to an automation system function to continue to operate the vehicle safely.

4. The driver attention estimator an attentiveness monitor configured to determine the attentiveness state of the driver during a first window having a first duration and to generate the first control signal; a human-machine interaction monitor configured to generate the assessment of whether the driver is paying sufficient attention during a second window having a second duration and to generate the second control signal; an awareness checker configured to determine whether the driver's perception of the environment of the vehicle corresponds to the environmental information and the sensor-based information and to generate a third control signal communicating an assessment of the driver's awareness state; and The apparatus of claim 3 , comprising:

5. The apparatus of claim 3 , wherein the awareness checker is further configured to communicate with the driver about the environment of the vehicle.

6. The awareness checker querying the driver about the perception of the environment of the vehicle; The apparatus of claim 5 , further configured to compare the response received from the driver with the environmental information and the sensor-based information received by the interface.

7. 5. The apparatus of claim 4, wherein the human-machine interaction monitor is further configured to monitor a plurality of driver attention states of the attentiveness monitor and the attention status of the driver provided by the attention checker.

8. the plurality of driver attention states of the attention monitor include one or more of an aware state, a temporarily unaware state, an unaware state, and an out-of-loop state; the awareness status communicated by the awareness checker comprises one or more of a satisfied state, an alert state, and a deactivation requested state; 8. The apparatus of claim 7.

9. 9. The apparatus of claim 8, wherein the human-machine interaction monitor is further configured to use a respective threshold value for each of the plurality of driver attention states of the attentiveness monitor and the attention status communicated by the attention checker.

10. 10. The apparatus of claim 9, wherein the human-machine interaction monitor is further configured to transfer operation of the vehicle to the driver and safely shut down automation system functions of the vehicle when the driver's perception differs from the environmental information and the sensor-based information received by the interface by a predetermined threshold.

11. 10. The apparatus of claim 9, wherein the respective thresholds for each of the plurality of driver attention states of the attention monitor and the attention status communicated by the attention checker are programmable.

12. The apparatus of claim 4 , wherein the awareness checker is further configured to customize itself over time by tracking and learning the driver's awareness profile.

13. 5. The apparatus of claim 4, wherein the control circuitry further comprises a feature mode manager configured to activate or maintain operation of an automation system of the vehicle when the threshold is not exceeded and to safely transfer operation of the vehicle from the automation system to the driver when the threshold is exceeded.

14. 14. The apparatus of claim 13, wherein the feature mode manager is configured to send a warning to the driver to take over control of the vehicle via a human-machine interface of the vehicle and reduce performance of an autopilot function of the automation system by generating a deceleration request to the vehicle platform.

15. 1. A method for controlling automation system functions of a vehicle, comprising: (i) receiving sensor-based information from a plurality of sensor signals from a vehicle platform; and (ii) environmental information relating to an environment of the vehicle; detecting whether a driver's attention state is attentive or inattentive in response to one or more of the plurality of sensor signals from the vehicle platform; assessing whether the driver is paying sufficient attention by monitoring the one or more of the plurality of sensor signals from the vehicle platform; maintaining the driver's awareness of the vehicle's environment by actively interacting with the driver to determine if the driver's perception of the vehicle's environment corresponds to the environmental information and the sensor-based information received by an interface or to alert the driver; A method comprising:

Images