Man-machine dialogue monitor

Abstract

The apparatus includes an interface and a control circuit. The interface may receive sensor signals from a vehicle platform and may present one or more control signals to the vehicle platform. The control circuit may (i) detect whether a driver is in an attentive or inattentive state in response to one or more of the sensor signals from the vehicle platform during a first window having a first duration, (ii) assess whether the driver is sufficiently attentive by monitoring the sensor signals and determining whether a change in the driver's attention state during a second window having a second duration greater than the first duration exceeds a threshold, and (iii) transition operation of the vehicle to the driver and safely suspend automation system functions of the vehicle when the threshold is exceeded.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. patent application Ser. No. 17 / 493,144, filed Oct. 4, 2021, entitled "HUMAN MACHINE INTERACTION MONITOR," which is assigned to the assignee of the present application and is hereby incorporated by reference in its entirety for all purposes.

[0002] The present invention relates generally to advanced driver assistance systems, and more particularly to a method and / or apparatus for implementing a man-machine dialogue monitor. [Background technology]

[0003] Off-the-shelf driver attention tracking features in today's market employ algorithms to track and categorize the driver's visual attention. However, due to numerous human factor-related challenges in engaging a driver in a driving task, it is not always possible to rely solely on the driver attention tracking feature to keep the driver functionally alert (and thereby meet safety goals). Foreseeable misuses, and the types of abuses of similar features documented in today's market, need to be taken into account. Misuses can result in edge cases where the driver attention tracking feature labels the driver as fully conscious, thereby activating automated features while the driver cannot intervene in case of any dangerous events. Thus, there is a need to detect drivers who are no longer attentive enough to act as a safety net for the driver attention tracking feature.

[0004] To ensure driver engagement during supervision of assistive (collaborative) driving automation, it would be desirable to implement man-machine dialogue monitoring. Summary of the Invention [Means for solving the problem]

[0005] At least some embodiments described herein relate to an apparatus including an interface and a control circuit. The interface may be configured to receive a plurality of sensor signals from a vehicle platform of a vehicle and to present one or more control signals to the vehicle platform. The control circuit may be configured to (i) detect whether a driver's attention state is attentive or inattentive in response to one or more of the plurality of sensor signals from the vehicle platform during a first window having a first duration, (ii) assess whether the driver is sufficiently attentive by monitoring one or more of the plurality of sensor signals from the vehicle platform and determining whether a change in the driver's attention state during a second window having a second duration greater than the first duration exceeds a threshold, and (iii) transition operation of the vehicle to the driver and safely suspend automation system functions of the vehicle when the threshold is exceeded.

[0006] Embodiments will become apparent from the following detailed description and the appended claims and drawings. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates a system according to one embodiment of the present disclosure. [Diagram 2] FIG. 1 illustrates an example implementation that utilizes gaze behavior as a measure of driver attentiveness. [Diagram 3] FIG. 2 is a flow diagram illustrating an exemplary operational state of a system according to one embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates an implementation of an advanced driver-assistance system (ADAS) man-machine dialogue monitor according to an exemplary embodiment of the present disclosure. [Diagram 5] FIG. 2 illustrates exemplary criteria for acceptable and unacceptable driver attentiveness. [Figure 6]FIG. 2 illustrates an exemplary operation of a system according to one embodiment of the present disclosure. [Figure 7] FIG. 2 illustrates an exemplary interaction between a driver and a system according to an exemplary embodiment of the present disclosure. [Figure 8] FIG. 2 illustrates an exemplary interaction between a driver and a system according to an exemplary embodiment of the present disclosure. [Figure 9] FIG. 2 illustrates an exemplary interaction between a driver and a system according to an exemplary embodiment of the present disclosure. [Figure 10] FIG. 2 illustrates an exemplary interaction between a driver and a system according to an exemplary embodiment of the present disclosure. [Figure 11] FIG. 2 illustrates an exemplary interaction between a driver and a system according to an exemplary embodiment of the present disclosure. [Figure 12] FIG. 2 illustrates an exemplary interaction between a driver and a system according to an exemplary embodiment of the present disclosure. [Figure 13] FIG. 2 illustrates an exemplary interaction between a driver and a system according to an exemplary embodiment of the present disclosure. [Figure 14] FIG. 1 illustrates an electronic control unit implementing an advanced driver assistance system (ADAS) function control system, according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Exemplary embodiments may (i) provide a driver awareness level elevation regime that generates warnings at several (e.g., three) different levels via the vehicle's human machine interface (HMI); (ii) detect insufficient attention related to short periods of shared visual attention (e.g., from engagement in secondary tasks); (iii) take into account predictable misuse and types of abuse of similar features documented in the market; (iv) detect and mitigate edge cases where the driver is labeled as fully aware, thereby activating automated features while the driver cannot intervene in the event of a dangerous event; and (v) detect and mitigate edge cases where the driver is no longer paying attention, such as when the driver is not aware of the driver's presence or absence of a warning. (vi) may detect insufficient detection of inattention when the HMI is not operational or there is driver misuse; (vii) may track gaze distribution over a longer duration than the duration that the awareness monitor tracks off-road gaze behavior; (viii) may utilize outputs from both man-machine interaction monitoring and awareness monitoring to affect kinesthetic and longitudinal control of the vehicle; and / or (ix) may be implemented as one or more integrated circuits.

[0009] Partial (collaborative) driving automation features are typically subject to safety objectives (e.g., Society of Automotive Engineers Level 2-3 (SAE L2+)) so that they are not operable unless the driver is attentive. An attentive driver can intervene in time to mitigate dangerous events that may occur due to functional limitations. With SAE L2+ automated systems that assume lateral and longitudinal steering functions, driver inattention is a major concern today. The safety case relies on a sufficiently attentive driver being able to take control in dangerous situations that reach limitations in the automated system's operational capabilities. There is an ever-growing need to design safe and intelligent collaborative driving systems to provide the necessary user value through automation, while still accommodating the user to adequate attention compliance. Thus, there is a need for a solution to systematically detect whether the driver is attentive and to reach an appropriate safety state when the driver is no longer attentive.

[0010] To best achieve attentional compliance, a clear and consistent mental model of the driver needs to be established. The driver's understanding of the capabilities and limitations of an automated system may fully characterize how the driver interacts with the automated system. An accurate representation of the mental model for the driver is critical to (i) how likely the driver will respond to a safety-critical situation that requires timely intervention, (ii) the development of trust in the system, which may lead to over-reliance or under-use, and (iii) the driver's overall conception of the system's operation.

[0011] The extent to which semi-automated systems affect the driver's information processing capabilities and cognitive offloading, including the possibility of the driver engaging in distracting secondary tasks, is a particularly complex concern. As the driver's role shifts from a full-time active operator to an intermittent passive supervisor, system design needs to ensure that the driver can still perceive significant changes in the driving environment and system status.

[0012] In various embodiments, methods and systems are provided for monitoring driver attention, including providing feedback to the driver, evaluating driver behavior over time, and enabling advanced driver assistance system (ADAS) functionality when driver attention is at an acceptable level. In various embodiments, a driver awareness estimator (DAE) may be implemented to monitor the driver's attention level over time and provide feedback (e.g., warnings) to the driver (e.g., using the vehicle's human-machine interface (HMI)). The HMI generally provides a connection between the driver and the vehicle, such that the driver's reactions to warnings may be observed by the driver awareness estimator system over an extended period of time. In one example, a system according to an embodiment of the present disclosure may utilize the observations made using the DAE to improve the quality of the driver attention assessment. In one example, various ADAS functions may be disabled when the driver behavior monitored over the extended period of time exceeds a certain threshold. In various embodiments, the system may provide improved driver attention. The system may also reduce the likelihood that a driver will circumvent (eg, cheat, abuse, etc.) the vehicle's driver monitoring system (DMS).

[0013] In various embodiments, the DAE generally comprises two separate functionalities: an attention (or awareness) monitor and a human machine interaction monitor (HMIM). The attention monitor may function to determine the driver's state (e.g., eyes on the road, inattention, momentary unconsciousness, out of the loop, dozing, etc.) during a short window of time (e.g., several seconds). In one example, the attention monitor may track eye blinks or the line of sight used by the driver, e.g., eye tracking information from a driver monitoring system of the vehicle, and then determine whether the driver is unconscious or inattentive. In one example, the attention monitor may perform eye tracking using a camera of the driver monitoring system. The attention monitor may also check whether a personal device or devices in the vehicle are being used, and then determine whether the driver is inattentive. The attention monitor may further determine the driver's state based on driving information such as speed, steering angle, and variability of the vehicle's speed. In one example, the attention monitor may provide a driver awareness level elevation regime that generates warnings at several (e.g., three) different levels via a human-machine interface (HMI) and may detect insufficient attention related to short periods of shared visual attention (e.g., from engagement in a secondary task). The attention monitor may be implemented similarly to existing attention tracking features on the market today that employ algorithms to track and categorize the driver's visual attention.

[0014] In various embodiments, the DAE does not rely solely on the attention monitor to keep the driver functionally alert (and thereby meet safety goals) due to numerous human factor-related challenges in engaging the driver in the driving task. In various embodiments, the DAE may take into account predictable misuse and the types of abuse of similar features documented in today's market. For example, misuse may result in edge cases where the attention monitor labels the driver as fully aware, thereby activating automated features while the driver cannot intervene in case of any dangerous events. Thus, the DAE according to an embodiment of the present disclosure generally includes additional functionality to systematically detect drivers who are no longer attentive enough to act as a safety net for the attention monitor. The additional functionality is generally provided by a man-machine interaction monitor (HMIM) according to an embodiment of the present disclosure. The HMIM generally provides an additional layer of safety on top of the attention monitor to provide a more multifaceted assessment of driver inattention.

[0015] Referring to FIG. 1, a diagram illustrating a system according to one embodiment of the present disclosure is shown. In one example, the system (or device) 90 may implement an Advanced Driver Assistance System (ADAS). In various embodiments, the system 90 may include a vehicle platform 92, a driver monitoring system (DMS) 94, a man-machine (or vehicle) interface (HMI) 96, and a function control module 100. In various embodiments, the vehicle platform 92, the driver monitoring system (DMS) 94, and the function control module 100 may be implemented as an Automotive Safety Integrity Level (ASIL), and the man-machine interface (HMI) 96 may be implemented as a Quality Management (QM).

[0016] Automotive Safety Integrity Level (ASIL) is a risk classification scheme defined by ISO26262 - Functional Safety for Road Vehicles standard. It is an adaptation of the Safety Integrity Level (SIL) used in IEC61508 for the automotive industry. ASIL classification helps define the safety requirements that are required to be in line with the ISO26262 standard to keep risks at an acceptable level. ASIL is established by performing a risk analysis of a potentially hazardous scenario by looking at the severity, exposure, and controllability of the vehicle operation scenario. The safety goal for that hazardous scenario in turn conveys the ASIL requirement. ASILs range from ASIL D, which represents the highest degree of risk of a hazardous scenario turning into a disaster, and the highest degree of stringency required to be applied in ensuring the resulting safety requirements, to QM, which represents an application that does not involve any automotive hazardous scenarios with unacceptable risks, and therefore no safety requirements to be managed under the ISO26262 safety process. Level QM, for "quality management," means that the risks associated with a hazardous event are not unreasonable and therefore do not require safety measures according to ISO 26262. The intervening levels (ASIL C, ASIL B, and ASIL A) are simply ranges of varying degrees of hazard risk levels and the degree of assurance required.

[0017] The standard defines functional safety as "free from unreasonable risk due to hazards caused by malfunctioning behavior of electrical or electronic systems." ASILs establish safety requirements based on the probability and severity of harm for automotive components to comply with ISO 26262. Systems such as airbags, antilock brakes, and power steering require an ASIL D rating (the highest stringency applicable to safety assurances) because the risks associated with their failures are greatest. At the other end of the safety spectrum, components such as windshield wiper systems only require an ASIL A rating. Headlights and brake lights would generally be ASIL B because rear lights would be at risk of rear-end collisions, while automatic emergency braking systems would generally be ASIL C due to risks associated with unintended deceleration.

[0018] In one example, the vehicle platform 92, the DMS 94, and the HMI 96 may provide input signals to the function control module 100. In one example, the vehicle platform 92 may provide an input signal (e.g., VEHICLE SPEED) communicating vehicle speed to the function control module 100. The DMS 94 may provide an input signal communicating information regarding driver awareness (e.g., driver eye movement, driver hand position, steering angle, etc.). In one example, the HMI 96 may provide a first input signal (e.g., ACTIVATION REQUEST) and a second input signal (e.g., DEACTIVATION REQUEST) to the function control module 100. The signal ACTIVATION REQUEST may communicate a request from the driver to activate an ADAS function controlled by the function control module 100. The signal DEACTIVATION REQUEST may communicate a request from the driver to deactivate an ADAS function controlled by the function control module 100. In some embodiments, the HMI 96 may optionally present input signals (e.g., DRIVER INFO) that communicate information about a particular driver operating the vehicle. In various embodiments, the signals VEHICLE SPEED and DECELERATION REQUEST may be implemented as ASIL, and the signals ACTIVATION REQUEST and DEACTIVATION REQUEST may be implemented as QM.

[0019] In one example, function control module 100 may provide output signals to vehicle platform 92 and HMI 96. In one example, function control module 100 may present an output signal (e.g., DECELERATION REQUEST) to vehicle platform 92. The signal DECELERATION REQUEST may be configured to enable function control module 100 to bring the vehicle to a safety stop. Function control module 100 may present a signal (e.g., DRIVER WARNING) to HMI 96. The signal DRIVER WARNING may communicate information to cause HMI 96 to present a particular warning to the driver. In various embodiments, the signal DRIVER WARNING may be implemented as a QM.

[0020] In one example, the function control module 100 may include a block (or circuit) 102, a block (or circuit) 104, and a block (or circuit) 106. The block 102 may be implemented as an attention (or awareness) monitor. The block 104 may be implemented as a man-machine interaction monitor (HMIM). The block 106 may be implemented as an ADAS function mode manager. In one example, the block 106 may be implemented as an autopilot mode manager. In various embodiments, the blocks 102, 104, and 106 are generally implemented as ASIL. In one example, a signal VEHICLE SPEED may be presented to a first input of the block 102, a first input of the block 104, and a first input of the block 106. A signal from the DMS 94 may be presented to a second input of the block 102 and a second input of the block 104. Block 102 may present a signal (e.g., AWARENESS LEVEL) to a third input of block 104 and a second input of block 106. The signal AWARENESS LEVEL may be implemented as an ASIL. Block 104 may present a signal (e.g., SUFFICIENTLY ATTENTIVE) to a third input of block 106. The signal SUFFICIENTLY ATTENTIVE may be implemented as an ASIL. In embodiments in which HMI 96 provides signal DRIVER INFO to function control module 100, signal DRIVER INFO may be presented to a fourth input of block 104.

[0021] In various embodiments, blocks 102 and 104 may be configured as a driver attention estimator (DAE) to systematically detect whether the driver is attentive and reach an appropriate safety state when the driver is no longer attentive. In one example, attention monitor 102 may provide a driver attention level elevation regime that generates warnings at several (e.g., three) different levels via HMI 96 and may detect insufficient attention related to short-term shared visual attention (e.g., from secondary task engagement). Attention monitor 102 may be implemented similarly to existing off-the-shelf attention tracking functions in the market today that employ algorithms to track and categorize the driver's visual attention.

[0022] In various embodiments, the driver attention estimator (DAE) does not rely solely on the attention monitor 102 to keep the driver functionally alert (and thereby meet safety goals) due to numerous human factor-related challenges in engaging the driver in the driving task. In various embodiments, the driver attention estimator (DAE) may take into account predictable misuse and the types of abuse of similar features documented in the market today. For example, misuse may result in edge cases where the attention monitor 102 labels the driver as fully aware, thereby activating automated features controlled by the feature control module 100 while the driver cannot intervene in case of any dangerous events. Thus, the driver attention estimator (DAE) according to an embodiment of the present disclosure generally utilizes the HMIM 104 to provide additional functionality to systematically detect drivers who are no longer attentive enough to act as a safety net for the attention monitor 102.

[0023] The HMIM 104 is generally configured to detect insufficient attention when the HMI 96 is not activated or there is driver misuse. In one example, the HMIM 104 may look at the driver's off-road gaze distribution pattern by analyzing the toggling behavior between the attention levels reported by the attention monitor 102 over a longer duration than the duration (or window) used by the attention monitor 102. In various embodiments, the long-term gaze distribution pattern may be used to affect the kinesthetic and longitudinal control of the vehicle platform 92. In one example, the HMIM 104 may focus on a longer-term assessment based on the toggling behavior between the attentiveness states reported by the attention monitor 102. By monitoring the driver's attention level (e.g., as captured by the time distribution of the attention states) within a given time window, an adjustable (or programmable) number of transitions and total allowable time within each state of attention may be defined. Using assisted driving (e.g., adaptive cruise control (ACC) etc.) gaze behavior as a criterion, the driver engagement may be calculated based on the gaze distribution pattern. The HMIM 104 generally assesses the driver's longer term gaze patterns and then triggers the transfer of control to the driver and the transition of the vehicle to a safe state (e.g., via signal DECELERATION REQUEST, etc.) to prevent the driver from repeatedly entering lower states of awareness for longer durations that may affect driver controllability.

[0024] Referring to FIG. 2, a diagram is shown illustrating an example implementation of a driver attention estimator that utilizes gaze behavior as a measure of driver attentiveness. The design of increasingly sophisticated supervised systems is complex due to the important role of the human in the loop. A successful design ensures that the driver can engage and take over when needed during warned obstacles, silent obstacles, and other transitions of control. Developing a forced vigilance system involves a deep understanding of user perception, cognition, and response behavior. In one example, the attention monitor 102 generally implements a driver attention level elevation regime that may generate warnings at several different levels via the HMI 96 and detects insufficient attention related to short-term shared visual attention (e.g., from engagement in a secondary task). In one example, the attention monitor 102 may observe the driver gaze behavior 108 over a time window 110 of short duration to determine the attention level. In one example, if for a short period of time (e.g., a few seconds) the driver is looking off the road 50% of the time, or for a longer period of time (e.g., 4-5 times longer than the short period) the driver is looking off the road 30% of the time, the attention monitor 102 may indicate that the driver is not aware. In one example, the attention monitor 102 may generate warnings at three different levels: temporarily unconscious, unconscious, and out of the loop. In one example, the number of warnings may include, but are not limited to, audio and visual reminders, haptic reminders (e.g., seat vibration), hands on, reduced thrust, request to take over, and deceleration to a safety stop.

[0025] In one example, the HMIM 104 may observe the driver gaze behavior 108 over a time window 112 of long duration to determine whether the driver is sufficiently aware. In one example, the HMIM 104 may detect insufficient attention when the HMI 96 is not operating as desired or there is misuse (e.g., assuming the HMI 96 is normally QM) by looking at the gaze distribution off the road for a longer period of time than the attention monitor 102. In one example, the HMIM 104 may be configured to utilize the HMI 96 and the attention monitor 102 to detect whether the HMI 96 is successfully transmitting signals to the driver. For example, a hardware failure or tracking failure generally means that the driver is not receiving signals from the HMI 96. In another example, the HMIM 104 may be configured to utilize the HMI 96 and the attention monitor 102 to detect whether the driver is misusing (abusing) the system 90. For example, a driver may misuse the system by continually maximizing off-road gaze time by bouncing between states of awareness and momentary unconsciousness.

[0026] 3, a flow diagram illustrating example operational states of an HMIM system according to an embodiment of the present disclosure is shown. In one example, a function control process 200 may comprise a plurality of states of an HMIM system according to an embodiment of the present disclosure. In one example, the plurality of states may comprise a number of function states and a number of driver awareness states. In various embodiments, the HMIM 104 monitors the driver's awareness level (e.g., as captured by a time distribution of the driver awareness states of a Driver Attention Estimator (DAE)) and defines an allowable number of transitions and / or an overall allowable time within a particular DAE state. In one example, control process (or method) 200 may comprise step (or state) 202, step (or state) 204, step (or state) 206, step (or state) 208, step (or state) 210, step (or state) 212, step (or state) 214, step (or state) 216, step (or state) 218, step (or state) 220, step (or state) 222, step (or state) 224, step (or state) 226, step (or state) 228, step (or state) 230, step (or state) 232, step (or state) 234, and step (or state) 236.

[0027] In one example, the process 200 may start at state 202 with ADAS function off and may transition to state 204. In state 204, the ADAS function is not ready for activation. The ADAS function may remain not ready for activation until operational design domain (ODD) conditions 210 are appropriate for activation. The operational design domain (ODD) safety concept ensures that Society of Automotive Engineers Level 2-3 (SAE L2+) driver assistance functions are acceptably safe by reducing exposure to challenging operating situations. Challenging operating situations are those that are determined to be outside the known capabilities of the advanced driver assistance system (ADAS) and are therefore considered dangerous. The goal of the ODD safety concept is to be able to identify at least 99% of the operating situations to minimize exposure to dangerous scenarios.

[0028] When ODD conditions 210 are appropriate for activation, process 200 may move to state 206. In state 206, the ADAS function is ready for activation. Process 200 may remain in state 206 until the driver is observed to be in state 212 (e.g., hands on the steering wheel and eyes on the road). When the driver is in state 212, process 200 may move to state 208 upon receiving a driver activation request 214. In state 208, the ADAS function is active. With the ADAS function active, process 200 may monitor the driver's level of attention as captured by the time distribution of DAE states 220-226.

[0029] Process 200 may define the allowed number of awareness state transitions and / or the total allowed time within a particular DAE state. In one example, process 200 may set a particular duration (e.g., N minutes) and number of transitions (e.g., I, J, and K) from driver awareness state 220 where the driver is deemed aware based on gaze observation 230 on the road to driver awareness states 222, 224, and 226, respectively. In one example, the process 200 may set a first criterion (or threshold) 232 for determining whether the driver is considered to be in the temporary unconscious state 222 (e.g., the driver enters state 222 I times or more), a second criterion (or threshold) 234 for determining whether the driver is considered to be in the unconscious state 224 (e.g., the driver enters state 224 J times or more), and a third criterion (or threshold) 236 for determining whether the driver is considered to be in the out-of-loop state 226 (e.g., the driver enters state 226 K times or more). In one example, the values ​​I, J, and K may represent a maximum allowed number of transitions of each driver awareness state during a selected specific duration. In one example, the process 200 may move to state 240 when one or more of the criteria 232, 234, and 236 are met during a selected specific duration. In one example, the duration N and the criteria 232, 234, and 236 may be programmable. In one example, the thresholds I, J, and K may be similar or different. In one example, the thresholds I, J, and K may be set based on a profile of a particular driver. In one example, the criteria 232, 234, and 236 may be modified as the HMIM 104 learns the behavior (e.g., gaze behavior) distribution of a particular driver.

[0030] In state 240, process 200 may notify the driver to take over operation of the vehicle and reduce vehicle propulsion to a certain speed deemed safe (e.g., 5 kph). Process 200 may then move to state 204, where the ADAS function remains not ready for activation until the driver takes a certain action (e.g., cycling the ignition switch).

[0031] Referring to FIG. 4, a diagram illustrating an implementation of a system 100 according to an exemplary embodiment of the present disclosure is shown. In one example, the device (or system) 100 may be fully or at least partially mounted within the vehicle 50. In one example, the system (or device) 100 may be implemented as part of an advanced driver assistance system (ADAS) electronic control unit (ECU) 90. In various embodiments, the system 100 implementing a driver attentiveness estimator (DAE) may be implemented within the ADAS ECU 90 of the vehicle 50. The ADAS ECU 90 may be connected to a vehicle platform 92 of the vehicle 50. The vehicle 50 may include a driver monitoring system (DMS) 94, a human-machine interface (HMI) 96, a forward looking camera (FLC) 250, a number of corner radar sensors 252a-252d, a number of frontal radar sensors (not shown), a forward looking radar (FLR) sensor 254, a high definition (HD) map receiver 260, a global navigation satellite system (GNSS) receiver 262, and an inertial measurement unit (IMU) 264. In some embodiments, the vehicle 50 may also include LIDAR and / or sonar sensors (not shown).

[0032] The forward-looking camera (FLC) 250 is generally used to detect and identify objects and road features ahead of the vehicle 50. In one example, the forward-looking camera (FLC) 250 may be configured to provide stereoscopic video with a 100-degree field of view (FOV). In one example, the forward-looking camera (FLC) 250 may be used to detect road markings (e.g., lane markings, etc.), road signs, traffic lights, structures, etc. The corner radar sensors 252a-d and the forward-looking radar (FLR) sensor 254 (and LIDAR and / or sonar sensors, when present) are generally used to detect and track objects. In one example, each of the corner radar sensors 252a-d may have a 140-degree FOV. In one example, the forward-looking radar sensor (FLR) 254 may have two FOVs: an 18-degree FOV for long-range sensing and a 90-degree FOV for short-range sensing. The IMU 264 generally reports the orientation, angular velocity and acceleration, and forces acting on the vehicle 50.

[0033] In one example, the DMS 94, the HD map receiver 260, the GNSS receiver 262, the FLC 250, the FCRs 252a-252b, and the FLR 254 may be connected to the system 90. In one example, the DMS 94, the HD map receiver 260, the GNSS receiver 262, the FLC 250, the FCRs 252a-252b, and the FLR 254 may be connected to the system 90 via one or more vehicle buses of the vehicle 50. In another example, the DMS 94, the HD map receiver 260, the GNSS receiver 262, the FLC 250, the FCRs 252a-252b, and the FLR 254 may be connected to the system 90 via a wireless protocol. In one example, the DMS 94 may communicate driver attention information to the system 90. The FLC 250 may communicate surrounding road information (e.g., lane width, marker type, lane marker crossing indication, and video) to the system 90. The GNSS receiver 262 may communicate position data (e.g., latitude values, longitude values, adjustment information, and reliability information) to the system 90. The HD map receiver 260 may transfer map data to the system 90.

[0034] The FLC 250 may implement an optical sensor. In various embodiments, the FLC 250 may be an optical camera. The FLC 250 is generally operable to provide surrounding road information (or image data) to the system 90. The road information may include, but is not limited to, lane width data, marker type data, lane change indicators, and video of the road ahead of the vehicle 50 within the field of view of the FLC 250. In various embodiments, the FLC 250 may be a color camera. Color may be useful to distinguish between solid white lane markers (e.g., the rightmost lane marker) from solid yellow lane markers (e.g., the leftmost lane marker). In various embodiments, the FLC 250 may provide an estimated lane width for at least the current lane in the center of the field of view of the FLC 250. In some embodiments, the FLC 250 may provide an estimated lane width for lanes adjacent to the center lane. In other embodiments, the FLC 250 may provide an estimated lane width for all of the lanes in the field of view of the FLC 250. The lane width may be determined using standard image recognition and standard analysis methods implemented in the FLC 250. The FLC 250 may also identify all lane markers in the field of view of the FLC 250. When the FLC 250 crosses a lane marker, the FLC 250 may notify the system 90 that a lane change is occurring. The identification of the lane markers and lane change may be determined using standard image recognition and standard analysis methods implemented in the FLC 250. The FLC 250 may transfer the road information to the system 90 via a vehicle bus or wireless protocol.

[0035] One or more other types of sensors may be used in conjunction with the FLC 250. Exemplary sensors may include, but are not limited to, radar sensors, light detection and ranging (LiDAR) sensors, inertial sensors, thermal imaging sensors, and / or acoustic sensors. Some of the sensors may detect objects on the side of the road to estimate the left and right boundaries of the road. From the left and right boundaries, the width of the road may be calculated. From the calculated width, an estimate may be made of how many lanes could possibly fit within that width based on standard lane widths. The sensor may then estimate the current lane that the vehicle 50 occupies based on the relative distance of the sensor on the vehicle 50 to the left and right boundaries of the road and the estimated number of lanes. Lane crossings may be determined by the sensor based on the estimated number of lanes and changes in the relative distance to the left and / or right boundaries.

[0036] The system 90 may implement a control circuit (e.g., an electronic control unit). The system 90 is generally operable to track a current lane occupied by the vehicle 50 and to correct the current position of the vehicle 50 to the center of the current lane. The tracking may be based on satellite position data received at the GNSS receiver 262, map data received from the HD map receiver 260, and road information received in the video detections from the FLC 250 and the radar detections received from the FCRs 252a-252b and the FLR 254. The satellite position data may include an adjustment value and a corresponding reliability value.

[0037] The GNSS receiver 262 may implement a satellite navigation device. In various embodiments, the GNSS receiver 262 may include a Global Positioning System (GPS) receiver. Other types of satellite navigation devices may be implemented to meet the design criteria of a particular application. The GNSS receiver 262 is generally operable to provide latitude and longitude data for the vehicle 50 based on GNSS signals received from several satellites. The GNSS receiver 262 may also be operable to adjust the latitude and longitude data based on adjustment values ​​and corresponding reliability values ​​received from the system 90. The reliability values ​​may range from 0 (e.g., unreliable) to 1 (e.g., reliable). If the reliability value is above a high threshold (e.g., >0.7), the GNSS receiver 262 may correct the latitude and longitude data for each adjustment value. If the reliability value is below a low threshold (e.g., <0.3), the GNSS receiver 262 may ignore the adjustment value. If the reliability value is between the high and low thresholds, the GNSS receiver 262 may apply a correction to both the latitude and longitude data that is linearly weighted based on the degree of reliability.

[0038] The HD map receiver 260 may implement a radio frequency receiver. The HD map receiver 260 may be operable to receive map data from an antenna (not shown). The map data may be converted to a digital format and presented to the system 90.

[0039] 5, a diagram illustrating exemplary criteria for acceptable and unacceptable driver attentiveness is shown. In one example, graph 300 shows curve 302 representing an acceptable driver awareness distribution and curve 304 representing an unacceptable driver attention distribution. In one example, curve 302 generally represents a gaze distribution that results in a desired controllability for a particular population. Curve 304 generally represents a gaze distribution that does not result in a desired controllability for a particular population.

[0040] In one example, the attention monitor 102 implements a driver awareness level elevation regime that may generally generate warnings at several different levels (e.g., 306, 308, and 310) via the HMI 96 to detect insufficient attention related to short-term shared visual attention (e.g., from engagement in a secondary task). In one example, the attention monitor 102 may observe the driver gaze behavior over a short duration to determine the awareness level. In one example, if the driver is off-road 50% of the time over a short period of time (e.g., a few seconds) or is off-road 30% of the time over a longer period of time (e.g., 4-5 times longer than the short period), the attention monitor 102 may indicate that the driver is not aware. In one example, the attention monitor 102 may generate warnings at three different levels: temporarily unconscious, unconscious, and out of the loop. However, other numbers of levels may be implemented to meet the design criteria of a particular application.

[0041] In one example, the number of warnings may include, but are not limited to, audio and visual reminders, haptic reminders (e.g., seat vibration), hands on, reduced propulsion, a request to take over, and decelerating to a safe stop. In one example, for a temporary unconscious level, the attention monitor 102 may generate a warning 306 comprising audio and visual reminders. For an unconscious level, the attention monitor 102 may generate a warning 308 comprising audio and visual reminders + seat vibration, hands on, and reduced propulsion. For an out-of-loop level, the attention monitor 102 may generate a warning 310 comprising audio and visual reminders, seat vibration, hands on, reduced propulsion + a request to take over, and decelerating the vehicle to a safe stop.

[0042] In one example, the HMIM 104 generally tracks the output of the attention monitor 102, which generally provides a level of inattention for the driver as shown on the x-axis. If at any time the gaze distribution changes from one that provides a desired controllability for a particular population (e.g., curve 302) to one that does not provide a desired controllability for a particular population (e.g., curve 304), the HMIM 104 may assert control and indicate that the driver is "insufficiently attentive" even if the attention monitor 102 at that moment says the driver is conscious. The HMIM 104 and the attention monitor 102 generally operate within different time horizons.

[0043] Referring to FIG. 6, a diagram illustrating an example operation of an HMIM system according to one embodiment of the disclosure is shown. In one example, the attention monitor 102 may be configured to present a signal having a first state indicating that the driver's eyes are on the road and a second state indicating that the driver's eyes are not on the road. In one example, the HMIM 104 may be configured to capture and increment (i.e., count or accumulate) inattention states (e.g., temporarily unconscious, unconscious, and out of the loop) as the inattention states are output by the attention monitor 102 during a particular period of time (e.g., 15 minutes). In the illustrated example, the driver is bounding between an awareness state and a temporarily unconscious state. Each time the driver enters a temporarily unconscious state, the HMIM 104 logs an incident of inattention (e.g., by incrementing a count of the number of transitions per inattention state). Once the HMIM 104 captures (accumulates) six incidents (transitions) within a minute window, the HMIM 104 transitions control back to the driver and sends the vehicle into a safe shutdown mode. In one example, a similar outcome may also be triggered by the driver entering an unconscious state three times or an out-of-loop state once within a minute window.

[0044] 7-13, diagrams illustrating an example interaction between a driver and a driver attentiveness estimation system are shown, according to an embodiment of the present disclosure. In one example, the HMIM 104 may be configured as described above with respect to FIG. 6. In one example, the HMIM 104 may be configured to monitor the output by the attention monitor 102 using a 15-minute period (monitoring duration). In one example, the HMIM 104 may be configured to transition control back to the driver and send the vehicle into a safe shutdown mode in response to capturing six incidents of the driver being temporarily unconscious within a 15-minute time window. Similar outcomes may also be triggered by the driver entering an unconscious state three times or an out-of-loop state once within a 15-minute time window.

[0045] Referring to FIG. 7, a diagram is shown showing the HMIM 104 capturing the first incident of the driver being temporarily unconscious within a 15 minute time window. Pictures 500a and 500b are shown showing a view inside the vehicle cockpit from the driver's perspective. Display 502a shows the current status against the criteria used by the HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode before any incident of inattention. A circle 504a is shown showing where the driver's attention is directed. In one example, the driver is driving hands-free on a highway with ADAS functions active. The driver's attention is directed to the road ahead. The attention monitor 102 has not reported any transitions from consciousness to temporary unconsciousness, unconsciousness, or out-of-loop.

[0046] In picture 500b, display 502b shows the current status against the criteria used by HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode after the first incident of the driver being labeled as temporarily unconscious by attention monitor 102 at 4 minutes into the 15 minute time window. A circle 504b is shown indicating where the driver's attention is directed when attention monitor 102 labels the driver as temporarily unconscious. In one example, the driver receives a text message and the driver's attention shifts from the road ahead to a phone call. HMI 96 warns the driver to pay attention to the road. HMIM 104 increments the temporarily unconscious criteria by one incident.

[0047] Referring to FIG. 8, a diagram is shown showing the HMIM 104 capturing a second incident of the driver being temporarily unconscious within the 15-minute time window. Picture 500c and picture 500d are shown showing a view inside the cockpit of the vehicle from the driver's perspective. Display 502c shows the current status against the criteria used by the HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode following the first incident of inattention at 4 minutes into the 15-minute window. A circle 504c is shown showing where the driver's attention is directed. In one example, the driver complied with the attention request. The driver is driving hands-free on the highway with the ADAS function still active. The driver's attention is directed to the road ahead.

[0048] In picture 500d, display 502d shows the current status against the criteria used by HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode after a second incident at 6 minutes into the 15 minute time window where the driver is labeled as temporarily unconscious by the attention monitor 102. A circle 504d is shown indicating where the driver's attention is directed when the attention monitor 102 labels the driver as temporarily unconscious. In one example, the driver receives another text message and the driver's attention shifts from the road ahead to the phone. The HMI 96 warns the driver to pay attention to the road. The HMIM 104 increments the temporarily unconscious criteria by one incident for a total of two incidents.

[0049] Referring to FIG. 9, a diagram is shown showing the HMIM 104 capturing a third incident of the driver being temporarily unconscious within the 15-minute time window. Picture 500e and picture 500f are shown showing a view inside the cockpit of the vehicle from the driver's perspective. Display 502e shows the current status against the criteria used by the HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode following a second incident of inattention at 6 minutes into the 15-minute window. A circle 504e is shown showing where the driver's attention is directed. In one example, the driver complies with the attention request. The driver is driving hands-free on the highway with the ADAS function still active. The driver's attention is again directed to the road ahead.

[0050] In picture 500f, display 502f shows the current status against the criteria used by HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode after the third incident of the driver being labeled as temporarily unconscious by attention monitor 102 at 10 minutes into the 15 minute time window. A circle 504f is shown indicating where the driver's attention is directed when attention monitor 102 labels the driver as temporarily unconscious. In one example, the driver becomes distracted looking for an item in the vehicle's glove box. HMI 96 warns the driver to pay attention to the road. HMIM 104 increments the temporarily unconscious criteria by one incident for a total of three incidents.

[0051] Referring to FIG. 10, a diagram is shown showing the HMIM 104 capturing a fourth incident of the driver being temporarily unconscious within a 15-minute time window. Picture 500g and picture 500h are shown showing a view inside the vehicle cockpit from the driver's perspective. Display 502g shows the current status against the criteria used by the HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode following a third incident of inattention at 10 minutes into the 15-minute window. A circle 504g is shown showing where the driver's attention is directed. In one example, the driver complied with the attention request. The driver is driving hands-free on a highway with ADAS functions still active. The driver's attention is again directed to the road ahead.

[0052] In picture 500h, display 502h shows the current status for the criteria used by HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode after the fourth incident of the driver being labeled as temporarily unconscious by attention monitor 102 at 12 minutes into the 15 minute time window. A circle 504h is shown indicating where the driver's attention is directed when attention monitor 102 labels the driver as temporarily unconscious. In one example, the driver becomes distracted looking for an item in the vehicle's glove box. HMI 96 warns the driver to pay attention to the road. HMIM 104 increments the temporarily unconscious criteria by one incident for a total of four incidents.

[0053] Referring to FIG. 11, a diagram is shown showing the HMIM 104 capturing a fifth incident of the driver being temporarily unconscious within a 15-minute time window. Pictures 500i and 500j are shown showing a view inside the vehicle cockpit from the driver's perspective. Display 502i shows the current status against the criteria used by the HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode following a fourth incident of inattention at 12 minutes into the 15-minute window. A circle 504i is shown showing where the driver's attention is directed. In one example, the driver complied with the attention request. The driver is driving hands-free on a highway with ADAS functions still active. The driver's attention is again directed to the road ahead.

[0054] In picture 500j, display 502j shows the current status against the criteria used by HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode after the fifth incident of the driver being labeled as temporarily unconscious by attention monitor 102 at 13 minutes into the 15 minute time window. A circle 504j is shown indicating where the driver's attention is directed when the attention monitor labels the driver as temporarily unconscious. In one example, the driver is distracted by looking at the scenery passing by the vehicle outside the vehicle window. HMI 96 warns the driver to pay attention to the road. HMIM 104 increments the temporarily unconscious criteria by one incident for a total of five incidents.

[0055] Referring to FIG. 12, a diagram is shown showing the HMIM 104 capturing a sixth incident of the driver being temporarily unconscious within a 15-minute time window. Picture 500k and picture 500l are shown showing a view inside the cockpit of the vehicle from the driver's perspective. Display 502k shows the current status against the criteria used by the HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode following the fifth incident of inattention at 13 minutes into the 15-minute window. A circle 504k is shown showing where the driver's attention is directed. In one example, the driver complied with the attention request. The driver is driving hands-free on a highway with ADAS functions still active. The driver's attention is directed to the road ahead.

[0056] In picture 500l, display 502l shows the current status for the criteria used by HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe stop mode after the sixth incident of the driver being labeled as temporarily unconscious by attention monitor 102 in the 15 minute into the 15 minute time window. A circle 504l is shown indicating where the driver's attention is directed when attention monitor 102 labels the driver as temporarily unconscious. In one example, the driver receives a call and the driver's attention shifts from the road ahead to the phone. HMI 96 warns the driver to pay attention to the road. HMIM 104 increments the temporarily unconscious criteria by one incident for a total of six incidents.

[0057] Referring to FIG. 13, a diagram is shown showing the HMIM 104 after capturing a sixth incident of driver temporary unconsciousness within a 15 minute time window. A picture 500m is shown showing a view inside the vehicle cockpit from the driver's perspective. A display 502m shows the current status against the criteria used by the HMIM 104 to determine whether to transition control back to the driver and send the vehicle into a safe shutdown mode following the sixth incident of inattention in 20 minutes. A circle 504m is shown showing where the driver's attention will be directed. In one example, the HMIM 104 responds to six incidents of driver inattention within a 15 minute period by transitioning control back to the driver and safely suspending ADAS functions. The HMI 96 alerts the driver to take control of the vehicle. The vehicle begins to decelerate to 5 kph.

[0058] Generally, when the HMIM 104 determines that something is wrong, the HMIM 104 requests that the driver take over operation of the vehicle and safely suspend the automated functions. Because suspending the activity of the automated functions at high speeds may also be unsafe, especially when the driver is known to be unaware, the HMIM 104 may use the HMI 96 to present a request for the driver to take over operation of the vehicle. In one example, the HMIM 104 may facilitate a safe transition by slowly downgrading operation of the automated functions to a safe state (e.g., decelerating to a safe speed and / or safe stop) before fully suspending collaborative operation. However, other strategies for suspending collaborative operation may be implemented to meet the design criteria of a particular situation or application.

[0059] Referring to FIG. 14, a diagram illustrating an electronic control module implementing an advanced driver assistance system (ADAS) function control system according to an exemplary embodiment of the present disclosure is shown. In one example, the device 800 may implement an electronic control unit or module (ECU). In one example, the electronic control module (ECU) 800 may be implemented as a domain controller (DC). In another example, the ECU 800 may be implemented as an active safety domain master (ASDM). In various embodiments, the ECU 800 may be configured to control activation of one or more features (or functions) of the ADAS components of the vehicle. In various embodiments, the driver attentiveness estimator 100 may be implemented within the ECU 800. In one example, the ECU 800 may be connected to the vehicle platform 92, the driver monitoring system (DMS) 94, the human-machine interface (HMI) 96, the circuit 802 (which may implement an electronic bus), and the map and sensors 804 of the vehicle. In one example, ECU 800 may be configured to (i) receive signals VEHICLE SPEED, DEACTIVATION REQUEST, ACTIVATION REQUEST, and DRIVER INFO from vehicle systems, and communicate signals DECELERATION REQUEST and DRIVER WARNING to vehicle systems.

[0060] In one example, the ECU 800 may be connected to a block (or circuit) 802. The circuit 802 may implement an electronic bus. The circuit 802 may be configured to transfer data between the ECU 800 and the vehicle platform 92, the DMS 94, the HMI 96, and the maps and sensors 804 (e.g., the HD map receiver 260, the GNSS receiver 262, the forward-looking camera (FLC) 250, the corner / side radar sensors 252a-252n, the forward-looking radar (FLR) sensor 254, and the inertial measurement unit 264). In some embodiments, the circuit 802 may be implemented as a vehicle controller area network (CAN) bus. The circuit 802 may be implemented as an electronic wired and / or wireless network. In general, the circuit 802 may connect one or more components of the vehicle 50 to enable sharing of information in the form of digital signals (e.g., a serial bus, an electronic bus connected by wiring and / or interfaces, a wireless interface, etc.).

[0061] ECU 800 generally comprises block (or circuit) 820, block (or circuit) 822, block (or circuit) 824, block (or circuit) 826, and block (or circuit) 828. Circuit 820 may implement a processor. Circuit 822 may implement a communication port. Circuit 824 may implement a filter. Circuit 826 may implement a clock. Circuit 828 may implement a memory. Other blocks (not shown) (e.g., I / O ports, power connectors, interfaces, etc.) may be implemented. The number and / or type of circuits implemented by module 800 may vary according to the design criteria of a particular implementation.

[0062] The circuit 820 may be implemented as a microcontroller, a multi-threaded microprocessor, or any combination thereof. The circuit 820 may include a block (or circuit) implementing the attention monitor 102, a block (or circuit) implementing the man-machine interaction monitor 104, and / or a block (or circuit) implementing the mode management 106. The circuit 820 may include other components (not shown). In some embodiments, the circuit 820 may be a combined (e.g., integrated) chipset that implements processing functionality. In some embodiments, the circuit 820 may include several separate circuits (e.g., a microcontroller, a multi-threaded microprocessor, a digital signal processor (DSP), a graphics processing unit (GPU), etc.). The design of the circuit 820 and / or the functionality of the various components of the circuit 820 may vary according to the design criteria of a particular implementation. The circuit 820 is shown transmitting data to and / or receiving data from the vehicle platform 92, the circuit 822, and / or the circuit 828.

[0063] The circuit 828 may include a block (or circuit) 860 and a block (or circuit) 862. The block 860 may store driver awareness (or attentiveness) estimator (DAE) data. The block 862 may store computer readable instructions (e.g., instructions readable by the circuit 820). The DAE data 860 may store various data sets 870a-870n. For example, the data sets 870a-870n may include a count of transitions to a temporary unconscious state 870a, a count of transitions to an unconscious state 870b, a count of transitions to an out-of-loop state 870c, a long-term gaze distribution 870d, driver information 870e, and / or other data 870n.

[0064] In one example, the other data 870n may comprise parameters (e.g., coefficients) used to convert data received from sensors (e.g., FLC, FLR, FCR, FCS, and IMU). The calibration data 870n may provide many sets of coefficients (e.g., one set of coefficients for each of the sensors). The calibration data 870n may be updatable. For example, the calibration data 870n may store current values ​​as coefficients for a sensor and data from sensor drift, and the module 800 may update the calibration data 870n to maintain accuracy. The format of the calibration data 870n may vary based on the design criteria of a particular implementation.

[0065] Various other types of data (e.g., other data 870n) may be stored as part of the DAE data 860. For example, other data 870n may store gaze distributions for multiple drivers. For example, other data 870n may store past data values ​​of the calibration data and / or current data values ​​of the calibration data. Past and current data values ​​of the calibration data may be compared to determine trends that are used to extrapolate and / or predict potential future values ​​for the calibration data.

[0066] The circuit 820 may be configured to execute stored computer-readable instructions (e.g., instructions 862 stored in the circuit 828). The circuit 820 may perform one or more steps based on the stored instructions 862. In one example, the steps of the instructions 862 may be executed / performed by the circuit 820 to implement one or more of the attention monitor 102, the man-machine interaction monitor 104, and the mode manager 106. The instructions executed by the circuit 820 and / or the order in which the instructions 862 are executed may vary according to design criteria of a particular implementation.

[0067] The circuitry 822 may enable the module 800 to communicate with external devices such as the maps and sensors 804, the vehicle platform 92, the driver monitoring system 94, and the man-machine interface 96. For example, the module 800 is shown connected to the external circuitry 802. In one example, information from the module 800 may be communicated to an infotainment device for display to the driver. In another example, a wireless connection (e.g., Wi-Fi, Bluetooth, cellular, etc.) to a portable computing device (e.g., a smartphone, a tablet computer, a notebook computer, a smartwatch, etc.) may enable information from the module 800 to be displayed to a user.

[0068] The circuit 824 may be configured to perform a linear quadratic estimation. For example, the circuit 824 may implement a Kalman filter. In general, the circuit 824 may operate recursively on input data to generate a statistically optimal estimate. For example, the circuit 824 may be used to calculate the position coordinate 870a and / or estimate the accuracy of the position coordinate 870a. In some embodiments, the circuit 824 may be implemented as a separate module. In some embodiments, the circuit 824 may be implemented as part of the circuit 828 (e.g., stored instructions 862). The implementation of the circuit 824 may vary according to the design criteria of a particular implementation.

[0069] The circuit 826 may be configured to determine and / or track time. The time determined by the circuit 826 may be stored as timestamp data 870c. In some embodiments, the circuit 826 may be configured to compare timestamps received from GNSS receivers.

[0070] The module 800 may be configured as a chipset, a system on a chip (SoC), and / or a discrete device. For example, the module 800 may be implemented as an electronic control unit (ECU). In some embodiments, the module 800 may be configured to control activation of one or more ADAS features / functions.

[0071] Given the absence of state-of-the-art ASIL in HMI warning messages, the underlying objective of the HMIM 104 according to one embodiment of the present disclosure is to provide monitoring functionality that ensures sufficient controllability of the supervising driver to a possible hazardous event. In one example, the HMIM 104 may achieve sufficient controllability by ensuring driver engagement. In one example, the HMIM 104 may ensure driver engagement by monitoring gaze distribution patterns as a measure of attentiveness. In various embodiments, several exemplary functionality iterations of the HMIM 104 may be implemented to mitigate false positives during decision making.

[0072] In one example, the HMIM 104 may check the delta change in eye gaze shift before and after an HMI warning to determine if the warning is being communicated to the driver and to guard against missing HMI messages. For example, the HMIM 104 may subscribe to a signal indicative of "eyes on the road" in real time from a camera of the driver monitoring system 94. The signal indicative of "eyes on the road" may be used as feedback to rapidly assess whether there is an improvement in gaze distribution after each rising warning. If not, the HMIM 104 may failsafe appropriately.

[0073] In another example, to guard against false positives from the DMS94 camera and attention monitor attention levels, when the attention monitor 102 reports the driver being “awake” for a longer duration than expected, the HMIM 104 may intentionally attempt to momentarily take the driver's eyes off the road. When the driver is reported as being “awake” for a longer duration, the HMIM 104 may send directed prompts to divert the driver's attention off the road (when it is determined to be safe to do so by subscribing to environmental information from on-board sensors, GNSS, HD maps, etc.) and verify whether the front-end signal chain (e.g., DMS94, attention monitor 102, etc.) detects the diverted attention. If not, the HMIM 104 may failsafe appropriately.

[0074] In another example, other cabin sensor information may be integrated into the HMIM 104 as inputs to form a holistic driver state estimate. In addition to gaze-on-road information, the HMIM 104 may subscribe to hands-on-steering-wheel, pedal information, seat sensors, seat belt status, etc. to form a holistic driver state estimation model. The HMIM 104 may leverage feedback from each of these inputs to detect and mitigate inattention.

[0075] In yet another example, the HMIM 104 may be developed using artificial intelligence / machine learning (AI / ML) based non-deterministic algorithms to baseline driver attentiveness profiles for each individual driver and track inattention against the specific driver's baseline. In various embodiments, each vehicle may implement a comprehensive HMIM 104 that may customize itself over time by tracking and learning about the driver's inattention profile by baselined against the same driver's gaze distribution during manual driving. Inattention during supervised driving may then be flagged if the inattention exceeds the threshold mentioned during the previous manual driving.

[0076] The terms "may" and "generally" when used herein with "is (are)" and verbs are intended to convey the intent that the description is exemplary and are considered to be broad enough to encompass both the specific examples presented in this disclosure and alternatives that may be derived based on this disclosure. The terms "may" and "generally" as used herein should not be construed as necessarily implying the desirability or possibility of omitting the corresponding element.

[0077] Designations of various components, modules, and / or circuits as "a" through "n," as used herein, disclose either a singular component, module, and / or circuit, or a plurality of such components, modules, and / or circuits, with the "n" designation applied to mean any particular integer. Different components, modules, and / or circuits, each having an instance (or occurrence) with a designation "a" through "n," may indicate that the different components, modules, and / or circuits may have a matching number of instances or different numbers of instances. An instance designated "a" may represent the first of multiple instances, and instance "n" may refer to the last of multiple instances, but does not imply a particular number of instances.

[0078] Although exemplary embodiments have been illustrated and described in detail, various changes in form and details may be made without departing from the scope of the disclosure. [Explanation of symbols]

[0079] 50 vehicles 90 Systems, devices, advanced driver assistance systems (ADAS) electronic control units (ECUs) 92 Vehicle Platform 94 Driver Monitoring System (DMS) 96 Man-machine interface (HMI) 100 Function Control Module 102 Attention Monitor 104 Man-machine interaction monitor (HMIM) 106 Mode Management, Mode Manager 108 Driver's gaze behavior 110 Short duration time windows 112 Long duration time windows 200 Functional Control Process 210 Operational Design Domain (ODD) Conditions 214 Driver Activation Request 220 Driver consciousness state 222 Temporary Unconsciousness 224 Unconsciousness 226 Out of Loop State 230 Roadside observation 232 First Criterion 234 Second Criterion 236 Third Criterion 250 Front monitoring camera (FLC) 252 Corner Radar Sensor 254 Forward Looking Radar (FLR) Sensor 260 High Definition (HD) Map Receiver 262 Global Navigation Satellite System (GNSS) Receiver 264 Inertial Measurement Unit (IMU) 302 Curve showing acceptable driver awareness distribution 304 Curve Representing Unacceptable Driver Attention Distribution 306 Warning 308 Warning 310 Warning 500 Pictures 502 Display 800 Equipment, Electronic Control Module (ECU) 802 External circuit 804 Maps and Sensors 860 DAE data 862 command 870 Datasets 870a Count of transitions to temporary unconsciousness, location coordinates 870b Counting transitions to unconsciousness 870c Count of transitions to out-of-loop states, timestamp data 870d Long-term gaze distribution 870e Driver Information 870n Other data, calibration data

Claims

1. 1. An apparatus comprising: an interface configured to receive a plurality of sensor signals from a vehicle platform of a vehicle and to present one or more control signals to the vehicle platform; a control circuit configured to: (i) detect whether a driver's attention state is attentive or inattentive in response to one or more of the plurality of sensor signals from the vehicle platform during a first window having a first duration; (ii) assess whether the driver is sufficiently attentive by monitoring the one or more of the plurality of sensor signals from the vehicle platform and determining whether a change in the driver's attention state during a second window having a second duration longer than the first duration exceeds a threshold; and (iii) transition operation of the vehicle to the driver and safely suspend automation system functions of the vehicle when the threshold is exceeded; Equipped with the control circuitry comprises a driver attention estimator configured to (i) generate a first control signal communicating the attention state of the driver during the first window having the first duration, and (ii) generate a second control signal communicating an assessment of whether the driver is sufficiently attentive for the automated system function to continue operating the vehicle; and the driver attention estimator comprises an attention monitor configured to determine the attention state of the driver during the first window having the first duration and generate the first control signal; and a man-machine interaction monitor configured to generate the assessment of whether the driver is sufficiently attentive during the second window having the second duration and generate the second control signal, and further configured to monitor a plurality of driver attention states in the attention monitor and use a respective threshold for each of the plurality of driver attention states in the attention monitor.

2. The apparatus of claim 1 , wherein the human-machine interaction monitor is further configured to assess whether a human-machine interface of the vehicle is successfully communicating with the driver.

3. The apparatus of claim 1 , wherein the human-machine interaction monitor is further configured to determine whether the driver is misusing the attention monitor.

4. The apparatus of claim 1 , wherein the plurality of driver attention states of the attention monitor comprises one or more of an attention state, a temporary unconscious state, an unconscious state, and an out-of-loop state.

5. 10. The apparatus of claim 1, wherein the respective thresholds for each of the plurality of driver attention states of the attention monitor are programmable.

6. 2. The apparatus of claim 1, wherein the respective thresholds for each of the plurality of driver attention states of the attention monitor comprise a maximum allowable number of transitions from the attention state to the corresponding driver attention state during the second window having the second duration.

7. 10. The apparatus of claim 1, wherein the human-machine interaction monitor is further configured to customize itself over time by tracking and learning the driver's inattention profile.

8. 8. The apparatus of claim 7, wherein the human-machine interaction monitor is further configured to learn the inattention profile of the driver by baselining the inattention profile of the driver against the driver's gaze distribution during manual driving.

9. 10. The apparatus of claim 1, further comprising a function mode manager, wherein the control circuitry is configured to activate or maintain operation of the automation system of the vehicle when the threshold is not exceeded and to safely transition operation of the vehicle from the automation system to the driver when the threshold is exceeded.

10. 10. The apparatus of claim 9, wherein the function mode manager is configured to downgrade execution of an autopilot function of the automation system by sending a warning to the driver via a human-machine interface of the vehicle to take over control of the vehicle and generating a deceleration request to the vehicle platform.

11. 1. A method for controlling an automation system function in a vehicle, comprising: receiving a plurality of sensor signals from a vehicle platform of a vehicle; detecting whether a driver's attention state is attentive or inattentive in response to one or more of the plurality of sensor signals from the vehicle platform during a first window having a first duration; assessing whether the driver is sufficiently attentive by monitoring the one or more of the plurality of sensor signals from the vehicle platform and determining whether a change in the attention state of the driver during a second window having a second duration longer than the first duration exceeds a threshold; transferring control of the vehicle to the driver and safely suspending the automated system functions of the vehicle when the threshold is exceeded; wherein the method comprises: and a man-machine interaction monitor configured to generate the assessment of whether the driver is sufficiently attentive during the second window having the second duration and generate the second control signal, and further configured to monitor a plurality of driver attention states in the attention monitor and use a respective threshold for each of the plurality of driver attention states in the attention monitor.

12. The method of claim 11 , further comprising using the human-machine interaction monitor to assess whether the vehicle's human-machine interface is successfully communicating with the driver.

13. 12. The method of claim 11, wherein the respective threshold for each of the plurality of driver attention states of the attention monitor comprises a maximum allowable number of transitions from an attention state to a corresponding driver attention state during the second window having the second duration.

Images