Vehicle human-machine interface alert generation based on obscured regions

Abstract

The invention provides vehicle human-machine interface alert generation based on an occlusion zone. An example vehicle control system includes: a plurality of sensors; a driver monitoring system; a human-machine interface configured to provide at least one of a visual alert, a haptic alert, and an audio alert to a driver; and a vehicle control module configured to identify at least one occlusion region based on the plurality of sensors, determine a predicted trajectory of the vehicle, score the at least one occlusion region based on an importance of the at least one occlusion region to the predicted trajectory of the vehicle, and determine a predicted trajectory of the vehicle. A driver attention direction is determined based on detection of at least one of a gaze direction and a head orientation of a driver of the vehicle, and an occlusion region alert is generated based on the score of the at least one occlusion region and the driver attention direction.

Description

[0001] Introduction

[0002] The information provided in this section is for the purpose of generally presenting the context of the present disclosure. To the extent that the work of the presently named inventors is described in this section and aspects of this description that may not otherwise be eligible as prior art at the time of filing are neither expressly nor implicitly admitted as prior art against the present disclosure. Technical Field

[0003] The present disclosure generally relates to occlusion - zone - based vehicle human - machine interface (HMI) alert generation, including alert escalation based on detected driver gaze direction and head orientation. Background Art

[0004] Vehicles use various sensors (such as cameras, radars, and lidar sensors) installed inside and around the vehicle to detect objects by scanning the environment around the vehicle. In some driving situations, areas around the vehicle may be occluded by one or more objects, which may compromise the safety of the vehicle driver. Summary of the Invention

[0005] An example vehicle control system includes: a plurality of sensors configured to sense the environment around the vehicle; a driver monitoring system configured to detect at least one of a driver's gaze direction and head orientation of a vehicle driver; a human - machine interface configured to provide at least one of a visual alert, a haptic alert, and an audio alert to the driver; and a vehicle control module configured to identify at least one occlusion zone based on the plurality of sensors, each occlusion zone indicating an area around the vehicle occluded by one or more objects detected by the plurality of sensors, determine a predicted trajectory of the vehicle, score the at least one occlusion zone based on the importance of the at least one occlusion zone to the predicted trajectory of the vehicle, determine a driver attention direction based on the detection of at least one of the driver's gaze direction and head orientation, and generate an occlusion - zone alert via the human - machine interface based on the score of the at least one occlusion zone and the driver attention direction.

[0006] In other features, the occlusion - zone alert includes at least one of a visual notification on a vehicle display at a position in a direction corresponding to the at least one occlusion zone, a directional haptic seat alert including vibration of a portion of the vehicle seat corresponding to the direction of the at least one occlusion zone, or a directional audio alert including audio generated by at least one vehicle speaker at a speaker position corresponding to the direction of the at least one occlusion zone.

[0007] In other features, the occlusion - zone alert includes all of the visual notification, the directional haptic seat alert, and the directional audio alert.

[0008] Among other features, the vehicle control module is configured to generate a map of the vehicle's surrounding environment, the map including one or more objects around the vehicle detected by a plurality of sensors, and selectively filter at least one occluded area from the map, the map including one or more filtered occluded areas after the selective filtering, wherein scoring the at least one occluded area includes scoring the one or more filtered occluded areas based on the importance of the one or more filtered occluded areas to the predicted trajectory of the vehicle.

[0009] Among other features, the vehicle control module is configured to modify the vehicle's trajectory based on the importance score of the filtered occluded area and use an advanced driver assistance system to advance the vehicle according to the modified trajectory.

[0010] Among other features, the vehicle control module is configured to obtain a driver history associated with the vehicle driver, the driver history including at least one of a historical driver attention score and a historical driver alert response score, and generate an occluded area alert at least in part based on the driver history.

[0011] Among other features, the vehicle control module is configured to, during generating the occluded area alert, determine an updated score of at least one occluded area and detect an updated driver attention direction, and terminate the occluded area alert in response to at least one of the updated score falling below an alert generation threshold, the updated driver attention direction corresponding to the direction of at least one occluded area, or the updated driver attention direction corresponding to the direction of the occluded area alert.

[0012] Among other features, the vehicle control module is configured to upgrade the intensity of the occluded area alert in response to the updated score remaining above the alert generation threshold and the updated driver attention direction not corresponding to the direction of at least one occluded area or the direction of the occluded area alert.

[0013] Among other features, the vehicle control module is configured to upgrade the intensity of the occluded area alert by increasing the visual size of a visual notification alert on the vehicle display and changing the visual color.

[0014] Among other features, the vehicle control module is configured to upgrade the intensity of the occluded area alert by flashing the visual notification alert on the vehicle display.

[0015] Among other features, the vehicle control module is configured to upgrade the intensity of the occluded area alert by activating directional tactile seat pulses in the vehicle seat.

[0016] Among other features, the vehicle control module is configured to upgrade the intensity of the occluded area alert by generating directional audio via at least one vehicle speaker.

[0017] An example method for controlling vehicle human-machine interface alerts includes: identifying at least one occlusion zone around the vehicle based on multiple sensors configured to sense the vehicle's surrounding environment, each occlusion zone indicating an area around the vehicle that is occluded by one or more objects detected by the multiple sensors; determining a predicted trajectory of the vehicle, scoring the at least one occlusion zone based on the importance of the at least one occlusion zone to the predicted trajectory of the vehicle, determining a driver attention direction based on detection by a vehicle driver monitoring system of at least one of the vehicle driver's gaze direction and head orientation, and generating an occlusion zone alert via the vehicle's human-machine interface based on the score of the at least one occlusion zone and the driver attention direction, wherein the human-machine interface is configured to provide at least one of a visual alert, a tactile alert, and an audio alert to the driver.

[0018] In other features, the occlusion zone alert includes at least one of a visual notification on a vehicle display at a location corresponding to the direction of the at least one occlusion zone, a directional tactile seat alert including vibration of a portion of the vehicle seat corresponding to the direction of the at least one occlusion zone, or a directional audio alert including audio generated by at least one vehicle speaker at a speaker location corresponding to the direction of the at least one occlusion zone.

[0019] In other features, the occlusion zone alert includes all of the visual notification, the directional tactile seat alert, and the directional audio alert.

[0020] In other features, the method includes generating a map of the vehicle's surrounding environment, the map including one or more objects around the vehicle detected by the multiple sensors, and selectively filtering at least one occlusion zone from the map, the map including one or more filtered occlusion zones after the selective filtering, wherein scoring the at least one occlusion zone includes scoring the one or more filtered occlusion zones based on the importance of the one or more filtered occlusion zones to the predicted trajectory of the vehicle.

[0021] In other features, the method includes modifying the trajectory of the vehicle based on the importance score of the filtered occlusion zone and advancing the vehicle according to the modified trajectory using an advanced driver assistance system.

[0022] In other features, the method includes obtaining a driver history associated with the vehicle driver, wherein the driver history includes at least one of a historical driver attention score and a historical driver alert response score, and generating the occlusion zone alert at least in part based on the driver history.

[0023] Among other features, the method includes determining an updated score for at least one occlusion zone and detecting an updated driver attention direction during generation of an occlusion zone alert, and terminating the occlusion zone alert in response to the updated score falling below an alert generation threshold, the updated driver attention direction corresponding to the direction of at least one occlusion zone, or the updated driver attention direction corresponding to the direction of the occlusion zone alert, at least one of which is satisfied.

[0024] Among other features, the method includes escalating the intensity of the occlusion zone alert in response to the updated score remaining above the alert generation threshold and the updated driver attention direction not corresponding to the direction of at least one occlusion zone or the direction of the occlusion zone alert.

[0025] This application also includes the following solutions:

[0026] Solution 1. A vehicle control system, comprising:

[0027] A plurality of sensors configured to sense the surrounding environment of the vehicle;

[0028] A driver monitoring system configured to detect at least one of the gaze direction and head orientation of a vehicle driver;

[0029] A human-machine interface configured to provide at least one of a visual alert, a tactile alert, and an audible alert to the driver; and

[0030] A vehicle control module configured to:

[0031] Identify at least one occlusion zone based on the plurality of sensors, each occlusion zone indicating an area around the vehicle that is occluded by one or more objects detected by the plurality of sensors;

[0032] Determine a predicted trajectory of the vehicle;

[0033] Score the at least one occlusion zone based on the importance of the at least one occlusion zone to the predicted trajectory of the vehicle;

[0034] Determine a driver attention direction based on detection of at least one of the gaze direction and head orientation of the vehicle driver; and

[0035] Generate an occlusion zone alert via the human-machine interface based on the score of the at least one occlusion zone and the driver attention direction.

[0036] Solution 2. The vehicle control system according to Solution 1, wherein the occlusion zone alert includes at least one of the following:

[0037] A visual notification on a vehicle display at a position corresponding to the direction of the at least one occlusion zone;

[0038] A directional haptic seat alert that includes vibration of a portion of a vehicle seat corresponding to the direction of the at least one occluded area; or

[0039] A directional audio alert that includes audio generated by at least one vehicle speaker at a speaker location corresponding to the direction of the at least one occluded area.

[0040] Solution 3. The vehicle control system according to Solution 2, wherein the occluded area alert includes all visual notifications, directional haptic seat alerts, and directional audio alerts.

[0041] Solution 4. The vehicle control system according to Solution 1, wherein the vehicle control module is configured to:

[0042] Generate a map of the vehicle's surrounding environment, the map including one or more objects around the vehicle detected by the plurality of sensors; and

[0043] Selectively filter the at least one occluded area from the map, the map including one or more filtered occluded areas after selective filtering, wherein scoring the at least one occluded area includes scoring the one or more filtered occluded areas based on their importance to the predicted trajectory of the vehicle.

[0044] Solution 5. The vehicle control system according to Solution 4, wherein the vehicle control module is configured to:

[0045] Modify the vehicle's trajectory based on the importance score of the filtered occluded area; and

[0046] Use an advanced driver assistance system to propel the vehicle according to the modified trajectory.

[0047] Solution 6. The vehicle control system according to Solution 1, wherein:

[0048] The vehicle control module is configured to obtain a driver history associated with the vehicle driver;

[0049] The driver history includes at least one of a historical driver attention score and a historical driver alert response score; and

[0050] Generate the occluded area alert at least in part based on the driver history.

[0051] Solution 7. The vehicle control system according to Solution 1, wherein the vehicle control module is configured to:

[0052] During generation of the occluded area alert, determine an updated score for the at least one occluded area and detect an updated driver attention direction; and

[0053] Terminate the occluded area alert in response to at least one of: an updated score falling below an alert generation threshold, an updated driver attention direction corresponding to the direction of the at least one occluded area, or an updated driver attention direction corresponding to the direction of an occluded area alert.

[0054] Solution 8. The vehicle control system according to Solution 7, wherein the vehicle control module is configured to upgrade the intensity of the occluded area alert in response to an updated score remaining above the alert generation threshold and an updated driver attention direction not corresponding to the direction of the at least one occluded area or the direction of the occluded area alert.

[0055] Solution 9. The vehicle control system according to Solution 8, wherein the vehicle control module is configured to upgrade the intensity of the occluded area alert by increasing the visual size of the visual notification alert on the vehicle display and changing the visual color.

[0056] Solution 10. The vehicle control system according to Solution 8, wherein the vehicle control module is configured to upgrade the intensity of the occluded area alert by flashing the visual notification alert on the vehicle display.

[0057] Solution 11. The vehicle control system according to Solution 8, wherein the vehicle control module is configured to upgrade the intensity of the occluded area alert by activating directional tactile seat pulses in the vehicle seat.

[0058] Solution 12. The vehicle control system according to Solution 8, wherein the vehicle control module is configured to upgrade the intensity of the occluded area alert by generating directional audio via at least one vehicle speaker.

[0059] Solution 13. A method for controlling a vehicle human-machine interface alert, the method comprising:

[0060] Identify at least one occluded area around the vehicle based on a plurality of sensors configured to sense the vehicle's surrounding environment, each occluded area indicating an area around the vehicle that is occluded by one or more objects detected by the plurality of sensors;

[0061] Determine the predicted trajectory of the vehicle;

[0062] Score the at least one occluded area based on the importance of the at least one occluded area to the predicted trajectory of the vehicle;

[0063] Determine the driver attention direction based on the detection by a vehicle driver monitoring system of at least one of the vehicle driver's gaze direction and head orientation; and

[0064] Generate an occlusion zone alert via a human-machine interface of the vehicle based on the score of the at least one occlusion zone and the driver's attention direction, wherein the human-machine interface is configured to provide at least one of a visual alert, a tactile alert, and an audio alert to the driver.

[0065] Solution 14. The method according to Solution 13, wherein the occlusion zone alert includes at least one of the following:

[0066] A visual notification on a vehicle display at a position corresponding to the direction of the at least one occlusion zone;

[0067] A directional tactile seat alert including vibration of a part of the vehicle seat corresponding to the direction of the at least one occlusion zone; or

[0068] A directional audio alert including audio generated by at least one vehicle speaker at a speaker position corresponding to the direction of the at least one occlusion zone.

[0069] Solution 15. The method according to Solution 14, wherein the occlusion zone alert includes all visual notifications, directional tactile seat alerts, and directional audio alerts.

[0070] Solution 16. The method according to Solution 13, further comprising:

[0071] Generating a map of the vehicle's surrounding environment, the map including one or more objects around the vehicle detected by the plurality of sensors; and

[0072] Selectively filtering the at least one occlusion zone from the map, the map including one or more filtered occlusion zones after the selective filtering, wherein scoring the at least one occlusion zone includes scoring the one or more filtered occlusion zones based on the importance of the one or more filtered occlusion zones to the predicted trajectory of the vehicle.

[0073] Solution 17. The method according to Solution 16, further comprising:

[0074] Modifying the trajectory of the vehicle based on the importance score of the filtered occlusion zone; and

[0075] Using an advanced driver assistance system to advance the vehicle according to the modified trajectory.

[0076] Solution 18. The method according to Solution 13, further comprising obtaining a driver history associated with the vehicle driver, wherein:

[0077] The driver history includes at least one of a historical driver attention score and a historical driver alert response score; and

[0078] Generate the occlusion zone alert based at least in part on the driver history.

[0079] Solution 19. The method according to Solution 13, further comprising:

[0080] During generating the occlusion zone alert, determine an updated score of the at least one occlusion zone and detect an updated driver attention direction; and

[0081] Terminate the occlusion zone alert in response to at least one of the updated score falling below the alert generation threshold, the updated driver attention direction corresponding to the direction of the at least one occlusion zone, or the updated driver attention direction corresponding to the direction of the occlusion zone alert.

[0082] Solution 20. The method according to Solution 19, further comprising upgrading the intensity of the occlusion zone alert in response to the updated score remaining above the alert generation threshold and the updated driver attention direction not corresponding to the direction of the at least one occlusion zone or the direction of the occlusion zone alert.

[0083] Based on the detailed description, claims, and drawings, additional fields of applicability of the present disclosure will become apparent. The detailed description and specific examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Brief Description of the Drawings

[0084] The present disclosure will be more fully understood in accordance with the detailed description and the drawings.

[0085] Figure 1 An example of a system including a vehicle communicating with a remote server and a system is shown.

[0086] Figure 2 An example of a system for calculating, filtering, and scoring occlusions around a vehicle to modify the vehicle trajectory is shown.

[0087] Figure 3 And Figure 4 An example of an occlusion zone around a vehicle, a filtered occlusion zone, and scoring of the filtered occlusion zone is shown.

[0088] Figure 5 An example of a method for calculating, filtering, and scoring occlusions around a vehicle to modify the vehicle trajectory is shown.

[0089] Figure 6 An example of a method for filtering occlusions is shown.

[0090] Figure 7 An example of a method for scoring occlusions is shown.

[0091] Figure 8illustrates an example method for training a neural network used in a scoring module of a system shown in Figure 2 to score occlusions.

[0092] Figure 9 is a block diagram of an example system for generating a human-machine interface alert based on a detected occlusion area.

[0093] Figure 10 is a schematic diagram illustrating an example clustering around a vehicle for grouping occlusion areas.

[0094] Figure 11 is a flowchart depicting an example process for generating an HMI alert based on a detected occlusion area.

[0095] Figure 12 is a flowchart depicting an example process for upgrading an HMI alert generated based on a process of Figure 11 .

[0096] In the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0097] Occlusions are prevalent in many driving scenarios where one or more objects in the surrounding environment at least partially block the driver's field of view, which may compromise the driver's safety. For example, an obstacle emerging from an occluded area (such as another vehicle) can be unsafe for a human driver who is inattentive or does not anticipate the obstacle.

[0098] In some example embodiments described herein, a vehicle control module automatically assigns an importance score to an occlusion area detected in the vehicle's surrounding environment. The presence of the occlusion area and the importance of the occlusion area can be communicated to the driver via one or more human-machine interfaces of the vehicle (such as a visual display, a tactile seat vibration, an audio alert, etc.) to notify the driver of a potential dangerous situation nearby.

[0099] In some examples, a motion planning subsystem (e.g., a navigation subsystem) in autonomous and semi-autonomous vehicles (hereinafter referred to as "vehicles") needs to consider occlusions to modify the vehicle's trajectory. A trajectory is different from a motion plan or a route. A motion plan is a route, which is a static element provided by the navigation subsystem, and the vehicle travels on this route from a source to a destination. A trajectory is a small part or segment (i.e., a subset) of the motion plan, which needs to be periodically updated depending on the dynamically changing surrounding environment around the vehicle to adjust the vehicle's movement (such as speed and steering) according to the motion plan to advance the vehicle. The present disclosure relates to periodically changing the vehicle's trajectory depending on the dynamically changing surrounding environment around the vehicle.

[0100] Broadly speaking, sensors on a vehicle (such as camera, radar, and lidar sensors) sense (perceive) the vehicle's surrounding environment and provide sensed data about the vehicle's surrounding environment to the vehicle's navigation subsystem. The navigation subsystem generates a map of the vehicle's surrounding environment based on the sensed data. The map is a snapshot of the vehicle's surrounding environment. Due to various static obstacles (such as buildings) and dynamic obstacles (such as other vehicles, pedestrians, cyclists, etc. around the vehicle), certain areas of the map may be occluded. The number of unobserved or unobservable (i.e., occluded) areas on the map may increase rapidly with multiple occluding objects. For example, additional vehicles, pedestrians, and objects such as cones and roadblocks used in road construction may appear in the map. The rapid increase in the number of occluded areas in the map increases the computational requirements for the motion planning subsystem.

[0101] The present disclosure provides a system and method to reduce the computational requirements for the motion planning subsystem by determining which occluded areas are unimportant, filtering out the unimportant occlusions from the map, and assigning importance scores to the filtered occluded areas. A heuristic filter first removes the unimportant occluded areas. A subsequent attention-based mechanism classifies the filtered occluded areas based on the importance of the filtered occluded areas to the vehicle's trajectory and assigns importance scores to the filtered occluded areas. The importance score indicates how important (relevant) the occluded area is to the vehicle's trajectory. The motion planning subsystem then processes only the filtered occluded areas based on the importance scores assigned to the filtered occluded areas to modify the vehicle's trajectory. The system and method of the present disclosure can be used to enhance the computations for both model-free planning and model-based planning, as well as enhance the computations for downstream tasks such as maneuvering the vehicle, providing notifications and warnings on the vehicle's human-machine interface (HMI), and so on.

[0102] The present disclosure is organized as follows. Refer to Figure 1 Examples of systems including a vehicle communicating with a remote server and system are shown and described. Refer to Figure 2 Examples of systems for computing, filtering, and scoring occlusions around a vehicle to modify the vehicle's trajectory are shown and described. Refer to Figure 3 and Figure 4 Examples of occluded areas around a vehicle, filtered occluded areas, and scoring of the filtered occluded areas are shown and described. Refer to Figure 5 Examples of methods for computing, filtering, and scoring occlusions around a vehicle to modify the vehicle's trajectory are shown and described. Refer to Figure 6 Examples of methods for filtering occlusions are shown and described. Refer to Figure 7 Examples of methods for scoring occlusions are shown and described. Refer to Figure 8 Examples of methods for training a neural network used in a scoring module to score occlusions are shown and described.

[0103] Figure 1 illustrates system 100, including vehicle 102, one or more servers 104 (e.g., located in the cloud), Global Positioning System (GPS) 105, and one or more mapping systems (hereinafter referred to as mapping systems) 106. Vehicle 102, server 104, GPS 105, and mapping system 106 communicate with each other via distributed communication system 108. For example, vehicle 102 may be an autonomous or semi-autonomous vehicle implementing the systems and methods of the present disclosure. For example, distributed communication system 108 may include one or more of a local area network (LAN), wide area network (WAN), cellular network, WiFi network, and the Internet. For example, server 104 may process some data from vehicle 102 and other vehicles ( Figure 1 not shown, but see Figure 3 and 4 the examples shown therein). Server 104 may provide information to vehicle 102 to assist in navigation and other subsystems of vehicle 102 while vehicle 102 is in motion.

[0104] Vehicle 102 includes a navigation subsystem 120, a communication subsystem 122, an infotainment subsystem 124, an autonomous subsystem 126, a steering subsystem 128, a braking subsystem 130, a plurality of sensors 132, and a propulsion subsystem 134. Navigation subsystem 120 communicates with server 104, GPS 105, and mapping system 106 via distributed communication system 108. Navigation subsystem 120 may communicate with GPS 105 directly or via communication subsystem 122. Navigation subsystem 120 implements the systems and methods of the present disclosure, as described in detail below with reference to Figure 2 continued.

[0105] Communication subsystem 122 may include one or more transceivers (e.g., cellular transceivers, WiFi transceivers, GPS receivers, and Bluetooth transceivers). The transceivers may communicate with distributed communication system 108, GPS 105, server 104, and mobile devices such as cellular phones. Communication subsystem 122 may also communicate directly with GPS 105. Additionally, communication subsystem 122 may communicate with other vehicles (not shown) using vehicle-to-vehicle (V2V) communication technology.

[0106] The navigation subsystem 120 communicates with the infotainment subsystem 124. The infotainment subsystem 124 may include a display screen (e.g., a touch screen) and multimedia devices (e.g., speakers and microphones) for audiovisual interaction with the occupants of the vehicle 102. The navigation subsystem 120 may provide maps and other audiovisual information to the occupants of the vehicle 102 via the infotainment subsystem 124. The navigation subsystem 120 may also receive audiovisual input from the occupants of the vehicle 102 via the infotainment subsystem 124. The navigation subsystem 120 may also receive input from the occupants of the vehicle 102 via a mobile device (such as a cellular phone).

[0107] The navigation subsystem 120 receives data from the sensors 132. For example, the sensors 132 may include sensors that provide the speed, heading, turn indication, etc. of the vehicle 102. The sensors 132 also include sensors located on the vehicle 102 that provide data about the environment around the vehicle 102, such as cameras, radars, lidars, and other sensors. The navigation subsystem 120 also receives mapping data (e.g., road maps, number of lanes, intersections, etc.) from the mapping system 106. The navigation subsystem 120 also receives GPS data (e.g., the location information of the vehicle 102) from the GPS receiver in the communication subsystem 122 (or directly from the GPS 105). The navigation subsystem 120 also receives data about other vehicles from the server 104. The navigation subsystem 120 adjusts the trajectory of the vehicle 102 based on all the data using the systems and methods of the present disclosure described in detail below.

[0108] The autonomous subsystem 126 controls the operation of the vehicle 102 by controlling the steering subsystem 128, the braking subsystem 130, and the propulsion subsystem 134 based on the adjusted trajectory received from the navigation subsystem 120. For example, the propulsion subsystem 134 may include a motor (not shown) that propels the vehicle 102. The propulsion subsystem 134 may also include an engine (not shown) that works in conjunction with the motor to propel the vehicle 102. The autonomous subsystem 126 controls the parameters of the motor and / or the engine according to the adjusted trajectory provided by the navigation subsystem 120.

[0109] Figure 2 An example of the navigation subsystem 120 for calculating, filtering, and scoring occlusions around the vehicle 102 according to the present disclosure to modify the trajectory of the vehicle 102 is shown. The navigation subsystem 120 includes a perception module 150, an occlusion calculation module 152, a filtering module 154, a scoring module 156, and a motion planning module 158. The modules are described in detail below.

[0110] The perception module 150 receives data about the surrounding environment of the vehicle 102 from the sensor 132, receives mapping data from the mapping system 106, receives GPS data from the GPS 105, and receives data about other vehicles from the server 104. The perception module 150 generates a map of the surrounding environment of the vehicle 102 (e.g., see Figure 3 , described below), which is a snapshot of the scene around the vehicle 102. The perception module 150 may also be referred to as a scene fusion module because it uses information from different sensors and fuses (e.g., combines or merges) the scenes captured by different sensors to generate a map.

[0111] Figure 3 FIG. shows an example map for illustrating a system and method for detecting, filtering, and scoring occlusions around the vehicle 102 to modify the trajectory of the vehicle 102 according to the present disclosure. Figure 3 The map shown in Figure 3 is just an example. Although the present disclosure is described by referring to the scene of the intersection shown in

[0112] as an example, the present disclosure is not limited thereto. Instead, the teachings of the present disclosure are applicable to any other scenario that the vehicle may encounter when traveling anywhere (e.g., the vehicle 102 passes another vehicle, the vehicle 102 is passed by another vehicle, in a parking lot and on a lane, etc.). Figure 2 In Figure 3 , the occlusion calculation module 152 includes an occlusion model that receives various inputs from a map of the surrounding environment of the vehicle 102 (e.g., the map shown in Figure 3 ). For example, the inputs to the occlusion model in the occlusion calculation module 152 include information about the objects around the vehicle 102 captured by the sensor 132 on the vehicle 102. The inputs include the state of the vehicle 102 (e.g., speed, heading, and lane) and the states of various moving and stationary objects around the vehicle 102 (such as other vehicles, buildings, and pedestrians around the vehicle 102). For example, in

[0113] the map shows the vehicle 102 and other vehicles 200-1, 200-2, 200-3, 200-4, 200-5, 200-6 (collectively referred to as other vehicles 200). For example, the map shows buildings 202-1, 202-2 (collectively referred to as buildings 202). Figure 3In [description], the map shows that vehicle 102 is traveling in the left lane of a two-lane road 210 towards another two-lane road 212. At the intersection of roads 210 and 212, the map shows a crosswalk 214.

[0114] Based on inputs from the map and the perception module 150, the occlusion model in the occlusion calculation module 152 identifies occlusion regions in the map of the environment around vehicle 102. For example, in Figure 3 [description], dashed lines are used to show the line of sight of vehicle 102. Based on the line of sight of vehicle 102, the map shows occlusion regions 220-1, 220-2, 220-3, 220-4, 220-5, 220-6, 220-7 (collectively referred to as occlusion regions 220).

[0115] The filtering module 154 preprocesses the output of the occlusion model, which is a map of the environment around vehicle 102 that includes the occlusion regions 220 identified by the occlusion model in the occlusion calculation module 152. The filtering module 154 filters out unimportant occlusion regions from the map of the environment around vehicle 102 as follows.

[0116] The filtering module 154 performs preprocessing for two reasons: first, to reduce the number of occlusion regions to the maximum number of occlusions to be tracked; and second, to eliminate the calculation of irrelevant occlusions. In addition to the output of the occlusion model, which is a map of the environment around the vehicle that includes the occlusion regions identified by the occlusion model, the filtering module 154 also receives various inputs. For example, additional inputs received by the filtering module 154 include the route, status, and current trajectory of vehicle 102; the status of moving objects (e.g., other vehicles 200 and pedestrians) around vehicle 102, and predictions (e.g., from server 104) regarding the movement of objects (e.g., vehicles 200) around vehicle 102 (e.g., where an object (e.g., vehicle 200) is about to be located relative to vehicle 102 (e.g., within the next few seconds)).

[0117] The filtering module 154 filters out irrelevant (unimportant) occlusion regions from the map based on heuristics. For example, the heuristics can include occlusions that do not intersect the route of vehicle 102, the size of the occlusion regions 220, and the proximity (e.g., distance) of the occlusion regions 220 to vehicle 102. For example, the heuristics can include occlusion regions (e.g., occlusion regions 220-1, 220-2) that are relatively far from vehicle 102, which can be ignored and filtered out from the map. In some examples, if all of the occlusion regions 220 are considered important (relevant) to the trajectory of vehicle 102, then none of the occlusion regions 220 are filtered out. Thus, in general, the filtering module 154 can filter out one or more of the occlusion regions 220 or filter out none of the occlusion regions 220 from the map.

[0118] In other examples, the heuristics can include occlusion regions caused by stationary objects such as a building at an intersection on the downstream portion of a one-way street where vehicle 102 is about to turn, and such occlusion regions can be ignored and filtered out. For example, the heuristics can include multiple occlusion regions on the lanes of vehicle 102 in front of and behind vehicle 102, and the farther regions among the multiple occlusion regions can be filtered out. For example, the heuristics can include the time evolution of occlusions and objects around vehicle 102. For example, if there is a closer leading vehicle in the same lane, the filtering module 154 can filter out the occlusion in front of vehicle 102 in the same lane. The filtering parameters used by the filtering module 154 can be adjusted based on the type of vehicle 102 (e.g., sedan, pickup truck, recreational vehicle (RV), etc.) and / or the possible presence of vulnerable road users (VRUs) in the occlusion region 220).

[0119] After preprocessing (filtering) the occlusion regions 220 in the map, the filtering module 154 outputs a map with filtered occlusion regions. Figure 4 An example of a map with filtered occlusion regions is shown. In Figure 4 as explained above,[[]]END]] Figure 3 the occlusion regions 220-1, 220-2 shown in Figure 4 are filtered out. Thus, the occlusion regions 220-3 to 220-7 that remain in the map (i.e., are not filtered out from the map) are referred to as filtered occlusion regions. Although not so labeled in Figure 6 for ease of the following description, the filtered occlusion regions 220-3 to 220-7 are referred to as filtered occlusion regions 221. A method of filtering occlusions is shown and described below with reference to

[0120] The scoring module 156 scores the filtered occlusion regions 221 in the map of the environment around vehicle 102 output by the filtering module 154. The scoring module 156 scores the filtered occlusion regions 221 to indicate the importance of each filtered occlusion region 221 that is critical to the trajectory of vehicle 102. For example, in Figure 3 and Figure 4 an example of the trajectory of vehicle 102 is shown at 230. The preprocessing (filtering) and scoring steps are performed before the motion planning module 158 processes the filtered occlusion regions 221 based on the importance scores of the filtered occlusion regions 221 to determine whether to change the trajectory 230 of vehicle 102, as described below.

[0121] The scoring module 156 includes a neural network called a deep importance network for occlusions (DINO), which is trained to score the importance of the filtered occlusion regions 221. A reference is made below toFigure 8 Describe the training method technology of DINO. The trained DINO utilizes an attention-based mechanism that receives input features regarding the trajectory 230 of vehicle 102 and regarding the filtered occlusion region 221, which is described below with reference to Figure 7 as follows. As described in the following reference Figure 7 the trained DINO uses an attention-based mechanism to generate an importance score for the filtered occlusion region 221. The motion planning module 158 can then change the trajectory 230 of vehicle 102 based on the importance score of the filtered occlusion region 221.

[0122] In reference Figure 7 the operation of the scoring module 156 is described in detail and in reference Figure 8 before describing in detail the method for training DINO, in reference Figure 5 an example of a method 250 executed by the navigation subsystem 120 of vehicle 102 is shown and described. The following description of method 250 briefly and broadly embodies (summarizes) the operations performed by each module of the navigation subsystem 120 of vehicle 102.

[0123] In Figure 5 at 252, method 250 (e.g., using the sensors 132 of vehicle 102) senses the surrounding environment of vehicle 102. At 254, method 250 (e.g., using the perception module 150) generates a map (snapshot) of the surrounding environment of vehicle 102. At 256, method 250 (e.g., using the occlusion calculation module 152) calculates the occlusion region 220 in the map. At 258, method 250 (e.g., using the filtering module 154) filters out unimportant occlusion regions from the map based on heuristics. At 260, method 250 (e.g., using the scoring module 156) generates an importance score for the filtered occlusion region 221. At 262, method 250 (e.g., using the motion planning module 158) modifies the trajectory of vehicle 102 based on the selected importance score.

[0124] Figure 6Illustrates method 280 performed by filtering module 154. The operation of filtering module 154 has been described in detail above. In the following description of method 280, the operation of filtering module 154 is summarized. At 282, filtering module 154 receives a map (snapshot) of the environment around vehicle 102 that includes occlusion region 220. At 284, filtering module 154 receives inputs including the route, status, and current trajectory of vehicle 102; the status of moving objects (e.g., other vehicles 200, pedestrians, etc.) around vehicle 102; and predictions regarding the movement of objects (e.g., other vehicles 200) around vehicle 102. At 286, filtering module 154 uses heuristics to consider, for example, occlusions that do not intersect the route of vehicle 102, the size of occlusion region 220, and the proximity (e.g., distance) of occlusion region 220 to vehicle 102, as well as the temporal evolution of the occlusions and the objects around vehicle 102. For example, the heuristics are embedded (e.g., encoded) in filtering module 154. At 288, filtering module 154 filters out (i.e., removes) unimportant occlusion regions from the map based on the inputs and heuristics, and determines a filtered occlusion region 221 that is relevant to the trajectory of vehicle 102.

[0125] Generally, the scene around vehicle 102 (i.e., the map of the environment around vehicle 102) can change dynamically. For example, as vehicle 102, other vehicles 200, and other objects (such as cyclists and pedestrians) continue to move, the positions of other vehicles 200 and the corresponding occlusion regions 220 can change. Thus, the process of detecting occlusion region 220, determining filtered occlusion region 221, calculating an importance score for filtered occlusion region 221, and changing the trajectory 230 of vehicle 102 based on the importance score can be repeated periodically (e.g., every second).

[0126] In addition, the modification of the trajectory 230 of vehicle 102 can generate alternative trajectories from which the autonomous subsystem 126 of vehicle 102 can select a trajectory. For example, if the trajectory of vehicle 102 is for a left turn, the modified trajectory can be used to perform the left turn differently (e.g., faster, slower, narrower, wider) than planned in the original trajectory, or to cause vehicle 102 to come to a complete stop. Thus, the change in the trajectory can include alternative decisions.

[0127] Figure 7 Method 300 performed by scoring module 156 is shown in detail. The attention-based mechanism utilized by the trained DINO in scoring module 156 receives specific inputs regarding vehicle 102 and filtered occlusion region 221, and generates an importance score for filtered occlusion region 221, as described in detail below.

[0128] For example, DINO in the scoring module 156 includes multiple embedders (e.g., neural networks) that receive inputs as follows. For vehicle 102, DINO in the scoring module 156 includes a query embedder, a key embedder, and a value embedder. For each filtered occlusion area 221, DINO in the scoring module 156 includes a key embedder and a value embedder. For the filtered occlusion area 221, the key embedder and the value embedder are shared by or among the filtered occlusion areas 221. The key embedder and the value embedder of vehicle 102 are different from the key embedder and the value embedder shared by the filtered occlusion areas 221.

[0129] In addition, DINO in the scoring module 156 does not use a query embedder for the filtered occlusion areas 221 because the relationships between the filtered occlusion areas 221 are not relevant to scoring the filtered occlusion areas 221. Instead, the spatial relationship between each filtered occlusion area 221 and vehicle 102 is relevant to scoring the filtered occlusion areas 221. Therefore, DINO in the scoring module 156 uses the query embedder of vehicle 102 but does not use the query embedder of the filtered occlusion areas 221.

[0130] Therefore, DINO in the scoring module 156 can include five embedders: a query embedder, a key embedder, and a value embedder for vehicle 102 (3 embedders), plus a key embedder and a value embedder shared among the filtered occlusion areas 221 (2 embedders). For example, each of the five embedders can be a separate neural network. The neural networks of DINO in the scoring module 156 receive the following features as inputs and generate the following vectors as outputs. For each neural network, the number of layers, the width of each layer, and the associated activation function are user-defined (i.e., selectable) parameters.

[0131] For example, let N denote the maximum number of filtered occlusion areas 221 to be tracked, where N is an integer greater than 1 and is selectable. In Figure 7 In the method 300 shown in, at 302, for vehicle 102, the query embedder, the key embedder, and the value embedder receive input features extracted from the trajectory 230 of vehicle 102. For example, the input features can include (x, y, v, a), where x and y are the 2D coordinates of vehicle 102, v is the speed, and a is the acceleration.

[0132] At 304, for each filtered occlusion area 221, the key embedder and the value embedder shared among the filtered occlusion areas 221 receive the input features of the filtered occlusion area 221. For example, the input features of the filtered occlusion area 221 include (s0, s end , l), where S o and S endare the starting and ending longitudinal coordinates of the filtered occlusion region 221, and l is information about the lane in which occlusion occurs.

[0133] At 306, the query embedder of vehicle 102 generates an output vector of dimension N Q The key embedder of vehicle 102 generates an output vector of dimension N Q The value embedder of vehicle 102 generates an output vector of dimension N V At 308, the key embedder of the filtered occlusion region 221 generates an output vector of dimension N Q The value embedder of the filtered occlusion region 221 generates an output vector of dimension N V of the output vector.

[0134] Subsequently, in DINO, the keys and values from vehicle 102 and the filtered occlusion region 221 are concatenated. For example, at 310, the key from vehicle 102 and the key from the filtered occlusion region 221 are concatenated to form a key matrix K of dimension (N + 1)×N Q At 312, the value from vehicle 102 and the value from the filtered occlusion region 221 are concatenated to form a value matrix V of dimension (N + 1)×N v of the value matrix.

[0135] At 314, a query matrix Q of dimension 1×Nq including the output of the query embedder of vehicle 102 and the transpose of the key matrix K including the outputs of the key embedders of vehicle 102 and the filtered occlusion region 221 are multiplied to generate an attention matrix of dimension 1×(N + 1). At 316, the attention matrix is divided by sqrt(N Q ). At 318, the softmax operator is applied row by row to the attention matrix; and at 320, the resulting matrix after applying the softmax operator is multiplied by the value matrix V including the outputs of the value embedders of vehicle 102 and the filtered occlusion region 221 to generate an output matrix of DINO with dimension 1×N V of the output matrix.

[0136] To infer the importance score of the filtered occlusion region 221 using the output of DINO, one of the following two methods can be selected. The first method forces N V = N. Using the first method, the output of the multiplication between the attention matrix and the value matrix has the correct dimension (i.e., the maximum number N of occlusion regions). In the second method, another layer or sequence of layers (referred to as a head) can be added to DINO. The head receives 1×N VThe vector (i.e., the result of the multiplication between the attention matrix and the value matrix) is used as input and generates an output of dimension 1×N. Similarly, like the neural network of the embedder, the structure (number of layers, layer width, and activation function) of the head can be user-defined (i.e., selectable). Thus, at 322, either N is forced to be equal to N, or the output matrix of DINO is input into the head to generate an output of dimension 1×N. At 324, the importance scores of the N filtered occluded regions 221 are inferred from the output of dimension 1×N. V At 324, the importance scores of the N filtered occluded regions 221 are inferred from the output of dimension 1×N.

[0137] Subsequently, the motion planning module 158 can select the filtered occluded regions 221 having importance scores greater than or equal to a selectable threshold and ignore the filtered occluded regions having importance scores less than the selectable threshold. The motion planning module 158 only processes those selected from the filtered occluded regions 221, which reduces the computational load of the motion planning module 158 and can change the trajectory 230 of the vehicle 102 based on those selected from the filtered occluded regions 221. Alternatively or additionally, the motion planning module 158 can only process those selected from the filtered occluded regions 221 and generate one or more alternative trajectories of the vehicle 102 based on those selected from the filtered occluded regions 221.

[0138] Figure 8 A method 350 for training DINO used in the scoring module 156 is shown in detail. For example, the DINO in the scoring module 156 is trained using reinforcement learning as follows. During training, as described above, the embedder (neural network) of DINO receives input features regarding the trajectory of the vehicle 102 and the occluded regions 220. For example, at 352, for the vehicle 102, the query embedder, key embedder, and value embedder receive input features extracted from the trajectory of the vehicle 102. For example, the input features can include (x, y, v, a), where x and y are the 2D coordinates of the vehicle 102, v is the speed, and a is the acceleration. For example, at 354, the key embedder and value embedder shared among the occluded regions 220 receive the input features of the occluded regions 220. For example, the input features of the occluded regions 220 include (s0, s end , l), where s o and s end are the starting and ending longitudinal coordinates of the filtered occluded region 221, and l is the information about the lane where the occlusion occurs.

[0139] At 356, the DINO in the scoring module 156 generates an output including the importance scores of the occluded regions 220. Based on the importance scores generated by DINO, the motion planning module 158 generates an action sequence (e.g., speed, acceleration, lane change, turn, etc.) for the trajectory of the vehicle 102.

[0140] At 358, DINO is trained using a return that includes two components. The return that includes two components is used to adjust the weights and biases of the neural network in DINO. The two components are balanced with each other such that the net return does not increase the computational load on the motion planning module 158 as follows, and also does not compromise the comfort, safety, and speed of the vehicle 102.

[0141] The first component of the return signal is a baseline return generated by the motion planning module 158. The motion planning module 158 generates the first component based on factors such as how fast the vehicle 102 can complete a trajectory, how much comfort (e.g., jerk) the trajectory can generate when the vehicle 102 completes the trajectory, and how safely the vehicle 102 can complete the trajectory. The motion planning module 158 can generate the first component by optimizing one or more of these factors. For example, the motion planning module 158 can emphasize one factor (e.g., comfort) while de-emphasizing another factor (e.g., speed). For example, the motion planning module 158 can use an equation or formula to maximize the baseline return such that the vehicle 102 can execute the trajectory at a speed with maximum comfort and safety.

[0142] In some examples, if only the baseline return component is used to train DINO, the number of occluded regions detected as important may increase, which in turn increases the computational load on the motion planning module 158. Accordingly, the present disclosure adds a second return component to the training signal used to train DINO. The second return component balances or cancels the baseline return component as follows.

[0143] For example, the second return component can be generated by summing the importance scores of the occluded regions, multiplying the sum of the importance scores by a negative coefficient, and adding the negative product to the first baseline return component. Alternatively, the second return component can be generated by only selecting those importance scores that are greater than a selected threshold. For example, the importance scores of occluded regions with high importance scores due to a pedestrian entering the road, due to an oncoming vehicle (such as an emergency vehicle) approaching the vehicle 102, etc. can be selected. Then, the second return component can be generated by summing the selected high importance scores, multiplying the sum by a negative coefficient, and adding the negative product to the first baseline return component. The second return component reduces the first return component such that DINO is not disproportionately biased to maximize factors such as comfort, safety, and speed of the vehicle 102 in the baseline return.

[0144] Thus, at 358, the return signal for training DINO includes the sum of a baseline return component and another return component that includes the product of a negative factor and the sum of importance scores of occluded regions. At 360, the learning signal from the return is used to adjust the weights and biases of five embedders (neural networks). The adjustment continues until DINO is trained. For example, training stops when the user decides to stop training based on different metrics such as the number of updates to the weights and biases of the neural network (embedder). As described above, the trained DINO is then used in the scoring module 156 of the navigation subsystem 120 of vehicle 102.

[0145] Figure 9 is a block diagram of an example system 400 for generating a human-machine interface alert based on detected occluded regions. System 400 can be executed by a vehicle control module 401 of a vehicle, for example. As Figure 9 shown, the vehicle control module receives a driver monitoring system (DMS) input 402, which can be used to determine the direction of the driver's attention.

[0146] For example, the vehicle can include one or more gaze tracking devices, head tracking devices, etc., which are configured to determine the direction of the driver's gaze or the orientation of the driver's head. This information can be used by the vehicle control module 401 to determine what direction the driver is looking in order to estimate the direction of the driver's attention or focus.

[0147] The vehicle control module 401 is configured to receive an occlusion data input 404, which can include egocentric occluded region data (e.g., data identifying occluded regions from the perspective of the vehicle driver). The occlusion data can include the location of the occlusion, the importance score of the occlusion, etc.

[0148] In various implementations, the example subsystems described herein can be used to detect occluded regions (e.g., based on vehicle sensors detecting one or more objects in the vehicle's surrounding environment), map occluded regions, filter occluded regions, generate scores for occluded regions, etc. The vehicle control module 401 can use this occlusion data as the occlusion data input 404 to determine alert generation 412.

[0149] As Figure 9 shown, the vehicle control module 401 is configured to receive a vehicle path prediction input 406. The vehicle path prediction input 406 can include data about the vehicle's current trajectory, predicted future trajectory of the vehicle, etc.

[0150] The vehicle path prediction input 406 can be generated using any suitable vehicle data, such as the current vehicle gear of the vehicle (e.g., forward or reverse), turn signal status (e.g., left or right turn signal), current speed, and heading, etc. The vehicle data can be classified into one of multiple vehicle action categories, such as the vehicle continuing straight, the vehicle reversing, a left lane change, a right lane change, a left turn, a right turn, etc.

[0151] The vehicle control module 401 optionally receives a driver historical data input 408, which can include a historical driver attention score, a historical driver response score to a human-machine interface (HMI) alert, etc. For example, the driver historical data input 408 can include the scores and response history of each historical driver of the vehicle (or other vehicles), which indicates how attentive the driver has been to occlusions in the past.

[0152] In various implementations, the driver historical score can be calculated differently for each HMI device. For example, the driver has a higher score in responding to seat vibration compared to a notification on a head-up display (HUD). For example, a driver monitoring system (DMS) camera can be used to implement face detection to match the stored driver historical score with a specific corresponding driver.

[0153] As Figure 9 shown, the occlusion data input 404 can be combined with the vehicle path prediction input 406 to perform HMI device selection 410. For example, the vehicle control module 401 can be configured to select which HMI device to use to alert the driver of the occlusion area based on the position of the occlusion area and the current trajectory of the vehicle.

[0154] Example HMI alert devices can include, but are not limited to, a head-up display (HUD) in the vehicle, an in-vehicle display, side mirrors, rearview mirrors, seat haptics (e.g., seat vibration), and an alert sound via the vehicle speakers. For example, if the predicted vehicle path is straight and the occlusion position is on the right or left side of the vehicle, the HMI device selection can include a head-up display and the left or right mirror (e.g., generating a visual alert on the side mirror on the same side as the occlusion to increase the likelihood that the driver notices the occlusion).

[0155] As another example, if the predicted vehicle path is a right lane change and the occlusion position is on the right side of the vehicle, the HMI device selection 410 can include a HUD, seat vibration, and a right mirror alert. If the driver is planning to move the vehicle to the right, providing seat vibration can provide an enhanced warning to alert the driver of the occlusion on the side of the vehicle. Similarly, if the predicted vehicle path is a left lane change and the occlusion position is on the left side of the vehicle, the HMI device selection 410 can include a HUD, seat vibration, and a left rearview mirror alert.

[0156] As another example, if the predicted vehicle path is in reverse and the occlusion location is on the left rear or right rear side of the vehicle, the HMI device selection 410 may include the front console and the rearview mirror. Since the driver may be more likely to look at the rearview mirror when reversing, or at the backup camera screen on the front console, providing an occlusion alert at these locations may increase the chance that the driver notices the alert (e.g., compared to a HUD or sideview mirror that the driver may not be looking at during a reverse maneuver).

[0157] As Figure 9 shown, alert generation 412 is determined based on the driver monitoring system input 402, the occlusion data input 404, the HMI device selection 410, and optionally the driver history data input 408. For example, the vehicle control module 401 may cluster the occlusion zones into regions around the vehicle.

[0158] Figure 10 is a diagram illustrating an example clustering around vehicle 502 for grouping detected occlusion zones. Figure 10 Illustrates six example zones: a right front (FR) cluster 504, a left front (FL) cluster 506, a right side (SR) cluster 508, a left side (SL) cluster 510, a right rear (RR) cluster 512, and a left rear (RL) cluster 514. Other example embodiments may include more or fewer clusters. The importance score for each cluster can be calculated as the maximum of the occlusion scores (or occlusion importance scores) of all the occlusion zones within that cluster.

[0159] The HMI score for each cluster can be updated based on multiple factors. For example, a higher occlusion importance score can result in a higher HMI score for that cluster. Optionally, when the driver history data indicates that the driver has had low historical attention or response to similar HMI alerts in the past, the cluster can receive a higher HMI score. If, based on the driver monitoring system input 402, the driver's gaze direction or head orientation is not in the direction of the occlusion cluster, the HMI score can be increased.

[0160] Based on the HMI device selection 410 and the occlusion cluster zones, the type of HMI alert can be different. In some examples, the HMI alert can include text, symbols, audio, seat vibration, etc. In various implementations, a coordinate transformation from the vehicle center view to the driver's eye center view can be performed.

[0161] If the HMI score exceeds a specified threshold after being modified based on the cluster importance score, driver history data, driver gaze and head orientation data, etc., the vehicle control module 401 can perform alert generation 412. Further details regarding an example process for determining HMI alert generation are discussed below with reference to Figure 11 additional details regarding an example process for determining HMI alert generation.

[0162] Figure 11 is a flowchart depicting an example process for generating an HMI alert based on a detected occlusion zone. Figure 11 The process can be implemented by, for example, Figure 9 vehicle control module 401. At 604, the process begins with detecting that the vehicle is in motion.

[0163] At 608, the vehicle control module is configured to identify occlusion zones in the driving environment (e.g., using vehicle sensors). Then at 612, the vehicle control module scores the occlusion zones by importance. Example processes for identifying and scoring occlusion zones are discussed further above.

[0164] At 616, the vehicle control module is configured to group the occlusion zones into regions around the vehicle. For example, the vehicle control module can group the occlusion zone locations and scores based on Figure 10 the six clusters illustrated in.

[0165] At 620, the control is configured to determine whether the importance score is greater than a high importance score threshold. If so, the control adds a high level value to the HMI alert score at 624. For example, if one of the clusters has an importance score indicating a high importance occlusion in that cluster, the likelihood of generating an HMI alert increases (e.g., by increasing the HMI alert score towards or above a specified HMI alert generation threshold).

[0166] At 628, the vehicle control module is configured to determine whether any of the clusters has an importance score below a low importance score threshold. If not (indicating that the cluster has an importance score between the high and low importance score thresholds), the control proceeds to 632 to add a medium level value to the HMI score.

[0167] At 636, the control determines whether the driver history score is less than a low threshold, which indicates low historical driver attention or alert responsiveness. If so, the control adds a high level value to the HMI score at 640 to increase the likelihood of generating an alert for the driver.

[0168] At 644, the vehicle control module is configured to determine whether the driver history score is greater than a high threshold. If not (indicating that the driver has had a medium level of attention or alert responsiveness historically), the control proceeds to 648 to add a medium level value to the HMI score.

[0169] At 652, the vehicle control module is configured to track the driver's eye direction and / or head orientation via a driver monitoring system. Then, at 656, the control determines whether the driver's gaze or head orientation is directed towards the direction of the occlusion zone. If not, a specified orientation negligence value is added to the HMI score. For example, if the driver is already looking in the direction of the occlusion zone, it may not be necessary to generate an HMI alert for the driver. If the driver is looking in a different direction from the detected occlusion zone, the vehicle control module is more likely to generate an HMI alert (e.g., due to a greater safety risk if the driver is looking in a different direction).

[0170] At 664, the control is configured to compile an overall HMI score. For example, the overall HMI score can include an initial HMI score calculated for each cluster, and then, if the occlusion zone has high importance, if the driver has historically low attention or alert responsiveness, if the driver is not looking in the direction of the occlusion, etc., the initial HMI score is modified by increasing the HMI score.

[0171] At 668, the vehicle control module is configured to determine the appropriate method and level for delivering an alert. For example, depending on the overall HMI score, the vehicle control module can determine whether to display only a visual notification, increase the intensity of the visual notification via size / color or blinking, generate a tactile seat alert, generate an audio alert via one or more speakers, etc.

[0172] At 672, the control generates the determined HMI alert, such as by providing the HMI alert at the HMI output device 416. The control then proceeds to 676 to apply HMI alert escalation if necessary. For example, after generating an HMI alert for a specified occlusion zone or cluster, the type and intensity of the HMI alert can be adapted or escalated.

[0173] In some example embodiments, the HMI alert can be escalated based on the driver's attention direction. For example, the vehicle control module 401 can use the driver's gaze direction and / or head orientation detected by the driver monitoring system to implement alert escalation 414 based on whether the driver is paying attention to the direction of the occlusion zone.

[0174] The alert can be escalated based on an occlusion importance score. For example, if the importance score of the occlusion zone increases over time (e.g., the driver's actions are inappropriate for the occlusion, such as the vehicle continuing to approach the occlusion zone at a high speed), the corresponding HMI alert can be escalated. Further details regarding an example process for escalating HMI alerts are discussed below with reference to Figure 12 Further details regarding an example process for escalating HMI alerts are discussed below.

[0175] Figure 12 depicts an example process for escalating an HMI alert based on Figure 11Flowchart of an example process for generating HMI alerts generated by the process Figure 12 The process can be implemented by, for example Figure 9 the vehicle control module 401. At 704, the process begins by transmitting an HMI alert to one or more HMI devices of the vehicle.

[0176] While the HMI alert is being generated, the control determines whether the driver has turned their gaze direction or head orientation towards the alert area. If so, the control proceeds to 716 to end the HMI alert. For example, once the driver turns their attention to the HMI alert (or the detected occlusion area), the likelihood that the driver is now in the occlusion area increases, and the HMI alert can end. Then, at 720, the control returns to identifying occlusion areas in the driving environment.

[0177] If at 708 the driver does not direct their attention towards the alert area, the control proceeds to 712 to determine whether the importance score of the occlusion area has dropped below a specified threshold (e.g., because the vehicle has passed the occlusion area, etc.). If so, the control ends the HMI alert at 716.

[0178] If at 708 the driver does not direct their attention towards the alert area and at 712 the occlusion importance score is still above the threshold, the control proceeds to 726 to determine whether the time count has exceeded a first time threshold. For example, if the driver continues to look in a direction other than the occlusion area and the occlusion area remains at a high importance score, the vehicle control module can be configured to increase the intensity of the HMI alert every second, every three seconds, every ten seconds, in time periods less than one second, etc.

[0179] If at 724 the time count has not exceeded the first threshold, the control proceeds to 744 to increase the visual size and / or the color of the HMI alert. Then, the control increments the time count at 760 within a specified time period.

[0180] If at 724 the time count has exceeded the first threshold, the control proceeds to 728 to determine whether the time count has exceeded a second threshold (longer than the first threshold). If not, the control proceeds to 748 to add a visual blink to the displayed HMI alert.

[0181] If at 728 the time count has exceeded the second threshold, the control proceeds to 732 to determine whether the time count has exceeded a third threshold (longer than the first and second thresholds). If not, the control proceeds to 752 to add a directional haptic seat pulse to the HMI alert.

[0182] If the time count exceeds a third threshold at 732, control proceeds to 736 to determine if the time count exceeds a fourth threshold (longer than the first, second, and third thresholds). If not, control proceeds to 756 to add directed (or non-directed) audio to the HMI alert.

[0183] If the time count exceeds the fourth threshold at 736, control may activate an advanced driver assistance system to attempt to avoid the occlusion zone or any object that may emerge from the occlusion zone. Example details for controlling the automated driving of a vehicle to avoid an object from an occlusion zone were discussed above.

[0184] If the HMI alert includes a visual alert, any suitable visual component of the vehicle may be used to display the visual alert. For example, the HMI alert may include an augmented reality HUD or a full windshield augmented reality (AR) display, a conventional HUD, an overhead display, or a reflective HUD, an in-vehicle display AR HUD, etc.

[0185] In various implementations, the visual alert may be upgraded by increasing the visual alert intensity in terms of size, color, contrast, brightness, blink frequency, etc. The audio alert may be upgraded by increasing the intensity in terms of frequency, tone, amplitude, localization, time period, etc. The haptic alert may be upgraded by increasing the intensity in terms of amplitude and frequency, by changing the waveform pattern, etc. Directed audio alerts and directed haptic seat alerts may be implemented, for example, by generating an audio alert via a speaker located on one side of the vehicle, generating a vibration on the driver seat side, etc.

[0186] The foregoing description is merely illustrative in nature and is not intended to limit the present disclosure, its application, or uses. The broad teachings of the present disclosure may be implemented in a variety of forms. Thus, while the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited since other modifications will become apparent after study of the drawings, the specification, and the following claims.

[0187] It should be understood that, without changing the principles of the present disclosure, one or more steps within a method may be performed in a different order (or simultaneously). Additionally, although each embodiment is described above as having certain features, any one or more of those features described with respect to any embodiment of the present disclosure may be implemented in any other embodiment and / or combined with the features of any other embodiment, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and arrangements of one or more embodiments with each other are still within the scope of the present disclosure.

[0188] The spatial and functional relationships between components (e.g., between modules, subsystems, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "joined," "coupled," "adjacent," "immediately adjacent," "on," "above," "below," and "disposed." Unless explicitly described as "direct," when describing the relationship between a first and a second component in the foregoing disclosure, the relationship can be a direct relationship in which no other intervening component exists between the first and second components, but can also be an indirect relationship in which one or more intervening components (either spatially or functionally) exist between the first and second components. As used herein, the phrase "at least one of A, B, and C" should be interpreted to mean a logical (A or B or C) using a non-exclusive logical "or," and should not be interpreted to mean "at least one of A, at least one of B, and at least one of C."

[0189] In the various figures, the direction of the arrows—as indicated by the arrowheads—generally indicates the information flow (such as data or instructions) of interest for that figure. For example, when components A and B exchange various information, but the information transmitted from component A to component B is relevant to that figure, the arrow can point from component A to component B. This one-way arrow does not imply that no other information is transmitted from component B to component A. Additionally, for the information sent from component A to component B, component B can send a request for that information or receive an acknowledgment.

[0190] In this application, including the definitions below, the term "module," the term "controller," or the term "subsystem" can be replaced with the term "circuit." The term "module" or the term "subsystem" can refer to, be part of, or include the following: application specific integrated circuit (ASIC); digital, analog, or mixed analog / digital discrete circuits; digital, analog, or mixed analog / digital integrated circuits; combinational logic circuits; field programmable gate arrays (FPGA); processor circuits that execute code (shared, dedicated, or grouped); memory circuits that store code executed by the processor circuits (shared, dedicated, or grouped); other suitable hardware components that provide the described functionality; or a combination of some or all of the foregoing, such as in a system-on-chip.

[0191] A module or subsystem can include one or more interface circuits. In some examples, the interface circuits can include wired or wireless interfaces connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module or subsystem of the present disclosure can be distributed among multiple modules or subsystems connected via the interface circuits. For example, multiple modules or subsystems can allow load balancing. In a further example, a server (also referred to as a remote server or cloud) can implement some functionality on behalf of a client module or subsystem.

[0192] As used above, the term "code" can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" encompasses a single processor circuit that executes some or all of the code from multiple modules or subsystems. The term "group processor circuit" encompasses a processor circuit that executes some or all of the code from one or more modules or subsystems in combination with additional processor circuits. References to multiple processor circuits encompass multiple processor circuits on separate die, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" encompasses a single memory circuit that stores some or all of the code from multiple modules or subsystems. The term "group memory circuit" encompasses a memory circuit that stores some or all of the code from one or more modules or subsystems in combination with additional memory.

[0193] The term "memory circuit" is a subset of the term "computer-readable medium". As used herein, the term "computer-readable medium" does not encompass transitory electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); thus, the term "computer-readable medium" can be considered tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0194] The apparatus and methods described in this application can be implemented in part or in whole by a special-purpose computer created by configuring a general-purpose computer to execute one or more specific functions embodied in a computer program. The functional blocks, flowchart components, and other elements described above serve as software specifications that can be transformed into a computer program by the routine work of a skilled technician or programmer.

[0195] A computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. The computer program may also include or rely on stored data. The computer program may cover a basic input / output system (BIOS) that interacts with the hardware of a dedicated computer, device drivers that interact with specific devices of a dedicated computer, one or more operating systems, user applications, background services, background applications, and the like.

[0196] A computer program may include: (i) descriptive text to be parsed, such as HTML (HyperText Markup Language), XML (eXtensible Markup Language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated by a compiler from source code, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, the source code may be written using the syntax of languages including: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Fifth Edition of the HyperText Markup Language), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK and

Claims

1. A vehicle control system, comprising: A plurality of sensors configured to sense the surrounding environment of the vehicle; A driver monitoring system configured to detect at least one of a gaze direction and a head orientation of a vehicle driver; A human-machine interface configured to provide at least one of a visual alert, a tactile alert, and an audible alert to the driver; And A vehicle control module configured to: Identify at least one occlusion zone based on the plurality of sensors, each occlusion zone indicating an area around the vehicle that is occluded by one or more objects detected by the plurality of sensors; Determine a predicted trajectory of the vehicle; Score the at least one occlusion zone based on the importance of the at least one occlusion zone to the predicted trajectory of the vehicle; Determine a driver attention direction based on the detection of at least one of a gaze direction and a head orientation of the vehicle driver; And Generate an occlusion zone alert via the human-machine interface based on the score of the at least one occlusion zone and the driver attention direction.

2. The vehicle control system according to claim 1, wherein the occlusion zone alert comprises at least one of the following: A visual notification on a vehicle display at a position corresponding to the direction of the at least one occlusion zone; A directional tactile seat alert including vibration of a part of the vehicle seat corresponding to the direction of the at least one occlusion zone; or A directional audio alert including audio generated by at least one vehicle speaker at a speaker position corresponding to the direction of the at least one occlusion zone.

3. The vehicle control system according to claim 2, wherein the occlusion zone alert comprises all visual notifications, directional tactile seat alerts, and directional audio alerts.

4. The vehicle control system according to claim 1, wherein the vehicle control module is configured to: Generate a map of the vehicle surrounding environment, the map including one or more objects around the vehicle detected by the plurality of sensors; and Selectively filter the at least one occlusion zone from the map, the map including one or more filtered occlusion zones after the selective filtering, wherein scoring the at least one occlusion zone comprises scoring the one or more filtered occlusion zones based on the importance of the one or more filtered occlusion zones to the predicted trajectory of the vehicle.

5. The vehicle control system according to claim 4, wherein the vehicle control module is configured to: Modify the trajectory of the vehicle based on the importance score of the filtered occlusion zone; and Advance the vehicle according to the modified trajectory using an advanced driver assistance system.

6. The vehicle control system according to claim 1, wherein: The vehicle control module is configured to obtain a driver history associated with the vehicle driver; The driver history includes at least one of a historical driver attention score and a historical driver alert response score; And Generate the occlusion zone alert at least in part based on the driver history.

7. The vehicle control system according to claim 1, wherein the vehicle control module is configured to: During generating the occlusion zone alert, determine an updated score of the at least one occlusion zone and detect an updated driver attention direction; and Terminate the occluded area alert in response to at least one of the updated score falling below the alert generation threshold, the updated driver attention direction corresponding to the direction of the at least one occluded area, or the updated driver attention direction corresponding to the direction of the occluded area alert.

8. The vehicle control system according to claim 7, wherein the vehicle control module is configured to upgrade the intensity of the occluded area alert in response to the updated score remaining above the alert generation threshold and the updated driver attention direction not corresponding to the direction of the at least one occluded area or the direction of the occluded area alert.

9. The vehicle control system according to claim 8, wherein the vehicle control module is configured to upgrade the intensity of the occluded area alert by increasing the visual size of the visual notification alert on the vehicle display and changing the visual color.

10. A method for controlling a vehicle human-machine interface alert, the method comprising: Identifying at least one occluded area around the vehicle based on a plurality of sensors configured to sense the vehicle's surrounding environment, each occluded area indicating an area around the vehicle that is occluded by one or more objects detected by the plurality of sensors; Determining a predicted trajectory of the vehicle; Scoring the at least one occluded area based on the importance of the at least one occluded area to the predicted trajectory of the vehicle; Determining a driver attention direction based on detection of at least one of a gaze direction and a head orientation of the vehicle driver by a vehicle driver monitoring system; And Generating an occluded area alert via the vehicle's human-machine interface based on the score of the at least one occluded area and the driver attention direction, wherein the human-machine interface is configured to provide at least one of a visual alert, a tactile alert, and an audio alert to the driver.

Images