Enhancing motion planning for autonomous and semi-autonomous vehicles based on occluded regions

Abstract

A system for a vehicle includes a sensor that senses an environment of the vehicle. The perception module generates a map of the surroundings of the vehicle. The map includes objects around the vehicle. The occlusion calculation module calculates an occluded area in the map. The shielded area is an area, which is shielded by one or more objects around the vehicle, around the vehicle. The filtering module does not filter the occluded area or filters one or more occluded areas from a map. After filtering, the map includes a plurality of filtered obscured areas. A scoring module scores the filtered occluded region based on an importance of the filtered occluded region to the trajectory of the vehicle. A motion planning module modifies the trajectory of the vehicle based on the importance score of the filtered occluded region. A propulsion module propels the vehicle according to the modified trajectory.

Description

[0001] introduce

[0002] The information provided in this section is for the purpose of generally presenting the context of the present disclosure. The work of the inventors currently listed (to the extent that it is described in this section) and aspects of the description that may not have been prior art at the time of filing are neither explicitly nor implicitly admitted as prior art against the present disclosure.

[0003] The present disclosure relates generally to autonomous and semi-autonomous vehicles, and more particularly to enhancing motion planning of autonomous and semi-autonomous vehicles based on occluded areas.

[0004] Autonomous and semi-autonomous vehicles (hereinafter "vehicles") receive motion plans from a navigation subsystem in the vehicle. The navigation subsystem utilizes various sensors installed in and around the vehicle (such as cameras, radio radars, and lidar sensors) to enhance the motion plan. These sensors scan the environment around the vehicle and provide data about the vehicle's surroundings to the navigation subsystem. The navigation subsystem uses this data to generate, update, and enhance the vehicle's motion plan. Other subsystems of the vehicle (such as the engine control subsystem, transmission subsystem, braking subsystem, steering subsystem, etc.) propel the vehicle according to the generated, updated, and enhanced motion plans. Summary of the invention

[0005] A system for a vehicle includes: a plurality of sensors configured to sense the surroundings of the vehicle; and a perception module configured to generate a map of the surroundings of the vehicle, wherein the map includes objects around the vehicle. The system includes an occlusion calculation module configured to calculate occluded areas in the map, wherein the occluded areas are areas around the vehicle that are occluded by one or more objects around the vehicle. The system includes a filtering module configured to not filter out the occluded areas or to filter out one or more occluded areas from the map, wherein after filtering, the map includes a plurality of filtered occluded areas. The system includes a scoring module configured to score the filtered occluded areas based on the importance of the filtered occluded areas to the trajectory of the vehicle. The system includes a motion planning module configured to modify the trajectory of the vehicle based on the importance scores of the filtered occluded areas. The system includes a propulsion module configured to propel the vehicle according to the modified trajectory.

[0006] In other features, the motion planning module is configured to: select one or more filtered occluded regions having an importance score greater than or equal to a threshold; and modify the trajectory of the vehicle based on the selected filtered occluded regions.

[0007] In other features, the scoring module includes a neural network configured to score the filtered occluded regions. The neural network is trained using a baseline reward component and a second reward component that balances the baseline reward component.

[0008] In other features, the second reward component comprises a product of a negative factor and a sum of the importance scores of the filtered occluded regions.

[0009] In other features, the scoring module includes: a first plurality of neural networks configured to receive features associated with the trajectory of the vehicle as input and generate a first output; and a second plurality of neural networks configured to receive features associated with the filtered occluded regions and generate a second output. The scoring module is configured to output the importance score of the filtered occluded regions based on the first output and the second output.

[0010] In other features, the second plurality of neural networks are shared among the filtered occluded regions.

[0011] In other features, the second plurality of neural networks is different from the first plurality of neural networks.

[0012] In other features, the filtering module is configured to filter out one or more occluded regions from the map based on a correlation of the occluded regions to the trajectory of the vehicle.

[0013] In other features, the filtering module is configured to not filter out the occluded areas or filter out one or more occluded areas from the map based on: the route, status and the trajectory of the vehicle; the status of the moving objects around the vehicle; predictions about the moving objects around the vehicle; and heuristics including: the occluded areas that do not intersect the route of the vehicle, the size and proximity of the occluded areas relative to the vehicle, and the time evolution of the occluded areas and the moving objects around the vehicle.

[0014] In other features, the occlusion calculation module is configured to calculate the occluded area based on data received from a mapping system that identifies stationary objects including buildings and road configurations along a route of the vehicle.

[0015] In yet other features, a method for a vehicle includes: sensing a surrounding environment of the vehicle; and generating a map of the surrounding environment of the vehicle, wherein the map includes objects around the vehicle. The method includes calculating occluded areas in the map, wherein the occluded areas are areas around the vehicle that are occluded by one or more objects around the vehicle. The method includes filtering one or more occluded areas from the map, wherein after filtering, the map includes a plurality of filtered occluded areas. The method includes scoring the filtered occluded areas based on the importance of the filtered occluded areas to the trajectory of the vehicle. The method includes modifying the trajectory of the vehicle based on the importance scores of the filtered occluded areas. The method includes propelling the vehicle according to the modified trajectory.

[0016] In other features, the method further comprises selecting one or more filtered occluded regions having importance scores greater than or equal to a threshold and modifying the trajectory of the vehicle based on the selected filtered occluded regions.

[0017] In other features, the method further comprises: scoring the filtered occluded regions using a neural network; and training the neural network using a baseline reward component and a second reward component that balances the baseline reward component.

[0018] In other features, the method further comprises generating the second reward component by multiplying a sum of the importance scores of the filtered occluded regions by a negative factor.

[0019] In other features, the method further comprises inputting features associated with the trajectory of the vehicle to a first plurality of neural networks to generate a first output. The method further comprises inputting features associated with the filtered occluded regions to a second plurality of neural networks to generate a second output. The method further comprises outputting the importance score of the filtered occluded regions based on the first output and the second output.

[0020] In other features, the second plurality of neural networks are shared among the filtered occluded regions.

[0021] In other features, the second plurality of neural networks is different from the first plurality of neural networks.

[0022] In other features, the method further comprises filtering one or more occluded regions from the map based on a correlation of the occluded regions to the trajectory of the vehicle.

[0023] In other features, the method also includes filtering one or more occluded areas from the map based on: the route, status, and trajectory of the vehicle; the status of moving objects around the vehicle; predictions about the moving objects around the vehicle; and heuristics including: the occluded areas that do not intersect the route of the vehicle, the size and proximity of the occluded areas relative to the vehicle, and the temporal evolution of the occluded areas and the moving objects around the vehicle.

[0024] In other features, the method further comprises calculating the occluded area based on data received from a mapping system that identifies stationary objects including buildings and road configurations along a route of the vehicle.

[0025] This disclosure provides the following examples:

[0026] Example 1. A system for a vehicle, comprising:

[0027] a plurality of sensors configured to sense the surrounding environment of the vehicle;

[0028] a perception module configured to generate a map of the vehicle's surroundings, the map including objects around the vehicle;

[0029] an occlusion calculation module, configured to calculate an occluded area in the map, wherein the occluded area is an area around the vehicle that is occluded by one or more objects around the vehicle;

[0030] a filtering module configured to not filter out the occluded area or to filter out one or more occluded areas from the map, wherein after filtering, the map includes a plurality of filtered occluded areas;

[0031] a scoring module configured to score the filtered occluded regions based on the importance of the filtered occluded regions to the trajectory of the vehicle;

[0032] a motion planning module configured to modify the trajectory of the vehicle based on the importance scores of the filtered occluded regions; and

[0033] A propulsion module is configured to propel the vehicle according to the modified trajectory.

[0034] Example 2. The system of Example 1, wherein the motion planning module is configured to:

[0035] selecting one or more filtered occluded regions having an importance score greater than or equal to a threshold; and

[0036] The trajectory of the vehicle is modified based on the selected filtered occluded regions.

[0037] Example 3. The system of Example 1, wherein:

[0038] The scoring module includes a neural network configured to score the filtered occluded regions; and

[0039] The neural network is trained using a baseline reward component and a second reward component that balances the baseline reward component.

[0040] Example 4. The system of Example 3, wherein the second reward component comprises a product of a negative factor and a sum of the importance scores of the filtered occluded regions.

[0041] Example 5. The system of Example 1, wherein the scoring module comprises:

[0042] a first plurality of neural networks configured to receive as input features associated with the trajectory of the vehicle and to generate a first output; and

[0043] a second plurality of neural networks configured to receive features associated with the filtered occluded regions and generate a second output,

[0044] The scoring module is configured to output the importance score of the filtered occluded region based on the first output and the second output.

[0045] Example 6. The system of Example 5, wherein the second plurality of neural networks are shared between the filtered occluded regions.

[0046] Example 7. The system of Example 5, wherein the second plurality of neural networks is different from the first plurality of neural networks.

[0047] Example 8. The system of Example 1, wherein the filtering module is configured to filter out one or more occluded regions from the map based on a correlation of the occluded regions with the trajectory of the vehicle.

[0048] Example 9. A system according to Example 1, wherein the filtering module is configured to not filter out the occluded areas or to filter out one or more occluded areas from the map based on: the route, status and the trajectory of the vehicle; the status of moving objects around the vehicle; predictions about the moving objects around the vehicle; and heuristics including: the occluded areas that do not intersect the route of the vehicle, the size of the occluded areas and their proximity to the vehicle, and the temporal evolution of the occluded areas and the moving objects around the vehicle.

[0049] Example 10. The system of Example 1, wherein the occlusion calculation module is configured to calculate the occluded area based on data received from a mapping system that identifies stationary objects including buildings and road configurations along the vehicle's route.

[0050] Example 11. A method for a vehicle, comprising:

[0051] sensing the surrounding environment of the vehicle;

[0052] generating a map of the surrounding environment of the vehicle, the map including objects around the vehicle;

[0053] Calculating an occluded area in the map, the occluded area being an area around the vehicle that is occluded by one or more objects around the vehicle;

[0054] filtering one or more occluded regions from the map, wherein after filtering, the map includes a plurality of filtered occluded regions;

[0055] scoring the filtered occluded regions based on importance of the filtered occluded regions to a trajectory of the vehicle;

[0056] modifying the trajectory of the vehicle based on the importance scores of the filtered occluded regions; and

[0057] The vehicle is propelled according to the modified trajectory.

[0058] Example 12. The method according to Example 11, further comprising:

[0059] selecting one or more filtered occluded regions having an importance score greater than or equal to a threshold; and

[0060] The trajectory of the vehicle is modified based on the selected filtered occluded regions.

[0061] Example 13. The method according to Example 11, further comprising:

[0062] Using a neural network to score the filtered occluded regions; and

[0063] The neural network is trained using a baseline reward component and a second reward component that balances the baseline reward component.

[0064] Example 14. The method of Example 13, further comprising generating the second reward component by multiplying a sum of the importance scores of the filtered occluded regions by a negative factor.

[0065] Example 15. The method according to Example 13, further comprising:

[0066] inputting features associated with the trajectory of the vehicle into a first plurality of neural networks to generate a first output;

[0067] inputting features associated with the filtered occluded regions into a second plurality of neural networks to generate a second output; and

[0068] The importance score of the filtered occluded region is output based on the first output and the second output.

[0069] Example 16. The method of Example 15, wherein the second plurality of neural networks are shared between the filtered occluded regions.

[0070] Example 17. The method of Example 15, wherein the second plurality of neural networks is different from the first plurality of neural networks.

[0071] Example 18. The method according to Example 11 also includes filtering one or more occluded areas from the map based on the correlation of the occluded areas with the trajectory of the vehicle.

[0072] Example 19. The method according to Example 11 also includes filtering out one or more occluded areas from the map based on: the route, status and the trajectory of the vehicle; the status of moving objects around the vehicle; predictions about the moving objects around the vehicle; and heuristics including: the occluded areas that do not intersect the route of the vehicle, the size of the occluded areas and their proximity to the vehicle, and the temporal evolution of the occluded areas and the moving objects around the vehicle.

[0073] Example 20. The method of Example 11 further includes calculating the occluded area based on data received from a mapping system that identifies stationary objects including buildings and road configurations along the route of the vehicle.

[0074] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims and drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The present disclosure will be more fully understood from the detailed description and accompanying drawings, in which:

[0076] Figure 1 An example of a system including a vehicle in communication with a remote server and system is shown;

[0077] Figure 2 An example of a system for calculating, filtering, and scoring occlusions around a vehicle in order to modify the vehicle's trajectory is shown;

[0078] Figure 3 and Figure 4 An example of an occluded area around a vehicle, a filtered occluded area, and a scored filtered occluded area is shown;

[0079] Figure 5 An example of a method for calculating, filtering, and scoring occlusions around a vehicle in order to modify the vehicle trajectory is shown;

[0080] Figure 6 An example of a method for filtering occlusions is shown;

[0081] Figure 7 An example of a method for scoring occlusion is shown; and

[0082] Figure 8 An example of a method for training a neural network is shown. Figure 2 The scoring module of the system shown in is used to score occlusion;

[0083] In the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0084] Occlusions are prevalent in many driving scenarios. The motion planning subsystem (e.g., navigation subsystem) in autonomous and semi-autonomous vehicles (hereinafter referred to as "vehicles") needs to take occlusions into account to modify the trajectory of the vehicle. A trajectory is different from a motion plan or route. A motion plan is a route provided by the navigation subsystem as a static element on which a vehicle drives from a starting point to a destination. A trajectory is a small portion or fragment (i.e., a subset) of the motion plan that needs to be periodically updated to adjust the vehicle's movement (e.g., speed and steering) based on the dynamically changing surrounding environment around the vehicle in order to propel the vehicle according to the motion plan. The present disclosure relates to periodically changing the trajectory of a vehicle based on the dynamically changing surrounding environment around the vehicle.

[0085] In summary, onboard sensors such as cameras, radio radars, and lidar sensors sense (perceive) the vehicle's surroundings and provide sensed data about the vehicle's surroundings to the vehicle's navigation subsystem. The navigation subsystem generates a map of the vehicle's surroundings based on the sensed data. The map is a snapshot of the vehicle's surroundings. Some areas of the map may be obscured due to various static obstacles (such as buildings) and dynamic obstacles (such as other vehicles around the vehicle, pedestrians, cyclists, etc.). The number of unobserved or unobservable (i.e., obscured) areas on the map can increase rapidly with multiple obstructing objects. For example, additional vehicles, pedestrians, objects used in road construction (such as cones and roadblocks) may appear in the map. The rapid increase in the number of obscured areas of the map increases the computational demands on the motion planning subsystem.

[0086] The present disclosure provides a system and method for reducing the computational requirements of a motion planning subsystem by determining which occluded areas are unimportant, filtering out unimportant occlusions from a map, and assigning importance scores to the filtered occluded areas. A heuristic filter first removes unimportant occluded areas. A subsequent attention-based mechanism classifies the filtered occluded areas based on their importance to the vehicle trajectory and assigns importance scores to the filtered occluded areas. The importance score indicates how important (relevant) the occluded area is to the vehicle trajectory. The motion planning subsystem then processes only the filtered occluded areas to modify the vehicle's trajectory based on the importance scores assigned to the filtered occluded areas. The systems and methods of the present disclosure can be used to enhance the computation of both model-free and model-based planning, as well as to enhance the computation of downstream tasks such as maneuvering the vehicle, providing notifications and warnings on the vehicle's human-machine interface (HMI), and the like.

[0087] This disclosure is organized as follows. Figure 1 , an example of a system including a vehicle in communication with a remote server and system is shown and described. Figure 2 , shows and describes an example of a system for calculating, filtering, and scoring occlusions around a vehicle in order to modify the vehicle's trajectory. Figure 3 and Figure 4 , shows and describes examples of occluded regions around a vehicle, filtered occluded regions, and scored filtered occluded regions. Figure 5 , shows and describes an example of a method for calculating, filtering, and scoring occlusions around a vehicle in order to modify the vehicle trajectory. Figure 6 , an example of a method for filtering occlusion is shown and described. Figure 7 , shows and describes an example of a method for scoring occlusion. Figure 8, an example of a method for training a neural network for scoring occlusions in a scoring module is shown and described.

[0088] Figure 1 A system 100 is shown, which includes a vehicle 102, one or more servers 104 (e.g., located in the cloud), a global positioning system (GPS) 105, and one or more mapping systems (hereinafter referred to as mapping systems) 106. The vehicle 102, the server 104, the GPS 105, and the mapping system 106 communicate with each other via a distributed communication system 108. For example, the vehicle 102 can be an autonomous or semi-autonomous vehicle that implements the systems and methods of the present disclosure. For example, the distributed communication system 108 can include one or more of the following: a local area network (LAN), a wide area network (WAN), a cellular network, a WiFi network, and the Internet. For example, the server 104 can handle the communication from the vehicle 102 and other vehicles ( Figure 1 Not shown, but see Figure 3 and 4 The server 104 may provide information to the vehicle 102 to assist in navigation and other subsystems of the vehicle 102 while driving the vehicle 102.

[0089] Vehicle 102 includes a navigation subsystem 120, a communication subsystem 122, an infotainment subsystem 124, an autonomous subsystem 126, a steering subsystem 128, a braking subsystem 130, a plurality of sensors 132, and a propulsion subsystem 134. Navigation subsystem 120 communicates with server 104, GPS 105, and mapping system 106 via distributed communication system 108. Navigation subsystem 120 may communicate with GPS 105 directly or via communication subsystem 122. Navigation subsystem 120 is described below with reference to FIG. Figure 2 The systems and methods of the present disclosure are implemented as described in detail in the accompanying drawings.

[0090] The communication subsystem 122 may include one or more transceivers (e.g., a cellular transceiver, a WiFi transceiver, a GPS receiver, and a Bluetooth transceiver). The transceiver may communicate with the distributed communication system 108, the GPS 105, the server 104, and a mobile device (such as a cellular phone). The communication subsystem 122 may also communicate directly with the GPS 105. In addition, the communication subsystem 122 may communicate with other vehicles (not shown) using vehicle-to-vehicle (V2V) communication technology.

[0091] The navigation subsystem 120 communicates with the infotainment subsystem 124. The infotainment subsystem 124 may include a display screen (e.g., a touch screen) and multimedia devices (e.g., speakers and microphones) for audio-visual interaction with occupants of the vehicle 102. The navigation subsystem 120 may provide maps and other audio-visual information to occupants of the vehicle 102 via the infotainment subsystem 124. The navigation subsystem 120 may also receive audio-visual input from occupants of the vehicle 102 via the infotainment subsystem 124. The navigation subsystem 120 may also receive input from occupants of the vehicle 102 via a mobile device (such as a cellular telephone).

[0092] The navigation subsystem 120 receives data from sensors 132. For example, the sensors 132 may include sensors that provide the speed, heading, turn instructions, and the like of the vehicle 102. The sensors 132 also include sensors such as cameras, radio radars, lidars, and other sensors located on the vehicle 102 and providing data about the environment surrounding the vehicle 102. The navigation subsystem 120 also receives mapping data (e.g., a map of the road, the number of lanes, intersections, etc.) from the mapping system 106. The navigation subsystem 120 also receives GPS data (e.g., location information of the vehicle 102) from a GPS receiver in the communication subsystem 122 (or directly from GPS 105). The navigation subsystem 120 also receives data about other vehicles from the server 104. The navigation subsystem 120 adjusts the trajectory of the vehicle 102 based on all of the data using the systems and methods of the present disclosure as described in detail below.

[0093] The autonomous subsystem 126 controls the operation of the vehicle 102 by controlling the steering subsystem 128, the braking subsystem 130, and the propulsion subsystem 134 based on the adjusted trajectory received from the navigation subsystem 120. For example, the propulsion subsystem 134 may include a motor (not shown) that propels the vehicle 102. The propulsion subsystem 134 may also include an engine (not shown) that works with the motor to propel the vehicle 102. The autonomous subsystem 126 controls the parameters of the motor and / or the engine according to the adjusted trajectory provided by the navigation subsystem 120.

[0094] Figure 2 An example of a navigation subsystem 120 according to the present disclosure is shown, which is used to calculate, filter and score occlusions around the vehicle 102 in order to modify the trajectory of the vehicle 102. The navigation subsystem 120 includes a perception module 150, an occlusion calculation module 152, a filtering module 154, a scoring module 156 and a motion planning module 158. Each module is described in detail below.

[0095] The perception module 150 receives data about the environment around the vehicle 102 from the sensors 132, receives mapping data from the mapping system 106, receives GPS data from the GPS 105, and receives data about other vehicles from the server 104. The perception module 150 generates a map of the environment around the vehicle 102 (e.g., see the following description). Figure 3 ), which is a snapshot of the scene around the vehicle 102. The perception module 150 may also be referred to as a scene fusion module 150 because it uses information from different sensors and fuses (e.g., combines or merges) the scenes captured by the different sensors to generate a map.

[0096] Figure 3 An example of a map is shown for illustrating systems and methods for detecting, filtering, and scoring occlusions around a vehicle 102 in order to modify a trajectory of the vehicle 102 in accordance with the present disclosure. Figure 3 The maps shown in are examples only. Figure 3 The scene of the intersection shown in the description of the present disclosure is described as an example, but the present disclosure is not limited thereto. Alternatively, the teachings of the present disclosure are applicable to any other scene that the vehicle may encounter when driving the vehicle anywhere (e.g., vehicle 102 overtakes another vehicle, vehicle 102 is overtaken by another vehicle, in parking lots and driving lanes, etc.).

[0097] exist Figure 2 , the occlusion calculation module 152 includes an occlusion model that is derived from a map of the vehicle 102's surroundings (e.g., Figure 3 102 ). For example, the input to the occlusion model in the occlusion computation module 152 includes information about objects around the vehicle 102 captured by the sensors 132 on the vehicle 102. The input includes the state of the vehicle 102 (e.g., speed, heading, and lane) and the state of various moving and stationary objects around the vehicle 102 (such as other vehicles, buildings, and pedestrians around the vehicle 102). For example, in Figure 3 In FIG. 1 , the map shows the vehicle 102 and other vehicles 200 - 1 , 200 - 2 , 200 - 3 , 200 - 4 , 200 - 5 , and 200 - 6 (collectively referred to as other vehicles 200 ). For example, the map shows buildings 202 - 1 and 202 - 2 (collectively referred to as buildings 202 ).

[0098] Inputs to the occlusion model in the occlusion computation module 152 include other information received from the mapping system 106, such as the number of lanes, approaching intersections, and road signs (e.g., traffic lights, stop signs, one-way signs, etc.). Figure 3, the map shows the vehicle 102 traveling in the left lane of a two-lane road 210 toward another two-lane road 212. At the intersection of the roads 210, 212, the map shows a crosswalk 214.

[0099] Based on input from the map and perception module 150, the occlusion model in the occlusion calculation module 152 identifies occluded areas in the map of the environment surrounding the vehicle 102. Figure 3 In FIG. 1 , a dotted line is used to illustrate the line of sight of the vehicle 102. Based on the line of sight of the vehicle 102, the map illustrates occluded areas 220-1, 220-2, 220-3, 220-4, 220-5, 220-6, and 220-7 (collectively referred to as occluded areas 220).

[0100] The filtering module 154 preprocesses the output of the occlusion model, which is a map of the surroundings of the vehicle 102, including the occluded regions 220 identified by the occlusion model in the occlusion computation module 152. The filtering module 154 filters unimportant occluded regions from the map of the surroundings of the vehicle 102, as described below.

[0101] The filtering module 154 performs this preprocessing for two reasons: first, to reduce the number of occluded areas to the maximum number of occlusions to be tracked; and second, to eliminate the calculation of irrelevant occlusions. The filtering module 154 receives various inputs and the output of the occlusion model, which is a map of the vehicle's surroundings including the occluded areas identified by the occlusion model. For example, additional inputs received by the filtering module 154 include the route, status, and current trajectory of the vehicle 102; the status of moving objects (e.g., vehicle 202 and pedestrians) around the vehicle 102, and predictions (e.g., from the server 104) about the movement of objects around the vehicle 102 (e.g., where the object (e.g., vehicle 200) will be relative to the vehicle 102 (e.g., in the next few seconds)).

[0102] The filtering module 154 filters out irrelevant (unimportant) occluded areas from the map based on heuristics. For example, the heuristics may include occlusions that do not intersect the route of the vehicle 102, the size of the occluded area 220, and the proximity (e.g., distance) of the occluded area 220 to the vehicle 102. For example, the heuristics may include occluded areas that are relatively far away from the vehicle 102 (e.g., occluded areas 220-1, 220-2), which may be ignored and filtered out from the map. In some examples, if all occluded areas 220 are considered important (relevant) to the trajectory of the vehicle 102, no occluded areas 220 will be filtered out. Therefore, in general, the filtering module 154 may not filter out any occluded areas 220 from the map, or may filter out one or more of the occluded areas 220 from the map.

[0103] In other examples, the heuristics may include occluded areas caused by stationary objects (such as buildings passed by the intersection of the downstream portion of the one-way street where the vehicle 102 is about to turn), which can be ignored and filtered out. For example, the heuristics may include multiple occluded areas in front of and behind the vehicle 102 on the lane of the vehicle 102, where the farther areas of the multiple occluded areas can be filtered out. For example, the heuristics may include the time evolution of objects and occlusions around the vehicle 102. For example, if there is a closer leading vehicle in the same lane, the filtering module 154 can filter out the occlusion in front of the vehicle 102 in the same lane. The filtering parameters used by the filtering module 154 can be adjusted based on the type of vehicle 102 (e.g., sedan, pickup truck, recreational vehicle (RV), etc.) and / or the possible presence of vulnerable road users (VRUs) in the occluded area 220.

[0104] After pre-processing (filtering) the occluded regions 220 in the map, the filtering module 154 outputs the map with the filtered occluded regions. Figure 4 An example of a map with filtered occluded areas is shown. Figure 4 middle, Figure 3 The occluded regions 220-1, 220-2 shown in FIG. 2 are filtered out as explained above. Therefore, the occluded regions 220-3 to 220-7 that remain in the map (ie, are not filtered out from the map) are referred to as filtered occluded regions. Figure 4 However, for the convenience of the following description, the filtered blocked areas 220-3 to 220-7 are referred to as the filtered blocked areas 221. Figure 6 Methods for filtering various occlusions are shown and described.

[0105] The scoring module 156 scores the filtered occluded regions 221 in the map of the surroundings of the vehicle 102 output by the filtering module 154. The scoring module 156 scores the filtered occluded regions 221 to indicate the importance of each of the filtered occluded regions 221 as being material to the trajectory of the vehicle 102. For example, in Figure 3 and 4 , an example of a trajectory of the vehicle 102 is shown at 230. Before the motion planning module 158 processes the filtered occluded regions 221 according to their importance scores to determine whether to change the trajectory 230 of the vehicle 102 as described below, a pre-processing (filtering) and scoring step is performed.

[0106] Scoring module 156 includes a neural network called Deep Importance Network of Occlusion (DINO) that is trained to score the importance of filtered occluded regions 221. Figure 8 The training methodology of DINO is described. The trained DINO utilizes an attention-based mechanism that receives input features about the trajectory 230 of the vehicle 102 and about the filtered occluded regions 221, as referenced below. Figure 7 Describe the mechanism. The trained DINO uses the following reference Figure 7 The described attention-based mechanism generates an importance score for the filtered occluded regions 221. The motion planning module 158 can then change the trajectory 230 of the vehicle 102 based on the importance score for the filtered occluded regions 221.

[0107] In reference Figure 7 The operation of the scoring module 156 is described in detail and reference Figure 8 Before describing in detail the method used to train DINO, refer to Figure 5 An example of a method 250 performed by the navigation subsystem 120 of the vehicle 102 is shown and described. The following description of the method 250 briefly and generally captures (summarizes) the operations performed by each module of the navigation subsystem 120 of the vehicle 102.

[0108] exist Figure 5 , at 252, the method 250 senses the surroundings of the vehicle 102 (e.g., using the sensors 132 of the vehicle 102). At 254, the method 250 generates a map (snapshot) of the surroundings of the vehicle 102 (e.g., using the perception module 150). At 256, the method 250 calculates the occluded areas 220 in the map (e.g., using the occlusion calculation module 152). At 258, the method 250 filters out unimportant occluded areas from the map based on a heuristic (e.g., using the filtering module 154). At 260, the method 250 generates an importance score for the filtered occluded areas 221 (e.g., using the scoring module 156). At 262, the method 250 modifies the trajectory of the vehicle 102 based on the selected importance score (e.g., using the motion planning module 158).

[0109] Figure 6A method 280 performed by the filtering module 154 is shown. The operation of the filtering module 154 has been described in detail above. In the following description of the method 280, the operation of the filtering module 154 is summarized. At 282, the filtering module 154 receives a map (snapshot) of the surrounding environment of the vehicle 102 including the occluded area 220. At 284, the filtering module 154 receives input including the following: the route, state, and current trajectory of the vehicle 102; the state of moving objects (e.g., other vehicles 200, pedestrians, etc.) around the vehicle 102; and predictions about the movement of objects (e.g., other vehicles 200) around the vehicle 102. At 286, the filtering module 154 uses heuristics to account for occlusions, such as not intersecting the route of the vehicle 102, the size of the occluded area 220 and the proximity (e.g., distance) to the vehicle 102, and the time evolution of occlusions and objects around the vehicle 102. For example, the heuristics are built into (e.g., encoded in) the filtering module 154. At 288 , the filter module 154 filters (ie, removes) unimportant occluded regions from the map based on the input and the heuristics, and determines a filtered occluded region 221 associated with the trajectory of the vehicle 102 .

[0110] Typically, the scene around the vehicle 102 (i.e., the map of the environment around the vehicle 102) may change dynamically. For example, as the vehicle 102, other vehicles 200, and other objects (such as cyclists and pedestrians) continue to move, the positions of the other vehicles 200 and the corresponding occluded areas 220 may change. Therefore, the process of detecting the occluded areas 220, determining the filtered occluded areas 221, calculating the importance scores of the filtered occluded areas 221, and changing the trajectory 230 of the vehicle 102 based on the importance scores may be repeated periodically (e.g., every second).

[0111] Additionally, the modification of the trajectory 230 of the vehicle 102 may generate an alternative trajectory from which the autonomous subsystem 126 of the vehicle 102 may select a trajectory. For example, if the trajectory of the vehicle 102 is to turn left, the modified trajectory may be to execute a left turn different from that planned in the original trajectory (e.g., faster, slower, narrower, wider), or to completely stop the vehicle 102. Thus, the change of trajectory may include an alternative decision.

[0112] Figure 7 The method 300 performed by the scoring module 156 is shown in detail. The attention-based mechanism utilized by the trained DINO in the scoring module 156 receives specific inputs regarding the vehicle 102 and the filtered occluded regions 221 and generates an importance score for the filtered occluded regions 221 as described in detail below.

[0113] For example, the DINO in the scoring module 156 includes multiple embedders (e.g., neural networks) that receive inputs as follows. For the vehicle 102, the DINO in the scoring module 156 includes a query embedder, a key embedder, and a value embedder. For each of the filtered occluded regions 221, the DINO in the scoring module 156 includes a key embedder and a value embedder. For the filtered occluded regions 221, the key embedder and the value embedder are shared by the filtered occluded regions 221 or shared between the filtered occluded regions 221. The key embedder and the value embedder for the vehicle 102 are different from the key embedder and the value embedder shared by the filtered occluded regions 221.

[0114] Furthermore, the DINO in the scoring module 156 does not use a query embedder for the filtered occluded regions 221 because the relationship between the filtered occluded regions 221 is not relevant to scoring the filtered occluded regions 221. Rather, the spatial relationship of each of the filtered occluded regions 221 to the vehicle 102 is relevant to scoring the filtered occluded regions 221. Therefore, the DINO in the scoring module 156 uses a query embedder for the vehicle 102, but does not use a query embedder for the filtered occluded regions 221.

[0115] Thus, the DINO in the scoring module 156 may include five embedders: a query embedder, a key embedder, and a value embedder (3 embedders) for the vehicle 102, plus a key embedder and a value embedder (2 embedders) shared between the filtered occluded regions 221. For example, each of the five embedders may be a separate neural network. The neural network of the DINO in the scoring module 156 receives the features described below as input and generates the vectors described below as output. For each neural network, the number of layers, the width of each layer, and the associated activation function are user-defined (i.e., selectable) parameters.

[0116] For example, let N represent the maximum number of filtered occluded regions 221 to be tracked, where N is an integer greater than 1, which is optional. Figure 7 In the method 300 shown in FIG. 1 , at 302, for the vehicle 102, the query embedder, the key embedder, and the value embedder receive input features extracted from the trajectory 230 of the vehicle 102. For example, the input features may include (x, y, v, a), where x and y are 2D coordinates, v is the velocity, and a is the acceleration of the vehicle 102.

[0117] At 304, for each of the filtered occluded regions 221, the key embedder and the value embedder shared between the filtered occluded regions 221 receive input features of the filtered occluded regions 221. For example, the input features of the filtered occluded regions 221 include (s 0 ,s end , l), where S o and S end are the start and end points of the longitudinal coordinates of the filtered occluded area 221 , and I is information about the lane where the occlusion occurs.

[0118] At 306 , the query embedder for the vehicle 102 generates the dimension N Q The key embedder for vehicle 102 generates a vector of dimension N Q The value embedder for vehicle 102 generates a dimension N V At 308, the key embedder for the filtered occluded region 221 generates a vector of dimension N Q The value embedder for the filtered occluded region 221 generates a vector of dimension N V The output vector of .

[0119] Subsequently, in DINO, the keys and values ​​from the vehicle 102 and the filtered occluded region 221 are concatenated. For example, at 310, the keys from the vehicle 102 and the keys from the filtered occluded region 221 are concatenated to form a dimension (N+1)×N Q At 312, the values ​​from the vehicle 102 and the values ​​from the filtered occluded region 221 are concatenated to form a matrix of dimension (N+1)×N v The value matrix V of .

[0120] At 314, the query matrix Q including the output of the query embedder for the vehicle 102 and the key matrix K including the output of the key embedder for the vehicle 102 and the filtered occluded region 221 are multiplied to generate an attention matrix of dimension 1×(N+1). At 316, the attention matrix is ​​divided by sqrt(N Q ). At 318, a softmax operator is applied row by row to the attention matrix; and at 320, the resultant matrix after applying the softmax operator is multiplied by a value matrix V including the output of the value embedder for the vehicle 102 and the filtered occluded region 221 to generate a matrix having dimensions 1×N V The output matrix of DINO.

[0121] Using the output of DINO to infer the importance score of the filtered occluded region 221, one of the following two methods can be selected. The first method forces N V= N. With the first approach, the output of the multiplication between the attention matrix and the value matrix has the correct dimensions (i.e., the maximum number of occluded regions N). In the second approach, another layer or series of layers (called a head) can be added to DINO. The head receives a 1×N V The vector (i.e., the result of multiplying the attention matrix by the value matrix) is taken as input and generates an output of dimension 1×N. Again, as with the neural network of the embedder, the structure of the head (the number of layers, the width of the layers, and the activation function) can be user-defined (i.e., selectable). Therefore, at 322, Nv=N is forced or the output matrix of DINO is input to the head to generate an output of dimension 1xN. At 324, the importance scores of the N filtered occluded regions 221 are inferred from the output of dimension 1xN.

[0122] Subsequently, the motion planning module 158 may select the filtered occluded areas 221 having importance scores greater than or equal to a selectable threshold, and ignore the filtered occluded areas having importance scores less than the selectable threshold. The motion planning module 158 processes only the selected ones of the filtered occluded areas 221, which reduces the computational load of the motion planning module 158, and the trajectory 230 of the vehicle 102 may be changed based on the selected ones of the filtered occluded areas 221. Alternatively or additionally, the motion planning module 158 may process only the selected ones of the filtered occluded areas 221, and generate one or more alternative trajectories for the vehicle 102 based on the selected ones of the filtered occluded areas 221.

[0123] Figure 8 A method 350 for training DINO used in the scoring module 156 is shown in detail. For example, reinforcement learning is used to train DINO in the scoring module 156 as follows. During training, the embedder (neural network) of DINO receives input features about the trajectory of the vehicle 102 and the occluded region 220 as described above. For example, at 352, for the vehicle 102, the query embedder, key embedder, and value embedder receive input features extracted from the trajectory of the vehicle 102. For example, the input features may include (x, y, v, a), where x and y are 2D coordinates, v is the velocity, and a is the acceleration of the vehicle 102. At 354, for example, the key embedder and value embedder shared between the occluded regions 220 receive input features of the occluded region 220. For example, the input features of the occluded region 220 include (s 0 ,s end , l), where S 0 and S end are the start and end points of the longitudinal coordinates of the filtered occluded area 221 , and I is information about the lane where the occlusion occurs.

[0124] At 356 , DINO in the scoring module 156 generates an output including an importance score for the occluded region 220 . Based on the importance score generated by DINO, the motion planning module 158 generates a sequence of actions (eg, speed, acceleration, lane changes, turns, etc.) for the trajectory of the vehicle 102 .

[0125] At 358, a reward comprising two components is used to train DINO. The reward comprising two components is used to adjust the weights and biases of the neural network in DINO. The two components are balanced with each other so that the net reward does not increase the computational load on the motion planning module 158 and does not compromise the comfort, safety, and speed of the vehicle 102, as described below.

[0126] The first component of the reward signal is a baseline reward generated by the motion planning module 158. The motion planning module 158 generates the first component based on factors such as how quickly the vehicle 102 can complete the trajectory, how much comfort (e.g., jerk) the trajectory can generate when the vehicle 102 completes the trajectory, and how safely the vehicle 102 can complete the trajectory. The motion planning module 158 can generate the first component by optimizing one or more of these factors. For example, the motion planning module 158 can focus on one factor (e.g., comfort) and reduce the importance of another factor (e.g., speed). For example, the motion planning module 158 can use an equation or formula to maximize the baseline reward so that the vehicle 102 can execute the trajectory at a speed with maximum comfort and safety.

[0127] In some examples, if only the baseline reward component is used to train DINO, the number of occluded regions detected as important may increase, which in turn increases the computational load of the motion planning module 158. Therefore, the present disclosure adds a second reward component to the training signal used to train DINO. The second reward component balances or offsets the baseline reward component, as described below.

[0128] For example, a second reward component may be generated by summing the importance scores of the occluded regions, multiplying the sum of the importance scores by a negative coefficient, and adding the negative product to the first baseline reward component. Alternatively, the second reward component may be generated by selecting only those importance scores that are greater than a selected threshold. For example, importance scores of occluded regions that have high importance scores due to pedestrians entering the road, due to oncoming vehicles (such as emergency vehicles) approaching the vehicle 102, etc. may be selected. The second reward component may then be generated by summing the selected high importance scores, multiplying the sum by a negative coefficient, and adding the negative product to the first baseline reward component. The second reward component reduces the first reward component so that DINO is not disproportionately biased toward maximizing factors in the baseline reward (such as the comfort, safety, and speed of the vehicle 102).

[0129] Thus, at 358, the reward signal used to train DINO includes the sum of a baseline reward component and another reward component that includes the product of a negative factor and the sum of importance scores for occluded regions. At 360, the learning signal from the reward is used to adjust the weights and biases of the five embedders (neural networks). This adjustment is continued until the DINO training is complete. For example, the training stops when the user decides to stop training based on a different metric, such as the number of updates to the weights and biases of the neural network. The trained DINO is then used in the scoring module 156 of the navigation subsystem 120 of the vehicle 102 as described above.

[0130] The foregoing description is merely illustrative in nature and is not intended to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in many forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited, because other modifications will become apparent after studying the drawings, description and appended claims.

[0131] It should be understood that one or more steps within the method can be performed in a different order (or simultaneously) without changing the principles of the present disclosure. In addition, although each embodiment is described above as having specific features, any one or more of those features described with respect to any embodiment of the present disclosure can be implemented in any other embodiment and / or combined with the features of any other embodiment, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the arrangement of one or more embodiments with respect to each other is still within the scope of the present disclosure.

[0132] The spatial and functional relationships between elements (e.g., between modules, subsystems, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "next to," "on top of," "above," "below," and "disposed." Unless explicitly described as "directly," when describing the relationship between a first element and a second element in the above disclosure, the relationship can be a direct relationship, in which there are no other intermediate elements between the first element and the second element, but can also be an indirect relationship, in which there are one or more intermediate elements (spatially or functionally) between the first and second elements. As used herein, the phrase "at least one of A, B, and C" should be interpreted to mean a logical (A or B or C), using a non-exclusive logical "or," and should not be interpreted to mean "at least one of A, at least one of B, and at least one of C."

[0133] In the drawings, the direction of the arrows, as indicated by the arrows, generally demonstrates the flow of information (e.g., data or instructions) of interest to the illustration. For example, when component A and component B exchange various information, but the information transmitted from component A to component B is relevant to the illustration, the arrow may be directed from component A to component B. This unidirectional arrow does not mean that no other information is transmitted from component B to component A. In addition, for information sent from component A to component B, component B may send a request or a receipt confirmation of the information to component A.

[0134] In this application, including the definitions below, the term "module", the term "controller" or the term "subsystem" may be replaced with the term "circuit". The term "module" or the term "subsystem" may refer to, be part of, or include: an application specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0135] The module or subsystem may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functions of any given module or subsystem of the present disclosure may be distributed between multiple modules or subsystems connected via interface circuits. For example, multiple modules or subsystems may allow load balancing. In a further example, a server (also referred to as a remote server or cloud) may perform some functions on behalf of a client module or subsystem.

[0136] The term "code" as used above may include software, firmware, and / or microcode, and may refer to a program, a routine, a function, a class, a data structure, and / or an object. The term "shared processor circuit" covers a single processor circuit that executes some or all of the code from multiple modules or subsystems. The term "group processor circuit" covers a processor circuit that executes some or all of the code from one or more modules or subsystems in combination with an additional processor circuit. References to multiple processor circuits cover multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" covers a single memory circuit that stores some or all of the code from multiple modules or subsystems. The term "group memory circuit" covers a memory circuit that stores some or all of the code from one or more modules or subsystems in combination with additional memory.

[0137] The term "memory circuit" is a subset of the term "computer-readable medium". As used herein, the term "computer-readable medium" does not cover transient electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); therefore, the term "computer-readable medium" may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory, tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0138] The apparatus and methods described in this application may be implemented in part or in whole by a special purpose computer created by configuring a general purpose computer to perform one or more specific functions embodied in a computer program. The above-mentioned function blocks, flow chart components and other elements serve as software specifications, which can be translated into a computer program by routine work of a skilled technician or programmer.

[0139] The computer program includes processor executable instructions stored on at least one non-transitory tangible computer readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0140] A computer program may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated by a compiler from source code, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, source code may be written using syntax from a language including: C, C++, C#, Objective C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language Fifth Edition), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK, and

Claims

1. A system for a vehicle, comprising: a plurality of sensors configured to sense the surrounding environment of the vehicle; a perception module configured to generate a map of the vehicle's surroundings, the map including objects around the vehicle; an occlusion calculation module, configured to calculate an occluded area in the map, wherein the occluded area is an area around the vehicle that is occluded by one or more objects around the vehicle; a filtering module configured to not filter out the occluded area or to filter out one or more occluded areas from the map, after which the map includes a plurality of filtered occluded areas; a scoring module configured to score the filtered occluded regions based on the importance of the filtered occluded regions to the trajectory of the vehicle; a motion planning module configured to modify the trajectory of the vehicle based on the importance scores of the filtered occluded regions; as well as A propulsion module is configured to propel the vehicle according to the modified trajectory.

2. The system of claim 1, wherein the motion planning module is configured to: selecting one or more filtered occluded regions having an importance score greater than or equal to a threshold; and The trajectory of the vehicle is modified based on the selected filtered occluded regions.

3. The system of claim 1, wherein: The scoring module includes a neural network configured to score the filtered occluded regions; as well as The neural network is trained using a baseline reward component and a second reward component that balances the baseline reward component. 4 . The system of claim 3 , wherein the second reward component comprises a product of a negative factor and a sum of the importance scores of the filtered occluded regions.

5. The system according to claim 1, wherein the scoring module comprises: a first plurality of neural networks configured to receive as input features associated with the trajectory of the vehicle and to generate a first output; as well as a second plurality of neural networks configured to receive features associated with the filtered occluded regions and generate a second output, The scoring module is configured to output the importance score of the filtered occluded region based on the first output and the second output.

6. The system of claim 5, wherein the second plurality of neural networks are shared between the filtered occluded regions.

7. The system of claim 5, wherein the second plurality of neural networks is different from the first plurality of neural networks. 8 . The system of claim 1 , wherein the filtering module is configured to filter out one or more occluded regions from the map based on a correlation of the occluded regions with the trajectory of the vehicle.

9. The system of claim 1 , wherein the filtering module is configured to not filter out the occluded areas or to filter out one or more occluded areas from the map based on: the route, state, and trajectory of the vehicle; the state of moving objects around the vehicle; predictions about the moving objects around the vehicle; and a heuristic comprising: the occluded areas that do not intersect the route of the vehicle, the size of the occluded areas and their proximity to the vehicle, and the temporal evolution of the occluded areas and the moving objects around the vehicle.

10. The system of claim 1, wherein the occlusion calculation module is configured to calculate the occluded area based on data received from a mapping system that identifies stationary objects including buildings and road configurations along a route of the vehicle.

Images