IMPROVING THE MOTION PLANNING OF AUTONOMOUS AND SEMI-AUTOMATIC VEHICLES BASED ON HIDDEN AREAS

Abstract

A vehicle system contains sensors that detect the vehicle's surroundings. A perception module generates a map of the vehicle's environment. The map includes objects around the vehicle. An occlusion calculation module calculates occlusion areas on the map. Occlusion areas are areas around the vehicle that are obscured by one or more of the objects surrounding the vehicle. A filtering module filters none, one, or more of the occlusion areas from the map. After filtering, the map includes multiple filtered occlusion areas. An evaluation module assesses the filtered occlusion areas based on their importance to the vehicle's trajectory. A motion planning module modifies the vehicle's trajectory based on the importance assessments of the filtered occlusion areas. A propulsion module propels the vehicle according to the modified trajectory.

Description

INTRODUCTION

[0001] The information provided in this section serves to generally illustrate the context of the disclosure. Work by the inventors identified herein, to the extent described in this section, as well as aspects of the description that do not otherwise qualify as prior art at the time of filing, are neither explicitly nor implicitly acknowledged as prior art to the present disclosure.

[0002] The present disclosure relates generally to autonomous and semi-autonomous vehicles, and more particularly to improving motion planning of autonomous and semi-autonomous vehicles based on occluded regions.

[0003] Autonomous and semi-autonomous vehicles (hereinafter "vehicles") receive a motion plan from a navigation subsystem within the vehicle. The navigation subsystem uses various sensors, such as cameras, radar, and lidar sensors, installed in and around the vehicle to improve the motion plan. These sensors scan the environment around the vehicle and provide data about the vehicle's surroundings to the navigation subsystem. The navigation subsystem uses the data to generate, update, and improve a motion plan for the vehicle. Other subsystems of the vehicle, such as the engine control subsystem, the transmission subsystem, the braking subsystem, the steering subsystem, etc., drive the vehicle in accordance with the generated, updated, and improved motion plan. SUMMARY

[0004] A system for a vehicle comprises a plurality of sensors configured to sense the surroundings of the vehicle, and a perception module configured to generate a map of the surroundings of the vehicle, the map comprising objects around the vehicle. The system comprises an occlusion calculation module configured to calculate occluded areas on the map, the occluded areas being areas around the vehicle that are occluded by one or more of the objects around the vehicle. The system comprises a filtering module configured to filter out none or one or more of the occluded areas from the map, the map comprising a plurality of filtered occluded areas after filtering. The system comprises an evaluation module configured to evaluate the filtered occluded areas based on the importance of the filtered occluded areas for a trajectory of the vehicle.The system includes a motion planning module configured to modify the vehicle's trajectory based on importance ratings of the filtered occluded regions. The system includes a propulsion module configured to propel the vehicle in accordance with the modified trajectory.

[0005] According to other features, the motion planning module is configured to select one or more of the filtered occluded regions having importance scores greater than or equal to a threshold and to change the trajectory of the vehicle based on the selected filtered occluded regions.

[0006] According to other features, the evaluation module includes a neural network configured to evaluate the filtered hidden regions. The neural network is trained using a base reward component and a second reward component balanced with the base reward component.

[0007] According to other features, the second reward component comprises a product of a negative factor and a sum of the importance ratings of the filtered occluded regions.

[0008] According to other features, the evaluation module comprises a first plurality of neural networks configured to receive features associated with the vehicle's trajectory as inputs and generate first outputs, and a second plurality of neural networks configured to receive features associated with the filtered occluded regions and generate second outputs. The evaluation module is configured to output importance ratings of the filtered occluded regions based on the first outputs and the second outputs.

[0009] According to other features, the second plurality of neural networks are shared among the filtered hidden regions.

[0010] According to other characteristics, the second plurality of neural networks is different from the first plurality of neural networks.

[0011] According to other features, the filtering module is configured to filter out one or more of the occluded areas from the map based on the relevance of the occluded areas to the trajectory of the vehicle.

[0012] According to other features, the filtering module is configured to filter out none or one or more of the occluded areas from the map based on the route, state, and trajectory of the vehicle; states of moving objects around the vehicle; predictions about the moving objects around the vehicle; and heuristics including the occluded areas that do not intersect the route of the vehicle, the size and proximity of the occluded areas relative to the vehicle, and the temporal evolution of the occluded areas and the moving objects around the vehicle.

[0013] According to other features, the occlusion calculation module is configured to calculate the occluded areas based on data received from a mapping system that identifies static objects, including buildings and road configuration along a route of the vehicle.

[0014] According to still other features, a method for a vehicle comprises sensing the surroundings of the vehicle and generating a map of the surroundings of the vehicle, wherein the map includes objects around the vehicle. The method comprises calculating occluded regions on the map, wherein the occluded regions are regions around the vehicle that are occluded by one or more of the objects around the vehicle. The method comprises filtering out none or one or more of the occluded regions from the map, wherein the map, after filtering, comprises a plurality of filtered occluded regions. The method comprises evaluating the filtered occluded regions based on the importance of the filtered occluded regions for a trajectory of the vehicle. The method comprises changing the trajectory of the vehicle based on importance ratings of the filtered occluded regions.The method includes driving the vehicle in accordance with the changed trajectory.

[0015] According to other features, the method further comprises selecting one or more of the filtered occluded regions having importance scores greater than or equal to a threshold and changing the trajectory of the vehicle based on the selected filtered occluded regions.

[0016] According to other features, the method further comprises evaluating the filtered hidden regions using a neural network and training the neural network using a base reward component and a second reward component balanced with the base reward component.

[0017] According to other features, the method further comprises generating the second reward component by multiplying a sum of the importance ratings of the filtered hidden regions by a negative factor.

[0018] According to other features, the method further comprises inputting features associated with the vehicle's trajectory to a first plurality of neural networks to generate first outputs. The method further comprises inputting features associated with the filtered occluded regions to a second plurality of neural networks to generate second outputs. The method further comprises outputting the importance ratings of the filtered occluded regions based on the first outputs and the second outputs.

[0019] According to other features, the second plurality of neural networks are shared among the filtered hidden regions.

[0020] According to other characteristics, the second plurality of neural networks is different from the first plurality of neural networks.

[0021] According to other features, the method further comprises filtering out one or more of the occluded areas from the map based on the relevance of the occluded areas to the trajectory of the vehicle.

[0022] According to other features, the method further comprises filtering one or more of the occluded regions from the map based on the route, state, and trajectory of the vehicle; states of moving objects around the vehicle; predictions about the moving objects around the vehicle; and heuristics including the occluded regions that do not intersect the route of the vehicle, the size and proximity of the occluded regions relative to the vehicle, and the temporal evolution of the occluded regions and the moving objects around the vehicle.

[0023] According to other features, the method further comprises calculating the obscured areas based on data received from a mapping system that identifies static objects, including buildings and road configuration along a path of the vehicle.

[0024] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims, and drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present disclosure will be more fully understood from the detailed description and the accompanying drawings, in which: Fig. 1 shows an example of a system including a vehicle communicating with remote servers and systems; Fig. 2 shows an example of a system for calculating, filtering and evaluating occlusions around a vehicle to change the vehicle's trajectories; Fig. 3 and Fig. 4 Examples of occluded areas, filtered occluded areas and evaluating filtered occluded areas around a vehicle; Fig. 5 shows an example of a method for calculating, filtering and evaluating occlusions for changing vehicle trajectories; Fig. 6 shows an example of a method for filtering occlusions; Fig. 7 an example of a method for evaluating occlusions; and Fig. 8 an example of a method for training neural networks used in an evaluation module of the Fig. 2 can be used to evaluate occlusions.

[0026] In the drawings, reference symbols may be used multiple times to identify similar and / or identical elements. DETAILED DESCRIPTION

[0027] Occlusions are ubiquitous in many driving scenarios. Motion planning subsystems (e.g., navigation subsystems) in autonomous and semi-autonomous vehicles (hereafter "the vehicles") must account for occlusions to modify vehicle trajectories. A trajectory is different from a motion plan or a route. A motion plan is a route, which is a static element provided by a navigation subsystem, along which a vehicle is driven from a source to a destination. A trajectory is a small section or segment (i.e., a subset) of the motion plan that must be periodically updated depending on the dynamically changing environment around the vehicle in order to adjust the vehicle's motion (e.g., speed and steering) to propel the vehicle in accordance with the motion plan.The present disclosure relates to periodically changing the trajectory of the vehicle depending on the dynamically changing environment around the vehicle.

[0028] Broadly speaking, the sensors onboard a vehicle, such as cameras, radar, and lidar sensors, sense (perceive) the vehicle's surroundings and provide sensed data about the vehicle's surroundings to a navigation subsystem on the vehicle. The navigation subsystem generates a map of the vehicle's surroundings based on the sensed data. The map is a snapshot of the vehicle's surroundings. Due to various static obstacles, such as buildings, and dynamic obstacles, such as other vehicles around the vehicle, pedestrians, cyclists, etc., some areas of the map may be occluded. The number of unobserved or unobservable (i.e., occluded) areas on the map can rapidly increase with multiple occluding objects. For example, additional vehicles, pedestrians, and objects such as traffic cones and barriers used in road construction may appear on the map.The rapid increase in the number of occluded areas of the map increases the computational demands on the motion planning subsystem.

[0029] The present disclosure provides a system and method for mitigating the computational demands on the motion planning subsystem by determining which occluded regions are unimportant, filtering the unimportant occlusions from the map, and assigning importance scores to the filtered occluded regions. A heuristic filter first removes unimportant occluded regions. A subsequent attention-based mechanism classifies the filtered occluded regions based on the importance of the filtered occluded regions to the vehicle's trajectory and assigns importance scores to the filtered occluded regions. The importance scores indicate how important (relevant) the occluded regions are to the vehicle's trajectory.The motion planning subsystem then processes only the filtered occluded regions to modify the vehicle's trajectory based on the importance scores assigned to the filtered occluded regions. The system and method of the present disclosure can be effectively used to improve the computation of both model-free and model-based planning, as well as to improve the computation of downstream tasks such as maneuvering the vehicle, providing messages and alerts to a human-machine interface (HMI) of the vehicle, and so on.

[0030] The present disclosure is organized as follows. Fig. 1 is an example of a system including a vehicle communicating with remote servers and systems. Fig. 2 shows and describes an example of a system for calculating, filtering, and evaluating occlusions around a vehicle to change the vehicle's trajectories. Fig. 3 and Fig. 4 shows and describes examples of hidden areas, filtered hidden areas, and the evaluation of the filtered hidden areas around the vehicle. Fig. 5 shows and describes an example of a method for calculating, filtering, and evaluating occlusions around a vehicle to change the vehicle's trajectories. Fig. 6 shows and describes an example of a method for filtering occlusions. Fig. Figure 7 shows and describes an example of a method for evaluating occlusions. Fig. 8, an example of a method for training neural networks used in an occlusion evaluation module is shown and described.

[0031] Fig. 1 shows a system 100 that includes a vehicle 102, one or more servers 104 (e.g., located in a cloud), a global positioning system (GPS) 105, and one or more mapping systems (hereinafter, the mapping system) 106. The vehicle 102, the servers 104, the GPS 105, and the mapping system 106 communicate with each other via a distributed communication system 108. The vehicle 102 may, for example, be an autonomous or semi-autonomous vehicle implementing the system and method of the present disclosure. The distributed communication system 108 may, for example, include a local area network (LAN), a wide area network (WAN), a cellular network, a Wi-Fi network, and / or the Internet. The servers 104 may, for example, receive some of the data from the vehicle 102 and from other vehicles (in Fig. 1 not shown, but see the in Fig. 3 and Fig. 4). The servers 104 may provide information to the vehicle 102 to assist navigation and other subsystems of the vehicle 102 in driving the vehicle 102.

[0032] The vehicle 102 includes a navigation subsystem 120, a communication subsystem 122, an infotainment subsystem 124, an autonomous subsystem 126, a steering subsystem 128, a braking subsystem 130, a plurality of sensors 132, and a propulsion subsystem 134. The navigation subsystem 120 communicates with the server 104, with the GPS 105, and with the mapping system 106 via the distributed communication system 108. The navigation subsystem 120 may communicate with the GPS 105 directly or via the communication subsystem 122. The navigation subsystem 120 implements the system and method of the present disclosure, as described below with reference to Fig. 2 and the following are described in detail.

[0033] The communication subsystem 122 may include one or more transceivers (e.g., a cellular transceiver, a Wi-Fi transceiver, a GPS transceiver, and a Bluetooth transceiver). The transceivers may communicate with the distributed communication system 108, with the GPS 105, with the servers 104, and with a mobile device such as a cellular phone. In addition, the communication subsystem 122 may communicate directly with the GPS 105. Furthermore, the communication subsystem 122 may communicate with other vehicles (not shown) using vehicle-to-vehicle (V2V) communication technology.

[0034] The navigation subsystem 120 communicates with the infotainment subsystem 124. The infotainment subsystem 124 may include a display screen (e.g., a touchscreen) and multimedia devices (e.g., a speaker and a microphone) for audiovisual interactions with occupants of the vehicle 102. The navigation subsystem 120 may provide maps and other audiovisual information to the occupants of the vehicle 102 via the infotainment subsystem 124. The navigation subsystem 120 may also receive audiovisual inputs from the occupants of the vehicle 102 via the infotainment subsystem 124. The navigation subsystem 120 may also receive inputs from the occupants of the vehicle 102 via a mobile device, such as a cellular phone.

[0035] The navigation subsystem 120 receives data from the sensors 132. The sensors 132 may include, for example, sensors that provide the speed, direction of travel, turn indication, etc. of the vehicle 102. The sensors 132 also include sensors such as cameras, radar, lidar, and other sensors located onboard the vehicle 102 that provide data about the surroundings of the vehicle 102. The navigation subsystem 120 also receives mapping data (e.g., a map of the road, the number of lanes, an intersection, etc.) from the mapping system 106. The navigation subsystem 120 also receives GPS data (e.g., location information of the vehicle 102) from the GPS receiver in the communication subsystem 122 (or directly from the GPS 105). The navigation subsystem 120 also receives data about other vehicles from the servers 104.The navigation subsystem 120 adjusts the trajectory of the vehicle 102 based on all data using the system and method of the present disclosure, as described in detail below.

[0036] The autonomous subsystem 126 controls the operations of the vehicle 102 by controlling the steering subsystem 128, the braking subsystem 130, and the propulsion subsystem 134 based on the set trajectory received from the navigation subsystem 120. The propulsion subsystem 134 may, for example, include a motor (not shown) that powers the vehicle 102. The propulsion subsystem 134 may also include a prime mover (not shown) that cooperates with the motor to propel the vehicle 102. The autonomous subsystem 126 controls parameters of the motor and / or prime mover in accordance with the set trajectory provided by the navigation subsystem 120.

[0037] Fig. Figure 2 shows an example of the navigation subsystem 120 for calculating, filtering, and evaluating occlusions around the vehicle 102 to change trajectories of the vehicle 102 according to the present disclosure. The navigation subsystem 120 includes a perception module 150, an occlusion calculation module 152, a filtering module 154, an evaluation module 156, and a motion planning module 158. The modules are described in detail below.

[0038] The perception module 150 receives data about the surroundings of the vehicle 102 from the sensors 132, the mapping data from the mapping system 106, the GPS data from the GPS 105, and data about other vehicles from the servers 104. The perception module 150 generates a map (see, e.g., Fig. 3, described below) of the surroundings of the vehicle 102, which is a snapshot of a scene surrounding the vehicle 102. The perception module 150 may also be called a scene fusion module 150 because it uses information from various sensors and fuses (e.g., combines or merges) the scenes captured by the various sensors to generate the map.

[0039] Fig. 3 shows an example of a map used to illustrate the system and method for detecting, filtering, and evaluating occlusions around the vehicle 102 to change trajectories of the vehicle 102 according to the present disclosure. Fig. The map shown in Figure 3 is only an example. Although the present disclosure is based on a Fig. While the scene shown in Figure 3, which includes an intersection, is described as an example, the present disclosure is not limited thereto. Rather, the teachings of the present disclosure are applicable to any other scenario that vehicles may encounter while being driven anywhere (e.g., vehicle 102 passing another vehicle, vehicle 102 being passed by another vehicle, parking lots and driveways, etc.).

[0040] The occlusion calculation module 152 in Fig. 2 includes an occlusion model that uses various inputs from the map of the surroundings of the vehicle 102 (e.g., from the Fig. 3). For example, the inputs to the occlusion model in the occlusion calculation module 152 include information about objects surrounding the vehicle 102 as detected by the sensors 132 on board the vehicle 102. The inputs include the state (e.g., the speed, direction of travel, and lane) of the vehicle 102 and states of various moving and stationary objects around the vehicle 102, such as other vehicles, buildings, and pedestrians around the vehicle 102. For example, the map in Fig. 3 Vehicle 102 and other vehicles 200-1, 200-2, 200-3, 200-4, 200-5, 200-6 (collectively referred to as the other vehicles 200). For example, the map shows buildings 202-1, 202-2 (collectively referred to as buildings 202).

[0041] The inputs to the occlusion model in the occlusion calculation module 152 include other information received from the mapping system 106, such as the number of lanes, an approaching intersection, and traffic signs (e.g., traffic lights, a stop sign, a one-way sign, etc.). For example, the map in Fig. 3, that the vehicle 102 is traveling in a left lane on a two-lane road 210 toward another two-lane road 212. At the intersection of roads 210 and 212, the map shows a pedestrian crossing 214.

[0042] The occlusion model in the occlusion calculation module 152 identifies occluded areas on the map of the surroundings of the vehicle 102 based on the inputs from the map and the perception module 150. For example, in Fig. 3 dashed lines are used to show lines of sight of the vehicle 102. Based on the lines of sight of the vehicle 102, the map shows obscured areas 220-1, 220-2, 220-3, 220-4, 220-5, 220-6, 220-7 (collectively, the obscured areas 220).

[0043] The filtering module 154 preprocesses the output of the occlusion model, which is the map of the surroundings of the vehicle 102 containing the occluded regions 220 identified by the occlusion model in the occlusion calculation module 152. The filtering module 154 filters out irrelevant occluded regions from the map of the surroundings of the vehicle 102 as follows.

[0044] The filtering module 154 performs preprocessing for two reasons: first, to reduce the number of occluded regions to a maximum number of trackable occlusions; and second, to eliminate calculations for irrelevant occlusions. The filtering module 154 receives various inputs in addition to the output of the occlusion model, which is the map of the vehicle's surroundings containing the occluded regions identified by the occlusion model. The additional inputs received by the filtering module 154 include, for example, the route, state, and current trajectory of the vehicle 102. the states of moving objects (e.g., vehicles 202 and pedestrians) around the vehicle 102 and predictions (e.g., from the servers 104) about the movement of objects around the vehicle 102 (e.g., where the objects (e.g., the vehicles 200) will be imminently located relative to the vehicle 102 (e.g., in the next few seconds)).

[0045] The filtering module 154 filters out irrelevant (unimportant) occluded areas from the map based on heuristics. The heuristics may include, for example, occlusions that do not intersect the path of the vehicle 102, the size of the occluded areas 220, and the proximity (e.g., distance) of the occluded areas 220 from the vehicle 102. The heuristics may include, for example, an occluded area that is relatively far from the vehicle 102 (e.g., occluded areas 220-1, 220-2) that may be ignored and filtered out from the map. According to some examples, if all of the occluded areas 220 are considered important (relevant) to the trajectory of the vehicle 102, none of the occluded areas 220 are filtered out. Accordingly, the filtering module 154 may generally filter out none or one or more of the occluded areas 220 from the map.

[0046] According to other examples, the heuristics may include an occlusion region caused by a stationary object, such as a building, beyond an intersection on a downstream portion of a one-way street onto which the vehicle 102 is about to turn, which may be ignored and filtered out. For example, the heuristics may include multiple occlusion regions in the lane of the vehicle 102 that are in front of and behind the vehicle 102, where the further ones of the multiple occlusion regions may be filtered out. For example, the heuristics may include the temporal evolution of occlusions and objects around the vehicle 102. For example, the filtering module 154 may filter out occlusion in front of the vehicle 102 in the same lane if there is a closer preceding vehicle in the same lane. The filtering parameters used by the filtering module 154 may be based on the type of the vehicle 102 (e.g.,a sedan, a van, a mobile home (RV), etc.) and / or the possible presence of a vulnerable road user (VRU) in the covert areas 220.

[0047] After preprocessing (filtering) the hidden areas 220 on the map, the filtering module 154 outputs a map with the filtered hidden areas. Fig. 4 shows an example of a map with the filtered hidden areas. In Fig. 4 are the Fig. 3 are filtered out, as explained above. Accordingly, the hidden areas 220-3 to 220-7 that remain on the map (ie, that are not filtered out from the map) are called the filtered hidden areas. Although they are shown in Fig. 4 are not designated as such, in order to facilitate the following description, the filtered hidden areas 220-3 to 220-7 are called the filtered hidden areas 221. In the following, Fig. 6 a method for filtering the occlusions is shown and described.

[0048] The evaluation module 156 evaluates the filtered occluded areas 221 on the map of the surroundings of the vehicle 102 output by the filtering module 154. The evaluation module 156 evaluates the filtered occluded areas 221 to indicate the importance of each of the filtered occluded areas 221 that is essential to the trajectory of the vehicle 102. For example, in Fig. 3 and Fig. 4, an example of the trajectory of the vehicle 102 is shown at 230. The preprocessing step (filtering step) and the evaluation step are performed before the motion planning module 158 processes the filtered occluded regions 221 in accordance with the importance scores of the filtered occluded regions 221 to determine whether to change the trajectory 230 of the vehicle 102, as described below.

[0049] The evaluation module 156 comprises a neural network, called deep importance network for occlusions (DINO), which is trained to evaluate the importance of the filtered occluded regions 221. The training methodology of the DINO is described below using Fig. 8. The trained DINO uses an attention-based mechanism that receives input features about the trajectory 230 of the vehicle 102 and about the filtered occluded areas 221, which are described below using Fig. 7. The trained DINO generates using the attention-based mechanism, as shown below with reference to Fig. 7, the importance ratings for the filtered occluded regions 221. The motion planning module 158 may then change the trajectory 230 of the vehicle 102 based on the importance ratings for the filtered occluded regions 221.

[0050] Before describing in detail the operation of the evaluation module 156 using Fig. 7 and the procedure for training the DINO using Fig. 8 is based on Fig. 5, an example of a method 250 performed by the navigation subsystem 120 of the vehicle 102 is shown and described. The following description of the method 250 briefly and comprehensively captures (summarizes) the operations performed by each module of the navigation subsystem 120 of the vehicle 102.

[0051] In Fig. 5, at 252, the method 250 senses the surroundings of the vehicle 102 (e.g., using the sensors 132 of the vehicle 102). At 254, the method 250 generates a map (a snapshot) of the surroundings of the vehicle 102 (e.g., using the perception module 150). At 256, the method 250 calculates (e.g., using the occlusion calculation module 152) the occluded areas 220 on the map. At 258, the method 250 filters out unimportant occluded areas from the map based on heuristics (e.g., using the filtering module 154). At 260, the method 250 generates (e.g., using the scoring module 156) importance scores for the filtered occluded regions 221. At 262, the method 250 changes (e.g., using the motion planning module 158) the trajectory of the vehicle 102 based on selected importance scores.

[0052] Fig. 6 shows a method 280 performed by the filtering module 154. The operations of the filtering module 154 have been described in detail above. The following description of the method 280 summarizes the operations of the filtering module 154. At 282, the filtering module 154 receives a map (a snapshot) of the environment of the vehicle 102, including the occluded regions 220. At 284, the filtering module 154 receives inputs including the track, state, and current trajectory of the vehicle 102; the states of moving objects (e.g., other vehicles 200, pedestrians, etc.) around the vehicle 102; and predictions about the movement of the objects (e.g., other vehicles 200) around the vehicle 102. At 286, the filtering module 154 leverages heuristics to, for example, B. Occlusions that do not intersect the path of the vehicle 102, the size and proximity (e.g.The heuristics are incorporated (e.g., coded into) the filtering module 154 to consider the distance (i.e., the distance) of the occluded regions 220 from the vehicle 102, the temporal evolution of the occlusions, and the objects around the vehicle 102. For example, the heuristics are incorporated (e.g., coded into) the filtering module 154. At 288, based on the inputs and the heuristics, the filtering module 154 filters out (i.e., removes) unimportant occluded regions from the map and determines the filtered occluded regions 221 relevant to the trajectory of the vehicle 102.

[0053] In general, the scene around vehicle 102 (i.e., the map of the surroundings of vehicle 102) may change dynamically. For example, the locations of other vehicles 200 and, accordingly, the occluded areas 220 may change as vehicle 102, other vehicles 200, and other objects such as cyclists and pedestrians move. Accordingly, the process of detecting occluded areas 220, determining filtered occluded areas 221, calculating importance scores of filtered occluded areas 221, and changing the trajectory 230 of vehicle 102 based on the importance scores may be repeated periodically (e.g., every second).

[0054] Furthermore, changing the trajectory 230 of the vehicle 102 may generate alternative trajectories from which the autonomous subsystem 126 of the vehicle 102 may select a trajectory. For example, if a trajectory for the vehicle 102 is for a left turn, the changed trajectory may be for executing the left turn differently (e.g., faster, slower, narrower, farther) than planned in the original trajectory, or for bringing the vehicle 102 to a complete stop. Thus, a trajectory change may include alternative decisions.

[0055] Fig. 7 illustrates in detail a method 300 performed by the evaluation module 156. The attention-based mechanism utilized by the trained DINO in the evaluation module 156 receives specific inputs about the vehicle 102 and the filtered occluded regions 221 and generates the importance scores for the filtered occluded regions 221, as described in detail below.

[0056] For example, the DINO in the evaluation module 156 includes a plurality of embedders (e.g., neural networks) that receive inputs as follows. For the vehicle 102, the DINO in the evaluation module 156 includes a query embedder, a key embedder, and a value embedder. The DINO in the evaluation module 156 includes a key embedder and a value embedder for each of the filtered hidden regions 221. The key embedder and the value embedder are shared for the filtered hidden regions 221 by or among the filtered hidden regions 221. The key embedder and the value embedder for the vehicle 102 are different from the key embedder and the value embedder shared by the filtered hidden regions 221.

[0057] Furthermore, the DINO in the evaluation module 156 does not use a query embedder for the filtered occluded regions 221, since the relationships between the filtered occluded regions 221 are not relevant for the evaluation of the filtered occluded regions 221. Rather, the spatial relationship of each of the filtered occluded regions 221 to the vehicle 102 is relevant for the evaluation of the filtered occluded regions 221. Thus, the DINO in the evaluation module 156 uses a query embedder for the vehicle 102, but does not use a query embedder for the filtered occluded regions 221.

[0058] Accordingly, the DINO in the evaluation module 156 may include five embedders: a query embedder, a key embedder, and a value embedder (3 embedders) for the vehicle 102, plus a key embedder and a value embedder (2 embedders) shared among the filtered hidden regions 221. For example, each of the five embedders may be a separate neural network. The neural networks of the DINO in the evaluation module 156 receive features, described below as inputs, and generate vectors, described below as outputs. For each neural network, the number of layers, the width of each layer, and associated activation functions are user-defined (i.e., selectable) parameters.

[0059] For example, let N denote a maximum number of filtered hidden regions to be tracked 221, where N is an integer greater than 1 that can be selected. According to the Fig. In the method 300 shown in Figure 7, the query embedder, the key embedder, and the value embedder receive input features for the vehicle 102 at 302, which are extracted from the trajectory 230 of the vehicle 102. The input features may include, for example, (x, y, v, a), where x and y are 2D coordinates, v is the velocity, and a is the acceleration of the vehicle 102.

[0060] At 304, the key embedder and the value embedder, which are shared among the filtered hidden regions 221, receive, for each of the filtered hidden regions 221, input features of the filtered hidden regions 221. For example, the input features of the filtered hidden regions 221 (s0, s end , l), where S o and S end are the start and end points of longitudinal coordinates of the filtered occluded area 221 and I is information about the lane in which the occlusion occurs.

[0061] At 306, the query embedder generates an output vector of dimension N for the vehicle 102 Q . The key embedder for the vehicle 102 generates an output vector of dimension N Q . The value embedder for the vehicle 102 generates an output vector of dimension N v . At 308, the key embedder for the filtered hidden region 221 generates an output vector of dimension N Q . The value embedder for the filtered hidden region 221 generates an output vector of dimension N v .

[0062] Subsequently, the keys and values ​​from the vehicle 102 and the filtered hidden regions 221 are concatenated in the DINO. For example, at 310, the keys from the vehicle 102 and the keys from the filtered hidden regions 221 are concatenated to form a key matrix K of dimension (N + 1) × N QAt 312, the values ​​from the vehicle 102 and the values ​​from the filtered hidden areas 221 are concatenated to form a value matrix V of dimension (N + 1) × N v to form.

[0063] At 314, a query matrix Q comprises the output of the query embedder for the vehicle 102 and the key matrix K comprises the outputs of the key embedders for the vehicle 102, and the filtered occluded regions 221 are multiplied to generate an attention matrix of dimension 1 × (N + 1). At 316, the attention matrix is ​​given by sqrt(N Q ). At 318, a softmax operator is applied to the attention matrix row by row; and at 320, a resulting matrix after application of the softmax operator is multiplied by the matrix V, which comprises the outputs of the value embedding for the vehicle 102 and the filtered occluded regions 221, to produce an output matrix of the DINO with dimension 1 × N vto generate.

[0064] To use the output of DINO to infer the importance scores for the filtered hidden regions 221, one of the following two approaches can be selected. A first approach enforces N v = N. In the first approach, the output of the multiplication between the attention matrix and the value matrix has the correct dimension (i.e., the maximum number of hidden regions, N). In a second approach, another layer or sequence of layers, called a head, can be added to the DINO. The head receives as input the 1 × N V-vector (i.e., the result of the multiplication between the attention matrix and the value matrix) and produces an output of dimension 1 × N. As with the neural networks of the embedders, the structure of the head (the number of layers, the width of the layers, and the activation function) can again be user-defined (i.e., selectable). Accordingly, at 322, either Nv = N is enforced or the output matrix of the DINO is input to a head to produce an output of dimension 1 × N. At 324, importance scores for the N filtered hidden regions 221 are inferred from the output of dimension 1 × N.

[0065] Subsequently, the motion planning module 158 may select the filtered occluded regions 221 with importance ratings greater than or equal to a selectable threshold and ignore the filtered occluded regions with importance ratings less than the selectable threshold. The motion planning module 158 processes only the selected ones of the filtered occluded regions 221, which reduces the computational load on the motion planning module 158, and may change the trajectory 230 of the vehicle 102 based on selected ones of the filtered occluded regions 221. Alternatively or additionally, the motion planning module 158 may process only the selected ones of the filtered occluded regions 221 and generate one or more alternative trajectories for the vehicle 102 based on the selected ones of the filtered occluded regions 221.

[0066] Fig.8 shows in detail a method 350 for training the DINO used in the evaluation module 156. For example, the DINO is trained in the evaluation module 156 using reinforcement learning as follows. During training, the DINO's embedders (neural networks) receive input features about the trajectory of the vehicle 102 and the occluded regions 220, as described above. For example, at 352, the query embedder, the key embedder, and the value embedder for the vehicle 102 receive input features extracted from the trajectory of the vehicle 102. The input features may include, for example, (x, y, v, a), where x and y are 2D coordinates, v is the velocity, and a is the acceleration of the vehicle 102. For example, at 354, the query embedder, the key embedder, and the value embedder receive input features extracted from the trajectory of the vehicle 102. B. the key embedder and the value embedder shared among the hidden regions 220, input features of the hidden regions 220.The input features of the hidden regions 220 include, for example, (s0, s. end , l), where s o , and s end are the start and end points of the longitudinal coordinates of the filtered occluded area 221 and I is information about the lane in which the occlusion occurs.

[0067] At 356, the DINO in the evaluation module 156 generates an output that includes the importance ratings for the occluded regions 220. The motion planning module 158 generates a sequence of actions (e.g., speed, acceleration, lane change, turn, etc.) for the trajectory of the vehicle 102 based on the importance ratings generated by the DINO.

[0068] At 358, the DINO is trained using a reward comprising two components. The reward comprising the two components is used to adjust the weights and biases of the neural networks in the DINO. The two components balance each other as follows, so that the net reward does not increase the computational load on the motion planning module 158 and also does not compromise the comfort, safety, or speed of the vehicle 102.

[0069] A first component of the reward signal is a base reward generated by the motion planning module 158. The motion planning module 158 generates the first component based on factors such as how quickly the vehicle 102 can complete the trajectory, how much comfort (e.g., jerk) the trajectory can generate while the vehicle 102 completes the trajectory, and how safely the vehicle 102 can complete the trajectory. The motion planning module 158 may generate the first component by optimizing one or more of these factors. For example, the motion planning module 158 may increase one factor (e.g., comfort) while decreasing another factor (e.g., speed).For example, the motion planning module 158 may use an equation or formula to maximize the base reward so that the vehicle 102 can execute the trajectory at a speed with maximum comfort and maximum safety.

[0070] According to some examples, if only the base reward component is used to train the DINO, the number of occluded regions detected as important may increase, which in turn increases the computational load on the motion planning module 158. Thus, the present disclosure adds a second reward component to the training signal used to train the DINO. The second reward component balances or compensates for the base reward component as follows.

[0071] The second reward component may be generated, for example, by summing the importance ratings of the occluded regions, multiplying the sum of the importance ratings by a negative coefficient, and adding the negative product to the first base reward component. Alternatively, the second reward component may be generated by selecting only those importance ratings greater than a selected threshold. For example, the importance ratings may be selected for occlusion regions with high importance ratings, e.g., because a pedestrian staggers into the road, because of an oncoming vehicle such as an emergency vehicle approaching vehicle 102, etc.The second reward component can then be generated by summing the selected high-importance ratings, multiplying the sum by a negative coefficient, and adding the negative product to the first base reward component. The second reward component reduces the first reward component so that the DINO does not receive an undue bias to maximize factors such as comfort, safety, and speed of the vehicle 102 in the base reward.

[0072] Accordingly, the reward signal used to train the DINO at 358 comprises a sum of the base reward component and another reward component comprising a product of a negative factor and a sum of importance ratings of the occluded regions. At 360, the weights and biases of the five embedders (neural networks) are adjusted using the learning signal from the reward. The adjustments continue until the DINO is trained. For example, training pauses if the user decides to pause training based on other metrics, such as the number of updates to the weights and biases of the neural networks. The trained DINO is then used in the evaluation module 156 of the navigation subsystem 120 of the vehicle 102, as described above.

[0073] The foregoing description is merely illustrative in nature and is not intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure may be implemented in a variety of forms. Thus, while this disclosure contains specific examples, the true scope of the disclosure is not intended to be so limited, since other changes will become apparent upon a study of the drawings, the specification, and the following claims.

[0074] It should be understood that one or more steps within a method may be performed in a different order (or simultaneously) without altering the principles of the present disclosure. Furthermore, although each of the embodiments has been described above as having certain features, one or more of these features described with respect to any embodiment of the disclosure may be implemented in and / or together with features of any of the other embodiments, even if this combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and interchanges of one or more embodiments with one another remain within the scope of this disclosure.

[0075] Spatial and functional relationships between elements (e.g., between modules, subsystems, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "adjacent," "on," "over," "under," and "disposed." When a relationship between a first and a second element is not explicitly described as "direct" in the above disclosure, that relationship may be a direct relationship, with no other intervening elements present between the first and second elements, but may also be an indirect relationship, with one or more intervening elements (either spatially or functionally) present between the first and second elements.As used herein, the phrase at least one of A, B, and C is intended to mean a logical (A OR B OR C) using a non-exclusive logical OR, and is not to be understood as meaning "at least one of A, at least one of B, and at least one of C."

[0076] In the figures, the direction of an arrow, as indicated by the arrowhead, generally illustrates the flow of information (such as data or instructions) of interest for the representation. For example, if an element A and an element B exchange a lot of information, but information passed from element A to element B is relevant for the representation, the arrow may point from element A to element B. This unidirectional arrow does not imply that no other information is passed from element B to element A. Furthermore, for information sent from element A to element B, element B may send requests for the information to element A or receive acknowledgments thereof.

[0077] In this application, including in the definitions below, the term "module", "controller", or "subsystem" may be replaced with the term "circuit". The term "module" or the term "subsystem" may refer to, be a part of, or include: an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0078] The module or subsystem may include one or more interface circuits. According to some examples, the interface circuits may include wired or wireless interfaces connected to a local area network (LAN), the Internet, a wide area network (WAN), or combinations thereof. The functionality of any given module or subsystem of the present disclosure may be distributed among multiple modules or subsystems connected via interface circuits. For example, multiple modules or subsystems may enable load balancing.

[0079] According to another example, a server (also known as a remote server or cloud) may perform some functionality on behalf of a client module or client subsystem.

[0080] The term code, as used above, may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term shared processor circuitry includes a single processor circuit that executes some or all of the code from multiple modules or subsystems. The term group processor circuitry includes a processor circuit that executes some or all of the code from one or more modules or subsystems along with additional processor circuitry. References to multiple processor circuits include multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above.The term shared memory circuit refers to a single memory circuit that stores some or all of the code from multiple modules or subsystems. The term group memory circuit refers to a memory circuit that stores some or all of the code from one or more modules or subsystems along with additional memories.

[0081] The term "memory circuit" is a subset of the term "computer-readable medium." As used herein, the term "computer-readable medium" does not include transitory electrical or electromagnetic signals that propagate through a medium (such as in a carrier wave); thus, the term "computer-readable medium" can be considered tangible and non-transitory.Non-limiting examples of a non-transitory, tangible computer-readable medium include non-volatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).

[0082] The devices and methods described in this application may be implemented partially or entirely by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions embodied in computer programs. The functional blocks, flowchart components, and other elements described above serve as software specifications that can be translated into computer programs through the routine work of a skilled technician or programmer.

[0083] The computer programs contain processor-executable instructions stored on at least one non-transitory, tangible computer-readable medium. Furthermore, the computer programs may contain or rely on stored data. The computer programs may include a basic input / output system (BIOS) that interacts with hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0084] The computer programs may contain: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language) or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, source code may be written using syntax from languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language, 5th Revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK and Python®.

Claims

[1] System for a vehicle, comprising: a plurality of sensors configured to sense the environment of the vehicle; a perception module configured to generate a map of the surroundings of the vehicle, the map including objects around the vehicle; an occlusion calculation module configured to calculate occluded areas on the map, wherein the occluded areas are areas around the vehicle that are occluded by one or more of the objects around the vehicle; a filtering module configured to filter out none or one or more of the occluded regions from the map, wherein the map comprises a plurality of filtered occluded regions after filtering; an evaluation module configured to evaluate the filtered occluded areas based on the importance of the filtered occluded areas for a trajectory of the vehicle; a motion planning module configured to modify the trajectory of the vehicle based on importance ratings of the filtered occluded regions; and a propulsion module configured to propel the vehicle in accordance with the modified trajectory. [2] The system of claim 1, wherein the motion planning module is configured to: Selecting one or more of the filtered hidden regions with importance ratings greater than or equal to a threshold; and Changing the vehicle's trajectory based on the selected filtered occluded areas. [3] The system of claim 1, wherein: the evaluation module comprises a neural network configured to evaluate the filtered hidden regions; and the neural network is trained using a base reward component and a second reward component that is balanced with the base reward component. [4] The system of claim 3, wherein the second reward component comprises a product of a negative factor and a sum of the importance ratings of the filtered hidden regions. [5] The system of claim 1, wherein the evaluation module comprises: a first plurality of neural networks configured to receive features associated with the trajectory of the vehicle as inputs and to generate first outputs; and a second plurality of neural networks configured to receive features associated with the filtered hidden regions and to generate second outputs, wherein the evaluation module is configured to output the importance ratings of the filtered hidden regions based on the first outputs and the second outputs. [6] The system of claim 5, wherein the second plurality of neural networks are shared among the filtered hidden regions. [7] The system of claim 5, wherein the second plurality of neural networks is different from the first plurality of neural networks. [8] The system of claim 1, wherein the filtering module is configured to filter out one or more of the occluded areas from the map based on the relevance of the occluded areas to the trajectory of the vehicle. [9] The system of claim 1, wherein the filtering module is configured to filter out none or one or more of the occluded regions from the map based on the route, state, and trajectory of the vehicle; states of moving objects around the vehicle; predictions about the moving objects around the vehicle; and heuristics including the occluded regions that do not intersect the route of the vehicle, the size and proximity of the occluded regions relative to the vehicle, and the temporal evolution of the occluded regions and the moving objects around the vehicle. [10] The system of claim 1, wherein the occlusion calculation module is configured to calculate the occluded areas based on data received from a mapping system that identifies static objects, including buildings and road configuration along a path of the vehicle.

Images