Top-down scene prediction based on action data

Abstract

Described herein are techniques for determining predictions in a top-down representation of an environment based on vehicle actions. Sensors of a first vehicle (e.g., an autonomous vehicle) can capture sensor data of the environment, which may include objects (e.g., vehicles or pedestrians) away from the first vehicle. A multi-channel image representing a top-down view of the objects and the environment can be generated based on the sensor data, map data, and / or action data. Environmental data (e.g., object range, speed, lane location, crosswalk location, etc.) can be encoded in the image. The action data can represent the first vehicle's target lane, trajectory, etc. Multiple images representing the environment over time can be generated and input into a prediction system configured to output predicted probabilities associated with possible future object locations, which may be based on the autonomous vehicle's actions.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This patent application claims priority to U.S. utility patent application Serial No. 16 / 504,147, filed July 5, 2019. Application Serial No. 16 / 504,147 is incorporated herein by reference in its entirety. [Background technology]

[0002] Prediction techniques can be used to determine the future state of entities in an environment, i.e., how a particular entity is likely to behave in the future. Current prediction techniques often involve physics-based modeling or road rule simulation to predict the future state of entities in an environment. [Brief description of the drawings]

[0003] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears, and the use of the same reference number in different drawings indicates similar or identical items or features.

[0004] [Figure 1] FIG. 1 is an illustrated flow diagram of an exemplary process for receiving object data and action data, inputting a multi-channel image representing an environment into a machine learning model, and receiving a prediction probability based on the object data and action data, in accordance with a disclosed example. [Figure 2A] FIG. 2A illustrates a block diagram of an exemplary system for implementing the techniques described herein. [Figure 2B] FIG. 2B shows a block diagram of an example training component for training a machine learning model to implement the techniques described herein. [Diagram 3]FIG. 3 illustrates an exemplary diagram of inputs to a predictive system and training data for training the predictive system, according to an example of the present disclosure. [Figure 4] FIG. 4 is an illustration of first and second predicted probabilities for a scenario including two double-parked vehicles, where the second predicted probability is based in part on action data, according to an example of this disclosure. [Diagram 5] FIG. 5 is an illustration of first and second predicted probabilities for a scenario involving double-parked vehicles, where the second predicted probability is based in part on action data, according to an example of this disclosure. [Figure 6] FIG. 6 is an illustration of first and second predicted probabilities for a scenario including an object turning in front of an autonomous vehicle, where the second predicted probability is based in part on action data, in accordance with an example of this disclosure. [Figure 7] FIG. 7 is an illustration of first and second predicted probabilities for a scenario including an autonomous vehicle traversing a junction, where the second predicted probability is based in part on action data, according to an example of this disclosure. [Figure 8] FIG. 8 is an illustration of first and second predicted probabilities for a scenario including an autonomous vehicle entering a blocked junction, where the second predicted probability is based in part on action data, in accordance with an example of this disclosure. [Figure 9] FIG. 9 is an illustration of first and second predicted probabilities for a scenario including an autonomous vehicle crossing another junction, where the second predicted probability is based in part on action data, in accordance with an example of this disclosure. [Figure 10] FIG. 10 illustrates an exemplary process for receiving sensor data and action data, generating a multi-channel image, evaluating prediction probabilities, and controlling a vehicle based on the prediction probabilities according to an example of this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Techniques for determining predictions based on a top-down representation of an environment and vehicle actions are described herein. A sensor of a first vehicle (e.g., an autonomous vehicle) can capture sensor data of the environment, which may include objects distant from the vehicle, such as a second vehicle or pedestrians. A multi-channel image (which may simply be referred to throughout as an image) encoding various parameters of the objects and / or environment in a top-down view can be generated based on the sensor data, map data, and / or action data. The sensor data can be used to generate one or more channels of the multi-channel image representing bounding boxes and speed information associated with the objects, map data, and other semantic information (e.g., speed limits, lane widths, lane edges, stop lines, traffic lights, traffic light status, etc.). The action data can be used to generate one or more channels of the multi-channel image representing a target lane, trajectory, etc. of the first vehicle (which may correspond to actions including, for example, a vehicle passing through an intersection, lane changes, turns, etc.). A plurality of images representing the environment over time can be generated and input into a prediction system configured to output a prediction probability associated with a possible future location of the object. A predicted probability may be generated or determined based on particular candidate actions, and the predicted probability may be evaluated to select or determine a candidate action for controlling the autonomous vehicle.

[0006] Autonomous driving in dense urban environments is challenging due to the complex reasoning often used to resolve multi-way interactions between objects. This reasoning can be time-critical and constantly evolving. The techniques described herein are targeted at driving scenarios, which may include, but are not limited to, urban intersections without traffic lights. At these junctions, multiple objects (e.g., vehicles, pedestrians, cyclists) often compete for the same shared space, so it is useful to predict the intent of the objects to successfully navigate the intersection. The techniques described herein are targeted at predicting the driving behavior of objects traversing through the junction given several snapshots of the environment's history. Some techniques, as described herein, use semantic segmentation-style approaches to formulate deep convolutional neural network (CNN) architectures that can forecast the future as a predicted probability (e.g., heatmaps).

[0007] In some examples, the predicted probability associated with an object may not be limited to a Gaussian distribution across a set of different operations, but may instead include a heat map indexed over time of where the object will be in the future (e.g., with an arbitrary distribution). In some examples, the intensity of the heat map may represent the probability that a cell or pixel will be occupied by an arbitrary object at a specified instance in time (e.g., an occupancy grid). In some examples, the heat map may capture both an arbitrary trajectory distribution per object and a distribution across distinct operations (e.g., candidate actions) without explicit operation specification. In some examples, this approach naturally aggregates many future modes into a single heat map. In some examples, some signals regarding the autonomous vehicle's intent can be used to resolve this multi-modality. In such examples (e.g., conditional on the intent of the action), such a heat map may include a more tightly packed probability distribution.

[0008] In order for a vehicle (such as an autonomous vehicle or an autonomous robot) to safely navigate an intersection, it should reason about the behavior of other objects in the world. In an example discussed below in connection with FIG. 1, an autonomous vehicle may cross an intersection. If the autonomous vehicle can infer whether to yield to an oncoming object, it can safely traverse the junction. In some examples, the techniques described herein may include conditioning the prediction probability (e.g., heat map) to the autonomous vehicle's intent (represented as action data). By conditioning the heat map to the autonomous vehicle's intent, the techniques described herein may include minimizing the multi-modality of the predicted heat map. That is, predictions of objects in the environment may be based on candidate actions proposed to be performed by the autonomous vehicle, and such predictions may include improved predictions for additional objects. The output of the machine learning model may be used in a planning system to determine a cost function based on the heat map output and search for an optimal action for the autonomous vehicle that minimizes the cost of the heat map output.

[0009] As described above, the technique may include generating a plurality of multi-channel images representative of an environment over time. The image or plurality of images may be input to a predictive system (e.g., a machine learning model) configured to output a predicted probability associated with a possible future location of an object. The predicted probability may be associated with a candidate action of a plurality of candidate actions. Each candidate action may be evaluated and a risk, cost, and / or reward associated with the candidate action may be determined, and a candidate action may be selected or determined based at least in part on the evaluation of the candidate action. The autonomous vehicle may be controlled based at least in part on the selected or determined candidate action.

[0010] The sensor data captured by the autonomous vehicle can include lidar data, radar data, image data, time of flight data, sonar data, etc. In some cases, the sensor data can be provided to a perception system configured to determine the type of object in the environment (e.g., vehicle, pedestrian, bicycle, animal, parked car, tree, building, etc.). Additionally, the perception system can determine motion information about the object in the environment based on the sensor data. The sensor data and any data based on the sensor data can be represented in a top-down view of the environment. For example, the image can represent the object as a two-dimensional bounding box representing the location of the object in the environment, as well as the extent of the object (e.g., the length and width of the object) and the classification of the object (e.g., vehicle, pedestrian, etc.). Motion information, such as speed information, can be represented as a velocity vector associated with the bounding box, although other representations are envisioned.

[0011] In some cases, the image input to the prediction system may be represented by individual channels of a multi-channel image, with each channel representing different information about the object and the environment the object is located in. For example, the channels may represent one or more of, but are not limited to, road network information (e.g., semantic labels indicating lanes, crosswalks, stop lines, lane dividers, stop signs, intersections, traffic lights, etc.), traffic light status (e.g., red, yellow, green, etc.), bounding box associated with the object, the object's speed in x and y directions, the object's acceleration in x and y directions, the object's flashing status (turn left, turn right, brake, reverse, hazard, no lights, etc.), etc., that may be determined based on sensor data or otherwise received / encoded in map data available to the system.

[0012] In some examples, the planning system can generate or otherwise determine one or more candidate actions for the autonomous vehicle to perform within the environment. For example, a first candidate action can include staying in the current lane, while a second candidate action can include a lane change to the left or right of the current lane. As another example, a third candidate action can include crossing through a junction, while a fourth candidate action can include a left or right adjustment at the junction. In some examples, the candidate actions can represent target lanes or candidate trajectories for the vehicle to follow within the environment as highlighted, illustrated, annotated, or otherwise represented in the multi-channel image.

[0013] In some examples, the candidate actions may be associated with one or more subgoals. For example, the subgoals may include speed or acceleration information, such as a target speed or acceleration, an average speed or acceleration, a maximum or minimum speed or acceleration, etc. In some examples, the subgoals may include, but are not limited to, a steering angle, a distance from an object, a distance from a reference line, etc. In some examples, the subgoals may be represented as channels of a multi-channel image.

[0014] In some examples, multiple channels can be input to the prediction system to generate predicted probabilities. In some examples, an image can be input directly to the prediction system (e.g., without generating multiple channels based on the image) to generate predicted probabilities.

[0015] In some examples, the number of channels may be reduced to fewer channels. As a non-limiting example, a simple three-channel (e.g., RGB) image may be used to represent multiple semantic information. In such examples, for example, a red channel may be used to indicate a bounding box, and the intensity of the red value may be associated with speed, acceleration, etc. In such examples, a second channel may encode lane width with the intensity indicating a speed limit, etc. Additionally, in such examples, a third channel may encode a candidate action by identifying a target lane associated with a particular candidate action.

[0016] In some examples, the prediction system can include a machine learning model trained to output data that can be used to generate one or more predicted trajectories. For example, the machine learning model can output coordinates (e.g., x and y coordinates) associated with an object (e.g., a third-party vehicle) at one or more times in the future (e.g., 1 second, 2 seconds, 3 seconds, etc.). In some examples, the machine learning model can output coordinates associated with the object and probability information associated with each coordinate. In some examples, the machine learning model can include a convolutional neural network (CNN), which may include one or more recurrent neural network (RNN) layers, such as, but not limited to, a long short-term memory (LSTM) layer. In some examples, the machine learning model can output a heat map associated with the predicted probabilities. In some examples, at least one predicted trajectory can be determined based at least in part on the heat map.

[0017] The heat map can represent a personalized region of the environment proximate to the autonomous vehicle. For example, the heat map can represent a 100 meter by 100 meter region around the autonomous vehicle. × 64 grid (or J ×The heatmap may represent a region of any size, and may represent an area of ​​any number of distinct portions. In some cases, the portions of the heatmap may be referred to as cells of the heatmap. Each cell may include a predicted probability representing the probability that an object is at a corresponding location in the environment at a time corresponding to the heatmap. In some examples, the machine learning model may output multiple heatmaps, and a heatmap of the multiple heatmaps may represent a probability prediction associated with the environment (e.g., objects and / or autonomous vehicles) at a particular time in the future (e.g., 0.5 seconds, 1 second, 3 seconds, 5 seconds, 10 seconds, etc.).

[0018] In some examples, one or more heat maps can be provided to the planning system instead of or in addition to at least one predicted trajectory.

[0019] In some examples, the techniques described herein can include generating possible scenarios as one or more probabilistic maps that take into account the interactions and present an interpretable marginal cost of the tree search. In some examples, the techniques include conditioning on the autonomous vehicle's intent via network inputs (e.g., as one or more channels of a multi-channel image, without constraining the autonomous vehicle's trajectory). Additionally, aspects of the present disclosure include machine learning algorithms (e.g., convolutional neural networks) that can predict the future state of objects around the autonomous vehicle as well as the autonomous vehicle itself as heat maps stacked over time. As described above, the heat maps (or predicted probabilities) output from the machine learning models can be conditioned on one or more candidate actions of the autonomous vehicle. Additionally, the techniques can include individualizing the autonomous vehicle's intent into goals and sub-goals, allowing the planning system to search for the best (or optimal) candidate actions for the autonomous vehicle. Additionally, aspects of the present disclosure include data indicative of different types of interactions between the autonomous vehicle and other objects in the environment.

[0020] The techniques described herein can improve the functionality of a computing device in several additional ways. In some cases, representing the environment and objects in the environment as a top-down view can represent a simplified representation of the environment for purposes of generating predicted probabilities and / or selecting from candidate actions. In some cases, the image can represent the environment without extracting specific features of the environment, which can simplify the generation of the prediction system and subsequently the generation of at least one predicted trajectory. In some cases, evaluating the predicted probabilities can enable the autonomous vehicle to generate a more accurate and / or safer trajectory for the autonomous vehicle to traverse the environment. For example, a predicted probability associated with a first candidate action can be evaluated to determine the likelihood of a collision or near-collision, which can enable the autonomous vehicle to select or determine another candidate action (e.g., change lanes, stop, etc.) to safely traverse the environment. In at least some examples described herein, conditioning predictions based on a top-down encoding of the environment in addition to the intended action can minimize (improve) the broadening of probability distribution functions associated with objects, resulting in safer decision-making of the system. These and other improvements to the functionality of a computer are described herein.

[0021] The techniques described herein can be implemented in several ways. Exemplary implementations are described below with reference to the following drawings. Although described in the context of an autonomous vehicle, the methods, apparatus, and systems described herein can be applied to various systems (e.g., sensor systems, or robotic platforms) and are not limited to autonomous vehicles. In one example, similar techniques can be utilized in driver-controlled vehicles where such systems can provide indications of whether it is safe to perform various operations. In another example, the techniques can be utilized in an aviation or nautical context, or in any system that includes objects or entities that can be associated with actions that are unknown to the system. Furthermore, sensor data can include any two-dimensional, three-dimensional, or multi-dimensional data, such as image data (e.g., stereo cameras, time-of-flight data, etc.), lidar data, radar data, sonar data, etc. In addition, the techniques described herein can be used with real data (e.g., captured using a sensor), simulated data (e.g., generated by a simulator), or any combination of the two.

[0022] FIG. 1 is an illustrated flow diagram of an exemplary process 100 for receiving object data and action data, inputting a multi-channel image representing an environment into a machine learning model, and receiving a prediction probability based on the object data and action data, in accordance with a disclosed example.

[0023] In operation 102, the process may include receiving object data representing objects in the environment. In some examples, the object data may be received from a perception system of the autonomous vehicle. For example, the object data may include, but is not limited to, one or more bounding boxes associated with the environment, speed and / or acceleration information, classification information, and the like. In some examples, operation 102 may include capturing sensor data of the environment. In some examples, the sensor data may be captured by one or more sensors of the autonomous vehicle. For example, the sensor data may include data captured by one or more of a lidar sensor, a radar sensor, an image sensor, a time of flight sensor, a sonar sensor, and the like. In at least some examples, additional environmental data may be received (e.g., map data of the environment encoding portions of lanes, reference lines, crosswalks, traffic devices and / or signs, etc.).

[0024] Example 104 illustrates an environment in which an autonomous vehicle 106 is traversing the environment. In some examples, the autonomous vehicle 106 can perform the operations of process 100, which can include capturing sensor data of the environment. For illustrative purposes, the vehicle capturing (or utilizing) the sensor data can be an autonomous vehicle configured to operate in accordance with a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle that can perform all safety-critical functions for the entire journey without expecting a driver (or passenger) to be in control of the vehicle at all times. In such an example, the vehicle can be configured to control all functions from start to stop, including all parking functions, and thus can be unmanned. This is merely an example, and the systems and methods described herein can be incorporated into any land, air, or water vehicle, including vehicles that require manual control at all times by a driver, to those that are partially or fully autonomously controlled. Additional details associated with the vehicle are described throughout this disclosure.

[0025] Objects 108 are also located within the environment. In some examples, the objects 108 can represent vehicles, pedestrians, bicycles, animals, etc. The environment shown in example 104 can include drivable and non-drivable areas. In some examples, the operations 102 can include locating the autonomous vehicle 106 within the environment and accessing map data associated with the environment. For example, the map data can include map elements such as lane markings, lane boundaries, one or more lane references 110 (e.g., indicating centerlines associated with lanes and / or available (e.g., legal) routes between lanes). Additional examples of map elements can include, but are not limited to, one or more lane elements, bike lane elements, crosswalk elements, intersection elements, lane division elements, traffic light elements, stop sign elements, stop line elements, yield symbol elements, yield line elements, parking lane elements, driveway elements, speed bump elements, joywalking areas (e.g., virtual crosswalks), track waypoints (e.g., known tracks), passenger pick-up points, sign location elements, geofence elements, etc.

[0026] In some examples, the sensor data captured in operation 102 may be used to determine information about the object 108, including, but not limited to, the type of object (e.g., semantic information indicative of a classification of the object, such as a vehicle, a pedestrian, a bicycle, an animal, etc.) In some examples, operation 102 may include determining the type of object 108, a bounding box associated with the object 108, and / or motion information associated with the object, as described herein.

[0027] At operation 112, the process may include receiving action data associated with candidate actions for an autonomous vehicle in an environment. Example 114 illustrates candidate action 116 (e.g., a first candidate action, such as a yield action) and candidate action 118 (e.g., a second candidate action, such as a "go through intersection" action).

[0028] That is, the candidate actions 116 may include an instruction from the planning system of the autonomous vehicle 106 to yield to the object 108 and refrain from crossing the intersection until the object 108 has crossed the intersection (or some other trigger, such as the passage of a certain period of time).

[0029] The candidate actions 118 may include traversing the junction shown in the example 104 before allowing the object 108 to traverse the junction.

[0030] In some examples, the action data can be represented in various ways. In some examples, the example 104 shows the action data as a current lane 120 and a target lane 122. In some examples, as described herein, information associated with the candidate actions 116 and 118 can be represented via the current lane 120 and / or the target lane 122, as well as a velocity profile and / or an acceleration profile (e.g., a subgoal).

[0031] In operation 124, the process can include inputting a multi-channel image representing a top-down view of the environment into the machine learning model, where the multi-channel image is based on the object data and the action data. Examples of such multi-channel images are shown in examples 126 and 128.

[0032] The example 126 represents a first multi-channel image associated with the first candidate action 116. For example, the example 126 includes a first channel 132, a second channel 134, and a third channel 136. In some examples, the first channel 132 may represent a bounding box, a position, an extent (e.g., length and width), etc. of the autonomous vehicle 106 and / or the object 108 within the environment. In some examples, the second channel 134 may represent action data associated with the candidate action 116. For example, the second channel 134 may indicate a current lane 120 and / or a target lane 122 associated with the candidate action 116. In some examples, the third channel 136 may represent additional action data, in this case corresponding to speed information 140 (e.g., V1) associated with the autonomous vehicle 106 performing the candidate action 116. In some examples, the speed information 140 may include an instantaneous speed, an average speed, etc. For example, where the candidate action 116 indicates a yielding behavior, the speed information 140 may represent a relatively low speed (or may represent a stopping speed while the autonomous vehicle 106 yields to the object 108). Although described in the context of speed, the speed information 140 may represent information associated with acceleration (e.g., average throughout the action, maximum acceleration associated with the action, etc.), steering angle, distance from a baseline or obstacle, etc.

[0033] Example 128 represents a second multi-channel image associated with the second candidate action 118. In some examples, some aspects of example 128 may be equivalent to some aspects of example 126. For example, example 128 may include a first channel 132, a second channel 134, and a third channel 142. In some examples, the third channel may include speed information 144 that may be associated with the second candidate action 118. For example, where the second candidate action 118 may represent an autonomous vehicle 106 crossing through an intersection before an object 108, the speed information 144 (e.g., V2) may represent a higher average speed (or speed profile) than the second speed information 140 associated with the first candidate action 116. As described above in the context of speed, the speed information 144 may represent information associated with acceleration (e.g., average throughout the action, maximum acceleration associated with the action, etc.), steering angle, distance from a reference line or obstacle, etc.

[0034] Of course, example 128 may include different action data than the action data represented in second channel 134 if the second candidate action 118 represents a different position than the first candidate action 116 (e.g., if the first candidate action 116 represents going straight and the second candidate action 118 represents a left or right turn).

[0035] In some examples, encoding the autonomous vehicle 106's intent (e.g., via channel 134) may be provided as partial future information without specifying the autonomous vehicle 106's exact trajectory. In some examples, intent may be encoded 1) by painting or otherwise indicating an entrance lane (e.g., current lane 120) and an exit lane (e.g., target lane 122); and 2) by providing separate blobs indicating high and low future average accelerations and speeds (e.g., as channels 136 and / or 142).

[0036] The first intent signal (e.g., second channel 134) above uses a target lane 122 associated with a particular candidate action. By providing the target lane 122, the technique can eliminate multi-modal outputs, such as for each junction exit, in favor of one that is based on the planned trajectory associated with the autonomous vehicle 106.

[0037] The second intent above (e.g., channels 136 and 142) indicates a subgoal and may include the intended speed and / or acceleration profile of the autonomous vehicle 106. In some cases, this subgoal intent provides an individualized value of the average acceleration of the autonomous vehicle 106 in the future up to a specified prediction horizon. In some cases, the subgoal intent indicates what the autonomous vehicle will do in the near future. For example, a sudden deceleration means the autonomous vehicle will brake for some other object. A sudden acceleration means the object is speeding up to pass a junction. Similar intents also apply to the speed dimension of the autonomous vehicle 106. The intent is designed to remove multi-modal output, but at the same time, does not constrain the output of the autonomous vehicle 106 and / or the object 108 to a single trajectory.

[0038] As can be appreciated, examples 126 and 128 can include multiple multi-channel images representing an environment at various times within the environment. For example, examples 126 and / or 128 can represent the history of autonomous vehicle 106 and object 108 (and other objects such as pedestrians and vehicles) at 0.5 second intervals over the past 4 seconds, although any number of instances and time periods can be used to represent an environment.

[0039] In some examples, instances 126 and / or 128 may include a time step (e.g., at time t0) and / or a time-lapse (e.g., at time t -mto time t0), such as turn signals, brake lights, hazard lights, etc. The additional channels may include, but are not limited to, kinematic information (also referred to as kinematic data, motion information, or motion data) of the autonomous vehicle 106 and / or object 108 (e.g., average speed or acceleration over a period of time, instantaneous speed or acceleration at a previous point in time, wheel angles, turn rate, yaw, pitch, roll, etc.), semantic information (e.g., vehicle classification, turn signal indicators, brake light indicators, double parked vehicle indication, etc.), occlusion grids (e.g., indicating areas of the environment that are not visible to one or more sensors of the autonomous vehicle 106 or that are predicted to be occluded, etc.), and the like.

[0040] As can be appreciated, operation 124 can include generating or determining a multi-channel image, as described herein. An example of generating or determining a multi-channel image is described in connection with U.S. Patent Application No. 16 / 151,607, entitled "Trajectory Prediction on Top-Down Scenes," filed October 4, 2018. U.S. Patent Application No. 16 / 151,607 is incorporated herein by reference in its entirety.

[0041] At operation 146, the process may include receiving a predicted probability associated with the object and / or the autonomous vehicle from the machine learning model and based on the multi-channel image. As can be appreciated, the multi-channel image may be based on action data, as discussed herein, and thus the predicted probability may be based on the action data.

[0042] Examples 148 and 150 each illustrate the output of a machine learning model. In some examples, examples 148 and 150 include heat maps that can represent predicted probabilities (e.g., from a top-down perspective of the environment) that indicate the likelihood that object 108 will be at a corresponding location in the environment based on a time and a respective candidate action corresponding to each heat map. That is, example 148 illustrates a prediction probability that object 108 will be at a corresponding location in the environment based on a time and a respective candidate action corresponding to each heat map. 1A , with the designation "A" corresponding to the first candidate action 116. Example 150 shows a candidate action 116 at time t 1B , with the "B" designation corresponding to the second candidate action 118. The examples 148 and 150 may represent heat maps and / or predicted probabilities of future autonomous vehicles 106 and / or objects 108. Of course, the example heat maps 148 and 150 may represent predicted probabilities in any time period and are not limited to the express times described herein.

[0043] Example 148 shows the predicted probabilities 152, 154, 156, and 158 In some examples, the prediction probabilities 152 and 154 represent possible locations of the object 108 in the environment based at least in part on the autonomous vehicle 106 performing the first candidate action 116. The prediction probabilities 156 and 158 represent possible locations of the autonomous vehicle 106 conditional on the action data (e.g., channels 134 and 136).

[0044] Example 150 represents prediction probabilities 160, 162, and 164. In some examples, prediction probability 160 represents a possible location of object 108 in the environment based at least in part on the autonomous vehicle 106 performing the second candidate action 118. Prediction probabilities 162 and 164 represent possible locations of the autonomous vehicle 106 conditional on the action data (e.g., channels 134 and 142).

[0045] Further, process 100 may include evaluating the candidate actions and / or controlling the autonomous vehicle 106 based at least in part on the candidate actions. For example, process 100 may include determining an amount of overlap of bounding boxes representing the autonomous vehicle 106 and a predicted probability associated with the object 108. In some examples, the process may include evaluating candidate trajectories of the autonomous vehicle 106 to determine an overlap between predicted future movements and the predicted probabilities 156, 158, 162, and / or 164.

[0046] 2A illustrates a block diagram of an example system 200 for implementing the techniques described herein. In at least one example, the system 200 may include a vehicle 202.

[0047] The vehicle 202 may include one or more vehicle computing devices 204 (also referred to as one vehicle computing device 204 or multiple vehicle computing devices 204), one or more sensor systems 206, one or more emitters 208, one or more communication connections 210, at least one direct connection 212, and one or more drive systems 214.

[0048] The vehicle computing device 204 may include one or more processors 216 and a memory 218 communicatively coupled to the one or more processors 216. In the illustrated example, the vehicle 202 is an autonomous vehicle, although the vehicle 202 may be any other type of vehicle. In the illustrated example, the memory 218 of the vehicle computing device 204 stores a position estimation component 220, a perception component 222, one or more maps 224, one or more system controllers 226, a prediction component 228 including an image generation component 230 and a machine learning component 232, and a planning component 234 including a trajectory generation component 236 and an evaluation component 238. While shown in FIG. 2 as residing in memory 218 for illustrative purposes, it is contemplated that the position estimation component 220, the perception component 222, the one or more maps 224, the one or more system controllers 226, the prediction component 228, the image generation component 230, the machine learning component 232, the planning component 234, the trajectory generation component 236, and the evaluation component 238 may additionally or alternatively be accessible to the vehicle 202 (e.g., stored remotely).

[0049] In at least one example, the position estimation component 220 can include functionality to receive data from the sensor system 206 to determine the position and / or orientation of the vehicle 202 (e.g., one or more of x, y, z positions, roll, pitch, or yaw). For example, the position estimation component 220 can include and / or request / receive a map of the environment and can continually determine the position and / or orientation of the autonomous vehicle within the map. In some examples, the position estimation component 220 can receive image data, LIDAR data, radar data, IMU data, GPS data, wheel encoder data, etc., utilizing SLAM (simultaneous localization and mapping), CLAMS (calibration, localization and mapping simultaneously), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine the position of the autonomous vehicle. In some examples, the position estimation component 220 can provide data to various components of the vehicle 202 as described herein to determine an initial position of the autonomous vehicle for generating a trajectory and / or for generating or receiving map data.

[0050] In some examples, the perception component 222 may include functionality to perform object detection, segmentation, and / or classification. In some examples, the perception component 222 may provide processed sensor data indicative of the presence of an entity proximate to the vehicle 202 and / or the classification of the entity as an entity type (e.g., automobile, pedestrian, bicycle, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional or alternative examples, the perception component 222 may provide processed sensor data indicative of one or more characteristics associated with a detected entity (e.g., a tracked object) and / or with the environment in which the entity is located. In some examples, the characteristics associated with the entity may include, but are not limited to, an x-position (global and / or local position), a y-position (global and / or local position), a z-position (global and / or local position), an orientation (e.g., roll, pitch, yaw), an entity type (e.g., classification), an entity's velocity, an entity's acceleration, an entity's range (size), etc. Characteristics associated with an environment may include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, the time of day, the day of the week, the season, weather conditions, darkness / light indications, etc.

[0051] The memory 218 may further include one or more maps 224 that may be used by the vehicle 202 to navigate within the environment. For purposes of this description, a map may be any number of data structures modeled in two, three, or N dimensions that may provide information about the environment, such as, but not limited to, topology (such as intersections), roadways, mountain ranges, roads, terrain, and the general environment. In some examples, the map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), and reflectance information (e.g., specular reflectance information, retroreflectance information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some examples, the map can be stored in a tile format, such that individual tiles of the map represent separate portions of the environment, and can be loaded into the working memory as needed, as described herein. In at least one example, the one or more maps 224 can include at least one map (e.g., an image and / or a mesh). In some examples, the vehicle 202 can be controlled at least in part based on the map 224. That is, the map 224 can be used in conjunction with the position estimation component 220, the perception component 222, the prediction component 228, and / or the planning component 234 to determine a position of the vehicle 202, identify objects in the environment, generate predicted probabilities associated with the objects and / or the vehicle 202, and / or generate a route and / or trajectory for navigating the environment.

[0052] In some examples, one or more maps 224 may be stored on a remote computing device (such as computing device 242) accessible via network 240. In some examples, multiple maps 224 may be stored, for example, based on a characteristic (e.g., entity type, time of day, day of the week, season, etc.). Storing multiple maps 224 may have similar memory requirements but increases the speed at which the data in the maps can be accessed.

[0053] In at least one example, the vehicle computing device 204 can include one or more system controllers 226, which can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 202. These system controllers 226 can communicate with and / or control corresponding systems of the drive system 214 and / or other components of the vehicle 202.

[0054] In general, the prediction component 228 can generate predicted trajectories of objects in the environment. For example, the prediction component 228 can generate one or more predicted trajectories of vehicles, pedestrians, animals, etc., within a threshold distance from the vehicle 202. In some examples, the prediction component 228 can measure traces of the objects and generate trajectories of the objects based on the observed and predicted behavior.

[0055] In some examples, the image generation component 230 can include functionality for generating images that represent a top-down view of a scene. Examples of such images generated by the image generation component 230 are shown in FIG. 1 as examples 104, 126, and 128.

[0056] In some examples, the image generation component 230 can generate an image representing an area around the vehicle 202. In some examples, the area can be based at least in part on an area visible from a sensor (e.g., sensor range), a receding horizon, an area associated with an action (e.g., crossing an intersection), and the like. In some examples, the image can represent an area of ​​100 meters by 100 meters around the vehicle 202, although any area is contemplated. The image generation component 230 can receive data about objects in the environment from the perception component 222, and data about the environment itself from the localization component 220, the perception component 222, and one or more maps 224. The image generation component 230 can generate a top-down view of the environment including objects in the environment (e.g., represented by bounding boxes as described herein), semantic information about the objects (e.g., classification type), motion information (e.g., speed information, acceleration information, etc.), and the like. The image generation component 230 may further generate image elements (e.g., symbols or other representations) within the image that represent various attributes of the object and the environment, including, but not limited to, one or more velocity vectors, one or more acceleration vectors, lighting indicators (e.g., the brake lights of the object, the turn signal lights of the object, etc., represented as respective indicator statuses), traffic light information (e.g., the status of the traffic light), lane information (e.g., whether a lane is a left-turn-only lane or a right-turn-only lane), etc.

[0057] Additionally, the image generation component 230 can generate one or more channels that include action data, as described herein. For example, the action data can include identifying a current lane and / or a target lane associated with a candidate action.

[0058] In some examples, the image generation component 230 can generate one or more images that include semantic information of the environment, as described in more detail herein.

[0059] In some examples, the image generation component 230 can generate one or more images that represent traces of objects in the environment. The images generated by the image generation component 230 can represent prior information about the state of the environment and objects in the environment that can be used to generate predicted probabilities, as described herein.

[0060] In any of the examples described herein, the image generation component 230 may generate an image having multiple channels, with each channel representing some information (semantic or otherwise).

[0061] In some examples, the machine learning component 232 (also referred to as the machine learning model component 232) may include functionality to receive images generated by the image generation component 230 and generate one or more heat maps including predicted probabilities, as described herein.

[0062] In some examples, the machine learning component 232 can include one or more machine learning models trained to output one or more heatmaps based on top-down images input to the models. In some examples, the machine learning component 232 can include a convolutional neural network (CNN), which may or may not include one or more recurrent neural network (RNN) layers, e.g., a long short-term memory (LSTM) layer.

[0063] In some examples, the heat map output by the machine learning component 232 can represent a personalized region of the environment proximate to the autonomous vehicle. For example, the heat map can represent a 100 meter by 100 meter area around the autonomous vehicle. ×64 grid (or J × The heatmap may represent a region of any size (e.g., a grid of size K). In some examples, the area represented by the heatmap may correspond to the area represented by the image input to the machine learning component 232. Of course, the heatmap may represent an area of ​​any size and may represent an area of ​​any number of discrete portions. In some cases, a portion of the heatmap may be referred to as a cell of the heatmap. Each cell may include a predicted probability representing the probability that the object is at a corresponding location in the environment at a time corresponding to the heatmap. In some examples, the machine learning model may output multiple heatmaps, where the heatmap of the multiple heatmaps may represent probability predictions associated with the object at a particular time in the future (e.g., 0.5 seconds, 1 second, 3 seconds, 5 seconds, 10 seconds, etc.).

[0064] In some examples, the machine learning component 232 can be trained to generate any number of heatmaps based on any number of input images. For example, the machine learning component 232 can receive N input images and output M heatmaps.

[0065] In general, the planning component 234 can determine a path for the vehicle 202 to follow to traverse through an environment.

[0066] In some examples, the trajectory generation component 236 can include functionality for determining various routes and trajectories as well as various levels of detail. For example, the trajectory generation component 236 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this description, the route can be a series of waypoints for travel between the two locations. As non-limiting examples, the waypoints include roads, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the trajectory generation component 236 can generate instructions for guiding the autonomous vehicle along at least a portion of a path from the first location to the second location. In at least one example, the trajectory generation component 236 can determine how to guide the autonomous vehicle from a first waypoint in the series of waypoints to a second waypoint in the series of waypoints. In some examples, the instructions can be a trajectory or a portion of a trajectory. In some examples, multiple trajectories can be generated substantially simultaneously (e.g., within technical tolerances) according to a receding horizon technique, and one of the multiple trajectories is selected for travel by the vehicle 202.

[0067] In some examples, the evaluation component 238 can include functionality to evaluate one or more candidate actions and / or candidate trajectories. For example, for each set of heatmaps including predicted probabilities associated with each action, the evaluation component 238 can determine one or more costs, amounts of risk, and / or rewards. In some examples, the evaluation component 238 can determine an amount of overlap between a candidate location of the vehicle 202 and predicted probabilities associated with an object in the environment. In some examples, the evaluation component 238 can determine a distance between a representation of the vehicle 202 (e.g., a bounding box or an extended bounding box) and a predicted probability and / or extended predicted probability associated with the object. In at least some examples, such an extension (to the bounding box as an input or to the predicted probability as an output) can be based on a safety factor to artificially increase the area around the likely location of the future object. In some examples, the evaluation component 238 can determine whether the trajectory of the vehicle 202 crosses through an area associated with a predicted probability (which may include extended predicted probability) associated with the vehicle 202. The evaluation component 238 can determine costs, risks, and / or rewards at individual time steps in the future and / or cumulatively for some or all of the time steps associated with the candidate actions. Thus, the evaluation component 238 can compare the costs, risks, and / or rewards of different candidate actions and can select an action for controlling the vehicle.

[0068] Examples of evaluation of candidate actions are described in connection with U.S. Patent Application No. 16 / 206,877, entitled "Probabilistic Risk Assessment for Trajectory Evaluation," filed November 30, 2018. U.S. Patent Application No. 16 / 206,877 is incorporated herein by reference in its entirety.

[0069] Additional costs to be considered by the evaluation component 238 may include, but are not limited to, position-based costs (e.g., based on the distance between the object and the predicted probability associated with the vehicle 202), speed costs (e.g., based on applying a speed while traversing an area associated with the predicted probability), acceleration costs (e.g., applying an acceleration limit to the entire candidate trajectory action), an expectation that the object will obey the rules of the road, and the like.

[0070] In some examples, the machine learning component 232 can provide the heatmap directly to the planning component 234, which can then select or determine an action for the vehicle 202 to take and / or generate a trajectory for the vehicle 202 to follow.

[0071] In some examples, some or all aspects of the components described herein may include any model, algorithm, and / or machine learning algorithm. For example, in some examples, the components in memory 218 (and memory 246 described below) may be implemented as neural networks.

[0072] As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer of a neural network can include another neural network, or can include any number of layers (whether convolutional or not). As can be understood in the context of this disclosure, a neural network can utilize machine learning, which can refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0073] Although described in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scarplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomy 3 (ID3), chi-squared automated interaction detection (CHAID), decision stump, conditional decision tree)), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average-one dependent estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-median, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backpropagation, The methods may include, but are not limited to, methods such as Hopfield networks, radial basis function networks (RBFNs), deep learning algorithms (e.g., deep Boltzmann machines (DBMs), deep brief networks (DBNs), convolutional neural networks (CNNs), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machines (GBMs), gradient boosting regression trees (GBRTs), random forests), support vector machines, supervised learning, unsupervised learning, and semi-supervised learning.

[0074] Additional examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

[0075] In at least one example, the sensor system 206 can include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), time of flight sensors, microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and the like. The sensor system 206 can include multiple instances of each of these or other types of sensors. For example, the lidar sensors can include individual lidar sensors located at corners, front, rear, sides, and / or top of the vehicle 202. As another example, the camera sensors can include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 202. The sensor system 206 can provide input to the vehicle computing device 204. Additionally or alternatively, the sensor system 206 can transmit sensor data over one or more networks 240 to one or more computing devices at a particular frequency in near real time after a predetermined period of time has elapsed.

[0076] The vehicle 202 may also include one or more emitters 208 for emitting light and / or sound, as described above. The example emitters 208 include interior audio and visual emitters for communicating with passengers of the vehicle 202. By way of example and not limitation, the interior emitters may include speakers, lights, symbols, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The example emitters 208 also include exterior emitters. By way of example and not limitation, the example exterior emitters may include lights for signaling a direction of travel or other indicators of the vehicle's actions (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with one or more pedestrians or other nearby vehicles, including acoustic beam steering technology.

[0077] The vehicle 202 may also include one or more communication connections 210 that enable communication between the vehicle 202 and one or more other local or remote computing devices. For example, the communication connections 210 may facilitate communication with other local computing devices on the vehicle 202 and / or the drive system 214. The communication connections 210 may also enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). The communication connections 210 may also enable the vehicle 202 to communicate with remotely operated computing devices or other remote services.

[0078] The communications connection 210 may include physical and / or logical interfaces for connecting the vehicle computing device 204 to another computing device or network, such as the network 240. For example, the communications connection 210 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices.

[0079] In at least one example, the vehicle 202 can include one or more drive systems 214. In some examples, the vehicle 202 can have a single drive system 214. In at least one example, when the vehicle 202 has multiple drive systems 214, individual drive systems 214 can be located at both ends of the vehicle 202 (e.g., front and rear, etc.). In at least one example, the drive system 214 can include one or more sensor systems for detecting conditions surrounding the drive system 214 and / or the vehicle 202. By way of example and not limitation, the sensor systems can include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive system, cameras or other imaging sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. for acoustically detecting objects surrounding the drive system. Some sensors, such as wheel encoders, can be specific to the drive system 214. In some cases, the sensor systems on the drive system 214 may overlap or complement corresponding systems on the vehicle 202 (e.g., the sensor system 206).

[0080] The drive system 214 can include many vehicle systems including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use in other vehicle systems, a steering system including a steering motor and a steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for braking force distribution to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the vehicle's exterior environment), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, and other electrical components). Additionally, the drive system 214 can include a drive system controller that can receive and pre-process data from the sensor systems and control the operation of various vehicle systems. In some examples, the drive system controller can include one or more processors and a memory communicatively coupled to the one or more processors. The memory can store one or more components that perform various functions of the drive system 214. In addition, drive system 214 also includes one or more communication connections that enable each drive system to communicate with one or more other local or remote computing devices.

[0081] In at least one example, the direct connection 212 can provide a physical interface for coupling one or more drive systems 214 with the body of the vehicle 202. For example, the direct connection 212 can enable the transfer of energy, fluid, air, data, etc. between the drive systems 214 and the vehicle. In some examples, the direct connection 212 can also removably secure the drive systems 214 to the body of the vehicle 202.

[0082] In some examples, the vehicle 202 can transmit sensor data over the network 240 to one or more computing devices 242. In some examples, the vehicle 202 can transmit raw sensor data to the computing device 242. In other examples, the vehicle 202 can transmit processed sensor data and / or representations of the sensor data to the computing device 242. In some examples, the vehicle 202 can transmit sensor data at a particular frequency to the computing device 242, such as after a predetermined period of time, in near real time, etc. In some examples, the vehicle 202 can transmit the sensor data (raw or processed) to the computing device 242 as one or more log files.

[0083] The computing device 242 may include a processor 244 and a memory 246 that stores a training component 248 .

[0084] In some examples, the training component 248 can include functionality to train the machine learning model to output one or more heatmaps including the predicted probabilities. For example, the training component 248 can receive a set of images representing a top-down view of an environment. In some cases, the set of images can represent objects traversing through the environment over a period of 5 seconds, 7 seconds, 10 seconds, 20 seconds, etc. At least a portion of the set of images can be used as input for training the machine learning model, and at least a portion of the set of images can be used as ground truth information for training the machine learning model. As a non-limiting example, a first set of a sequence of images (e.g., 3, 4, 5, or more) can be input to the machine learning model. In some examples, the intent, goals, sub-goals, etc. of the vehicle 202 can also be input for training (e.g., as may be encoded as channels of a multi-channel image). Then, a second set of images (or trajectory information associated therewith, e.g., by extracting position and / or velocity from the images) of the sequence of images immediately preceding the first set can be used as ground truth for training the model. Thus, by providing images of the recorded trajectory of an object traversing an environment, the prediction component 228 can be trained to output one or more heat maps containing prediction probabilities, as described herein.

[0085] In some examples, the training component 248 can include training data generated by a simulator. For example, the simulated training data can represent instances in which the vehicle collided with or nearly collided with objects in the environment to provide additional training examples.

[0086] Additional details of the training component 248 are described below in connection with FIG. 2B, as well as throughout this disclosure.

[0087] Additional examples of data for training are described in connection with FIG. 3, as well as throughout this disclosure.

[0088] The processor 216 of the vehicle 202 and the processor 244 of the computing device 242 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processors 216 and 244 may comprise one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into registers or other electronic data that can be stored in memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement the encoded instructions.

[0089] The memories 218 and 246 are examples of non-transitory computer-readable media. The memories 218 and 246 can store an operating system and one or more software applications, instructions, programs, and / or data that implement the methods and functions attributed to the various systems described herein. In various implementations, the memory can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, and those shown in the accompanying drawings are merely examples relevant to the description herein.

[0090] In some examples, the memories 218 and 246 may include at least a working memory and a storage memory. For example, the working memory may be a limited capacity, high-speed memory (e.g., cache memory) used to store data to be manipulated by the processors 216 and 244. In some examples, the memories 218 and 246 may include a storage memory, which may be a relatively large capacity, slower memory used for long-term storage of data. In some cases, the processors 216 and 244 may not be able to directly manipulate data stored in the storage memory and may need to load the data into the working memory in order to perform operations based on the data, as described herein.

[0091] 2 is shown as a distributed system, it should be noted that in alternative examples, components of the vehicle 202 may be associated with the computing device 242 and / or components of the computing device 242 may be associated with the vehicle 202. That is, the vehicle 202 may perform one or more of the functions associated with the computing device 242, and vice versa.

[0092] FIG. 2B shows a block diagram 250 of an example training component for training a machine learning model that implements the techniques described herein.

[0093] In some examples, training data 258 may include data captured by a vehicle as it traverses through an environment. Such training data may include action data 262 that may represent actions to be performed by an autonomous vehicle if such autonomous vehicle captured training data 258. In some examples, input 252 may be based at least in part on action data 262 and training data 258.

[0094] An input 252 including one or more multi-channel images can be input to the machine learning component 232. In some examples, the machine learning component 232 can include a ResNet architecture. In some examples, the machine learning component 232 can include a ResNet architecture as a base backbone. In some examples, action data can be input along with the input 252, and the action data can be represented as action data 262.

[0095] In some examples, the output feature maps can be passed through a 1×1 convolution to obtain the final channel dimensions, which can be bilinearly upsampled (e.g., via upsampling component 254) to match the input resolution and represented as output 256 (e.g., heatmaps and / or predicted probabilities).

[0096] In some examples, the output logits from the machine learning component 232 can be compared to training data 258 (e.g., ground truth representing an occupancy map) using a sigmoid cross-entropy loss.

[0097] The cross-entropy loss and / or probability mass component 260 may include a function that compares the output logits to the training data 258. The resulting loss matrix is ​​then calculated using the function

[0098]

number

[0099] The occupied and unoccupied pixels in the training data 258 can be balanced according to i is the loss matrix obtained from the sigmoid cross entropy loss at the i-th future step, and N is the number of future steps to be predicted. occupied and 1 unoccupiedis the indicator matrix representing the occupancy and non-occupancy map of future objects, and the term n occupied and n unoccupied represents the number of future occupied and unoccupied cells. Balancing occupancy ensures that an equal (or nearly equal) number of occupied and unoccupied gradients are passed through backpropagation, and that prediction component 228 is not biased towards predicting all cells as occupied or all cells as unoccupied.

[0100] The cost of dense segmentation specified in equation (1) above may not preserve the total number of entities in the scene. In some examples, the cross-entropy loss and / or probability mass component 260 may be calculated by multiplying the total probability mass predicted by the prediction component 228 in the scene (e.g., the sum of the probabilities associated with the heatmaps) by

[0101]

number

[0102] We can include an additional cost to minimize the difference between the actual number of occupied cells as i represents the occupancy map at a future step i. By adding a probability mass component, the prediction component 228 can distribute the total probability mass among the available cells without creating additional probability mass. For example, if an autonomous vehicle has two possible modes, then the network distributes the probability of occupancy between the two modes and may not add any additional occupancy probability.

[0103] Additionally, constraining or otherwise managing the mass of a scene limits the availability of the network for adding objects to the environment, which may simplify processing.

[0104] FIG. 3 illustrates an example diagram 300 of inputs to a predictive system and training data for training the predictive system, according to an example of this disclosure.

[0105] For example, the multi-channel images 302 represent inputs to the prediction component 228 and represent past objects. For example, the time associated with each image is shown in the lower right corner (e.g., t -8 , t -7 , t -6 , t -5 , t -4 , t -3 , t -2 , t -1 , and t0). As mentioned above, each frame may further consist of several channels.

[0106] For example, each channel may represent one or more of, but is not limited to, a bounding box (e.g., a bounding box represents a vehicle 304 or object 306, and the color or channel of the bounding box represents the classification of the object associated with the bounding box); the speed of the bounding box in the x and y directions (where the speed may be represented as a number encoded into an area representing the area of ​​another channel (e.g., channels 136 and 142) corresponding to the bounding box, with the x and y speeds being represented in different channels); the acceleration of the object in the x and y directions (where the acceleration may be represented as a vector in one channel or multiple channels, with the x and y accelerations being represented in different channels); the blinker status associated with the bounding box (left turn, right turn, braking, reversing, hazards, no lights, etc.); road network information (e.g., semantic labels indicating lanes, reference lines (e.g., reference lines 308), crosswalks, stop signs, intersections, traffic lights, etc.); the state of the traffic lights (e.g., red light, yellow light, green light, etc.); In some examples, multiple channels can be input into the predictive system to generate at least one heat map.

[0107] Further, the action data may be represented as a current lane 310 and a target lane 312. In some examples, the action data may be -7 to t0 can be encoded into one or more images that represent the environment.

[0108] The top-down features are represented as image blobs by stacking some or all of the above feature images. In some examples, the resolution of the multi-channel image 302 can be set to 15 cm per pixel based on the resolution of the sensor resolution. In some examples, a single frame of the multi-channel image 302 can correspond to an area of ​​size 50 meters by 50 meters, although any area can be selected. In some examples, the multi-channel image 302 can be rotated and translated such that the vehicle 304 (e.g., an autonomous vehicle) faces left to right and is located approximately in the center of the multi-channel image 302 with a configurable horizontal offset. In some examples, the environment represented by the multi-channel image can represent a junction with no signal for prediction, in which case the signal status of each lane may not be included in the top-down representation.

[0109] The training data 314 represents the ground truth of all objects and vehicles in the environment. The training data 314 can include any number of future frames. Figure 3 shows training data at times t1, t2, t3, t4, t5, t6, t7, and t8. The training data 314 can correspond to the training data 258 of Figure 2, for example, for purposes of training the machine learning component 232.

[0110] FIG. 4 is an illustration 400 of first and second predicted probabilities for a scenario including two double parked vehicles, where the second predicted probability is based in part on action data, according to an example of this disclosure.

[0111] Predicted probabilities 402 refer to a series of eight frames (labeled t1 through t8) that illustrate the output of prediction component 228, whereby predicted probabilities 402 are not based in part on action data. In the first frame of predicted probabilities 402 (denoted as frame t1), the scenario depicts a vehicle 406 and objects 408 and 410. Arrow 412 indicates the direction of traffic associated with the environment. Similar representations of traffic direction are used throughout this disclosure.

[0112] Objects 408 and 410 in the scenario represent two double parked vehicles. A predicted probability 414 is output by the prediction component 228 representing a predicted probability associated with vehicle 406 at time t1. Predicted probabilities 416 and 418 are associated with objects 408 and 410, respectively. Predicted probability 402 is shown evolving over time as determined by the prediction component 228.

[0113] Predicted probabilities 404 refer to a series of eight frames (labeled t1 through t8) that represent the output of prediction component 228, whereby the predicted probabilities are based in part on action data 420 (shown as grey shaded lanes). Predicted probability 422 is associated with vehicle 406, while predicted probabilities 424 and 426 are associated with objects 408 and 410, respectively. Predicted probabilities 404 are shown to evolve over time, as determined by prediction component 228.

[0114] FIG. 5 is an illustration 500 of first and second predicted probabilities for a scenario involving double-parked vehicles, where the second predicted probability is based in part on action data, according to an example of this disclosure.

[0115] The predicted probabilities 502 refer to a series of eight frames (labeled t1 through t8) that illustrate the output of the prediction component 228, whereby the predicted probabilities 502 are not based in part on action data. In the first frame of the predicted probabilities 502 (denoted as frame t1), the scenario represents a vehicle 506 and objects 508, 510, and 512.

[0116] Object 508 represents double parked vehicles. Objects 510 and 512 represent oncoming traffic relative to vehicle 506.

[0117] Notably, predicted probabilities 502 represent non-continuous predicted probabilities associated with vehicle 506. For example, at time t, predicted probabilities 514 and 516 are associated with vehicle 506. At time t, predicted probabilities 518 and 520 are associated with vehicle 506. And, at time t, predicted probabilities 522 and 524 are associated with vehicle 506. Predicted probabilities 502 are shown to evolve over time, as determined by prediction component 228.

[0118] The predicted probabilities 504 refer to a series of eight frames (labeled t1 through t8) that represent the output of the prediction component 228, whereby the predicted probabilities are based in part on the action data 526 (shown as gray shaded lanes).

[0119] Notably, the predicted probability 504 associated with the vehicle 506 does not show non-adjacent areas, but instead represents the vehicle 506 traversing around the object 508. The predicted probability 504 is shown to evolve over time, as determined by the prediction component 228.

[0120] FIG. 6 is an illustration 600 of first and second predicted probabilities for a scenario involving an object turning in front of an autonomous vehicle, where the second predicted probability is based in part on action data, according to an example of this disclosure.

[0121] The predicted probabilities 602 refer to a series of eight frames (labeled t1 through t8) that represent the output of the prediction component 228, whereby the predicted probabilities 602 are not based in part on action data. In the first frame of the predicted probabilities 602 (denoted as frame t1), the scenario depicts a vehicle 606 and an object 608.

[0122] Object 608 represents a vehicle turning into the lane in front of vehicle 606 .

[0123] Notably, the predicted probability associated with region 610 of frame t2 (and corresponding regions in frames t3-t8) is associated with object 608, while predicted probability 612 is associated with vehicle 606. Predicted probability 602 is shown evolving over time, as determined by prediction component 228.

[0124] Prediction probability 604 refers to a series of eight frames (labeled t1 through t8) that represent the output of prediction component 228, whereby the prediction probability is based in part on action data 614 and 616 (shown as gray shaded lanes). Action data 614 can correspond to a current (or exit) lane, while action data 616 can correspond to a target (or entrance) lane.

[0125] The predicted probability associated with region 618 of frame t2 (and corresponding regions of frames t3-t8) is associated with object 608, while predicted probability 620 is associated with vehicle 606. Predicted probability 604 is shown evolving over time, as determined by prediction component 228.

[0126] FIG. 7 is an illustration 700 of first and second predicted probabilities for a scenario including an autonomous vehicle traversing a junction, where the second predicted probability is based in part on action data, according to an example of this disclosure.

[0127] The predicted probabilities 702 refer to a series of eight frames (labeled t1 through t8) that represent the output of the prediction component 228, whereby the predicted probabilities 702 are not based in part on action data. In the first frame of the predicted probabilities 702 (denoted as frame t1), the scenario represents a vehicle 706 and an object 708.

[0128] The scenario in Figure 7 is a vehicle 706 7 represents a vehicle 706 crossing a junction in front of an object 708 representing a vehicle turning into the lane in front of it.

[0129] Of note, predicted probabilities 710, 712, 714, and 716 (e.g., within their respective dashed lines) represent predicted probabilities associated with vehicle 706 and illustrate a multi-modal output of predicted probabilities. Predicted probability 702 is shown evolving over time as determined by prediction component 228.

[0130] Prediction probability 704 refers to a series of eight frames (labeled t1 through t8) that represent the output of prediction component 228, whereby the prediction probability is based in part on action data 718 and 720 (shown as gray shaded lanes). Action data 718 can correspond to a current (or exit) lane, while action data 720 can correspond to a target (or entrance) lane.

[0131] Predicted probabilities 722, 724, 726, and 728 are associated with object 708, while the respective remaining predicted probabilities are associated with vehicle 706. Thus, predicted probability 704 represents a crisper prediction of the location associated with vehicle 706, which can lead to more accurate predictions. Predicted probabilities 704 are shown to evolve over time, as determined by prediction component 228.

[0132] FIG. 8 is an illustration 800 of first and second predicted probabilities for a scenario involving an autonomous vehicle entering a blocked junction, where the second predicted probability is based in part on action data, in accordance with an example of this disclosure.

[0133] The predicted probabilities 802 refer to a series of eight frames (labeled t1 through t8) that illustrate the output of the prediction component 228, whereby the predicted probabilities 802 are not based in part on action data. In the first frame of the predicted probabilities 802 (denoted as frame t1), the scenario represents a vehicle 806 and objects 808, 810, and 812.

[0134] The scenario in FIG. 8 involves a right turn and objects 808 and 810 It represents a vehicle 806 at a junction intending to continue to.

[0135] Of note, predicted probabilities 814, 816, 818, and 820 (e.g., within their respective dashed lines) represent predicted probabilities associated with vehicle 806 and illustrate a multi-modal output of predicted probabilities. Predicted probability 802 is shown evolving over time as determined by prediction component 228.

[0136] Prediction probability 804 refers to a series of eight frames (labeled t1 through t8) that represent the output of prediction component 228, whereby the prediction probability is based in part on action data 822 and 824 (shown as gray shaded lanes). Action data 822 can correspond to a current (or exit) lane, while action data 824 can correspond to a target (or entrance) lane.

[0137] Predicted probabilities 826, 828, 830, and 832 are associated with vehicle 806, while the remaining predicted probabilities are associated with objects 808, 810, or 812, respectively. Thus, predicted probabilities 804 represent crisper predictions of the location associated with vehicle 806, which can lead to more accurate predictions. Predicted probabilities 804 are shown to evolve over time, as determined by prediction component 228.

[0138] FIG. 9 is an illustration 900 of first and second predicted probabilities for a scenario including an autonomous vehicle crossing another junction, where the second predicted probability is based in part on action data, according to an example of this disclosure.

[0139] The predicted probabilities 902 refer to a series of eight frames (labeled t1 through t8) that illustrate the output of the prediction component 228, whereby the predicted probabilities 902 are not based in part on action data. In the first frame of the predicted probabilities 902 (denoted as frame t1), the scenario represents a vehicle 806 and objects 808, 810, and 812.

[0140] The scenario in Figure 9 represents a vehicle 906 at a junction intending to cross the junction. In this example, the target lane of the vehicle 906 is not blocked, so the vehicle 906 proceeds towards the goal. In the unconditional case (e.g., represented by prediction probability 902), the network proposes that the vehicle 906 can go right, left, or straight, but does not know which one the vehicle 906 will do.

[0141] Of note, predicted probabilities 914, 916, 918, and 920 (e.g., within their respective dashed lines) represent predicted probabilities associated with vehicle 906 and illustrate the variance modal output of predicted probabilities. Predicted probability 902 is shown evolving over time as determined by prediction component 228.

[0142] Prediction probability 904 refers to a series of eight frames (labeled t1 through t8) that represent the output of prediction component 228, whereby the prediction probability is based in part on action data 922 and 924 (shown as gray shaded lanes). Action data 922 can correspond to a current (or exit) lane, while action data 924 can correspond to a target (or entrance) lane.

[0143] Prediction probabilities 926, 928, 930, and 932 are associated with vehicle 906, while the remaining prediction probabilities are associated with objects 908, 910, or 912, respectively. Thus, prediction probability 904 represents a crisper prediction of the location associated with vehicle 906, which may lead to more accurate predictions. Prediction probability 904 is 、 It is shown to evolve over time.

[0144] 1, 2B, and 10 illustrate exemplary processes according to embodiments of the present disclosure. These processes are illustrated as logical flow graphs, with each operation representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable recording media that, when executed on one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to perform the process.

[0145] 10 illustrates an example process 1000 for receiving sensor data and action data, generating a multi-channel image, evaluating prediction probabilities, and controlling a vehicle based on the prediction probabilities, according to an example of this disclosure. For example, some or all of the process 1000 may be performed by one or more components in FIG. 2 as described herein. For example, some or all of the process 1000 may be performed by the vehicle computing device 204 and / or the computing device 242.

[0146] At operation 1002, the process may include receiving object data of the environment, the object data representing objects. In some examples, the object data may represent data output by a perception system of the autonomous vehicle. For example, the object data may include, but is not limited to, bounding box information, speed / acceleration information, classification, and the like. In some examples, the operation 1002 may include capturing sensor data of the environment using sensors of the autonomous vehicle. For example, the sensor data may include data captured by one or more of a lidar sensor, a radar sensor, an image sensor, a time of flight sensor, a sonar sensor, and the like. In some examples, the operation 1002 may include capturing sensor data using multiple sensors and fusing or combining the sensor data into a detailed and useful representation of the environment. In at least some examples, additional data may be received (e.g., map data) that may encode additional data about the environment (e.g., crosswalk locations, lane boundaries, and the like).

[0147] At operation 1004, the process may include receiving action data associated with candidate actions for the vehicle to perform within the environment. In some examples, the action data may include a current lane, a target lane, and / or speed or acceleration information, such as a speed profile or acceleration profile, associated with each action. Examples of candidate actions may include, but are not limited to, an in-lane action, a left turn action, a right turn action, a lane change action, a stop action, a yield action, a merge action, etc.

[0148] At operation 1006, the process may include generating a multi-channel image representing a top-down view of the environment based at least in part on the sensor data and the action data. Examples of such multi-channel images are described throughout this disclosure. For example, the multi-channel image may represent objects as bounding boxes, velocities of the objects, positions of the objects within the environment, etc. In some examples, the multi-channel image may include acceleration information (instead of or in addition to velocity information).

[0149] In some examples, the action data may be represented as a current lane and a target lane, speed information, or acceleration information encoded into channels of a multi-channel image.

[0150] Additional information that can be included in multi-channel images is the classification of the object associated with the bounding box, the velocity of the bounding box in the x-direction, y-direction, and / or z-direction (velocity can be represented as a number or vector in one channel or multiple channels, and x-velocity, y-velocity, and / or z-velocity can be represented as a vector). speedmay include, but are not limited to, the following: x, y, and / or z acceleration (wherein x, y, and z are represented in different channels), the acceleration of the object in the x, y, and / or z directions (wherein the acceleration may be represented as a number or vector in one channel or multiple channels, with the x acceleration, y acceleration, and / or z acceleration being represented in different channels), the pose of the vehicle (e.g., x, y, z, roll, pitch, yaw), the rate of change of the pose, the blinker status associated with the bounding box (e.g., turn left, turn right, brake, reverse, hazard, no lights, etc.), road network information (e.g., semantic labels indicating lanes, crosswalks, stop signs, intersections, traffic lights, etc.), the state of the traffic lights (e.g., red light, yellow light, green light, etc.), etc. In such multi-channel images, semantic information about the objects and the environment may be encoded in one or more channels, as described in detail herein.

[0151] As described herein, operation 1006 may include generating a plurality of images representing the motion of objects in an environment over time.

[0152] At operation 1008, the process may include determining that a candidate action has been considered. For example, operation 1008 may include determining whether a multi-channel image has been generated for each candidate action of the plurality of candidate actions. If not all (or not substantially all) of the candidate actions have been considered (e.g., "no" at operation 1008), the process may return to operation 1006 to generate additional multi-channel images. If all candidate actions have been considered (e.g., "yes" at operation 1008), the process may proceed to operation 1010. Of course, in some examples, operations may be performed in parallel, depending on the implementation.

[0153] At operation 1010, the process may include inputting the multi-channel image into a machine learning model. Such a machine learning model may be trained to generate a plurality of predicted probabilities (e.g., as a heat map). Additional details of the machine learning model, heat maps, and predicted probabilities are described throughout this disclosure.

[0154] At operation 1012, the process may include receiving, from the machine learning model, a predicted probability associated with the object and / or vehicle. Additional details of the machine learning model, the heatmap, and the predicted probabilities are described throughout this disclosure.

[0155] At operation 1014, the process may include evaluating the candidate actions based at least in part on the predicted probabilities. As described herein, operation 1014 may include determining one or more costs, risks, and / or rewards associated with the candidate actions. For example, the candidate actions may be evaluated to determine, for a trajectory associated with the candidate actions, an amount of overlap between a region associated with the autonomous vehicle along the candidate trajectory and one or more predicted probabilities. In some examples, the amount of overlap between a region associated with the autonomous vehicle and a predicted probability of the object may represent an amount of risk associated with such a candidate action. In some examples, the amount of overlap between a region associated with the autonomous vehicle and a predicted probability of the autonomous vehicle may represent an extent to which the autonomous vehicle will remain within the predicted probability. In some examples, the candidate actions may be evaluated based on one or more distances between an expected position associated with the autonomous vehicle and a predicted probability associated with the object.

[0156] Operation 1014 may further include selecting or otherwise determining a candidate action based at least in part on the evaluation. For example, costs associated with the candidate actions may be compared, and the candidate action associated with the lowest cost may be selected to control the autonomous vehicle.

[0157] At operation 1016, the process may include controlling the vehicle based at least in part on the predicted probability. In some examples, operation 1016 may include determining a trajectory for the autonomous vehicle. In some examples, the trajectory may be based on candidate actions selected to be performed by the vehicle, as described herein, and may represent a safer and / or smoother trajectory as compared to a trajectory generated without evaluating the predicted probability. In some examples, operation 1016 may include controlling one or more of steering, braking, and / or acceleration of the autonomous vehicle to traverse an environment.

[0158] Thus, the operations described herein improve predictions and provide a way for planning systems to explore possible futures.

[0159] [Example of invention content] A: A method for implementing the present invention includes: receiving sensor data of an environment captured by a sensor of an autonomous vehicle; receiving action data associated with candidate actions to be performed within the environment by the autonomous vehicle, the candidate actions including one or more of an in-lane action, a turn action, or a lane change action, the action data indicating the candidate actions; and generating a multi-channel image representing a top-down view of the environment based at least in part on the sensor data and the action data, the multi-channel image including bounding boxes associated with vehicles in the environment proximate to the autonomous vehicle, a front view of the environment, and a back view of the environment. and one or more computer-readable media storing the instructions executable by the one or more processors to cause the one or more processors to perform operations including: inputting the multi-channel images into a machine learning model trained to generate a heat map including predicted probabilities of possible positions associated with the vehicle, representing one or more kinematic or semantic information associated with the vehicle and the candidate actions associated with the autonomous vehicle, determining a cost associated with the candidate actions based at least in part on the heat map, and determining a trajectory for the autonomous vehicle to traverse the environment based at least in part on the cost associated with the candidate actions.

[0160] B: Generating the multi-channel image representing the candidate action associated with the autonomous vehicle includes generating a channel of the multi-channel image representing a target lane in the environment associated with the candidate action, the target lane representing a lane that the autonomous vehicle is intended to occupy. 1. The system described in paragraph A.

[0161] C: The system described in paragraph A or B, wherein generating the multi-channel image representing the candidate action associated with the autonomous vehicle includes generating a channel of the multi-channel image representing a target speed or a target acceleration of the autonomous vehicle associated with the candidate action.

[0162] D: The system described in any of paragraphs A to C, wherein the predicted probability is represented as an occupancy grid associated with a future time, and a cell of the occupancy grid indicates the probability that the vehicle will be present in the area associated with the cell at a future time.

[0163] E: the candidate action is a first candidate action, the heat map is a first heat map, the predicted probability is a first predicted probability, and the cost is a first cost, and the operations further include determining a second heat map including a second predicted probability associated with the vehicle based at least in part on the sensor data and a second candidate action, determining a second cost based at least in part on the second heat map, selecting a selected action from the first candidate action or the second candidate action based at least in part on the first cost and the second cost, and determining the trajectory based at least in part on the selected action. A system according to any of paragraphs A to D.

[0164] F: A method comprising: receiving object data representing objects in an environment; receiving action data associated with candidate actions for a vehicle to perform within the environment; generating a multi-channel image representing a top-down view of the environment based at least in part on the object data and the action data, the multi-channel image representing the objects, motion information associated with the objects, and the candidate actions associated with the vehicle; inputting the multi-channel image into a machine learning model; receiving from the machine learning model a predicted probability associated with the objects; and controlling the vehicle to traverse the environment based at least in part on the predicted probability.

[0165] G: The method of paragraph F, wherein the prediction probability is a first prediction probability, and the method further includes receiving a second prediction probability associated with the vehicle from the machine learning model.

[0166] H: the candidate action includes at least one of an in-lane action, a lane change action, or a turn action, and generating the multi-channel image representing the candidate action associated with the vehicle includes generating a channel of the multi-channel image representing a target lane associated with the candidate action. The method described in paragraph F or G.

[0167] I: The method of paragraph H, wherein the channel is a first channel, and generating the multi-channel image representing the candidate action associated with the vehicle further includes a second channel of the multi-channel image representing a target velocity or a target acceleration of the vehicle associated with the candidate action.

[0168] J: The method of any of paragraphs F to I, wherein the multi-channel image further includes one or more channels containing additional object information for additional objects in the environment, and the multi-channel image is one of a plurality of multi-channel images associated with one or more previous times before the current time.

[0169] K: The method of paragraph J, further comprising receiving from the machine learning model a plurality of predicted probabilities representing the environment at one or more future times after the current time.

[0170] L: The method of any of paragraphs F-K, wherein the predicted probability is a first predicted probability associated with a first time after the current time, the method further including receiving a second predicted probability associated with the object from the machine learning model, the second predicted probability being associated with a second time after the first time, and an amount of first probability associated with the first predicted probability within a threshold amount of an amount of second probability associated with the second predicted probability.

[0171] M: The method of any of paragraphs F-L, wherein the object data is based at least in part on at least one of image data, lidar data, radar data, or time-of-flight data.

[0172] N: The method of any of paragraphs F-M, wherein the machine learning model includes a convolutional neural network.

[0173] O. The method of any of paragraphs F-N, wherein the action data is first action data, the candidate action is a first candidate action, the multi-channel image is a first multi-channel image, the top-down view of the environment is a first top-down view of the environment, and the predicted probability is a first predicted probability associated with the first candidate action, the method further including receiving second action data associated with a second candidate action for the vehicle to perform within the environment, generating a second multi-channel image representing a second top-down view of the environment based at least in part on the object data and the second action data, inputting the second multi-channel image into the machine learning model, and receiving from the machine learning model a second predicted probability associated with the object.

[0174] P. The method of paragraph O, further comprising: determining a first cost associated with the first predicted probability; determining a second cost based at least in part on the second predicted probability; selecting one of the first candidate action or the second candidate action as a selected action and based at least in part on the first cost and the second cost; and controlling the vehicle to traverse the environment based at least in part on the selected action.

[0175] Q: One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations including receiving object data representing objects within an environment; receiving action data associated with candidate actions for a vehicle to perform within the environment, the candidate actions including one or more of an in-lane action, a turn action, or a lane change action, and the action data indicating the candidate actions; generating a multi-channel image representing the object, motion information associated with the object, and the candidate actions associated with the vehicle based at least in part on the object data and the action data; inputting the multi-channel image into a machine learning model; receiving from the machine learning model a predicted probability associated with the object; and controlling the vehicle to traverse the environment based at least in part on the predicted probability and the candidate actions.

[0176] R: The non-transitory computer-readable medium of paragraph Q, wherein generating the multi-channel image representing the candidate action associated with the vehicle includes generating a channel of the multi-channel image representing a target lane associated with the candidate action, the target lane indicating a lane that the vehicle is intended to occupy.

[0177] S: The non-transitory computer-readable medium of paragraph Q or S, wherein the channel is a first channel, and generating the multi-channel image representing the candidate action associated with the vehicle further includes generating a second channel of the multi-channel image representing a target velocity or a target acceleration of the vehicle associated with the candidate action.

[0178] T: The non-transitory computer-readable medium of any of paragraphs Q-T, further comprising: receiving second action data, wherein the action data is first action data, the candidate action is a first candidate action, the multi-channel image is a first multi-channel image, the predicted probability is a first predicted probability associated with the first candidate action, and the operation is associated with a second candidate action for the vehicle to perform in the environment; generating a second multi-channel image representing the object, the motion information, and the second candidate action based at least in part on the object data and the second action data; inputting the second multi-channel image into the machine learning model; and receiving from the machine learning model a second predicted probability associated with the object.

[0179] Although the above example subject matter is described with respect to one particular implementation, it should be understood in the context of this document that the example subject matter can also be implemented via a method, device, system, computer-readable medium, and / or other implementations. Furthermore, any of the examples A-T can be implemented alone or in combination with one or more of the other examples A-T.

[0180] [Conclusion] One or more examples of the technology described herein have been described; however, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0181] In the illustrative description, reference is made to the accompanying drawings, which form a part hereof, showing by way of example specific examples of the claimed subject matter. It is to be understood that other examples may be used and modifications or substitutions, such as structural changes, may be made. Such examples, modifications or substitutions do not necessarily depart from the intended scope of the claimed subject matter. Although the steps herein may be presented in a particular order, in some cases the order may be changed, such as providing certain inputs at different times or in a different order, without changing the functionality of the described systems and methods. The disclosed procedures may also be performed in different orders. Furthermore, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to reordering, calculations may also be decomposed into sub-calculations that produce the same results.

Claims

1. receiving object data representing objects in an environment, the environment including an intersection that a vehicle is about to cross; receiving action data associated with candidate actions for the vehicle to perform within the environment; generating a multi-channel image representing a top-down view of the environment based at least in part on the object data and the action data, the multi-channel image including an image representing the object, an image representing motion information associated with the object, an image representing the candidate actions associated with the vehicle, and one or more channel images representing the environment, the multi-channel image being one of a plurality of multi-channel images associated with one or more previous times prior to a current time; inputting the multi-channel images into a machine learning model; receiving a predicted probability associated with the object from the machine learning model, the predicted probability representing at least one possible location of the object; and determining a trajectory of the vehicle based at least in part on the predicted probabilities and the candidate actions; and controlling the vehicle to traverse the intersection based at least in part on the predicted probability; the machine learning model is trained to output a plurality of prediction probabilities, including the prediction probabilities associated with the object and / or prediction probabilities associated with possible positions of the vehicle; method.

2. The predicted probability is a first predicted probability, and the method comprises: receiving a second prediction probability associated with the vehicle from the machine learning model that is different from the first prediction probability; The method of claim 1 , wherein determining the vehicle's trajectory is based at least in part on the first prediction probability, the second prediction probability, and the candidate actions.

3. The candidate actions include: Action in the lane lane change action, or Includes at least one turn action, Generating the images representative of the candidate actions associated with the vehicle includes: a first channel image representing a target lane associated with the candidate action; or generating at least one of a second channel image representing a target velocity or a target acceleration of the vehicle associated with the candidate action; The method according to claim 1 or claim 2.

4. 2. The method of claim 1, further comprising receiving, from the machine learning model, the predicted probabilities associated with the objects in the environment and / or the predicted probabilities associated with possible positions of the vehicle at one or more future times after the current time.

5. the predicted probability associated with the object is a first predicted probability associated with a first time after a current time; The method comprises: receiving a second predicted probability from the machine learning model that is different from the first predicted probability associated with the object, the second predicted probability being associated with a second time after the first time; a first amount of probability associated with the first predicted probability is within a threshold amount of a second amount of probability associated with the second predicted probability.

5. The method according to any one of claims 1 to 4.

6. the object data is based at least in part on at least one of image data, lidar data, radar data, or time-of-flight data; The machine learning model includes a convolutional neural network.

6. The method according to any one of claims 1 to 5.

7. the predicted probabilities associated with the objects are represented as an occupancy grid associated with future times; a cell of the occupancy grid indicating a probability that the vehicle will be within a region associated with the cell at the future time; The method of claim 1.

8. the action data is a first action data, the candidate action is a first candidate action, the multi-channel image is a first multi-channel image; the top-down view of the environment is a first top-down view of the environment; the predicted probability is a first predicted probability associated with the first candidate action; the trajectory is a first trajectory, The method receiving second action data different from the first action data associated with a second candidate action different from the first candidate action for the vehicle to perform within the environment; generating a second multi-channel image different from the first multi-channel image representing a second top-down view of the environment different from the first top-down view of the environment based at least in part on the object data and the second action data; inputting the second multi-channel image into the machine learning model; receiving a second predicted probability from the machine learning model that is different from the first predicted probability associated with the object, the second predicted probability representing at least one possible location of the object; and determining a second trajectory for the vehicle that differs from the first trajectory based at least in part on the second predicted probability and the second candidate action; controlling the vehicle to traverse the intersection based at least in part on the second trajectory; and The method of claim 1 , further comprising:

9. evaluating the first candidate action and the second candidate action to determine a first cost and a second cost; selecting one of the first candidate action or the second candidate action as a selected action and based at least in part on comparing the first cost and the second cost; determining a third trajectory for the vehicle different from the first trajectory and the second trajectory based at least in part on the selected candidate action; controlling the vehicle to traverse the intersection based at least in part on the third trajectory; and The method of claim 8 further comprising:

10. A computer program comprising coded instructions which, when executed on a computer, performs the method of any one of claims 1 to 9.

11. one or more processors; When the command is executed, the system receiving object data representing objects in an environment, the environment including an intersection that a vehicle is about to cross; receiving action data associated with candidate actions for the vehicle to perform within the environment, the candidate actions including one or more of a in-lane action, a turn action, or a lane change action, the action data being indicative of the candidate actions; generating a multi-channel image based at least in part on the object data and the action data, the multi-channel image including an image representing the object, an image representing motion information associated with the object, an image representing the candidate actions associated with the vehicle, and one or more channel images representing the environment, the multi-channel image being one of a plurality of multi-channel images associated with one or more previous times prior to a current time; inputting the multi-channel images into a machine learning model; receiving a predicted probability associated with the object from the machine learning model, the predicted probability representing at least one possible location of the object; and determining a trajectory of the vehicle based at least in part on the predicted probabilities and the candidate actions; controlling the vehicle to traverse the intersection based at least in part on the predicted probabilities and the candidate actions; one or more non-transitory computer-readable media storing instructions executable by the one or more processors to cause the one or more processors to perform operations including: Including, The machine learning model is trained to output the predicted probability associated with the object. system.

12. 12. The system of claim 11, wherein generating the image representing the candidate action associated with the vehicle includes generating a channel image representing a target lane associated with the candidate action, the target lane indicating a lane that the vehicle is intended to occupy.

13. 13. The system of claim 12, wherein the channel image is a first channel image, and generating the image representing the candidate action associated with the vehicle further comprises generating a second channel image representing a target velocity or a target acceleration of the vehicle associated with the candidate action.

14. the action data is a first action data, the candidate action is a first candidate action, the multi-channel image is a first multi-channel image; the image representing the object is a first image representing the object; the image representing motion information associated with the object is a first image representing the motion information; the image representing the candidate action associated with the vehicle is a first image representing the candidate action; the predicted probability is a first predicted probability associated with the first candidate action; the trajectory is a first trajectory, The operation is receiving second action data different from the first action data associated with a second candidate action different from the first candidate action for the vehicle to perform within the environment; generating a second multi-channel image based at least in part on the object data and the second action data, the second multi-channel image including a second image representing the object, a second image representing the motion information, and a second image representing the second candidate action; inputting the second multi-channel image into the machine learning model; receiving a second predicted probability from the machine learning model that is different from the first predicted probability associated with the object, the second predicted probability representing at least one possible location of the object; and determining a second trajectory for the vehicle that differs from the first trajectory based at least in part on the second predicted probability and the second candidate action; controlling the vehicle to traverse the intersection based at least in part on the second trajectory; and The system of claim 12 or claim 13, further comprising:

Images