Trajectory prediction for top-down scenes

Abstract

A method for determining predicted trajectories based on a top-down representation of the environment is described. Sensors on a first vehicle can capture sensor data of the environment, including agents other than the first vehicle, such as a second vehicle or pedestrians. Multichannel images, representing a top-down view of the agents and environment and containing semantic information, can be generated based on the sensor data. Semantic information can include bounding boxes and velocity information associated with the agents, map data, and other semantic information. It is possible to generate a variety of images representing the environment over time. The images can be input into a prediction system configured to output a heatmap consisting of predicted probabilities associated with possible future locations of the agents. The predicted trajectory is generated based on the predicted probabilities and can be output to control the operation of the first vehicle.

Description

[Technical field]

[0001] The present invention relates to trajectory prediction for top-down scenes. [Background technology]

[0002] This patent application claims priority to U.S. utility patent application Ser. No. 16 / 151607, filed Oct. 4, 2018. Application Ser. No. 16 / 151607 is incorporated by reference herein in its entirety.

[0003] Prediction techniques can be used to determine the future states of entities in an environment, i.e., how a particular entity is likely to behave in the future. Current prediction techniques often involve physics-based modeling or traffic rule simulations to predict the future states of entities in an environment. [Brief description of the drawings]

[0004] The detailed description will now be described with reference to the accompanying drawings, in which the left-most digit of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical components or features.

[0005] [Figure 1] FIG. 1 is a pictorial flow diagram of an exemplary process for capturing sensor data, inputting an image representing a top-down view of an environment into a prediction system, receiving a heat map associated with a prediction probability, and generating a predicted trajectory based at least in part on the prediction probability, in accordance with an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram of an example system for implementing the techniques described herein. [Diagram 3]FIG. 1 is a pictorial flow diagram of an example process for receiving heat maps associated with predicted probabilities, determining predicted points based at least in part on the highest probability in each heat map, and generating one or more predicted trajectories based at least in part on the predicted points, in accordance with an embodiment of the present disclosure. [Figure 4A] FIG. 1 is an exemplary diagram of inputs to a prediction system for determining one or more predicted trajectories, according to an embodiment of the present disclosure. [Figure 4B] FIG. 1 is an exemplary diagram of inputs to a prediction system for determining one or more predicted trajectories, according to an embodiment of the present disclosure. [Diagram 5] 1 is a diagram of various predicted trajectories as output by a prediction system, according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates an exemplary process for generating a predicted trajectory based at least in part on a top-down representation of an environment that includes information associated with an agent in the environment, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] The present disclosure is directed to determining a predicted trajectory based at least in part on a top-down representation of an environment. For example, sensors of an autonomous vehicle can capture sensor data of an environment, which may include agents separate from the autonomous vehicle, such as other vehicles or pedestrians. An image representing the environment from a top-down perspective can be generated based at least in part on the sensor data. In some examples, the image may include semantic information of the environment and / or the agent. With respect to the agent, such semantic information may comprise a bounding box having a range (e.g., identifying the bounding box as being associated with the vehicle), movement information associated with the bounding box (e.g., speed, acceleration, etc.), classification information (whether the box represents a vehicle, a pedestrian, a bicyclist, a motorcyclist, etc.). With respect to the environment, the image may comprise semantic information indicating speed limits, lane widths, lane ends, stop lines, traffic signals, traffic signal conditions, etc., as described in more detail herein.

[0007] A plurality of images can be generated that represent the environment over time. The image or plurality of images can be input to a predictive system (e.g., a machine learning model) configured to output a heat map with predicted probabilities associated with possible future locations of the agent. One or more predicted trajectories can be generated based on the predicted probabilities and output to a planning system to control operation of the autonomous vehicle.

[0008] The sensor data captured by the autonomous vehicle may include LIDAR data, RADAR data, image data, time of flight data, SONAR data, etc. In some cases, the sensor data may be provided to a perception system configured to determine a type of agent in the environment (e.g., vehicle, pedestrian, bicycle, animal, parked car, tree, building, etc.). In addition, the perception system may determine movement information about the agent in the environment based on the sensor data. The sensor data and any data based on the sensor data may be represented in a top-down view of the environment. For example, the image may represent the agent as a two-dimensional bounding box representing the location of the agent in the environment as well as the agent's range (e.g., the agent's length and width) and the agent's classification (e.g., vehicle, pedestrian, etc.). Movement information, such as speed information, may be represented as a velocity vector associated with the bounding box, although other representations are contemplated.

[0009] In some cases, the image to be input to the prediction system can be represented by individual channels of a multi-channel image, with each channel representing different information about the agent and the environment in which the agent is located. For example, the channels may represent, but are not limited to, one or more of road network information (e.g., semantic labels indicating lanes, crosswalks, stop lines, lane dividers, stop signs, intersections, traffic signals, etc.), traffic signal status (e.g., red light, yellow light, green light, etc.), a bounding box associated with the agent, the agent's speed in x and y directions, the agent's acceleration in x and y directions, the agent's turn signal status (e.g., turn left, turn right, brake, reverse, hazard, no lights, etc.), etc. In some examples, multiple channels can be input to the prediction system to generate at least one predicted trajectory. In some examples, an image can be directly input to the prediction system to generate at least one predicted trajectory (e.g., without generating multiple channels based on an image).

[0010] In some examples, the number of channels may be reduced to a smaller number of channels. As a non-limiting example, a simple three-channel (e.g., RGB) image may be used to represent a variety of semantic information. In such examples, the red channel may be used, for example, to indicate a bounding box, and the intensity of the red value may be associated with speed, acceleration, etc. Furthermore, in such examples, a second channel may encode lane width, with the intensity indicating a speed limit, etc.

[0011] In some examples, the prediction system may include a machine learning model trained to output data that may be used to generate one or more predicted trajectories. For example, the machine learning model may output coordinates (e.g., x and y coordinates) associated with an agent (e.g., a third-party vehicle) at one or more times in the future (e.g., 1 second, 2 seconds, 3 seconds, etc.). In some examples, the machine learning model may output coordinates associated with the agent as well as probability information associated with each coordinate. In some examples, such probability information may be represented as an ellipse or other polygon associated with a threshold probability level (e.g., a 65% probability that the location of the agent at a particular time is within the area represented by the ellipse). The predicted trajectories may be generated by interpolating between the coordinates output by the machine learning model. In some examples, the machine learning model may include a convolutional neural network (CNN), which may include one or more recurrent neural network (RNN) layers, such as, but not limited to, a long short-term memory (LSTM) layer. In some examples, the machine learning model may output a heat map associated with the prediction probability. In some examples, at least one predicted trajectory may be determined based at least in part on the heat map.

[0012] A heatmap can represent a discretized region of the environment proximate to the autonomous vehicle. For example, a heatmap can represent a 64×64 grid (or a grid of size J×K) representing a 100 meter×100 meter region around the autonomous vehicle. Of course, a heatmap can represent any size region and any number of discrete portions of the region. In some cases, a portion of a heatmap may be referred to as a cell of the heatmap. Each cell can comprise a predicted probability representing the probability that an agent will be at a corresponding location in the environment at a time corresponding to the heatmap. In some cases, the machine learning model can output multiple heatmaps, where the heatmaps of the multiple heatmaps can represent probability predictions associated with an agent at a particular time in the future (e.g., 0.5 seconds, 1 second, 3 seconds, 5 seconds, 10 seconds, etc.).

[0013] To determine a predicted trajectory from the heatmaps, each heatmap can be individually analyzed to determine the highest probability associated with a cell of the heatmap. In some cases, the area of ​​the heatmap associated with the highest probability may be referred to as a predicted point. In some examples, the predicted point can be selected based at least in part on the highest probability point and one or more costs and / or constraints associated with the vehicle dynamics of the resulting predicted trajectory. As an example, if the prediction system outputs five heatmaps, the techniques described herein can include determining at least five predicted points. A first predicted point can be selected based on corresponding to the area of ​​highest probability in the first heatmap. A second predicted point can be selected based on the area of ​​highest probability in the second heatmap and a cost associated with the trajectory between the first predicted point and the second predicted point. For example, such costs or constraints can be used to enforce physical correctness (e.g., to ensure that the predicted trajectory follows a physically plausible trajectory). Use of the predicted points can determine a predicted trajectory. In some cases, the predicted trajectory may be determined by interpolating between the points or fitting a curve to the points (e.g., fitting one or more of a polynomial curve, a Bezier curve, a clothoid curve, etc.). In some cases, generating the predicted trajectory and / or selecting the predicted points may be based at least in part on one or more costs and / or constraints associated with the vehicle dynamics to prevent or reduce occurrences where the predicted trajectory represents an irrational trajectory (e.g., a subsequent point involves a "teleportation" of the vehicle with a physically impossible movement of the vehicle). Examples of such costs may include, but are not limited to, speed costs (e.g., constant velocity costs), acceleration costs, expectations that the agent may obey traffic rules, etc.

[0014] As will be appreciated, the heatmap may comprise prediction probabilities that may represent multiple predicted trajectories for an agent in an environment. In the context of a vehicle traversing an environment, a first predicted trajectory may represent a vehicle turning left through an intersection, while a second predicted trajectory may represent a vehicle going straight through the intersection. If the first predicted trajectory has a higher probability than the second predicted trajectory (e.g., because the sensor data may have captured a left turn indicator (e.g., blinker or turn signal) of the vehicle), the operation may include masking, covering, or otherwise removing the prediction probability of the heatmap corresponding to the first predicted trajectory. The masked heatmap may then be normalized (e.g., the prediction probabilities may be scaled between 0 and 1), and the highest probability of the masked heatmap may be determined as the prediction point. The second trajectory may be based at least in part on the prediction points associated with the masked heatmap. That is, the second set of prediction points may be used to generate a second predicted trajectory by evaluating one or more cost functions to determine the second trajectory. This masking process and determining a predicted trajectory can be repeated until the trajectory's probability does not meet or exceed a prediction threshold. At least one predicted trajectory can be provided to a planning system of the autonomous vehicle, whereby the autonomous vehicle can be controlled based at least in part on the at least one predicted trajectory. In at least other examples, all possible predicted trajectories and / or their corresponding uncertainties can be output to such a planning system.

[0015] In some cases, one or more heat maps can be provided to the planning system instead of or in addition to at least one predicted trajectory.

[0016] The techniques described herein can improve the functionality of a computing device in many additional ways. In some cases, representing the environment and agents in the environment as a top-down view can represent a simplified view of the environment and generate at least one predicted trajectory. In some cases, the image can represent the environment without extracting specific features of the environment, which can simplify the generation of the prediction system and the subsequent generation of the at least one predicted trajectory. In some cases, generating the at least one predicted trajectory can be provided to a planner system of the autonomous vehicle, which can enable the autonomous vehicle to generate a more accurate and / or safer trajectory for the autonomous vehicle to traverse the environment. For example, a predicted trajectory that indicates a potential collision or near-collision can enable the autonomous vehicle to alter the trajectory (e.g., change lanes, stop, etc.) to safely traverse the environment. These and other improvements to the functionality of a computer are described herein.

[0017] The techniques described herein may be implemented in several ways. Exemplary implementations are provided below with reference to the following figures. Although described in the context of an autonomous vehicle, the methods, apparatus, and systems described herein may be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In another example, the techniques may be utilized in an aviation or nautical context, or in any system with agents or entities that may be associated with behaviors unknown to the system. Additionally, although described in the context of LIDAR data, the sensor data may include any two-dimensional, three-dimensional, or multi-dimensional data, such as image data (e.g., stereo cameras, time-of-flight data, etc.), RADAR data, SONAR data, etc. Additionally, the techniques described herein may be used with real data (e.g., captured using a sensor), simulated data (e.g., generated by a simulator), or any combination of the two.

[0018] FIG. 1 is a pictorial flow diagram 100 of an exemplary process for capturing sensor data, inputting an image representing a top-down view of an environment into a prediction system, receiving a heat map associated with a prediction probability, and generating a predicted trajectory based at least in part on the prediction probability, in accordance with an embodiment of the present disclosure.

[0019] At operation 102, the process may include capturing sensor data of the environment. In some examples, the sensor data may be captured by one or more sensors on the autonomous vehicle. For example, the sensor data may include data captured by one or more of a LIDAR sensor, a RADAR sensor, an image sensor, a time-of-flight sensor, a SONAR sensor, etc.

[0020] Example 104 illustrates an environment in which an autonomous vehicle 106 is traversing the environment. In some examples, the autonomous vehicle 106 may perform the operations of process 100, which may include capturing sensor data of the environment. For purposes of illustration, the vehicle capturing (or utilizing) the sensor data may be an autonomous vehicle configured to operate in accordance with a Level 5 classification issued by the US National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions during an entire trip with no driver (or passenger) expected to control the vehicle at any time. In such an example, the vehicle may not require a driver, as it may be configured to control all functions from starting to stopping, including all parking functions. This is by way of example only, and the systems and methods described herein may be incorporated into any land, air, or water vehicle, including vehicles ranging from those that must be manually controlled by a driver at all times to those that are partially or fully autonomously controlled. Further details associated with the vehicles are described throughout this disclosure.

[0021] Agents 108 are also located in the environment. In some examples, agents 108 may represent vehicles, pedestrians, bicycles, animals, etc. The environment shown in example 104 may include a drivable area 110 and a non-drivable area 112. In some examples, operation 102 may include locating autonomous vehicle 106 in the environment and accessing map data associated with the environment. For example, the map data may comprise map elements such as crosswalk element 114 indicating an area of ​​the environment corresponding to a crosswalk. Additional examples of map elements may include, without limitation, one or more of lane elements, bicycle lane elements, crosswalk elements, intersection elements, lane divider elements, traffic signal elements, stop sign elements, stop line elements, yield sign elements, yield line elements, parking lane elements, roadway elements, speed bump elements, red light areas (e.g., virtual crosswalks), trajectory waypoints (e.g., known trajectories), passenger pick-up points, sign location elements, geofence elements, etc.

[0022] In some cases, the sensor data captured in operation 102 may be used to determine information about the agent 108, including, but not limited to, the type of agent (e.g., semantic information indicative of the agent's classification, such as a vehicle, pedestrian, bicycle, animal, etc.). In some cases, operation 102 may include determining the type of agent 108, a bounding box associated with the agent 108, and / or movement information associated with the agent, as described herein.

[0023] In operation 116, the process may include inputting images representing the environment into the predictive system. In some cases, the images may represent the environment from a top-down perspective, as shown in images 118, 120, and 122. That is, images 118, 120, and 122 may represent the environment shown in example 104 over time. For example, image 118 represents the environment at time T -N , and the image 120 may represent the environment at time T -N+1and image 122 may represent the environment at time T0. For example, image 118 may represent the environment two seconds ago, image 120 may represent the environment one second ago, and image 122 may represent the environment currently (e.g., time=0). Of course, images 118, 120, and 122 may represent the environment at any period of time and are not limited to the explicit times described herein.

[0024] As introduced above, image 118 is taken at time T -N The image 118 may represent the environment at the time depicted and may include a bounding box 124 representing the location and extent (e.g., length and width) of the agent 108 at the depicted time. Additionally, the image 118 may include a bounding box 126 representing the autonomous vehicle 106. As can be appreciated, the image 118, as described herein, may include any number of static objects (e.g., buildings, trees, curbs, sidewalks, map elements, etc.) or dynamic objects (e.g., agents) in the environment at any time and is not limited to one agent (e.g., the bounding box 124 representing the agent 108).

[0025] The image 118 may further include velocity information 128, represented in the image 118 as a velocity vector associated with the bounding box 124. As described above, the image 118 may include additional information that represents objects in the environment or the state of objects in the environment. For example, the image 118 may include lane or pathway information indicating that the lane associated with the bounding box 126 (e.g., associated with the autonomous vehicle 106) is controlled by a traffic light, which for simplicity may correspond to a red light. Thus, in this example, the position of the bounding box 126 may remain constant in the images 118, 120, 122, while the autonomous vehicle 106 may be moving through the environment while capturing sensor data and determining at least one predicted trajectory, as described herein.

[0026] As will be appreciated, bounding box 124 (e.g., agent 108) may be associated with different location and / or movement information over time. Thus, image 120 may include speed information 130 that is the same or different from speed information 128, and image 122 may include speed information 132 that is the same or different from speed information 128 and 130. Additionally, the location of bounding box 124 is updated throughout images 118, 120, and 122 to indicate the movement of agent 108 as it traverses the environment.

[0027] Additionally, as described below in the context of Figures 4A and 4B, an image input to the prediction system may be decomposed into multiple channels prior to input to the prediction system (e.g., Figure 4A) or may be input to the prediction system without generating multiple channels (e.g., Figure 4B).

[0028] At operation 134, the process may include receiving heat maps associated with predicted probabilities. In some examples, operation 134 may include receiving heat maps 136, 138, and 140 from a prediction system, where heat maps 136, 138, and 140 are based at least in part on images 118, 120, and 122 input to the prediction system at operation 116. In some cases, heat maps 136, 138, and 140 may represent predicted probabilities indicating the probability that agent 108 will be at a corresponding location in the environment at a time corresponding to the respective heat map (e.g., from a top-down perspective of the environment). That is, heat maps 136, 138, and 140 may represent predicted probabilities of agent 108 in the future.

[0029] Heatmap 136 may represent predicted probabilities 142 associated with agent 108 at time T1, heatmap 138 may represent predicted probabilities 144 associated with agent 108 at time T2, and heatmap 140 may represent predicted probabilities 142 associated with agent 108 at time T3, M , 138, and 140. Of course, heat maps 136, 138, and 140 can represent predicted probabilities for any period of time and are not limited to the explicit times described herein.

[0030] In some cases, the predicted probabilities 142, 144, and 146 may represent a probability distribution over the area represented by the respective heat maps 136, 138, and 140.

[0031] In operation 148, the operation may include determining a predicted trajectory based at least in part on the prediction probability. In some cases, operation 148 may include determining at least one predicted trajectory associated with the agent 108.

[0032] Example 150 illustrates predicted trajectories 152 and 154 associated with agent 108 determined by autonomous vehicle 106. In some cases, predicted trajectory 152 may represent a most likely predicted trajectory of agent 108 traversing the environment based on sensor data captured by autonomous vehicle 106 and based on output from a prediction system. In some cases, predicted trajectory 154 may represent a second most likely predicted trajectory of agent 108 traversing the environment based on the factors discussed above. That is, operation 148 may include ranking or associating a confidence value or probability associated with each of the predicted trajectories to determine the most likely trajectory for agent 108 to follow. In some cases, predicted trajectories 152 and 154 may be associated with individual uncertainties. Further details of predicted trajectories are described below with respect to FIGS. 3 and 5 and throughout this disclosure.

[0033] In some cases, process 100 may be performed in parallel for each agent in the environment. In some cases, process 100 may be performed on a single set of images to generate at least one trajectory for each agent of multiple agents in the environment.

[0034] 2 shows a block diagram of an example system 200 for implementing the techniques described herein. In at least one example, the system 200 may include a vehicle 202.

[0035] The vehicle 202 may include a vehicle computing device 204, one or more sensor systems 206, one or more emitters 208, one or more communication connections 210, at least one direct connection 212, and one or more drive systems 214.

[0036] The vehicle computing device 204 may include one or more processors 216 and a memory 218 communicatively coupled to the one or more processors 216. In the illustrated example, the vehicle 202 is an autonomous vehicle, although the vehicle 202 may be any other type of vehicle. In the illustrated example, the memory 218 of the vehicle computing device 204 stores a localization component 220, a perception component 222, a planning component 224, one or more system controllers 226, one or more maps 228, and a prediction component 230 comprising an image generation component 232, a heatmap generation component 234, a predicted points component 236, and a trajectory generation component 238. While shown in FIG. 2 as residing in memory 218 for purposes of illustration, it is contemplated that the localization component 220, the perception component 222, the planning component 224, the one or more system controllers 226, the one or more maps 228, the prediction component 230, the image generation component 232, the heat map generation component 234, the predicted points component 236, and the trajectory generation component 238 may additionally or alternatively be accessible to the vehicle 202 (e.g., may be stored remotely).

[0037] In at least one example, the localization component 220 can include functionality for receiving data from the sensor system 206 to determine the position and / or orientation (e.g., one or more of x position, y position, z position, roll, pitch, or yaw) of the vehicle 202. For example, the localization component 220 can include and / or request / receive a map of the environment and can continuously determine the position and / or orientation of the autonomous vehicle within the map. In some cases, the localization component 220 can utilize SLAM (simultaneous localization and mapping), CLAMS (simultaneous calibration, localization, and mapping), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc. to receive image data, LIDAR data, radar data, IMU data, GPS data, wheel encoder data, etc. to precisely determine the position of the autonomous vehicle. In some cases, the localization component 220 can provide data to various components of the vehicle 202 as described herein to determine an initial position of the autonomous vehicle for generating a trajectory and / or for generating map data.

[0038] In some cases, the perception component 222 may include functionality to perform object detection, segmentation, and / or classification. In some examples, the perception component 222 may provide processed sensor data indicative of the presence of an entity proximate to the vehicle 202 and / or the classification of the entity as an entity type (e.g., car, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional or alternative examples, the perception component 222 may provide processed sensor data indicative of one or more characteristics associated with the detected entity (e.g., tracked object) and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity may include, but are not limited to, x-position (global and / or local position), y-position (global and / or local position), z-position (global and / or local position), orientation (e.g., roll, pitch, yaw), entity type (e.g., classification), velocity of the entity, acceleration of the entity, range (size) of the entity, etc. Characteristics associated with an environment may include, but are not limited to, the presence of other entities in the environment, the state of other entities in the environment, the time of day, the day of the week, the season, weather conditions, indications of darkness / light, etc.

[0039] In general, the planning component 224 can determine a path for the vehicle 202 to follow to traverse through an environment. For example, the planning component 224 can determine various routes and trajectories and various levels of detail. For example, the planning component 224 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this description, the route can be a series of waypoints to travel between the two locations. As non-limiting examples, the waypoints include streets, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 224 can generate instructions for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 224 can determine how to guide the autonomous vehicle from a first waypoint in the series of waypoints to a second waypoint in the series of waypoints. In some examples, the instructions can be a trajectory or a portion of a trajectory. In some examples, multiple trajectories can be generated substantially simultaneously (e.g., within engineering tolerances) according to a receding horizon approach, where one of the multiple trajectories is selected for vehicle 202 to navigate.

[0040] In at least one example, the vehicle computing device 204 may include one or more system controllers 226 that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 202. These system controllers 226 may communicate with and / or control corresponding systems of the drive system 214 and / or other components of the vehicle 202.

[0041] The memory 218 may further include one or more maps 228 that may be used by the vehicle 202 to navigate within the environment. For purposes of this description, a map may generally be any number of data structures modeled in two, three, or n-dimensions that may provide information about the environment, such as, but not limited to, topology (such as intersections), streets, mountains, roads, terrain, and the environment. In some cases, the map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectance information (e.g., specularity information, retroreflectivity information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some cases, the map may be stored in a tile format, such that individual tiles of the map represent separate portions of the environment, and may be loaded into the working memory as needed, as described herein. In at least one example, the one or more maps 228 may include at least one map (e.g., an image and / or a mesh). In some examples, the vehicle 202 may be controlled based at least in part on the map 228. That is, the map 228 may be used in conjunction with the localization component 220, the perception component 222, and / or the planning component 224 to determine a position of the vehicle 202, identify objects in the environment, and / or generate a route and / or trajectory for navigating within the environment.

[0042] In some examples, one or more maps 228 may be stored on a remote computing device (such as computing device 242) accessible via network 240. In some examples, multiple maps 228 may be stored, for example, based on a characteristic (e.g., type of entity, time of day, day of the week, season, etc.). Storing multiple maps 228 may have similar memory requirements but increase the speed at which data in the maps can be accessed.

[0043] In general, the prediction component 230 may generate predicted trajectories for objects in the environment. For example, the prediction component 230 may generate one or more predicted trajectories for vehicles, pedestrians, animals, etc. within a threshold distance from the vehicle 202. In some cases, the prediction component 230 may measure traces of the objects and generate trajectories for the objects based on the observed and predicted behavior.

[0044] In some examples, the image generation component 232 may include functionality for generating images that represent a top-down view of a scene. Examples of such images generated by the image generation component 232 are shown in FIG. 1 as images 118, 120, and 122.

[0045] In some examples, the image generation component 232 can generate an image representing an area (e.g., a receding horizon) around the vehicle 202. In some examples, the image may represent an area of ​​100 meters by 100 meters around the vehicle 202, although any area is contemplated. The image generation component 232 can receive data about agents in the environment from the perception component 222, and can receive data about the environment itself from the localization component 220, the perception component 222, and one or more maps 228. The image generation component 232 can generate a top-down view of the environment including agents in the environment (e.g., represented by bounding boxes as described herein), semantic information about the agents (e.g., classification type), movement information (e.g., speed information, acceleration information, etc.), and the like. The image generation component 232 may further generate graphical elements (e.g., symbols) in the image that represent various attributes of the agent and the environment, including, but not limited to, one or more velocity vectors, one or more acceleration vectors, lighting indicators (e.g., the agent's brake lights, the agent's turn signal lights, etc., represented as respective indicator statuses), traffic signal information (e.g., the status of a traffic signal), lane information (e.g., whether a lane is a left-turn-only lane or a right-turn-only lane), etc.

[0046] In some examples, the image generation component 232 may generate one or more images that comprise semantic information of the environment, as described in more detail herein.

[0047] In some examples, the image generation component 232 can generate one or more images that represent a trace of the agent in the environment. As described above in FIG. 1, the images 118, 120, and 122 can represent prior information about the state of the environment and the agent in the environment that can be used to generate at least one predicted trajectory, as described herein.

[0048] In any of the examples described herein, the image generation component 232 may generate an image having multiple channels, with each channel representing some information (eg, semantic).

[0049] In some examples, the heatmap generation component 234 can include functionality to receive images generated by the image generation component 232 and generate one or more heatmaps including predicted probabilities, as described herein.

[0050] In some cases, the heatmap generation component 234 can include one or more machine learning models trained to output one or more heatmaps based on top-down images input to the model. In some examples, the heatmap generation component 234 can include one or more recurrent neural network (RNN) layers, e.g., a convolutional neural network (CNN) that may or may not include a long short-term memory (LSTM) layer.

[0051] In some examples, the heatmap output by the heatmap generation component 234 can represent a discretized region of the environment proximate to the autonomous vehicle. For example, the heatmap can represent a 64×64 grid (or a grid of size J×K) representing a 100 meter×100 meter region around the autonomous vehicle. In some examples, the area represented by the heatmap can correspond to the area represented by the image input to the heatmap generation component 234. Of course, the heatmap can represent an area of ​​any size and can represent any number of discrete portions of an area. In some cases, a portion of the heatmap may be referred to as a cell of the heatmap. Each cell can comprise a predicted probability representing the probability that the agent will be at a corresponding location in the environment at the time corresponding to the heatmap. In some cases, the machine learning model can output multiple heatmaps, where the heatmap of the multiple heatmaps can represent a probability prediction associated with the agent at a particular time in the future (e.g., 0.5 seconds, 1 second, 3 seconds, 5 seconds, 10 seconds, etc.).

[0052] In some examples, the heatmap generation component 234 can be trained to generate any number of heatmaps based on any number of input images. For example, the heatmap generation component 234 can receive N input images and output M heatmaps.

[0053] In some examples, the predicted point component 236 can include functionality for determining a predicted point representing a predicted location of the agent in the environment. In some examples, the predicted point component 236 can receive at least one heat map from the heat map generation component 234. The predicted point component 236 can determine a predicted point associated with the heat map based at least in part on a cell associated with a highest probability and / or based at least in part on a cost associated with generating a predicted trajectory associated with the predicted point.

[0054] For example, the predicted point component 236 may select a point, cell, or region of the heat map as a predicted point based at least in part on evaluating one or more cost functions associated with vehicle dynamics. In at least some examples, such costs may be associated with preventing or reducing occurrences where the predicted trajectory represents an irrational trajectory (e.g., involving vehicle "teleportation," too large changes in speed, etc.). Examples of such costs may include, without limitation, position-based costs (e.g., limiting the allowed distance between predicted points), speed costs (e.g., constant speed costs that enforce constant speed throughout the predicted trajectory), acceleration costs (e.g., enforcing acceleration limits throughout the predicted trajectory), expectations that the agent will obey traffic rules, etc. In at least some examples, the probabilities associated with the cells may be multiplied by the costs (which, in at least some examples, may be normalized), such that the point (e.g., candidate point) associated with the highest value of the cost multiplied by the probability is selected as the predicted point.

[0055] Further, in some cases, the predicted points component 236 can determine predicted points for the heat maps and provide the predicted points to the trajectory generation component 238 based at least in part on the predicted points to determine a predicted trajectory.

[0056] In some cases, after determining the predicted points for each heatmap, the predicted points component 236 can receive the predicted trajectory from the trajectory generation component 238 and can mask the area of ​​the heatmap associated with the predicted trajectory to generate a masked heatmap. The predicted points component 236 can normalize the remaining unmasked prediction probabilities and can determine predicted points associated with the masked heatmap. Additional predicted points can be determined for each masked heatmap, and the additional predicted points can be provided to the trajectory generation component 238 to determine additional predicted trajectories. In some cases, the masking and predicted points determining operations can be repeated until the prediction probabilities or probabilities of the resulting predicted trajectories fall below a threshold. In other examples, such a process can be repeated until the heatmap is completely masked. Further details of determining predicted points are described below with respect to FIG. 3.

[0057] In some cases, the predicted points component 236 can receive the image generated by the image generation component 232 to generate one or more predicted points directly (e.g., without determining the predicted points from a heatmap provided by the heatmap generation component 234). In such cases, the predicted points component 236 can comprise a CNN that may or may not include one or more RNN layers (which may be LSTM layers).

[0058] In some examples, the trajectory generation component 238 can include functionality to receive one or more predicted points from the predicted points component 236 to generate one or more predicted trajectories. In some examples, the trajectory generation component 238 can generate a curve based at least in part on the predicted points as a predicted trajectory.

[0059] In some cases, the trajectory generation component 238 can provide at least one predicted trajectory to the planning component 224, which can then generate a trajectory for the vehicle 202 to follow.

[0060] In some cases, the heatmap generation component 234 can provide a heatmap directly to the planning component 224 (e.g., with or without a predicted trajectory), which can then generate a trajectory for the vehicle 202 to follow.

[0061] In some cases, some or all aspects of the components described herein may include any models, algorithms, and / or machine learning algorithms. For example, in some cases, the components in memory 218 (and memory 246 described below) may be implemented as neural networks.

[0062] As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer in a neural network may also comprise another neural network, or may comprise any number of layers (whether convolutional or not). As may be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0063] Although described in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, machine learning algorithms include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least-angle regression (LARS)), decision tree algorithms (e.g., classification and regression tree (CART), iterative dichotomiser 3 (ID3), Chi-squared automatic interaction detection (CHAID), decision strain, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average one-dependence estimator (AD)), and Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average one-dependence estimator (AD)). estimators (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-means, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backpropagation, Hopfield network, radial basis function network (RBFN)), deep learning algorithms (e.g., Deep Boltzmann Machine (DBM), deep belief network (DeepBelief Networks (DBN), Convolutional Neural Networks (CNN), Stacked Autoencoders), Dimensionality Reduction Algorithms (e.g. Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), Ensemble Algorithms (e.g. Boosting, Bootstrap Aggregation (Bagging), Adaboost, Stack Generalization (Blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees), These can include GRBFT (Grain-Based Random Forests), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, and more.

[0064] Additional examples of architectures include neural networks such as ResNet70, ResNet101, VGG, DenseNet, and PointNet.

[0065] In at least one example, the sensor system 206 may include LIDAR sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), time-of-flight sensors, microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and the like. The sensor system 206 may include multiple instances of each of these or other types of sensors. For example, the LIDAR sensors may include individual LIDAR sensors located on the edges, front, back, sides, and / or top of the vehicle 202. As another example, the camera sensors may include multiple cameras disposed at various locations on the exterior and / or interior of the vehicle 202. The sensor system 206 may provide input to the vehicle computing device 204. Additionally or alternatively, the sensor system 206 may send sensor data over one or more networks 240 to one or more computing devices at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0066] The vehicle 202 may also include one or more emitters 208 for emitting light and / or sound, as described above. The emitters 208 in this example include internal audiovisual emitters for communicating with occupants of the vehicle 202. By way of example and not limitation, the internal emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 208 in this example also include external emitters. By way of example and not limitation, the external emitters in this example include lights or other indicators of vehicle actions for signaling heading (e.g., indicator lights, signs, light arrays, etc.), one or more of which may comprise acoustic beam steering technology, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles.

[0067] Vehicle 202 may also include one or more communication connections 210 enabling communication between vehicle 202 and one or more other local or remote computer devices. For example, communication connections 210 may facilitate communication with other local computing devices on vehicle 202 and / or drive system 214. Communication connections 210 may also enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Communication connections 210 may also enable vehicle 202 to communicate with remote teleoperated computing devices or other remote services.

[0068] The communication connection 210 may include a physical and / or logical interface for connecting the vehicle computing device 204 to another computing device or a network, such as the network 240. For example, the communication connection 210 may enable Wi-Fi based communications over frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that allows each computing device to interface with other computing devices.

[0069] In at least one example, the vehicle 202 may include one or more drive systems 214. In some examples, the vehicle 202 may have a single drive system 214. In at least one example, when the vehicle 202 has multiple drive systems 214, the individual drive systems 214 may be located at both ends of the vehicle 202 (e.g., front and back, etc.). In at least one example, the drive system 214 may include one or more sensor systems to detect conditions of the drive system 214 and / or surroundings of the vehicle 202. By way of example and not limitation, the sensor systems may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive system, cameras or other image sensors, ultrasonic sensors for acoustically detecting objects around the drive system, LIDAR sensors, RADAR sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive system 214. In some cases, sensor systems on drive system 214 may overlap or complement corresponding systems on vehicle 202 (eg, sensor system 206).

[0070] The drive system 214 may include many of the vehicle systems, including a high-voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking forces to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights for illuminating the exterior surroundings of the vehicle), as well as one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 214 may receive data from sensor systems and pre-process it to control the operation of various vehicle systems. It may include a drive system controller. In some examples, the drive system controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more components for performing various functions of the drive system 214. Additionally, drive system 214 also includes one or more communication connections that enable the respective drive system to communicate with one or more other local or remote computing devices.

[0071] In at least one example, the direct connection 212 may provide a physical interface to couple the body of the vehicle 202 and one or more drive systems 214. For example, the direct connection 212 may enable the transfer of energy, fluid, air, data, etc. between the drive systems 214 and the vehicle. In some cases, the direct connection 212 may also removably secure the drive systems 214 to the body of the vehicle 202.

[0072] In some examples, the vehicle 202 may send sensor data to one or more computing devices 242 over the network 240. In some examples, the vehicle 202 may send raw sensor data to the computing device 242. In other examples, the vehicle 202 may send processed sensor data and / or representations of the sensor data to the computing device 242. In some examples, the vehicle 202 may send sensor data to the computing device 242 at a particular frequency, after a predetermined period of time, in near real-time, etc. In some cases, the vehicle 202 may send sensor data (raw or processed) to the computing device 242 as one or more log files.

[0073] The computing device 242 may include a processor 244 and a memory 246 that stores a training component 248 and a prediction component 250 .

[0074] In some cases, the training component 248 can include functionality to train the machine learning model to output one or more predicted points, where one or more confidence values ​​are associated with the one or more predicted points, or one or more heat maps include the prediction probabilities. For example, the training component 248 can receive a set of images representing a top-down view of an environment. In some cases, the set of images can represent an agent traversing through the environment for a period of time, such as 5 seconds, 7 seconds, 10 seconds, 20 seconds, etc. At least a portion of the set of images can be used as input to train the machine learning model, and at least a portion of the set of images can be used as ground truth information for training the machine learning model. As a non-limiting example, a first set of images in a series (e.g., 3, 4, 5 or more) can be input to the machine learning model. A second set of images in the series of images immediately preceding the first set (or trajectory information associated therewith, e.g., by extracting position and / or velocity from the images) can then be used as ground truth for training the model. Thus, by providing images of the recorded trajectory of the agent traversing the environment, the prediction components 230 and 250 can be trained to output one or more heat maps including predicted points, confidence values ​​associated with the predicted points, or prediction probabilities, as described herein.

[0075] In some cases, prediction component 250 may correspond substantially to prediction component 230 and may include functionality for generating predicted trajectories of objects in the environment. For example, prediction component 250 may generate one or more predicted trajectories for vehicles, pedestrians, animals, etc. within a threshold distance from vehicle 202. In some cases, prediction component 250 may measure traces of the objects and generate trajectories for the objects based on observed and predicted behavior.

[0076] The processor 216 of the vehicle 202 and the processor 244 of the computing device 242 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processors 216 and 244 may comprise one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data to convert the electronic data to other electronic data that may be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0077] The memories 218 and 246 are examples of non-transitory computer-readable media. The memories 218 and 246 can store an operating system and one or more software applications, instructions, programs, and / or data to implement the functions attributed to the methods and various systems described herein. In various implementations, the memories can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous random access memory (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, program, and physical components, and those shown in the accompanying figures are only examples associated with the description herein.

[0078] In some cases, memories 218 and 246 may include at least working memory and storage memory. For example, working memory may be a limited capacity high-speed memory (e.g., cache memory) used to store data to be operated on by processors 216 and 244. In some cases, memories 218 and 246 may include storage memory, which may be a relatively large capacity lower speed memory used for long-term storage of data. In some cases, processors 216 and 244 may not be able to operate directly on data stored in storage memory, as described herein, and data may need to be loaded into working memory to perform operations based on the data.

[0079] 2 is illustrated as a distributed system, it should be noted that in alternative examples, components of vehicle 202 may be associated with computing device 242 and / or components of computing device 242 may be associated with vehicle 202. That is, vehicle 202 may perform one or more of the functions associated with computing device 242 and vice versa.

[0080] FIG. 3 is a pictorial flow diagram of an example process 300 for receiving heat maps associated with predicted probabilities, determining predicted points based at least in part on the highest probability in each heat map, and generating one or more predicted trajectories based at least in part on the predicted points, in accordance with an embodiment of the present disclosure.

[0081] At operation 302, the process may include receiving a heat map associated with the predicted probabilities. In some cases, operation 302 may include receiving heat maps 136, 138, and 140, as described above with respect to FIG. 1. Heat map 136 may include predicted probabilities 142, which may represent the probability of a location of agent 108 in the environment at time T1. Heat map 138 may include predicted probabilities 144, which may represent the probability of a location of agent 108 in the environment at time T2. Heat map 140 may also include predicted probabilities 145, which may represent the probability of a location of agent 108 in the environment at time T3. M The prediction probability 146 may represent the probability of the location of the agent 108 in the environment at

[0082] At operation 304, the process may include determining a predicted point based at least in part on the highest probability and the cost. Figure 3 shows examples 306, 308, and 310 representing predicted points determined from heat maps 136, 138, and 140, respectively.

[0083] The example 306 shows a predicted point 312 that represents the highest of the predicted probabilities 142 included in the heatmap 136. In some cases, for a first heatmap in a series of heatmaps, the first predicted point (e.g., predicted point 312) may correspond to the highest probability of the heatmap 136.

[0084] Example 308 illustrates a predicted point 314 based at least in part on the highest of the predicted probabilities 144 included in the heatmap 138 and one or more costs. For example, the predicted point 314 can be selected to maximize the probability associated with the predicted point 314 and to penalize the selection of a point (e.g., a candidate point) as a predicted point based on one or more cost functions associated with the vehicle dynamics to prevent or reduce occurrences where the predicted trajectory represents an irrational trajectory (e.g., involving a "teleportation" of the vehicle) or has a high associated cost. Examples of such costs and / or constraints can include, but are not limited to, position costs (e.g., such that the selected subsequent predicted point follows a physically plausible trajectory), speed costs (e.g., constant velocity costs), acceleration costs, expectations that the agent may obey traffic rules, etc.

[0085] Example 310 shows a candidate point 316 and a predicted point 318. In some examples, the candidate point 316 may represent the highest probability of the predicted probabilities 146 included in the heat map 140. In some examples, the predicted point 318 may represent a probability less than the highest probability for the candidate point 316. However, selecting the candidate point 316 as the predicted point incurs a high cost because such a trajectory based on the predicted points 312 and 314 and the candidate point 316 would represent the agent 108 “teleporting” from a location associated with the predicted point 314 to a location associated with the candidate point 316.

[0086] Alternatively, operation 304 may include selecting predicted point 318 based at least in part on a cost associated with a predicted trajectory based at least in part on predicted points 312, 314, and 318.

[0087] In operation 320, the process may include determining a predicted trajectory based on the predicted points. As shown in one example 322, operation 320 may interpolate between predicted points 312, 314, and 318 to determine a predicted trajectory 324.

[0088] In operation 326, the process may include masking the region associated with the predicted trajectory. Examples 328, 330, and 332 show heat maps 136, 138, and 140, respectively, with masks 334, 336, and 338 obscuring (wholly or partially) the predicted probabilities 142, 144, and 146, respectively. In some examples, mask 334 may represent a region associated with predicted point 312. For example, mask 334 may represent a circle or other polygon centered on or associated with predicted point 312. In some examples, mask 334 may represent a circle having a radius of one meter, although mask 334 may represent any size. In some examples, mask 336 may be associated with predicted point 314 and mask 338 may be associated with predicted point 318.

[0089] In some examples, the mask may be based at least in part on the predicted trajectory 324. For example, the mask may correspond to lane elements represented in the map data that are traversed by the predicted trajectory 324.

[0090] In some examples, operation 326 may generate masked heatmaps that correspond to heatmaps 136, 138, and 140, respectively.

[0091] At operation 340, the process may include determining whether the predicted probabilities are below a threshold. For example, operation 340 may include determining whether an unmasked portion of the predicted probabilities associated with each heatmap is below a threshold. If "no" (indicating there are predicted probabilities that meet or exceed the threshold), the process may return to operation 304 to determine one or more predicted points based on the masked heatmap, which may then be used to determine additional predicted trajectories.

[0092] For example, because candidate points 316 were not masked by mask 338 in example 332, such predicted points can be used to determine additional predicted trajectories.

[0093] In some examples, operation 340 or 304 may include normalizing the unmasked prediction probabilities (e.g., scaling the unmasked prediction probabilities between 0 and 1) prior to determining the additional predicted trajectories.

[0094] When there are no predicted probabilities below the threshold (e.g., "yes" at operation 340), the process may proceed to operation 342. In at least some examples, if any unmasked portions of the heatmap remain (or there are unmasked portions below some threshold amount), the operation may proceed to operation 304. In such examples, all possible (or likely) trajectories may be determined according to the techniques described herein.

[0095] At operation 342, the process may include outputting the predicted trajectory to a planning system to generate a trajectory for controlling the autonomous vehicle. Example 344 illustrates predicted trajectory 324 (e.g., based at least in part on predicted points 312, 314, and 318) and predicted trajectory 346 (e.g., based at least in part on candidate point 316, which may have been determined to be a predicted point for predicted trajectory 346). In some examples, the predicted trajectory may be output to planning component 224 of autonomous vehicle 106. Planning system 224 may generate one or more trajectories for autonomous vehicle 106 to follow based at least in part on at least one predicted trajectory. In at least some examples, the trajectory may comprise an uncertainty associated with each predicted point, an uncertainty of the trajectory as a whole, or a heat map.

[0096] 4A and 4B are example diagrams of inputs to a prediction system for determining one or more predicted trajectories, according to an embodiment of the present disclosure.

[0097] FIG. 4A shows an example 400 in which an input image 402 is used to generate multiple channels 404, which are then input to the prediction component 230 to determine one or more predicted trajectories 406.

[0098] In some examples, individual aspects of the input image 402 can be used to generate individual channels of the channels 404. For example, the individual channels can represent one or more of, without limitation, a bounding box (e.g., the bounding box 124, where the color of the bounding box represents the classification of the agent associated with the bounding box), the x and y velocity of the bounding box (where the velocity can be represented as a vector in one channel or in multiple channels, and the x and y velocities can be represented in different channels), the x and y acceleration of the agent (where the acceleration can be represented as a vector in one channel or in multiple channels, and the x and y accelerations can be represented in multiple channels), a turn signal status associated with the bounding box (e.g., turn left, turn right, brake, reverse, hazard, no lights, etc.), road network information (e.g., semantic labels indicating lanes, crosswalks, stop signs, intersections, traffic signals, etc.), traffic signal status (e.g., red light, yellow light, green light, etc.), etc. In some examples, multiple channels may be input into the prediction system to generate at least one predicted trajectory.

[0099] 4B illustrates an example 408 in which an input image 402 is input to a prediction component 230 to determine a predicted trajectory 406. In contrast to example 400, example 408 may not include generating multiple channels associated with image 402, and may instead include inputting image 402 directly to prediction component 230.

[0100] 5 is a diagram 500 of various predicted outputs 502 by a prediction system according to an embodiment of the present disclosure. Diagram 500 provides three representative examples 504, 506, and 508 of predicted outputs 502 output by prediction component 230 as described herein.

[0101] Example 504 illustrates predicted points 510 and an associated predicted trajectory 512. In some cases, predicted points 510 can be output directly by predicted points component 236 of Figure 2. In some cases, predicted trajectory 512 can be generated by interpolating between predicted points 510, which may include smoothing predicted trajectory 512 to optimize vehicle dynamics, comfort, etc.

[0102] The example 504 includes individual predicted points 514 associated with individual uncertainty information 516. In some examples, the uncertainty information 516 may represent an area of ​​the environment indicating a particular probability that the location of the agent 108 will be located with the area represented by the uncertainty information 516 at a time associated with the predicted point 514. In some examples, the certainty information may be output by the predicted point component 236 in association with the predicted point 514.

[0103] Example 506 illustrates another embodiment of a predicted trajectory. In some examples, predicted trajectory 512 and predicted trajectory 518 can be output by prediction component 230 according to the techniques described in FIG. 3 and throughout this disclosure. That is, example 506 can represent a predicted trajectory based at least in part on one or more heat maps, which can include a variety of trajectories with probabilities, rankings, or weights associated with each trajectory. In some cases, each predicted trajectory 512 and 518 can be associated with a respective uncertainty.

[0104] Example 508 illustrates the “raw” heatmaps 136, 138, and 140 that may be output by the forecasting component 230 for use in the planning component 224. That is, in some cases, the forecasting component 230 may output the heatmaps 136, 138, and 140 in addition to or instead of the output illustrated by examples 504 and 506.

[0105] Thus, diagram 500 illustrates the flexible output of the prediction component 230 described herein, whereby the output format of the predicted output 502 is based on the particular implementation of the system.

[0106] 1, 3, and 6 illustrate exemplary processes according to embodiments of the present disclosure. These processes are illustrated as logical flow graphs, with each operation representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described should not be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0107] 6 illustrates an example process 600 for generating a predicted trajectory based at least in part on a top-down representation of an environment that includes information associated with agents in the environment, according to an embodiment of the disclosure. For example, some or all of the process 600 may be performed by one or more components in FIG. 2 as described herein. For example, some or all of the process 600 may be performed by the vehicle computing device 204 and / or the computing device 242.

[0108] At operation 602, the process may include capturing sensor data of the environment using sensors of the autonomous vehicle. For example, the sensor data may include data captured by one or more of a LIDAR sensor, a RADAR sensor, an image sensor, a time-of-flight sensor, a SONAR sensor, etc. In some examples, operation 602 may include capturing sensor data using multiple sensors and fusing or combining the sensor data into a detailed and information representation of the environment.

[0109] At operation 604, the process may include determining whether an agent is represented in the sensor data. For example, operation 604 may include inputting at least a portion of the sensor data to a detector or providing the data to a detection or perception algorithm (e.g., perception component 222) to identify one or more agents in the environment. Examples of agents include, but are not limited to, vehicles, pedestrians, bicycles, skateboards, scooters, animals, etc. If an agent is not detected (e.g., "no" at operation 604), the process returns to operation 602 to capture additional sensor data. If an agent is detected (e.g., "yes" at operation 604), the process returns to operation 602 to capture additional sensor data.

[0110] At operation 606, the process may include generating an image representing a top-down view of the environment based at least in part on the sensor data, the image comprising a bounding box associated with the agent and velocity information associated with the agent. In some examples, the image may comprise acceleration information (instead of or in addition to the velocity information).

[0111] Additional information that may be included in the image may include, but is not limited to, a color of the bounding box, which represents a classification of an agent associated with the bounding box, a velocity of the bounding box in the x and y directions (wherein the velocity may be represented as a vector in one channel or in multiple channels, and the x and y velocities are represented in different channels), an acceleration of the agent in the x and y directions (wherein the acceleration may be represented as a vector in one channel or in multiple channels, and the x and y accelerations may be represented in different channels), a turn signal status associated with the bounding box (e.g., turn left, turn right, brake, reverse, hazard, no lights, etc.), road network information (e.g., semantic labels indicating lanes, crosswalks, stop signs, intersections, traffic signals, etc.), traffic signal status (e.g., red light, yellow light, green light, etc.), etc. In such an image, semantic information about the agent and the environment may be encoded in one or more channels, as described in detail herein.

[0112] As described herein, operation 606 may include generating a number of images representative of the behavior of the agent in the environment over time.

[0113] At operation 608, the process may include inputting the image into a machine learning model trained to generate a heat map associated with a prediction probability. In some examples, operation 608 may include generating multiple channels based on the image and inputting the multiple channels into the machine learning model. In some examples, the image may be input into the machine learning model without generating multiple channels. Further details of the machine learning model, the heat map, and the prediction probability are described throughout this disclosure.

[0114] At operation 610, the process may include determining a predicted trajectory associated with the agent based at least in part on the heat map. Further details of determining a predicted trajectory based at least in part on the heat map are described above with respect to FIG. 3 as well as throughout this disclosure.

[0115] At operation 612, the process may include determining a trajectory for the autonomous vehicle based at least in part on the predicted trajectory. In some examples, a trajectory based on the predicted trajectory may represent a safer and / or smoother trajectory compared to a trajectory generated without the predicted trajectory, because the planning component 224 that generates the trajectory more closely anticipates actions of entities proximate to the autonomous vehicle.

[0116] At operation 614, the process may include controlling the autonomous vehicle based at least in part on the trajectory. In some examples, operation 614 may include controlling one or more of steering, braking, and / or accelerating the autonomous vehicle to traverse through an environment.

[0117] Example clauses A: A system comprising: one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors, the instructions which, when executed, cause the system to perform operations comprising: capturing sensor data of an environment using at least one sensor of an autonomous vehicle; generating an image representing a top-down view of the environment based at least in part on the sensor data, the image comprising a bounding box associated with the vehicle and speed information associated with the vehicle, the vehicle being separate from the autonomous vehicle; inputting the image into a machine learning model trained to generate a heat map associated with predicted probabilities of possible locations associated with the vehicle; determining a predicted trajectory associated with the vehicle based at least in part on the heat map; and determining a trajectory for traveling by the autonomous vehicle through the environment based at least in part on the predicted trajectory.

[0118] B: The system of paragraph A, wherein determining the predicted trajectory comprises determining the predicted points based at least in part on a highest probability portion associated with a portion of the heat map and a cost associated with determining a predicted trajectory based at least in part on the predicted points.

[0119] C: The system of paragraph B, wherein the predicted point is a first predicted point, the highest probability portion is the first highest probability portion, the portion is the first portion, the cost is the first cost, and the predicted trajectory is the first predicted trajectory, and the operation further comprises masking a second portion of the heatmap associated with the first predicted point as a masked heatmap, the second portion of the heatmap comprising at least the first portion; determining a second highest probability portion associated with a third portion of the masked heatmap as a second predicted point; and determining a second predicted trajectory based at least in part on the second predicted point and a second cost associated with the second predicted point and determining a second predicted trajectory based at least in part on the second predicted point.

[0120] D: The system described in any one of paragraphs A to C, wherein the image has a first channel representing a bounding box and semantic information associated with the vehicle and a second channel representing speed information associated with the vehicle, and the operation further includes inputting the first channel and the second channel into a machine learning model.

[0121] E: The system of any one of paragraphs A to D, wherein the machine learning model comprises a convolutional neural network (CNN) and a long short-term memory (LSTM) component coupled to the CNN.

[0122] F: A method comprising: receiving sensor data of an environment; generating an image representing a top-down view of an agent in the environment, the image including semantic information associated with the agent or the environment; inputting the image to an algorithm configured to generate a probability distribution associated with a predicted location of the agent; determining a predicted trajectory associated with the agent based at least in part on the probability distribution; and controlling an autonomous vehicle based at least in part on the predicted trajectory.

[0123] G: The method of paragraph F, wherein the probability distribution is represented as a heat map that discretizes a portion of the environment, and the predicted probabilities of the probability distribution are associated with portions of the heat map.

[0124] H: The method of paragraph G, wherein the step of determining a predicted trajectory comprises determining a predicted point based at least in part on a highest probability portion of the heat map and a cost associated with the predicted trajectory that is based at least in part on the predicted point.

[0125] I: The method of paragraph H, wherein the predicted point is a first predicted point, the most likely portion is the first most likely portion, the portion is the first portion, the cost is the first cost, and the predicted trajectory is the first predicted trajectory, and the method further comprises: masking a second portion of the heatmap associated with the first predicted point as a masked heatmap, the second portion of the heatmap comprising at least the first portion; and determining a second predicted point based at least in part on the second most likely portion of the heatmap and a second cost associated with determining a second predicted trajectory based at least in part on the second predicted point.

[0126] J: The method of any of paragraphs F-I, wherein the image comprises a plurality of channels, a channel of the plurality of channels comprising at least one of a location of an agent, a range of the agent, a bounding box associated with the agent, speed information associated with the agent, acceleration information associated with the agent, a turn signal status of the agent, map data associated with the environment, or traffic signal information associated with the environment.

[0127] K: The method of any one of paragraphs F-J, wherein the algorithm is a machine learning model comprising a convolutional neural network trained to generate a heat map representing a probability distribution, and the semantic information comprises a classification of the agent and speed information associated with the agent.

[0128] L: The method of any of paragraphs F-K, wherein controlling the autonomous vehicle based at least in part on the predicted trajectory comprises generating a trajectory for the autonomous vehicle to follow to traverse the environment, and controlling the autonomous vehicle based at least in part on the trajectory.

[0129] M: The method of any of paragraphs F-L, further comprising a step of capturing sensor data of the environment using sensors on the autonomous vehicle, the sensors comprising one or more of a LIDAR sensor, a RADAR sensor, an image sensor, or a time-of-flight sensor.

[0130] N: The method of any of paragraphs F-M, wherein the predicted trajectory is associated with an uncertainty.

[0131] O: The method of any of paragraphs F-N, wherein the sensor data represents an environment at a first time, the probability distribution is a first probability distribution associated with a second time after the first time, and the predicted location is a first predicted location, and the algorithm is further configured to generate a second probability distribution associated with a second predicted location of the agent at a third time after the second time.

[0132] P: A non-transitory computer-readable medium having instructions stored thereon that, when executed, cause one or more processors to perform operations comprising receiving sensor data of an environment; generating an image representing a top-down view of an agent in the environment, the image including semantic information associated with the agent or the environment; inputting the image to an algorithm configured to generate a probability distribution associated with a predicted location of the agent; determining a predicted trajectory associated with the agent based at least in part on the probability distribution; and controlling the autonomous vehicle based at least in part on the predicted trajectory.

[0133] Q: The non-transitory computer-readable medium of paragraph P, wherein the probability distribution is represented as a heat map that discretizes a portion of the environment, and a predicted probability of the probability distribution is associated with a portion of the heat map.

[0134] R: The non-transitory computer-readable medium of paragraph Q, wherein determining the predicted trajectory comprises determining the predicted points based at least in part on the highest probability portion and a cost associated with the heat map.

[0135] S: The non-transitory computer-readable medium of paragraph R, wherein the predicted point is a first predicted point, the highest probability portion is the first highest probability portion, the portion is the first portion, the cost is the first cost, and the predicted trajectory is the first predicted trajectory, and the operation further comprises masking a second portion of the heatmap associated with the first predicted point as a masked heatmap, the second portion of the heatmap comprising at least the first portion; and determining a second predicted point based at least in part on a second highest probability portion associated with a third portion of the masked heatmap and a second cost associated with determining a second predicted trajectory based at least in part on the second predicted point.

[0136] T: A non-transitory computer-readable medium described in any of paragraphs P to S, wherein the sensor data represents an environment at a first time, and the probability distribution is a first probability distribution associated with a second time after the first time, and the probability distribution is generated before the second time.

[0137] Although the example clauses described above are described with respect to one particular implementation, it should be understood in the context of this specification that the content of the example clauses may also be implemented via a method, device, system, computer-readable medium, and / or another implementation.

[0138] conclusion One or more examples of the techniques described herein have been described; however, various modifications, additions, permutations, and equivalents thereof fall within the scope of the techniques described herein.

[0139] In describing the examples, reference is made to the accompanying drawings, which form a part of this specification, which show, by way of example, specific examples of the claimed subject matter. It is understood that other examples can be used and that changes or modifications, such as structural changes, can be made. Such examples, changes or modifications do not necessarily depart from the intended scope of the claimed subject matter. Although steps in this specification may be presented in a certain order, in some cases the order can be changed, and thus some inputs are provided at different times or in different orders without changing the functionality of the systems and methods described. The procedures disclosed may also be performed in different orders. Furthermore, the various calculations in this specification do not have to be performed in the order disclosed, and other examples using alternative orders of calculations can be easily implemented. In addition to being reordered, calculations can also be decomposed into sub-calculations with the same results.

Claims

1. 1. A system comprising: one or more processors; one or more computer-readable media storing instructions executable by the one or more processors, the instructions, when executed, causing the system to perform operations; and wherein the operation comprises: receiving sensor data of an environment captured by a sensor of the vehicle; generating a multi-channel image representing a top-down view of the environment based at least in part on the sensor data, the multi-channel image comprising a first channel associated with a location of an object in the environment and a second channel associated with a velocity of the object; determining a heat map associated with predicted probabilities of possible locations associated with the object based at least in part on the multi-channel image; determining a predicted trajectory associated with the object based at least in part on the heat map; determining a trajectory for the vehicle to travel within the environment based at least in part on the predicted trajectory; A system comprising:

2. 2. The system of claim 1, The step of determining a predicted trajectory comprises: determining a predicted point for the location based at least in part on a highest probability portion associated with a portion of the heat map and a cost associated with determining the predicted trajectory based at least in part on the predicted point. A system comprising:

3. 3. The system of claim 2, the predicted point is a first predicted point; the most probable portion is a first most probable portion; the first most probable portion is a first portion; the cost is a first cost, The predicted trajectory is a first predicted trajectory, The operation includes: masking a second portion of the heatmap associated with the first predicted trajectory as a masked heatmap, the second portion of the heatmap comprising at least the first portion; determining a second most probable portion associated with a third portion of the masked heatmap as a second predicted point for the location; determining a second predicted trajectory based at least in part on the second predicted points and a second cost associated with determining the second predicted trajectory based at least in part on the second predicted points; The system further comprises:

4. A system according to any one of claims 2 to 3, The system, wherein the operations further comprise inputting the first channel and the second channel into a machine learning model trained to output the heatmap associated with the prediction probability based on the first channel and the second channel in response to an image input.

5. 4. The system of claim 1, wherein the operations further comprise inputting the multi-channel image into a machine learning model to determine the heat map; The machine learning model is a model trained to output the heat map associated with the prediction probability in response to an image input, Convolutional Neural Networks (CNNs) a long short-term memory (LSTM) component coupled to the CNN; A system comprising:

6. 2. The system of claim 1, wherein the action comprises: determining the predicted trajectory based at least in part further on a cost associated with determining the predicted trajectory and reducing a probability of selecting the predicted trajectory that corresponds to irrational dynamics of the object; The system further comprises:

7. 1. A method comprising: receiving sensor data of an environment; generating a multi-channel image representing a top-down view of an object in the environment, the multi-channel image comprising a first channel associated with a location of the object and a second channel associated with a velocity of the object; inputting the multi-channel image into an algorithm configured to generate a probability distribution associated with a predicted location of the object; determining a predicted trajectory associated with the object based at least in part on the probability distribution; controlling a vehicle based at least in part on the predicted trajectory; A method for providing the above.

8. 8. The method of claim 7, wherein the probability distribution is represented as a heat map that discretizes a portion of the environment; The method, wherein the predicted probability of the probability distribution is associated with a portion of the heat map.

9. 9. The method of claim 8, wherein the step of determining a predicted trajectory comprises: determining a predicted point for the location based at least in part on a highest probability portion of the heat map and a cost associated with determining the predicted trajectory based at least in part on the predicted point. A method for providing the above.

10. 10. The method of claim 9, the predicted point is a first predicted point; the most probable portion is a first most probable portion; the first most probable portion is a first portion; the cost is a first cost, The predicted trajectory is a first predicted trajectory, The method comprises: masking a second portion of the heatmap associated with the first predicted trajectory as a masked heatmap, the second portion of the heatmap comprising at least the first portion; determining a second predicted point for the location based at least in part on a second most probable portion of the heat map and a second cost associated with determining a second predicted trajectory based at least in part on the second predicted point; A method for providing the above.

11. 11. The method according to claim 7, wherein the multi-channel image is: the scope of said object, A bounding box associated with the object; acceleration information associated with the object; A turn signal status of the object; map data associated with said environment; or Traffic signal information associated with said environment The method further comprising a channel representing at least one of:

12. 11. The method of claim 7, wherein the algorithm is a machine learning model comprising a convolutional neural network trained to generate a heat map representing the probability distribution based on the multi-channel image in response to an image input, the machine learning model comprising: The multi-channel image is A classification of the object; or A bounding box associated with the object The method further comprising a channel representing at least one of:

13. 13. The method of claim 7, wherein the vehicle is an autonomous vehicle, and controlling the autonomous vehicle based at least in part on a predicted trajectory comprises: generating a trajectory for the autonomous vehicle to follow to traverse the environment; controlling the autonomous vehicle based at least in part on the trajectory; A method for providing the above.

14. 14. The method according to claim 7, wherein the step of capturing sensor data of the environment using sensors mounted on the vehicle comprises: LIDAR sensor, RADAR sensor, Image sensor, or Time-of-flight sensor, The method comprises one or more of the following:

15. 15. The method according to any one of claims 7 to 14, comprising: the sensor data represents the environment at a first time; the probability distribution being a first probability distribution associated with a predicted probability of a possible location associated with the object at a second time after the first time; the predicted location is a first predicted location; the algorithm is further configured to generate a second probability distribution associated with a predicted probability of a possible location associated with the object at a third time after the second time; The step of determining a predicted trajectory comprises: determining a second predicted trajectory associated with the object at the third time based at least in part on the first probability distribution and the second probability distribution; The method, wherein controlling the vehicle includes controlling the vehicle based at least in part on the second predicted trajectory.

16. 8. The method of claim 7, determining the predicted trajectory based at least in part further on a cost associated with determining the predicted trajectory and reducing a probability of selecting the predicted trajectory that corresponds to irrational dynamics of the object; The method further comprises:

17. A computer program comprising coded instructions which, when executed on a computer, performs the method according to any one of claims 7 to 16.

Images