Behavioral prediction using scene-centric representations

Abstract

A method performed by one or more computers, the method comprising: obtaining scene context data characterizing a scene in an environment at a current point in time, where the scene context data comprises features of the scene in a scene-centered coordinate system; generating a scene-centered encoded representation of the scene in the environment by processing the scene context data using an encoder neural network; for each target agent: obtaining agent-specific features of the target agent, processing the agent-specific features of the target agent and the scene-centered encoded representation of the scene using a fusion neural network to generate a fused scene representation of the target agent, and processing the fused scene representation of the target agent using a decoder neural network to generate a fused scene representation of the target agent. A trajectory prediction output is generated for the target agent in the agent-centered coordinate system of the target agent.

Description

Technical Field

[0001] This specification is concerned with predicting the future trajectory of an agent in an environment. Background Art

[0002] The environment may be a real-world environment, and the agent may be, for example, a vehicle, a cyclist, a pedestrian, or other vehicle in the environment. Predicting the future trajectory of an agent is a task required, for example, for motion planning by an autonomous vehicle.

[0003] Autonomous vehicles include self-driving cars, ships, and aircraft. Autonomous vehicles use various onboard sensors and computer systems to detect nearby objects and use such detections to make control and navigation decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a diagram of an example system.

[0005] Figure 2 The trajectory prediction model is shown.

[0006] Figure 3 Show Figure 2 A detailed version of the trajectory prediction model.

[0007] Figure 4 is a flow chart of an example process for trajectory prediction.

[0008] Figure 5 is a flowchart of an example process for training a trajectory prediction model.

[0009] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0010] This specification describes a system implemented as a computer program on one or more computers in one or more locations that generates trajectory predictions for one or more target agents in an environment, such as vehicles, cyclists, or pedestrians. Each trajectory prediction is a prediction that defines a future trajectory of the corresponding target agent starting from a current point in time.

[0011] Predicting the future behavior of road users is a challenging and important problem for autonomous driving and control of other types of autonomous vehicles. Furthermore, when trajectory predictions need to be generated on board a vehicle, there are strict latency requirements for generating trajectory predictions so that they can be used to make timely control decisions for the autonomous vehicle.

[0012] One approach for generating trajectory predictions is to use an agent-centric neural network. The agent-centric neural network processes corresponding agent-centric inputs for each agent in the scene. Each agent-centric input is calculated separately and is in an agent-centric coordinate system, where the corresponding agent is located at the origin of the coordinate system.

[0013] However, using agent-centric neural networks to make predictions is computationally expensive and can make it difficult to generate high-quality predictions within a latency budget. This is because, for agent-centric neural networks, the computational cost required to make predictions for all agents in a scene can potentially scale quadratically with the number of agents and scene elements in the scene. In particular, agent-centric networks require corresponding agent-centric inputs for each agent in the scene.

[0014] To address these issues, this specification describes a trajectory prediction model that includes a scene-centric encoder neural network, a fusion neural network, and an agent-centric decoder.

[0015] The encoder neural network is described as "scene-centric" because all agents in the environment are encoded in a single input along with information about the environment. The encoder uses a predetermined point in the scene as the origin and encodes the environment information and the agents in the environment relative to the predetermined point to produce a scene-centric encoded representation.

[0016] The fusion neural network can process the scene-centric encoded representation of the scene and the specific features of the target agent to generate a fused scene representation of the target agent.

[0017] The agent-centric decoder may process the fused scene representation with respect to the target agent to generate a trajectory prediction output with respect to the target agent in an agent-centric coordinate system with respect to the target agent.

[0018] Because the encoder is scene-centric, a single encoded representation is used for all agents, rather than requiring the encoder to process separate, agent-centric inputs for all agents. The fused neural network uses agent-specific features to refine the scene-centric encoded representation to generate a fused scene representation that includes context specific to the target agent, thereby improving the performance of the trajectory prediction model. By using the fused neural network, the trajectory prediction model does not need to transform the scene-centric input to an agent-centric coordinate system to use the agent-centric decoder. The fused model can use agent-specific features to refine the scene-centric encoded representation to generate appropriate inputs for the agent-centric decoder.

[0019] This can allow a trained trajectory prediction model to achieve performance that matches or exceeds that of a trajectory prediction neural network with an agent-centric encoder, while being much more computationally efficient and generating predictions with reduced latency. In particular, because the fused neural network is relatively computationally efficient compared to a scene-centric encoder, most of the processing to generate the input to the decoder only needs to be performed once (rather than requiring separate processing for each agent), significantly improving the computational efficiency of the process while maintaining or improving the accuracy of the predictions.

[0020] The described techniques allow an onboard trajectory prediction model to make predictions within the latency requirements of an autonomous vehicle, but with an accuracy that matches or exceeds the accuracy of an agent-centric model that cannot be deployed on-board within the latency budget by leveraging a fused neural network.

[0021] Figure 1 is a diagram of an example system 100 . System 100 includes an onboard system 110 and a training system 120 .

[0022] The vehicle-mounted system 120 is located on a vehicle 122 . Figure 1 The vehicle 122 in FIG. 1 is shown as an automobile, but the in-vehicle system 120 may be located on any suitable type of vehicle.

[0023] In some cases, the vehicle 122 is an autonomous vehicle. An autonomous vehicle can be a fully autonomous vehicle that determines and executes fully autonomous driving decisions in order to navigate through an environment. An autonomous vehicle can also be a semi-autonomous vehicle that uses predictions to assist a human driver. For example, if a prediction indicates that the human driver is about to collide with another vehicle, the vehicle 122 can autonomously apply the brakes. As another example, the vehicle 122 can have an advanced driver assistance system (ADAS) that assists the human driver of the vehicle 122 in driving the vehicle 122 by detecting potentially unsafe conditions and alerting the human driver or otherwise responding to the unsafe conditions. As a specific example, the vehicle 122 can alert the driver of the vehicle 122 or take autonomous driving actions when an obstacle is detected, when the vehicle deviates from the lane of travel, or when an object is detected in the blind spot of the human driver.

[0024] The vehicle system 120 includes one or more perception subsystems 132. The perception subsystem 132 can generate input data 155 representing a scene including one or more agents in the environment and the vehicle 122. The agent can be other road users, such as vehicles, cyclists, pedestrians, etc., within the range of one or more sensors of the vehicle 122.

[0025] The data representing the scene may include data representing objects in the scene, agent state history, and vehicle state history.

[0026] For any given object, the data characterizing the object in the scene may include one or more of the given object's object type, object attributes (e.g., a child, a special vehicle, and a turn signal), object characteristics (e.g., human activity, signs), object trajectory, object coordinates, object speed, object direction of travel, object curvature, and the like.

[0027] For all surrounding agents in the environment, the agent state history may include a current state at a current point in time and a previous state at one or more corresponding previous points in time. In other words, for all surrounding agents in the environment, the agent state history may include data characterizing the agent's previous trajectory in the environment up to the current point in time. The state of the agent at a point in time may include the position of the agent at the point in time, and optionally, the values ​​of a predetermined set of motion parameters at the point in time. As specific examples, the motion parameters may include the agent's direction of travel, the agent's velocity, and / or the agent's acceleration.

[0028] The vehicle state history may also include data characterizing a current state of the vehicle 122 at a current point in time and a previous state of the vehicle 122 at one or more corresponding previous points in time.

[0029] For example, the perception subsystem 132 may include a combination of sensor components that receive reflections of electromagnetic radiation, such as a lidar system that detects reflections of laser light, a radar system that detects reflections of radio waves, and a camera system that detects reflections of visible light, and may use measurements from those sensors to generate data characterizing a scene.

[0030] In addition, the perception subsystem 132 can obtain predetermined environmental information, such as information identifying lanes, traffic signs, crosswalks, and other road features that can be found in a road map or an environmental map. In some embodiments, the vehicle system 120 can obtain navigation history information (e.g., trajectory and speed, direction of travel, etc.) of the vehicle 122, such as from a driving log of the vehicle, and can provide the navigation history information as part of the input data 155.

[0031] Using predetermined environmental information and information captured by sensors, the perception subsystem 132 may generate input data 155 that characterizes the scene.

[0032] For example, the perception subsystem 132 may use images or videos of the agent over a period of time captured by a camera sensor, point cloud data of the agent captured by a lidar sensor, road information (e.g., lanes and stop signs), the positions of surrounding objects (e.g., other vehicles and pedestrians), etc. to generate data representing a scene 155 including the agent in the environment.

[0033] The perception system 132 provides input data 155 to the on-board machine learning subsystem 134 .

[0034] The onboard machine learning subsystem 134 implements the operation of the trajectory prediction model 102. The trajectory prediction model 102 may include an encoder model, a fusion model, and a decoder model. Figure 2 and Figure 3 The trajectory prediction model 102 is described in more detail.

[0035] Trajectory prediction model 102 may process input data 155 to generate, for each of one or more of the surrounding agents, a corresponding trajectory prediction output 166. Trajectory prediction output 166 for a given agent characterizes the future trajectory of the agent after a current point in time.

[0036] More specifically, the trajectory prediction output 166 for a given agent represents a likelihood distribution over possible future trajectories that the agent may follow, eg, a probability distribution or another distribution specifying a respective likelihood score for each of a set of possible future trajectories.

[0037] Each possible future trajectory includes data specifying a sequence of a plurality of waypoint spatial positions in the environment, each waypoint spatial position corresponding to a possible position of the agent at a corresponding future time point after the future time point.

[0038] As a specific example, trajectory prediction output 166 may include data defining a plurality of possible future trajectories and a corresponding likelihood score for each of the plurality of possible future trajectories representing the likelihood that the possible future trajectory will most closely resemble an actual trajectory followed by the surrounding agent.

[0039] For each future trajectory, the data defining the future trajectory may be a corresponding waypoint position at each future time point in a fixed set of future time points, which represents the position of the agent at the corresponding time point.

[0040] Alternatively, the data defining the future trajectory may be parameters of a probability distribution of a first possible future trajectory. That is, the data may be parameters of a corresponding parameter probability distribution for each waypoint spatial position in the possible future trajectory, i.e., a corresponding distribution over the spatial position of each time point in the trajectory. As a specific example, the parameter probability distribution for a given waypoint spatial position may be a normal probability distribution of the spatial position, and the data defining the parameters of the normal probability distribution may include (i) a mean of the normal probability distribution, and (ii) a covariance parameter of the normal probability distribution.

[0041] The onboard system 120 also includes a planning subsystem 136. The planning subsystem 136 may make autonomous or semi-autonomous driving decisions for the vehicle 122, for example, by generating a planned vehicle path that represents a path that the vehicle 122 will take in the future.

[0042] The onboard system 120 may provide the trajectory prediction output 166 generated by the trajectory prediction model 102 to one or more other onboard systems of the vehicle 122 , such as the planning subsystem 136 .

[0043] When the planning subsystem 136 receives the trajectory prediction output 166, the planning system 136 can use the trajectory prediction output 166 to generate a planning decision for the future trajectory of the planned vehicle, that is, to generate a new planned vehicle path. For example, the trajectory prediction output 166 may include a prediction that a certain surrounding agent may cut in front of the vehicle 122 at a certain future time point, potentially causing a collision. In this example, the planning system 136 can generate a new planned vehicle path that avoids the potential collision, and, for example, by autonomously controlling the steering of the vehicle, cause the vehicle 122 to follow the new planned path and avoid the potential collision.

[0044] The onboard machine learning subsystem 134 may also use the input data 155 to generate training data 108. The training data 108 may be used to train the trajectory prediction model 102. The onboard system 120 may provide the training data 108 to the training system 110 in offline batches or in an online manner, e.g., continuously whenever the training data 108 is generated.

[0045] Training system 110 is typically hosted within a data center 112, which may be a distributed computing system having hundreds or thousands of computers in one or more locations.

[0046] The training system 110 includes a machine learning training subsystem 114 that can implement the operation of the trajectory prediction model 102, which is configured to generate trajectory predictions for each of one or more agents in the scene. The machine learning training subsystem 114 includes a plurality of computing devices having software or hardware modules that implement corresponding operations of the machine learning model, for example, corresponding operations of each layer of a neural network according to the architecture of the neural network.

[0047] The training trajectory prediction model typically has the same architecture and parameters as the onboard trajectory prediction model 102. However, the training system 110 does not need to use the same hardware to compute the operations of the trajectory prediction model 102. In other words, the training system 110 may use only CPUs, highly parallelized hardware, or some combination of these.

[0048] The machine learning training subsystem 114 can calculate the operation of the trajectory prediction model, for example, the operation of each layer of the neural network, using the current parameter values ​​115 stored in the set of model parameter values ​​170. Although shown as logically separated, the model parameter values ​​170 and the software or hardware modules that perform the operations can actually be located on the same computing device or on the same storage device.

[0049] The machine learning training subsystem 114 may receive as input training examples 123. The training examples 123 may be labeled training data 125 stored in a database. Each training example includes an input representing a scene, including one or more agents and vehicles in an environment and one or more labels indicating a ground truth trajectory of the agent.

[0050] The machine learning training subsystem 114 may generate an error prediction 135 for each training example 123. Each error prediction 135 represents an estimate of the error between a ground truth trajectory of one or more of the plurality of agents in the scene and a trajectory of one or more of the plurality of agents generated by the trajectory prediction model 102 being trained. The training engine 116 analyzes the error predictions 135 and compares the error predictions to the labels in the training examples 123 using a loss function, such as a classification loss or a regression loss function. The training engine 116 then generates updated model parameter values ​​145 by using an appropriate update technique, such as stochastic gradient descent with back-propagation. The training engine 116 may then use the updated model parameter values ​​145 to update the set of model parameter values ​​170.

[0051] After training is completed, the training system 110 can provide the final model parameter value set 171 to the vehicle system 120 for making fully autonomous or semi-autonomous driving decisions. For example, the training system 110 can provide the final model parameter value set 171 to the trajectory prediction model 102 running in the vehicle system 120 to generate trajectory predictions 166 for one or more of the multiple agents in the scene. The training system 110 can provide the final model parameter value set 171 to the vehicle system 120 via a wired or wireless connection.

[0052] Reference below Figure 5 Training trajectory prediction model 102 is described in more detail.

[0053] Figure 2 An example trajectory prediction model 200 is shown.

[0054] like Figure 2 As shown, the trajectory prediction model 200 includes an encoder model 204 , a fusion model 210 , and a decoder model 214 .

[0055] The encoder model 204 is described as "scene-centric" because all agents in the environment are encoded within a single input along with information about the environment. The encoder model 204 uses a shared point in the scene as an origin and encodes the environment information and agents in the environment relative to a predetermined point to generate a scene-centric encoded representation 206. For example, the shared point may be the location of the autonomous vehicle at the current time.

[0056] The fusion model 210 may process the scene-centric encoded representation 206 of the scene and specific features of the target agent to generate a fused scene representation of the target agent.

[0057] The agent-centric decoder 214 may process the fused scene representation of the target agent to generate a trajectory prediction output for the target agent in an agent-centric coordinate system for the target agent.

[0058] Because the encoder model 204 is scene-centric, a single encoded representation is used for all agents, rather than requiring the encoder model to process separate, agent-centric inputs for all agents.

[0059] The fusion model 210 refines the scene-centric encoded representation 206 using the agent-specific features 208 to generate a fused scene representation 212 that includes context specific to the target agent, thereby improving the performance of the trajectory prediction model 200 .

[0060] This may allow the trajectory prediction model 200 to achieve performance matching or exceeding that of a trajectory prediction neural network with an agent-centric encoder, while being much more computationally efficient and generating predictions with reduced latency.

[0061] Optionally, the fusion model 210 and the decoder model 214 may batch their processing and generate trajectory predictions for all agents in parallel.

[0062] In more detail, the encoder model 204 receives the scene context data 202 and generates a scene-centric encoded representation 206 of the scene.

[0063] The fusion model 210 receives the scene-centric encoded representation 206 of the scene and corresponding agent-specific features 208 for each of a plurality of target agents, and generates a corresponding fused scene representation 212 for each of the target agents from the scene-centric encoded representation 206 and the corresponding agent-specific features 208 for the target agent.

[0064] The decoder model 214 receives the fused scene representation 212 of the target agent for each target agent and generates a trajectory prediction output 216 for the target agent from the fused scene representation 212 of the target agent.

[0065] The scene context data 202 represents the scene in the environment at the current time step in a scene-centric coordinate system. The scene-centric coordinate system is a coordinate system centered at a fixed point in the scene, for example, at the location of the autonomous vehicle at the current time point. All features of all agents at all time points are in a shared scene-centric coordinate system. The scene can include multiple agents in the environment, such as vehicles, cyclists, pedestrians, etc.

[0066] The scene context data 202 may include data characterizing objects in the scene, context information about the environment, navigation history information of one or more agents in the scene, and target agent historical context data.

[0067] For any given object, the data characterizing the object in the scene may include one or more of the object type, object attributes (e.g., child, special vehicle, and turn signal), object coordinates, object speed, object heading, object curvature, etc. of the given object.

[0068] The context information of the environment may include information identifying lanes, traffic signs, crosswalks, and other road features that may be found in a road map or map of the environment. The context information may include sensor data characterizing the scene. For example, the scene context data 202 may include data generated from data captured by one or more sensors of the autonomous vehicle, and the target agent may be an agent in the vicinity of the autonomous vehicle.

[0069] The navigation history may include a series of positions, velocities, and orientations of the autonomous vehicle in the environment at predetermined intervals over a period of time, for example, every 0.1 seconds over the last two seconds.

[0070] Target agent historical context data characterizing current states and previous states of multiple target agents may include: for all target agents in the environment, the current state at the current time point and the previous state at one or more corresponding previous time points. In other words, for the target agents in the environment, the target agent historical context data may include data characterizing the previous trajectory of the agent in the environment up to the current time point. The state of the agent at a time point may include the position of the agent at that time point, and optionally, the values ​​of a predetermined set of motion parameters at that time point. As a specific example, the motion parameters may include the direction of travel of the agent, the speed of the agent, and / or the acceleration of the agent.

[0071] The encoder model 204 can be any appropriate type of machine learning model, such as a neural network model or another type of machine learning model. For example, the encoder model 204 can be a self-attention encoder, which can encode one or more modalities across time and space dimensions. Driving scenes can include multimodal data, such as road information, traffic light status, agent history, and agent interaction. Some modalities can have both spatial dimensions (i.e., data at different locations within the scene) and temporal dimensions (i.e., data at different time points). The encoder model 204 may include one or more attention encoders that summarize the driving scene. One or more attention encoders may include a multi-axis attention encoder and / or a factorized attention encoder. The encoder model 204 may include both spatial encoder blocks and temporal encoder blocks that can encode data with different modalities.

[0072] The encoder model 204 may be used to generate a scene-centric encoded representation 206 of a scene represented by the scene context data 202. The scene-centric encoded representation 206 of the scene represents the scene context data 202 in a scene-centric coordinate system, such as centered on a predetermined point in the scene.

[0073] In some embodiments, the scene-centric encoded representation 206 of the scene includes a sequence of scene embeddings. Each scene embedding in the embedding sequence can be an ordered set of values, such as a vector, matrix, or other tensor of values.

[0074] The fusion model 210 can be any suitable type of machine learning model, such as a neural network model or another type of machine learning model. For example, the fusion model can be a late fusion encoder model with one or more cross-attention layers. The late fusion encoder model has dedicated attention encoders for each modality.

[0075] The fusion model 210 processes the scene-centric encoded representation 206 of the scene and the agent-specific features 208 of each target agent in the scene to generate a fused scene representation 212 of the target agent. The fusion model 210 can generate the fused scene representation 212 for all target agents in the scene. The fused scene representation 212 for a given agent is latent information that is not explicitly in a particular coordinate system, but provides contextual information that is specific to the given agent and can be used to make accurate trajectory predictions for the target agent. For example, the fused scene representation 212 can include a vector sequence that provides contextual information specific to the given agent.

[0076] The agent-specific features 208 for a given target agent may include agent historical context data characterizing a current state and previous states of the target agent in a context-centric coordinate system. The agent historical data for a given target agent may include a current state at a current point in time and previous states at one or more corresponding previous points in time. The state of the target agent at a point in time may include the position of the target agent at that point in time, and optionally, the values ​​of a set of motion parameters predetermined at that point in time. As specific examples, the motion parameters may include a direction of travel of the target agent, a velocity of the target agent, and / or an acceleration of the target agent.

[0077] In some implementations, the agent-specific features include features of the target agent and additional features of the target agent from the scene context data 202. For example, the scene context data 202 may include target agent history data characterizing the current state and previous states of the target agent, and the agent-specific features 208 may include the target agent history data from the scene context data and the agent-specific features of each target agent.

[0078] In some implementations, agent-specific features 208 for a given target agent include a combination of features in a context-centric coordinate system and features in an agent-centric coordinate system. The agent-centric coordinate system for a target agent is a coordinate system that is centered on the target agent.

[0079] Features in an agent-specific coordinate system for a given target agent may include any suitable features expressed in an agent-centric coordinate system.

[0080] For example, features may include a state history of a target agent expressed in an agent-centric coordinate system of the target agent, a current state or state history of one or more other agents expressed in an agent-centric coordinate system of the target agent, or both.

[0081] As another example, features may also include features representing road map information, such as road information or other landmarks in a coordinate system centered on the agent.

[0082] As another example, a feature may include one or more "anchor" future trajectories for a given agent, expressed in an agent-centric coordinate system. An anchor future trajectory is an example future trajectory that may be traversed by a given agent. For example, these anchor future trajectories may be predetermined based on the type of the given agent, or may be predicted by another more computationally efficient prediction model. In some cases, the system may include only anchor trajectories for certain agent types, while excluding anchors for other agent types.

[0083] In some implementations, the fusion model 210 can generate a sequence of agent embeddings from agent-specific features.

[0084] Then, the fusion model 210 can process the agent embedding sequence and the encoding representation to generate a fused scene representation. For example, the scene-centric encoding representation of the scene can be a sequence of scene embeddings, and the fusion neural network can have at least one cross-attention neural network block that performs cross-attention between the sequence of scene embeddings and the sequence of agent embeddings.

[0085] The decoder model 214 can be any suitable type of machine learning model, such as a neural network model or another type of machine learning model. For example, the decoder model 214 can be a neural network including a stack of one or more standard transformer cross-attention blocks.

[0086] The decoder model 214 may be used to generate trajectory prediction output 216 for each target agent in the scene from the fused scene representation of the target agent.

[0087] The trajectory prediction output 216 for a given target agent represents a probability distribution over possible future trajectories that the agent may follow, e.g., a probability distribution or another distribution that specifies a respective likelihood score for each of a set of possible future trajectories. The trajectory prediction output 216 for a given target agent is in an agent-centered coordinate system centered at the location of the target agent at the current time.

[0088] Each possible future trajectory includes data specifying a sequence of a plurality of waypoint spatial positions in the environment, each waypoint spatial position corresponding to a possible position of the target agent at a corresponding future time point after the future time point.

[0089] As a specific example, trajectory prediction output 216 may include data defining a plurality of possible future trajectories and a corresponding likelihood score for each of the plurality of possible future trajectories that represents the likelihood that the possible future trajectory will be closest to the actual trajectory followed by the target agent.

[0090] For each future trajectory, the data defining the future trajectory may be a corresponding waypoint position at each future time point in a fixed set of future time points, which represents the position of the target agent at the corresponding time point.

[0091] Alternatively, the data defining the future trajectory may be parameters of a probability distribution about a first possible future trajectory. That is, the data may be parameters of a corresponding parameter probability distribution about the spatial position of each waypoint in the possible future trajectory, i.e., a corresponding distribution about the spatial position of each time point in the trajectory. As a specific example, the parameter probability distribution of the spatial position of a given waypoint may be a normal probability distribution of the spatial position, and the data defining the parameters of the normal probability distribution may include (i) the mean of the normal probability distribution, and (ii) the covariance parameters of the normal probability distribution.

[0092] Figure 3 Show Figure 2 Detailed example of trajectory prediction model 200.

[0093] More specifically, Figure 3 For three future time steps T 1 、T 2 and T 3 An example of trajectory prediction model 200 in an example of making predictions.

[0094] exist Figure 3 In the example of , B refers to the batch size, which specifies the number of scenes for which predictions are made in parallel; T refers to the number of time steps for which scene context data is available; Nc refers to the number of context agents in a given scene, i.e., the number of context agents represented by the scene context data, and Nm refers to the number of “modeled” agents, i.e., the number of target agents for which predictions are made.

[0095] The encoder model 204 receives scene context data representing the scene in a context-centric coordinate system. As part of the scene context data, the encoder model 204 may receive road map information 306, traffic light information 304, motion history (i.e., agent state history) 302. The encoder model 204 may use corresponding linear projections 308, 310, and 312 for each type of data 302, 304, and 306 to map the data to a shared dimension.

[0096] For the road map information 306, the road map encoder may use polylines to encode road elements from the 3D high definition map using MLP to generate Vr features, ie, one Vr feature per polyline with a size of Fr.

[0097] For the traffic light information 304 , the traffic signal encoder may use a recurrent neural network, such as LSTM, to encode the traffic light information to generate Vt features of size Ft, ie, one Vt feature for each traffic light.

[0098] For motion history and agent interactions 302, the motion encoder can use a transformer neural network to encode the sequence of past observations of each agent. For example, the system can generate corresponding features of size Fc for each contextual agent at each time step, and generate features of size Fs for the autonomous vehicle at each time step.

[0099] The encoder model 204 can process the combined sequence of embeddings of each type of scene context data through multiple self-attention blocks to generate a scene-centric encoded representation 332 of the scene in a scene-centric coordinate system, which scene-centric encoded representation 332 includes a sequence of scene embeddings, i.e., a sequence of V scene embeddings of size F.

[0100] For each of the target agents, the fusion model 210 can process the scene-centric encoded representation 332 and the agent-specific state history 314 of the target agent in a scene-centric coordinate system. The one or more target agents are a proper subset of all agents 302 in the scene for which predictions are to be made. The fusion model can include an agent history encoder 316 to encode the agent-specific state history 314 of the target agent in the sequence of agent embeddings 334.

[0101] In some examples, the fusion model 210 can have at least one crisscross attention neural network block that performs crisscross attention between a sequence of scene embeddings 332 and a sequence of agent embeddings 334 to generate a fused representation of each target agent. The sequence of scene embeddings 332 is used to generate a key / value, and the sequence of agent embeddings 334 is used to generate a query for the crisscross attention. In some examples, the fusion model 210 can receive road map information in an agent-centric coordinate system for each target agent. The fusion model 210 can include an additional encoder that encodes the agent-centric road map information.

[0102] For example, the fusion model 210 may include only one or more cross-attention blocks, or may include one or more self-attention blocks that apply self-attention on a sequence of agent embeddings 334 in addition to the cross-attention blocks. However, in general, the fusion model 210 is significantly more computationally efficient than the encoder 204. More specifically, the computational resources required to encode the agent-specific features 314 and perform the operations of the fusion model 210 are significantly less than the computational resources required to encode the scene context data and perform the operations of the encoder 204. Therefore, it is computationally significantly more efficient to encode only the agent-specific features 314 and perform the operations of the fusion model 210 for each scene than to encode the scene context data separately for each target agent, i.e., represented in the agent-centric coordinate system of the agent.

[0103] The decoder model 214 can then process the fused representation of the target agent using a decoder to generate trajectory predictions 326, 328, and 330. The decoder can be, for example, a multi-layer perceptron (MLP), a transformer, or a recurrent neural network (RNN). For example, the decoder can include a stack of one or more transformer cross-attention blocks, where the initial query is fed in and then cross-attended with the scene embedding 332 to produce a trajectory. The decoder can also include one or more self-attention blocks that self-attention the initial query. Typically, the initial query includes a set of learning queries 320 that are learned during training of the decoder model 214. In some embodiments, the initial query also includes a set of "anchor" queries 318 that are not learned but represent pre-defined or predicted "anchor" future trajectories as described above. As described above, in some cases, the system only uses anchor trajectories for certain types of agents.

[0104] The decoder model 214 may use a respective linear projection 322 and 324 on each initial query 318 and 320 to map the learned initial queries to a shared dimension.

[0105] The decoder model 214 can output a Gaussian mixture to represent possible trajectories that the agent can take. For example, the decoder model 214 can process the updated query using the output neural network head to generate parameters of the Gaussian mixture for a given agent. For example, the system can output, for each future time step and for each mode of the mixture, the probability of the mode, the mean of the mode at the future time step, and optionally, parameters defining the variance of the mode at the future time step.

[0106] In order to generate corresponding trajectory prediction outputs 326, 328, and 330 for each target agent using the decoder model 214, the fusion model 210 needs to generate a fused scene representation for the target agent. The system then processes the fused scene representation to generate corresponding trajectory prediction outputs 326, 328, and 330 for each of the target agents.

[0107] Figure 4 is a flow chart of an example process 400 for trajectory prediction. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a trajectory prediction model (e.g., appropriately programmed according to the present specification) Figure 1 The vehicle-mounted system 120) may perform process 400.

[0108] The system obtains scene context data representing a scene in the environment and a set of target agents in the environment (step 402). The scene context data represents the scene in the environment at the current time step in a coordinate system centered on the scene. The target agents can be other road users, such as vehicles, cyclists, pedestrians, etc.

[0109] Scene context data may include context information of the environment (e.g., information identifying lanes, traffic signs, crosswalks, and other road features that may be found in a road map or an environment map). The context information may include sensor data that characterizes the scene. For example, the scene context data may include data generated from data captured by one or more sensors of an autonomous vehicle, and the target agent may be an agent in the vicinity of the autonomous vehicle.

[0110] The scene context data may include target agent historical context data characterizing the current state and previous states of a set of target agents. For example, for all target agents in the environment, the scene context data 202 may include the current state at the current time point and the previous state at one or more corresponding previous time points.

[0111] The system processes the scene context data to generate a scene-centric encoded representation of the scene represented by the scene context data using an encoder model (step 404).

[0112] In some embodiments, the scene-centric encoded representation of the scene includes a sequence of scene embeddings. The embedding sequence can be an ordered set of values, such as a vector, matrix, or other tensor of values.

[0113] For each target agent in the set of target agents, the system obtains agent-specific characteristics of the target agent (step 406). The agent-specific characteristics of a given target agent may include agent historical context data characterizing a current state and a previous state of the target agent in a coordinate system centered on the context. The agent historical data for a given target agent may include a current state at a current point in time and a previous state at one or more corresponding previous points in time. The state of the target agent at a point in time may include the position of the target agent at that point in time, and optionally, the values ​​of a predetermined set of motion parameters at that point in time. As specific examples, the motion parameters may include a direction of travel of the target agent, a velocity of the target agent, and / or an acceleration of the target agent.

[0114] In some implementations, the agent-specific features include features of the target agent and additional features of the target agent from the scene context data 202. For example, the scene context data 202 may include target agent history data characterizing the current state and previous states of the target agent, and the agent-specific features 208 may include the target agent history data from the scene context data and the agent-specific features of each target agent.

[0115] In some implementations, the agent-specific features for a given target agent include a combination of features in a context-centric coordinate system and features in an agent-centric coordinate system.The features in the agent-specific coordinate system may include a state history for each of the target agents.

[0116] The system processes the agent-specific features of the target agent and the scene-centric encoded representation of the scene using the fusion model to generate a fused scene representation of the target agent (step 408).

[0117] The system processes the fused scene representation of the target agent to generate a trajectory prediction output for the target agent (step 410). The trajectory prediction output for a given target agent may include a future trajectory of the target agent after a current time point in the target agent's agent-centric coordinate system. The trajectory prediction output may define a probability distribution of possible future trajectories of the target agent after the current time point. For example, the trajectory prediction output may include a corresponding likelihood score for each of a plurality of future trajectories and, optionally, data defining each of the plurality of future trajectories. For example, the plurality of future trajectories may be a pattern of a Gaussian mixture as described above.

[0118] Figure 5 is a flow chart of an example process 500 for training a trajectory prediction system. For convenience, process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., Figure 1 The training system 110) can perform process 500.

[0119] At a high level, to perform process 500, the system obtains a mini-batch of training examples and updates the current values ​​of the model parameters using the mini-batch.

[0120] By repeatedly updating the model parameter values ​​in this way for different mini-batches of training examples, the system can train the trajectory prediction neural network to generate accurate trajectory prediction outputs.

[0121] The system obtains a batch of training examples (step 502). Each training example includes scene data representing a scene that may include one or more agents at a first point in time. The training example also includes ground truth agent trajectories for all or some of the one or more agents in the scene, the ground truth agent trajectories identifying actual trajectories followed by the agents after the first point in time.

[0122] The system samples a fixed number of agents from the batch (step 504). For example, a batch may have 500 training examples, where a portion of the training examples have few agents in the corresponding scenes, i.e., corresponding to sparse scenes sparsely populated with agents, and other training examples have many agents, i.e., corresponding to crowded scenes densely populated with agents. Instead of training on many examples with few agents, the system samples a fixed number of agents across all scenes in a batch of training examples.

[0123] In some examples, the system may select a fixed number of agents from the scenes in the batch, either randomly or based on each agent's priority score. The fixed number of agents may be the number of agents that most scenes in the batch are likely to contain.

[0124] In some examples, the system can select the number of agents to sample from a particular training example based on the crowdedness of the scene as represented by its scene data. The number of agents sampled from the scene can be higher (i.e., 15) for dense scenes and lower (i.e., 2) for sparse scenes.

[0125] In some examples, the system may classify agents as vehicle agents or non-vehicle agents. The system may select a fixed number of vehicle agents to sample and a fixed number of non-vehicle agents to sample.

[0126] The system jointly trains the encoder neural network, the fusion neural network, and the decoder neural network to minimize the loss function using the ground truth trajectory of the agent sampled from the batch of training examples (step 506).

[0127] For example, the system can determine the gradients of the model parameters with respect to the loss function. The system can calculate the gradients, for example, by backpropagation. The system can then use the gradients to update the model parameters, for example, by applying an optimizer to the gradients with respect to the model parameters.

[0128] The loss function may include one or more terms that, for each training example and for each of a corresponding plurality of agents in a corresponding scene represented by the training example, measure the difference between (i) the agent's ground truth trajectory and (ii) the trajectory among the trajectory prediction outputs generated by the decoder neural network for the agent that is closest to the ground truth trajectory.

[0129] For example, when the trajectory prediction is a Gaussian mixture, the loss can have: a classification loss term that measures the log probability assigned by the trajectory closest to the ground truth trajectory in the trajectory prediction output; and a regression loss term that measures the log probability assigned to the ground truth trajectory by the probability distribution that defines the closest trajectory in the trajectory prediction output.

[0130] Although the above description describes trajectory predictions made on a vehicle after training, as another example, trajectory predictions can be made in a computer simulation of a real-world environment navigated by a simulated autonomous vehicle and a target agent. Generating these predictions in simulation can help control the simulated vehicle and help test the realism of certain situations encountered in the simulation. More generally, generating these predictions in simulation can be part of testing control software for a real-world autonomous vehicle before deploying the software on the autonomous vehicle, part of training one or more machine learning models that will later be deployed on the autonomous vehicle, or both.

[0131] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing device or for controlling the operation of a data processing device. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or additionally, program instructions may be encoded on an artificially generated propagation signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0132] The term "data processing device" refers to data processing hardware and covers all kinds of devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The device may also be or further include an off-the-shelf or customized parallel processing subsystem, such as a GPU or another special processing subsystem. The device may also be or further include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may optionally include code that creates an execution environment for a computer program, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0133] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0134] For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0135] As used in this specification, "engine" or "software engine" refers to a software-implemented input / output system that provides an output different from the input. An engine can be a coding block of a function, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine can be implemented on any suitable type of computing device including one or more processors and a computer-readable medium, such as a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smart phone, or other fixed or portable device. In addition, two or more of the engines can be implemented on the same computing device or on different computing devices.

[0136] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit (e.g., an FPGA or ASIC) or a combination of a special purpose logic circuit and one or more programmed computers.

[0137] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing or running instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated into the special logic circuit. Typically, the computer will also include one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from it or transmit data to it or both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0138] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CDROM and DVD-ROM disks.

[0139] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse, trackball, or presence-sensitive display or other surface through which the user can provide input to the computer). Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device (e.g., a smart phone) running a messaging application and receiving a response message from the user in return.

[0140] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0141] The computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input therefrom. Data generated at the user device, for example, the result of a user interaction, may be received from the device at the server.

[0142] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be cut out of the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0143] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequence, or that all of the operations shown be performed, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0144] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: Obtaining scene context data representing a scene in the environment at a current point in time, wherein the scene includes a set of agents, the set of agents includes a plurality of target agents, and wherein the scene context data includes features of the scene in a coordinate system centered on the scene; Generate a scene-centric encoded representation of the scene in the environment by processing the scene context data using an encoder neural network; For each target proxy: obtaining agent-specific characteristics of a target agent; processing the agent-specific features of the target agent and the scene-centric encoded representation of the scene using a fused neural network to generate a fused scene representation of the target agent; and The fused scene representation of the target agent is processed using a decoder neural network to generate a trajectory prediction output for the target agent, which predicts a future trajectory of the target agent after a current time point in an agent-centered coordinate system of the target agent.

2. The method according to claim 1, wherein: The scene-centric encoded representation of the scene in the environment includes a sequence of scene embeddings.

3. The method according to claim 2, further comprising: Generate a sequence of agent embeddings from agent-specific features using a fused neural network.

4. The method according to claim 3, wherein: The fusion neural network includes at least one criss-cross attention neural network block that performs criss-cross attention between the sequence of scene embeddings and the sequence of agent embeddings.

5. A method according to any one of the preceding claims, wherein: The agent-specific features of the target agent include agent history context data characterizing the current state and previous states of the target agent in a context-centric coordinate system.

6. A method according to any one of the preceding claims, wherein: The trajectory prediction output defines a probability distribution over possible future trajectories of the target agent after the current time point.

7. A method according to any one of the preceding claims, wherein: The scene context data includes data generated from data captured by one or more sensors of the autonomous vehicle, and The multiple target agents are agents near the autonomous vehicle in the environment.

8. The method according to claim 7, further comprising: (i) trajectory prediction outputs of the plurality of target agents, (ii) data derived from the trajectory prediction outputs, or (iii) both are provided to an onboard system of the autonomous vehicle for use in controlling the autonomous vehicle.

9. The method according to claim 8, wherein: Generate trajectory prediction output onboard the autonomous vehicle.

10. A method according to any one of the preceding claims, wherein: The context data includes data generated from data that simulates data that would be captured by one or more sensors of the autonomous vehicle in a real-world environment, and The plurality of target agents are agents that are in the vicinity of a simulated autonomous vehicle in a computer simulation.

11. The method according to claim 10, further comprising: (i) trajectory prediction output, (ii) data derived from the trajectory prediction output, or (iii) both are provided for use in controlling a simulated autonomous vehicle in a computer simulation.

12. A method according to any one of the preceding claims, wherein: The scene context data includes target agent historical context data characterizing current states and previous states of the plurality of target agents.

13. The method according to claim 12, wherein: The agent-specific features are a subset of target agent history data that characterizes current and previous states of the plurality of target agents.

14. A method according to any one of the preceding claims, wherein: The scene context data includes road map context data representing road features in the scene.

15. A method according to any one of the preceding claims, wherein: The scene context data includes traffic signal context data characterizing at least respective current states of one or more traffic signals in the scene.

16. A method according to any one of the preceding claims, wherein: The agent-specific features of the target agent include a combination of features in a context-centric coordinate system and features in an agent-centric coordinate system.

17. A system comprising: one or more computers; as well as One or more storage devices communicatively coupled to one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-16.

18. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-16.

Images