Method, system, and computer-readable storage medium for predicting vehicle intention

Abstract

Methods, systems, and apparatus for generating vehicle intent predictions using a neural network, including a computer program encoded on a computer storage medium. One of the methods includes: obtaining input representing one or more vehicles in an environment; generating features for each of the vehicles from the input; and, for each of the vehicles: processing the features of the vehicle using each of a plurality of intent-specific neural networks, wherein each of the intent-specific neural networks corresponds to a respective intent from a set of intents, and wherein each intent-specific neural network is configured to process the features of the vehicle to generate an output corresponding to the intent.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 16 / 723,787, filed on December 20, 2019. Background Art

[0003] This specification relates to autonomous vehicles.

[0004] Autonomous vehicles include self-driving cars, boats, and aircraft. They use various on-board sensors and computer systems to detect nearby objects and use these detections to predict their intentions and trajectories. Accurately and timely predictions of road users' intentions and trajectories are crucial for making control and navigation decisions.

[0005] Some autonomous vehicles have computer systems that implement neural networks for classifying objects within images. For example, a neural network can be used to determine that an image captured by an onboard camera is likely an image of a nearby car.

[0006] A neural network, or simply a network, is a machine learning model that uses multiple layers of operations to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers between the input layer and the output layer. The output of each layer serves as the input to another layer in the network (e.g., the next hidden layer or the output layer).

[0007] Each layer of a neural network specifies one or more transformation operations to be performed on the inputs of that layer. Some neural network layers have operations called neurons. Each neuron receives one or more inputs and generates an output that is received by another neural network layer. Typically, each neuron receives inputs from other neurons, and each neuron provides outputs to one or more other neurons.

[0008] The architecture of a neural network specifies what layers are included in the network, their properties, and how the neurons in each layer of the network are connected. In other words, the architecture specifies which layers provide their output as input to which other layers, and how the output is provided.

[0009] The transformation operation of each layer is performed by a computer that has installed a software module that implements the transformation operation. Therefore, when a layer is described as performing an operation, it means that the computer that implements the transformation operation of the layer performs the operation.

[0010] Each layer uses the current values ​​of that layer's parameter set to generate one or more outputs. Training the network therefore involves continuously performing forward passes on the input, computing gradients, and updating the current values ​​of each layer's parameter set. Once the neural network is trained, the resulting parameter set can be used to make predictions in a production system.

[0011] A convolutional neural network consists of convolutional neural network layers. A convolutional neural network layer has neuron connectivity that exploits spatially local correlations in the input data. To this end, convolutional neural network layers have sparse connectivity, where neurons in one convolutional layer receive input only from a small subset of neurons in the previous layer. The other neurons from which a neuron receives input define its receptive field.

[0012] A convolutional neural network layer has one or more parameters that define one or more filters of each layer, where each filter has one or more parameters. A convolutional neural network layer generates an output by performing a convolution of each neuron's filter with the layer's input.

[0013] Furthermore, each convolutional network layer can have neurons arranged in three dimensions, with depth, width, and height dimensions. The width and height dimensions correspond to the two-dimensional features of the layer's input. The depth dimension comprises one or more deep sublayers of neurons. Convolutional neural networks employ weight sharing, so that all neurons in a deep sublayer have the same weights. This provides translation invariance when detecting features in the input.

[0014] Convolutional neural networks can also include fully connected layers and other types of layers. Neurons in a fully connected layer receive input from every neuron in the previous neural network layer.

[0015] Autonomous and semi-autonomous vehicle systems can use full vehicle predictions to make driving decisions. A full vehicle prediction is a prediction about the region of space occupied by the vehicle. The predicted region of space can include spaces not observable by the set of onboard sensors used to make the prediction.

[0016] Autonomous vehicle systems can make full vehicle predictions using human-programmed logic that specifies precisely how the outputs of onboard sensors should be combined, transformed, and weighted to calculate the full vehicle prediction. Summary of the Invention

[0017] This specification describes systems and techniques for predicting vehicle intent using neural networks.

[0018] In a first aspect, a method is provided that includes obtaining input representing one or more vehicles in an environment; generating features of each of the vehicles from the input; and for each of the vehicles: processing the features of the vehicle using each of a plurality of intent-specific neural networks, wherein each of the intent-specific neural networks corresponds to a respective intent in a set of intents, and wherein each intent-specific neural network is configured to process the features of the vehicle to generate an output corresponding to the intent, the output comprising: (i) a confidence score representing a predicted likelihood that the vehicle will follow the intent corresponding to the intent-specific neural network, and (ii) a predicted trajectory that the vehicle will follow in a future time period if the vehicle follows the intent corresponding to the intent-specific neural network.

[0019] The first aspect may also optionally include one or more of the following features.

[0020] In some embodiments, the method also includes providing outputs from the plurality of intent-specific neural networks of the one or more vehicles to a planning system that plans navigation of the autonomous vehicle in the environment.

[0021] In some implementations, for each of the one or more vehicles, the input includes an appearance embedding of the vehicle that represents the appearance of the vehicle as sensed by one or more sensors of particular other vehicles in the environment.

[0022] In some embodiments, the input includes one or more images of an environment, and generating features for each of the vehicles includes: processing the one or more images of the environment using a convolutional neural network to generate a feature map, the feature map including a corresponding feature vector for each of a plurality of locations in the environment; and generating features for each of the vehicles based on the feature vectors in the feature map.

[0023] In some embodiments, generating features for each of the vehicles based on feature vectors in a feature map includes: obtaining a position of the vehicle in each of one or more images of the environment; obtaining a plurality of cropped feature vectors by cropping the feature map based on each position of the vehicle, including a corresponding feature vector for each position of the vehicle; and generating a trajectory feature map for the vehicle by performing an average pooling operation on the plurality of cropped feature vectors.

[0024] In some embodiments, generating features for each of the vehicles based on feature vectors in the feature map includes generating a contextual feature map for the vehicle by performing an average pooling operation on the feature vectors in the feature map.

[0025] In some embodiments, generating features for each of the vehicles based on the feature vectors in the feature map includes: applying a self-attention mechanism to the feature map to generate an attended feature map; and generating features based on the attended feature map.

[0026] In some implementations, for each of the one or more vehicles, the predicted trajectory includes a predicted position of the vehicle at each of a plurality of future time steps.

[0027] In some implementations, each intent-specific neural network is one or more fully connected neural network layers configured to generate an output corresponding to the intent from the features.

[0028] In some implementations, each intent-specific neural network includes: one or more fully connected neural network layers configured to generate confidence scores; and one or more autoregressive neural network layers configured to autoregressively generate predicted trajectories.

[0029] In some embodiments, one or more autoregressive neural network layers are recurrent neural network layers.

[0030] In some embodiments, for each of the one or more vehicles, the predicted trajectory includes a predicted position of the vehicle for each of a plurality of future time steps, wherein the plurality of future time steps are divided into a plurality of partitions, and wherein the autoregressive neural network layer is configured to generate, for each partition, a predicted position of the vehicle for that partition conditioned on predicted positions of earlier partitions.

[0031] In some implementations, each partition includes more than one future time step.

[0032] In some embodiments, each intent is a different possible long-term motion goal for the vehicle, and wherein each predicted trajectory is a sequence of predicted positions of the vehicle.

[0033] In some embodiments, the set of intentions includes one or more of: go straight, turn left, turn right, change lane left, change lane right, remain stationary, or reverse.

[0034] In a second aspect, a system is provided comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations including the operations of the method of the first aspect.

[0035] In a third aspect, one or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations including the operations of the method of the first aspect are provided.

[0036] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0037] Traditionally, autonomous vehicles have implemented onboard behavioral prediction (BP) systems to predict the intentions or trajectories of nearby vehicles. BP systems rely primarily on heuristic rules to make these predictions. However, for situations not covered by the heuristic rules, BP systems may not be able to predict the correct trajectory in a timely manner.

[0038] An autonomous or semi-autonomous vehicle system can use a vehicle intention prediction neural network comprising a plurality of intention-specific neural networks to predict the intentions of one or more nearby vehicles. The vehicle intention neural network can use raw input data representing one or more vehicles in the environment, thereby reducing the risk of failure in traditional heuristic-based approaches. The system can determine a confidence score representing the predicted likelihood that a nearby vehicle will follow the intention corresponding to each intention-specific neural network. Additionally, the system can predict the trajectory that the nearby vehicle will follow in a future time period if the vehicle follows the intention corresponding to each intention-specific neural network. The system can provide the vehicle intention output to a planning system, and the planning system can plan the navigation of the autonomous vehicle based on the predicted intentions of the nearby vehicles. More generally, by being able to accurately predict the intentions of nearby vehicles and the trajectories that these vehicles are likely to follow given the predicted intentions, autonomous vehicles can make better autonomous driving decisions or provide better semi-autonomous driving recommendations to the vehicle's operator.

[0039] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a diagram of an example system.

[0041] Figure 2 An example architecture of a vehicle intent prediction neural network for making vehicle intent predictions from input data is shown.

[0042] Figure 3 An example architecture of a vehicle intent prediction neural network using multiple intent-specific neural networks is shown.

[0043] Figure 4 is a flowchart of an example process for computing vehicle intent predictions from input data.

[0044] Figure 5 is a flowchart of an example process for training a vehicle intent prediction neural network.

[0045] Similar reference numbers and designations in different drawings indicate similar elements. DETAILED DESCRIPTION

[0046] This specification describes how a vehicle (e.g., an autonomous or semi-autonomous vehicle) can use a machine learning model (e.g., a fully learned neural network) to predict the vehicle's intent and trajectory by utilizing input data that characterizes the vehicle's appearance and contextual information in an environment. As used in this specification, a "fully learned" machine learning model is a model that is trained to compute desired predictions. In other words, the fully learned model generates output based solely on training data rather than on human-programmed decision logic. A vehicle intent prediction can include one or more predicted intents that the vehicle is likely to follow and one or more corresponding predicted trajectories that the vehicle will follow in a future time period.

[0047] A vehicle intent describes where the vehicle plans to go, i.e., the long-term action goal. Vehicle intent can be organized into multiple categories, each corresponding to a different intent. Examples of intent categories can include any of: going straight, turning left, turning right, changing lanes left, changing lanes right, stopping, and reversing. A trajectory describes where the vehicle might go, i.e., the short-term execution result. In some cases, the short-term execution result is consistent with the long-term action goal. However, in some cases, the execution result can differ from the original intent.

[0048] For example, car A plans to turn left at an intersection, but it stops at a red light. Therefore, the short-term execution result (i.e., trajectory) is static. As another example, car A plans to make a left lane change, but there is another car B driving parallel to it in the left lane. Therefore, the short-term execution of car A may be to accelerate or decelerate in a straight line to avoid car B and then change lanes.

[0049] Accurately and timely predicting a vehicle's intention and trajectory is crucial for solving many autonomous driving problems. This specification describes a system for trajectory and intention prediction that uses machine learning models to efficiently model scene-level context and appearance information, including the road map, the dynamics of perceived objects, and the self-driving car (SDC) itself.

[0050] Figure 1 is a diagram of an example system 100. System 100 includes a training system 110 and an onboard system 120.

[0051] The onboard system 120 is physically located on the vehicle 122. On the vehicle 122 means that the onboard system 120 includes components that travel with the vehicle 122, such as a power source, computing hardware, and sensors. Figure 1Vehicle 122 in FIG. 1 is shown as a motor vehicle, but onboard system 120 can be located on any suitable vehicle type. Vehicle 122 can be a fully autonomous vehicle that uses vehicle intent prediction to inform fully autonomous driving decisions. Vehicle 122 can also be a semi-autonomous vehicle that uses vehicle intent prediction to assist a human driver. For example, if vehicle intent prediction indicates that a human driver is about to collide with another vehicle, vehicle 122 can autonomously apply the brakes.

[0052] The vehicle-mounted system 120 includes one or more perception subsystems 132. The perception subsystem 132 can capture information about objects in the scene, including object type, object attributes (e.g., children, special vehicles, and turn signals), object properties (e.g., human activity, signs), object coordinates, object speed, object heading, and object curvature. For example, the perception subsystem can include a combination of sensor components that receive reflections of electromagnetic radiation, such as a lidar system that detects reflections of laser light, a radar system that detects reflections of radio waves, and a camera system that detects reflections of visible light. Additionally, the perception subsystem 132 can obtain predetermined environmental information, such as information identifying lanes, stop signs, intersections, and the like.

[0053] Using predetermined environmental information and sensor-captured information, the perception subsystem 132 can generate surrounding context information and object appearance information that can help predict vehicle intent. The context information can be road context information represented by a sequence of top-down rendered images, i.e., a road map. In each rendered image, road information (e.g., lanes and stop signs), the positions of surrounding objects (e.g., vehicles and pedestrians), and the vehicle itself are rendered in the image from a top-down perspective. Appearance information can generally be any information that characterizes the visual appearance of the vehicle.

[0054] The perception subsystem 132 provides input data 155 to the onboard neural network subsystem 134. The input data 155 may include one or more channels of contextual information and appearance information captured by the perception subsystem 132. For example, the input data may include a sequence of top-down rendered images representing one or more vehicles in the environment. In some embodiments, the input data 155 may include embedding information derived from data generated by one or more sensors, the embedding information representing the scene near the autonomous vehicle. For example, the input data 155 may be or may include an appearance embedding of a vehicle or a camera patch. The appearance embedding may be computed using a previously trained appearance embedding neural network.

[0055] The perception subsystem 132 or the onboard neural network subsystem 134, or some combination of the two, can transform the raw perception data into multiple channels of input data 155 representing the appearance and contextual information of one or more vehicles in the environment. To do this, the onboard system 120 can project the various characteristics of the raw perception data into a common coordinate system.

[0056] The onboard neural network subsystem 134 implements the operations of each layer of the vehicle intent prediction neural network that is trained to make vehicle intent and trajectory predictions. Therefore, the onboard neural network subsystem 134 includes one or more computing devices having software or hardware modules that implement the corresponding operations of each layer of the neural network according to the architecture of the neural network.

[0057] Onboard neural network subsystem 134 may implement the operations of each layer of the neural network by loading a set of model parameter values ​​172 received from training system 110. Although shown as logically separate, model parameter values ​​170 and the software or hardware modules that perform the operations may actually be located on the same computing device or, in the case of executing software modules, stored within the same memory device.

[0058] The onboard neural network subsystem 134 may use hardware acceleration or other specialized computing devices to implement the operations of one or more layers of the neural network. For example, some operations of some layers may be performed by highly parallelized hardware, such as a graphics processing unit or another specialized computing device. In other words, not all operations of every layer need to be performed by the central processing unit (CPU) of the onboard neural network subsystem 134.

[0059] The onboard neural network subsystem 134 uses the input data 155 to generate one or more vehicle intent predictions 165. The onboard neural network subsystem 134 may provide the one or more vehicle intent predictions 165 to the planning subsystem 136, the user interface subsystem 138, or both.

[0060] Each vehicle intention prediction includes corresponding predicted probabilities for a plurality of possible intentions and a corresponding predicted trajectory for each possible intention. As discussed above, vehicle intentions can be organized into a plurality of categories, such as one or more of going straight, turning left, turning right, changing lanes left, changing lanes right, stationary, and reversing. For example, a vehicle intention prediction may include a probability of 0.8 for turning left, and a probability of 0.2 for going straight. Each vehicle intention prediction also includes a predicted trajectory, which is the predicted trajectory that the vehicle will follow in a future time period if the vehicle follows the intention. In the above example, the vehicle intention prediction may include a predicted trajectory of turning left during the next 10 seconds, and a predicted trajectory of going straight during the next 10 seconds. In some embodiments, the predicted trajectory may include a predicted position of the vehicle at each of a plurality of future time steps.

[0061] When planning subsystem 136 receives one or more vehicle intention predictions 165, planning subsystem 136 can use the one or more vehicle intention predictions 165 to make fully autonomous or semi-autonomous driving decisions. For example, planning subsystem 136 can generate a fully autonomous plan based on the predicted intentions and trajectories of nearby vehicles driving in parallel with the vehicle itself, and navigate around the nearby vehicles. As another example, determining that an oncoming vehicle is planning to turn left at an intersection despite being currently stopped at a stop sign may cause planning subsystem 136 to appropriately yield to the oncoming vehicle until the oncoming vehicle completes its left turn.

[0062] The user interface subsystem 138 can receive the vehicle intent prediction 165 and can generate a user interface presentation that indicates one or more intents and / or trajectories of nearby vehicles. For example, the user interface subsystem 138 can generate a user interface presentation with image or video data that includes a representation of the short-term trajectory and long-term intent of a nearby truck in the environment. The onboard display device can then display the user interface presentation for a passenger of the vehicle 122.

[0063] Onboard neural network subsystem 134 may also use input data 155 to generate training examples 123. Onboard system 120 may provide training examples 123 to training system 110 in an offline batch or online manner, e.g., continuously whenever training examples 123 are generated.

[0064] Training system 110 is typically hosted within a data center 112, which may be a distributed computing system having hundreds or thousands of computers in one or more locations.

[0065] The training system 110 includes a training neural network subsystem 114 that can implement the operations of each layer of a neural network configured to make vehicle intent predictions from input data representing one or more vehicles in an environment. The training neural network subsystem 114 includes multiple computing devices having software or hardware modules that implement the corresponding operations of each layer of the neural network according to the architecture of the neural network.

[0066] The training neural network typically has the same architecture and parameters as the on-board neural network. However, the training system 110 does not need to use the same hardware to calculate the operations of each layer. In other words, the training system 110 can use only CPUs, highly parallelized hardware, or some combination of these.

[0067] The training neural network subsystem 114 may calculate the operation of each layer of the neural network using the current parameter values ​​115 stored in the set of model parameter values ​​170. Although shown as logically separate, the model parameter values ​​170 and the software or hardware modules that perform the operations may actually be located on the same computing device or the same memory device.

[0068] The training neural network subsystem 114 may receive as input training examples 123. The training examples 123 include input data and one or more labels indicating the intent and trajectory of one or more vehicles within the spatial region represented by the input data.

[0069] The training neural network subsystem 114 can generate one or more vehicle intent predictions 135 for each training example 123. Each vehicle intent prediction 135 represents the predicted likelihood that the vehicle will follow the corresponding intent, as well as the predicted trajectory that the vehicle will follow in the future time period if the vehicle follows the corresponding intent. The training engine 116 analyzes the vehicle intent predictions 135 and compares the vehicle intent predictions to the labels in the training examples 123. The training engine 116 then generates updated model parameter values ​​145 by using an appropriate update technique (e.g., stochastic gradient descent with backpropagation). The training engine 116 can then use the updated model parameter values ​​145 to update the set of model parameter values ​​170.

[0070] After training is complete, the training system 110 can provide the final set of model parameter values ​​171 to the vehicle system 120 for making fully autonomous or semi-autonomous driving decisions. The training system 110 can provide the final set of model parameter values ​​171 via a wired or wireless connection to the vehicle system 120.

[0071] Figure 2 An example architecture of a vehicle intent prediction neural network 200 for making vehicle intent predictions from input data 202 is shown. The input data 202 can be data generated or derived from the perception subsystem 132 (i.e., from sensor data) that represents one or more vehicles in the environment. The input data 202 can include multiple frames of rendered images, with a fixed time interval between each frame. For example, the input can include (i) 10 frames of rendered images representing historical trajectory characteristics of one or more vehicles in the scene, with 0.1 seconds between each two frames, and (ii) appearance information of one or more vehicles in the scene. The rendered image of the input data 202 can include information identifying lanes, vehicle positions, and a road map that includes tracked objects in the vicinity of the vehicle. In some embodiments, the input rendered image can also include vehicle heading information by rendering the heading information with a complementary color.

[0072] In some embodiments, the appearance information is an appearance embedding of the input data 202 that represents the appearance of a vehicle and is generated from sensor data captured by one or more sensors of a particular other vehicle in the environment. For example, the appearance embedding of car A can be generated from a camera image captured by a camera mounted on car B behind car A. The appearance embedding can represent the current visual appearance of car A, for example, whether the left turn signal of car A is currently on.

[0073] Network 200 includes a context encoder neural network 204 that takes input data 202 as input and outputs a context feature map that encodes context information in the input data 202. For example, context encoder neural network 204 can be a convolutional neural network that performs feature encoding from an initial set of measurement data to construct derived values, i.e., a feature set, that are intended to provide information about the properties of the initial set of measurement data. For example, a previously trained convolutional neural network can encode a rendered image and generate a road context feature map that includes the feature set.

[0074] The network 200 also includes a feature assembler neural network 206 that constructs and assembles one or more context features and one or more appearance features for each vehicle in the environment. Feature assembly can be implemented by cascading layers or additional layers in the neural network architecture. The feature assembler neural network 206 takes as input the context feature map encoded by the context encoder neural network 204. The feature assembler neural network 206 can assemble the context feature map directly or perform further encoding / embedding operations on the input context feature map before assembling. Figure 3 More details regarding the feature assembler neural network 206 are discussed.

[0075] In some embodiments, the input data 202 may include previously computed appearance embeddings, or other types of embedded representations of the environment. The feature assembler neural network 206 may take as input these previously computed embedding representations for assembly. For example, if the input data 202 includes appearance embeddings for a vehicle, the feature assembler neural network 206 may assemble the appearance embeddings directly with the context feature map using cascade layers. In some embodiments, the feature assembler neural network 206 may further process the previously computed embedding representations before assembling. For example, if the input data 202 includes appearance embeddings for a vehicle, the feature assembler neural network 206 may generate an appearance feature map from the previously computed appearance embeddings for the vehicle by processing the embeddings using multiple convolutional neural network layers, and then assemble the generated appearance feature map with the context feature map using cascade layers.

[0076] The network 200 also includes a plurality of intent-specific neural networks 208 ( 1 ), 208 ( 2 ) through 208 (N) that predict intent categories and trajectories.

[0077] Each intent-specific neural network 208(1), 208(2) to 208(N) corresponds to a corresponding intent in the set of intents. Each intent-specific neural network is configured to process features of a vehicle to generate an output corresponding to the intent. For example, when vehicle intents are organized into seven categories, the network 200 may have N=7 intent-specific neural networks 208(1) to 208(7) configured to generate intent predictions for each of the seven possible intent categories.

[0078] The outputs from each intent-specific neural network 208(1), 208(2) to 208(N) include: (i) a confidence score representing the predicted likelihood that the vehicle will follow the intent corresponding to the intent-specific neural network, and (ii) a predicted trajectory that the vehicle will follow in a future time period if the vehicle follows the intent corresponding to the intent-specific neural network.

[0079] In some embodiments, the output confidence scores from the N intent-specific neural networks 208(1), 208(2) through 208(N) can represent a probability distribution of the predicted intent among possible intent categories. The sum of all confidence scores can be equal to 1. For example, for the seven intent categories described in the example above, the output confidence scores can be [0.1, 0.1, 0.4, 0.3, 0.05, 0.025, 0.025].

[0080] Reference below Figure 3 The architecture of the intent-specific neural network is described in more detail.

[0081] Figure 3 An example architecture of a vehicle intent prediction neural network 300 using multiple intent specific neural networks 316(1), 316(2) through 316(N) is shown. The neural network 300 is Figure 2 A more detailed example of the network 200 in FIG. Input data to the neural network 300 includes a rendered image 302 and an appearance embedding 304. The vehicle intent prediction neural network 300 is configured to generate intent predictions for one or more vehicles from the input rendered image 302 and appearance embedding 304.

[0082] The vehicle intent prediction neural network 300 can generate intent predictions for a single vehicle at a time or for multiple vehicles at a time. To generate intent predictions for multiple vehicles at a time, the system can process the rendered image 302 to generate corresponding inputs to the intent-specific neural network for each of the vehicles, which are then processed.

[0083] Rendered image 302 may include a road map sequence representing a sequence of historical frames of the vehicle and its surrounding vehicles. The rendering position may be fixed at the vehicle's latest position. For example, rendered image 302 includes three road maps at three historical time steps, i.e., three frames, wherein the rendering position is fixed at the vehicle's latest position in the third frame. In rendered image 302, the gray rectangle represents vehicle 329, and the white rectangles represent two other vehicles near vehicle 329. The sequence of rendered images 302 shows that vehicle 329 is moving in the middle lane, and vehicle 330 to the left of vehicle 329 is changing to the middle lane by cutting in in front of vehicle 329.

[0084] In some embodiments, the rendered image 302 can be generated by a rendering processor in the vehicle system 120 of the vehicle 122. The rendering processor can synchronize the vehicle pose and one or more tracked objects in the vicinity of the vehicle to a desired frequency and can maintain a history buffer of tracked objects and vehicle poses. For example, information about the pose of one or more tracked objects in the vicinity of the vehicle can come at a variable frequency, and the rendering processor can generate the rendered image at a fixed frequency of 10 Hz.

[0085] The appearance embedding 304 of a vehicle can represent the appearance of the vehicle as sensed by one or more sensors of specific other vehicles in the environment. In some embodiments, the appearance embedding 304 can include a camera patch of the vehicle of interest. In some embodiments, the appearance embedding 304 can include an appearance feature map generated from one or more camera images using a pre-trained appearance embedding neural network. The appearance embedding 304 can include features that identify information such as the vehicle's turn signal, heading, one or more tracked objects in the vicinity of the vehicle, object types, etc. For example, the appearance embedding 304 can include camera image patches and / or features extracted from a camera image that can indicate whether the vehicle's left turn signal is currently on.

[0086] The network 300 includes a context encoder 306 that uses a convolutional neural network (CNN) backbone. The context encoder 306 takes rendered images 302 as input and outputs a road context feature map 308 for each vehicle in the environment. In some embodiments, the context encoder 306 can process one or more rendered images 302 of the environment to generate a feature map that includes a corresponding feature vector for each of a plurality of positions of each vehicle in the environment. For example, the context encoder 306 can generate a feature map that includes three corresponding feature vectors for three positions of the vehicle 329 in three rendered images 302.

[0087] Typically, the context encoder 306 uses a convolutional neural network comprising a plurality of convolutional layers and optionally a plurality of deconvolutional layers. Each convolutional layer or deconvolutional layer has parameters whose values ​​define the filters of the layer.

[0088] In some embodiments, the context encoder 306 can use an AlexNet-type CNN backbone (Krizhevsky, Alex et al., "Imagenet classification with deep convolutional neural networks," Advances in Neural Information Processing Systems, 2012), including multiple convolutional layers, pooling layers, and optionally multiple fully connected layers. For example, from an input rendered image of dimension 256×256, a 4-layer AlexNet with four 2×2 pooling layers can generate a road context feature vector of dimension 16×16×32. From three input rendered images 302, each of dimension 256×256, a 4-layer AlexNet can generate a road context feature map of dimension 16×16×32×3. The road context feature map includes three feature vectors, each of dimension 16×16×32, corresponding to each of the three rendered images 302. Each feature vector includes features for each of the three vehicles in the rendered image.

[0089] In some embodiments, the context encoder 306 may use a ResNet-type CNN backbone (He, Kaiming et al., "Deep residual learning for image recognition," IEEE Conference on Computer Vision and Pattern Recognition, 2016). For example, the context encoder 306 may use block 1 of ResNet-18 as a backbone to generate the road context feature map 308. The ResNet-type backbone may provide the benefit of lower latency, for example, reducing the amount of computation required, without reducing the accuracy of the vehicle intent prediction neural network 300.

[0090] In some embodiments, based on the feature vectors in the road context feature map 308 , the network 300 may be configured to generate one or more other feature maps, such as a trajectory feature map 310 and a context feature map 312 .

[0091] In some embodiments, the network 300 can be configured to generate a trajectory feature map 310 using the following steps: (i) for a vehicle 330 in the environment, obtaining the position of the vehicle 330 in each input rendered image 302 of the environment; (ii) cropping the road context feature map 308 based on each position of the vehicle to obtain a plurality of cropped feature vectors 331, 332, and 333 including a corresponding feature vector for each position of the vehicle; and (iii) performing an average pooling operation, i.e., temporal average pooling, on the plurality of cropped feature vectors to generate a trajectory feature map 310 for the vehicle 330. The trajectory feature map 310 can capture the local features of each vehicle in the environment by extracting feature patches based on the vehicle position on the feature map.

[0092] For example, based on the road context feature map 308 having dimensions of 16×16×32×3 and the position of the vehicle 330 in each of the three input rendered images, the network 300 can be configured to obtain three cropped feature vectors centered on the vehicle 330, each having dimensions of 4×4×32. The network 300 can then be configured to generate a trajectory feature map 310 of the vehicle 330 by performing a temporal average pooling operation on the three cropped feature vectors. The generated trajectory feature map 310 has dimensions of 4×4×32.

[0093] In some embodiments, the network 300 can be configured to generate a context feature map 312 of the vehicle 330 by performing an average pooling operation, i.e., spatial average pooling, on the feature vectors in the road context feature map 308. The context feature map 312 can capture global features by performing spatial average pooling on the entire feature map.

[0094] For example, based on the road context feature map 308 having dimensions of 16×16×32×3, the network 300 may be configured to generate a context feature map having dimensions of 4×4×32×3 by performing a spatial average pooling operation on the feature vectors.

[0095] In some embodiments, before generating one or more other feature maps, such as the trajectory feature map 310 or the context feature map 312, the network 300 can be configured to apply a self-attention mechanism to the road context feature map 308 to generate an attended feature map 309. The network 300 can then be configured to generate one or more other feature maps, such as the trajectory feature map 310 or the context feature map 312, based on the attended feature map 309. The attended feature map 309 can implicitly model the interaction of factors (i.e., vehicles) and thus help generate more meaningful feature maps, such as the context feature map 312.

[0096] The attention mechanism (Vaswani, Ashish et al., “Attention is all you need”, Advances in Neural Information Processing Systems, 2017) can be described as mapping a query and a set of key-value pairs to an output, where the query, key, value, and output are all vectors. For example, the network 300 can be configured to implement the following scaled dot product attention function:

[0097]

[0098] Where Q is a set of queries packed into a matrix, K is a vector of keys, V is a vector of values, and d k is the dimension of the key, K T is the transfer matrix of vector K, and softmax() is the softmax function. The network 300 can be configured to apply the attention function in equation (1) to the road context feature map 308 to generate an attention feature map 309. Based on the attention feature map 309, the network can be configured to generate a context feature map 312.

[0099] The network 300 includes a feature assembler neural network 314 that assembles the appearance feature map 305, the trajectory feature map 310, and the context feature map 312 for each vehicle in the environment. The feature assembler neural network 314 can be implemented as a cascade of layers or additional layers. For example, for a vehicle 330 in the environment, the feature assembler neural network 314 can concatenate or average the appearance feature map corresponding to the vehicle, the trajectory features of the vehicle, and the context features to generate a single tensor.

[0100] The appearance feature map 305 can be the same as the appearance embedding 304. In some embodiments, the feature assembler neural network 314 can perform further encoding / embedding operations on the input appearance embedding 304 and can use one or more convolutional layers to generate the appearance feature map 305. Similarly, the trajectory feature map 310 and / or the context feature map 312 can also be generated or further processed by the feature assembler neural network 314 using one or more convolutional layers.

[0101] The network 300 includes a fixed number of intent-specific neural networks 316(1), 316(2) through 316(N) that can output a fixed number of intent predictions 318(1), 318(2) through 318(N). The intent-specific neural network 316(2) is an example of one of the intent-specific neural networks 316(1), 316(2) through 316(N). The intent prediction 318(2) is an example of one of the intent predictions 318(1), 318(2) through 318(N). For example, when vehicle intent is organized into seven categories, the network 300 can have N=7 intent-specific neural networks 316(1) through 316(7) configured to generate intent predictions 318(1) through 318(7) for each of the seven possible intent categories.

[0102] Each intent-specific neural network (e.g., intent-specific neural network 316(2)) takes the assembled feature map as input and outputs an intent prediction, such as intent prediction 318(2), including an intent confidence score 322 and a prediction trajectory 324. Each intent-specific neural network may include one or more fully connected neural network layers configured to generate an intent prediction for the corresponding intent from the assembled feature map. Each intent-specific neural network may be an intent decoder that decodes the assembled feature map.

[0103] In some embodiments, each intent-specific neural network (e.g., intent-specific neural network 316(2)) can include: (i) one or more fully connected neural network layers configured to generate intent confidence scores 322; and (ii) one or more autoregressive neural network layers configured to autoregressively generate predicted trajectories 324. Intent confidence scores 322 and predicted trajectories 324 are outputs of intent-specific neural network 316(2).

[0104] Each intent confidence score 322 represents a predicted likelihood that the vehicle will follow the intent corresponding to the intent-specific neural network 316(2). Each predicted trajectory 324 represents a trajectory that the vehicle will follow in a future time period if the vehicle follows the intent 322 corresponding to the intent-specific neural network 316(2). For example, the intent-specific neural network 316(2) for a left turn intent may output an intent prediction 318(2) that includes: (i) a left turn intent confidence score of 0.5, i.e., there is a 50% probability that the vehicle is planned to turn left; and (ii) a predicted left turn trajectory within the next 10 seconds.

[0105] In some embodiments, the intent-specific neural networks 316(1), 316(2) through 316(N) can share one or more neural network layers. For example, the intent-specific neural networks 316(1), 316(2) through 316(N) can share one or more fully connected layers. As another example, the intent-specific neural networks 316(1), 316(2) through 316(N) can share a classification layer that outputs N confidence scores in an output header of the classification layer. The shared classification layer can output a probability distribution over N possible intents, and the sum of all N confidence scores is equal to 1.

[0106] In some embodiments, the predicted trajectory 324 may include the predicted position of the vehicle at each of a plurality of future time steps. In some embodiments, the predicted trajectory may include predicted coordinates, such as (x, y), at each of a plurality of future time steps at regular intervals (e.g., every 0.1 seconds). For example, the predicted trajectory 324 may include the (x, y) coordinates at the next 0.1 seconds, the next 0.2 seconds, and the next 1.0 seconds.

[0107] In some embodiments, each intent-specific neural network (e.g., intent-specific neural network 316(2)) can include one or more autoregressive neural network layers, including recurrent neural network (RNN) layers (Cleeremans, Axel et al., "Finite state automata and simple recurrent networks," Neural Computation 1.3 (1989): 372-381). For example, each intent-specific neural network can include a vanilla RNN that follows the following equation:

[0108] intent=f CLS (h0),

[0109] h t =f RNN (h t-1 ,traj t-1 ,intent),

[0110] traj t =f TRAJ (h t ).

[0111] Among them, h t represents the hidden state of the RNN at step t. The hidden state of the first step h0 can be taken from the yjr context encoder 306 or set to a predetermined default value. RNN represents the loop update function, i.e., the processing for RNN, intent represents the discretized intention label, such as turn left or turn right, and trajt Represents the predicted trajectory deviation at step t. traj t is implemented using a neural network—for example, by applying one or more fully connected layers to the hidden state h t To implement—— TRAJ To speed up the inference, in some embodiments, each prediction step of the RNN predicts multiple future steps of coordinates. That is, traj t Coordinates for multiple future time steps can be included. Therefore, the RNN does not have to perform processing for every future time step in the predicted trajectory.

[0112] In some embodiments, each intent-specific neural network can include an output head for predicting trajectory 324, which includes multiple neural network layers to predict the mean and variance of a Gaussian distribution of trajectory coordinates. During training of neural network 300, a loss term (e.g., negative log-likelihood loss) can balance accuracy (e.g., L2 distance to the mean) and uncertainty (e.g., variance). Therefore, after training, neural network 300 can be more robust to outliers.

[0113] In some embodiments, each intent-specific neural network can generate a trajectory multiple steps at a time. When the predicted trajectory includes predicted positions of the vehicle for multiple future time steps, the multiple future time steps can be divided into multiple partitions. The autoregressive neural network layer can be configured to generate, for each partition, a predicted position of the vehicle for that partition conditioned on the predicted positions of earlier partitions.

[0114] In some implementations, each partition may include more than one future time step. Generating trajectories multiple time steps at a time can help reduce latency issues in RNNs by reducing the number of processing steps required to predict the entire trajectory.

[0115] For example, a plurality of future time steps t(0), t(1), t(2), t(3), t(4) ... can be divided into a plurality of groups of time steps. Each group can include k time steps, resulting in group 1 having k time steps t(0), t(1), ..., t(k-1), group 2 having k time steps t(k), t(k+1), ..., t(2k-1), and so on. The autoregressive neural network layer can include an RNN layer that is configured to generate a predicted position of the vehicle for each group at each output head, rather than generating a predicted position for one time step at each output head. For example, a first output head of the RNN can output a predicted position at time steps t(0), t(1), ..., t(k-1). A second output head of the RNN can output a predicted position at time steps t(k), t(k+1), ..., t(2k-1) conditioned on the predicted position of the earlier group at time steps t(0), t(1), ..., t(k-1). Instead of generating a single output t(0) at the first output head and a single output t(1) at the second output head, the RNN is configured to generate k predicted positions from each output head. This RNN architecture can help reduce RNN latency because predictions for more than one position can be generated simultaneously from a single output head of the RNN.

[0116] In some embodiments, each intent-specific neural network may include three output heads: (i) a classification head for intent classification; (ii) a regression head for multi-trajectory prediction; and optionally, (iii) a regression head for orientation prediction. That is, each intent-specific neural network may also have a third output head that predicts the orientation of the vehicle if the vehicle has the corresponding intent, where the orientation is the position of a specific part of the vehicle (e.g., the front of the vehicle) in a specific coordinate system. Each output head typically includes one or more neural network layers having an architecture that allows the output head to map the input of the head to the corresponding type of output desired from the head.

[0117] Figure 4 is a flow chart of an example process 400 for computing vehicle intent predictions from input data. Figure 4 The example process in

[15] uses a forward inference pass through a neural network that has already been trained or is currently being trained to make vehicle intent predictions. Therefore, the example process can be used during training or to make predictions from unlabeled inputs, for example, in a production system. The process will be described as being performed by a suitably programmed neural network system.

[0118] The system obtains input data representing one or more vehicles in an environment (402). The input data may include one or more images of the environment containing the one or more vehicles. For example, the input data may include a sequence of rendered images that include information identifying one or more tracked objects near the vehicle, and historical trajectory information for the one or more vehicles in the environment. For each of the vehicles in the environment, the input data may include an appearance embedding of the vehicle that represents the visual appearance of the vehicle as sensed by one or more sensors of a particular other vehicle in the environment. For example, the appearance embedding may include a camera image patch of a left turn signal for the vehicle.

[0119] The system generates features for each of the vehicles from the input data (404). Figure 2 and Figure 3 The architecture of an example neural network suitable for generating features for each of the vehicles from input is described in greater detail. The generated features may include a trajectory feature map and a context feature map. The system may use a feature assembler neural network to assemble the generated trajectory feature map, the generated context feature map, and the appearance feature map obtained from the appearance embedding. The assembled feature map may include local features and global features of the vehicle.

[0120] For each of the vehicles, the system processes the characteristics of the vehicle using each of a plurality of intent-specific neural networks (406). Figure 2 and Figure 3 The architecture of an example intent-specific neural network suitable for generating the intent and trajectory of each of the vehicles from the assembled feature maps is described in more detail. Each intent-specific neural network can include a first output head to predict a confidence score for a specific intent and can include a second output head to predict a corresponding predicted trajectory for the specific intent.

[0121] The system receives a corresponding intent prediction for each of the vehicles from the output of each intent-specific neural network (408). As described above, each intent prediction can include a confidence score for the intent and a predicted trajectory for the corresponding intent. The predicted trajectory can include a predicted position, such as coordinates, of the vehicle at each of a plurality of future time steps.

[0122] After generating a vehicle intent prediction, the system can take different actions depending on whether the forward inference pass is being used for training or in a production system. During training, the vehicle intent prediction can be used to calculate one or more errors based on the labels in the training examples used. The errors can be used to calculate updated weights for the neural network, which can be calculated and updated using any suitable training technique (e.g., gradient descent with backpropagation).

[0123] When used in an in-vehicle production system, vehicle intention predictions can be provided to a planning subsystem that makes decisions about how the vehicle should react to the predicted intentions of vehicles in the environment.

[0124] Figure 5 is a flow chart of an example process for training a vehicle intention prediction neural network. The process will be described as a process performed by a suitably programmed neural network system (e.g., Figure 1 ) is executed by the training system 110).

[0125] The system receives a plurality of training examples, each having input data representing one or more vehicles in an environment and corresponding vehicle intent information (502). As discussed above, the input data can be generated from the perception subsystem 132 of the onboard system 120 or another system. The vehicle intent information can be one or more labels, where each label can indicate the intent of the vehicle and a corresponding trajectory.

[0126] Labels can be automatically obtained using rule-based methods. For example, lane change intention can be determined based on whether a lane change has already occurred in an adjacent lane within k seconds. For example, left or right turn intention can be determined based on vehicle curvature information being greater than a threshold. For example, stationary intention, straight ahead intention, or reverse intention can be determined based on vehicle heading information and speed information. Alternatively, or in combination, labels can be obtained through swarm computing labeling performed by human labelers.

[0127] The system uses the training examples to train a vehicle intent prediction neural network (504). The system can use multiple intent-specific neural networks to process local and global features of the vehicles in the environment and generate outputs corresponding to the intent of each of the multiple intent-specific neural networks. Each intent prediction can include a confidence score and a predicted trajectory that the vehicle will follow in the future time period.

[0128] The system can compare the intent predictions with the labels in the training examples. The system can calculate a loss that can measure the difference between the intent predictions and the labels in the training examples. The loss can include: (1) an intent classification loss, such as a sigmoid loss; and (2) a trajectory regression loss, such as a smoothed L1 loss between the predicted coordinates and the labeled coordinates at a series of future time steps. In some embodiments, the loss can also include an actual loss on the predicted trajectory. For example, a predicted trajectory with a zigzag shape is unrealistic, and a predicted trajectory with a zigzag shape may generate a higher actual loss value, which indicates that the predicted trajectory is unlikely to occur in the real world by measuring the frequency and level of changes in the vehicle's orientation. For example, a loss function including a trajectory regression loss with a smoothed L1 loss, an intent classification loss with a sigmoid loss, and an actual loss on the predicted trajectory can be formulated as follows:

[0129]

[0130] Among them, the true intention is c and the predicted intention is The ground-truth trajectory corresponding to the intention c is trajectory t, and corresponds to the intention The predicted trajectory is the trajectory

[0131] In some embodiments, the vehicle intent prediction neural network can predict the mean and variance of a Gaussian distribution of trajectory coordinates. In other words, instead of directly regressing the coordinates, the system predicts the parameters of a probability distribution (i.e., a Gaussian distribution) over the possible values ​​of each of the coordinates. In this case, the trajectory prediction output head can have one or more output layers that predict the corresponding means of the x and y coordinates, and one or more output layers that predict the corresponding variances of the x and y coordinates. For example, for both the mean and variance, the neural network can have a fully connected layer with a relu nonlinearity, where the final output has the same size of 2, corresponding to the x and y coordinates. The final fully connected layer of the mean network has no nonlinearity, while the final fully connected layer of the variance network has a softplus nonlinearity to make the variance non-negative.

[0132] In these embodiments, instead of using L2 or other distance-based loss for training, the system can directly compute the negative log-likelihood loss of the Gaussian probability to perform maximum likelihood training. Because the loss term balances the L2 distance between the ground truth trajectory and the predicted mean with the uncertainty based on the prediction variance, training the neural network model with this loss can be more robust to outliers.

[0133] The system can generate updated model parameter values ​​based on the loss by using an appropriate update technique (e.g., stochastic gradient descent with backpropagation). The system can then use the updated model parameter values ​​to update the set of model parameter values.

[0134] In some embodiments, when each intent-specific neural network includes one or more RNN layers, each intent-specific neural network can include an output head for predicting the trajectory 324, which includes multiple neural network layers for curriculum learning (Bengio, Yoshua et al., "Curriculum learning", Proceedings of the 26th International Conference on Machine Learning, ACM, 2009) to help generate long sequences. Curriculum learning can include multiple curriculum strategies, such as a teacher forcing strategy and a scheduled sampling strategy. The teaching forcing strategy can allow the neural network to learn from the ground truth traj t-1 Predict traj t , instead of predicting traj t-1 The scheduled sampling strategy can apply a teaching-forcing strategy at the beginning of the training process and then gradually remove the teaching-forcing strategy.

[0135] This specification uses the term "configuration" in conjunction with system and computer program components. When referring to a system of one or more computers configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. When referring to one or more computer programs configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0136] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or additionally, program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical signal, optical signal, or electromagnetic signal, wherein the artificially generated propagated signal is generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus.

[0137] The term "data processing apparatus" refers to data processing hardware and includes all kinds of apparatuses, devices and machines for processing data (including, for example, a programmable processor, a computer, or multiple processors or computers). The apparatus may also be or also include an off-the-shelf or customized parallel processing subsystem, such as a GPU or another special-purpose processing subsystem. The apparatus may also be or also include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code for creating an operating environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0138] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and it may be deployed in any form, including as a standalone program or as a module, component, subroutine or other unit suitable for a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines or code portions). A computer program may be deployed to be run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0139] As used in this specification, an "engine" or "software engine" refers to a software-implemented input / output system that provides outputs distinct from inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device (e.g., a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device) comprising one or more processors and a computer-readable medium. Additionally, two or more of the engines can be implemented on the same computing device or different computing devices.

[0140] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry (e.g., an FPGA or ASIC), or by a combination of special purpose logic circuitry and one or more programmed computers.

[0141] The computer that is suitable for running a computer program can be based on a general or special microprocessor or both or any other kind of central processing unit.Usually, the central processing unit will receive instructions and data from a read-only memory or random access memory or both.The basic element of a computer is a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data.The central processing unit and memory can be supplemented by a special logic circuit or incorporated into the special logic circuit.Usually, a computer will also include one or more large-capacity storage devices (for example, magnetic disks, magneto-optical disks or optical disks) for storing data or be operably coupled to receive data from the one or more large-capacity storage devices or to transfer data thereto.However, a computer does not need to have such a device.In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device (for example, a universal serial bus (USB) flash drive), to give only a few examples.

[0142] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks.

[0143] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse, trackball, or presence-sensitive display or other surface through which the user can provide input to the computer). Other types of devices may also be used to provide for interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound input, voice input, or tactile input. In addition, a computer may interact with a user by transmitting documents to and receiving documents from a device used by the user; for example, by transmitting a web page to a web browser on a user's device in response to a request received from the web browser. In addition, a computer may interact with a user by transmitting text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving a response message from the user in exchange.

[0144] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the claimed combination in some cases, and the claimed combination may be directed to subcombinations or variations of subcombinations.

[0145] Similarly, although operations are depicted in the accompanying drawings in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown, or in a sequential order, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged as multiple software products.

[0146] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the claims. For example, the actions can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

Claims

1. A method for predicting vehicle intention, comprising: obtaining, by an autonomous vehicle navigating in the environment, input representing one or more other vehicles in the environment; generating, by the autonomous vehicle, a characteristic of each of the one or more other vehicles from the input; For each of the one or more other vehicles: The autonomous vehicle processes features of the other vehicle using each of a plurality of intent-specific neural networks, wherein each of the intent-specific neural networks corresponds to a respective intent in a set of intents, wherein each intent in the set of intents is an action goal of the other vehicle at a first time frame, and wherein each intent-specific neural network is configured to process the features of the other vehicle to generate an output corresponding to the intent, the output comprising: (i) a confidence score, representing the predicted likelihood that other vehicles will follow the intent corresponding to the intent-specific neural network, and (ii) a predicted trajectory that the other vehicle will follow in a future time period if the other vehicle follows the intent corresponding to the intent-specific neural network, wherein the predicted trajectory is a sequence of predicted positions of the other vehicle over a second time frame that is shorter than the first time frame; as well as The autonomous vehicle is controlled to navigate in an environment based on outputs generated by a plurality of intent-specific neural networks for each of the one or more other vehicles.

2. The method according to claim 1, wherein Controlling the autonomous vehicle to navigate the environment based on the output includes: Outputs generated by the plurality of intent-specific neural networks for each of the one or more other vehicles are provided to a planning system that plans navigation of the autonomous vehicle in the environment.

3. The method according to claim 1, wherein For each of the one or more other vehicles, the input includes an appearance embedding of the other vehicle, the appearance embedding representing an appearance of the other vehicle as sensed by one or more sensors of the particular other vehicle in the environment.

4. The method according to claim 1, wherein The input comprises one or more images of an environment, and wherein generating a feature of each of the one or more other vehicles comprises: processing one or more images of an environment using a convolutional neural network to generate a feature map comprising a corresponding feature vector for each of a plurality of locations in the environment; and A feature is generated for each of the one or more other vehicles based on the feature vectors in the feature map.

5. The method according to claim 4, wherein Generating features for each of the one or more other vehicles based on the feature vector in the feature graph includes: obtaining a position of the other vehicle in each of the one or more images of the environment; obtaining a plurality of cropped feature vectors by cropping the feature map based on each position of the other vehicles, including a corresponding feature vector for each position of the other vehicles; and A trajectory feature map of the other vehicle is generated by performing an average pooling operation on the plurality of cropped feature vectors.

6. The method according to claim 4, wherein: Generating features for each of the one or more other vehicles based on the feature vector in the feature graph includes: The context feature map of the other vehicle is generated by performing an average pooling operation on the feature vectors in the feature map.

7. The method according to claim 4, wherein: Generating features for each of the one or more other vehicles based on the feature vector in the feature graph includes: applying a self-attention mechanism to the feature map to generate an attended feature map; and Features are generated based on the attended feature map.

8. The method according to claim 1, wherein For each of the one or more other vehicles, the predicted trajectory includes a predicted position of the other vehicle at each of a plurality of future time steps.

9. The method according to claim 1, wherein: Each intent-specific neural network is one or more fully connected neural network layers configured to generate outputs corresponding to the intent from features.

10. The method according to claim 1, wherein Each intent-specific neural network consists of: one or more fully connected neural network layers configured to generate confidence scores; and One or more autoregressive neural network layers are configured to autoregressively generate predicted trajectories.

11. The method according to claim 10, wherein: The one or more autoregressive neural network layers are recurrent neural network layers.

12. The method according to claim 10, wherein: For each of the one or more other vehicles, the predicted trajectory includes a predicted position of the other vehicle for each of a plurality of future time steps, wherein the plurality of future time steps are divided into a plurality of partitions, and wherein the autoregressive neural network layer is configured to generate, for each partition, a predicted position of the other vehicle for that partition conditioned on a predicted position of an earlier partition.

13. The method according to claim 12, wherein: Each partition includes more than one future time step.

14. The method according to claim 1, wherein Each predicted trajectory is a sequence of predicted positions of the other vehicles.

15. The method according to claim 1, wherein The intention set includes one or more of the following: go straight, turn left, turn right, change lane left, change lane right, stay stationary, or reverse.

16. A system for predicting vehicle intent, comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform operations comprising: obtaining, by an autonomous vehicle navigating in the environment, input representing one or more other vehicles in the environment; generating, by the autonomous vehicle, a characteristic of each of the one or more other vehicles from the input; For each of the one or more other vehicles: The autonomous vehicle processes features of the other vehicle using each of a plurality of intent-specific neural networks, wherein each of the intent-specific neural networks corresponds to a respective intent in a set of intents, wherein each intent in the set of intents is an action goal of the other vehicle at a first time frame, and wherein each intent-specific neural network is configured to process the features of the other vehicle to generate an output corresponding to the intent, the output comprising: (i) a confidence score, representing the predicted likelihood that other vehicles will follow the intent corresponding to the intent-specific neural network, and (ii) a predicted trajectory that the other vehicle will follow in a future time period if the other vehicle follows the intent corresponding to the intent-specific neural network, wherein the predicted trajectory is a sequence of predicted positions of the other vehicle over a second time frame that is shorter than the first time frame; as well as The autonomous vehicle is controlled to navigate in an environment based on outputs generated by a plurality of intent-specific neural networks for each of the one or more other vehicles.

17. The system according to claim 16, wherein: Controlling the autonomous vehicle to navigate the environment based on the output includes: Outputs generated by the plurality of intent-specific neural networks for each of the one or more other vehicles are provided to a planning system that plans navigation of the autonomous vehicle in the environment.

18. The system according to claim 16, wherein: For each of the one or more other vehicles, the input includes an appearance embedding of the other vehicle, the appearance embedding representing an appearance of the other vehicle as sensed by one or more sensors of the particular other vehicle in the environment.

19. The system according to claim 16, wherein: The input comprises one or more images of an environment, and wherein generating a feature of each of the one or more other vehicles comprises: processing one or more images of an environment using a convolutional neural network to generate a feature map comprising a corresponding feature vector for each of a plurality of locations in the environment; and A feature is generated for each of the one or more other vehicles based on the feature vectors in the feature map.

20. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: obtaining, by an autonomous vehicle navigating in the environment, input representing one or more other vehicles in the environment; generating, by the autonomous vehicle, a characteristic of each of the one or more other vehicles from the input; For each of the one or more other vehicles: The autonomous vehicle processes features of the other vehicle using each of a plurality of intent-specific neural networks, wherein each of the intent-specific neural networks corresponds to a respective intent in a set of intents, wherein each intent in the set of intents is an action goal of the other vehicle at a first time frame, and wherein each intent-specific neural network is configured to process the features of the other vehicle to generate an output corresponding to the intent, the output comprising: (i) a confidence score, representing the predicted likelihood that other vehicles will follow the intent corresponding to the intent-specific neural network, and (ii) a predicted trajectory that the other vehicle will follow in a future time period if the other vehicle follows the intent corresponding to the intent-specific neural network, wherein the predicted trajectory is a sequence of predicted positions of the other vehicle over a second time frame that is shorter than the first time frame; as well as The autonomous vehicle is controlled to navigate in an environment based on outputs generated by a plurality of intent-specific neural networks for each of the one or more other vehicles.

Images