Predicting plugging probability of surrounding factors

Abstract

Methods, systems, and devices for generating merge probabilities for factors surrounding a vehicle, including computer programs encoded on computer storage media. One of the methods includes obtaining factor trajectory data for one or more factors in an environment; obtaining vehicle trajectory data for a vehicle in the environment; and processing, using a neural network, a network input generated from the factor trajectory data and the vehicle trajectory data to generate a merge output, wherein the merge output includes a respective merge probability for each of a plurality of locations in the environment, wherein the respective merge probability for each location as a current location of one or more factors characterizes a likelihood that the factor at the current location will intersect a planned future location of the vehicle within a predetermined amount of time.

Description

BACKGROUND

[0001] This specification relates to autonomous vehicles.

[0002] Autonomous vehicles include self-driving cars, boats, and airplanes. Autonomous vehicles use various on-board sensors and computer systems to detect nearby objects, and use such detections to make control and navigation decisions.

[0003] Some autonomous vehicles have on-board computer systems that implement neural networks, other types of machine learning models, or both, for various prediction tasks, such as object classification within an image. For example, a neural network can be used to determine that an image captured by an on-board camera is likely to be an image of a nearby car. A neural network, or simply a network, is a machine learning model that employs multiple layers of operations to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers between an input layer and an output layer. The output of each layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer.

[0004] Each layer of a neural network specifies one or more transformation operations to be performed on the input to that layer. Some neural network layers have operations that are referred to as neurons. Each neuron receives one or more inputs and generates an output that is received by another neural network layer. Typically, each neuron receives inputs from other neurons and each neuron provides outputs to one or more other neurons.

[0005] The architecture of a neural network specifies what layers are included in the network and their properties, and how the neurons of each layer of the network are connected. In other words, the architecture specifies which layers provide their outputs as inputs to which other layers, and how the outputs are provided.

[0006] The transformation operations of each layer are performed by a computer that has installed software modules that implement the transformation operations. Thus, describing a layer as performing an operation means that the computer that implements the transformation operations of that layer performs the operation.

[0007] Each layer generates one or more outputs using a current value of a set of parameters for that layer. Thus, training a neural network involves performing forward passes on inputs in succession, computing gradient values, and using the computed gradient values to update the current values of the sets of parameters for each layer, e.g., using gradient descent. Once a neural network is trained, the final set of parameter values can be used to make predictions in a production system. SUMMARY

[0008] This specification describes how a vehicle (e.g., an autonomous or semi-autonomous vehicle) can generate a prediction of whether one or more surrounding agents are to cut in front of the vehicle. The cut-in prediction is generated by a machine learning model, which is referred to in this specification as a “cut-in neural network.” That is, given an input that characterizes the environment surrounding the vehicle, the output of the cut-in neural network is one or more probabilities that represent the likelihood that one or more agents will “cut in” (i.e., enter the vehicle’s planned path within a certain amount of time (e.g., 3 seconds or 5 seconds)). This specification also describes how a computer system can train the cut-in neural network using training examples generated by vehicles operating in the real world.

[0009] Once the cut-in neural network has been trained, the trained cut-in neural network can be deployed on a vehicle and can be used by the vehicle to make autonomous or semi-autonomous driving decisions.

[0010] According to a first embodiment, there is provided a method comprising: obtaining agent trajectory data for one or more agents in an environment, the agent trajectory data for each agent comprising a current location and a current value for a set of predetermined motion parameters of the agent, and a previous location and a previous value for the set of predetermined motion parameters of the agent at each of one or more previous time points; obtaining vehicle trajectory data for a vehicle in the environment, the vehicle trajectory comprising a current location and a current value for a set of predetermined motion parameters of the vehicle, a previous location and a previous value for the set of predetermined motion parameters of the vehicle at each of one or more previous time points, and a planned future location and a planned future value for the set of predetermined motion parameters of the vehicle at each of one or more future time points; and processing, using a neural network, a network input generated from the agent trajectory data and the vehicle trajectory data to generate a cut-in output, wherein the cut-in output comprises a respective cut-in probability for each of a plurality of locations in the environment, wherein the respective cut-in probability for each location that is a current location of one or more agents characterizes a likelihood that the agent at the current location will intersect with the planned future location of the vehicle within a predetermined amount of time.

[0011] In some implementations, the agent trajectory data further comprises a predicted future location and a predicted future value for the set of predetermined motion parameters of the agent at each of the one or more future time points.

[0012] In some implementations, for each of the one or more future time points, the vehicle trajectory data further comprises a plurality of planned future locations and a plurality of planned future values for each of the set of predetermined motion parameters of the vehicle.

[0013] In some implementations, the plurality of locations comprises non-agent locations that are not any current location of the one or more agents, and wherein the neural network has been trained to generate a null probability for the non-agent locations.

[0014] In some embodiments, the network input includes at least one of: an image that characterizes a road lane in the environment, or an image that characterizes a bounding box of one or more factors in the environment.

[0015] In some embodiments, the network input includes a respective concatenation of multiple channels corresponding to each factor and the vehicle, and wherein in each concatenation: each channel is represented as a two-dimensional array of data values; each position in each channel corresponds to a respective location in the environment; corresponding positions in different channels correspond to the same location in the environment; the channels include a temporal channel and a respective motion channel corresponding to each motion parameter in a predetermined set of motion parameters; and for each particular location in the environment occupied by a factor or the vehicle at a particular time point: a position in the temporal channel corresponding to the particular location defines the particular time point; and for each motion channel, a position in the motion channel corresponding to the particular location defines a value of the motion parameter of the motion channel corresponding to the particular time point.

[0016] In some embodiments, the predetermined set of motion parameters includes at least one of: an orientation of the factor or the vehicle at the time point, a velocity of the factor or the vehicle at the time point, or an acceleration of the factor or the vehicle at the time point.

[0017] In some embodiments, the jam output includes a two-dimensional array of data values, wherein each position in the array corresponds to a respective location in the environment, and wherein the data values each characterize a jam probability of the respective location.

[0018] In some embodiments, the method further includes generating jam neural network features, including one or more of: extracting features from the jam output, or extracting features from one or more intermediate outputs of the neural network; and using the jam neural network features to generate factor jam probabilities of one or more factors in the environment.

[0019] In some embodiments, the features extracted from the jam output include one or more of: an average jam probability of multiple locations in a predetermined region of the environment, or a maximum jam probability of multiple locations in a predetermined region of the environment.

[0020] In some embodiments, using the jam neural network features to generate the factor jam probabilities includes providing the jam neural network features as a first input to a different jam machine learning model that operates on the jam neural network features and other features of the factors, other features of the vehicle, or both.

[0021] In some embodiments, the different jam machine learning model is a random decision forest.

[0022] In some embodiments, the method further comprises computing a respective initial factor-jam probability for each of the one or more factors, including: receiving data specifying one or more predicted paths for the factor, the predicted paths including one or more predicted future locations; receiving data specifying a respective predicted path probability for each of the one or more predicted paths for the factor, wherein the predicted path probability for a given predicted path characterizes a likelihood that the factor will travel on the predicted path; determining which of the predicted paths intersect a planned future location of the vehicle within a predetermined amount of time; and generating the respective initial factor-jam probability by combining the respective predicted path probabilities for those predicted paths that intersect the planned future location of the vehicle; and providing the initial factor-jam probability as a second input to the random decision forest.

[0023] In some embodiments, the method further comprises: for each of the one or more surrounding factors, extracting one or more features characterizing a predicted path of the surrounding factor from the predicted path of the surrounding factor, wherein the one or more features include one or more of: a time to nearest collision with the vehicle, a time to nearest collision with one or more other surrounding factors, one or more features related to traffic lights, one or more features related to stop signs, or a maximum lateral acceleration of the surrounding factor; and providing the one or more features characterizing the predicted path of each surrounding factor as a third input to the different-jam machine learning model.

[0024] In some embodiments, the method further comprises: determining one or more features characterizing the surrounding factor, wherein the one or more features include one or more of: a velocity of the surrounding factor, an acceleration of the surrounding factor, or an orientation of the surrounding factor; and providing the one or more features characterizing the surrounding factor as a fourth input to the different-jam machine learning model.

[0025] According to a second embodiment, there is provided a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method according to any one of the embodiments of the first embodiment.

[0026] According to a third embodiment, there is provided one or more non-transitory storage media storing instructions that are operable, when executed by one or more computers, to cause the one or more computers to perform the method according to any one of the embodiments of the first embodiment.

[0027] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0028] Some existing techniques for predicting cut-ins rely on predicting future paths of surrounding agents. For a given surrounding agent, a separate prediction system will generate one or more predicted future paths for the agent, each with a corresponding probability. The cut-in prediction system will then receive these predicted paths and infer a cut-in probability from them. Thus, rather than training a cut-in prediction system directly on positive and negative cut-in training examples, cut-in predictions are inferred indirectly. In some embodiments described in this specification, a cut-in neural network is trained directly on cut-in training examples generated by vehicles operating in the real world. Specifically, these cut-in training examples are collected over many driving hours by many different autonomous vehicles. This direct training can yield more accurate predictions and allow vehicles to predict potential cut-ins earlier, i.e., further ahead of potential impacts, compared to existing techniques.

[0029] Training a cut-in neural network directly on cut-in training examples can also greatly simplify the problem of predicting cut-ins; some existing techniques require cut-in prediction systems to explicitly model complex behaviors of surrounding agents. There are many failure modes inherent in some such techniques. As a specific example, some existing techniques can fail to predict that a surrounding agent performs an illegal maneuver that causes the agent to cut in front of the vehicle. Some embodiments of the cut-in neural network described in this specification have been shown to be able to capture this scenario.

[0030] In some embodiments described in this specification, the cut-in neural network can also learn nonlinear features of the surrounding environment that characterize interactions between one or more surrounding agents and the vehicle that are important for predicting a cut-in probability. The above-described existing techniques rely on predicted future paths of individual isolated agents when generating cut-in predictions for that agent and cannot account for nonlinear interactions.

[0031] In some embodiments described in this specification, the cut-in neural network is part of a larger cut-in prediction pipeline, where it is combined with other cut-in prediction engines to generate more accurate cut-in predictions than existing techniques.

[0032] In some embodiments described in this specification, the cut-in prediction system can generate a respective cut-in probability for each of one or more surrounding agents. This is an improvement over existing techniques, some of which can only generate predictions for a single surrounding agent, while others can only generate an overall probability of a cut-in occurring without specifying the particular surrounding agent responsible.

[0033] In some embodiments described in this specification, the cut-in neural network generates trajectory representation data using a data representation system that defines trajectories of a vehicle and one or more surrounding factors as a set of two-dimensional (2D) “lanes.” The cut-in neural network can process the trajectory representation data, e.g., using a convolutional neural network, to generate an output that defines a respective cut-in prediction for each surrounding factor. Defining trajectories as a set of 2D lanes is more compact than defining trajectories as a set of 3D lanes (e.g., a sequence of video frames). Moreover, unlike 1D representations, e.g., such as a feature vector, 2D lanes can be processed by multi-dimensional convolutional filters in a convolutional neural network. Thus, the cut-in neural network can consume less computational resources (e.g., memory, computational power, or both) by defining trajectories as a set of 2D lanes than if the on-vehicle system defined trajectories as a set of 3D lanes. Moreover, by processing the set of 2D lanes representing trajectories using multi-dimensional convolutional filters, the convolutional neural network can learn to exploit complex interactions between different spatial regions of the trajectory representation data, thereby generating more accurate cut-in predictions.

[0034] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a diagram of an example system.

[0036] Figure 2A is a block diagram of an example data representation system for generating vehicle trajectory data.

[0037] Figure 2B is a block diagram of an example data representation system for generating factor trajectory data.

[0038] Figure 3 is an illustration of an example process by which a data representation system can generate lanes.

[0039] Figure 4 is a block diagram of an example cut-in prediction system.

[0040] Figure 5 is an illustration of an example cut-in output corresponding to an environment.

[0041] Figure 6 is a flowchart of an example process for generating a cut-in output.

[0042] Figure 7 is a flowchart of an example process for extracting features from a cut-in output to generate factor cut-in probabilities.

[0043] Figure 8is a flowchart of an example process for generating initial factor cut-in probabilities.

[0044] Figure 9 is a flowchart of an example process for generating training examples for training a cut-in neural network.

[0045] Like reference numbers and designations in different drawings indicate like elements. DETAILED DESCRIPTION

[0046] This specification describes how a vehicle (e.g., an autonomous or semi-autonomous vehicle) can use a trained machine learning model (referred to in this specification as a“cut-in neural network”) to predict whether one or more surrounding factors are to cut in front of the vehicle. In this specification, to“cut in front of the vehicle” means to enter the vehicle’s planned path within an amount of time. For example, if the vehicle’s planned path includes a right turn, then a factor can cut in front of the vehicle by entering the vehicle’s planned path to travel after the right turn, even if the vehicle has not yet performed the right turn. In this specification, without loss of generality, a“surrounding factor” can refer to a vehicle, a bicycle, a pedestrian, a boat, a drone, or any other moving object. This specification also describes how training examples generated by the vehicle can be used to effectively train the cut-in neural network to make predictions accurately and reliably.

[0047] Figure 1 is a diagram of an example system 100. The system 100 includes an on-vehicle system 110 and a training system 120.

[0048] The on-vehicle system 110 is located on a vehicle 102. Figure 1 The vehicle 102 in is shown as a motor vehicle, but the on-vehicle system 102 can be located on any appropriate vehicle type. The vehicle 102 can be a fully autonomous vehicle that determines and executes fully autonomous driving decisions in order to navigate through an environment. The vehicle 102 can also be a semi-autonomous vehicle that uses predictions to assist a human driver. For example, if a prediction indicates that a human driver is going to collide with another vehicle, the vehicle 102 can autonomously apply brakes. The on-vehicle system 110 includes one or more sensor subsystems 140. The sensor subsystems 140 include a combination of components that receive reflections of electromagnetic radiation, such as lidar systems that detect reflections of laser light, radar systems that detect reflections of radio waves, and camera systems that detect reflections of visible light.

[0049] Sensor data generated by a given sensor generally indicates the distance, direction, and intensity of reflected radiation. For example, a sensor can emit one or more pulses of electromagnetic radiation in a particular direction, and can measure the intensity of any reflections and the time at which the reflections are received. Distance can be computed by determining the time taken between a pulse and its corresponding reflection. A sensor can continuously scan a particular space in angle, azimuth, or both. For example, scanning in azimuth can allow a sensor to detect multiple objects along the same line of sight.

[0050] The sensor subsystem 140 or other components of the vehicle 102 can also classify a set of one or more raw sensor measurements from one or more sensors as a measurement of another factor. A set of sensor measurements can be represented in any of a variety of ways, depending on the kind of sensor measurements captured. For example, each set of raw laser sensor measurements can be represented as a three-dimensional point cloud, with each point having an intensity and a location. In some implementations, the location is represented as a range and elevation pair. Each set of camera sensor measurements can be represented as an image patch, such as an RGB image patch.

[0051] Once the sensor subsystem 140 classifies one or more sets of raw sensor measurements as measurements of respective other factors, the sensor subsystem 140 compiles the raw sensor measurements into a collection of raw data 142 and transmits the raw data 142 to the data representation system 150.

[0052] The on-vehicle system 110 also includes a factor prediction system 144. For each surrounding factor detected by the sensor subsystem 140, the factor prediction system 144 generates a predicted factor path. The predicted factor path characterizes a path that the surrounding factor is predicted to take in the future. The factor prediction system transmits all predicted factor paths 146 to the data representation system 150.

[0053] The on-vehicle system 110 also includes a path planning system 160. The path planning system 160 generates a planned vehicle path 165, which characterizes a path that the vehicle 102 will take in the future. The path planning system 160 transmits the planned vehicle path 165 to the data representation system 150.

[0054] The data representation system 150 on the vehicle 102 also receives the raw sensor data 142 from the sensor system 140 and the predicted factor paths from the factor prediction system 144, and generates factor trajectory data 152. For each surrounding factor, the factor trajectory data 152 can characterize a current state at a current time point, previous states at one or more respective previous time points, and predicted future states at one or more respective future time points. The state of a factor at a given time point can include a location of the factor at the given time point, as well as values of a predetermined set of motion parameters. The motion parameters can include an orientation of the factor, a velocity of the factor, and an acceleration of the factor. Reference is made below to Figure 2B An example data representation system 150 and factor trajectory data 152 are described in more detail.

[0055] The data representation system 150 also receives the planned vehicle path 165 from the path planning system 160, and generates vehicle trajectory data 154. Similar to the factor trajectory data 152, the vehicle trajectory data 154 can characterize, for the vehicle, a current state at a current time point, previous states at one or more respective previous time points, and planned future states at one or more respective future time points. In some implementations, the vehicle trajectory data 154 can characterize multiple possible planned future states for each future time point. The data representation system 150 can invoke previous states of the vehicle 102 in order to generate the vehicle trajectory data 154. Reference is made below to Figure 2A An example data representation system 150 and vehicle trajectory data 152 are described in more detail.

[0056] The data representation system 150 provides the factor trajectory data 152 and the vehicle trajectory data 154 to a cut-in prediction system 170, also on the vehicle 102. The cut-in prediction system uses the trajectory data to generate a cut-in prediction 172 that predicts whether one or more surrounding factors will cut in front of the vehicle 102 within a predetermined amount of time, e.g., 3 seconds or 5 seconds. Reference is made below to Figure 4 This process is described in more detail.

[0057] The on-vehicle system 100 can provide the cut-in prediction 172 generated by the cut-in prediction system 170 to the path planning system 160, to the user interface system 174, or to both.

[0058] When the path planning system 160 receives the cut-in prediction 172, the path planning system 160 can use the cut-in prediction 172 to generate a new planned vehicle path that characterizes a path that the vehicle 102 will take in the future. For example, the cut-in prediction 172 can contain a prediction that a particular surrounding agent is likely to cut in front of the vehicle 102, potentially causing a collision. In this example, the path planning system 160 can generate a new planned vehicle path that avoids the potential collision. That is, the path planning system 160 can modify the planned future trajectory of the vehicle 102 to avoid the potential collision that can result from the likely cut-in from another agent.

[0059] When the user interface system 174 receives the cut-in prediction 172, the user interface system 174 can use the cut-in prediction 172 to present information to the driver of the agent 102 to help the driver safely operate the agent 102. The user interface system 174 can present the information to the driver of the agent 102 through any appropriate means, such as an audio message sent through a speaker system of the agent 102 or an alert displayed on a visual display system in the agent, such as an LCD display on the dashboard of the agent 102. In a particular example, the cut-in prediction 172 can contain a prediction that a particular surrounding agent is likely to cut in front of the agent 102, potentially causing a collision. In this example, the user interface system 174 can present an alert message to the driver of the agent 102 with instructions to adjust the trajectory of the agent 102 to avoid the collision or to notify the driver of the agent of the potential collision with the particular surrounding agent.

[0060] To generate the cut-in prediction 172, the cut-in prediction system 170 can use the trained parameter values 135 obtained from the cut-in model parameter store 130 in the training system 120.

[0061] The training system 120 is typically hosted within a data center 124, which can be a distributed computing system with hundreds or thousands of computers in one or more locations.

[0062] The training system 120 includes a training data store 180 that stores all training data used to train the parameter values of the cut-in prediction system 170. The training data store 180 receives raw training examples from agents operating in the real world. For example, the training data store 180 can receive raw training examples 175 from the agent 102, as well as one or more other agents in communication with the training system 120. The raw training examples 175 can be processed by the training system 120 to generate new training examples. The raw training examples 175 include agent trajectory data 152 and vehicle trajectory data 154, which together can be used as input to a new training example. The raw training examples 175 also include outcome data characterizing whether one or more surrounding agents actually cut in front of the vehicle 102 within a predetermined amount of time of generating the cut-in prediction 172. This outcome data can be used to generate ground truth output for the new training example. The process for generating new training examples from raw training examples is described in more detail below. Figure 9 The process for generating new training examples from raw training examples is described in more detail below.

[0063] The training data store 180 provides the training examples 185 to a training engine 190 also housed in the training system 120. The training engine uses the training examples 185 to update the model parameters to be used by the cut-in prediction system 170, and provides the updated model parameters 195 to the cut-in model parameters store 130. Once the parameter values of the cut-in prediction system 170 have been fully trained, the training system 120 can transmit the trained parameter values 135 to the cut-in prediction system 170, e.g., over a wired or wireless connection.

[0064] Figure 2A is a block diagram of an example data representation system 200 for generating vehicle trajectory data 220. The data representation system 200 is an example of a system of one or more computers implementing the systems, components, and techniques described below.

[0065] The data representation system 200 processes a planned vehicle path 210 characterizing a path that a vehicle is planned to travel to generate vehicle trajectory data 220. The data representation system 200 can also invoke a previous vehicle path characterizing a path that the vehicle has previously traveled to generate the vehicle trajectory data 220.

[0066] The vehicle trajectory data 220 consists of multiple“lanes.” Each lane is a two-dimensional array of data values representing a“top-down” view of the environment in the vicinity of the vehicle. Each position in each lane corresponds to a respective location in the environment and can be indexed by respective coordinates. Each lane has the same dimensions (i.e., the same number of rows and columns), the same top-down view of the environment, and is aligned (i.e., registered) with the others. That is, positions indexed by the same coordinates in different lanes correspond to the same location in the environment. In other words, corresponding positions in different lanes correspond to the same location in the environment.

[0067] To represent the trajectory of the vehicle, the vehicle trajectory data 220 includes a vehicle time lane and respective vehicle motion parameter lanes corresponding to each of a predetermined number of motion parameters. Each motion parameter characterizes a respective property of the motion of the vehicle, such as the velocity, acceleration, or orientation of the vehicle. As Figure 2A As depicted, the vehicle trajectory data 220 can include a vehicle time lane 230, a vehicle velocity lane 224, a vehicle acceleration lane 226, and a vehicle orientation lane 228.

[0068] The vehicle time lane and the vehicle motion parameter lanes represent both a previous trajectory of the vehicle up to and including a current time point, and a planned future trajectory of the vehicle after the current time point, which is derived from the planned vehicle path 210. More specifically, the planned future trajectory of the vehicle specifies, for each of a plurality of future time points after the current time point, a location in the environment that can be occupied by the vehicle at the future time point. For each of the plurality of future time points, the planned future trajectory also specifies a possible value for each of a predetermined number of motion parameters characterizing the motion of the vehicle at the future time point.

[0069] In some implementations, the planned vehicle path 210 includes multiple possible future paths of the vehicle. That is, for each future time point, the planned vehicle path 210 specifies multiple possible locations in the environment that can be occupied by the vehicle at the future time point, and multiple possible values for each of the motion parameters. In these cases, the generated vehicle trajectory data 220 can characterize each possible future trajectory of the vehicle.

[0070] The following description will refer to both the previous vehicle trajectory up to and including the current time point and the planned future vehicle trajectory after the current time point together as the“full trajectory” of the vehicle.

[0071] The vehicle time channel represents respective time points at which the vehicle occupies different locations in the environment throughout the full trajectory of the vehicle. Each motion parameter channel represents values of a respective motion parameter that characterizes the motion of the vehicle as the vehicle occupies different locations in the environment throughout the full trajectory of the vehicle. Specifically, for each location in the environment that is occupied by the vehicle, the data value in the vehicle time channel corresponding to that location defines a time point at which the vehicle occupies that location. In addition, the data values in each respective vehicle motion parameter channel corresponding to that location define values of the respective motion parameter that characterizes the motion of the vehicle as the vehicle occupies that location in the environment.

[0072] In general, the environment can include multiple locations that are not included in the previous trajectory or the candidate future trajectory of the agent. For these locations, the data representation system 200 can set the data values in the vehicle time channel and the vehicle motion parameter channels corresponding to these locations to default values (e.g., a value of 0 or a value of -1).

[0073] When the vehicle occupies the same location in the environment at multiple time points, the data representation system 200 can set the respective data values in the vehicle time channel and the motion parameter channels corresponding to that location to any appropriate values. For example, the data representation system 200 can set the respective data values in the vehicle time channel and the motion parameter channels corresponding to that location according to the last time point at which the vehicle occupies that location. That is, the data representation system 200 can set the data value in the vehicle time channel corresponding to that location to define the last time point at which the vehicle occupies that location. In addition, the data representation system 200 can set the respective data values in the vehicle motion parameter channels to define values of the respective motion parameter that characterizes the motion of the vehicle as the vehicle last occupies that location.

[0074] In some implementations, in addition to the time channel and the motion parameter channels, the data representation system 200 can include other channels in the vehicle trajectory data 220. For example, the vehicle trajectory data 220 can include a road map channel, a vehicle localization channel, or both. The road map channel represents known geometry of the environment in the vicinity of the vehicle. For example, the road map channel can represent locations of different features of the environment, including: roads, different lanes in the road, crosswalks, traffic lights, construction zones, school zones, etc. The vehicle localization channel represents the location of the vehicle in the environment at the current time point (e.g., defined by a bounding box). In some implementations, the vehicle localization channel can represent the location of the vehicle in the environment at the last time point at which the vehicle occupies a location in the environment. In some implementations, the vehicle localization channel can represent the location of the vehicle in the environment at the last time point at which the vehicle occupies a location in the environment that is included in the candidate future trajectory of the vehicle. Figure 2A In the example depicted in FIG. 2, the road map channel and the vehicle localization channel are superimposed on a single channel 222.

[0075] Figure 2B FIG. 2 is a block diagram of an example data representation system 200 for generating agent trajectory data 260. The data representation system 200 is an example of a system implemented as computer programs on one or more computers in one or more locations that implement the systems, components, and techniques described below.

[0076] The data representation system 200 processes the raw sensor data 240 and the predicted factor path data 250 to generate factor trajectory data 260 that characterizes factors in the environment in the vicinity of the vehicle.

[0077] To represent the trajectories of the surrounding factors, the factor trajectory data 260 includes a factor time channel and respective factor motion parameter channels corresponding to each of a predetermined number of motion parameters. As Figure 2B As depicted, the factor trajectory data 260 can include a factor time channel 270, a factor velocity channel 364, a factor acceleration channel 266, and a factor orientation channel 268.

[0078] The factor time channel and the factor motion parameter channels jointly represent a previous trajectory of the factor up to a current time point and a predicted future trajectory of the factor after the current time point, which is derived from the predicted factor path data 250. The predicted future trajectory of the factor specifies, for each of a plurality of future time points after the current time point, a location in the environment that can be occupied by the factor at the future time point. For each of the plurality of future time points, the predicted future trajectory also specifies a possible value of each of a predetermined number of motion parameters that characterizes the motion of the factor at the future time point. The following description will refer to the previous factor trajectory up to and including the current time point and the planned future factor trajectory after the current time point together as the “full trajectory” of the factor.

[0079] The factor time channel jointly represents respective time points at which the factor occupies different locations in the environment in the full trajectory of the factor. Each factor motion parameter channel represents values of the respective motion parameter that characterizes the motion of the factor when the factor occupies different locations in the environment in the full trajectory of the factor. Specifically, for each location in the environment that is occupied by one of the factors, the data value in the factor time channel corresponding to the location defines the time point at which the factor occupies the location. In addition, the data values in each respective factor motion parameter channel corresponding to the location define values of the respective motion parameter that characterizes the motion of the factor when the factor occupies the location in the environment.

[0080] In general, the environment can include a plurality of locations that are not included in the previous trajectory or the predicted future trajectory of any factor. For these locations, the data representation system 200 can set the data values in the factor time channel and the factor motion parameter channels corresponding to these locations to default values (e.g., a value of 0 or a value of -1).

[0081] When a factor occupies the same location in the environment at multiple time points, the data representation system 200 can set the respective data values in the factor time channel and the factor motion parameter channel corresponding to that location to any suitable value. For example, the data representation system 200 can set the respective data values in the factor time channel and the factor motion parameter channel corresponding to that location according to the last time point at which one of the factors occupied that location.

[0082] In some implementations, a factor position channel (e.g., factor position channel 262) jointly represents the location (e.g., defined by a bounding box) of each vehicle in the environment at the current time point.

[0083] By jointly representing the trajectories of factors using a single factor time channel and a single factor motion parameter corresponding to each motion parameter, the data representation system 200 can generate factor trajectory data 260 having a predetermined dimension, regardless of the (variable) number of factors. In this way, the factor trajectory data 260 generated by the data representation system 200 can be easily processed by a platoon prediction system configured to process factor trajectory data 260 of a predetermined dimension.

[0084] Referring to Figure 2A and 2B In some implementations, the data representation system 200 jointly generates the vehicle trajectory data 220 and the factor trajectory data 260 by aligning and concatenating the generated channels at a channel-by-channel level, as depicted in 280 in Figure 2A and Figure 2B That is, the combined trajectory data is generated by concatenating all of the generated channels. By implicitly representing the respective trajectories of vehicles and factors in this format, the data representation system 200 can generate a representation that is both compact and can be efficiently processed by a platoon prediction system. In particular, the platoon prediction system can include a convolutional neural network. The platoon prediction system can process the combined trajectory data using convolutional neural network layers defined by multi-dimensional (e.g., two-dimensional or three-dimensional) convolutional filters, thereby enabling the platoon prediction system to learn complex spatial relationships between different spatial regions of the environment.

[0085] Figure 3 is an illustration of an example process by which the data representation system can generate the vehicle time channel 302, the vehicle velocity channel 304, and the vehicle orientation channel 306, which represent the trajectory of the vehicle up to and including the current time point and the planned future trajectory of the vehicle after the current time point.

[0086] The data defining the previous trajectory of the vehicle and the planned future trajectory of the vehicle is represented in tabular format 308. Each row of the table defines a time point, a location in the environment occupied by the vehicle at that time point, a velocity of the vehicle at that time point, and an orientation of the vehicle at that time point.

[0087] For example, the first row of table 308 defines that at time point -3 (e.g., 3 seconds before the current time point), the location occupied by the vehicle in the environment is defined by coordinates [-3, -2] in the lane reference frame, the speed of the vehicle is 45 mph, and the orientation of the vehicle is 45 degrees (e.g., clockwise from north). The coordinates [-3, -2] are expressed in the lane reference frame and can be mapped to express a location in the environment reference frame. For example, the coordinates in the lane reference frame can represent 5-foot increments in a north-south-east-west environment reference frame centered on the vehicle. In this example, the coordinates [-3, -2] in the lane reference frame can represent a location in the environment 15 feet west and 10 feet south of the location occupied by the vehicle at the current time point. As another example, the last row of table 308 defines that at time point +2 (e.g., 2 seconds after the current time point), the location occupied by the vehicle (i.e., according to the planned future trajectory) is defined by coordinates [2, 0] in the lane reference frame, the speed of the agent is 49 mph, and the orientation of the agent is 90 degrees.

[0088] The data representation system sets the data values in the vehicle time lanes corresponding to locations [-3, -2], [-2, -1], [-1, 0], [0, 0], [1, 0], and [2, 0] to the respective time points -3, -2, -1, 0, +1, and +2 at which the vehicle occupies the respective locations. For clarity, the data values in the vehicle time lanes corresponding to other locations are shown as blank. Typically, the data representation system sets the data values in the vehicle time lanes corresponding to other locations to a default value (e.g., a value of 0 or a value of -1). Similarly, the data representation system sets the data values in the vehicle speed lanes corresponding to locations [-3, -2], [-2, -1], [-1, 0], [0, 0], [1, 0], and [2, 0] to the respective speed values 45, 47, 51, 49, 49, and 49 that define the speed of the vehicle (in mph) when the vehicle occupies the respective locations. Similarly, the data representation system sets the data values in the vehicle orientation lanes corresponding to locations [-3, -2], [-2, -1], [-1, 0], [0, 0], [1, 0], and [2, 0] to the respective orientation values 45, 45, 90, 90, 90, and 90 that define the orientation of the vehicle (in degrees) when the vehicle occupies the respective locations.

[0089] In some implementations, the data representation system determines the lane reference frame such that the location occupied by the vehicle at the current time point corresponds to the data value in the “center” of each lane. Generating trajectory data with a consistent lane reference frame can facilitate training a platooning prediction system on the trajectory data.

[0090] Figure 4is a block diagram of an example cut-in prediction system 400. Cut-in prediction system 400 is an example of a system implemented as computer programs on one or more computers in one or more locations that implement the systems, components, and techniques described below.

[0091] Data representation system 410 provides factor trajectory data 412 and vehicle trajectory data 414 to cut-in prediction system 400. For example, the data representation system can provide trajectory data in a channel format as described above with reference to Figure 2A 、 Figure 2B and Figure 3 .

[0092] Cut-in neural network 430 processes factor trajectory data 412 and vehicle trajectory data 414 to generate cut-in output 435. In some implementations, cut-in neural network 430 is a convolutional neural network. Such a convolutional neural network can learn to exploit complex interactions between different spatial regions of the trajectory data by processing the two-dimensional channels using multi-dimensional convolutional filters, generating more accurate predictions when the trajectory data is represented as two-dimensional arrays (e.g., channels) as depicted in Figure 2A and Figure 2B .

[0093] Cut-in output 435 can include respective cut-in probabilities for one or more locations in the environment. The cut-in probability for a location occupied by a surrounding factor at a current time point characterizes the likelihood that the respective surrounding factor will cut in front of the vehicle within a predetermined amount of time. For example, a cut-in probability of 0.9 can represent that the cut-in neural network predicts that there is a 90% likelihood that the surrounding factor will cut in front of the vehicle within the predetermined amount of time.

[0094] In some implementations, cut-in output 435 is a two-dimensional array of data values similar to the input trajectory data depicted in Figure 2A and Figure 2B , where the array represents a “top-down” view of the environment near the vehicle. Each position in the array corresponds to a respective location in the environment, and the value at each position corresponds to the cut-in probability for the respective location. An example two-dimensional cut-in output is described in more detail below with reference to Figure 5 .

[0095] As a particular example, cut-in neural network 430 can be a convolutional neural network with several convolutional layers and several max-pooling layers that reduce the size of the input array. These convolutional layers can be followed by one or more fully connected layers that can learn non-linear features of the input array. The network can then reshape the output of the fully connected layers into an output array. Finally, the network can have several transpose convolutional layers that increase the size of the output array, such that cut-in output 435 is an array of the same size as the input array.

[0096] In some implementations, the cut-in output 435 is the final output of the cut-in prediction system 400, and the cut-in output is used by the vehicle to make a strategic decision, e.g., whether to change course to avoid a collision with a surrounding agent that will cut in. In some other implementations, such as the example in Figure 4 the cut-in output 435 is used as input to a different cut-in machine learning model 450.

[0097] The cut-in machine learning model 450 takes as input the cut-in output 435 (or data derived from the cut-in output 435) and other features (e.g., features of the surrounding agents and / or other features of the vehicle), and generates as output an agent cut-in probability 455. The agent cut-in probability 455 characterizes the probability that the corresponding agent will cut in front of the vehicle within a predetermined amount of time.

[0098] In some implementations, the cut-in machine learning model 450 extracts features from the cut-in output 435, and uses the extracted features to generate the agent cut-in probability. This process is described in more detail below with reference to Figure 7 .

[0099] In some implementations, the cut-in prediction system 400 also includes an initial cut-in prediction engine 440 that generates an initial agent cut-in probability 445 that is also used as input to the cut-in machine learning model 450. In some such implementations, the cut-in machine learning model 450 extracts further features from the initial agent cut-in probability 445, and uses the further extracted features as input. This process is described in more detail below with reference to Figure 8 .

[0100] Figure 5 is an illustration of an example cut-in output 500 corresponding to the environment 502.

[0101] The environment 502 contains a vehicle 510 and three surrounding agents 504, 506, and 508. Agent 504 appears likely to cut in front of the vehicle 510, while agents 506 and 508 appear unlikely to cut in front of the vehicle 510.

[0102] The cut-in output 500 is an example two-dimensional array generated by a trained cut-in neural network (e.g., the cut-in neural network 430 in Figure 4 . The array represents a “top-down” view of the environment 502, where each location corresponds to a respective location in the environment 502. For example, location 514 corresponds to the location occupied by agent 504, location 516 corresponds to the location occupied by agent 506, and location 518 corresponds to the location occupied by agent 508.

[0103] In some implementations, the input to the cut-in neural network is represented as a two-dimensional array (e.g., the cut-in output 435 in Figure 2A and Figure 2BIn some implementations of the trajectory data (depicted in FIG. 6), the jam output array has the same dimensions as the input trajectory array, such that a position in the output array represents the same location in the environment as the corresponding position in the input array.

[0104] The value at each position corresponds to a jam probability for the respective location. In some implementations, an image depicting the jam output can be created, where a respective block of one or more pixels corresponds to each position in the array, and a higher jam probability is depicted as a darker grayscale value for the respective block of pixels. Such an image is useful during training of the jam neural network, for example, to allow a user to visually inspect the progress of the training network.

[0105] For example, position 514 has a dark grayscale value corresponding to a high jam probability, because the corresponding factor 504 has a high likelihood of jamming. Positions 516 and 518 have light grayscale values corresponding to low jam probabilities, because the corresponding factors 506 and 508 have low likelihoods of jamming.

[0106] In some implementations, positions in the jam output representing locations in the environment not occupied by surrounding factors can have a default value, such as 0, or a white or black grayscale value. During the training process, the jam neural network learns to generate the default value for locations not occupied by factors, because the ground truth output is formatted in this way. In some other implementations, the jam neural network can be trained using a loss function that ignores positions in the jam output representing locations in the environment not occupied by surrounding factors, so the jam neural network can assign any value to these locations.

[0107] Figure 6 is a flowchart of an example process 600 for generating a jam output. For convenience, the process 600 will be described as performed by a system of one or more computers located in one or more locations. For example, a system of one or more computers (e.g., the on-vehicle system 110 of FIG. 1) appropriately programmed in accordance with this specification can perform the process 600. Figure 1 The process 600 can be performed by the on-vehicle system 110 of FIG. 1.

[0108] The system obtains factor trajectory data for one or more factors in the environment (step 602). As described above, the factor trajectory data characterizes, for each factor surrounding the vehicle, a current state at a current time point, a previous state at one or more respective previous time points, and a predicted future state at one or more respective future time points. The state of a factor at a given time point can include a location of the factor at the given time point, as well as values for a predetermined set of motion parameters. Example factor trajectory data is described above with reference to FIG. 5. Figure 2B

[0109] ​The system obtains vehicle trajectory data (step 604). As described above, the vehicle trajectory data characterizes, for a vehicle, a current state at a current time point, a previous state at one or more respective previous time points, and a planned future state at one or more respective future time points. The above reference Figure 2A An example vehicle trajectory data is described.

[0110] The system processes the factor trajectory data and the vehicle trajectory data to generate a jam probability output (step 606). The jam probability output characterizes jam probabilities for one or more locations in the environment. The above reference Figure 5 An example jam probability output is described.

[0111] Figure 7 is a flowchart of an example process 700 for extracting features from a jam probability output to generate factor jam probabilities. For convenience, the process 700 will be described as performed by a system of one or more computers located in one or more locations. For example, a jam prediction system (e.g., the jam prediction system 400 of Figure 4 may perform the process 700, appropriately programmed in accordance with this specification.

[0112] The system extracts features from the jam probability output (step 702). The jam probability output is the final output of a jam neural network (e.g., the jam neural network 430 of Figure 4 The features extracted from the jam probability output can include average jam probabilities for locations in a given region of the environment. For example, the system can compute an average jam probability for all locations within 10 feet of a given surrounding factor, and an average jam probability for all locations within 20 feet of the given surrounding factor, and include these two averages as features. The features can also include maximum jam probabilities for locations in a given region of the environment. For example, the system can compute a maximum jam probability for all locations within 10 feet of a given surrounding factor, and a maximum jam probability for all locations within 20 feet of the given surrounding factor, and include these two maximums as features. In some implementations, the system extracts a set of features for each surrounding factor using a portion of the jam probability output that characterizes a region of the environment that includes the surrounding factor.

[0113] The system extracts features from intermediate outputs of the jam neural network (step 704). For example, the features extracted from the intermediate outputs can include outputs of one of the intermediate layers of the jam neural network, as the intermediate layers can include useful information learned during training. The features extracted from the jam probability output and the features extracted from the intermediate outputs are collectively referred to as “jam neural network features”

[0114] The system provides the jam neural network features to another jam machine learning model (706). The jam machine learning model is different from the jam neural network, e.g., the jam neural network 430 of Figure 4the clogging machine learning model 450. In some implementations, the clogging machine learning model also takes as input the initial factor clogging probabilities generated by another initial clogging prediction engine (e.g., the initial clogging prediction engine 440 in the system 400). In some implementations, the clogging machine learning model is a random decision forest or other decision tree-based machine learning model. Figure 4

[0115] The system uses the clogging machine learning model to generate factor clogging probabilities for one or more surrounding factors (step 708). As described above, a factor clogging probability characterizes the probability that the corresponding surrounding factor will clog in front of the vehicle within a predetermined amount of time.

[0116] Figure 8 is a flowchart of an example process 800 for generating initial factor clogging probabilities. For convenience, the process 800 will be described as being performed by a system of one or more computers located in one or more locations. For example, an initial clogging prediction engine (e.g., the initial clogging prediction engine 440 of the system 400) appropriately programmed according to the present specification can perform the process 800. Figure 4

[0117] The system receives one or more predicted paths of a surrounding factor near a vehicle (step 802). A predicted path characterizes a possible path that the surrounding factor will take, as predicted by the vehicle.

[0118] The system receives a predicted path probability for each predicted path of the surrounding factor (step 804). A respective predicted path probability for a predicted path characterizes the likelihood that the surrounding factor will take that predicted path.

[0119] The system determines which predicted paths will intersect the planned trajectory of the vehicle within a predetermined amount of time (step 806). The system can receive the planned trajectory of the vehicle from a path planning system of the vehicle (e.g., the path planning system 160 of the system 400). Figure 1

[0120] The system generates an initial factor clogging probability for the surrounding factor (step 808). In some implementations, the system sums the corresponding predicted path probabilities for the predicted paths that intersect the planned path of the vehicle and outputs that sum as the initial factor clogging probability. For example, there can be four predicted paths: path A with a probability of 0.50, path B with a probability of 0.30, path C with a probability of 0.10, and path D with a probability of 0.10. If only path C and path D intersect the planned path of the vehicle, then the system will determine that the initial factor clogging probability is 0.20.

[0121] The system provides the initial clogging probability to another clogging machine learning model (step 810). The clogging machine learning model is different from the initial clogging prediction engine, e.g., the initial clogging prediction engine 440 of the system 400. Figure 4 ​​​The factor insertion machine learning model 450. The factor insertion machine learning model generates factor insertion probabilities using the initial factor insertion probabilities.

[0122] The factor insertion machine learning model can also use other features in conjunction with the initial factor insertion probabilities to generate the factor insertion probabilities. For example, the factor insertion machine learning model can use features extracted from the factor insertion neural network (e.g., Figure 4 The factor insertion neural network 430) to generate the factor insertion probabilities.

[0123] In some implementations, the system can extract features from data characterizing each surrounding factor and provide the extracted features to the factor insertion machine learning model. For example, the system can extract a current velocity of each surrounding factor, a current acceleration of each surrounding factor, and / or an orientation of each surrounding factor.

[0124] In some implementations, the system can extract features from the predicted path of a given surrounding factor and provide the extracted features to the factor insertion machine learning model. As a particular example, the system can extract a feature characterizing whether the given surrounding factor is likely to collide with one or more other surrounding factors. The system can also extract a closest collision time at which the given surrounding factor is likely to collide with a vehicle or another surrounding factor. As another particular example, the system can extract traffic light information characterizing one or more traffic lights on the predicted path of the given surrounding factor. In particular, the system can extract a feature characterizing how far away the traffic light is from the given surrounding factor and whether there is a crosswalk at the traffic light. As another particular example, the system can extract parking sign information characterizing one or more parking signs on the predicted path of the given surrounding factor. In particular, the system can extract a feature characterizing how far away the parking sign is from the given surrounding factor and whether there is a crosswalk at the parking sign. As another particular example, the system can extract a feature characterizing a maximum lateral acceleration of the given surrounding factor.

[0125] Figure 9 is a flowchart of an example process 900 for generating training examples for training a factor insertion neural network. For convenience, the process 900 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., Figure 1 The training system 120, appropriately programmed in accordance with this specification, can perform process 900.

[0126] The system obtains raw training examples (step 901). The system can obtain the raw training examples from vehicles navigating through the real world. As described above with reference to Figure 1 The raw training examples include three components: i) vehicle trajectory data, ii) factor trajectory data, and iii) outcome data.

[0127] The system extracts vehicle trajectory data (step 902). The vehicle trajectory data is from the viewpoint of the vehicle at the first time, and characterizes a current state at the first time point and one or more previous states at respective previous time points before the first time point. As previously described, a state of an agent at a given time point can include a location of the agent at the given time point, and values of a predetermined set of motion parameters. The vehicle trajectory data also includes a first planned path from the viewpoint of the vehicle at the first time point. That is, the first planned path characterizes a path that the vehicle has intended to take at the first time point. In some embodiments, the vehicle trajectory data includes reference Figure 2A to the described time and motion channels. In some other embodiments, the system generates the time and motion channels using data in the original training examples. The vehicle trajectory data is part of the input for the training examples.

[0128] The system extracts agent trajectory data (step 904). The agent trajectory data is also from the viewpoint of the vehicle at the first time, and characterizes a current state at the first time point and one or more previous states at respective previous time points before the first time point for each surrounding agent of the vehicle. The agent trajectory data also includes a predicted path of the respective surrounding agent from the viewpoint of the vehicle at the first time point. That is, the predicted path of the respective agent characterizes a path that the vehicle believes the agent will take at the first time point. In some embodiments, the agent trajectory data includes reference Figure 2B to the described time and motion channels. In some other embodiments, the system generates the time and motion channels using data in the original training examples. The agent trajectory data is part of the input for the training examples.

[0129] The system extracts outcome data (step 906). The outcome data characterizes whether one or more respective surrounding agents actually cut into the first planned path within a predetermined amount of time after the first time point. For example, if the first time point is time t, and the predetermined amount of time is 3 time steps, the outcome data characterizes whether the respective agent cut into the first planned path between time t and time t+3.

[0130] Importantly, the outcome data does not characterize whether the respective agent actually collided with the vehicle, or whether the respective agent cut into a second planned path from the viewpoint of the vehicle at time t+1 or t+2. This is because the vehicle can have changed its planned path between time t and time t+3, perhaps to avoid a collision with the agent. This does not change the fact that the agent would have cut into the first planned path of the vehicle if the vehicle had not changed course. The system operates exclusively from the viewpoint of the first time t, determining whether a surrounding agent will cut into the first planned path of time t between time t and time t+3.

[0131] In particular, if the surrounding factor partially but not completely overlaps with the planned path of the vehicle, e.g., if a portion of the surrounding factor (e.g., the front bumper of the factor) enters the lane of the vehicle, but the entire surrounding factor does not enter the lane, the result data can characterize that the corresponding surrounding factor did indeed cut into the first planned path of the vehicle. If the surrounding factor enters a given portion of the planned path of the vehicle after the vehicle has already passed through the given portion of the planned path, e.g., if the surrounding factor enters a portion of the planned path that the vehicle occupied at time t+1 at time t+2, the result data can characterize that the corresponding surrounding factor did not cut into the first planned path of the vehicle.

[0132] The system uses the output data to generate a ground truth output (step 908). In some implementations, the ground truth output is a two-dimensional array of data values, where each position in the array corresponds to a respective location in the environment, and the value at each position corresponds to a cut-in label for the respective location. The cut-in label for a given position in the ground truth output can be a value within a certain range, e.g., [0, 1] or [80, 250]. In particular, for positions that correspond to locations of factors that cut into the first planned path of the vehicle, the system can set the maximum value of the cut-in label, e.g., 1 for the range [0, 1] or 250 for the range [80, 250]. In some implementations, for positions that correspond to locations of factors that did not cut into the first planned path of the vehicle, the system sets some small non-zero value, e.g., 0.05 or 0.01 for the range [0, 1] or 80 for the range [80, 250]. This is done so that the ground truth output encodes the fact that there was a factor at the corresponding location, and the fact that the factor did not cut into the front of the vehicle. This helps the cut-in neural network learn the format that the cut-in output should take, i.e., non-zero values for positions that correspond to locations of factors, and provides a richer error signal for updating the network parameters. In some other implementations, for positions that correspond to locations of factors that did not cut into the first planned path of the vehicle, the system sets the minimum value, e.g., 80 for the range [80, 250]. For positions that correspond to locations that were not occupied by a surrounding factor, the system sets a default value, e.g., 0.

[0133] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0134] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, a special purpose logic device, e.g., a GPU or another specialized processing subsystem, or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0135] A computer program, which can also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and

[0136] The processes and logic flows described in this specification can be performed by one or more programmable computers, each computer including a processor, a storage memory, communication interface, and numerous other elements and functionalities typical of today's computing devices. The processes and logic flows can also be performed by special purpose logic circuitry, for example, an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. The processes and logic flows can be performed by mechanical computer processors, such as optical, electrical, hydraulic, pneumatic, electro-mechanical, and the like.

[0137] The computer used to run the program can be of various architectures from a general purpose computer to a dedicated computer. Generally, the computer will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data. These devices can be, for example, magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, for example, a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0138] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0139] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse, trackball, or a presence-sensitive display or other surface by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device of the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0140] Although this specification contains many specific implementation details, these should not be construed as limiting the scope of any inventions or of what can be claimed, but as describing specifications that can be employed by particular embodiments of the inventions. Some features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0141] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order nor that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0142] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain circumstances, multitasking and parallel processing can be advantageous.

Claims

1. A method comprising: obtaining factor trajectory data for one or more factors in an environment, the factor trajectory data for each factor comprising: a current location and a current value of a predetermined set of motion parameters of the factor; and a previous location and a previous value of the predetermined set of motion parameters of the factor at each of one or more previous time points; obtaining vehicle trajectory data for a vehicle in the environment, the vehicle trajectory data comprising: a current location and a current value of the predetermined set of motion parameters of the vehicle; a previous location and a previous value of the predetermined set of motion parameters of the vehicle at each of one or more previous time points; and a planned future location and a planned future value of the predetermined set of motion parameters of the vehicle at each of one or more future time points; and processing, using a neural network, a network input generated from the factor trajectory data and the vehicle trajectory data to generate a jam output, wherein the jam output comprises a respective jam probability for each of a plurality of locations in the environment, wherein the respective jam probability for each location that is a current location of one of the one or more factors characterizes a likelihood that the factor at the current location will intersect with a planned future location of the vehicle within a predetermined amount of time.

2. The method of claim 1, wherein, the factor trajectory data further comprises a predicted future location and a predicted future value of the predetermined set of motion parameters of the factor at each of the one or more future time points.

3. The method of claim 1 or 2, wherein, for each of the one or more future time points, the vehicle trajectory data further comprises a plurality of planned future locations and a plurality of planned future values for each of the predetermined set of motion parameters of the vehicle.

4. The method of claim 1 or 2, wherein, the plurality of locations comprises non-factor locations that are not any of the current locations of the one or more factors, and wherein the neural network has been trained to generate null probabilities for the non-factor locations.

5. The method of claim 1 or 2, wherein, the network input comprises at least one of: an image representing road lanes in the environment; or an image representing a bounding box of the one or more factors in the environment.

6. The method of claim 1 or 2, wherein: the network input comprises a respective concatenation of a plurality of channels corresponding to each factor and the vehicle, and wherein in each concatenation: each channel is represented as a two-dimensional array of data values; each position in each channel corresponds to a respective location in the environment; corresponding positions in different channels correspond to the same location in the environment; the channels comprise a time channel and a respective motion channel corresponding to each motion parameter in the predetermined set of motion parameters; and for each particular location in the environment occupied by a factor or the vehicle at a particular time point: a position in the time channel corresponding to the particular location defines the particular time point; and for each motion channel, a position in the motion channel corresponding to the particular location defines a value of the motion parameter of the motion channel corresponding to the particular time point.

7. The method of claim 1 or 2, wherein, the predetermined set of motion parameters comprises at least one of: an orientation of the factor or the vehicle at the time point, a velocity of the factor or the vehicle at the time point, or an acceleration of the factor or the vehicle at the time point.

8. The method of claim 1 or 2, wherein, the jam output comprises a two-dimensional array of data values, wherein each position in the array corresponds to a respective location in the environment, and wherein the data values each characterize a jam probability for the respective location. the factor trajectory data further comprises a predicted future location and a predicted future value of the predetermined set of motion parameters of the factor at each of the one or more future time points. for each of the one or more future time points, the vehicle trajectory data further comprises a plurality of planned future locations and a plurality of planned future values for each of the predetermined set of motion parameters of the vehicle. the plurality of locations comprises non-factor locations that are not any of the current locations of the one or more factors, and wherein the neural network has been trained to generate null probabilities for the non-factor locations. the network input comprises at least one of: an image representing road lanes in the environment; or an image representing a bounding box of the one or more factors in the environment.

6. The method of claim 1 or 2, wherein: the network input comprises a respective concatenation of a plurality of channels corresponding to each factor and the vehicle, and wherein in each concatenation: each channel is represented as a two-dimensional array of data values; each position in each channel corresponds to a respective location in the environment; corresponding positions in different channels correspond to the same location in the environment; the channels comprise a time channel and a respective motion channel corresponding to each motion parameter in the predetermined set of motion parameters; and for each particular location in the environment occupied by a factor or the vehicle at a particular time point: a position in the time channel corresponding to the particular location defines the particular time point; and for each motion channel, a position in the motion channel corresponding to the particular location defines a value of the motion parameter of the motion channel corresponding to the particular time point. the predetermined set of motion parameters comprises at least one of: an orientation of the factor or the vehicle at the time point, a velocity of the factor or the vehicle at the time point, or an acceleration of the factor or the vehicle at the time point. the jam output comprises a two-dimensional array of data values, wherein each position in the array corresponds to a respective location in the environment, and wherein the data values each characterize a jam probability for the respective location.

9. The method of claim 1 or 2, further comprising: generating cut-in neural network features, including one or more of: extracting features from the cut-in output, or extracting features from one or more intermediate outputs of the neural network; and using the cut-in neural network features to generate factor cut-in probabilities for one or more factors in the environment.

10. The method of claim 9, wherein, the features extracted from the cut-in output include one or more of: average cut-in probabilities for a plurality of locations in a predetermined region of the environment, or maximum cut-in probabilities for a plurality of locations in a predetermined region of the environment.

11. The method of claim 9, wherein, using the cut-in neural network features to generate the factor cut-in probabilities includes providing the cut-in neural network features as a first input to a different cut-in machine learning model that operates on the cut-in neural network features and other features of the factor, other features of the vehicle, or both.

12. The method of claim 11, wherein, the different cut-in machine learning model is a random decision forest.

13. The method of claim 11 or 12, further comprising: computing a respective initial factor cut-in probability for each of the one or more factors, including: receiving data specifying one or more predicted paths of the factor, the predicted paths including one or more predicted future locations; receiving data specifying a respective predicted path probability for each of the one or more predicted paths of the factor, wherein the predicted path probability for a given predicted path characterizes a likelihood that the factor will travel on the predicted path; determining which of the predicted paths intersect the vehicle’s planned future locations within a predetermined amount of time; and generating the respective initial factor cut-in probability by combining the respective predicted path probabilities of those predicted paths that intersect the vehicle’s planned future locations; and providing the initial factor cut-in probability as a second input to the different cut-in machine learning model.

14. The method of claim 13, further comprising: for each of the one or more surrounding factors, extracting one or more features characterizing the predicted path of the surrounding factor from the predicted path of the surrounding factor, wherein the one or more features include one or more of: a time of a most recent collision with the vehicle, a time of a most recent collision with one or more other surrounding factors, one or more features related to traffic lights, one or more features related to stop signs, or a maximum lateral acceleration of the surrounding factor; and providing the one or more features characterizing the predicted path of each surrounding factor as a third input to the different cut-in machine learning model.

15. The method of claim 11 or 12, further comprising: determining one or more features characterizing the surrounding factor, wherein the one or more features include one or more of: a velocity of the surrounding factor, an acceleration of the surrounding factor, or an orientation of the surrounding factor; and providing the one or more features characterizing the surrounding factor as a fourth input to the different cut-in machine learning model.

16. A system comprising one or more computers and one or more storage devices storing instructions that are operable when executed by the one or more computers to cause the one or more computers to perform the methods of any of claims 1-15.

17. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1-15.

Images