behavioral prediction of peripheral factors

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting occupancy of factors. One of the methods includes obtaining scene data characterizing a current scene in an environment; and processing, using a neural network, a neural network input comprising the scene data to generate a neural network output, wherein: the neural network output comprises respective occupancy outputs corresponding to a plurality of factor types at one or more future time points; the occupancy output for each factor type at a first future time point comprises respective occupancy probabilities for a plurality of locations in the environment; and in the occupancy output for each factor type at the first future time point, the respective occupancy probability for each location characterizes a likelihood that a factor of that factor type will occupy that location at the first future time point.

Description

Technical Field

[0001] This manual pertains to autonomous vehicles. Background Technology

[0002] Autonomous vehicles include self-driving cars, boats, and aircraft. Autonomous vehicles use various onboard sensors and computer systems to detect nearby objects and use this detection to make control and navigation decisions.

[0003] Some autonomous vehicles have onboard computer systems that implement neural networks, other types of machine learning models, or both, for various prediction tasks, such as object classification within images. For example, a neural network can be used to determine that an image captured by an onboard camera is likely an image of a nearby car. A neural network (or simply a network) is a machine learning model that employs multiple layers of operations to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers located between input and output layers. The output of each layer serves as the input to another layer in the network (e.g., the next hidden layer or output layer).

[0004] Each layer of a neural network specifies one or more transformation operations to be performed on the inputs of that layer. Some neural network layers have operations called neurons. Each neuron receives one or more inputs and generates an output that is received by another neural network layer. Typically, each neuron receives inputs from other neurons and provides outputs to one or more other neurons.

[0005] The architecture of a neural network specifies which layers are included in the network, their properties, and how the neurons in each layer are connected. In other words, the architecture specifies which layers provide their outputs as input to which other layers, and how the outputs are provided.

[0006] The transformation operations for each layer are performed by a computer that has the software module that implements the transformation operations installed. Therefore, describing a layer as performing operations means that the computer that implements the transformation operations for that layer performs those operations.

[0007] Each layer uses the current values ​​of its parameter set to generate one or more outputs. Therefore, training a neural network involves continuously performing a forward pass on the input, computing gradient values, and updating the current values ​​of each layer's parameter set using the computed gradient values ​​(e.g., using gradient descent). Once the neural network is trained, the final set of parameter values ​​can be used to make predictions in a production system. Summary of the Invention

[0008] This specification describes how an airborne system, such as that of an autonomous or semi-autonomous vehicle, can use an occupation prediction system to generate one or more occupation outputs. The system can simultaneously generate occupation outputs for each of one or more agent types and for each of one or more future time points. For each of multiple locations in the vehicle's environment, the occupation output for a given agent type and a given future time point characterizes the corresponding predicted probability that the location will be occupied by an agent of that given agent type at that given future time point.

[0009] An occupancy prediction system can receive scene data representing the current state of the environment as input and process the scene data using a neural network to generate p occupancy outputs, each corresponding to one of q factor types and one of r future time points, where generally p = q × r. In some implementations, each of the p occupancy outputs can be generated in parallel, conditioned on a common representation of the input scene data generated by the neural network.

[0010] Once the occupancy prediction system has been trained, it can be deployed on a vehicle and used by the vehicle to make autonomous or semi-autonomous driving decisions.

[0011] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages.

[0012] Some existing techniques for generating predicted occupancy outputs rely on predicted future paths of surrounding factors. For a given surrounding factor, a separate prediction system generates one or more predicted future paths for that factor, each with a corresponding probability. An occupancy prediction system then receives these predicted paths and, for each location in the environment, infers the occupancy probability from the predicted paths. In some cases, the predicted future paths are restricted to conforming to a road map provided as input to the system; that is, the system cannot predict that the surrounding factors will navigate outside the boundaries of the road map, for example, by making an illegal turn or veering into a different lane. These existing techniques yield poor results when the predicted paths are inaccurate or unavailable.

[0013] These existing technologies also do not directly model the interactions between different surrounding factors, but instead rely on the predicted future paths of isolated individual factors. Using the techniques described in this specification, the occupancy prediction system can learn the nonlinear characteristics of the surrounding environment, explicitly considering the interactions between one or more surrounding factors and the vehicle. Furthermore, the system does not rely entirely on road maps. Instead, the system described in this specification can generate predictions for surrounding factors that will violate the road map. As a specific example, the system can determine that a factor in the right-turn lane is driving too fast to make a right turn and therefore predicts that the factor will continue straight through the intersection, even if such a maneuver is illegal. Additionally, in some embodiments described herein, the system can receive the vehicle's own planned path and incorporate that planned path into the predicted occupancy output.

[0014] Using the techniques described in this specification, the occupancy prediction system can generate a corresponding occupancy probability for each of multiple surrounding factors in a single forward pass. This is an improvement over prior art, which generates occupancy predictions for a single surrounding factor at a time and then combines the occupancy predictions for each surrounding factor to generate an occupancy output. Regardless of the number of surrounding factors in the environment, generating the occupancy output in a single forward pass can significantly reduce the computation time of the occupancy prediction system, a particularly useful feature for systems deployed on vehicles.

[0015] As a specific example, in dense urban environments, there are often many pedestrians moving within the environment, and generating predicted paths for each pedestrian may be impractical or impossible. These scenarios require a more comprehensive approach to the environment. The system described in this specification can generate occupancy outputs for a large number of pedestrians, predicting whether any pedestrian will occupy that location for each position in the environment, regardless of which particular pedestrian will occupy that location. Therefore, the system can model the behavior of multiple factors together without specifically tracking any single factor.

[0016] Some other existing systems rely on multiple neural networks, each generating a prediction for a single, specific future time point, and each must be trained independently. Then, at the inference time, the onboard system must run each of these neural networks individually to generate the corresponding occupation output for each future time point. However, time constraints often make it infeasible for an onboard system to perform more than one or more inference calls while making occupation predictions. Some systems described in this specification can generate occupation predictions for multiple different future time points in parallel and efficiently.

[0017] Some existing techniques rely on different models for corresponding types of surrounding factors, for example, by training each model to predict occupancy for a single factor type and then running each model separately at inference time. Such techniques do not model the interactions between different types of factors because the different models do not interact. Using the techniques described in this specification, an occupancy prediction system can, for example, generate occupancy predictions for multiple factor types simultaneously in a single forward pass, thereby allowing different occupancy predictions to share information, for example, through shared parameters.

[0018] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become clear from the description, drawings, and claims. Attached Figure Description

[0019] Figure 1 This is a diagram of the example system.

[0020] Figure 2A and Figure 2B This is a diagram of an example architecture for generating the dominant output of a neural network.

[0021] Figure 3A and Figure 3B This is an example of the output diagram corresponding to the vehicle's surrounding environment.

[0022] Figure 4 This is a flowchart of an example process used to generate the occupied output.

[0023] The same reference numerals and names in different figures indicate the same elements. Detailed Implementation

[0024] This specification describes how a vehicle (e.g., an autonomous vehicle or a semi-autonomous vehicle) can use a trained machine learning model (referred to in this specification as an "occupancy prediction system") to generate an occupancy output corresponding to each of one or more future time points and each of one or more types of surrounding factors.

[0025] In this specification, without loss of generality, "surrounding factors" can refer to vehicles, bicycles, pedestrians, ships, drones, or any other moving object in the environment. This specification also describes how to effectively train an occupancy prediction system to make accurate and reliable predictions using training examples generated from vehicles.

[0026] Figure 1 This is a diagram of example system 100. System 100 includes airborne system 110 and training system 120.

[0027] The airborne system 110 is located on the vehicle 102. Figure 1The vehicle 102 is shown as a car, but the onboard system 102 can be located on any suitable vehicle type. The vehicle 102 can be a fully autonomous vehicle that determines and executes fully autonomous driving decisions to navigate its environment. The vehicle 102 can also be a semi-autonomous vehicle that uses prediction to assist a human driver. For example, if a prediction indicates that a collision with another vehicle is imminent, the vehicle 102 can autonomously apply the brakes.

[0028] The airborne system 110 includes one or more sensor subsystems 130. The sensor subsystem 130 includes a combination of components that receive electromagnetic radiation reflections (e.g., a LIDAR system for detecting laser reflections, a radar system for detecting radio wave reflections, and a camera system for detecting visible light reflections).

[0029] Sensor data generated by a given sensor typically indicates the distance, direction, and intensity of reflected radiation. For example, a sensor can transmit one or more pulses of electromagnetic radiation in a specific direction and can measure the intensity of any reflection as well as the time it takes to receive it. Distance can be calculated by determining the time elapsed between a pulse and its corresponding reflection. A sensor can continuously scan a specific space at an angle, azimuth, or both. For example, azimuth scanning allows a sensor to detect multiple objects along the same line of sight.

[0030] Other components of the sensor subsystem 130 or vehicle 102 can also classify multiple sets of one or more raw sensor measurements from one or more sensors into measures of additional factors. A set of sensor measurements can be represented in any of a variety of ways, depending on the type of sensor measurements captured. For example, each set of raw laser sensor measurements can be represented as a three-dimensional point cloud, where each point has intensity and location in a specific two-dimensional or three-dimensional coordinate space. In some embodiments, location is represented as a range and elevation pair. Each set of camera sensor measurements can be represented as an image patch, such as an RGB image patch.

[0031] Once the sensor subsystem 130 classifies one or more sets of raw sensor measurements into measures of other corresponding factors, the sensor subsystem 130 can compile the raw sensor measurements into a set of raw data 132 and send the raw data 132 to the data representation system 140.

[0032] The data representation system 140 on vehicle 102 also receives raw sensor data 132 from sensor system 130 and generates scene data 142. Scene data 142 characterizes the current state of the environment surrounding vehicle 102.

[0033] For example, for all surrounding factors in the environment, scene data may include the current state at the current time point and the previous state at one or more corresponding previous time points. The state of a factor at a given time point may include the factor's position at that time point and the values ​​of a predetermined set of motion parameters at that time point. As a specific example, motion parameters may include the factor's trajectory, the factor's velocity, and / or the factor's acceleration. In some embodiments, scene data may also include the current state of a vehicle at the current time point and the vehicle's previous state at one or more corresponding previous time points. In some embodiments, scene data may also include a road map of the environment, which may include road lanes, pedestrian crossings, traffic lights, stop signs, etc.

[0034] Across different time points, different surrounding factors represented by the scene data may or may not be tracked. That is, although the scene data includes the state of each factor at each time point, the system may or may not associate a specific factor at a first time point with the same specific factor at a second time point. Therefore, the representation of the same factor across different time points in the scene data can be independent, i.e., untracked. Tracking is not necessary because the occupancy prediction system 150 can generate occupancy outputs for specific factor types, which do not predict the occupancy of individual factors of that factor type, but rather predict the global behavior of all factors of that factor type. Removing the requirement to track individual surrounding factors across time points allows for more efficient and accurate occupancy prediction.

[0035] As a specific example, scene data may include one or more top-down rendered images of the environment at each time point, each image depicting surrounding factors within the environment. For instance, scene data may include a single top-down rendered image of the environment at each time point, where factors of different factor types are depicted in different colors. As another example, different top-down rendered images may be available for each factor type and each time point, where a given top-down rendered image depicts only factors of a specific factor type at a particular time point.

[0036] In some implementations, the surrounding factors can be depicted as bounding boxes in a top-down rendered image. A bounding box is a pixel box in an image that identifies the boundary or edge of a corresponding object. In some other implementations, the surrounding factors can be depicted as contours in a top-down rendered image. Like bounding boxes, contours are another way of representing an estimate of the spatial extent of a corresponding object. Typically, unlike bounding boxes, contours do not depict an estimate of the spatial extent of the portion of the object occluded by the sensors of vehicle 102, and therefore can be represented by shapes that are more complex than boxes.

[0037] In some implementations, bounding boxes are used to depict some factor types, while contours are used to depict other factor types. For example, bounding boxes may be a more effective representation of vehicles, while contours may be a more effective representation of pedestrians.

[0038] In some implementations, top-down rendering of the image also includes depictions of road lanes and other road map features.

[0039] The data representation system 140 provides scene data 142 to the occupancy prediction system 150, which is also on the vehicle 102.

[0040] Occupancy prediction system 150 processes scene data 142 to generate occupancy outputs 152 corresponding to each of multiple future time points and each of the factor types. That is, if there are r future time points and q different factor types, the occupancy prediction system can generate p occupancy outputs, where p = q × r. As a specific example, there may be r = 5 different future time points, extending one second into the future in increments of 0.2 seconds. As another specific example, there may be r = 10 different future time points, extending 10 seconds into the future in increments of 1 second. As a specific example, there may be q = 3 different factor types, corresponding to vehicles, pedestrians, and cyclists.

[0041] Each occupancy output represents the predicted state of the environment for a specific factor type at a specific future time point. Each location in the occupancy output corresponds to a position in the environment and has an occupancy probability representing the likelihood that a surrounding factor of a specific factor type will occupy the corresponding position in the environment at that specific future time point. (Reference) Figure 2A and Figure 2B An example architecture occupying 150 of the prediction system is described.

[0042] The onboard system 110 also includes a path planning system 160. The path planning system 160 can make autonomous or semi-autonomous driving decisions for the vehicle 102, for example, by generating a planned vehicle path characterizing the path the vehicle 102 will take in the future. In some embodiments, the path planning system 160 provides a currently planned path 162 to an occupancy prediction system 150, and the occupancy prediction system processes the planned path 162 and scene data 142 to generate an occupancy output 152. For example, the planned path 162 can be represented using a top-down rendered image of the environment depicting the planned path through the environment, such as points depicting the planned positions of the vehicle 102 at corresponding future time points in the image, or a single geometric curve depicting the entire planned path in the image.

[0043] In an implementation where the occupancy prediction system 150 processes the planned path 162 of a vehicle, the occupancy output 152 is conditional on the current planned path 162. That is, each occupancy output 152 represents a predicted state of the environment at a specific future time point for a particular factor type, assuming that the vehicle 102 follows the planned path 162 through the environment until that specific future time point. In some implementations, the path planning system 160 can provide multiple planned paths 162, and the occupancy prediction system 150 can generate a set of occupancy outputs 152 corresponding to each of the planned paths 162.

[0044] In some implementations, for each future time point and each factor type, the occupation prediction system 150 can generate two occupation outputs: a first occupation output predicting the occupation of factors currently in the environment and a second occupation output predicting the occupation of factors not currently in the environment but potentially entering it. (See below for reference.) Figure 2A and Figure 2B Let's discuss this process in more detail.

[0045] The airborne system 100 can provide the occupation output 152 generated by the occupation prediction system 150 to one or more other airborne systems of the vehicle 102, such as the path planning system 160 and / or the user interface system 165.

[0046] When the path planning system 160 receives the occupancy output 152, it can use the occupancy output 152 to generate a new planned vehicle path. For example, the occupancy output 152 may contain predictions that certain surrounding factors are likely to overtake vehicle 102 at a specific future time point, potentially causing a collision. In this example, the path planning system 160 can generate a new planned vehicle path that avoids the potential collision. For example, the path planning system 160 can generate a trajectory that navigates through the environment, allowing vehicle 102 to navigate through locations in the environment at the corresponding future time point, such that each location has a low occupancy probability for each factor type at the corresponding future time point according to the occupancy output 152. Vehicle 102 can then navigate to follow the new planned path and avoid the potential collision.

[0047] When the user interface system 165 receives the occupation output 152, the user interface system 165 can use the occupation output 152 to present information to the driver of vehicle 102 to assist the driver in safely operating vehicle 102. The user interface system 165 can present the information to the driver of vehicle 102 by any appropriate means, such as audio messages transmitted via the vehicle 102's speaker system or alarms displayed on a visual display system in the vehicle (e.g., an LCD display on the vehicle 102's dashboard). In a particular example, the occupation output 152 may contain a prediction that a particular ambient factor may overtake in front of vehicle 102, potentially causing a collision. In this example, the user interface system 165 may present the driver of vehicle 102 with instructions to adjust the vehicle 102's trajectory to avoid a collision or notify the driver of a potential collision with a particular ambient factor.

[0048] In order to generate occupancy output 152, occupancy prediction system 150 can use trained parameter values ​​195 obtained from occupancy model parameter storage 190 in training system 120.

[0049] The training system 120 is typically hosted within a data center 124, which may be a distributed computing system with hundreds or thousands of computers in one or more locations.

[0050] Training system 120 includes a training data store 170 that stores all training data for training parameter values ​​for occupancy prediction system 150. Training data store 170 receives raw training examples from factors operating in the real world. For example, training data store 170 may receive raw training examples 155 from vehicle 102 and one or more other factors communicating with training system 120. Training system 120 may process raw training examples 155 to generate new training examples. Raw training examples 155 may include scene data 142 that can be used as input to new training examples. Raw training examples 155 may also include outgoing data characterizing the state of the environment surrounding vehicle 102 at one or more future time points for one or more factor types. This outgoing data can be used to generate ground-truth outputs for new training examples, where each ground-truth output is an occupancy output corresponding to a specific future time point and a specific factor type. In some implementations, each baseline truth value for a future time point and factor type occupies the output, having a "1" in the location corresponding to the position occupied by the surrounding factors of that factor type in the environment at that future time point, and a "0" in all other locations.

[0051] Training data storage 170 provides training examples 175 to training engine 180, which is also hosted in training system 120. Training engine 180 uses training examples 175 to update model parameters that will be used by occupancy prediction system 150, and provides the updated model parameters 185 to occupancy model parameter storage 190. Once the parameter values ​​of occupancy prediction system 150 have been fully trained, training system 120 can send the trained parameter values ​​195 to occupancy prediction system 150, for example, via a wired or wireless connection.

[0052] Figure 2A This is a diagram of an example architecture for a neural network 200 used to generate occupational output. For example, the neural network could be an occupational prediction system (e.g., Figure 1 The part of the occupancy prediction system 150 described in the text.

[0053] Neural network 200 receives input scene data 202, which represents the current state of the environment surrounding the vehicle. Neural network 200 is a feedforward neural network that generates multiple different occupancy outputs in a single forward pass. Each occupancy output can predict the occupancy probability for each of multiple locations in the environment at a specific future time point, for a specific factor type.

[0054] The shared neural network block 210 receives scene data 202 and processes it to generate a shared embedding 212 of the scene data 202. In this specification, an embedding is an ordered set of numerical values ​​representing an input in a specific embedding space. For example, an embedding can be a floating-point vector or other numerical values ​​with a fixed dimension.

[0055] The shared neural network block 210 is called "shared" because it is shared by each factor type and each future time point. In other words, the parameters of the shared neural network block 210 contribute to the occupancy output for each factor type and each future time point.

[0056] In some implementations, scene data 202 includes multiple channels of images, such as channels of a top-down image of the environment, where each top-down image corresponds to a specific factor type at a particular previous or current time point. In these implementations, the shared neural network block 210 may include one or more convolutional neural network layers capable of processing the input channels to generate a shared embedding 212. Each convolutional layer may be defined by a multidimensional (e.g., two-dimensional or three-dimensional) convolutional filter, enabling the shared neural network block 210 to learn complex spatial relationships between different spatial regions of the environment and encode the spatial relationships in the shared embedding 212. As a specific example, the shared neural network block 210 may include several convolutional layers and several max-pooling layers, the max-pooling layers reducing the input channel size such that the shared embedding 212 includes one or more channels smaller than the input channels.

[0057] Following the shared neural network block 210, the neural network 200 is divided into different branches corresponding to the respective factor types. That is, the shared neural network block 210 can provide a shared embedding 212 to each of the q different factor type neural network blocks. For example... Figure 2A As shown, there are three factor type neural network blocks 220a-220c, corresponding to three different factor types A, B and C.

[0058] Each factor type neural network block 220a-220c processes the shared embedding 212 to generate a corresponding factor type embedding. For example, factor type neural network block 220b processes the shared embedding 212 to generate a factor type embedding 222 corresponding to factor type B. Each factor type neural network block 220a-220c has different parameters, allowing each factor type neural network block 220a-220c to extract factor type-specific information from the shared embedding 212 and encode that information in the corresponding factor type embedding. In some embodiments, multiple factor type neural network blocks of neural network 200 can process the shared embedding 212 in parallel.

[0059] After each factor type neural network block 220a-220c, neural network 200 is further divided into different branches corresponding to the corresponding future time points. For example, factor type neural network block 220b provides factor type embedding 222 to each of the three future time point neural network blocks 230a-230c corresponding to three different future time points A, B, and C.

[0060] Each future time point neural network block 230a-230c processes factor type embedding 222 to generate a corresponding occupying output. Future time point neural network block 230a generates an occupying output 232a corresponding to factor type B and future time point A. Future time point neural network block 230b generates an occupying output 232b corresponding to factor type B and future time point B. Future time point neural network block 230c generates an occupying output 232c corresponding to factor type B and future time point C. Each future time point neural network block 230a-230c has different parameters, allowing each future time point neural network block 230a-230c to extract information specific to the corresponding future time point from the factor type embedding 222 and use this information to generate the corresponding occupying output. In some embodiments, multiple future time point neural network blocks of neural network 200 can process the corresponding factor type embeddings in parallel.

[0061] In some implementations, the final layer of each future time point neural network block receives a nonnormalized occupancy output, which includes nonnormalized values ​​for each location in the environment, as input, and applies a sigmoid activation function to each nonnormalized value to normalize it to between 0 and 1. Therefore, each value in the corresponding occupancy output can represent a probability, between 0 and 1, that a factor of the corresponding factor type will occupy that location at the corresponding future time point.

[0062] In some implementations, future time point neural network blocks corresponding to the same future time point and different factor types may share some or all of their parameters. For example, future time point neural network block 230a corresponding to future time point A and factor type B may share parameters with different future time point neural network blocks below factor type neural network block 220a and corresponding to future time point A and factor type A. In some implementations, no neural network blocks in neural network 200 share parameters; instead, during the training of neural network 200, the parameters of each neural network block are learned independently.

[0063] In some implementations, occupancy outputs 232a-232c are top-down images of the environment. That is, each occupancy output for a specific future time point and a specific factor type can include a top-down image, where each pixel corresponds to a location in the environment and has a value characterizing the predicted occupancy probability of the corresponding location in the environment for a specific factor type and a specific future time point.

[0064] In some embodiments where the occupying outputs 232a-232c are top-down images and the scene data 202 includes one or more channels of the top-down image, the occupying outputs 232a-232c have the same resolution as the input channels of the scene data 202. That is, the occupying outputs 232a-232c and the input channels of the scene data 202 can have the same number of elements (e.g., pixels), such that each position represented by an element in the input channel of the scene data 202 is also represented by an element in the occupying outputs 232a-232c. In some such embodiments, the factor-type neural network block and / or the future time-point neural network block can have one or more deconvolution layers that increase the size of the shared embedding 212, such that the occupying outputs have the same resolution as the scene data 202.

[0065] During the training of neural network 200, the training system can determine the error of each occupation output based on the corresponding baseline true occupation output. For example, the training system can use the cross-entropy loss between the predicted occupation probability at a specific location and the "true" occupation probability at that specific location, where the true occupation probability is 1 if a factor of the corresponding factor type is present at that location at the corresponding future time point, and 0 otherwise. The training system can then backpropagate each error through the neural network to update the parameters of neural network 200. For example, the training system can use the error in occupation output 232a to update the parameters in future time point neural network block 230a, factor type neural network block 220b, and shared neural network block 210. As another example, the training system can use the error in occupation output 232b to update the parameters in future time point neural network block 230b, factor type neural network block 220b, and shared neural network block 210. Therefore, shared neural network block 210 can be updated based on the error in each of the occupation outputs, allowing shared neural network block 210 to learn and generate a shared embedding that can be used to generate occupation outputs for each future time point and each factor type. Similarly, each factor type neural network block can be updated based on the error in each occupied output corresponding to the same factor type and the corresponding future time point, thereby allowing the factor type neural network block to learn the generated factor type embedding, which can be used to generate occupied output for each future time point.

[0066] Although neural network 200 is shown to generate occupied outputs corresponding to three future time points and three factor types, it should be understood that neural network 200 can be configured to generate occupied outputs for any number of future time points and factor types.

[0067] In some implementations, each occupation output 232i includes a first occupation output corresponding to a surrounding factor currently in the environment and a second occupation output corresponding to a surrounding factor currently not in the environment. That is, for an occupation output corresponding to a specific factor type and a specific future time point, the first occupation output can predict, for each location in the environment, whether that location will be occupied by a factor of a specific factor type currently in the environment that will occupy that location at a specific future time point. The second occupation output can predict, for each location in the environment, whether that location will be occupied by a factor of a specific type currently not in the environment that will occupy that location at a specific future time point; that is, whether a factor of a specific type will enter the environment from a location outside the environment and occupy that location at a specific future time point. For example, each future time point neural network block 230i can have a first branch generating the first occupation output and a second branch generating the second occupation output. In these implementations, the neural network 200 can be trained using a baseline ground truth occupation corresponding to a factor currently in the environment in the first occupation output and a baseline ground truth occupation corresponding to a factor currently not in the environment in the second occupation output.

[0068] Figure 2B This is an illustration of an example architecture for another neural network 250 used to generate occupation output. For example, the neural network could be an occupation prediction system (e.g., Figure 1 The part of the occupancy prediction system 150 described in the text.

[0069] The neural network 250 receives input scene data 252, which represents the current state of the environment surrounding the vehicle. The neural network 250 is a feedforward neural network that generates multiple different dominant outputs in a single forward pass.

[0070] The shared neural network block 260 receives scene data 252 and processes the scene data 252 to generate a shared embedding 262 of the scene data 252. As previously described, the shared neural network block 260 may include one or more convolutional neural network layers capable of processing input channels in the scene data 252 to generate the shared embedding 262.

[0071] Following the shared neural network block 260, the neural network 250 is divided into different branches corresponding to the respective future time points. That is, the shared neural network block 260 can provide a shared embedding 212 to each of the r different future time point neural network blocks. For example... Figure 2B As shown, there are three future time points, neural network blocks 270a-270c, which correspond to three future time points A, B, and C.

[0072] Each future time point neural network block 270a-270c processes the shared embedding 262 to generate a corresponding future time point embedding. For example, future time point neural network block 270b processes the shared embedding 262 to generate a future time point embedding 272 corresponding to future time point B. Each future time point neural network block 270a-270c has different parameters, allowing each future time point neural network block 270a-270c to extract information specific to the corresponding future time point from the shared embedding 262 and encode this information in the corresponding future time point embedding. In some embodiments, multiple future time point neural network blocks of neural network 250 can process the shared embedding 262 in parallel.

[0073] After each future time point neural network block 270a-270c, neural network 250 is again divided into different branches corresponding to the corresponding factor types. For example, future time point neural network block 270b provides future time point embedding 272 to each of the three factor type neural network blocks 280a-280c ​​corresponding to three different factor types A, B, and C.

[0074] Each factor type neural network block 280a-280c ​​processes the future time point embedding 272 to generate a corresponding occupation output. For example, factor type neural network block 280a generates an occupation output 282a corresponding to future time point B and factor type A. Each factor type neural network block 280a-280c ​​has different parameters, allowing each factor type neural network block 280a-280c ​​to extract factor type-specific information from the future time point embedding 272 and use that information to generate the corresponding occupation output. In some implementations, multiple factor type neural network blocks of neural network 250 can process the corresponding future time point embeddings in parallel.

[0075] In some implementations, the final layer of each factor type neural network block receives a nonnormalized occupancy output, which includes nonnormalized values ​​for each location in the environment, as input, and applies a sigmoid activation function to each nonnormalized value to normalize it to between 0 and 1. Therefore, each value in the corresponding occupancy output can represent the probability, between 0 and 1, that a factor of the corresponding factor type will occupy that location at various future time points.

[0076] In some implementations, neural network blocks of the same factor type but corresponding to different future time points may share some or all of their parameters. For example, factor type neural network block 280a corresponding to factor type A and future time point B may share parameters with neural network blocks of different factor types, below future time point neural network block 270a and corresponding to factor type A and future time point A. In some implementations, no neural network blocks in neural network 250 share parameters; instead, the parameters of each neural network block are learned independently during the training of neural network 250.

[0077] In some implementations, the occupies outputs 282a-282c are top-down images of the environment. In some implementations, the top-down image has the same resolution as the input channel of the scene data 252. In some such implementations, the factor-type neural network block and / or the future time-point neural network block may have one or more deconvolution layers that increase the size of the shared embedding 262 such that the occupies the output has the same resolution as the scene data 252.

[0078] During the training of neural network 250, the training system can determine the error of each occupation output based on the corresponding baseline ground truth occupation output. For example, the training system can use the cross-entropy loss between the predicted occupation probability at a specific location and the true occupation probability at that specific location. The training system can then backpropagate each error through the neural network to update the parameters of neural network 250. For example, the training system can use the error in occupation output 282a to update the parameters in factor type neural network block 280a, future time point neural network block 270b, and shared neural network block 260. Therefore, shared neural network block 260 can be updated based on the error in each of the occupation outputs, allowing shared neural network block 260 to learn and generate a shared embedding that can be used to generate occupation outputs for each future time point and each factor type. Similarly, each future time point neural network block can be updated based on the error in each occupation output corresponding to the same future time point and corresponding factor type, allowing future time point neural network blocks to learn and generate future time point embeddings that can be used to generate occupation outputs for each factor type.

[0079] Although neural network 250 is shown to generate occupation outputs corresponding to three future time points and three factor types, it should be understood that neural network 250 can be configured to generate occupation outputs for any number of future time points and factor types.

[0080] As previously described, in some embodiments, each occupation output 282i includes a first occupation output corresponding to a surrounding factor currently in the environment and a second occupation output corresponding to a surrounding factor currently not in the environment. For example, each factor-type neural network block 280i may have a first branch generating the first occupation output and a second branch generating the second occupation output. In these embodiments, the neural network 250 may be trained using a baseline ground truth occupation corresponding to a currently in-environment factor in the first occupation output and a baseline ground truth occupation corresponding to a currently not-in-environment factor in the second occupation output.

[0081] Figure 3A The illustrations corresponding to the vehicle's surrounding environment occupy outputs 310 and 320. The environment contains two surrounding factors of a specific factor type. Although the two surrounding factors are shown as a car, it should be understood that the surrounding factors can be any appropriate factor type, such as pedestrians, cyclists, etc.

[0082] Occupancy outputs 310 and 320 are generated by a trained occupancy prediction system (e.g., ...). Figure 1 The example two-dimensional arrays generated by the occupancy prediction system 150 depicted in the figure. Each array represents a "top-down" view of the environment, where each location corresponds to a corresponding position in the environment.

[0083] In some implementations where the input to the occupancy prediction system is scene data represented as a top-down image of the environment, the occupancy output array may have the same dimensions as the input channel, such that the localization in the output array represents the same location in the environment as the corresponding localization in the input channel.

[0084] The value at each location in the occupancy output corresponds to the occupancy probability at that location. In some implementations, an image depicting the occupancy output can be created where corresponding blocks of one or more pixels correspond to each location in the array, and higher occupancy probabilities are depicted as darker grayscale values ​​for the corresponding pixel blocks. Such images can be useful during the training of the occupancy prediction system, for example, allowing a user to visually check the progress of the system's training.

[0085] The first occupancy output 310 corresponds to a future time point in the relatively near future, such as 1 second. For each of the two surrounding factors, the location with a non-zero occupancy probability in the first occupancy output 310 is shown in dark gray. For example, a dark gray location could indicate an occupancy probability of 1 / 7 ≈ 0.14, since there are 7 non-zero occupancy probabilities for each surrounding factor.

[0086] The first occupancy output 310 is simplified; it should be understood that the corresponding occupancy probability for each location may be different. For example, there may be one or more locations that are accessible to two surrounding factors. In this case, the occupancy probability of these locations will be higher than the occupancy probability of the surrounding locations, because the occupancy probability characterizes the collective likelihood that any one or two of the surrounding factors will occupy that location. As another example, a single surrounding factor may occupy multiple different locations in the environment at future points in time, for example, if the surrounding factor is on the boundary between a first location and a second location in the environment.

[0087] The second occupancy output 320 corresponds to a future time point relatively far from the first occupancy output 310, for example, 2 seconds. For each of the two surrounding factors, locations in the second occupancy output 320 with a non-zero occupancy probability are shown in light gray. For example, a light gray location could indicate an occupancy probability of 1 / 23 ≈ 0.04, since there are 23 non-zero occupancy probabilities for each surrounding factor. Here, because the future time point is further away, the occupancy probability distribution is more uniform, thus the behavior of the surrounding factors is more uncertain. Generally, the longer the time span (horizon), the greater the diffusion of the occupancy probability.

[0088] Figure 3B This is another example of the environment corresponding to the vehicle's surroundings, occupying an output of 330. The environment contains multiple pedestrians in a crowd.

[0089] The occupancy output 330 is generated by a trained occupancy prediction system (e.g., Figure 1 The example two-dimensional arrays generated by the occupancy prediction system 150 depicted in the figure. Each array represents a "top-down" view of the environment, where each location corresponds to a corresponding position in the environment.

[0090] The value at each location in the occupied output corresponds to the occupancy probability of that location. In some implementations, an image depicting the occupied output can be created, wherein corresponding blocks of one or more pixels correspond to each location in the array, and a higher occupancy probability is depicted as a darker grayscale value for the corresponding pixel block.

[0091] Occupancy output 330 corresponds to a future time point, such as 2 seconds in the future. Locations in occupancy output 330 with a non-zero probability of being occupied by any pedestrian at the future time point are shown in gray shading. For example, locations with a high occupancy probability are shown in dark gray, locations with a slightly lower occupancy probability are shown in medium gray, and locations with an even lower but non-zero occupancy probability are shown in light gray.

[0092] Occupied output 330 indicates that no pedestrians are tracked on a time-by-time basis; for each location, occupied output 330 represents the collective probability that any one or more pedestrians will occupy that location at a future time point, regardless of which pedestrians might do so.

[0093] In dense urban environments (such as those shown in Figure 3), vehicle-based factor tracking systems struggle to perfectly segment and track different pedestrians over time. Furthermore, because vehicle sensors observe the environment from a single perspective, there may be significant occlusion at different times, making segmentation and tracking difficult. Therefore, occupancy prediction systems that rely on tracking individual pedestrians may perform poorly in these dense urban environments due to the excessive noise in their input. On the other hand, probabilistic occupancy outputs, such as those shown in Figure 3, can express the uncertainties in the environment and generate accurate predictions without tracking individual pedestrians.

[0094] In some implementations, for each location in the environment, occupancy outputs 310, 320, and 330 predict whether any surrounding factors of appropriate type (including surrounding factors currently in the environment and surrounding factors not currently in the environment) will occupy that location at a future time point. In these implementations, during training of the occupancy prediction system, the system may process training examples where surrounding factors not in the environment at the current time point corresponding to the training example enter the environment and are therefore represented in the baseline ground truth occupancy output at future time points. By generating parameter updates for the occupancy prediction system using the errors in the occupancy output generated by the system in response to processing such training examples, the training system can train the occupancy prediction system to predict when surrounding factors will enter the environment using input scene data. As a specific example, if road map information in the input scene data characterizes a pedestrian crossing near the edge of the environment for pedestrian movement into the environment, the occupancy prediction system can know that a pedestrian may enter the environment.

[0095] In some other implementations, for each location in the environment, the occupancy outputs 310, 320, and 330 only predict whether one of the surrounding factors of the appropriate factor type currently in the environment will occupy that location at a future point in time.

[0096] In any of these implementations, the occupancy prediction system can be trained to predict that one or more surrounding factors are likely to leave the environment before a future time point. During training, the occupancy prediction system can process training examples where surrounding factors that were in the environment at the current time point corresponding to the training example leave the environment and are therefore not represented in the baseline ground truth occupancy output. By generating parameter updates for the occupancy prediction system using errors in the occupancy output generated by the occupancy prediction system in response to processing such training examples, the training system can train the occupancy prediction system to predict when surrounding factors will leave the environment using input scene data. As a specific example, if the training examples characterize surrounding factors near the edge of the environment moving rapidly toward the edge, the occupancy prediction system can determine that the surrounding factors are likely to leave the environment.

[0097] Figure 4 This is a flowchart of an example process 400 for generating occupancy output. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, an occupancy prediction system appropriately programmed according to this specification (e.g., Figure 1 The occupancy prediction system (150) can execute process 400.

[0098] The system obtains scene data characterizing the current scene in the environment surrounding the vehicle (step 402). For example, for each of a variety of factor types, the scene data may include the location of the factor of that factor type in the environment at one or more previous time points. For each previous time point, the scene data may include a top-down rendered image of the environment at the previous time point depicting each factor of that factor type.

[0099] The system uses a neural network to process neural network inputs including scene data to generate a neural network output including occupancy outputs for each of multiple factor types and for each of multiple future time points (step 404). The occupancy output for each factor type at each future time point may include a corresponding occupancy probability for each of multiple locations in the environment, wherein the occupancy probability for each location characterizes the likelihood that a factor of that factor type will occupy that location at the future time point.

[0100] In some implementations, the occupancy output includes multiple cells in a map, each cell corresponding to a location in the environment, and having the probability that a factor of a certain type will occupy a corresponding real-world location at a given future time point.

[0101] Neural networks can, for example, have similar characteristics to... Figure 2A and Figure 2B The architecture described in the text.

[0102] In some implementations, the neural network input also includes vehicle data of the vehicle in the environment, wherein the vehicle data includes the vehicle's planned path. For example, the vehicle's planned path may include the vehicle's planned position in the environment at each of one or more future time steps. In these implementations, the neural network output may be conditioned on the vehicle path.

[0103] The system provides the neural network output as input to a planning system that plans the vehicle's navigation in the environment (step 406). For example, the vehicle's planning system may include an optimization-based planner configured to process the neural network output to generate constraints in the optimization for each specific location in the environment, each location having an occupancy probability exceeding a predetermined threshold (e.g., a threshold of 0, 0.05, or 0.1) in the occupancy output for a corresponding factor type at a corresponding specific future time point. As a specific example, the planning system may generate constraints in the optimization for each specific location in the environment, each location having a non-zero occupancy probability in the occupancy output for a corresponding factor type at a corresponding specific future time point. The constraints for a specific location may correspond to a specific future time point where a factor of the corresponding factor type is predicted to occupy that specific location. The constraints may penalize the planned path of the vehicle, planning the vehicle to be in a specific location at a specific future time point.

[0104] As a specific example, the planning system can determine which cells in each occupied output have a predicted occupancy probability higher than a threshold occupancy probability. These cells correspond to specific points in the spatiotemporal environment. The planning system can then add constraints to the optimization framework of the optimization-based planner that penalize planning paths that occupy one or more specific points in the environment. These constraints can be hard constraints (where the planner is not allowed to generate planning paths that occupy these points) or soft constraints (where such planning paths are penalized but still allowed), depending on the optimization framework already deployed for the planner. For example, if the planning system establishes a high threshold, it can add hard constraints to these points because these points correspond to relatively high collision probabilities.

[0105] For example, as described above, in some implementations, the input to a trained occupancy prediction system that generates occupancy output may include one or more top-down rendered images of the environment, where, for example, bounding boxes or contours are used to depict surrounding factors in the environment. In these implementations, the spatial extent of the surrounding factors is implicitly modeled in the input; thus, the trained occupancy prediction system can determine which locations in the environment will be occupied by any portion of the spatial extent of the surrounding factors. The vehicle planning system can then determine which potential future locations of the vehicle will overlap with any portion of the surrounding factors at a given future time point, thereby avoiding collisions. Some existing planning systems rely on predicted trajectories of the surrounding factors, which do not encode information about the spatial extent of the surrounding factors. Therefore, these existing systems must explicitly model the spatial extent of the surrounding factors at each point along the trajectory to determine which potential future locations of the vehicle will overlap with any portion of the surrounding factors, which is unreliable and computationally expensive.

[0106] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their equivalents), or combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0107] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including programmable processors, computers, or multiple processors or computers. The device may also be or may include off-the-shelf or custom-designed parallel processing subsystems, such as GPUs or other types of dedicated processing subsystems. The device may also be or may include dedicated logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0108] A computer program, also referred to or described as a program, software, software application, application, module, software module, script, or code, can be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form (including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment). A program may, but does not need to, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located in one location or distributed across multiple locations and interconnected via a data communication network.

[0109] For a computer system to be configured to perform a specific operation or action, it means that the system has software, firmware, hardware, or a combination thereof installed thereon, which, in operation, cause the system to perform the operation or action. For a computer program to be configured to perform a specific operation or action, it means that the program includes instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0110] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs different from the inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device (e.g., server, mobile phone, tablet computer, laptop computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device) including one or more processors and computer-readable media. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0111] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (such as an FPGA or ASIC), or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0112] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented or incorporated therein by special-purpose logic circuitry. Generally, a computer will also include or be operatively coupled to one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, to receive data from or deliver data to, or both, these mass storage devices. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in other devices (e.g., mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, GPS receivers, or portable storage devices (e.g., Universal Serial Bus (USB) flash drives, to name just a few).

[0113] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0114] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse, trackball, or presence-sensitive display or other surface through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone), running a messaging application, and then receiving response messages from the user.

[0115] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server; or middleware components, such as an application server; or frontend components, such as a client computer with a graphical user interface, a web browser, or an application that allows users to interact with embodiments of the subject matter described in this specification; or any combination of one or more such backend components, middleware components, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0116] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship arises from computer programs running on their respective computers, and they have a client-server relationship. In some embodiments, the server sends data, such as HTML pages, to a user device, for example, with the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interactions, may be received at the server from the device.

[0117] In addition to the embodiments described above, the following embodiments are also innovative:

[0118] Example 1 is a method comprising:

[0119] Obtain scene data representing the current scene in the environment; and

[0120] A neural network is used to process the neural network input, which includes scene data, to generate a neural network output, where:

[0121] The neural network output includes corresponding occupancy outputs for various factor types at one or more future time points;

[0122] For each factor type, the occupancy output at the first future time point includes the corresponding occupancy probability at multiple locations in the environment; and

[0123] In the occupancy output for each factor type at the first future time point, the corresponding occupancy probability for each position represents the likelihood that a factor of that factor type will occupy that position at the first future time point.

[0124] Example 2 is the method of Example 1, wherein, for each factor type, the scenario data includes the corresponding location of the factor of that factor type in the environment at one or more previous time points.

[0125] Example 3 is the method of Example 2, wherein, for a first prior time point, the scene data includes:

[0126] The environment is rendered from top to bottom at a first previous time point, wherein the rendered from top to bottom image includes one or more pixels occupying a factor that depicts the factor type.

[0127] Example 4 is the method of Example 3, wherein the top-down rendered image of a specific factor type at a first prior time point includes one or more of the following:

[0128] Multiple first pixels of the bounding box of a factor of this specific factor type;

[0129] Multiple second pixels depicting the outline of a factor of that specific factor type; or

[0130] Depicts multiple third pixels of road lanes in the environment.

[0131] Example 5 is a method of any of Examples 1-4, wherein the occupancy output for each factor type at a first future time point includes multiple units corresponding to the position in the environment, wherein each unit has the occupancy probability that the factor of the corresponding factor type will occupy the corresponding position at the first future time point.

[0132] Example 6 is the method of Example 5, wherein the resolution of the occupied output is the same as the resolution of the scene data.

[0133] Example 7 is a method of any of Examples 1-6, wherein the scene data characterizes the environment at each of a plurality of previous time points, and wherein a first representation of the first factor at a first previous time point is independent of a second representation of the first factor at a second previous time point.

[0134] Example 8 is a method of any of Examples 1-7, wherein the neural network is a feedforward convolutional neural network, comprising:

[0135] Multiple shared neural network layers process scene data to generate shared embeddings; and

[0136] Multiple blocks of a second neural network layer, each block of a second neural network layer corresponding to a corresponding factor type, wherein each block of a second neural network layer processes a shared embedding to generate an occupied output for the corresponding factor type.

[0137] Example 9 is the method of Example 8, wherein the neural network processes shared embeddings to generate occupy outputs in parallel for each factor type.

[0138] Example 10 is a method of any of Examples 8 or 9, wherein each block of the second neural network layer includes a plurality of blocks of the third neural network layer, each block of the third neural network layer corresponding to a corresponding future time point, wherein, for each block of the second neural network layer, processing the shared embedding includes:

[0139] Multiple second neural network layers are used to process shared embeddings to generate factor type embeddings for corresponding factor types; and

[0140] For each block of the third neural network layer, the factor type embedding is processed to generate the occupancy output for the corresponding factor type at the corresponding future time point.

[0141] Example 11 is the method of Example 10, wherein, for each factor type, the neural network processes the factor type embedding to generate, in parallel, an occupation output for each future time point for that factor type.

[0142] Example 12 is a method of any of Examples 1-11, wherein the neural network input includes vehicle data, which includes the planned location of the vehicle in the environment at each of one or more future time points.

[0143] Example 13 is a method of any of Examples 1-12, further comprising:

[0144] The output of the neural network is fed as input to a planning system that plans the vehicle's navigation in the environment.

[0145] Example 14 is the method of Example 13, wherein the vehicle planning system includes an optimization-based planner configured to process neural network outputs to generate constraints on each location in the environment, each location in the environment having an occupancy probability exceeding a predetermined threshold in the corresponding occupancy output.

[0146] Example 15 is a method of any of Examples 1-14, wherein the factor type includes one or more of the following:

[0147] Vehicle type

[0148] Cyclist type, or

[0149] Pedestrian type.

[0150] Example 16 is a method of any of Examples 1-15, wherein, for each factor type, the neural network output includes:

[0151] The first occupancy output includes corresponding first occupancy probabilities for multiple locations in the environment, each first occupancy probability representing the likelihood that a factor of that type in the environment at the current time point will occupy that location at a first future time point.

[0152] The second occupancy output includes corresponding second occupancy probabilities for multiple locations in the environment, which characterize the probability that a factor of a factor type that is not in the environment at the current time point will occupy that location at a first future time point.

[0153] Example 17 is a system comprising: one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, are operable to cause the one or more computers to perform the method of any of Examples 1 to 16.

[0154] Example 18 is a computer storage medium encoded with a computer program that includes instructions that, when executed by a data processing apparatus, are operable to cause the data processing apparatus to perform the method of any of Examples 1 to 16.

[0155] Although this specification contains numerous details of specific embodiments, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0156] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all shown operations to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0157] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions stated in the claims can be performed in a different order and the desired result can still be obtained. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for a vehicle, comprising: Obtain scene data representing the current scene in the environment; as well as A neural network is used to process the neural network input, which includes scene data, to generate a neural network output, where: The neural network output includes corresponding occupation outputs for multiple factor types at one or more future time points; For each factor type, the occupancy output at the first future time point includes the corresponding occupancy probability at multiple locations in the environment; and In the occupancy output for each factor type at the first future time point, the corresponding occupancy probability for each position represents the likelihood that a factor of that factor type will occupy that position at the first future time point.

2. The method according to claim 1, wherein, For each factor type, the scenario data includes the corresponding location of the factor of that factor type in the environment at one or more previous time points.

3. The method according to claim 2, wherein, For the first previous time point, the scene data includes: The environment is rendered from top to bottom at a first previous time point, wherein the rendered from top to bottom image includes one or more pixels occupying a factor that depicts a specific factor type.

4. The method according to claim 3, wherein, The top-down rendered image at the first previous time point includes one or more of the following: Multiple first pixels of the bounding box of a factor of a specific factor type; Multiple second pixels that depict the outline of a factor of a specific factor type; or Depicts multiple third pixels of road lanes in the environment.

5. The method according to any one of claims 1-4, wherein, The occupancy output for each factor type at a first future time point includes multiple units corresponding to positions in the environment, wherein each unit has the occupancy probability that a factor of the corresponding factor type will occupy a corresponding position at the first future time point.

6. The method according to claim 5, wherein, The resolution of the occupied output is the same as the resolution of the scene data.

7. The method according to claim 1, wherein, The scene data represents the environment at each of a plurality of previous time points, and wherein a first representation of the first factor at a first previous time point is independent of a second representation of the first factor at a second previous time point.

8. The method according to claim 1, wherein, The neural network is a feedforward convolutional neural network, including: Multiple shared neural network layers process scene data to generate shared embeddings; and Multiple blocks of a second neural network layer, each block of a second neural network layer corresponding to a corresponding factor type, wherein each block of a second neural network layer processes a shared embedding to generate an occupied output for the corresponding factor type.

9. The method according to claim 8, wherein, The neural network processes shared embeddings to generate occupy outputs in parallel for each factor type.

10. The method according to any one of claims 8 or 9, wherein, Each block of the second neural network layer includes multiple blocks of the third neural network layer, each block of the third neural network layer corresponding to a corresponding future time point. For each block of the second neural network layer, processing the shared embedding includes: Multiple second neural network layers are used to process shared embeddings to generate factor type embeddings for corresponding factor types; and For each block of the third neural network layer, the factor type embedding is processed to generate the occupancy output for the corresponding factor type at the corresponding future time point.

11. The method according to claim 10, wherein, For each factor type, the neural network processes the factor type embedding to generate, in parallel, occupancy outputs for each future time point for that factor type.

12. The method according to claim 1, wherein, The neural network input includes vehicle data, which includes the planned location of the vehicle in the environment at each of one or more future time points.

13. The method according to claim 1, further comprising: The output of the neural network is provided as input to a planning system that plans the vehicle's navigation in the environment.

14. The method according to claim 13, wherein, The vehicle planning system includes an optimization-based planner configured to process neural network outputs to generate constraints for each location in the environment, each location having an occupancy probability exceeding a predetermined threshold in a corresponding occupancy output.

15. The method according to claim 1, wherein, The factor type includes one or more of the following: Vehicle type Cyclist type, or Pedestrian type.

16. The method according to claim 1, wherein, For each factor type, the neural network output includes: The first occupancy output includes a corresponding first occupancy probability for each of a plurality of locations in the environment, the first occupancy probability representing the probability that a factor of the factor type in the environment at the current time point will occupy the location at a first future time point, and The second occupancy output includes a corresponding second occupancy probability for each of a plurality of locations in the environment, the second occupancy probability representing the probability that a factor of the factor type that is not in the environment at the current time point will occupy the location at a first future time point.

17. A system for a vehicle, comprising: One or more computers and one or more storage devices storing instructions, which, when executed by one or more computers, are operable to cause one or more computers to perform the method of any one of claims 1 to 16.

18. One or more non-transitory storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1-16.

Images