spatial-temporal embedding

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a spatio-temporal embedding of a sequence of point clouds. One of the methods includes obtaining a time series comprising a respective point cloud input corresponding to each of a plurality of time points, each point cloud input comprising point cloud data generated from sensor data captured by one or more sensors of a vehicle at the respective time point; processing each point cloud input using a first neural network to generate a respective spatial embedding characterizing the point cloud input; and processing the spatial embeddings of the point cloud inputs using a second neural network to generate a spatio-temporal embedding characterizing the point cloud inputs in the time series.

Description

TECHNICAL FIELD

[0001] This specification relates to autonomous vehicles. BACKGROUND

[0002] Autonomous vehicles include self-driving cars, boats, and aircraft. Autonomous vehicles use various on-board sensors and computer systems to detect nearby objects, and use such detections to make control and navigation decisions.

[0003] Some autonomous vehicles have on-board computer systems that implement neural networks, other types of machine learning models, or both, for various prediction tasks, such as classification of objects within an image. For example, a neural network can be used to determine that an image captured by an on-board camera is likely to be an image of a nearby car. A neural network (or simply, a network) is a machine learning model that employs multiple layers of operations to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers between an input layer and an output layer. The output of each layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer.

[0004] Each layer of a neural network specifies one or more transformation operations to be performed on the input to the layer. Some neural network layers have operations that are referred to as neurons. Each neuron receives one or more inputs and generates an output that is received by another layer of the neural network. Typically, each neuron receives inputs from other neurons, and each neuron provides outputs to one or more other neurons.

[0005] The architecture of a neural network specifies the layers included in the network and the properties of the layers, and how the neurons of each layer in the network are connected. In other words, the architecture specifies which layers provide their outputs as inputs to which other layers, and how the outputs are provided.

[0006] The transformation operations of each layer are performed by a computer that has installed software modules that implement the transformation operations, and so describing a layer as performing an operation means that the computer that implements the transformation operations of the layer performs the operation.

[0007] Each layer generates one or more outputs using a current value of a parameter set for the layer. Training a neural network thus involves performing a forward pass on the inputs continuously, computing gradient values, and updating the current values of the parameter sets for each layer using the computed gradient values (e.g., using gradient descent). Once the neural network is trained, the final parameter set values can be used to make predictions in a production system. SUMMARY

[0008] The specification describes a system implemented as computer programs on one or more computers in one or more locations that processes a sequence of point clouds using a spatio-temporal embedding neural network to generate a spatio-temporal embedding of the sequence of point clouds. Each point cloud in the sequence of point clouds corresponds to a respective different time point. The system can provide the spatio-temporal embedding of the sequence of point clouds to one or more downstream systems configured to receive the spatio-temporal embedding of the sequence of point clouds as input.

[0009] For example, the system can be an onboard system of an autonomous or semi-autonomous vehicle. Each point cloud in the sequence of point clouds can represent an environment surrounding the vehicle and be generated from sensor data captured by one or more sensors onboard the vehicle. In this example, the system can provide the spatio-temporal embedding of the sequence of point clouds to one or more downstream systems configured to process the spatio-temporal embedding to generate predictions about the environment surrounding the vehicle.

[0010] In some implementations, the spatio-temporal embedding neural network includes i) a first neural network that processes each point cloud in the sequence of point clouds to generate a respective spatial embedding of each point cloud in the sequence of point clouds, and ii) a second neural network that processes the spatial embeddings of the point clouds in the sequence of point clouds to generate the spatio-temporal embedding of the sequence of point clouds. As a particular example, the first neural network can include a voxel embedding neural network and a spatial embedding neural network. The voxel embedding neural network can partition an environment represented by the point clouds into a grid of voxels and generate a respective embedding of each voxel in the grid of voxels. The spatial embedding neural network can process the respective embeddings of the voxels in the grid of voxels to generate the spatial embedding of the point clouds.

[0011] In this specification, an embedding is an ordered set of numerical values that represent an input in a particular embedding space. For example, an embedding can be a vector of floating point or other numerical values of fixed dimension. In this specification, a spatio-temporal embedding is an embedding of an input that encodes both spatial and temporal information of the input. That is, the input can represent a plurality of locations and a plurality of time points, and the spatio-temporal embedding of the input can encode information across the plurality of locations and the plurality of time points.

[0012] A point cloud is a collection of data points defined by a given coordinate system. For example, in a three-dimensional coordinate system, a point cloud can define a shape of some real or synthetic physical system, where each point in the point cloud is defined by a respective coordinate in the coordinate system. In some cases, each point in the point cloud can include additional information, e.g., intensity. In this specification, “point cloud” refers to a three-dimensional point cloud.

[0013] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0014] In some implementations, the systems described in this specification can provide machine-learned spatio-temporal embeddings of sequences of point clouds to multiple different downstream systems to perform respective downstream tasks. Each downstream system thus does not need to generate its own respective embedding, improving the computational and temporal efficiency of systems that are required to perform all of multiple tasks, e.g., an on-board system of an autonomous vehicle. The improved efficiency can be particularly important in time-limited and / or resource-limited environments, e.g., when the system is deployed on-board a vehicle.

[0015] In some implementations, the systems described in this specification can use spatio-temporal embeddings for one or more downstream tasks for which the spatio-temporal embedding neural network was not explicitly trained. For example, the system can train the spatio-temporal embedding neural network using a downstream task of object heat map prediction, in which an object prediction neural network generates an image that characterizes predicted locations of objects in the surrounding. During training, the spatio-temporal embedding neural network generates an embedding that is used by the object prediction neural network, and based on an error of the object prediction neural network, the parameters of the spatio-temporal embedding neural network are updated using backpropagation. After training, the spatio-temporal embedding pairs generated by the spatio-temporal embedding neural network encode spatio-temporal information for a different task (object type classification) without any modification. In some implementations, the system trains the spatio-temporal embedding neural network using multiple different downstream tasks, such that the spatio-temporal embedding encodes different domains of information. The system can then provide the trained spatio-temporal embedding to a different downstream task that is not one of the tasks for which the spatio-temporal embedding neural network was trained.

[0016] Some existing systems use recurrent neural networks (such as LSTMs and GRUs) to encode sequences of elements (e.g., sequences of text or sequences of audio). However, it is not possible to embed sequences of point clouds into a high-dimensional feature space using these conventional techniques. The raw data is very high-dimensional, and the task of encoding information for each point cloud in a sequence of point clouds (each of which can contain millions of points) is not amenable to traditional recurrent neural networks. Using the techniques described in this specification, the system can distill the information contained in a sequence of point clouds (each containing millions of points) into a single embedding that captures rich spatio-temporal features of the environment represented by the sequence of point clouds.

[0017] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a diagram of an example system that includes a spatio-temporal embedding neural network.

[0019] Figure 2 Figure 3A Figure 3B is a diagram of an example spatio-temporal embedding neural network.

[0020] Figure 4 is a diagram of an example voxel embedding neural network.

[0021] Figure 5 is a flowchart of an example process for generating spatio-temporal embeddings of a sequence of point clouds.

[0022] Like reference numbers and designations in different drawings indicate like elements. DETAILED DESCRIPTION

[0023] This specification describes systems that process a sequence of point clouds using a spatio-temporal embedding neural network to generate spatio-temporal embeddings of the sequence of point clouds.

[0024] Figure 1 is a diagram of an example system 100 that includes a spatio-temporal embedding neural network 110. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, where the systems, components, and techniques described below can be implemented.

[0025] The spatio-temporal embedding neural network 100 has been configured through training to receive a sequence of point clouds 102a-n and process the sequence of point clouds 102a-n to generate spatio-temporal embeddings 112 of the sequence of point clouds 102a-n. Each of the point clouds 102a-n can represent the same environment at a respective different point in time. For example, each point cloud in the sequence of point clouds 102a-n can be generated from sensor data captured by one or more sensors onboard a vehicle operating in the environment.

[0026] In some implementations, the spatio-temporal embedding neural network 110 generates exactly one spatio-temporal embedding 112 of the sequence of point clouds 102a-n. In some other implementations, the spatio-temporal embedding neural network 110 generates a respective spatio-temporal embedding 112 for each point cloud 102a-n. For example, for each point cloud 102a-n, the spatio-temporal embedding neural network 110 can process the point cloud and each previous point cloud in the sequence of point clouds 102a-n to generate a respective spatio-temporal embedding 112 for the point cloud. As another example, for each point cloud 102a-n, the spatio-temporal embedding neural network 110 can process an intermediate representation of the point cloud and each previous point cloud in the sequence of point clouds 102a-n (e.g., an intermediate representation generated by the spatio-temporal embedding neural network 110 when generating a spatio-temporal embedding 112 corresponding to the previous point cloud) to generate a respective spatio-temporal embedding 112 for the point cloud.

[0027] ​​In some implementations, the spatio-temporal embedding neural network 110 is configured to receive as input a sequence of exactly N point clouds 102a-n to generate a spatio-temporal embedding 112 of the sequence of point clouds 102a-n. In some other implementations, the spatio-temporal embedding neural network 110 can receive a sequence comprising any number of point clouds 102a-n and generate a spatio-temporal embedding 112 of the sequence.

[0028] In some implementations, the spatio-temporal embedding neural network 110 receives each point cloud in the sequence of point clouds 102a-n simultaneously and processes the sequence of point clouds 102a-n to generate the spatio-temporal embedding 112. In some other implementations, the spatio-temporal embedding neural network 110 can receive each point cloud in the sequence of point clouds 102a-n at a respective different point in time. At each point in time at which the spatio-temporal embedding neural network 110 receives a point cloud, the spatio-temporal embedding neural network 110 can generate a spatio-temporal embedding 112 of the received point cloud by processing the received point cloud and i) each previous point cloud received at a previous point in time, ii) an intermediate representation of each previous point cloud received at a previous point in time (e.g., an intermediate representation generated by the spatio-temporal embedding neural network 110 at a previous point in time corresponding to a previous point cloud for which the spatio-temporal embedding neural network 110 generated a spatio-temporal embedding 112), or iii) both.

[0029] The spatio-temporal embedding neural network 110 can have any appropriate neural network architecture for generating the spatio-temporal embedding 112. Reference is made below to Figures 2-3B An example spatio-temporal embedding neural network is discussed in more detail.

[0030] After generating the spatio-temporal embedding 112 of the sequence of point clouds 102a-n, the spatio-temporal embedding neural network 110 can provide the spatio-temporal embedding 112 to M downstream systems 120a-m, where M > 1. Each of the one or more downstream systems 120a-m is configured to process the spatio-temporal embedding 112 to perform a respective downstream task.

[0031] For example, the downstream task of one of the downstream systems 120a-m can be object intent prediction, where the downstream system processes the spatio-temporal embedding 112 to generate a prediction of an intent trajectory of one or more objects (e.g., one or more pedestrians or one or more surrounding vehicles) in an environment represented by the sequence of point clouds 102a-n.

[0032] As another example, the downstream task of one of the downstream systems 120a-m can be object interaction prediction, where the downstream system processes the spatio-temporal embedding 112 to generate a prediction of a future interaction between two or more objects in an environment represented by the sequence of point clouds 102a-n.

[0033] As another example, a downstream task of one of the downstream systems 120a-m can be pedestrian action recognition, in which the downstream system processes the spatiotemporal embedding 112 to generate a prediction of respective actions of one or more pedestrians in the environment represented by the sequence of point clouds 102a-n.

[0034] As another example, a downstream task of one of the downstream systems 120a-m can be pedestrian gesture recognition, in which the downstream system processes the spatiotemporal embedding 112 to generate a classification of respective gestures performed by one or more pedestrians in the environment represented by the sequence of point clouds 102a-n.

[0035] As another example, a downstream task of one of the downstream systems 120a-m can be object heatmap prediction, in which the downstream system processes the spatiotemporal embedding 112 to generate an image that characterizes predicted current locations of one or more objects in the environment represented by the sequence of point clouds 102a-n.

[0036] As another example, a downstream task of one of the downstream systems 120a-m can be future trajectory prediction, in which the downstream system processes the spatiotemporal embedding 112 to generate a prediction of future trajectories of one or more objects in the environment represented by the sequence of point clouds 102a-n. For example, the downstream system can process the spatiotemporal embedding 112 to generate respective predicted heatmaps for one or more future time points that characterize predicted locations of the one or more objects at the future time points.

[0037] As another example, a downstream task of one of the downstream systems 120a-m can be object type classification, in which the downstream system processes the spatiotemporal embedding 112 to generate a classification of one or more objects in the environment represented by the sequence of point clouds 102a-n.

[0038] In some implementations, one or more of the downstream systems 120a-m can query respective subsets of the spatiotemporal embedding 112 and process only the subsets of the spatiotemporal embedding 112 to perform respective downstream tasks. For example, the spatiotemporal embedding 112 can include a respective voxel embedding for each voxel in a voxel grid imposed on the environment represented by the sequence of point clouds 102a-n. The downstream systems 120a-m can process only the voxel embeddings in the spatiotemporal embedding 112 that are relevant to the respective downstream task. That is, the downstream systems can only need the voxel embeddings that correspond to voxels in a particular region of the environment represented by the sequence of point clouds 102a-n.

[0039] In some implementations, the system 100 can be an onboard system of a vehicle (e.g., an autonomous or semi-autonomous vehicle) operating in an environment represented by a sequence of point clouds 102a-n. For example, each of the point clouds 102a-n can be generated from sensor data captured by one or more sensors (e.g., one or more lidar sensors or one or more radar sensors) onboard the vehicle at respective different points in time. As a particular example, each time the vehicle generates a new point cloud, the system 100 can provide the new point cloud to the spatio-temporal embedding neural network 100 to generate a new spatio-temporal embedding 112 that encodes information from the new point cloud and each of one or more previous point clouds generated at previous points in time. The M downstream systems 120a-m can then use the spatio-temporal embedding 112 to make driving decisions for the vehicle.

[0040] In some other implementations, the system 100 can be a component of a server system (e.g., a server system hosted within a data center) that can be a distributed computing system having hundreds or thousands of computers in one or more locations. For example, the spatio-temporal embedding neural network 110 can generate spatio-temporal embeddings 112 to be used as training examples for training the M downstream systems 120a-m. As a particular example, a training system can process the spatio-temporal embeddings 112 using the downstream systems to generate system outputs, and the training system can use errors in the system outputs to update one or more parameters in the downstream systems.

[0041] In some implementations, the spatio-temporal embedding neural network 110 can be trained simultaneously with one or more of the M downstream systems 120a-m. For example, a training system can process the spatio-temporal embeddings 112 using the downstream systems to generate system outputs, and then use errors in the system outputs to update i) one or more parameters of the downstream systems and ii) one or more parameters of the spatio-temporal embedding neural network 110. In this way, the spatio-temporal embedding neural network 110 can be trained to generate spatio-temporal embeddings 112 that include information useful to the downstream systems.

[0042] In some implementations, one or more of the M downstream systems 120a-m can be configured to perform a downstream task for which the spatio-temporal embedding neural network 110 has not been trained. That is, during training, the training system can process the temporal embeddings 112 generated by the spatio-temporal embedding neural network 110 to perform a first task. Then, during inference (e.g., after the spatio-temporal embedding neural network 110 has been deployed to an on-board system of a vehicle), the system 100 can process the spatio-temporal embeddings 112 generated by the spatio-temporal embedding neural network 110 to perform a second task that is different from the first task. Thus, the information that the spatio-temporal embedding neural network 110 learns to encode into the spatio-temporal embeddings 112 during training can also be used for a different task that was not performed during training.

[0043] As a particular example, the training system can train the spatio-temporal embedding neural network 110 on a downstream task of object heat map prediction, in which the downstream system processes the spatio-temporal embeddings 112 to generate an image that characterizes predicted locations of one or more objects in the environment represented by the sequence of point clouds 102a-n. The training system can use a per-pixel weighted sigmoid cross-entropy loss to determine an error between the generated image and a ground-truth image. The training system can then backpropagate the error through the downstream system to the spatio-temporal embedding neural network 110.

[0044] Figure 2 is a diagram of an example spatio-temporal embedding neural network 200 that receives and processes a sequence of point clouds to generate spatio-temporal embeddings of the sequence of point clouds. Figure 3A and Figure 3B are diagrams of example spatio-temporal embedding neural networks 300 and 350 that receive each point cloud in a sequence of point clouds at a respective different point in time and generate, for each received point cloud, a respective spatio-temporal embedding that represents the received point cloud and each previous point cloud in the sequence of point clouds.

[0045] Referring to Figure 2 , the spatio-temporal embedding neural network 200 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0046] The spatio-temporal embedding neural network 200 has been configured through training to receive and process a sequence of point clouds 202a-n to generate spatio-temporal embeddings 232n of the sequence of point clouds 202a-n. Each point cloud in the sequence of point clouds 202a-n represents the same environment at a respective different point in time.

[0047] The spatio-temporal embedding neural network 200 includes three subnetworks: a voxel embedding neural network 210, a spatial embedding neural network 220, and a temporal embedding neural network 230.

[0048] For each point cloud 202 in the sequence of point clouds 202a-n, the voxel embedding neural network 210 has been configured through training to receive the point cloud 202 and generate a respective voxelized embedding 212 of the point cloud 202. The voxelized embedding 212 of the point cloud 202 encodes information about the points of the point cloud 202 in each voxel of a plurality of voxels for the point cloud 202.

[0049] In some implementations, the voxel embedding neural network 210 generates the respective voxelized embedding 212 for each point cloud 202 in the sequence of point clouds 202a-n in parallel.

[0050] For convenience, the following description will refer to the voxel embedding neural network 210 processing a single point cloud 202; it should be understood that the voxel embedding neural network 210 can perform the techniques described below for each point cloud 202 in the sequence of point clouds 202a-n.

[0051] The voxel embedding neural network 210 can divide the environment represented by the point cloud 202 into a voxel grid including a plurality of voxels, where each voxel represents a volume of space in the environment. The voxel grid can be predetermined; that is, each voxel in the voxel grid can have a predetermined shape and size. For example, the voxel grid can be a Cartesian grid of the environment, where each voxel in the voxel grid represents a cubic or cuboid voxel volume of space in the environment.

[0052] The voxel embedding neural network 210 can assign each point in the point cloud 202 to the voxel in which the point is located. The voxel embedding neural network 210 can then process the point cloud 202a-n to generate a respective voxelized embedding 212 of the point cloud 202. The voxelized embedding 212 of the point cloud 202 can include a respective voxel embedding for each voxel in the voxel grid of the point cloud 202. The voxel embedding for a respective voxel encodes information from each point assigned to the voxel.

[0053] The voxel embedding neural network 210 can have any appropriate network architecture to generate the voxelized embedding 212 of the point cloud 202.

[0054] For example, the voxel embedding neural network 210 can process each point in the point cloud 202 using one or more neural network layers (e.g., one or more fully connected neural network layers) to generate a respective point embedding for the point. For a voxel in the voxel grid, the voxel embedding neural network 210 can combine the respective point embeddings for each point assigned to the voxel to generate a voxel embedding for the voxel. For example, for each voxel in the voxel grid, the voxel embedding neural network 210 can process the point embeddings for the points assigned to the voxel using a max-pooling layer to generate an initial voxel embedding. The voxel embedding neural network 210 can then process each initial voxel embedding layer in the voxel grid using one or more convolutional neural networks to generate a respective voxel embedding.

[0055] As another example, the voxel embedding neural network 210 can generate multiple different voxel grids corresponding to respective different points of view and generate a respective view-dependent voxelized embedding for each generated voxel grid. The voxel embedding neural network 210 can then combine the multiple view-dependent voxelized embeddings to generate voxelized embeddings 212a-n for the point clouds 202a-n. In these implementations, the voxelized embeddings 212a-n can include a respective voxel embedding for each voxel in a particular voxel grid of the multiple different voxel grids. Reference is made below to FIG. 3 for further details. Figure 4 Example voxel embedding neural networks are discussed in further detail.

[0056] For each point cloud 202 in the sequence of point clouds 202a-n, the spatial embedding neural network 220 has been configured through training to receive the voxelized embedding 212 for the point cloud 202 and process the voxelized embedding 212 to generate a respective spatial embedding 222 for the point cloud 202. The spatial embedding 222 for the point cloud 202 encodes spatial information for the environment represented by the point cloud 202 at the time the sensor data used to generate the point cloud 202 was captured.

[0057] In some implementations, the spatial embedding neural network 220 generates the respective spatial embedding 222 for each point cloud 202 in the sequence of point clouds 202a-n in parallel.

[0058] In some implementations, the spatial embedding 222 includes a respective spatial embedding for each voxel represented by the voxelized embedding 212 for the point cloud 202.

[0059] The spatial embedding neural network 220 can have any appropriate network architecture to generate the spatial embeddings 222a-n for the point clouds 202a-n.

[0060] For example, the spatial embedding neural network 220 can be a convolutional neural network that performs spatial convolutions on the voxel embeddings of the voxels in the voxelized embedding 212 to generate corresponding spatial embeddings 222 of the point clouds 202. That is, the spatial embedding neural network 220 can process the voxel embeddings of the voxels by convolving one or more learned convolutional filters over the voxel grid.

[0061] As a particular example, the spatial embedding neural network 220 can include one or more inception modules. An inception module receives a module input and separately processes the module input using a plurality of different stacks of one or more convolutional neural network layers. The inception module can then combine the respective outputs of the plurality of stacks of convolutional neural network layers (e.g., using concatenation) to generate a module output of the inception module.

[0062] For example, each stack of convolutional neural network layers of an inception module can process the module input using different sized convolutional filters (e.g., (1x1), (3x3), and (5x5) convolutional filters). In some implementations, one or more stacks of convolutional neural network layers can factorize a respective NxN convolutional filter into a first 1xN filter and a second Nx1 filter.

[0063] In some implementations, one or more inception modules of the spatial embedding neural network 220 further include a max-pooling layer. In some such implementations, a subsequent of the max-pooling layer can be one or more convolutional neural network layers. As described above, an inception module can combine (e.g., using concatenation) the output of the max-pooling layer (or a subsequent convolutional neural network layer) with the outputs of the plurality of stacks of convolutional neural network layers.

[0064] As another particular example, the spatial embedding neural network 220 can include one or more ResNet blocks. A ResNet block is a block of multiple neural network layers (e.g., convolutional neural network layers) that includes one or more skip connections between the respective neural network layers.

[0065] In some implementations, the spatio-temporal embedding neural network 200 does not include the voxel embedding neural network 210, and the spatio-temporal embedding neural network 200 provides the sequence of point clouds 202a-n directly to the spatial embedding neural network 220. That is, the spatial embedding neural network 220 can be configured to directly process the point clouds 202a-n to generate the corresponding spatial embeddings 222a-n.

[0066] The temporal embedding neural network 230 has been configured through training to receive the spatial embeddings 222a-n of the point clouds 202a-n and to process the spatial embeddings 222a-n to generate the spatio-temporal embedding 232. The spatio-temporal embedding 232 encodes temporal information from the sequence of spatial embeddings 222a-n as well as spatial information from each spatial embedding 222a-n.

[0067] In some implementations, the spatio-temporal embedding 232 includes a respective spatio-temporal embedding for each voxel represented by the voxelized embedding 212 of the point cloud 202.

[0068] The temporal embedding neural network 230 can have any appropriate network architecture that generates the spatio-temporal embedding 232 of the point clouds 202a-n.

[0069] For example, the temporal embedding neural network 230 can process the sequence of spatial embeddings 222a-n using a convolutional neural network. As a particular example, the temporal embedding neural network 230 can include one or more one-dimensional convolutional neural network layers that process the sequence of spatial embeddings 222a-n along the temporal dimension.

[0070] As another example, the temporal embedding neural network 230 can process the sequence of spatial embeddings 222a-n using a recurrent neural network. The recurrent neural network can process each spatial embedding 222a-n in turn, and the output of the recurrent neural network after processing the final spatial embedding 222n corresponding to the final point cloud 202n can be the spatio-temporal embedding 232 of the sequence of point clouds 202a-n.

[0071] As a particular example, at a first processing time point in a sequence of N processing time points, the recurrent neural network can process a first spatial embedding 222a corresponding to a first point cloud 202a. Then, at each subsequent processing time point i in the sequence of N processing time points, the recurrent neural network can process i) an i-th spatial embedding 222i corresponding to an i-th point cloud 202i and ii) an output of the recurrent neural network at processing time point i-1 and / or an intermediate output generated by the recurrent neural network at processing time point i-1 (e.g., an output of one or more hidden layers of the recurrent neural network that is not the output of an output layer of the recurrent neural network). The output of the recurrent neural network at the final processing time point N can be the spatio-temporal embedding 232.

[0072] As another example, the temporal embedding neural network 230 can process the sequence of spatial embeddings 222a-n using an attention-based neural network. As a particular example, the temporal embedding neural network 230 can include a transformer-based neural network that includes one or more self-attention neural network layers.

[0073] After generating the spatio-temporal embedding 232 of the sequence of point clouds 202a-n, the spatio-temporal embedding neural network 200 can provide the spatio-temporal embedding 232 to one or more downstream systems, e.g., the above-referenced Figure 2 one or more downstream systems are described.

[0074] Referring to Figure 3A , the spatio-temporal embedding neural network 300 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0075] The spatio-temporal embedding neural network 300 has been configured through training to receive a new point cloud 302 at each time point in a sequence of time points and process the point cloud 302 to generate a spatio-temporal embedding 332 that encodes information from the point cloud 302 and each previous point cloud received at a respective previous time point.

[0076] For example, if the spatio-temporal embedding neural network 300 is deployed on-board a vehicle, and the point clouds are generated from sensor data captured by one or more sensors on-board the vehicle, the spatio-temporal embedding neural network 300 can receive each new point cloud 302 as the point cloud 302 is generated from the sensor data.

[0077] The spatio-temporal embedding neural network 300 includes three sub-networks: a voxel embedding neural network 310, a spatial embedding neural network 320, and a temporal embedding neural network 330. The spatio-temporal embedding neural network 300 also includes a data store 340.

[0078] For each point cloud 302 received by the spatio-temporal embedding neural network 300, the voxel embedding neural network 310 has been configured through training to receive the point cloud 302 and generate a respective voxelized embedding 312 of the point cloud 302. The voxel embedding neural network 310 can have any appropriate network architecture, e.g., the above-referenced Figure 2 network architectures described.

[0079] For each point cloud 302 received by the spatio-temporal embedding neural network 300, the spatial embedding neural network 320 has been configured through training to receive the voxelized embedding 312 of the point cloud 302 and process the voxelized embedding 312 to generate a respective spatial embedding 322 of the point cloud 302. The spatial embedding neural network 320 can have any appropriate architecture, e.g., the above-referenced Figure 2 network architectures described.

[0080] In some implementations, the spatio-temporal embedding neural network 300 does not include the voxel embedding neural network 310, but instead provides the sequence of point clouds 302a-n directly to the spatial embedding neural network 320.

[0081] The temporal embedding neural network 330 has been configured through training to receive the spatial embedding 322 of the new point cloud 302 and process the spatial embedding 322 of the new point cloud 302 to generate a spatio-temporal embedding 332 that represents the new point cloud and a previous point cloud received at a previous time point by the spatio-temporal embedding neural network 300. The temporal embedding neural network 330 can have any appropriate network architecture, e.g., the network architectures described above with reference to Figure 2 the network architectures described above with reference to

[0082] The data store 340 is configured to store previous spatio-temporal embeddings 342 generated by the temporal embedding neural network 330 at previous time points. The previous spatio-temporal embeddings 342 correspond to previous point clouds received at the previous time points and encode information from the previous point clouds and each other point cloud received at the respective previous time points.

[0083] When generating the new spatio-temporal embedding 332 of the new point cloud 302, the spatio-temporal embedding neural network 330 can retrieve the previous spatio-temporal embedding 342 from the data store 340 and process the previous spatio-temporal embedding 342 to generate the new spatio-temporal embedding 332.

[0084] Instead of or in addition to storing the previous spatio-temporal embeddings 342, the data store 340 can store intermediate outputs generated by the temporal embedding neural network 330 at previous time points as the temporal embedding neural network 330 is generating the previous spatio-temporal embeddings 342. For example, the intermediate outputs can include respective outputs of one or more hidden layers of the temporal embedding neural network 330 (e.g., cell states of an LSTM network). The temporal embedding neural network 330 can then retrieve the intermediate outputs generated at previous time points when generating the new spatio-temporal embedding 332 of the new point cloud 302.

[0085] As a particular example, the temporal embedding neural network 330 can include a recurrent neural network (e.g., an LSTM network) that processes i) the new spatial embedding 322 of the new point cloud 302 and ii) the previous spatio-temporal embedding 342 and / or the intermediate outputs generated at the previous time point.

[0086] In some implementations, one or more sensors that capture the sensor data to generate the point clouds 302 can move through the environment represented by the point clouds 302. For example, if the spatio-temporal embedding neural network 300 is deployed on-board a vehicle and the point clouds represent an environment in which the vehicle is operating, then the vehicle can move through the environment as the respective point clouds are generated.

[0087] In some such implementations, the spatio-temporal embedding neural network 300 also receives, as input at each time point, a set of motion parameters 304. The motion parameters 304 can describe a positioning of the vehicle in the environment represented by the new point cloud 302. Alternatively or additionally, the motion parameters 304 can describe a motion of the vehicle through the environment represented by the new point cloud 302. As a particular example, the motion parameters 302 can include one or more of: a position of the vehicle, a heading of the vehicle, a velocity of the vehicle, an acceleration of the vehicle, or a jerk of the vehicle.

[0088] The temporal embedding neural network 330 can receive the motion parameters 304 and process the motion parameters 304 along with the new spatial embedding 322 and the previous spatio-temporal embeddings 342 (and / or intermediate outputs generated at previous time points) to generate the spatio-temporal embedding 332. For example, the temporal embedding neural network 330 can concatenate the motion parameters 304 to the new spatial embedding 322 to generate a concatenated representation, which is then processed as input. The previous spatio-temporal embeddings 342 (and / or intermediate outputs generated at previous time points) can encode respective motion parameters from each previous time point, so the temporal embedding neural network 330 can have access to the motion parameters of the current time point and all previous time points when generating the new spatio-temporal embedding 332.

[0089] Using motion parameters of the vehicle when generating a new point cloud can encode ego-motion into the spatio-temporal embedding 332. Because the vehicle can move through the environment, without incorporating ego-motion, the spatio-temporal embedding 332 can be inaccurate due to the changing position of objects in the environment relative to the vehicle as the vehicle moves. To overcome this problem without incorporating ego-motion, the spatio-temporal embedding neural network 300 can recompute a respective spatial embedding 322 for each previous point cloud received at a respective previous time point relative to the new positioning of the vehicle in the environment. The temporal embedding neural network 330 can then process (e.g., using a recurrent neural network as described above) each recomputed previous spatial embedding and the new spatial embedding 322 to generate the spatio-temporal embedding 332. However, recomputing each spatial embedding at each time point can be extremely costly in terms of time and computation. Using the motion parameters 304 to incorporate ego-motion can be a significantly more efficient way to generate accurate spatio-temporal embeddings 332.

[0090] After generating the spatio-temporal embedding 332 corresponding to the new point cloud 302, the spatio-temporal embedding neural network 300 can provide the spatio-temporal embedding 332 to one or more downstream systems. The spatio-temporal embedding neural network 300 can also provide the new spatio-temporal embedding 332 to the data store 340 to be used at a subsequent time point when the spatio-temporal embedding neural network 300 receives a subsequent point cloud.

[0091] Returning to Figure 3BThe spatio-temporal embedding neural network 350 is an example of a system implemented as computer programs on one or more computers in one or more locations, where the systems, components, and techniques described below can be implemented.

[0092] The spatio-temporal embedding neural network 350 has been configured through training to receive a new point cloud 352 at each time point in a sequence of time points and process the point cloud 352 to generate a spatio-temporal embedding 382 that encodes information from the point cloud 352 and each previous point cloud received at a respective previous time point.

[0093] The spatio-temporal embedding neural network 350 includes three sub-networks: a voxel embedding neural network 360, a spatial embedding neural network 370, and a temporal embedding neural network 380. The spatio-temporal embedding neural network 350 also includes a data store 390.

[0094] For each point cloud 352 received by the spatio-temporal embedding neural network 350, the voxel embedding neural network 360 has been configured through training to receive the point cloud 352 and generate a respective voxelized embedding 362 of the point cloud 352. The voxel embedding neural network 360 can have any appropriate network architecture, e.g., the network architectures described above with reference to Figure 2 .

[0095] For each point cloud 352 received by the spatio-temporal embedding neural network 350, the spatial embedding neural network 370 has been configured through training to receive the voxelized embedding 362 of the point cloud 352 and process the voxelized embedding 362 to generate a respective spatial embedding 372 of the point cloud 352. The spatial embedding neural network 370 can have any appropriate architecture, e.g., the network architectures described above with reference to Figure 2 .

[0096] In some implementations, the spatio-temporal embedding neural network 350 does not include the voxel embedding neural network 360, but instead provides the sequence of point clouds 352a-n directly to the spatial embedding neural network 370.

[0097] The temporal embedding neural network 380 has been configured through training to receive the spatial embedding 372 of the new point cloud 352 and process the spatial embedding 372 of the new point cloud 352 to generate a spatio-temporal embedding 382 that represents the new point cloud and previous point clouds received by the spatio-temporal embedding neural network 350 at previous time points. The temporal embedding neural network 380 can have any appropriate network architecture, e.g., the network architectures described above with reference to Figure 2 .

[0098] The data store 390 is configured to store previous spatial embeddings 392 generated by the temporal embedding neural network 370 at respective previous time points. The previous spatial embeddings 392 correspond to respective previous point clouds received at the respective previous time points. For example, the data store 390 can store P previous spatial embeddings 392 corresponding to P previous time points at which the spatio-temporal embedding neural network 350 received a point cloud, P > 1.

[0099] When generating a new spatio-temporal embedding 382 of a new point cloud 352, the temporal embedding neural network 380 can retrieve P previous spatial embeddings 392 from the data store 390 and process the previous spatial embeddings 392 to generate the new spatio-temporal embedding 382.

[0100] As a particular example, the temporal embedding neural network 380 can include a convolutional neural network that processes i) the previous spatial embeddings 392 corresponding to respective previous point clouds and ii) the new spatial embedding 372 corresponding to the new point cloud 352 using one-dimensional convolutions along the temporal dimension.

[0101] As described above with reference to FIG. 3, in some implementations, one or more sensors that capture sensor data to generate the point clouds 352 can move through the environment represented by the point clouds 352, e.g., if the spatio-temporal embedding neural network 350 is deployed on-board a vehicle.

[0102] In some such implementations, the spatio-temporal embedding neural network 350 also receives, as input at each time point, a set of motion parameters 354. The motion parameters 354 can describe a positioning and / or motion of the vehicle in the environment represented by the new point cloud 352.

[0103] The temporal embedding neural network 380 can receive the motion parameters 354 and process the motion parameters 354 along with the new spatial embedding 372 and the previous spatial embeddings 392 to generate the spatio-temporal embedding 382.

[0104] In some implementations, the temporal embedding neural network 380 processes the new spatial embedding 372 in accordance with the motion parameters 354 to center a coordinate system of the new spatial embedding 372 at a common center point in the environment. For example, if the spatial embedding 372 includes a respective spatial embedding for each voxel represented by the voxelized embedding 362, the temporal embedding neural network 380 can transform the spatial embedding in accordance with i) a current location of the vehicle in the environment identified in the motion parameters 354 and ii) a stationary common center point in the environment. As a particular example, the common center point can be a location of the vehicle in the environment at the first time point corresponding to the first point cloud 352 received by the spatio-temporal embedding neural network 350.

[0105] The coordinate system of each previous spatial embedding 392 can also be centered at a common center point in the environment (e.g., because the temporal embedding neural network 380 centered each previous spatial embedding 392 at the respective previous time point using the respective motion parameter). Thus, each spatial embedding 372 and 392 can be represented from a common frame of reference.

[0106] In some other implementations, the data store 390 stores respective motion parameters corresponding to each previous time point of the previous spatial embeddings 392. Then, to generate the spatio-temporal embedding 382, the temporal embedding neural network 380 can process i) the new spatial embedding 372 and the new motion parameter 354 and ii) the P previous spatial embeddings 392 and the P sets of their corresponding motion parameters.

[0107] As described above with reference to Figure 3A the temporal embedding neural network 350 can avoid re-computing a respective spatial embedding of each previous point cloud at each time point and then processing (e.g., using the convolutional neural network described above) the re-computed previous spatial embeddings and the new spatial embedding 372 to generate the spatio-temporal embedding 382.

[0108] After generating the spatio-temporal embedding 382 corresponding to the new point cloud 352, the spatio-temporal embedding neural network 350 can provide the spatio-temporal embedding 382 to one or more downstream systems. The spatio-temporal embedding neural network 350 can also provide the new spatial embedding 372 generated by the spatial embedding neural network 370 that will be used at a subsequent time point when the spatio-temporal embedding neural network 350 receives a subsequent point cloud to the data store 390.

[0109] Figure 4 is a diagram of an example voxel embedding neural network 400. The voxel embedding neural network 400 is an example of a system implemented as computer programs on one or more computers in one or more locations, where the systems, components, and techniques described below can be implemented. The voxel embedding neural network 400 can be a component of a spatio-temporal embedding neural network (e.g., the spatio-temporal embedding neural network 110 depicted in Figure 1 FIG. 1) configured to generate spatio-temporal embeddings of a sequence of point clouds.

[0110] The voxel embedding neural network 400 has been configured through training to receive a point cloud 402 and generate a voxelized embedding 432 of the point cloud 402. The point cloud 402 contains K points.

[0111] The voxel embedding neural network 400 includes a preprocessing engine 410, Q view neural networks 420a-q (Q > 1), and a combination engine 430. In some implementations, three coordinates representing a location in an environment represented by the point cloud 402 define each point in the point cloud 402. Each point can also have one or more additional features, e.g., an intensity value.

[0112] The preprocessing engine 410 is configured to receive the point cloud 402 and generate a view network input 412.

[0113] In some implementations, the preprocessing engine 410 processes each point in the point cloud using one or more neural network layers (e.g., one or more fully connected layers). The neural network layers can embed each point into a high-dimensional feature space.

[0114] In some implementations, the preprocessing engine 410 also obtains one or more images 404 depicting the same environment as the point cloud 402. The preprocessing engine 410 can embed information from the images 404 into the points in the point cloud 402.

[0115] For example, the images 404 can be color images, and the preprocessing engine 410 can project color values (e.g., RGB values) from the images 404 onto the points in the point cloud 402. That is, the preprocessing engine 410 can determine, for each of the one or more points in the point cloud 402, a corresponding pixel in the images 404. The point in the point cloud 402 and the corresponding pixel in the images 404 each depict the same object in the environment represented by the point cloud 402 and the images 404. The preprocessing engine 410 can then assign the color value of the corresponding pixel to the point.

[0116] Alternatively or additionally, the preprocessing engine 410 can process the one or more images 404 using a neural network (e.g., a convolutional neural network) to generate a respective feature map for each image 404. The preprocessing engine 410 can then project the computed features of the feature maps onto the points in the point cloud 402. That is, the preprocessing engine 410 can determine, for each of the one or more points in the point cloud 402, a corresponding pixel in the feature maps, and then assign the features of the respective pixel to the point.

[0117] After generating the view network input 412, the pre-processing engine 410 can provide the view network input 412 to each of the Q view neural networks 420a-q. That is, each of the Q view neural networks 420a-q receives the same input 412. The view network input 412 can be a K x L matrix, where each row in the matrix corresponds to a point of the K points in the point cloud 402 and includes the L-dimensional feature embedding for that point generated by the pre-processing engine 410. In some implementations, L = 3, where the view network input 412 includes respective coordinates for each point in the environment represented by the point cloud 402.

[0118] Each view neural network 420a-q processes the view network input 412 with respect to a different viewpoint to generate a respective view embedding 422a-q. Each view embedding 422a-q is an embedding of the view network input 412 with respect to the corresponding viewpoint in a high-dimensional feature space that can capture object semantics and shape information.

[0119] For example, a view neural network 420 can process the view network input 412 with respect to an aerial viewpoint (i.e., viewing the environment from above) to generate a view embedding 422 corresponding to the aerial viewpoint. As another example, a view neural network 420 can process the view network input 412 with respect to a perspective viewpoint (i.e., viewing the environment from the viewpoint of one or more sensors of the vehicle in the environment) to generate a view embedding 422 corresponding to the perspective viewpoint.

[0120] Each view neural network 420 can first partition the environment represented by the point cloud 402 into a grid of voxels including a plurality of voxels, where each voxel represents a volume of space in the environment. Each view neural network 420 partitions the environment into voxels according to the corresponding viewpoint. For example, a perspective view neural network 420 can partition the environment into voxels that radiate outward from the vehicle in the environment, while an aerial view neural network 420 can partition the environment into a grid of cubic or cuboid voxels viewed from above the environment. After partitioning the environment into voxels, each view neural network 420 can assign each point in the view network input 412 to the voxel at which the point resides.

[0121] In some implementations, each view neural network 420 can use dynamic voxelization; that is, the view neural networks 420a-q do not fix the number of voxels or the number of points per voxel in advance. Rather, each view neural network 420 can dynamically process all voxels and all points per voxel to generate the respective view embedding.

[0122] As a particular example, each view neural network 420 can process each point in the view network input 412 (e.g., by processing each point using one or more fully connected neural network layers) to generate a view-dependent point embedding for the point. The neural network 420 can then aggregate the respective point embeddings for each point in a voxel to generate an initial voxel embedding for the voxel. For example, the view neural network 420 can apply a max-pooling layer to the point embeddings for the points in the voxel. The view neural network 420 can then process the respective initial voxel embedding for each voxel in the voxel grid to generate a view embedding 422. For example, the view neural network 420 can apply one or more convolutional neural network layers to each initial voxel embedding in the voxel grid.

[0123] Each view embedding 422a-q can include a respective voxel embedding for each voxel in the voxel grid corresponding to the respective view neural network 420a-q.

[0124] The combination engine 430 can obtain the respective view embeddings 422a-q for the view neural networks 420a-q and combine the view embeddings 422a-q to generate the voxelized embedding 432.

[0125] For example, for each point in the point cloud 402, the combination engine 430 can generate a final point embedding for the point by combining the voxel embeddings in the view embeddings 422a-q for the voxel to which the point is assigned for each view embedding 422a-q. As a particular example, the combination engine 430 can generate a final point embedding for a point by concatenating the respective voxel embeddings for the voxel to which the point is assigned in the respective view embeddings 422a-q.

[0126] The combination engine 430 can then combine the respective final point embeddings for each point in the point cloud 402 to generate the voxelized embedding 432.

[0127] As a particular example, the combination engine 430 can divide the environment represented by the point cloud 402 into a final voxel grid including a plurality of final voxels. The voxelized embedding 432 can include a respective final voxel embedding for each final voxel in the final voxel grid. In some implementations, the final voxel grid can be the same as the respective voxel grid corresponding to one of the view neural networks 420a-q. For example, the final voxel grid can be an aerial voxel grid (also referred to as a Cartesian voxel grid) having cubic or cuboid final voxels.

[0128] The combination engine 430 can assign each point in the point cloud 402 to a final voxel in the final voxel grid that the point is in. The combination engine 430 can then combine the final point embeddings of the points assigned to each final voxel to generate a final voxel embedding for the final voxel for each final voxel. As a particular example, the combination engine 430 can process the final point embeddings using a max-pooling layer to generate the final voxel embedding. After generating the final voxel embedding for each final voxel in the final voxel grid, the combination engine 430 can output a voxelized embedding 432 that includes each final voxel embedding.

[0129] The voxel embedding neural network 400 can provide the voxelized embedding 432 to a spatial embedding neural network, e.g., the spatial embedding neural network 220 depicted in Figure 2 FIG. 5B.

[0130] Figure 5 is a flow diagram of an example process for generating a spatio-temporal embedding of a sequence of point clouds. For convenience, the process 500 will be described as performed by a system of one or more computers located in one or more locations. For example, a system, e.g., the system 100, described in FIG. 5B, appropriately programmed, can perform the process 500. Figure 1

[0131] The system obtains a time series that includes respective point cloud inputs for each of a plurality of time points (step 502). Each point cloud input can include point cloud data generated from sensor data captured by one or more sensors of a vehicle at the respective time point.

[0132] The system processes each point cloud input using a first neural network to generate a respective spatial embedding of the point cloud input that characterizes the point cloud input (step 504).

[0133] The system processes the spatial embeddings of the point cloud inputs using a second neural network to generate a spatio-temporal embedding that characterizes the point cloud inputs in the time series (step 506).

[0134] After generating the spatio-temporal embedding, the system can provide the spatio-temporal embedding to each of a plurality of task-specific downstream neural networks. Each task-specific downstream neural network can be configured to process the spatio-temporal embedding to generate a prediction output for a different prediction task.

[0135] In some implementations, the first neural network and the second neural network can be jointly trained on a particular prediction task. In some such implementations, the particular prediction task is not one of the prediction tasks corresponding to the task-specific downstream neural networks.

[0136] ​The term "configured" is used in the specification to mean that a system, comprising one or more computers, has been programmed with software, firmware, hardware or a combination thereof to perform particular operations or actions corresponding to the configuration. The term "configured" is used in the specification to mean that one or more computer programs, when executed, cause the computer to perform particular operations or actions.

[0137] Embodiments of the functional operations and the subject matter described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0138] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0139] A computer program, which can also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or code, of the software). A computer program can be deployed for execution on one or more computers or other programmable machines, these machines located at one site or distributed across multiple sites and

[0140] In this specification, the term“database” is used broadly to refer to any collection of data: the data need not be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, an index database can include multiple collections of data, each of which can be organized and accessed differently.

[0141] Similarly, in this specification, the term“engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may

[0142] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0143] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0144] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0145] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0146] The data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing the common and compute-intensive parts of machine learning training or production (i.e., inference, workloads).

[0147] The machine learning models can be implemented and deployed using a machine learning framework (e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework).

[0148] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0149] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established using computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the device). Data generated at the client device (e.g., a result of the user interaction) can be received from the device at the server.

[0150] In addition to the embodiments described above, the following embodiments also have innovations:

[0151] Embodiment 1 is a method comprising:

[0152] obtaining a time series comprising respective point cloud inputs corresponding to each of a plurality of time points, each point cloud input comprising point cloud data generated from sensor data captured by one or more sensors of a vehicle at the respective time point;

[0153] processing each point cloud input using a first neural network to generate a respective spatial embedding characterizing the point cloud input; and

[0154] processing the spatial embeddings of the point cloud inputs using a second neural network to generate a spatio-temporal embedding characterizing the point cloud inputs in the time series.

[0155] Example 2 is the method of Example 1, further comprising:

[0156] processing the spatiotemporal embedding using a neural network of each specialized task of the plurality of specialized tasks of the neural network, wherein each specialized task of the neural network is configured to process the spatiotemporal embedding to generate a prediction output for a different prediction task.

[0157] Example 3 is the method of Example 2, wherein the first neural network and the second neural network have been jointly trained on a first prediction task, and wherein the first prediction task is not one of the different prediction tasks corresponding to the plurality of specialized tasks of the neural network.

[0158] Example 4 is the method of any of Examples 1-3, wherein processing each point cloud input comprises:

[0159] partitioning the point cloud data into a plurality of voxels,

[0160] generating a feature representation comprising a feature of each voxel, and

[0161] processing the feature representation using the first neural network to generate a spatial embedding.

[0162] Example 5 is the method of Example 4, wherein generating the feature representation comprises:

[0163] processing the point cloud data using one or more view neural networks, wherein a view neural network extracts features from the point cloud data with respect to a viewpoint; and

[0164] combining outputs of the one or more view neural networks to generate the feature representation.

[0165] Example 6 is the method of Example 5, wherein the one or more view neural networks comprise an aerial view neural network that extracts features with respect to an aerial view and a perspective view neural network that extracts features with respect to a perspective view.

[0166] Example 7 is the method of any of Examples 5 or 6, wherein processing the point cloud data using the one or more view neural networks comprises processing each point in the point cloud data using a fully connected layer shared by the one or more view neural networks to embed the point into a high-dimensional feature space.

[0167] Example 8 is the method of any of Examples 5-7, wherein combining the outputs of the one or more view neural networks comprises concatenating the outputs of the one or more view neural networks.

[0168] Example 9 is the method of any of Examples 1-8, wherein processing the spatial embedding using the second neural network comprises processing the spatial embedding using a one-dimensional convolutional neural network layer.

[0169] Example 10 is a system comprising: one or more computers and one or more storage devices storing instructions that are operable

[0170] Example 11 is a computer storage medium encoded with a computer program, the program comprising instructions operable to program a data processing apparatus to perform the method of any of Examples 1 to 9.

[0171] Although this description contains many specifics, these should not be construed as limiting the scope of any invention or of what can be claimed, but merely as describing a particular implementation, or implementations, of certain embodiments. Certain features that are described in this description in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0172] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring such order nor should it be interpreted that all operations are to be performed to achieve desirable results. In some cases, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0173] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases multitasking and parallel processing can be advantageous.

Claims

1. A method for processing point cloud inputs, comprising: obtaining a time series comprising a respective point cloud input corresponding to each of a plurality of time points, each point cloud input comprising point cloud data generated from sensor data captured by one or more sensors of a vehicle at the respective time point; processing each point cloud input using a first neural network to generate a respective spatial embedding characterizing the point cloud input, the processing each point cloud input comprising, for each point cloud input: partitioning the point cloud data into a plurality of voxels, generating a feature representation comprising a feature for each voxel, and processing the feature representation using the first neural network to generate the spatial embedding; and processing the spatial embeddings of the point cloud inputs using a second neural network to generate a spatio-temporal embedding characterizing the point cloud inputs in the time series.

2. The method of claim 1, further comprising: processing the spatio-temporal embedding using each of a plurality of task-specific neural networks, wherein each task-specific neural network is configured to process the spatio-temporal embedding to generate a prediction output for a different prediction task.

3. The method of claim 2, wherein, the first neural network and the second neural network have been jointly trained on a first prediction task, and wherein the first prediction task is not one of the different prediction tasks corresponding to the plurality of task-specific neural networks.

4. The method of claim 1, wherein, generating the feature representation comprises: processing the point cloud data using one or more view neural networks, wherein a view neural network extracts features from the point cloud data with respect to a viewpoint; and combining outputs of the one or more view neural networks to generate the feature representation.

5. The method of claim 4, wherein, the one or more view neural networks comprise a bird’s eye view neural network that extracts features with respect to a bird’s eye view and a perspective view neural network that extracts features with respect to a perspective view.

6. The method of any one of claims 4 or 5, wherein, processing the point cloud data using the one or more view neural networks comprises processing each point in the point cloud data with a fully connected layer shared by the one or more view neural networks to embed the point into a high-dimensional feature space.

7. The method of any one of claims 4 or 5, wherein, combining the outputs of the one or more view neural networks comprises concatenating the outputs of the one or more view neural networks.

8. The method of any one of claims 1-5, wherein, processing the spatial embedding using the second neural network comprises processing the spatial embedding with a one-dimensional convolutional neural network layer.

9. A system for processing point cloud inputs, comprising one or more computers and one or more storage devices storing instructions that are operable to cause the one or more computers to perform operations comprising: obtaining a time series comprising a respective point cloud input corresponding to each of a plurality of time points, each point cloud input comprising point cloud data generated from sensor data captured by one or more sensors of a vehicle at the respective time point; processing each point cloud input using a first neural network to generate a respective spatial embedding characterizing the point cloud input, the processing each point cloud input comprising, for each point cloud input: partitioning the point cloud data into a plurality of voxels, generating a feature representation comprising a feature for each voxel, and processing the feature representation using the first neural network to generate the spatial embedding; and processing the spatial embeddings of the point cloud inputs using a second neural network to generate a spatio-temporal embedding characterizing the point cloud inputs in the time series. processing the spatial embedding of the point cloud input using a second neural network to generate a spatio-temporal embedding that characterizes the point cloud input in the time series.

10. The system of claim 9, wherein, The operations further include: processing the spatio-temporal embedding using a neural network of each specialized task of a plurality of specialized tasks of neural networks, wherein each specialized task of the neural networks is configured to process the spatio-temporal embedding to generate a prediction output for a different prediction task.

11. The system of claim 10, wherein, The first neural network and the second neural network have been jointly trained on a first prediction task, and wherein the first prediction task is not one of the different prediction tasks corresponding to the plurality of specialized tasks of neural networks.

12. The system of claim 9, wherein, Generating the feature representation includes: processing the point cloud data using one or more view neural networks, wherein a view neural network extracts features from the point cloud data with respect to a viewpoint; and combining outputs of the one or more view neural networks to generate the feature representation.

13. The system of any one of claims 9-12, wherein, Processing the spatial embedding using the second neural network includes processing the spatial embedding with a one-dimensional convolutional neural network layer.

14. One or more non-transitory computer storage media encoded with computer program instructions that, when executed by a plurality of computers, cause operations performed by the plurality of computers, the operations comprising: obtaining a time series comprising a respective point cloud input corresponding to each of a plurality of time points, each point cloud input comprising point cloud data generated from sensor data captured by one or more sensors of a vehicle at the respective time point; processing each point cloud input using a first neural network to generate a respective spatial embedding that characterizes the point cloud input, the processing each point cloud input including, for each point cloud input: partitioning the point cloud data into a plurality of voxels, generating a feature representation comprising features of each voxel, and processing the feature representation using the first neural network to generate the spatial embedding; and processing the spatial embedding of the point cloud input using a second neural network to generate a spatio-temporal embedding that characterizes the point cloud input in the time series.

15. The non-transitory computer storage medium of claim 14, wherein, The operations further include: processing the spatio-temporal embedding using a neural network of each specialized task of a plurality of specialized tasks of neural networks, wherein each specialized task of the neural networks is configured to process the spatio-temporal embedding to generate a prediction output for a different prediction task.

16. The non-transitory computer storage medium of claim 15, wherein, The first neural network and the second neural network have been jointly trained on a first prediction task, and wherein the first prediction task is not one of the different prediction tasks corresponding to the plurality of specialized tasks of neural networks.

17. The non-transitory computer storage medium of any of claims 14-16, wherein, Processing the spatial embedding using the second neural network includes processing the spatial embedding with a one-dimensional convolutional neural network layer.

18. The non-transitory computer storage medium of claim 14, wherein, Generating the feature representation includes: processing the point cloud data using one or more view neural networks, wherein a view neural network extracts features from the point cloud data with respect to a viewpoint; and combining outputs of the one or more view neural networks to generate the feature representation.

19. The non-transitory computer storage medium of claim 18, wherein, The one or more view neural networks include a bird’s eye view neural network that extracts features with respect to a bird’s eye view and a perspective view neural network that extracts features with respect to a perspective view.

20. The non-transitory computer storage medium of claim 18 or 19, wherein, Processing the point cloud data using the one or more view neural networks includes processing each point in the point cloud data with a fully connected layer shared by the one or more view neural networks to embed the point into a high-dimensional feature space. Processing the point cloud data using the one or more view neural networks includes processing each point in the point cloud data with a fully connected layer shared by the one or more view neural networks to embed the point into a high-dimensional feature space.

Images