3D position prediction from images

Abstract

Methods, systems and apparatus for predicting the position of a three-dimensional object from an image, including a computer program encoded on a computer storage medium. One of the methods includes obtaining an image sequence, the image sequence including a corresponding image captured by a camera at each of a plurality of time steps at the time step; for each image in the sequence, generating a corresponding pseudo-lidar feature of a corresponding pseudo-lidar representation of an area in the image that has been determined to depict a first object; for a particular image at a particular time step in the sequence, generating an image patch feature of an area in the particular image that has been determined to depict the first object; and generating a prediction from the corresponding pseudo-lidar feature and the image patch feature, the prediction characterizing the position of the first object in a three-dimensional coordinate system at the particular time step in the sequence.

Description

Technical Field

[0001] This description relates to predicting the position of objects in an environment. Background Art

[0002] The environment may be a real-world environment, and the objects may be, for example, vehicles or other objects in the environment. For example, the prediction may be made by an autonomous vehicle. Autonomous vehicles include self-driving cars, boats, and aircraft. Autonomous vehicles use various onboard sensors and computer systems to detect nearby objects and use such detection to make control and navigation decisions. Summary of the invention BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1A is an illustration of generating location predictions for example objects in an image.

[0004] Figure 1B is a schematic diagram of an example system.

[0005] Figure 2 is a flow chart of an example process for generating location prediction output.

[0006] Figure 3 An example of generating position predictions for objects depicted in an image sequence is shown.

[0007] Figure 4 is a flow diagram of another example process for generating position prediction output.

[0008] The same reference numbers and names in different drawings represent the same elements. DETAILED DESCRIPTION

[0009] This specification describes a system implemented as a computer program on one or more computers in one or more locations that generates three-dimensional position predictions for objects detected in one or more images.

[0010] The position prediction is called a "three-dimensional" position prediction because it incorporates the depth of the object, i.e., the distance of the object from the sensor or from another point that captured the image or images. This is in contrast to a "two-dimensional" position prediction that only identifies the area in the image that depicts the object.

[0011] Determining the three-dimensional position of objects in the environment is an important problem for many tasks. For example, an autonomous vehicle uses one or more sensors to sense objects in the vicinity of the autonomous vehicle. Determining the three-dimensional position of the sensed objects can help the autonomous vehicle safely make fully autonomous driving decisions or provide semi-autonomous driving assistance to human drivers.

[0012] However, while some sensors, such as lidar sensors, can accurately measure the distance from the sensor to an object sensed by the sensor, some autonomous vehicles may not be equipped with such sensors. Even when a vehicle is equipped with such a sensor, the distance measurement may not be available for all objects in a given scene, for example, if an object is outside the range of the sensor, if the sensor malfunctions, if an object is obscured from the sensor, or if the sensor cannot sense an object for another reason.

[0013] Therefore, when the autonomous vehicle does not have sensors capable of accurately measuring distances, or when distance measurements from such sensors are unavailable, determining the three-dimensional position of an object directly from camera images (i.e., without relying on measurements from other sensors) may be critical to the operation of the autonomous vehicle.

[0014] However, a camera image is a two-dimensional representation of the scene captured in the image. That is, a camera image only provides the intensity value of each pixel in a two-dimensional grid and does not directly include any three-dimensional information. Therefore, determining the three-dimensional position of an object directly from a camera image is a challenging problem.

[0015] Some systems attempt to predict the three-dimensional position of an object depicted in a given image by generating a dense depth map for the given image. A dense depth map assigns a predicted depth value to each pixel of the given image, such that the depth value for a given pixel represents the predicted three-dimensional distance, i.e., depth, from the camera that captured the given image to the portion of the scene depicted at that given pixel. These systems then estimate the distance of an object using the depth map directly, for example by determining that the three-dimensional distance to the object is the predicted depth value for pixels in the region of the image corresponding to the object. However, because accurately estimating the depth of each pixel in an image is difficult, these dense depth maps are noisy and the resulting object depths may be inaccurate.

[0016] This specification describes techniques for accurately estimating per-object depth of objects detected in a camera image using both an initial dense depth estimate and image features extracted from the camera image. In other words, the system described in this specification generates three-dimensional position predictions using a combination of "pseudo-lidar" features, which are calculated using the initial depth estimate, and image patch features, which are calculated directly from the intensity values ​​of pixels in the camera image. The pseudo-lidar features are called "pseudo-lidar" because they are generated from a pseudo-lidar representation that represents some or all pixels in a camera image as three-dimensional points using an initial depth estimate for the image, similar to a point cloud that would be generated from measurements by a lidar sensor.

[0017] By using the combined features to generate predictions, the disclosed system can accurately estimate a three-dimensional position using only one or more camera images. More specifically, the disclosed system can use image patch features to enhance an initial depth estimate, thereby producing a more accurate position prediction than a position prediction generated directly from the initial depth estimate.

[0018] In some cases, the disclosed system generates pseudo lidar features using a single image. In some other cases, when generating pseudo lidar features, the disclosed system combines information from multiple camera images taken at different times. By combining multiple camera images, the system can further improve the accuracy of the three-dimensional position prediction. For example, using multiple camera images allows the disclosed system to take into account the fact that a single 2D view of a scene, i.e., the view depicted in a single image, can be explained by many plausible 3D scenes.

[0019] Figure 1A is an illustration of generating position predictions for an example object 103 in an image 104. Figure 1A As can be seen in , image 104 is a perspective image, and the depth of object 103 , ie, the distance of object 103 from the camera that captured image 103 , cannot be directly obtained from image 104 .

[0020] For ease of illustration, various position predictions of the example object 103 are shown as corresponding bounding boxes in a bird's eye view (BEV) coordinate system. That is, although the above position predictions specify the three-dimensional position of the object, the position predictions are displayed in a two-dimensional BEV coordinate system that displays the depth of the object, but does not display the elevation of the object.

[0021] More specifically, Figure 1A The example of 104 shows a conventional position prediction 105 relative to a ground truth bounding box 106 representing the actual three-dimensional position of the object 103 when the image 104 was captured. For example, the conventional position prediction 105 can be generated using conventional monocular 3D detection techniques that attempt to generate the prediction directly from the image 104. For example, conventional techniques can generate a depth map that assigns a predicted depth to each pixel of the image, and then use the predicted depth to map the two-dimensional bounding box to the three-dimensional bounding box.

[0022] from Figure 1AAs can be seen from the example of , there is a significant error between the traditional position prediction 105 and the ground truth bounding box 106. More specifically, the error can be primarily attributed to depth error - while the traditional technique predicts the size, shape, and orientation of the object 103 relatively accurately, the traditional technique does not accurately predict the depth of the object 103. In particular, because the image 104 is a two-dimensional representation of the environment, and the depth of the object is a three-dimensional property, it may be difficult for the traditional technique to generate a representation of the image 104 from which the depth can be accurately estimated. For example, directly estimating the depth of each pixel in the image from a two-dimensional image may be prone to error.

[0023] Figure 1A The example also shows a position prediction 107 of the same object 103 generated using the techniques described in this specification, for example, the position prediction 107 will be referred to below. Figure 1B The onboard system 110 described in more detail is generated from Figure 1A As can be seen from the example of , there is no noticeable error between position prediction 107 and ground truth bounding box 106. In particular, unlike conventional position prediction 105, position prediction 107 generated by system 110 accurately predicts the depth of the object. This is because, as described in more detail below, system 110 uses both image features and pseudo lidar features generated using the initial depth prediction to enhance the initial depth prediction of image 104. By combining these two types of features, system 110 generates a significantly improved object depth prediction over the initial depth prediction. Therefore, position prediction 107 is significantly more accurate than conventional position prediction 105.

[0024] Figure 1B is a schematic diagram of an example system 100. System 100 includes an onboard system 110 and a training system 120.

[0025] The onboard system 110 is located on the carrier 102 . Figure 1BThe vehicle 102 in FIG. 1 is shown as a car, but the onboard system 102 can be located onboard any suitable vehicle type. In some cases, the vehicle 102 is an autonomous vehicle. An autonomous vehicle can be a fully autonomous vehicle that determines and executes fully autonomous driving decisions in order to navigate in an environment. An autonomous vehicle can also be a semi-autonomous vehicle that uses predictions to assist a human driver. For example, if a prediction indicates that a human driver is about to collide with another vehicle, the vehicle 102 can autonomously apply the brakes. As another example, the vehicle 102 can have an advanced driver assistance system (ADAS) that assists the human driver of the vehicle 102 in driving the vehicle 102 by detecting potential unsafe situations and warning the human driver or otherwise responding to unsafe situations. As a specific example, when an obstacle is detected, when the vehicle leaves the driving lane, or when an object is detected in the blind spot of the human driver, the vehicle 120 can warn the driver of the vehicle 102 or take autonomous driving actions.

[0026] The onboard system 110 includes one or more sensor subsystems 130. The sensor subsystem 130 includes a camera sensor that generates a camera image by detecting reflections of visible light, and may include other sensors, such as a lidar system that detects reflections of laser light, a radar system that detects reflections of radio waves, etc. As the vehicle 102 navigates the environment, the various sensors capture measurements of the environment. For example, a camera sensor may repeatedly capture images during navigation.

[0027] The sensor subsystem 130 or other components of the vehicle 102 may also classify portions of the sensor measurements from one or more sensors as metrics of objects in the environment surrounding the vehicle 102 .

[0028] For example, subsystem 130 or other components may perform object detection on images captured by a camera sensor to identify areas in the images that depict objects. Subsystem 130 or other components may use any of a variety of two-dimensional object detection techniques.

[0029] As a specific example, subsystem 130 may use an object detection neural network to process images captured by a camera sensor to generate an object detection output for each image that includes a corresponding set of bounding boxes, where each bounding box in a given image depicts a corresponding object, i.e., enclosing a portion of the given image that subsystem 130 has determined depicts the object.

[0030] In some embodiments, subsystem 130 may then apply an object tracker to bounding boxes in a temporal sequence of images (i.e., a sequence ordered according to the time at which each image was captured) to generate object tracklet data for one or more objects. The object tracklet data for a given object identifies a corresponding bounding box in each of the image sequence that depicts the given object. That is, the object tracklet data for a given object identifies which bounding box in each image depicts the given object. For example, subsystem 130 may apply a Kalman filter-based object tracker to the object detection output for images in the sequence to generate object tracklet data. As another example, subsystem 130 may apply an object tracking neural network to the object detection output for images in the sequence to generate object tracklet data.

[0031] Once the sensor subsystem 130 generates the object detection output, the sensor subsystem 130 may send the image sequence, object detection output, and, when used, object trajectory segment data to a position prediction system 150 that is also onboard the vehicle 102 .

[0032] The position prediction system 150 processes the images, object detection outputs, and object trajectory segment data when used, to generate a corresponding position prediction output 152 for each of one or more objects that have been detected in one or more images.

[0033] When the position prediction system 150 operates on a time series of multiple images, the corresponding position prediction output 152 of a given object is a prediction of the position of the given object in a three-dimensional coordinate system at a specific time step (e.g., the last time step) in the time series. In some embodiments, the three-dimensional coordinate system is a coordinate system centered on a specific position of the autonomous vehicle 102 (e.g., the position of a camera sensor that captured the time series or a different fixed position on the autonomous vehicle). For example, the prediction can be a prediction of the depth of the object relative to the camera at the last time step in the time series of images. Specifically, the depth prediction is a predicted depth value that represents the distance from a specified point on the object (e.g., the center of the object) to the camera at the time step. As another example, the prediction can be a prediction of a three-dimensional area in a three-dimensional coordinate system that corresponds to the predicted position of the object relative to the camera at the last time step.

[0034] The following are some examples of reference Figure 2 and 3 The generation of position prediction output 152 as system 150 operates on a sequence of images is described in greater detail.

[0035] When the position prediction system 150 operates on a single image captured at a particular time step, the corresponding position prediction output 152 for a given object is a prediction that characterizes the position of the given object in a three-dimensional coordinate system at the particular time step. For example, the prediction can be a prediction of the depth of the given object relative to the camera at the particular time step. Specifically, the depth prediction is a predicted depth value that represents the distance of a specified point on the object (e.g., the center of the object) from the camera at the particular time step. As another example, the prediction can be a prediction of a three-dimensional area in a three-dimensional coordinate system that corresponds to the predicted position of the given object relative to the camera at the last time step.

[0036] The following are some examples of reference Figure 4 Generating position prediction output 152 when system 150 operates on a single image is described in greater detail.

[0037] The onboard system 110 also includes a planning system 160. The planning system 160 can make autonomous or semi-autonomous driving decisions for the vehicle 102, for example, by generating a planned vehicle path that represents a path that the vehicle 102 will take in the future.

[0038] The onboard system 100 may provide the position prediction output 152 generated by the position prediction system 150 to one or more other onboard systems of the vehicle 102 , such as the planning system 160 and / or the user interface system 165 .

[0039] When the planning system 160 receives the position prediction output 152, the planning system 160 can use the position prediction output 152 to generate a planning decision for planning the future trajectory of the vehicle, that is, to generate a new planned vehicle path. For example, the position prediction output 152 may include a prediction that the position of a given object in the environment intersects with the current planned path of the vehicle 102, thereby causing a collision. In this example, the planning system 160 can generate a new planned vehicle path that avoids a potential collision, for example by autonomously controlling the steering of the vehicle, and cause the vehicle 102 to follow the new planned path and avoid the potential collision.

[0040] When the user interface system 165 receives the position prediction output 152, the user interface system 165 may use the position prediction output 152 to present information to the driver of the vehicle 102 to assist the driver in safely operating the vehicle 102. The user interface system 165 may present information to the driver of the agent 102 in any suitable manner, such as, for example, by an audio message transmitted via a speaker system of the vehicle 102, or by an alert displayed on a visual display system in the agent (e.g., a liquid crystal display on the dashboard of the vehicle 102). In a particular example, the position prediction output 152 may include a prediction that a particular object is within a threshold distance of the vehicle 102 and may result in a collision. In this example, the user interface system 165 may present a warning message to the driver of the vehicle 102 with instructions to adjust the trajectory of the vehicle 102 to avoid the collision, or notify the driver of the vehicle 102 that a collision with a particular surrounding agent is possible.

[0041] To generate position prediction output 152 , position prediction system 150 may use trained parameter values ​​195 obtained from model parameter storage 190 in training system 120 , ie, trained model parameter values ​​for a set of neural networks used by position prediction system 150 .

[0042] Training system 120 is typically hosted within a data center 124, which may be a distributed computing system having hundreds or thousands of computers in one or more locations.

[0043] The training system 120 includes a training data storage device 170 that stores training data for training the position prediction system (i.e., determining the training parameter values ​​195 of the machine learning model used by the position prediction system 150). The training data storage device 170 receives raw training examples from vehicles operating in the real world. For example, the training data storage device 170 can receive raw training examples 155 from the vehicle 102 and one or more other agents communicating with the training system 120. The raw training examples 155 can be processed by the training system 120 to generate new training examples. The raw training examples 155 can include object detection data, such as object detection data 142, which can be used as input for new training examples. The raw training examples 155 can also include ground truth data representing the position of objects in the environment around the vehicle 102 at one or more future time points. This data can be used to generate a ground truth position output of one or more objects near the vehicle at a time point represented by the object detection data. Each ground truth position output represents the actual three-dimensional position of the corresponding object. For example, the ground truth position output can identify the depth of the corresponding object relative to the camera. As another example, the ground truth location output may identify a three-dimensional area occupied by the corresponding object.

[0044] Training data storage 170 provides training examples 175 to training engine 180, which is also hosted in training system 120. Training engine 180 uses training examples 175 to update model parameters to be used by position prediction system 150, and provides updated model parameters 185 to model parameter storage 190. Once the parameter values ​​for position prediction system 150 have been fully trained, training system 120 may send trained parameter values ​​195 to position prediction system 150, for example, via a wired or wireless connection.

[0045] Training location prediction system 150 is described in more detail below.

[0046] Although this specification describes generating position predictions on an autonomous vehicle, more generally, the described techniques may be implemented on any system of one or more computers that receive images of a scene in an environment.

[0047] As an example, position prediction can be performed on different types of agents that have camera sensors and interact with objects as they navigate an environment. For example, position prediction can be performed by one or more computers embedded in a robot or other agent.

[0048] As another example, the position prediction can be made by one or more computers remote from the agent and receiving images captured by the camera sensor of the agent. In some of these examples, the one or more computers can use the position prediction to generate control decisions for controlling the agent, and then provide the control decisions to the agent for execution by the agent.

[0049] As another example, position predictions may be made in a computer simulation of a real-world environment navigated by a simulated autonomous vehicle and target agent. Generating these predictions in simulation may be helpful in controlling the simulated vehicle and testing the realism of certain situations encountered in the simulation. More generally, generating these predictions in simulation may be part of testing control software for a real-world autonomous vehicle before the software is deployed on the autonomous vehicle, or part of training one or more machine learning models that will later be deployed on the autonomous vehicle, or both.

[0050] Figure 2 is a flow chart of an example process 200 for generating a position prediction output of an object. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a position prediction system appropriately programmed according to the present specification, such as Figure 1B The location prediction system 150 may perform process 200 .

[0051] At any given point in time, the system may execute process 200 to generate a corresponding position prediction for each of one or more objects detected in a time series of images captured by a camera sensor of the vehicle.

[0052] The system obtains a time sequence of images including a plurality of images (step 202). The sequence is referred to as a "time" sequence because, in some embodiments, the images are arranged according to the time at which they were captured (i.e., timestamp). For example, the most recent image is the last image in the time sequence of images captured by the camera sensor, and the least recent image is the first image in the time sequence.

[0053] The system generates an initial depth estimate for each image in the time series (step 204). The initial depth estimate for each image in the time series assigns a corresponding estimated depth value to each pixel in the image. The corresponding estimated depth value for a given pixel represents the predicted distance of the scene depicted at that pixel from the camera that captured the image.

[0054] To generate an initial depth estimate for a given image, the system can process the given image using a depth estimation neural network. The depth estimation neural network can be, for example, a convolutional neural network that processes the image to generate a depth map that assigns a corresponding estimated depth value to each pixel of the image. In some embodiments, the system obtains a pre-trained depth estimation neural network. In some other embodiments, the system jointly trains the depth estimation neural network and other neural networks used to generate position predictions.

[0055] The system may then perform steps 206 - 214 for each of the one or more objects detected in the sequence of images.

[0056] The system obtains object trajectory segment data for the object (step 206). For each image in the time series, the object trajectory segment data for the object identifies regions in the image that have been determined to depict a given object. For example, each region may be a two-dimensional bounding box. Figure 1B As described above, trajectory segment data may be generated by applying a two-dimensional object detector.

[0057] The system generates, for each image in the time series, a corresponding pseudo-lidar representation of the region in the image based on the initial depth estimate for the image (step 208 ).

[0058] The pseudo lidar representation of a given area in a given image represents each pixel in the area in the given image as a point in a three-dimensional coordinate system. That is, the system transforms the two-dimensional pixels in the given area into a three-dimensional coordinate system using an initial depth estimate.

[0059] Specifically, to generate a pseudo-lidar representation of a given area, the system maps each pixel in the two-dimensional area to a three-dimensional coordinate system based on the estimated depth value of the pixel in the initial depth estimate of the image and the properties of the camera.

[0060] As a specific example, the system can perform the mapping using the horizontal and vertical focal lengths of the camera and the pixel position corresponding to the center of the camera. In some embodiments, the pixel value at a position (u, v) in a given image is mapped to a three-dimensional point (x, y, z) in a three-dimensional coordinate system as follows:

[0061] z=d(u,v),

[0062] x=(uC x )xz / f x ,as well as

[0063] y=(v–C y )xz / f y ,

[0064] where d(u,v) is the initial depth estimate for the pixel at position (u,v), and f x is the horizontal focal length, f y is the vertical focal length, and (C x ,C y ) is the pixel position corresponding to the camera center.

[0065] Optionally, when generating the pseudo-lidar representation, the system can also compensate for the ego-motion of the camera when capturing the temporal sequence of images. For example, the system can generate an initial pseudo-lidar representation as described above, and then transform each initial pseudo-lidar representation into the coordinate system of the camera as of the last image in the sequence using the ego-motion of the camera between images in the sequence.

[0066] As a specific example, the system can first project each initial pseudo-lidar representation into a global coordinate system using the camera's ego-motion, and then project the representation from the global coordinate system into the coordinate system of the camera as of the last image.

[0067] The system generates corresponding features of the pseudo-lidar representation for each pseudo-lidar representation (step 210). For example, the system can generate features by processing the pseudo-lidar representation using a pseudo-lidar feature extraction neural network that generates image pseudo-lidar features as output. The pseudo-lidar feature extraction neural network can be any suitable neural network configured to process a collection of three-dimensional points, such as an encoder neural network of any suitable 3D object detection neural network. For example, the neural network can be a convolutional neural network that arranges the points into an image and processes the image to generate features. An example of such a neural network is PatchNet, which is described in more detail in Ma et al., Rethinking Pseudo-LiDAR Representation, arXiv:2008.04582.

[0068] The system generates image patch features for regions in an image at a particular time step in the time series (step 212). For example, the image at the particular time step may be the last image in the time series, i.e., the most recent image. However, more generally, the particular time step may be any time step in the time series.

[0069] For example, the system can use an image feature extraction neural network to process an image at a specific time step in a sequence to generate image features for the image. The system can then select a subset of the image features corresponding to the blocks of the image within the region as image block features. That is, the system can generate image block features by cropping the image feature portion corresponding to the region. The feature extraction neural network can be, for example, a convolutional encoder neural network that maps an image to a feature map. One example of a convolutional neural network that can be used is a ResNet backbone neural network that includes multiple residual convolution blocks. Another example of a convolutional neural network that can be used is an EfficientNet backbone neural network.

[0070] The system generates a prediction representing the position of the object in the three-dimensional coordinate system at a particular time step in the time series from the corresponding pseudo-lidar features and the image patch features (step 214).

[0071] As a specific example, the system can combine corresponding pseudo lidar features and image patch features to generate a combined feature, and process the combined feature using a decoder neural network to generate a prediction.

[0072] In some embodiments, the system generates a combined feature by concatenating the corresponding pseudo-lidar features and the image patch features. Thus, in these embodiments, the input to the decoder neural network is the concatenation of the corresponding pseudo-lidar features for each region and the image patch features of the image at a particular time step in the time series.

[0073] In some other embodiments, the system first generates a combined pseudo-lidar feature by combining the corresponding pseudo-lidar features of the images in the sequence. For example, the system can process the corresponding pseudo-lidar features through a feed-forward neural network, such as a multi-layer perceptron (MLP), or can concatenate or average the corresponding pseudo-lidar features. The system then combines, for example, concatenates the combined pseudo-lidar features and the image patch features. Therefore, in these embodiments, the input to the decoder neural network is the concatenation of the combined pseudo-lidar features and the image patch features of the images in the time series.

[0074] The decoder neural network can have any suitable architecture that allows the decoder neural network to map a set of features to predictions.

[0075] For example, the decoder neural network can be a multi-layer perceptron (MLP) neural network that processes the combined features to generate a prediction. When the prediction is a depth estimate of an object, the output of the decoder neural network can be a single regressed depth value representing the predicted depth of the object. When the prediction specifies a three-dimensional region corresponding to the predicted position of the object at a particular time step, the output of the decoder neural network can be a regressed value for each of a set of coordinates (e.g., coordinates of a three-dimensional bounding box) defining the three-dimensional region in a three-dimensional coordinate system.

[0076] Figure 3 An example of generating a position prediction 152 for an object 302 depicted in an image sequence including three images 304, 306, and 308 is shown. The generation of the position prediction may be performed by a system of one or more computers located at one or more locations (e.g., Figure 1B The position prediction system 150 is used to perform.

[0077] To generate the position prediction, the system applies a two-dimensional object tracker to images 304, 306, and 308 to generate as output object track segment data for object 302. The object track segment data identifies bounding box 314 in image 304, bounding box 316 in image 306, and bounding box 318 in image 308.

[0078] The system then generates, from the last image in the sequence, image 318 , an image patch 328 that includes only the portion of image 318 that is within bounding box 318 .

[0079] The system also generates a corresponding pseudo-lidar representation 330 for each of the bounding boxes 314, 316, and 318, which represents the pixels within the bounding box as three-dimensional points for each bounding box. As described above, the system can generate pseudo-lidar representation 330 using a corresponding initial depth estimate for each of the images 304, 306, and 308.

[0080] The system then generates image patch features 338 for image patch 328 and pseudo lidar features 340 for each of pseudo lidar representations 330 .

[0081] The system then generates a position prediction 152 for object 302 from image patch features 338 of image patch 328 and pseudo lidar features 340 of each of pseudo lidar representations 330 .

[0082] exist Figure 3 In the example of , the system generates a combined pseudo lidar feature 350 from feature 340 and then concatenates the combined pseudo lidar feature 350 and the image patch feature 338 to generate a combined feature 360. The system then processes the combined feature 360 ​​to generate the position prediction 152.

[0083] Figure 4 is a flow chart of another example process 400 for generating a position prediction output of an object. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a position prediction system appropriately programmed according to the present specification, such as Figure 1B The location prediction system 150 may perform process 400 .

[0084] At any given point in time, the system may execute process 400 to generate a corresponding position prediction for each of one or more objects detected in an image captured by a camera sensor of the vehicle.

[0085] The system obtains an image captured by a camera sensor (step 402).

[0086] The system generates an initial depth estimate for the image (step 404). The initial depth estimate for the image assigns a corresponding estimated depth value to each pixel in the image. The corresponding estimated depth value for a given pixel represents the predicted distance of the scene depicted at that pixel from the camera that captured the image.

[0087] To generate an initial depth estimate for an image, the system can process the image using a depth estimation neural network. The depth estimation neural network can be, for example, a convolutional neural network that processes the image to generate a depth map that assigns a corresponding estimated depth value to each pixel of the image. In some embodiments, the system obtains a pre-trained depth estimation neural network. In some other embodiments, the system jointly trains the depth estimation neural network with other neural networks used to generate position predictions.

[0088] The system may then perform steps 406 - 412 for each of the one or more objects detected in the sequence of images.

[0089] The system obtains object detection data for the object (step 406). The object detection data for the object identifies a two-dimensional bounding box in the image that has been determined to depict a given object. The object detection data may be generated by applying a two-dimensional object detector, as described above with reference to Figure 1B described.

[0090] The system generates a pseudo-lidar representation of a two-dimensional bounding box in the image based on the initial depth estimate of the image (step 408).

[0091] The pseudo lidar representation of a given bounding box in a given image represents each pixel within the bounding box in the given image as a point in a 3D coordinate system. That is, the system transforms the 2D pixels in the given bounding box into a 3D coordinate system using the initial depth estimate.

[0092] Specifically, to generate a pseudo-lidar representation of a given bounding box, the system maps each pixel within the 2D bounding box to a 3D coordinate system based on the estimated depth value of the pixel in the initial depth estimate of the image and the properties of the camera.

[0093] As a specific example, the system can perform the mapping using the horizontal and vertical focal lengths of the camera and the pixel position corresponding to the center of the camera. In some embodiments, the pixel value at a position (u, v) in a given image is mapped to a three-dimensional point (x, y, z) in a three-dimensional coordinate system as follows:

[0094] z=d(u,v),

[0095] x=(uC x )xz / f x ,as well as

[0096] y=(v–C y )xz / f y ,

[0097] where d(u,v) is the initial depth estimate for the pixel at position (u,v), and f x is the horizontal focal length, f yis the vertical focal length, and (C x ,C y ) is the pixel position corresponding to the camera center.

[0098] The system generates features for the pseudo-lidar representation (step 410). For example, the system can generate features by processing the pseudo-lidar representation using a pseudo-lidar feature extraction neural network that generates pseudo-lidar features as output. The pseudo-lidar feature extraction neural network can be any suitable neural network configured to process a collection of three-dimensional points, such as an encoder neural network of any suitable 3D object detection neural network. For example, the neural network can be a convolutional neural network that arranges the points into an image and processes the image to generate features. An example of such a neural network is PatchNet, which is described in more detail in Ma et al., Rethinking Pseudo-LiDAR Representation, arXiv:2008.04582.

[0099] The system generates image patch features for two-dimensional bounding boxes in the image (step 412).

[0100] For example, the system may process the last image using an image feature extraction neural network to generate image features of the image. The system may then select a subset of the image features corresponding to a block of the image within a two-dimensional bounding box as the image block features. That is, the system may generate the image block features by cropping the portion of the image features corresponding to the bounding box.

[0101] The system generates a prediction representing the position of the object in the three-dimensional coordinate system from the corresponding pseudo-lidar features and the image patch features (step 414).

[0102] As a specific example, the system can combine the pseudo lidar features and the image patch features to generate a combined feature, and process the combined feature using a decoder neural network to generate a prediction. In some embodiments, the system generates the combined feature by concatenating the pseudo lidar features and the image patch features.

[0103] The decoder neural network can have any suitable architecture that allows the decoder neural network to map a set of features to predictions.

[0104] For example, the decoder neural network can be a multi-layer perceptron (MLP) neural network that processes the combined features to generate a prediction. When the prediction is a depth estimate of an object, the output of the decoder neural network can be a single regressed depth value representing the predicted depth of the object. When the prediction specifies a three-dimensional region corresponding to the predicted position of the object, the output of the decoder neural network can be a regressed value for each of the coordinates of the three-dimensional region in a three-dimensional coordinate system (e.g., the coordinates of a three-dimensional bounding box).

[0105] As described above, prior to using the neural network described above, such as the neural network used in performing process 200 or process 400, to perform inference (e.g., on an autonomous vehicle), a training system trains the neural network on training data. The training data includes a plurality of training examples, where each training example includes (i) one or more images of an environment and (ii) ground truth data for at least one of the images.

[0106] In particular, the ground truth data includes ground truth three-dimensional positions of one or more of the objects depicted in at least one of the images, such as ground truth per-object depths or ground truth three-dimensional bounding boxes.

[0107] The system trains the above neural networks together, i.e., jointly, to minimize an objective function that measures, for each of one or more objects, the error between a position prediction generated by the system for the object and the ground truth position of the object. For example, when the position prediction is a depth estimate, the error can be, for example, the squared error between the depth estimate and the ground truth depth. When the position prediction is a three-dimensional bounding box, the error can be the error between the predicted three-dimensional bounding box and the ground truth three-dimensional bounding box, such as the sum of the squared errors of each coordinate in the coordinates of the bounding box.

[0108] The system can train the neural network on the objective function using any suitable machine learning training technique, such as a gradient descent technique with any suitable optimizer, such as stochastic gradient descent, Adam, rmsProp, or Adafactor.

[0109] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0110] The term "data processing apparatus" refers to data processing hardware and includes various devices, apparatuses, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or further include an off-the-shelf or custom parallel processing subsystem, such as a GPU or another special processing subsystem. The apparatus may also be or further include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0111] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document; in a single file dedicated to the program in question; or in multiple coordinated files, such as files storing one or more modules, subroutines or code portions. A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0112] For a system consisting of one or more computers, being configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, when operated, causes the system to perform the operation or action. For one or more computer programs, being configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0113] As used in this specification, an "engine" or "software engine" refers to a software-implemented input / output system that provides outputs that are different from inputs. An engine can be a coded functional block, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine can be implemented on any suitable type of computing device, such as a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smart phone, or other fixed or portable devices including one or more processors and a computer-readable medium. In addition, two or more of the engines can be implemented on the same computing device, or on different computing devices.

[0114] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. These processes and logic flows can also be performed by a dedicated logic circuit, such as an FPGA or ASIC, or by a combination of a dedicated logic circuit and one or more programmed computers.

[0115] A computer suitable for executing a computer program may be based on a general or special purpose microprocessor or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by a dedicated logic circuit, or included in a dedicated logic circuit. Typically, a computer will also include or be operably coupled to one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, to receive data from it or to transmit data to it, or both. However, a computer does not need to have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0116] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0117] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse, trackball, or presence sensitive display or other surface through which the user can provide input to the computer). Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser in response to a request received from a web browser on the user's device. In addition, the computer may interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smart phone), running a messaging application, and receiving a response message from the user.

[0118] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0119] A computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device, for example, to display data to a user who interacts with the device acting as a client and to receive user input from the user. Data generated at the user device, such as the result of a user interaction, can be received from the device at the server.

[0120] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope of the claimed protection, but rather as descriptions of features that may be peculiar to a particular embodiment of a particular invention. Certain features described in this specification in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations, and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may point to a variant of a sub-combination or a sub-combination.

[0121] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order or sequence shown, or that all of the operations shown be performed, in order to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0122] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes described in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: obtaining a time sequence of images, the time sequence comprising, at each of a plurality of time steps, a corresponding image captured by a camera at the time step; for each image in the time series, generating a corresponding pseudo lidar feature of a corresponding pseudo lidar representation of an area in the image that has been determined to depict the first object; For a specific image at a specific time step in the time series, generating an image patch feature of a region in the specific image that has been determined to depict a first object, wherein the image patch feature is generated from intensity values ​​of pixels in the image; and Generate a prediction representing a position of a first object in a three-dimensional coordinate system at a specific time step in a time series from the corresponding pseudo-lidar features and the image patch features, wherein the generating includes combining the corresponding pseudo-lidar features and the image patch features to generate a combined feature, and processing the combined feature using a decoder neural network to generate the prediction.

2. The method of claim 1 , wherein the prediction comprises an updated depth estimate that estimates the depth of a specified point on the first object at a particular time step in the time series, wherein the updated depth estimate is a predicted distance from the specified point on the first object to the camera at the particular time step.

3. The method according to claim 1, wherein: The prediction specifies a three-dimensional region corresponding to a predicted position of the first object relative to the camera at the particular time step.

4. The method according to claim 1, wherein combining the corresponding pseudo lidar features and image block features comprises concatenating the corresponding pseudo lidar features and image block features.

5. The method according to claim 1, wherein generating image patch features of a region in an image at a specific time step in the time series comprises: Processing the image using an image feature extraction neural network to generate image features of the image; and A subset of the image features corresponding to blocks of the last image within the region is selected as image block features.

6. The method according to claim 1, further comprising: For each image in the time series, generating an initial depth estimate that assigns a corresponding estimated depth value to each pixel in the image; and Using the initial depth estimate for the image, generate a corresponding pseudo-lidar representation for each image in the time series.

7. The method according to claim 6, wherein: For each image in the time series, generating an initial depth estimate that assigns a corresponding estimated depth value to each pixel in the image includes: Process the image using a depth estimation neural network to generate an initial depth estimate for the image.

8. The method of claim 7, wherein generating the pseudo lidar representation comprises: Each pixel within a region of the image that has been determined to depict the first object is mapped to a three-dimensional coordinate system based on estimated depth values ​​of the pixels in the initial depth estimation of the image and properties of the camera.

9. The method of claim 8, wherein the properties of the camera include horizontal and vertical focal lengths of the camera.

10. The method of claim 1 , wherein generating a corresponding pseudo-lidar feature for each pseudo-lidar representation comprises: The pseudo-lidar representation is processed using a pseudo-lidar feature extraction neural network to generate pseudo-lidar features of the pseudo-lidar representation.

11. A method performed by one or more computers, the method comprising: obtaining a time sequence of images, the time sequence comprising, at each of a plurality of time steps, a corresponding image captured by a camera at the time step; For each image in the time series, generating an initial depth estimate that assigns a corresponding estimated depth value to each pixel in the image; obtaining object trajectory segment data for a first object, the object trajectory segment data identifying, for each image in the time series, a corresponding two-dimensional bounding box in the image that has been determined to depict the first object; generating, for each image in the time series, a corresponding pseudo-lidar representation of a two-dimensional bounding box in the image based on an initial depth estimate for the image; generating a corresponding pseudo-lidar feature for each of the pseudo-lidar representations; generating image patch features of a two-dimensional bounding box in a last image in the time series, wherein the image patch features are generated from intensity values ​​of pixels in the image; and Generate a prediction representing the position of the first object in the three-dimensional coordinate system at a last time step of the time series from the corresponding pseudo lidar features and the image patch features, wherein the generating includes combining the corresponding pseudo lidar features and the image patch features to generate a combined feature, and processing the combined feature using a decoder neural network to generate the prediction.

12. A system for three-dimensional position prediction, comprising one or more computers and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more computers, the one or more computers perform corresponding operations of any of the methods described in the preceding claims.

13. One or more computer-readable storage media storing instructions which, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method of any one of claims 1-11.

Images