Remote distance estimation using a reference object

Abstract

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, are provided for generating a distance estimate for a target object depicted in an image of a scene in an environment. The system obtains data specifying (i) a target portion of the image, the target portion of the image depicting a target object detected in the image, and (ii) one or more reference portions of the image, each reference portion depicting a respective reference object detected in the image. For each of the one or more reference objects, the system also obtains a respective distance measurement for the reference object, the distance measurement being a measurement of a distance from the reference object to a specified location in the environment. The system processes the obtained data to generate a distance estimate for the target object that is an estimate of a distance from the target object to the specified location in the environment.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 114,518, filed November 16, 2020, the disclosure of which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present invention relates to estimating distances to objects in an environment using neural networks, also known as localizing objects in an environment. BACKGROUND

[0004] Localizing objects in an environment is a task required for motion planning, for example, by an autonomous vehicle.

[0005] Autonomous vehicles include self-driving cars, boats, and aircraft. Autonomous vehicles use various on-board sensors and computer systems to detect nearby objects and use such detection to make control and navigation decisions.

[0006] Estimating respective distances of objects, such as other vehicles or pedestrians, from an autonomous vehicle is a safety-critical task for autonomous driving. While many autonomous vehicles are equipped with LiDAR sensors that can make distance measurements, some objects detected in images generated by a camera sensor of an autonomous vehicle can be outside the sensing range of the LiDAR sensor of the autonomous vehicle. SUMMARY

[0007] This specification describes a system implemented as computer programs on one or more computers in one or more locations that performs object localization, also known as localizing objects in an environment, by estimating distances to objects in an environment using neural networks.

[0008] For example, object localization can be performed by a vehicle’s on-board computer system of an autonomous vehicle navigating through an environment using images (and optionally, other data) generated by one or more sensors of the autonomous vehicle, e.g., a camera sensor, and optionally, a LiDAR sensor. The vehicle’s planning system can use object localization to make planning decisions to plan a future trajectory of the vehicle, e.g., by generating or revising a future trajectory to avoid a collision with any of the objects.

[0009] In particular, the system determines a distance estimate for a target object in an image depicted with one or more reference objects. For example, the system can apply an object detector to the image to determine that the image depicts the target object as well as the one or more reference objects.

[0010] The system obtains data specifying (i) a target portion of the image that depicts a target object detected in the image, (ii) one or more reference portions of the image, each reference portion depicting a respective reference object detected in the image.

[0011] For each of the one or more reference objects, the system obtains a respective distance measurement for the reference object, which is a measurement of a distance from the reference object to a specified point in the environment. These measurements can be obtained using a LiDAR sensor of the autonomous vehicle or from a map of the environment maintained by the autonomous vehicle, for example. Thus, the distance from the reference object to the specified point is known, while the distance from the target object to the specified point is not known.

[0012] The system generates a network input from the portion of the image that depicts the target object, the respective portion of the image that depicts each of the one or more reference objects, and the respective distance measurement for the reference object.

[0013] The system then processes the network input using a neural network to generate a distance estimate for the target object, which is an estimate of a distance from the target object to the specified point in the environment. Thus, the system uses the known distances to the reference objects and the relationships between the reference objects and the target object in order to accurately estimate the distance to the target object.

[0014] The details of one or more implementations of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 An example target distance estimation system is shown.

[0016] Figure 2 An example in-vehicle system is shown.

[0017] Figure 3A is a flow diagram showing an example process for target distance estimation.

[0018] Figure 3B is a flow diagram showing an example process for generating target-reference pair embeddings.

[0019] Figure 3C is a flow diagram showing an example process for generating a target distance estimate from a target reference pair embedding.

[0020] Like reference numbers and designations in different drawings indicate like elements. DETAILED DESCRIPTION

[0021] The specification describes a system implemented as computer programs on one or more computers in one or more locations that performs object localization, also known as localizing objects in an environment, by estimating distances to objects in the environment using neural networks.

[0022] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0023] Estimating distances to objects is a safety-critical task for applications such as robot control and autonomous driving. For example, for scenarios such as highway driving, heavy truck driving, and wet road driving, it is necessary to estimate distances to remote objects, such as objects that are outside the typical LiDAR range. For example, on a rural highway with a standard speed limit, a passenger vehicle can need a long distance to come to a complete stop in an emergency. In some cases, this large distance can greatly exceed the typical LiDAR sensing range. Therefore, it is important to sense the distance to objects outside the LiDAR sensing range even for passenger vehicles driving on good road conditions. The stopping distance required for heavy trucks or in bad road conditions, such as snow, ice, or rain, can increase significantly. Given that it can be unsafe to brake suddenly on a highway, it is still crucial to estimate distances to objects beyond the minimum required stopping distance in order to provide sufficient time for gradual deceleration or lane changing. Therefore, to leave sufficient time for appropriate reactions and ensure safety, autonomous driving systems need to estimate distances to remote objects.

[0024] Most existing LiDAR techniques do not meet the remote sensing requirements. On the other hand, camera sensing captures objects at greater distances and captures rich semantic information such as object appearance, geometry, and contextual cues. However, camera images do not provide depth information by themselves. Conventional methods have been developed to estimate object distances based on camera images. However, these methods tend to produce inaccurate results about remote objects due to challenges such as errors in size estimation and lack of context or other relevant signals in the scene.

[0025] The specification provides techniques for remote distance estimation that leverage reference objects with known distance measurements. Given a reference with known distance, the provided system uses a machine learning model to localize remote target objects. Certain other objects in the scene whose distances are readily available, such as objects within the sensing range of a LiDAR sensor or objects that can be localized using an available map, can obtain reference distances.

[0026] In some implementations, the provided system efficiently encodes and leverages relationships between target and reference objects by generating pair embeddings between the target object and the reference objects. This is in contrast to traditional approaches that emphasize object size and appearance for distance estimation, which give less informative cues for remote objects.

[0027] In some implementations, the provided system also processes the target-reference pair embeddings using an attention module that fuses information from different reference objects, and generates the target distance estimate using a neural network layer that processes the combined target-reference pair embeddings. This approach takes into account both global and local features of the target-reference relationships, as well as the relevance of different reference objects for distance estimation, further improving the performance of the system.

[0028] Overall, the provided system employs techniques that significantly improve the accuracy of distance estimation for remote target objects relative to traditional approaches, without requiring additional special hardware beyond typical existing sensors (e.g., sensors configured on an autonomous vehicle), such as camera and LiDAR sensors.

[0029] Other advantages of the subject matter will become apparent from the following description, the drawings, and the claims.

[0030] Figure 1 An example of a target distance estimation system 100 is shown. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0031] In general, the system 100 estimates the distance of a target object depicted in an image 110 of a scene in an environment. In one example, the image 110 can be an image obtained by an image sensor (such as by a monocular camera) deployed on an agent (such as an autonomous vehicle) navigating through the environment. The system 100 estimates the distance of the target object from the autonomous vehicle (e.g., the distance between the image sensor and the center of the target object). A planning system of the vehicle can use the estimated target distance to make planning decisions to plan a future trajectory of the vehicle, e.g., by generating or modifying a future trajectory to avoid collision with any object. Reference Figure 2 An example of an autonomous vehicle that deploys the target distance estimation system 100 is described. Other examples of agents include robots, drones, and the like.

[0032] As input, the system 100 obtains data specifying (i) a target portion 112 in the image 110, which depicts a target object detected in the image, and (ii) one or more reference portions 114 in the image 110, each of which depicts a respective reference object.

[0033] The target object can be a dynamic object in the environment, such as another car on the road, or a particular road feature, such as a traffic light. In some implementations, the target object can be outside the sensing range of a localization sensor (such as a LiDAR system) configured on the autonomous vehicle.

[0034] The reference objects can be other objects sharing the environment, such as other vehicles or pedestrians on the road with the autonomous vehicle, or map features, such as road signs, street signs, intersections, landmarks, etc. In some implementations, some or all of the reference objects can be within the sensing range of a localization sensor (such as a LiDAR system).

[0035] The system 100 or another system can detect the target object and the reference objects in the image 110 using any appropriate image-based object detection technique (e.g., by applying a conventional object detection neural network to the image 110), and select the target portion and the reference portion of the image, e.g., by obtaining respective bounding boxes of the detected target object and reference objects from the image.

[0036] For each of the one or more reference objects, the system also obtains a respective distance measurement 114d, which is a measure of the distance from the reference object to a specified location in the environment.

[0037] In some implementations, the specified location is the location of the image sensor (e.g., camera) that captured the image in the environment.

[0038] In some implementations, the respective distance measurements can be obtained from sensor readings of the scene in the environment (e.g., that have been generated by a LiDAR sensor also deployed on the agent).

[0039] In some implementations, some or all of the reference objects can be map features in the environment, and the respective distance measurements are obtained from map data of the scene in the environment. That is, one or more of the reference objects is an object identified in a map of the environment (i.e., whose location is indicated in the map of the environment), and the system can obtain the distance measurement by computing the distance between the specified location and the location indicated in the map for that object. For example, a reference object can be a road sign identified in a map of the environment, and the system can determine the distance measurement for the road sign by computing the distance from the location of the autonomous vehicle to the location indicated in the map for that road sign.

[0040] The system 100 can generate a network input from the portion of the image depicting the target object 112, the respective portion of the image depicting each of the one or more reference objects 114, and the respective distance measurement 114d of the reference object, and process the network input using a neural network to generate the target distance estimate 150.

[0041] In some implementations, the network input includes one or more pair embeddings 130, each pair embedding corresponding to a different target-reference pair including the target object and one of the reference objects. Specifically, the system 100 includes a pair embedding generation engine 120 to generate a respective target-reference pair embedding 130 for each target-reference pair including the target object and a reference object. As used in this specification, an embedding is an ordered collection of numerical values, such as a vector of floating-point or other type of numerical values having a fixed dimension.

[0042] In general, each target-reference pair embedding 130 can encode not only features of the target object and the reference object, but also features of the pairwise relationship between the target object and the reference object.

[0043] For each target-reference pair, the pair embedding generation engine 120 can generate a geographic distance embedding 132 encoding a geographic distance relationship between the target object and the reference object using a geographic distance embedding neural network 122. The geographic distance relationship between two objects reflects a distance relationship between the two objects, a size relationship between the two objects, or both. Specifically, the pair embedding generation engine 120 can generate an embedding input and process the embedding input using the geographic distance embedding neural network 122 to generate the geographic distance embedding 132. The embedding input includes data indicative of attributes of the target portion 112 and the reference portion 114 of the pair, and the distance measurement 114d of the reference object of the pair.

[0044] In some implementations, the attributes indicated by the embedding input include one or more of: a relative position difference between a center of the target portion and a center of the reference portion, or a relative scale between a size of the target portion and a size of the reference portion. For example, the embedding input can include one or more of: center coordinates of bounding boxes of the target and reference objects, a relative position difference between center coordinates of bounding boxes of the target and reference objects, sizes of bounding boxes of the target and reference objects, or a relative scale between sizes of bounding boxes of the target and reference objects.

[0045] The neural network 122 can have any appropriate architecture that allows the neural network to map the embedding input to an embedding. As a particular example, in some implementations, the geographic distance embedding neural network 122 used to process the embedding input to generate the geographic distance embedding is a multi-layer perceptron (MLP).

[0046] In some implementations, the pair embedding generation engine 120 can generate a union embedding 134 for a target-reference pair using the feature extraction neural network 124. Specifically, the pair embedding generation engine 120 identifies a union bounding box 116 that is part of the image that includes both the target portion of the pair and the reference portion of the reference object. The pair embedding generation engine 120 then generates the union embedding 134 by processing the portion of the image within the union bounding box 116 using the feature extraction neural network 124.

[0047] The neural network 124 can have any appropriate architecture that allows the neural network to map the union bounding box to an embedding. For example, the feature extraction neural network 124 can include a convolutional neural network. In one particular example, the pair embedding generation engine 120 utilizes a convolutional neural network (e.g., ResNet) to process the image 110 to generate a feature map of the image, and then utilizes an ROIAlign operation to process the feature map and the union bounding box 116 to crop a union embedding 134 from the feature map that corresponds to the union bounding box 116. Example techniques for applying the ROIAlign operation are described in “Fast R-CNN,” Girshick, arXiv: 1504.08083 [cs.CV], 2015 and “Mask R-CNN,” He et al., arXiv: 1703.06870 [cs.CV], 2017, the entire contents of which are incorporated by reference.

[0048] The pair embedding generation engine 120 can generate a pairwise embedding for a target-reference pair by combining (e.g., by concatenating) the geographic distance embedding 132 and the target-reference union embedding 134.

[0049] The pair embedding generation engine 120 can also obtain respective feature embeddings for the target object and the reference object in the target-reference pair. The target object feature embedding 136 for the target object includes features extracted from the portion of the image that depicts the target object. The reference object feature embedding 138 for the reference object includes features extracted from the portion of the image that depicts the reference object.

[0050] In some implementations, similar to generating the union embedding, the pair embedding generation engine 120 can utilize an ROIAlign operation to process a feature map of the image 110 and the respective bounding box of the target or reference object to generate the feature embeddings for the target object and the reference object.

[0051] The pair embedding generation engine 120 can generate the pair embedding 130 by combining (e.g., by concatenating) the pair embeddings (including the geo-distance embedding 132 and the joint embedding 134), the target object feature embedding 136, and the reference object feature embedding 138.

[0052] The pair embedding generation engine 120 repeats the above process for each target-reference pair to generate a plurality of target-reference pair embeddings 130 for the network input.

[0053] The distance estimation engine 140 generates the distance estimate 150 for the target object by processing the network input including the pair embeddings 130 using a neural network.

[0054] In some implementations, the distance estimation engine 140 first combines the pair embeddings 130 to generate a combined embedding 146, and processes the combined embedding 146 using one or more neural network layers (such as a multi-layer perceptron (MLP) 148) to generate the distance estimate 150.

[0055] In some implementations, the distance estimation engine 140 combines the pair embeddings by averaging the pair embeddings.

[0056] In some other implementations, the distance estimation engine 140 combines the pair embeddings by generating a respective attention weight 145 for each of the target-reference pair embeddings 130, and computing a weighted sum of the pair embeddings (where each pair embedding is weighted by the respective attention weight for the pair embedding) as the combined embedding 146. This process can improve the performance of the target distance estimate because it can take into account different importance of different reference objects for the distance estimate. For example, when locating a distant car depicted in the upper right corner of the image, a vehicle in the lower left corner region of the image can be less relevant.

[0057] To guide the model to pay more attention to the reference objects with the most amount of information, the distance estimation engine 140 can use an attention-based module to generate the attention weights 145 for the weighted sum of the target-reference pair embeddings.

[0058] In some implementations, the distance estimation engine 140 updates the target-reference pair embeddings 130 to include local target-reference pair features (from the target-reference pair embeddings) as well as global features that depict higher-order relationships between the target-reference pairs.

[0059] In one particular example, the attention module includes a graph neural network followed by one or more neural network layers to generate global features from the target-reference pair embeddings, and one or more additional neural network layers to generate the attention weights for combining the target-reference pair embeddings from the global features.

[0060] More specifically, in this particular example, the distance estimation engine 140 uses a graph neural network 142 to process an input graph indicating the information of the embeddings to generate updated embeddings, and then uses additional neural network layers (such as an MLP and an average pooling layer) to process the updated embeddings to generate a global embedding that includes global features of the target-reference pairs. The input graph can include the target objects and the reference objects as nodes in the graph, and also include edges connecting each target-object pair and the pair embeddings as edge features. An example technique of applying a graph neural network to update embeddings is described in “VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation,” Gao, et al., arXiv:2005.04259 [cs.CV], 2020, the entire contents of which are incorporated by reference.

[0061] The distance estimation engine 140 can combine each original target-reference pair embedding with the global embedding, e.g., by concatenation, to generate a respective global-local pair embedding for each target-reference pair. The distance estimation engine 140 then processes the global-local pair embedding using one or more neural network layers (such as a fully connected layer followed by a softmax layer 144) to generate a respective attention weight 145.

[0062] The distance estimation engine 140 combines the target-reference pair embeddings 130 by computing a weighted sum of the pair embeddings, where each pair embedding is weighted by the respective attention weight 145 of the pair embedding, to generate a combined embedding 146.

[0063] The distance estimation engine 140 then processes the combined embedding 146 using one or more neural network layers (e.g., an MLP 148) to generate a distance estimate 150.

[0064] Before using the above-described neural networks, i.e., the neural networks including the neural networks 122, 142, 144, and 148, to localize target objects, the system 100 or another system can perform training of the neural networks using training examples. Each training example can include an image of a scene, respective bounding boxes of target objects and reference objects, distance measurements of the reference objects, and ground-truth labels of the target object distances. The training system can update the network parameters of the neural networks in the system 100 based on a distance loss that measures a difference between target distance estimates generated by the networks and the ground-truth labels, using any suitable optimizer for neural network training (e.g., SGD, Adam, or rmsProp).

[0065] In some implementations, to encourage the model to learn pair-wise relationships between the target and reference objects, the training system can train using additional supervision. For example, during the training phase, the training system can embed for each target-reference pair an additional relative (or residual) distance head. The relative distance Ad between the target object and the reference object can be used as additional supervision for updating the network parameters of the neural network. Each relative distance head processes the respective target-reference pair embedding and generates a predicted relative distance Ad between the target object and the respective reference object before the embeddings are combined. The training system can supplement the distance loss with a loss that measures the difference between the predicted Ad and the relative distance computed based on ground truth, to update the network parameters.

[0066] Figure 2 An example on-board system 220 and training system 210 are shown.

[0067] The on-board system 220 is located on the vehicle 202 and makes control decisions for the autonomous vehicle. The on-board system 220 is composed of hardware and software components, some or all of which are physically located on the vehicle 202. The components of the on-board system 220 will be described in more detail below.

[0068] In some cases, the on-board system 220 can make fully autonomous or partially autonomous driving decisions (i.e., driving decisions made independent of a driver of the vehicle 202), present information to a driver of the vehicle 202 to assist the driver in safely operating the vehicle, or both. For example, in response to determining that a target object is likely to interact with the vehicle 202, the on-board system 220 can autonomously apply the brakes of the vehicle 202 or autonomously change the trajectory of the vehicle 202 to prevent an undesired interaction between the vehicle 202 and the target object.

[0069] Although Figure 2 The vehicle 202 in FIG. 1 is depicted as a car, and examples in this document are described with reference to a car, but in general the vehicle 202 can be any kind of vehicle. For example, in addition to a car, the vehicle 202 can be another autonomous vehicle that travels along a road, such as a truck or a motorcycle. Furthermore, the on-board system 220 can include components other than those shown in FIG. 1 (e.g., a collision detection system or a navigation system). Figure 2

[0070] ​To be able to safely control the autonomous vehicle 202, the on-board system 220 includes a sensor system 240 that enables the on-board system 220 to capture sensor data in the environment of the vehicle 202. More specifically, the sensor system 240 includes one or more sensors, some or all of which are configured to receive reflections of electromagnetic radiation from the environment of the vehicle 202. For example, the sensor system 240 can include a camera and one or more laser sensors (e.g., LIDAR laser sensors) configured to detect reflections of laser light. The sensor system 240 can generate images that depict the environment of the target object and one or more reference objects.

[0071] The sensor system 240 continuously (i.e., at each of a plurality of points in time) captures raw sensor data that can indicate the direction, intensity, and distance traveled by the reflected radiation. For example, a sensor in the sensor system 240 can emit one or more pulses of electromagnetic radiation in a particular direction, and can measure the intensity of any reflections and the time at which the reflections are received. The distance can be computed by determining the time elapsed between the emission of a pulse and the receipt of its reflection. Each sensor can continuously scan a particular space in an angle, an azimuth, or both. For example, scanning in azimuth can let the sensor detect multiple objects along the same line of sight. The sensor subsystem 240 can also include a combination of components that receive reflections of electromagnetic radiation, such as a LiDAR system that detects reflections of laser light.

[0072] The sensor subsystem 240 or other components of the vehicle 202 can also classify a set of one or more raw sensor measurements from one or more sensors as a measure of another agent in the environment. The set of sensor measurements can be represented in any of a variety of ways, depending on the kind of sensor measurements that are captured. For example, a set of each raw laser sensor measurement can be represented as a three-dimensional point cloud, with each point having an intensity and a location. In some implementations, the location is represented as a range and elevation pair. Thus, when an object is within the sensing range of the LiDAR sensor system, the sensor subsystem 240 can determine the distance of the object from the vehicle 202 using the sensor measurements generated by the LiDAR sensor system.

[0073] A set of each camera sensor measurement can be represented as an image, e.g., an RGB image. In some cases, because the range of a camera sensor is generally farther than the range of a LiDAR sensor system, certain objects detected in the image can be outside the range of the LiDAR sensor system. Thus, the sensor subsystem 240 cannot directly use the LiDAR sensor system to localize these objects.

[0074] When the object detected in the image 252 generated by the camera sensor includes a target object that cannot be localized using the LiDAR sensor system, the sensor subsystem 240 can provide the image 252 to the target distance estimation system 200 on the vehicle 202.

[0075] The target distance estimation system 200 includes a machine learning model 274 for processing the image 252 to estimate a distance 276 of the target object from the vehicle 202. Referring to Figure 1 Examples of target distance estimation systems and machine learning models are described.

[0076] The planning system 290 can use the target distance estimate 276 to make fully autonomous driving decisions, i.e., to update a planned trajectory of the vehicle 202. For example, if the target distance estimate 276 is below a threshold, or more generally, when a criterion depending on the target distance estimate 276 is met, the planning system 290 can generate a fully autonomous plan to brake or change lanes. In this example, the planning system 290 can generate a fully autonomous control output, e.g., to apply the brakes of the vehicle 202, such that the autonomous vehicle 202 slows down and maintains a safe distance from the target object.

[0077] The fully autonomous driving decisions generated by the planning system 290 can be implemented by a control system of the vehicle 202. For example, in response to receiving a fully autonomous driving decision generated by the planning system 290 indicating that the brakes of the vehicle should be applied, the control system can send an electronic signal to a brake control unit of the vehicle. In response to receiving the electronic signal, the brake control unit can mechanically apply the brakes of the vehicle.

[0078] The training system 210, which is typically remote from the on-board system 220 (e.g., in a data center remote from the autonomous vehicle), generates trained parameter values 235 for one or more machine learning models 274.

[0079] The training system 210 includes a training data store 280 that stores training data for training each of the machine learning models 274. The training data store 280 receives raw training examples from vehicles operating in the real world. For example, the training data store 280 can receive raw training examples 275 from the vehicle 202 in communication with the training system 210. The training system 210 can process each raw training example 275 to generate new training examples for training one or more machine learning models 274.

[0080] The training data store 280 provides training examples 282 to a training engine 285, which is also housed in the training system 210. The training engine 285 uses the training examples 285 to update model parameters of the machine learning model 274, and provides the updated model parameters 295 to the model parameter store 230. Once the parameter values for the model 274 have been fully trained, the training system 210 can transmit the trained parameter values 235 to the in-vehicle system 220, e.g., over a wired or wireless connection.

[0081] The training system 210 is typically hosted within a data center 224, which can be a distributed computing system having hundreds or thousands of computers in one or more locations.

[0082] Figure 3A is a flowchart illustrating an example process 300 for performing target distance estimation. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a target distance estimation system (e.g., the target distance estimation system 100 of Figure 1 FIG. 1) appropriately programmed in accordance with this specification can perform process 300.

[0083] An object of the process 300 is to estimate a distance of a target object depicted in an image of a scene in an environment.

[0084] In step 310, the system obtains data specifying (i) a target portion of the image that depicts a target object detected in the image and (ii) one or more reference portions of the image, each reference portion depicting a respective reference object.

[0085] In step 320, for each of the one or more reference objects, the system obtains a respective distance measurement for the reference object as a measure of a distance from the reference object to a specified location in the environment.

[0086] In some implementations, the specified location is a location of an image sensor (e.g., a camera) that captured the image in the environment.

[0087] In some implementations, the respective distance measurements can be obtained from sensor readings of the scene in the environment (e.g., that have been generated by a LiDAR sensor).

[0088] In some implementations, some of the reference objects can be map features in the environment, and the respective distance measurements are obtained from map data of the scene in the environment.

[0089] In step 330, the system generates a network input from the image portion that depicts the target object, the respective portion of the image that depicts each of the one or more reference objects, and the respective distance measurement for the reference object.

[0090] In some implementations, the network input includes one or more pair embeddings, where each pair embedding corresponds to a target-reference pair formed by one of the target object and one of the reference objects. Specifically, the system performs step 340 to generate, for each of the one or more pairs, a respective target-reference pair embedding, where each pair includes a respective one of the target object and the reference objects.

[0091] Referring to Figure 3B An example process for step 340 of generating pair embeddings is described. In the example process, the network input includes a plurality of respective target-reference pair embeddings for a plurality of target-reference pairs, each target-reference pair being formed by a target object and a respective one of a plurality of reference objects.

[0092] In general, each target-reference pair embedding can encode not only features of the target object and the reference object, but also a pair-wise relationship between the target object and the reference object.

[0093] Referring to Figure 3B , the system performs steps 342-348 for each target-reference pair to generate a respective target-reference pair embedding.

[0094] In step 342, the system generates a geo-distance embedding that encodes a geo-distance relationship between the target object and the reference object. Specifically, the system generates an embedding input, and processes the embedding input using one or more neural network layers to generate the geo-distance embedding.

[0095] The embedding input includes data indicative of attributes of the target portion and the reference portion of the reference object in the pair, and a distance measurement of the reference object in the pair.

[0096] In some implementations, the attributes include one or more of: a relative positional offset between center coordinates of a center of the target portion and a center of the reference portion, or a relative scale between a size of the target portion and a size of the reference portion.

[0097] In some implementations, the one or more neural network layers used to process the embedding input to generate the geo-distance embedding is a multi-layer perceptron (MLP).

[0098] In step 344, the system generates a joint embedding for the target-reference pair. Specifically, the system identifies a joint bounding box that is part of the image, the part of the image including the target portion and the reference portion of the reference object in the pair. The system then generates the joint embedding by processing the joint bounding box using a feature extraction neural network.

[0099] The system can generate the pair embedding for the target-reference pair by combining (e.g., concatenating) the geo-distance embedding and the target-reference joint embedding.

[0100] In step 346, the system obtains respective feature embeddings of the target object and the reference object in the target-reference pair. The target object feature embedding of the target object includes features extracted from the image portion depicting the target object. The reference object feature embedding of the reference object includes features extracted from the image portion depicting the reference object.

[0101] In step 348, the system generates a pair embedding by combining (e.g., by concatenating) the pair embeddings (including the geographic distance embedding and the joint embedding), the target object feature embedding, and the reference object feature embedding.

[0102] The system repeats steps 342-348 for each target-reference pair to generate a plurality of target-reference pair embeddings for the network input.

[0103] Returning to Figure 3A In step 350, the system generates a distance estimate for the target object by processing the network input using the neural network. The distance estimate for the target object is an estimate of a distance from the target object to a specified point in the environment.

[0104] In some implementations, step 350 to generate the target distance estimate includes combining the pair embeddings to generate a combined embedding, and processing the combined embedding using one or more neural network layers to generate the distance estimate.

[0105] In some implementations, the system combines the pair embeddings by averaging the pair embeddings.

[0106] In some other implementations, the system combines the pair embeddings by generating respective attention weights for each of the target-reference pair embeddings, and computing a weighted sum of the pair embeddings, where each pair embedding is weighted by its respective attention weight.

[0107] Referring to Figure 3C An example process for step 350 to generate the target distance estimate is described.

[0108] Referring to Figure 3C In step 352, the system updates the target-reference pair embeddings.

[0109] In one particular example, the system can use a graph neural network to process a graph input including information of the pair embeddings to generate updated embeddings, and then use an MLP and average pooling to process the updated embeddings to generate a global embedding including global features of the target-reference pair. The system can combine each original target-reference pair embedding with the global embedding, e.g., by concatenation, to generate a respective global-local pair embedding.

[0110] In step 354, the system processes the global-local pair embeddings using one or more neural network layers to generate respective attention weights.

[0111] For example, the one or more neural network layers can include a fully connected layer followed by a softmax layer.

[0112] In step 356, the system combines the target-reference pair embeddings by computing a weighted sum of the target-reference pair embeddings, where each pair embedding is weighted by the respective attention weight for the pair embedding, to generate a combined embedding.

[0113] In step 358, the system processes the combined embedding using one or more neural network layers to generate a distance estimate.

[0114] For example, the one or more neural network layers can include a multi-layer perceptron (MLP).

[0115] This specification uses the term“configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions, the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions, the one or more programs include instructions that in operation cause a data processing apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0116] The term“data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0117] A computer program (which can also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and

[0118] In this specification, the term“database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or multiple locations. Thus, for example, an index database can include multiple collections of data, each of which can be organized and accessed differently.

[0119] Similarly, in this specification, the term“engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer.

[0120] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0121] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0122] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0123] The data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing general and compute-intensive parts of machine learning training or production (i.e., inference, workloads).

[0124] The machine learning models can be implemented and deployed using a machine learning framework, such as a TensorFlow framework, a Microsoft Cognitive Tool kit framework, an Apache Singa framework, or an Apache MXNet framework.

[0125] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0126] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., data packets representing an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the device), and the client device transmits data to the server (e.g., for purposes of generating a web page displayed in a browser on the client device).

[0127] Although this specification contains many specific implementation details, these should not be construed as limiting the scope of any inventions, but merely as illustrating typical features of the embodiments. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0128] Similarly, although operations are shown in the accompanying drawings and described in the claims in a specific order, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or that all shown operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be construed as requiring such separation in all embodiments. Rather, it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0129] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers for generating a distance estimate for a target object depicted in an image obtained by a camera sensor that captures a scene in an environment, the method comprising: obtaining data that specifies (i) a target portion of the image that depicts a target object detected in the image, and (ii) a plurality of reference portions of the image that each depict a respective reference object, wherein each of the plurality of reference objects is within range of a LiDAR sensor and the target object is outside range of the LiDAR sensor; for each of the plurality of reference objects, obtaining a respective distance measurement for the reference object from the LiDAR sensor, the distance measurement being a measure of distance from the reference object to a specified location in the environment; generating a plurality of pair embeddings from the portion of the image that depicts the target object, the respective portion of the image that depicts each of the plurality of reference objects, and the respective distance measurement for the reference object from the LiDAR sensor, wherein the plurality of pair embeddings includes a respective pair embedding for a respective pair of objects for each of the plurality of reference objects, the respective pair of objects including the target object and the reference object; and processing the plurality of pair embeddings using a neural network to generate an estimated distance from the target object that is outside range of the LiDAR sensor to the specified location in the environment.

2. The method of claim 1, wherein, the respective reference object is a map feature in the environment, and wherein the distance measurement is obtained from map data for the scene in the environment.

3. The method of claim 1, wherein, generating the respective pair embedding includes, for each of the pairs: generating a pair embedding for the pair.

4. The method of claim 3, wherein, the pair embedding includes a geo-distance embedding, and wherein generating the geo-distance embedding includes: generating an embedding input from attributes of the target portion and the reference portion of the pair and the distance measurement for the reference object in the pair; and generating the geo-distance embedding by processing the embedding input using one or more neural network layers.

5. The method of claim 4, wherein, the attributes include one or more of: a relative positional offset between a center of the target portion and a center of the reference portion; or a relative scale between a size of the target portion and a size of the reference portion.

6. The method of claim 4, wherein, the one or more neural network layers are MLPs.

7. The method of claim 4, wherein, the pair embedding includes a joint embedding, and wherein generating the joint embedding includes: identifying a joint bounding box that is a portion of the image that includes both the target portion and the reference portion of the pair; and generating the joint embedding by processing the joint bounding box using a feature extraction neural network.

8. The method of claim 7, wherein, the pair embedding is a concatenation of the geo-distance embedding and a target-reference joint embedding.

9. The method of claim 8, wherein, the pair embedding includes: a target object feature embedding of the target object that includes features extracted from the portion of the image that depicts the target object; and a reference object feature embedding of the reference object in the pair that includes features extracted from the portion of the image that depicts the reference object.

10. The method of claim 9, wherein, processing the pair embeddings using one or more neural network layers to generate the distance estimate. combining the pair embeddings to generate a combined embedding; and processing the combined embedding using one or more neural network layers to generate the distance estimate.

11. The method of claim 10, wherein, The one or more neural network layers are multilayer perceptrons (MLPs).

12. The method of claim 10, wherein, Combining the pair embeddings to generate a combined embedding includes: averaging the pair embeddings.

13. The method of claim 10, wherein, Combining the pair embeddings to generate a combined embedding includes: generating respective attention weights for each of the pair embeddings; and computing a weighted sum of the pair embeddings, where each pair embedding is weighted by the respective attention weight for the pair embedding.

14. The method of claim 13, wherein, Generating respective attention weights for each of the pair embeddings includes: processing the pair embeddings using a graph neural network to generate a respective updated pair embedding for each pair; and processing the updated pair embeddings using one or more neural network layers to generate the respective attention weights.

15. The method of claim 14, wherein, The one or more neural network layers are fully connected layers followed by softmax layers.

16. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform a method for generating a distance estimate for a target object depicted in an image obtained by a camera sensor that captures a scene in an environment, the method comprising: obtaining data specifying (i) a target portion of the image that depicts a target object detected in the image and (ii) a plurality of reference portions of the image that each depict a respective reference object, where each of the plurality of reference objects is within a range of a LiDAR sensor and the target object is outside the range of the LiDAR sensor; for each of the plurality of reference objects, obtaining a respective distance measurement for the reference object from the LiDAR sensor, the distance measurement being a measure of a distance from the reference object to a specified location in the environment; generating a plurality of pair embeddings from the portion of the image that depicts the target object, a respective portion of the image that depicts each of the plurality of reference objects, and the respective distance measurement for the reference object from the LiDAR sensor, where the plurality of pair embeddings includes a respective pair embedding for a respective pair of objects for each of the plurality of reference objects, the respective pair of objects including the target object and the reference object; and processing the plurality of pair embeddings using a neural network to generate an estimated distance from the target object, which is outside the range of the LiDAR sensor, to the specified location in the environment.

17. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform a method for generating a distance estimate for a target object depicted in an image obtained by a camera sensor that captures a scene in an environment, the method comprising: obtaining data specifying (i) a target portion of the image that depicts a target object detected in the image, and (ii) a plurality of reference portions of the image that each depict a respective reference object, wherein each of the plurality of reference objects is within range of a LiDAR sensor and the target object is outside range of the LiDAR sensor; for each of the plurality of reference objects, obtaining a respective distance measurement for the reference object from the LiDAR sensor, the distance measurement being a measurement of a distance from the reference object to a specified location in the environment; generating a plurality of pair embeddings from the portion of the image that depicts the target object, a respective portion of the image that depicts each of the plurality of reference objects, and the respective distance measurement for the reference object from the LiDAR sensor, wherein the plurality of pair embeddings includes a respective pair embedding for a respective pair of objects for each of the plurality of reference objects, the respective pair of objects including the target object and the reference object; and processing the plurality of pair embeddings using a neural network to generate an estimated distance from the target object that is outside range of the LiDAR sensor to the specified location in the environment.

Images