Depth refinement from images

Abstract

A depth estimate of an object produced by one or more sensors of the vehicle may be refined using the locations of environmental attributes proximate the object. Captured images of the object proximate the environmental attributes may be analyzed to determine where the object is located relative to the environmental attributes. A machine learning model may be used to detect the environmental attributes, and the locations of the environmental attributes may be determined from map data. A probability of the object's location may be determined based on the known locations of the environmental attributes. The object's location probability may be used to refine a depth estimate produced by other means, such as monocular depth estimation from images using computer vision.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. patent application Ser. No. 16 / 452,304, filed Jun. 25, 2019, entitled “Depth Refinement from Images,” the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Various methods, devices, and systems are utilized to guide an autonomous vehicle through an environment that includes multiple static and dynamic objects. For example, an autonomous vehicle may utilize route planning methods, devices, and systems to guide the autonomous vehicle through congested areas having other moving and stationary vehicles (autonomous or otherwise), people, buildings, and other objects. In some examples, the autonomous vehicle may make decisions to ensure the safety of passengers and surrounding people and objects while navigating the environment. Various sensors may be used to gather information about objects in the surrounding environment, which may be used by the autonomous vehicle to make decisions on how to navigate the environment. Accurately predicting the depth of an object in the environment (e.g., the distance of the object from the autonomous vehicle) may be a challenge in some cases. [Brief description of the drawings]

[0003] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical items or features according to examples of the present disclosure.

[0004] [Figure 1] FIG. 1 is an illustrated flowchart of refining object depth estimation from a vehicle using environmental attributes, according to an example of the present disclosure. [Diagram 2] FIG. 2 is an illustration of objects detected in an environment for which a vehicle may determine and refine a depth estimate, in accordance with an example of this disclosure. [Diagram 3] FIG. 3 is an illustration of a 3D point cloud including objects in the environment shown in FIG. 2 and depth estimation of the objects in the environment, according to an example of the present disclosure. [Figure 4] FIG. 4 is an illustration of the three-dimensional point cloud of FIG. 3 further illustrating the location of environmental attributes according to an example of the present disclosure. [Diagram 5] FIG. 5 is an illustration of the three-dimensional point cloud of FIG. 3 further illustrating refined depth estimation of objects based on location of environmental attributes according to an example of the present disclosure. [Figure 6] FIG. 6 is an illustration of the use of a machine learning model to identify environmental attributes used to update object depth estimates, according to an example of this disclosure. [Figure 7] FIG. 7 illustrates a block diagram of an example system for implementing the techniques described herein. [Figure 8] FIG. 8 shows an exemplary process for identifying environmental attributes using a machine learning model, determining locations of the environmental attributes from map data, and using the locations of the environmental attributes to refine a determination of the depth of an object from a vehicle, according to an example of this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] This disclosure relates to techniques for accurately determining the depth of an object in an environment by taking into account one or more other attributes in the environment. In some examples, the techniques can include refining an initial depth estimate of the object made by one or more sensors (e.g., of a vehicle, an autonomous vehicle) using the location of environmental attributes proximate to the object. In some examples, the environmental attributes are features that typically remain in the same place in the environment, where the location of the environmental attributes is known (e.g., stored in map data of the environment). Examples of environmental attributes can include features such as, for example, sidewalks, crosswalks, bike lanes, lane markers on drivable surfaces, signs, fences, grass, or other ground cover materials.

[0006] The captured image of an object in proximity to an environmental attribute may be analyzed to determine where the object is located relative to the environmental attribute (e.g., above, behind, in front, next to, etc.). The object's location relative to the environmental attribute may then be used to refine an object's depth estimate, determined, for example, by a monocular depth estimate (e.g., using a machine learning model) corresponding to individual pixels of the image. In some cases, an object's depth estimate made using data generated from a single image may be inaccurate for a variety of reasons, such as the measurement accuracy decreasing (increasing uncertainty) the further away from the camera that captured the image. Even a small depth inaccuracy (e.g., a few inches or centimeters) may be the difference between a pedestrian on a sidewalk or street, a child in front of or behind a chain link fence, a bicycle in a bike lane or lane, etc. Thus, additional mechanisms for refining the depth estimate, such as the techniques described herein that use environmental attributes, improve the safety of autonomous vehicles in a variety of different scenarios.

[0007] Sensor data captured by an autonomous vehicle can include data captured by camera sensors, such as individual images, videos, etc., as well as data captured by other sensors, such as lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, etc. In some cases, the sensor data can be provided to a perception system configured to determine or classify types of objects in the environment (e.g., vehicles, pedestrians, bicycles, motorcycles, animals, parked cars, trees, buildings, etc.). Additionally, in some examples, sensor data, such as images, may be used to determine the depth of objects in the environment around the vehicle, e.g., the distance from the vehicle to various objects in the environment.

[0008] The sensor data may be captured by the autonomous vehicle as it travels through the environment. In some examples, the autonomous vehicle may include one or more cameras configured to capture images (e.g., individual images and / or video) of the surrounding environment, among other sensors. The images captured by the one or more cameras may be analyzed to detect objects, independent of or in combination with detection of the objects by other sensors of the vehicle. Once detected, in some examples, the images may be cropped to a region of interest around the object based on a bounding box that encloses the object, for example, based on a size (centimeters, pixels, etc.) surrounding the individual object. The objects detected in the images captured by the camera may be dynamic objects, such as pedestrians, bicycles, vehicles, etc., and generally do not remain stationary long enough to be incorporated into the map data of the environment. In some examples, an initial depth prediction of the object from the autonomous vehicle may be determined by inputting the image of the object into a machine learning model, such as a deep neural network, trained to predict the depth of a surface shown or represented in individual pixels of the image. Alternatively or additionally, sensor data from one or more modalities may be combined using data fusion techniques, such as Kalman filters and / or deep neural networks, to determine an initial depth estimate of the object from the autonomous vehicle. However, examples are contemplated in which the determination of an initial depth estimate from surfaces shown in the image may be omitted, and the depth estimate may be based on a comparison of the object's position to environmental attributes, as described below. Alternatively or additionally, a single machine learning model may be configured to determine, in some examples, in a single operation, a single depth estimate from both surfaces shown in the image, along with the object's relative position to the environmental attributes.

[0009] An attribute identifier component of the autonomous vehicle may determine environmental attributes in proximity to objects shown in the image (and / or shown in a cropped portion of the image). The attribute identifier component may determine environmental attributes in some cases by semantically segmenting the image and labeling pixels of the image that are associated with labels of environmental attributes. An environmental attribute may be considered to be “proximate” to an object if it intersects with the object as shown in the image or is within a threshold distance (e.g., a few pixels, a few centimeters, etc.). In some cases, a machine learning model may be trained to classify environmental attributes such as sidewalks, crosswalks, bike lanes, fences, etc. shown in the images input to the machine learning model. For example, the machine learning model may output environmental attributes in proximity to objects shown in the image and / or provide a relative position of the object with respect to the environmental attributes (e.g., above, behind, in front, next to, etc.). The relative position of the object with respect to the environmental attributes may be represented as a vector, coordinates, or other mechanism that conveys relative positions between entities when output by the machine learning model.

[0010] A position determination component of an autonomous vehicle may determine a location of an environmental attribute from map data. In some examples, the position determination component may have access to one or more maps that include various environmental attributes, such as roads, sidewalks, buildings, signs, crosswalks, etc. The position determination component may associate environmental attributes received from a machine learning model with environmental attributes included in the map data. In some examples, the position determination component may determine a probability of an object's location based on the location of the environmental attribute in the map data. For example, the position determination component may receive an indication from a machine learning model that a bicycle (e.g., an object) is behind a lane marker (e.g., an environmental attribute). Based on the location of the lane marker on the map, the position determination component may determine that the bicycle is in a bicycle lane behind the lane marker relative to the vehicle and is not in a lane in front of the lane marker relative to the vehicle.

[0011] In an example, the position determination component refines the depth estimate of the object based on the probability of the object's location, e.g., relative to the environmental attributes. For example, the vehicle may utilize the vehicle's known location (e.g., determined from a system such as simultaneous localization and mapping (SLAM)) and the location of the environmental attributes from map data to update the object's depth estimate. Continuing with the example above, the vehicle may use a three-dimensional point cloud to predict the depth of a surface from the vehicle and determine an initial depth estimate of a bicycle at 9 meters. If the vehicle determines the environmental attributes of a lane marker to be 9.5 meters from the vehicle, the vehicle may refine the depth estimate of the bicycle to 10 meters from the vehicle, recognizing that the bicycle is behind the lane marker.

[0012] The techniques described herein can improve the functionality of a computing device in several ways. As described above, the attribute identifier component may use just a single image (or a cropped portion of a single image) to make a reliable depth estimate of an object in the environment surrounding the vehicle. As a result, significantly fewer processing resources are used compared to conventional techniques that require complex image analysis algorithms applied to images captured by multiple cameras and / or other sensors to determine the depth of the object. Because the depth determination can be made from an image from a single camera, the position determination component may determine the depth of more objects in the environment than would be possible if images from multiple cameras were required. In some cases, as described above, the described techniques are more accurate than other depth measurement mechanisms. For example, depth estimates from monocular images may be limited in range and may decrease in accuracy as surfaces move away from the camera used to capture the image. These estimates may be improved by refining an initial depth estimate of the object based on the object's position relative to known locations of environmental attributes proximate to the object, as described herein. In an example of utilizing data fusion to estimate the depth of an object, the initial depth estimate and the refined depth estimate may be used as constraints and / or costs for estimating the depth. Controlling a vehicle based in part on the depth of an object determined from image data can improve vehicle safety by determining the depth of an object faster and earlier, thus allowing the vehicle to make faster trajectory determinations. Additionally, as described, controlling a vehicle based in part on determining the depth of an object from an image can reduce processing resources, with only one image being evaluated by the processing resources at a time during depth determination. Furthermore, techniques for controlling a vehicle based in part on determining the depth of an object from an image using environmental attributes can increase the reliability with which the vehicle can avoid collisions with oncoming vehicles and / or pedestrians by determining depth earlier and more accurately, which may improve safety outcomes, performance, and / or accuracy.These and other improvements to the functionality of the computer are described herein.

[0013] The techniques described herein can be implemented in several ways. Exemplary embodiments are presented below with reference to the figures. Although described in the context of an autonomous vehicle, the methods, apparatus, and systems described herein can be applied to a variety of systems (e.g., sensor systems, or robotic platforms) and are not limited to autonomous vehicles. In one example, the techniques described herein can be utilized in a driver-controlled vehicle where such a system can provide an indication to the driver of the vehicle of whether it is safe to perform various operations. In another example, the techniques can be utilized in an aviation or nautical context, or in any system involving objects or entities that can be associated with actions unknown to the system. Such techniques can also be used, for example, in a manufacturing and assembly context to inspect components as they move down an assembly line. In addition, the techniques described herein can be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0014] FIG. 1 is an illustrated flowchart 100 for refining depth estimation of objects from a vehicle using environmental attributes, according to an example of the present disclosure.

[0015] The operation 102 includes determining a depth estimate of an object in the environment. The depth estimate may be determined by inputting an image of the object into a deep neural network or other machine learning model configured to predict the depth of a surface shown in the image at individual pixels of the image. For example, the example 104 includes a vehicle 106 having a sensor 108 that may be configured to capture information of the surrounding environment. In some examples, the vehicle 106 may determine a depth 110 of the object 112 from the vehicle 106 based on data captured by the sensor 108. For example, the sensor 108 may be a camera that captures an image of the environment around the vehicle 106 showing the object 112. The vehicle 106 may unproject rays from the image onto a three-dimensional surface mesh of the environment, which may be used as an estimate of the projected position of the object 112. The vehicle 106 may use the projected position of the object 112 to determine an initial depth estimate of the object 112 from the vehicle 106, such as based on a known position of the vehicle 106 using a system such as SLAM.

[0016] The operation 114 includes determining environmental attributes proximate the object. In some examples, the environmental attributes may be determined from an image captured by the sensor 108 of the vehicle 106, the image showing the object 112 and at least a portion of the environment surrounding the object 112. For example, consider an example image 116 showing the object 112 captured by the sensor 108 of the vehicle 106. The image 116 may be captured by the sensor 108, or in some cases may be a cropped portion of the image to an area surrounding the object 112, for example, based on a predefined overall size and / or a predefined boundary size surrounding the object 112.

[0017] In an example, environmental attributes 118 proximate to object 112 may be determined by inputting image 116 to a machine learning model trained to identify different environmental attributes. For example, the machine learning model may be trained as a classifier to detect environmental attributes such as sidewalks, bike lanes, traffic lanes, crosswalks, etc. In some cases, the machine learning model may be trained to classify different environmental attributes according to the type of object shown in an input image, such as image 116. For example, if the object shown in the input image is a pedestrian, the machine learning model may classify the environmental attributes into one or more classifications related to pedestrians, a first sidewalk parallel to the direction of travel of vehicle 106, a second sidewalk perpendicular to the direction of travel of vehicle 106, a crosswalk (e.g., parallel or perpendicular to the direction of travel of the vehicle), a lane of a particular traffic, etc. In another example, if the object shown in the input image is another vehicle, the machine learning model may classify the environmental attributes into one or more classifications associated with the vehicle, such as an oncoming lane associated with the vehicle 106, a lane shared by the vehicle 106 and other vehicles, a lane traveling in the same direction as the vehicle 106 (although not necessarily shared with the vehicle 106), a parking space, or a roadway, to name a few. In at least some examples, the context portions stored in the map (e.g., the location and range of the environmental attributes 118) may be projected onto the image 116 along with associated depth information. Such depth information may then be associated with the object 112 (as further defined herein) to refine the estimation. If the object shown in the input image is a two-wheeled vehicle, such as a bicycle, motorcycle, scooter, or the like, the machine learning model may classify the environmental attributes into one or more classifications associated with the two-wheeled vehicle, such as, for example, a bike lane, a sidewalk, or a roadway. Of course, these examples of environmental attributes are non-limiting, and any environmental attribute may be used as a classification by the machine learning model.

[0018] The machine learning model may output environmental attributes 118, and in some cases may also output the position of the object 112 relative to the environmental attributes 118. For example, the machine learning model may output an indication that the object 112 is above the environmental attributes 118 and / or behind a portion of the environmental attributes 118. In this example, the object 112 is a pedestrian, the environmental attributes 118 are sidewalks, and the pedestrian is on the sidewalk and behind the curb or edge of the sidewalk, but not in front of the sidewalk (e.g., in the street). Such information regarding the position of the object 112 relative to the environmental attributes 118, when utilized along with map data of the environment as described, may significantly improve depth estimation without requiring multiple images from various cameras to detect depth.

[0019] For example, the operation 120 includes determining a probability of the object's location based at least in part on the location of the environmental attribute from the map data. The vehicle 106 may have access to one or more maps of the surrounding environment, which may provide the location of the environmental attribute 118 as part of the map data. After receiving an indication of the environmental attribute 118 from the machine learning model, the vehicle 106 may utilize the map data to determine a likely location of the object 112 based on the proximity of the object 112 to the environmental attribute 118. Additionally, the probability of the object 112's location may increase with the location of the object 112 relative to the environmental attribute 118 output by the machine learning model (e.g., the object 112 is in front of, behind, above, below, left, right, etc. of the environmental attribute 118), as described. Thus, the probability of the object 112's location may correspond to the location of the object 112 in the environment based on the map data.

[0020] The operation 122 includes refining the depth estimate of the object based at least in part on the probability of the position. For example, the example 124 shows the vehicle 106 with the object 112 in a position based on the first depth estimate and an object 126 corresponding to the object 112 in a position based on the refined depth estimate (or refined distance estimate). The example 124 shows how the depth estimate of the object 112 is refined based on the position of the object 112 relative to the environmental attributes 118. In some examples, refining the depth estimate may include using a second depth estimate determined from the object's position probability, e.g., replacing the first depth estimate with the second depth estimate. However, in some cases, the first and second depth estimates may be combined (e.g., based on a weighted average), and the likelihood of the first depth estimate may be compared to the likelihood of the second depth estimate to select the first or second depth estimate in the final depth estimate, etc. As described in connection with FIG. 6, other techniques for refining the depth estimate are contemplated. The vehicle 106 may be controlled based on the refined depth estimate determined in operation 122, such as by selecting a trajectory to follow based on the refined depth estimate of the object 126 from the vehicle 106.

[0021] 2 is an illustration of an example image 200 of objects detected in an environment for which a vehicle may determine and refine a depth estimate, according to an example of this disclosure. For example, the example image 200 may be an image captured by a sensor of the vehicle 202, such as a camera, as the vehicle 202 travels through the environment.

[0022] The exemplary image 200 may include various objects throughout the environment, such as an object 204, a crosswalk 206, etc. In some examples, the vehicle 202 may want to determine the depth of the object 204 and further determine how to navigate a portion of the environment, such as a junction shown in front of the vehicle 202. For example, if the object 204 is on the sidewalk, the vehicle 202 may decide to proceed through the junction, but if the object 204 is in the crosswalk 206, then the vehicle 202 may decide to stop and maintain its current position without entering the junction, e.g., to avoid disrupting traffic moving vertically through the junction. In some examples, the vehicle 202 may generate a crop 208 of the exemplary image 200 and input this into a machine learning model to determine environmental attributes proximate to the object 204. In this manner, processing resources may be saved by analyzing a portion of the image with environmental attributes related to the location of the object 204, rather than the entire image.

[0023] FIG. 3 is an illustration 300 of a 3D point cloud including objects in the environment shown in FIG. 2 and depth estimation of the objects in the environment, according to an example of this disclosure.

[0024] For example, the exemplary image 200 may be input to a machine learning model, such as a deep neural network or other type of machine learning model, trained to predict the depth of a surface shown in the image for each of the pixels in the image. In some examples, the machine learning model may output a three-dimensional point cloud 302 that corresponds to the predicted depths at each of the pixels in the input image. Diagram 300 provides a top-down view of a point cloud corresponding to the two-dimensional, exemplary image 200.

[0025] For example, indicator 304 indicates a depth estimate of object 204 shown in example image 200 from vehicle 202, generated from three-dimensional point cloud 302. In some examples, the probability that object 204 is at the exact location of indicator 304 may be relatively low, for example, based on the distance of object 204 from a camera used to capture example image 200 on vehicle 202. Thus, to improve the depth estimate of object 204, known locations of environmental attributes may be utilized to refine and increase accuracy of the depth estimate.

[0026] For example, consider FIG. 4, which is an illustration 400 of the three-dimensional point cloud 302 of FIG. 3, further illustrating the location of the environmental attribute, according to an example of the present disclosure. The illustration 400 includes an environmental attribute 402 that may correspond to the crosswalk 206 of FIG. 2. In an example, a machine learning model may be used as described herein to identify the crosswalk as an environmental attribute proximate to the object 204 from the crop 208. For example, the machine learning model may associate the object 204 with the attribute in image space to determine the relative location of the object 204 with respect to the environmental attribute. The vehicle 202 may obtain map data including the known location of the crosswalk 206, which may be used to determine the location of the environmental attribute 402 with respect to the three-dimensional point cloud 302. As shown, the depth estimation of the object 204 based on the three-dimensional point cloud 302 places the indicator 304 in a position immediately behind the environmental attribute 402. In other words, the depth estimation of the object 204 based on the three-dimensional point cloud 302 indicates that the object 204 is behind the crosswalk 206 and in a traffic lane.

[0027] However, consider FIG. 5, which is an illustration 500 of the three-dimensional point cloud 302 of FIG. 3, which further illustrates a refined depth estimation of an object based on the location of the environmental attributes 402, according to an example of this disclosure. As described herein, the location of the object 204 may be determined relative to the crosswalk 206, for example, based on how the object 204 and the crosswalk 206 are shown in the crop 208. For example, the machine learning model may output an indication that the object 204 is within the crosswalk 206, rather than in front of the crosswalk 206, behind the crosswalk 206, etc., based on the crop 208. Because the vehicle 202 has access to map data indicating the location of the environmental attributes 402, the vehicle 202 can determine a probability of the location of the object 204 based on the known location of the environmental attributes 402 (e.g., the crosswalk 206) and the location of the object 204 relative to the environmental attributes 402 (e.g., "in" the crosswalk 206). Thus, the vehicle 202 may update the depth estimate from indicator 304 “in” the environmental attribute 402 to indicator 502 , which corresponds to the object 204 being “in” the crosswalk 206 .

[0028] 3, 4, and 5 include a pedestrian in a crosswalk, other examples utilizing relative location are also contemplated. In some examples, the relative position of the object 204 with respect to the environmental attributes 402 may further take into account additional environmental attributes, such as the ground, drivable surfaces, and / or edges or intersections of the object 204 with the environmental attributes 402. For example, a determination may be made that the object 204 is within the crosswalk 206 if an edge of the crosswalk 206 intersects with the object 204.

[0029] FIG. 6 is an illustration 600 of the use of a machine learning model to identify environmental attributes used to update depth estimates of objects, according to an example of this disclosure.

[0030] Example 602 illustrates a top view of a vehicle 604 traveling through an environment, and an object 608 (e.g., similar to example 104 of FIG. 1) detected to be at a depth 606 from the vehicle 604. The depth 606 may be an estimate of a probability of a location of the object 608 in the environment from the vehicle 604, such as based on a monocular depth estimate as described above in connection with FIG. 3. For example, example 610 illustrates an ellipse 612 generally at the location of the object 608, where the ellipse 612 may correspond to a potential location of the object 608 in the environment (e.g., an uncertainty that may be output from such a machine-learned depth estimate). A first object location probability 614 may correspond to a Gaussian or normal distribution of the potential locations of the object 608 along the major axis of the ellipse 612. An x-axis of the first object location probability 614 may correspond to a location or location along the major axis of the ellipse 612, while a y-axis of the first object location probability 614 may correspond to a probability that the object 608 is at a particular location or location.

[0031] An image captured by the vehicle 604 depicting the object 608 and at least a portion of the environment surrounding the object 608 may be input to a machine learning model 616. The machine learning model 616 may be configured to classify environmental attributes shown in the image and may be further configured to output a location of the object relative to the environmental attributes (e.g., behind, in front, next to, etc.) as described herein. For example, example 618 may correspond to an image of the object 608 captured by the vehicle 604 along with environmental attributes 620 identified by the machine learning model 616. The machine learning model 616 may also output an indication that the object 608 is on a sidewalk, behind a curb, or the like, providing additional information to refine the initial depth 606 from the vehicle 604 to the object 608. In some cases, the machine learning model 616 may further determine additional information regarding the relative positions of the object 608 and the environmental attributes 620, such as an estimated distance between the object 608 and the environmental attributes 620, the angle between the object 608 and the environmental attributes 620, and others that may be used to further refine the depth of the object 608. Additionally, in some examples, map data may be input into the machine learning model along with an image of the object 608, such as in cases where environmental attributes that may help determine depth are occluded in the image.

[0032] Example 622 shows a vehicle 604 and an ellipse 612 as described in relation to example 610, and further shows an ellipse 624 illustrating a refined estimate of a potential location of an object 608 in the environment from the ellipse 612. The second object position probability 626 may correspond to a Gaussian or normal distribution of the potential locations of the object 608 along the axis of the ellipse 624. As above, the x-axis of the second object position probability 626 may correspond to a location or place along the major axis of the ellipse 624, while the y-axis of the second object position probability 626 may correspond to a probability that the object 608 is at a particular location or place. When comparing the second object position probability 626 to the first object position probability 614, the location or place of the object 608 with the highest probability is moved along the x-axis, while the probability that the object 608 is at a particular location or place is higher. Thus, in this example, the location of the object 608 in the environment is more likely to be farther away from the vehicle 604 than the initial depth 606 determined by the vehicle 604.

[0033] In some examples, the first object position probability 614 and the second object position probability 626 may be combined into a multimodal position probability 628. Generally, a multimodal distribution has multiple modes (e.g., modes occur above a threshold probability of a particular position). By combining the first object position probability 614 and the second object position probability 626. The vehicle 604 may evaluate which of the position or location estimates is most likely for the object 608 (e.g., a maximum likelihood depth estimate) and then use the most likely position or location estimate to calculate a revised depth estimate. In other examples, the first object position probability 614 and / or the second object position probability 626 may be averaged (e.g., by a weighted average) or otherwise combined to determine a revised depth estimate.

[0034] Additionally, one or more of the first object position probability 614, the second object position probability 626, and / or the multi-modal position probability 628 may be mapped to a three-dimensional location, such as in voxel space, providing the vehicle 604 with a three-dimensional image of the surrounding environment. The vehicle 604 may then use the probability of the depth of the object 608 in voxel space, such as by combining the probability of occupancy with information from other sensor modalities, such as lidar, radar, time-of-flight, etc., in determining the probability of occupancy within the voxel grid. In this manner, the vehicle 604 may develop a more complete and accurate understanding of its surroundings, enabling the vehicle 604 to navigate through the environment more safely and with better outcomes.

[0035] 7 shows a block diagram of an example system 700 for implementing the techniques described herein. In at least one example, the system 700 can include a vehicle 702, such as an autonomous, semi-autonomous, or manually controlled vehicle.

[0036] The vehicle 702 may include a vehicle computing device 704, one or more sensor systems 706, one or more emitters 708, one or more communication connections 710, at least one direct connection 712, and one or more drive systems 714.

[0037] The vehicle computing device 704 may include one or more processors 716 and a memory 718 communicatively coupled to the one or more processors 716. In the illustrated example, the vehicle 702 is an autonomous vehicle, although the vehicle 702 may be any other type of vehicle or robotic platform. In the illustrated example, the memory 718 of the vehicle computing device 704 stores a localization component 720, a perception component 722, one or more maps 724, one or more system controllers 726, a depth determination component 728, an attribute identifier component 730, a position determination component 732, and a planning component 734. While shown in FIG. 7 as residing in memory 718 for illustrative purposes, it is contemplated that the localization component 720, the perception component 722, the one or more maps 724, the one or more system controllers 726, the depth determination component 728, the attribute identifier component 730, the position determination component 732, and the planning component 734 may additionally or alternatively be accessible to the vehicle 702 (e.g., stored in memory separate from the vehicle 702 or otherwise accessible).

[0038] In at least one example, the localization component 720 can include functionality to receive data from the sensor system 706 to determine the position and / or orientation of the vehicle 702 (e.g., one or more of x, y, z positions, roll, pitch, or yaw). For example, the localization component 720 can include and / or request / receive a map of the environment and can continually determine the position and / or orientation of the autonomous vehicle within the map. In some examples, the localization component 720 can receive image data, lidar data, radar data, time of flight data, IMU data, GPS data, wheel encoder data, etc., utilizing SLAM (simultaneous localization and mapping), CLAMS (calibration, localization and mapping simultaneously), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine the position of the autonomous vehicle. In some examples, the localization component 720 can provide data to various components of the vehicle 702 to determine to search map data to generate a trajectory, as discussed herein, and to determine an initial position of the autonomous vehicle.

[0039] In some examples, the perception component 722 can include functionality for performing object detection, segmentation, and / or classification. In some examples, the perception component 722 can provide processed sensor data indicative of the presence of entities and objects proximate to the vehicle 702 and / or classification of the entities as entity types (e.g., automobiles, wheels, pedestrians, bicycles, animals, buildings, trees, road surfaces, curbs, sidewalks, traffic lights, stop signs, lane markers, unknown, etc.). In additional or alternative examples, the perception component 722 can provide processed sensor data indicative of one or more characteristics associated with the detected entities (e.g., tracked objects) and / or the environment in which the entities are located. In some examples, the characteristics associated with the entities can include, without limitation, x-position (global and / or local position), y-position (global and / or local position), z-position (global and / or local position), orientation (e.g., roll, pitch, yaw), entity type (e.g., classification), velocity of the entity, acceleration of the entity, range (size) of the entity, etc. Characteristics associated with an environment may include, but are not limited to, the presence of other entities in the environment, the state of other entities in the environment, the time of day, the day of the week, the season, weather conditions, dark / light indications, and the like.

[0040] In those examples in which the perception component 722 performs detection, the perception component 722 may output detections of objects detected in the image and / or associated environmental attributes of nearby objects. Such detections may include two-dimensional bounding boxes (which may be used to later crop the image) and / or masks of the detected objects. In some examples, such detections may utilize machine learning approaches (e.g., scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG), etc.) followed by a support vector machine (SVM) to classify objects shown in the images received from the cameras of the sensor system 706. Alternatively or additionally, the detections may utilize deep learning approaches based on convolutional neural networks (CNNs) to classify objects shown in the images received from the cameras of the sensor system 706. In examples, the detections may utilize one or more object detection techniques (or others) in accordance with the described techniques to detect objects shown in the images and / or to detect environmental attributes (e.g., nearby objects) shown in the images.

[0041] The memory 718 may further include one or more maps 724 that the vehicle 702 may use to navigate within the environment and / or to determine the location of environmental attributes. For purposes of illustration, a map may be any number of data structures modeled in two, three, or N dimensions that may provide information about the environment, such as, but not limited to, topology (such as intersections), streets, mountain ranges, roads, terrain, and the general environment. In some examples, the map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., lidar information, radar information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectance information (e.g., specular reflectance information, retroreflectance information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some examples, the map can be stored in a tile format and loaded into the working memory as needed, such that individual tiles of the map represent separate portions of the environment. In at least one example, the one or more maps 724 can include at least one map (e.g., an image and / or a mesh). In some examples, the vehicle 702 can be controlled at least in part based on the map 724. That is, the map 724 can be used in conjunction with the localization component 720, the perception component 722, the depth determination component 728, or the planning component 734 to determine a position of the vehicle 702, identify objects and / or environmental attributes within the environment, and / or generate a route and / or trajectory for navigating within the environment.

[0042] In some examples, one or more maps 724 can be stored on a remote computing device (such as computing device 738) accessible via network 736. In some examples, multiple maps 724 can be stored, for example, based on a characteristic (e.g., type of entity, time of day, day of the week, season, etc.). Storing multiple maps 724 can have similar memory requirements but increases the speed at which the data in the maps can be accessed. In some examples, one or more maps 724 can store sizes or dimensions of objects associated with individual locations in the environment. For example, as the vehicle 702 travels through the environment and a map representing an area proximate to the vehicle 702 is loaded into memory, one or more sizes or dimensions of objects associated with the locations can also be loaded into memory.

[0043] In general, the depth determination component 728 can estimate the depth of objects in the environment surrounding the vehicle 702 and refine the depth estimation using the positions of environmental attributes based on data provided by the sensor system 706. In some examples, the depth determination component 728 can provide information generated by the attribute identifier component 730 and / or the position determination component 732 to the planning component 734 to determine when and / or how to control the vehicle 702 to navigate the environment. As described herein, the depth determination component 728 can receive image data, map data, lidar data, etc. to determine information about objects in the environment.

[0044] From an image (or a cropped image) of an object, the attribute identifier component 730 can determine classifications of environmental attributes proximate the object, and in some cases, can also determine the location of the object relative to the environmental attributes. For example, the attribute identifier component 730 can utilize a machine learning model trained to classify environmental attributes into categories, such as sidewalks, bike lanes, crosswalks, or lanes, to name a few. In some examples, the attribute identifier component 730 can determine different classifications based on the particular objects identified in the image, such as a first set of classifications of vehicles (e.g., oncoming traffic lanes, lanes traveling in the same direction as the vehicle 702, parking spaces, roadways, etc.), a second set of classifications of pedestrians (e.g., sidewalks parallel to the direction of travel of the vehicle 702, sidewalks perpendicular to the direction of travel of the vehicle 702, crosswalks, traffic lanes, etc.), a third set of classifications of two-wheeled vehicles, such as bicycles, motorcycles, and / or scooters (e.g., bike lanes, sidewalks, lanes, etc.). Additionally, the attribute identifier component 730 may refine parameters of the machine learning model by comparing the predicted environmental attributes provided by the machine learning model to environmental attributes indicated in map data of the surrounding environment (e.g., provided by map 724) to more accurately classify the environmental attributes. In some examples, the attribute identifier component 730 can provide information regarding the environmental attributes and / or the position of objects relative to the environmental attributes to the planning component 734 for use in controlling the vehicle 702.

[0045] The position determination component 732 can determine a revised depth estimate of the object from the vehicle 702 from the environmental attributes, and in some cases from the position of the object relative to the environmental attributes. For example, the position determination component may determine a position probability of the object based at least in part on the known positions of the environmental attributes proximate the object, and the position of the object relative to the environmental attributes. Additionally, the position determination component 732 may update the depth estimate provided by a neural network that predicts a depth estimate for individual pixels of an image of the environment. For example, the position determination component 732 may utilize a multi-modal distribution of the probability of the object's position provided from the first depth estimate and the depth estimate from the relationship between the object and the environmental attributes. The position determination component 732 in some examples may select the position with the largest mode in the multi-modal distribution as the predicted position of the object, although other examples are contemplated and / or described. The position determination component 732 may then calculate the depth of the object from the vehicle 702 using the known position of the vehicle (e.g., determined using SLAM or CLAMS) and the predicted position of the object.

[0046] In general, the planning component 734 can determine a path that the vehicle 702 will follow to travel through an environment. For example, the planning component 734 can determine various routes and trajectories, as well as various levels of detail. For example, the planning component 734 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this description, the route can be a series of waypoints for travel between the two locations. As non-limiting examples, the waypoints include roads, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 734 can generate instructions for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 734 can determine how to guide the autonomous vehicle from a first waypoint in the series of waypoints to a second waypoint in the series of waypoints. In some examples, the instructions can be a trajectory or a portion of a trajectory. In some examples, multiple trajectories can be generated substantially simultaneously (e.g., within technical tolerances) in accordance with the receding horizon technique, and one of the multiple trajectories is selected for travel of vehicle 702.

[0047] In some examples, the planning component 734 can generate one or more trajectories for the vehicle 702 based at least in part on an estimated depth of objects in the environment determined from environmental attributes as discussed herein. In some examples, the planning component 734 can evaluate one or more trajectories for the vehicle 702 using temporal logic, such as linear temporal logic and / or signal temporal logic.

[0048] In at least one example, the vehicle computing device 704 can include one or more system controllers 726, which can be configured to control the steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 702. These system controllers 726 can communicate with and / or control corresponding systems of the drive system 714 and / or other components of the vehicle 702.

[0049] As can be appreciated, the components described herein (e.g., localization component 720, perception component 722, one or more maps 724, one or more system controllers 726, depth determination component 728, attribute identifier component 730, position determination component 732, and planning component 734) have been described separately for purposes of explanation. However, the operations performed by the various components may be combined with or performed by any other component. As an example, functions described in connection with the depth determination component 728, the attribute identifier component 730, and / or the position determination component 732 may be performed by the perception component 722 to reduce the amount of data transferred by the system.

[0050] In at least one example, the sensor system 706 can include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, time of flight, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and the like. The sensor system 706 can include multiple instances of each of these or other types of sensors. For example, the lidar sensors can include individual lidar sensors located at corners, front, rear, sides, and / or top of the vehicle 702. As another example, the camera sensors can include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 702. The sensor system 706 can provide input to the vehicle computing device 704. Additionally or alternatively, the sensor system 706 can transmit sensor data over one or more networks 736 to one or more computing devices at a particular frequency in near real time after a predetermined period of time has elapsed.

[0051] The vehicle 702 may also include one or more emitters 708 that emit light and / or sound, as described above. The emitters 708 in this example include internal audio and visual emitters for communicating with passengers of the vehicle 702. By way of example, and not by way of limitation, the internal emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 708 in this example also include external emitters. By way of example, and not by way of limitation, the external emitters in this example include lights for signaling a direction of travel or other indicators of the operation of the vehicle (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with one or more pedestrians or other nearby vehicles with acoustic beam steering technology.

[0052] The vehicle 702 may also include one or more communication connections 710 that enable communication between the vehicle 702 and one or more other local or remote computing devices. For example, the communication connections 710 may facilitate communication with other local computing devices on the vehicle 702 and / or the drive system 714. The communication connections 710 may also enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). The communication connections 710 may also enable the vehicle 702 to communicate with remotely operated computing devices or other remote services.

[0053] The communications connection 710 may include physical and / or logical interfaces for connecting the vehicle computing device 704 to another computing device or network, such as the network 736. For example, the communications connection 710 may include Wi-Fi based communications, such as over frequencies defined by the IEEE 802.11 standard, short range radio frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4 G. 4 G LTE, 5 G), or any suitable wired or wireless communication protocol that allows each computing device to interface with other computing devices.

[0054] In at least one example, the vehicle 702 can include one or more drive systems 714. In some examples, the vehicle 702 can have a single drive system 714. In at least one example, when the vehicle 702 has multiple drive systems 714, individual drive systems 714 can be located at both ends of the vehicle 702 (e.g., the front and rear, etc.). In at least one example, the drive system 714 can include one or more sensor systems for detecting conditions surrounding the drive system 714 and / or the vehicle 702. By way of example and not limitation, the sensor systems can include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive module, cameras or other imaging sensors, ultrasonic sensors for acoustically detecting objects surrounding the drive system, LIDAR sensors, radar sensors, etc. Some sensors, such as wheel encoders, can be specific to the drive system 714. In some cases, the sensor systems on the drive system 714 may overlap or complement corresponding systems on the vehicle 702 (eg, sensor system 706).

[0055] The drive system 714 can include many vehicle systems including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for braking force distribution to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the vehicle's exterior environment), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, and other electrical components). Additionally, the drive system 714 can include a drive system controller that can receive and pre-process data from the sensor systems and control the operation of various vehicle systems. In some examples, the drive system controller can include one or more processors and a memory communicatively coupled to the one or more processors. The memory can store one or more components that perform various functions of the drive system 714. Additionally, drive system 714 also includes one or more communications connections that enable each drive system to communicate with one or more other local or remote computing devices.

[0056] In at least one example, the direct connection 712 can provide a physical interface for coupling one or more drive systems 714 with the body of the vehicle 702. For example, the direct connection 712 can enable the transfer of energy, fluid, air, data, etc. between the drive systems 714 and the vehicle. In some examples, the direct connection 712 can further removably secure the drive systems 714 to the body of the vehicle 702.

[0057] In at least one example, the localization component 720, the perception component 722, the one or more maps 724, the one or more system controllers 726, the depth determination component 728, the attribute identifier component 730, the position determination component 732, and the planning component 734 can process the sensor data as described above and transmit their respective outputs to one or more computing devices 738 over one or more networks 736. In at least one example, the localization component 720, the perception component 722, the one or more maps 724, the one or more system controllers 726, the depth determination component 728, the attribute identifier component 730, the position determination component 732, and the planning component 734 can transmit their respective outputs to the computing device 738 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0058] In some examples, the vehicle 702 can transmit sensor data to one or more computing devices 738 via the network 736. In some examples, the vehicle 702 can transmit raw sensor data to the computing device 738. In other examples, the vehicle 702 can transmit processed sensor data and / or representations of the sensor data to the computing device 738. In some examples, the vehicle 702 can transmit sensor data at a particular frequency, after a predetermined period of time, in near real-time, etc. In some cases, the vehicle 702 can transmit sensor data (raw or processed) to the computing device 738 as one or more log files.

[0059] The computing device 738 may include a processor 740 and a memory 742 that stores a training component 744 .

[0060] In some examples, the training component 744 can include functionality to train one or more models to detect object and / or environmental attributes, determine (classify or regress) object and / or environmental attributes, determine a location of an object relative to the environmental attributes, etc. In some examples, the training component 744 can communicate information generated by the one or more models to the vehicle computing device 704 to modify how to control the vehicle 702 in response to various conditions.

[0061] For example, some or all aspects of the components described herein may include any models, algorithms, and / or machine learning algorithms. For example, in some examples, the components in memory 742 (and memory 718 described above) may be implemented as neural networks. In some examples, the training component 744 may utilize a neural network to generate and / or execute one or more models to improve various aspects of object trajectory estimation for use in trajectory planning of the vehicle 702.

[0062] As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer of a neural network may also contain another neural network, or may contain any number of layers (whether convolutional or not). As can be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on trained parameters.

[0063] Although described in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, but not limited to, machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scarplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomy 3 (ID3), chi-squared automated interaction detection (CHAID), decision stump, conditional decision tree)), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average-one dependent estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-median, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backbone, etc.), and the like. These can include methods such as linear propagation, Hopfield networks, and radial basis function networks (RBFNs), deep learning algorithms (e.g., deep Boltzmann machines (DBMs), deep brain networks (DBNs), convolutional neural networks (CNNs), and stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), and flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machines (GBMs), gradient boosting regression trees (GBRTs), and random forests), support vector machines, and methods such as supervised learning, unsupervised learning, and semi-supervised learning.

[0064] Additional examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

[0065] The processor 716 of the vehicle 702 and the processor 740 of the computing device 738 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processors 716 and 740 may comprise one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or part of a device that processes electronic data and converts it into registers or other electronic data that can be stored in memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0066] Memory 718 and 742 are examples of non-transitory computer readable media. Memory 718 and memory 742 can store an operating system and one or more software applications, instructions, programs, and / or data that implement the methods and functions attributed to the various systems described herein. In various implementations, memory can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, of which those illustrated in the accompanying drawings are merely examples relevant to the description herein.

[0067] 7 is shown as a distributed system, it should be noted that in alternative examples, components of the vehicle 702 may be associated with the computing device 738 and / or components of the computing device 738 may be associated with the vehicle 702. That is, the vehicle 702 may perform one or more of the functions associated with the computing device 738, and vice versa. Additionally, aspects of the depth determination component 728 and / or the planning component 734 may be performed on any of the devices described herein.

[0068] 8 illustrates an exemplary process 800 for identifying environmental attributes using a machine learning model, determining the location of the environmental attributes from map data, and refining the depth of an object from a vehicle using the location of the environmental attributes, according to an example of this disclosure. For example, some or all of process 800 may be performed by one or more components in FIG. 7 as described herein. For example, some or all of process 800 may be performed by vehicle computing device 704, computing device 738, or any other computing device or combination of computing devices. Additionally, any of the operations described in exemplary process 800 may be performed in parallel, in a different order than shown in process 800, omitted, in combination with other processes, etc.

[0069] At operation 802, the process may include capturing an image showing objects in the environment. For example, the image may be captured by a sensor, such as a camera, on a vehicle traveling through the environment.

[0070] At operation 804, the process may include determining a first depth of the object from the vehicle. In some examples, the first depth may be determined by inputting the image into a neural network trained to predict depths of surfaces shown at individual pixels in the image, and receiving a predicted depth at the location of the object from the neural network.

[0071] At operation 806, the process may include associating at least a portion of the image with environmental attributes. In at least some examples, such association may be performed by inputting at least a portion of the image into a machine learning model trained to determine environmental attributes. For example, the environmental attributes may be objects such as sidewalks, crosswalks, lanes, bike lanes, and others, as described and further contemplated herein. In some examples, the environmental attributes may generally be persistent objects in the environment.

[0072] At operation 808, the process may include receiving, from the machine learning model, environmental attributes associated with the object depicted in the image. For example, the environmental attributes may be one or more environmental attributes that the machine learning classifier is trained to detect. In some cases, the machine learning model may also output a location of the object relative to the environmental attribute, such as indicating that the object is above, below, next to, in front of, and / or behind the environmental attribute, to name a few. The relative location of the object relative to the environmental attribute may be represented as a vector, coordinates, or other mechanism that conveys relative location between entities when output by the machine learning model.

[0073] At operation 810, a determination is made as to whether an environmental attribute is included in the map data. In some cases, the environmental attribute may be newly added to the environment, changed in some way, occluded, or otherwise inappropriate to use to determine the depth of the object. In such cases, the process may proceed to operation 812, where the first depth estimate is used to determine the depth of the object, or another environmental attribute is selected. For example, if new lane markers have been painted on a road that has not yet been added to the map data, the process may select another environmental attribute, such as a sidewalk, that is included in the map data to refine the first depth estimate.

[0074] Otherwise, at operation 814, the process may include determining a probability of the object's location in the environment based at least in part on the location of the environmental attribute in the map data. In some examples, the probability of the object's location may be based on a multi-modal distribution of the probability of the object's location determined by the first depth estimate combined with a probability of the object's location determined by the object's location relative to the environmental attribute.

[0075] At operation 816, the process may include determining a second depth of the object from the vehicle based at least in part on the first depth and the probability of the object's location. For example, the highest mode of the multimodal distribution may be selected as the second depth of the object, refined from the first depth of the object. In some examples, the second depth of the object may be used to determine an uncertainty associated with the first depth of the object, such as by comparing the relative position of the object to environmental attributes, as described herein. The refined depth of the object may be used to control the vehicle to navigate the environment to avoid collisions with the object based on the refined depth.

[0076] [Example of invention content] A: A system comprising: one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors, the instructions, when executed, causing the system to perform operations including receiving an image from a sensor of an autonomous vehicle showing an object in an environment and determining a first depth of the object; inputting at least a portion of the image associated with the object to the machine learning model, the machine learning model being trained to output an association between one or more environmental attributes and the object depicted in the image input to the machine learning model; receiving an output from the machine learning model indicative of the environmental attributes associated with the object; determining a second depth associated with a position of the object in the environment based at least in part on the output and map data associated with the environmental attributes; and determining an estimated depth of the object based at least in part on the first depth and the second depth associated with the position of the object in the environment.

[0077] B: The system of paragraph A, wherein the operation further includes controlling the autonomous vehicle to navigate the environment based at least in part on the estimated depth of the object.

[0078] C: The system of claim A, wherein the first depth is based at least in part on a first distribution and the second depth is based at least in part on a second distribution.

[0079] D: The system of paragraph C, wherein the operations further include combining the first distribution and the second distribution into a multi-modal distribution associated with a depth of the object, and mapping the multi-modal distribution associated with the depth to a three-dimensional position.

[0080] E: A method comprising: receiving an image showing an object in an environment; determining a first distance to the object based at least in part on the image; inputting at least a portion of the image associated with the object into a machine learning model; receiving output from the machine learning model indicative of environmental attributes associated with the object; determining a second distance to the object based at least in part on the output and map data associated with the environmental attributes; and determining a refined distance to the object based at least in part on the first distance and the second distance.

[0081] F: The method of paragraph E, wherein the first distance of the object is based at least in part on a monocular depth estimate from the image.

[0082] G: The method of paragraph E, wherein determining the first distance to the object includes inputting an image to a neural network trained to predict a depth of a surface of the environment from an image, and receiving a predicted depth of the surface corresponding to pixels of the image, wherein the first distance is based at least in part on one of the predicted depths associated with the object shown in the image.

[0083] H: The method of paragraph E, further comprising determining a position of the object relative to the environmental attribute, wherein determining the second distance is further based on the position of the object relative to the environmental attribute.

[0084] I: The method of paragraph E, wherein the object is a first vehicle and the machine learning model includes a classifier trained to classify environmental attributes into one or more of: a first lane that is an oncoming lane, a second lane shared by the first vehicle and a second vehicle, a third lane traveling in the same direction as the second vehicle, a parking lot, or a roadway.

[0085] J: The method of paragraph E, wherein the object is a pedestrian and the machine learning model includes a classifier trained to classify environmental attributes into one or more of a first sidewalk parallel to a direction of travel of the vehicle, a second sidewalk perpendicular to the direction of travel of the vehicle, a corner, a crosswalk, or a lane.

[0086] K: The method of paragraph E, wherein the object is a motorcycle and the machine learning model includes a classifier trained to classify environmental attributes into one or more of a bike lane, a corner, a sidewalk, or a lane.

[0087] L: One or more computer-readable media having stored thereon instructions that, when executed by one or more processors, perform operations including: determining a first depth of an object represented in an image captured by a sensor of a vehicle; inputting at least a portion of the image associated with the object into a machine learning model; receiving an output from the machine learning model indicative of an environmental attribute associated with the object; determining a second depth of the object based at least in part on the output indicative of the environmental attribute received from the machine learning model and map data associated with the environmental attribute; and determining a refined depth of the object based at least in part on the first depth and the second depth.

[0088] M: The one or more computer-readable media of paragraph L, wherein the first depth is associated with a first distribution and the second depth is associated with a second distribution.

[0089] N: the operations combine the first distribution and the second distribution into a multi-modal distribution; and map the multi-modal distribution to a three-dimensional position; The one or more computer-readable media of paragraph M, further comprising:

[0090] O: The one or more computer-readable media of paragraph M, wherein the operations further include combining the first distribution and the second distribution into a multimodal distribution and determining a maximum likelihood depth estimate based at least in part on the multimodal distribution.

[0091] P: The one or more computer-readable media of paragraph L, wherein the first depth of the object is based at least in part on a monocular depth estimate from the image.

[0092] Q: The one or more computer-readable media described in paragraph P, wherein determining the first depth of the object includes inputting an image to a neural network trained to predict a depth of a surface of the environment from the image, and receiving a predicted depth of the surface corresponding to pixels of the image, wherein the monocular depth estimation is based at least in part on a predicted depth of one of the predicted depths associated with the object shown in the image.

[0093] R: The one or more computer-readable media described in paragraph L, wherein the operation further includes determining a position of the object relative to the environmental attribute, and determining an inaccuracy associated with the first depth of the object is further based on the position of the object relative to the environmental attribute.

[0094] S: The one or more computer-readable media of paragraph L, wherein the machine learning model includes a classifier trained to classify environmental attributes into one or more of: a sidewalk, a bike lane, a corner, a lane, or a crosswalk.

[0095] T: The one or more computer-readable media described in paragraph L, wherein the operation further includes controlling the vehicle to navigate an environment based at least in part on the refined depth of the object.

[0096] Although the above example subject matter has been described with respect to one particular implementation, it should be understood in the context of this document that the example subject matter may also be implemented via methods, devices, systems and / or computer-readable media, and / or other implementations. Additionally, any of the examples A-T may be implemented alone or in combination with one or more of the other examples A-T.

[0097] [Conclusion] One or more examples of the technology described herein have been described; however, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0098] In the exemplary description, reference is made to the accompanying drawings, which form a part hereof, which show by way of illustration specific examples of the claimed subject matter. It is to be understood that other examples can be used and modifications or substitutions, such as structural changes, can be made. Such examples, modifications or substitutions do not necessarily depart from the intended scope of the claimed subject matter. Although individual examples are described herein as having particular features or components, the features and components of the individual examples can be combined and used together. Although the operations herein can be presented in a particular order, in some cases the order can be changed, such as providing certain inputs at different times or in a different order, without changing the functionality of the described systems and methods. The disclosed procedures can also be performed in different orders. Furthermore, the various calculations herein need not be performed in the order disclosed, and other examples can be readily implemented using alternative orders of calculations. In addition to reordering, calculations can also be decomposed into sub-calculations that produce the same results.

Claims

1. one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors, the instructions, when executed, causing the system to: determining a first depth of an object represented in an image captured by a sensor of the vehicle; inputting at least a portion of the image associated with the object into a machine learning model; receiving an output from the machine learning model indicative of environmental attributes proximate to the object; determining a second depth of the object based at least in part on the output indicative of the environmental attribute received from the machine learning model and map data associated with the environmental attribute; determining a refined depth of the object based at least in part on the first depth and the second depth; Causes a system to perform operations including

2. The system of claim 1 , wherein the manipulation further comprises controlling the vehicle to navigate an environment based at least in part on the refined depth of the object.

3. The system of claim 1 , wherein the first depth is based at least in part on a first distribution and the second depth is based at least in part on a second distribution.

4. the operation combining the first distribution and the second distribution into a multi-modal distribution associated with a depth of the object; mapping the depth-associated multi-modal distribution to a three-dimensional position; The system of claim 3 further comprising:

5. The operation is combining the first distribution and the second distribution into a multimodal distribution; determining a maximum likelihood depth estimate based at least in part on the multimodal distribution; and The system of claim 3 or claim 4, further comprising:

6. The system of claim 1 , wherein the first depth of the object is based at least in part on a monocular depth estimate from the image.

7. 7. The system of claim 1, wherein the operation further includes determining a position of the object relative to the environmental attribute, and determining the position of the object relative to the environmental attribute is based on determining an inaccuracy associated with the first depth of the object.

8. receiving an image showing objects in an environment; determining a first distance to the object based at least in part on the image; inputting at least a portion of the image associated with the object into a machine learning model; receiving an output from the machine learning model indicative of environmental attributes proximate to the object; determining a second distance to the object based at least in part on the output and map data associated with the environmental attribute; and determining a refined distance of the object based at least in part on the first distance and the second distance; A method comprising:

9. The method of claim 8 , wherein the first distance of the object is based at least in part on a monocular depth estimate from the image.

10. Determining the first distance to the object includes: inputting the images into a neural network trained to predict surface depths of the environment from the images; receiving a predicted depth of the surface corresponding to pixels of the image; Including, the first distance is based at least in part on a predicted depth of one of the predicted depths associated with the object shown in the image. The method according to claim 8 or claim 9.

11. The method of claim 8 , further comprising determining a position of the object relative to the environmental attribute, and wherein determining the second distance is further based on the position of the object relative to the environmental attribute.

12. The object is a first vehicle, and the machine learning model is configured to estimate environmental attributes. a first lane which is an oncoming lane for a second vehicle for which the distance to the object is to be determined; a second lane shared by the second vehicle and the first vehicle; a third lane traveling in the same direction as the second vehicle and different from the second lane; Parking lot, or roadway, 12. The method of claim 8, comprising a classifier trained to classify into one or more of:

13. The object is a pedestrian, and the machine learning model is a first sidewalk parallel to the direction of travel of the vehicle; a second walkway perpendicular to the direction of travel of the vehicle; corner, Crosswalk, or Lane, 13. The method of claim 8, comprising a classifier trained to classify into one or more of:

14. The object is a motorcycle, and the machine learning model is Bike lanes, corner, Sidewalk, or Lane, 14. The method of claim 8, comprising a classifier trained to classify into one or more of:

15. A computer program comprising instructions which, when executed on a computer, performs the method according to any one of claims 8 to 14.

Images