Depth Data Model Training with Upsampling, Loss, and Loss Balancing

Abstract

Techniques for training an ML model to determine depth data based on image data are described herein. The training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to an ML model, which can output predicted disparity and / or depth data. The predicted disparity data can be used with second image data (e.g., right image) to reconstruct the first image. The difference between the first and reconstructed images can be used to determine a loss. The loss can include pixel, smoothing, structural similarity, and / or consistency loss. Furthermore, the difference between the depth data and the predicted depth data and / or the difference between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on various losses. Thus, techniques can train the ML model using self-supervised training and supervised training.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This patent application claims priority to U.S. utility patent application entitled “DEPTH DATA MODEL TRAINING WITH UPSAMPLING, LOSSES, AND LOSS BALANCING” having Serial No. 16 / 684,568, filed November 14, 2019, and claims priority to U.S. utility patent application entitled “DEPTH DATA MODEL TRAINING” having Serial No. 16 / 684,554, filed November 14, 2019. Application Serial Nos. 16 / 684,568 and 16 / 684,554 are incorporated by reference herein in their entireties. [Background technology]

[0002] A vehicle can use sensors to capture sensor data and detect objects in an environment. Accurate and precise sensor data can, for example, assist an autonomous vehicle in traversing an environment. In some examples, sensors may have a limited range and / or provide low density data associated with the environment. [Brief description of the drawings]

[0003] The detailed description will be set forth with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or features.

[0004] [Figure 1] FIG. 1 is an illustrative flow diagram of an exemplary process for training a machine learning model based on stereo image data, depth data, and various losses. [Diagram 2] FIG. 2 is a schematic flow diagram of an example process for training a machine learning model based on determining a loss based on a difference between image data and reconstructed image data. [Diagram 3]FIG. 3 is an illustrative flow diagram of an exemplary process for training a machine learning model based on losses associated with stereo image data. [Figure 4] FIG. 4 is an illustrative flow diagram of an example process for training a machine learning model based on losses associated with self-supervised outputs and / or supervised outputs. [Diagram 5] FIG. 5 is a block diagram of an example computer system for implementing the techniques described herein. [Figure 6] FIG. 6 is an exemplary process for training a machine learning model to generate image-based depth data. [Figure 7] FIG. 7 is another exemplary process for training a machine learning model to generate image-based depth data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The present disclosure describes a system, method, and apparatus for training a machine learning model to determine depth data based on image data. The training can be based on stereo image data and depth data (e.g., lidar data). When fully trained, the machine learning model can receive image data (e.g., a single image) and output predicted depth information associated with pixels in the image. The machine learning model can be trained using stereo images based on scene differences between a left image and a right image of a stereo image pair. For example, a first image (e.g., a left image) can be input to a machine learning model that can output predicted depth data. The predicted depth data can be used in combination with a second image data (e.g., a right image) to reconstruct the first image. The difference between the first image and the reconstructed first image can be used to determine a loss. Examples of losses include, but are not limited to, pixel loss, smoothing loss, structural similarity loss, and / or consistency loss. Furthermore, the difference between the depth data (e.g., representing a ground truth depth) and the predicted depth data can be determined, and the machine learning model can be trained based on the losses described herein. Thus, the technique can train machine learning models using self-supervised training (e.g., based on stereo images) and supervised training (e.g., based on depth data).

[0006] The machine learning model may be trained based at least in part on various losses, including but not limited to pixel loss, smoothing loss, structural similarity loss, consistency loss (e.g., left / right consistency loss, depth / disparity loss, etc.), etc. Details of the various losses are described throughout this disclosure.

[0007] In some examples, the techniques described herein may include determining pixel loss. For example, the operation may include determining a difference between a pixel in the left image data and a corresponding pixel in the reconstructed left image data. In some examples, the pixel loss may represent a difference in intensity values ​​(e.g., R, G, B values) between corresponding pixels. As can be appreciated, the operation may include determining a difference between a pixel in the right image data and a corresponding pixel in the reconstructed right image data (e.g., pixels representing the same portion of the environment). In some examples, if the pixel loss is above a threshold, this may indicate that the pixel in the image data does not correspond to a pixel in the reconstructed image data (e.g., due to occlusion), in which case the pixel loss associated with the particular pixel may be ignored for training purposes.

[0008] In some examples, the techniques described herein may include determining a smoothing loss. For example, the operations may include performing averaging of patches of the image before, after, or in combination with determining the loss described herein. In some examples, the smoothing loss may be de-weighted based on edges detected in the image data. In some examples, the smoothing loss may be evaluated for a single image.

[0009] In some examples, the techniques described herein can include determining a structural similarity loss. For example, the operations can include comparing patches or regions of the first image and the reconstructed first image. Operations can be performed on various patches, such as determining the average RGB value of the patches and comparing data from the first patch (e.g., from the first image) with data from the second patch (e.g., from the reconstructed first image). In some examples, the structural similarity loss can include comparing the location and / or presence of edges or discontinuities in the corresponding first and second patches. Thus, the structural similarity loss can determine information about the general layout or structure of the respective images, where the differences between the patches can represent the loss.

[0010] In some examples, the techniques described herein can include determining a consistency loss. For example, the techniques described herein can be used to determine a first reconstructed image (e.g., a reconstructed left image based on the right image data and the first predicted depth information) and a second reconstructed image (e.g., a reconstructed right image based on the left image data and the second predicted depth information). A consistency loss (e.g., a left / right consistency loss) can be determined based on a difference between the first reconstructed image and the second reconstructed image.

[0011] In some examples, the consistency loss can enforce consistency between data from different output heads of the machine learning model. For example, a first output head of the machine learning model can output disparity data, predicted depth data, and / or inverse predicted depth data based on a self-supervised technique described herein (e.g., based on an image-based supervision). In some examples, a second output head of the machine learning model can output depth data based on a fully supervised technique described herein (e.g., based on a lidar-based (or depth database) supervision). The consistency loss can enforce consistency between two or more outputs such that the depth information (or disparity information) output by the first output head is consistent with the depth information output by the second output head. The difference between the outputs can be used as a loss to minimize the difference between the outputs when the machine learning model is trained.

[0012] In some examples, the structural similarity loss and / or consistency loss may be based at least in part on classification information associated with pixels or patches of data, e.g., structural similarity and / or consistency may be enforced only between patches or pixels having the same classification, or other segmentation information.

[0013] In some cases, the machine learning model can output various predicted depth data. For example, the machine learning model can output depth data, inverse depth data, disparity data, etc. In some examples, the depth data and / or inverse depth data can represent measurements between an origin (e.g., a position associated with an image sensor or depth sensor) and an object in the environment (which in some examples may be determined along a ray from the origin that passes through pixel coordinates). In some examples, the disparity data can represent a horizontal shift of a pixel (or data element) between a first image (e.g., a left image) and a second image (e.g., a right image). In some examples, the disparity data can be related to the depth data by intrinsic and / or extrinsic information associated with a stereoscopic image sensor.

[0014] In some examples, the machine learning model can determine the depth data and the disparity data separately. As described above, and in some examples, the difference between the depth data and the disparity data can be determined, which can represent a consistency loss (e.g., a depth / disparity consistency loss) to enforce consistency between various predicted data.

[0015] In some examples, the output of the machine learning model can represent a discrete output or can represent a continuous output value. For example, the machine learning model can determine discrete depth portions / bins associated with the image data. Such bins can be linear or non-linear in either depth, inverse depth, disparity, or any other representation of depth. In some examples, the machine learning model can output continuous depth values ​​as a continuous output (e.g., the machine learning model can output a depth value of 10.65 meters without performing such a binning operation). Thus, the continuous offset can provide a gradual transition between depth values ​​regardless of whether discrete depth bins are used. In at least some examples, a combination of binning and offsetting can be used (e.g., the model can output a "coarse" measurement of the bin in addition to a fine offset (as may be measured from the "center" of the bin). In some examples, the machine learning algorithm can use a loss function and / or a softmax loss associated with the depth bin to determine the continuous offset.

[0016] In some examples, aspects of the machine learning model may include determining a feature map at a first resolution or level of detail, then upsampling the feature map to determine different information at various levels. In some examples, the feature map may correspond to an intermediate output of a portion of the machine learning model, such as an encoder or decoder portion of the machine learning model. In some examples, the techniques may include upsampling one or more feature maps based on local plane guidance, which may include predicting a unit vector associated with a direction and / or magnitude for upsampling. In some examples, the local plane guidance may be an example of a shape-based upsampling technique (e.g., upsampling using a sphere, a plane, or other shape) as opposed to a bilinear upsampling technique.

[0017] In some examples, any of the losses may represent L1 loss (mean absolute error), L2 loss (mean squared error), likelihood loss, log loss, cross entropy loss, hinge loss, Huber loss, etc.

[0018] In some examples, the techniques described herein can use loss balancing to combine, aggregate, or otherwise determine losses for training a machine learning model. For example, various losses described herein, such as pixel loss, smoothing loss, structural similarity loss, left / right consistency loss, depth / disparity consistency loss, etc., can be combined using loss balancing techniques to determine a "final" loss for training (e.g., backpropagation through the machine learning model). In some examples, the losses can be combined or aggregated based at least in part on one or more weights. In some examples, the weights can be learned as an aspect of the training operation. In some examples, such learning can be based on homoscedasticity. In some examples, homoscedasticity can determine one or more weights associated with the losses based at least in part on uncertainties associated with the various losses.

[0019] The machine learning model can be trained using the training image data and the training lidar data as ground truth for training the machine learning model. Thus, the techniques described herein may include unsupervised training, self-supervised training, semi-supervised training, fully supervised training, or any combination thereof. Additional examples of training machine learning models can be found, for example, in U.S. Patent Application No. 15 / 803,682, entitled "Dense Depth Estimation of Image Data," filed November 03, 2017, which is incorporated herein by reference in its entirety. In some examples, the training image data can include data representing data at a higher resolution or at a higher density compared to the training lidar data. For purposes of illustration only, the training image data can include data from three channels (e.g., red, green, blue) each having millions of pixels, while the training lidar data corresponding to the training image data can include on the order of hundreds of thousands of points, or less. Thus, based on the difference between the amount of training image data and the corresponding training lidar data, it can be understood that a particular pixel of the image data may not have a corresponding lidar measurement. In some examples, the operations described herein can provide depth data (e.g., single depth data) corresponding to the image data such that some or all of the individual pixels of the training image data can be associated with depth data.

[0020] In some examples, the machine learning algorithm can be trained using additional channels of data including, for example, three channels corresponding to RGB data, one channel corresponding to a binary indication (e.g., a binary channel) indicating whether lidar (or other depth data) is available for a particular pixel, and / or one channel that can correspond to a depth measurement associated with a particular pixel. In some examples, the depth measurement can be considered as a ground truth, where the machine learning model being trained can determine the depth data to minimize the difference between the ground truth and the generated depth data. As can be appreciated, any number of channels and / or types of data can be used to train the machine learning model and as input to the deployed machine learning algorithm. In some examples, the binary channel can be omitted.

[0021] A trained machine learning model can be sent or otherwise implemented to generate depth data from data from a single image sensor. That is, the trained machine learning model can receive image data as input and output depth data associated with some or all pixels of the image data. In some examples, the depth data can be stored for subsequent processing. For example, some applications or systems of autonomous vehicles can use depth data for localization, perception (e.g., detection, identification, segmentation, classification, tracking, etc. of objects in an environment), relative depth data generation, etc. As can be appreciated, these applications are examples, and such examples and uses of depth data or measured depth data are not intended to be limiting. Additional examples of determining and using image-based depth data can be found, for example, in U.S. Patent Application No. 16 / 408,407, entitled "Depth from Vision," filed May 9, 2019, which is incorporated herein by reference in its entirety.

[0022] In some examples, image-based segmentation techniques can be used to remove dynamic objects to improve training operations. For example, image-based segmentation techniques can be used to identify and remove / discard data associated with dynamic objects represented in the image data, such that the remaining data can represent static objects in the environment. For example, depth data associated with dynamic objects (e.g., vehicles, pedestrians, bicyclists, animals, debris, etc.) can be filtered out of the image data so that disparity data predicted by the machine learning model can be based on the stereo image sensor configuration rather than the movement of objects in the environment. Lidar-based segmentation techniques can also be used to identify and remove / discard data associated with dynamic objects, for example, for purposes of training machine learning models.

[0023] As introduced above, the vehicle can use the depth data generated by the machine learning model to perform operations, including perception operations, such as to detect, identify, segment, classify, and / or track objects in the environment, among other operations. In some examples, the depth data can be used to generate three-dimensional bounding boxes (or otherwise masks) associated with the objects.

[0024] In some examples, the techniques described herein can be implemented in a system that includes an image sensor (e.g., a red-green (RGB) camera, an intensity camera (grayscale), an infrared camera, an ultraviolet camera, etc.), a depth camera (e.g., an RGB-D camera), a time-of-flight sensor, a lidar sensor, a radar sensor, a sonar sensor, etc., to provide redundancy to the system in the event of a hardware or software failure. For example, if a depth camera, a lidar sensor, and / or a radar sensor is occluded or malfunctioning, the techniques described herein can be used with the image sensor to provide redundancy and / or backup to ensure that dense depth information is available under many circumstances. Thus, the techniques described herein can provide further improvements to, for example, machine vision systems.

[0025] The depth data generation techniques described herein can improve the functionality of a computing device by providing additional depth data for performing subsequent actions to control an autonomous vehicle (or other system). For example, the depth data associated with image data can enable subsequent processes such as localization, perception (e.g., detection, identification, segmentation, classification, tracking, etc.), route planning, trajectory generation, etc. to be performed more accurately, may require less processing power, and / or may require less memory. For example, in some examples, faster and / or more accurate segmentation can be used in generating a trajectory for an autonomous vehicle, which can improve the safety of occupants of the autonomous vehicle. Additionally, in some examples, the techniques described herein can be used to verify sensor calibration, provide error checking or voting to determine if a sensor measurement is inaccurate (e.g., by comparing the depth measurement to another depth sensor), and / or be used as a fallback if another sensor is occluded or invalid. In some examples, training a machine learning model using such self-supervised and supervised techniques (which together may include semi-supervised training) provides a machine learning model that may output more accurate depth estimates than models trained without these techniques. These and other improvements to the capabilities of the computer are described herein.

[0026] The techniques described herein can be implemented in many ways. Exemplary embodiments are provided below with reference to the following drawings. Although described in the context of an autonomous vehicle, the methods, apparatus, and systems described herein can be applied to a variety of systems (e.g., sensor systems, or robotic platforms) and are not limited to autonomous vehicles. In one example, similar techniques can be utilized in a vehicle under the control of a driver, and such a system can provide an indication of whether it is safe to perform various operations. In another example, the techniques can be utilized in a manufacturing assembly line context, an aerial surveying context, or a marine context. Furthermore, the techniques described herein can be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0027] FIG. 1 is an illustrative flow diagram 100 of an exemplary process for training a machine learning model based on stereo image data, depth data, and various losses.

[0028] The operations may include capturing and / or receiving image data from a stereo image sensor 102. For example, the stereo image sensor 102 may capture a right image 104 and a left image 106 of an environment. As illustrated in the right image 104 and the left image 106, the image data may represent an environment that the autonomous vehicle is traversing.

[0029] The operations may further include capturing and / or receiving depth data from a lidar sensor 108 (also referred to as lidar sensor 108). For example, the lidar sensor 108 may capture depth data 110 representing at least a portion of the environment represented by the right image 104 and / or the left image 106. In some examples, the lidar sensor 108 may be aligned and / or calibrated in combination with the stereo image sensor 102 such that the sensors capture the same portion of the environment. In some examples, the depth data may be captured by a time-of-flight sensor in addition to or instead of the depth data captured by the lidar sensor.

[0030] As mentioned above, the right image 104 and / or the left image 106 can represent millions of RGB pixels representing the environment. The stereoscopic image sensor 102 can include a left image sensor and a right image sensor (e.g., as shown in FIG. 2 ) in which the sensors capture at least partially overlapping fields of view. In some examples, the lidar sensor 108 can capture at least a portion of the environment corresponding to at least a portion of the field of view associated with the left image sensor or the right image sensor. In some examples, image data can be captured by various image sensors associated with different orientations. That is, the techniques described herein are not limited to stereoscopic image sensors and can include image sensors having multi-view geometries that share some common visibility.

[0031] Image data corresponding to the right image 104 and the left image 106, as well as the depth data 110, may be input to the machine learning component 112 to train the machine learning component 112. As described herein, the machine learning component 112 may be trained to ultimately output depth data based on the input image data. In some examples, the machine learning component 112 may be trained at least in part based on various components 114. In some examples, such components 114 may represent various loss components and / or architecture components, as described herein.

[0032] For example, components 114 may include, without limitation, one or more of a pixel-wise loss (also referred to as pixel loss), a smoothing loss component, a structural similarity loss component, a consistency loss component (e.g., left / right consistency, depth / disparity consistency, edge-based consistency, segmentation-based consistency, etc.), a scaling component, an ordinal regression component, an L2 loss component, a loss balancing component, a local planar guidance component, a homoscedasticity component, etc. Additional details of the foregoing components are described above and throughout this disclosure.

[0033] Once the machine learning component 112 is trained, it can be used to determine depth data for an environment based on image data (e.g., image data from a single image sensor). Such depth data can be used in connection with an autonomous vehicle traversing an environment. In some examples, the depth data can be stored for subsequent processing. For example, some applications or systems of an autonomous vehicle can use the depth data for localization, perception (e.g., detection, identification, segmentation, classification, tracking, etc. of objects in an environment), relative depth data generation, etc. As can be appreciated, these applications are examples and such examples and the use of depth data or measured depth data are not intended to be limiting.

[0034] FIG. 2 is a schematic flow diagram 200 of an example process for training a machine learning model based on determining a loss based on a difference between image data and reconstructed image data.

[0035] The operations may include capturing and / or receiving image data captured by a stereo image sensor as described herein. For example, a first image sensor 202 (e.g., a left image sensor of a stereo image sensor pair) may capture a left image 106 of an environment. A second image sensor 204 (e.g., a right image sensor of a stereo image pair) may capture a right image 104 of an environment. The left image 106 may represent a field of view 206 associated with the first image sensor 202, while the right image 104 may represent a field of view 208 associated with the second image sensor 204. At least a portion of the fields of view may overlap, as illustrated as overlap portion 210. As can be appreciated, at least a portion of the fields of view 206 and 208 may not overlap.

[0036] The left image 106 may then be input to a machine learning component 212, which may correspond to the machine learning component 112 of Figure 1. Although Figure 2 shows the left image 106 being input to the machine learning component 212, operations may include inputting the right image 104 to the machine learning component in series or in parallel with the input of the left image 106, as described in connection with Figure 3, as well as throughout this disclosure.

[0037] In some examples, the operations may include rectifying the left image 106 and the right image 104 such that corresponding points in the left image 106 and the right image 104 are in the same row of pixels. In some examples, rectifying the image data may simplify operations for determining disparity and / or warping operations.

[0038] The machine learning component 212 can determine predicted inverse depth data 214 based at least in part on the left image 106. In some examples, the machine learning component 212 can output depth data, disparity data, etc. In some examples, the predicted inverse depth data 214 can represent inverse depth values ​​for computational efficiency.

[0039] The operations may include inputting the predicted inverse depth data 214 and the right image 104 to an inverse warping component 216. The inverse warping component 216 may include functionality to warp the right image 104 based on the predicted inverse depth data 214, as well as based on internal camera parameters (e.g., lens distortion, focal length, etc.) and / or external camera parameters (e.g., pose (e.g., orientation) of individual image sensors and stereo image sensors, etc.). For example, given the predicted inverse depth data 214, the right image 104, and camera specific information, the inverse warping component 216 may determine the disparity between points and use the disparity to determine a reconstructed left image 218.

[0040] The reconstructed left image 218 and the left image 106 can be input to a loss component 220 to determine differences or losses between the left image 106 and the reconstructed left image 218. In some examples, areas where the right image 104 does not overlap with the left image 106 can result in null data for the reconstructed left image 218. Such null data can be represented as regions 222, which can be downweighted or excluded when determining the loss for training.

[0041] As described herein, the loss component 220 can determine various losses for training the machine learning component 212. For example, the loss component can determine a pixel loss, a smoothing loss, a structural similarity loss, a consistency loss, etc.

[0042] In some examples, the operations may include generating a reconstructed right image based on the left image data and generating predicted inverse depth data (or predicted disparity data) based on the left image data (or vice versa).

[0043] 3 is a schematic flow diagram 300 of an example process for training a machine learning model based on losses associated with stereo image data. The aspects of FIG. 3 may be performed by one or more components described in connection with FIG. 1 and FIG. 2, which may be omitted for clarity.

[0044] For example, the operations may include capturing or receiving a left image 106 and determining predicted inverse depth data 302 (e.g., based on the left image 106). Additionally, the operations may include capturing or receiving a right image 104 and determining predicted inverse depth data 304 (e.g., based on the right image 104). Determining the predicted depth data may be performed in series or in parallel, within technical tolerances.

[0045] The operations may further include determining a reconstructed left image 306 based on the predicted inverse depth data 302 and the right image 104, as described above in Figure 2. Additionally, the operations may include determining a reconstructed right image 308 based on the predicted inverse depth data 304 and the left image 106, as described herein.

[0046] One or more losses (e.g., loss 310) can be determined based at least in part on the reconstructed left image 306 and the left image 106. In some examples, the loss 310 can represent one or more of a pixel loss, a smoothing loss, a structural similarity loss, a consistency loss (e.g., left / right consistency, edge-based consistency, segmentation-based consistency, etc.). In some examples, the consistency loss (e.g., left / right consistency loss) can represent a total or otherwise aggregated difference between pixels in the image data and the reconstructed image data. In some examples, the consistency loss can correspond to a difference between the reconstructed images (e.g., the reconstructed left image 306 and the reconstructed right image 308).

[0047] Similarly, one or more losses (e.g., loss 312) can be determined based at least in part on the reconstructed right image 308 and the right image 104. In some examples, a category / type of loss 312 can substantially correspond to loss 310.

[0048] Losses 310 and 312 can be combined, summed, or otherwise aggregated to determine loss 314. A machine learning model as described herein can be trained based at least in part on loss 314.

[0049] FIG. 4 is an illustrative flow diagram 400 of an example process for training a machine learning model based on losses associated with self-supervised outputs and supervised outputs.

[0050] As described above, the training operations may include capturing and / or receiving image data captured by a stereo image sensor (e.g., represented as right image 104 and left image 106). The training operations may further include capturing and / or receiving depth data 110 (e.g., captured by one or more lidar and / or time-of-flight sensors). The right image 104, the left image 106, and the depth data 110 may be input to the machine learning component 402.

[0051] In some examples, the machine learning component 402 can include a neural network, a convolutional neural network, a recurrent neural network, etc. In some examples, the machine learning component 402 can correspond to the machine learning components 112 and 212 described in Figures 1 and 2, respectively.

[0052] The machine learning component 402 can determine one or more feature maps based at least in part on the image data (e.g., the right image 104 and / or the left image 106) and / or the depth data 110. In some examples, the machine learning component 402 can output the one or more feature maps to the upsampling component 404. In some examples, the upsampling component 404 may be a component separate from the machine learning component 402 or may be incorporated into the machine learning component 402. In some examples, the upsampling component 404 may represent or be part of a decoder portion of a machine learning model.

[0053] In some examples, the upsampling component may upsample one or more feature maps to different resolution levels based at least in part on the local plane guidance, in some examples, the local plane guidance may include predicting unit vectors associated with the orientation and / or magnitude of individual pixels or features for upsampling.

[0054] The machine learning component 402 and / or the upsampling component 404 can determine outputs 406, which may include self-supervised outputs 408 (e.g., inverse depth map, disparity, depth, etc.) and / or supervised outputs 410 (e.g., depth, etc.).

[0055] In some examples, the self-supervised output 408 may be based at least in part on losses determined in association with the reconstructed image data, as described herein.

[0056] In some examples, the teacher output 410 may be based at least in part on comparing the predicted depth data to the depth data 110 as ground truth data.

[0057] In some examples, the machine learning component 402 may include separate output heads of a neural network to output a self-supervised output 408 and a supervised output 410. The output 406 may be input to a loss component 412 to determine various losses based on the data.

[0058] For example, the loss component 412 can determine a consistency loss and / or loss balancing. For example, the consistency loss can determine the difference between the self-supervised output 408 (e.g., represented as disparity data) and the teacher output 410 (e.g., represented as depth data). Such a consistency loss can be referred to as a depth / disparity consistency loss.

[0059] In some examples, the consistency loss can be one-sided (e.g., using the teacher output as a ground truth for determining a first loss associated with the self-supervised output, or using the self-supervised output as a ground truth for determining a second loss associated with the teacher output), or the consistency loss can be two-sided (e.g., using the teacher output as a ground truth for determining a first loss associated with the self-supervised output, using the self-supervised output as a ground truth for determining a second loss associated with the teacher output, and determining a third loss based on the first loss and the second loss). In some examples, the consistency loss can be an edge-based consistency loss (e.g., edge-based consistency between an image and a reconstructed image) or a segmentation-based consistency loss (e.g., consistency of instance segmentation between classifications of data or between data).

[0060] In some examples, losses associated with supervised training (e.g., based on lidar data representing ground truth) can be based at least in part on ordinal regression. In some examples, losses based on ordinal regression can be proportional to or related to the difference between the error and the ground truth value.

[0061] The loss component 412 may further include functionality for loss balancing. For example, the operations may include determining various losses, such as pixel loss, smoothing loss, structural similarity loss, consistency loss, etc. Such losses may be summed or aggregated to determine an aggregated loss to backpropagate through the machine learning component 402 for training. In some examples, the losses may be summed according to weights and / or uncertainties associated with each loss. In some examples, the weights may be learned as part of the training process. In some examples, the weights associated with the losses may be determined based at least in part on homoscedasticity.

[0062] 5 illustrates a block diagram of an example system 500 for implementing the techniques described herein. In at least one example, the system 500 can include a vehicle 502. In the illustrated example system 500, the vehicle 502 is an autonomous vehicle, although the vehicle 502 can be any other type of vehicle.

[0063] The vehicle 502 may be a driverless vehicle, such as an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions of an entire journey without the expectation of a driver (or passenger) controlling the vehicle at all times. In such an example, the vehicle 502 may be configured to control all functions from the beginning to the completion of a journey, including all parking functions, and may therefore not include a driver and / or controls for driving the vehicle 502, such as a steering wheel, accelerator pedal, and / or brake pedal. This is merely an example, and the systems and methods described herein may be incorporated into any land, air, or water vehicle, including vehicles that require manual control at all times by a driver, to those that are partially or fully autonomously controlled.

[0064] The vehicle 502 may include one or more computing devices 504, one or more sensor systems 506, one or more emitters 508, one or more communication connections 510 (also referred to as communication devices and / or modems), at least one direct connection 512 (e.g., for physically coupling with the vehicle 502 to exchange data and / or provide power), and one or more drive systems 514. The one or more sensor systems 506 may be configured to capture sensor data associated with the environment.

[0065] The one or more sensor systems 506 can include time-of-flight sensors, position sensors (GPS, compass, etc.), inertial sensors (inertial measurement unit (IMU), accelerometer, magnetometer, gyroscope, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (RGB, IR, intensity, depth, etc.), microphone sensors, environmental sensors (temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc. The one or more sensor systems 506 can include multiple instances of each of these or other types of sensors. For example, a time-of-flight sensor can include individual time-of-flight sensors positioned at the corners, front, rear, sides, and / or top of the vehicle 502. As another example, a camera sensor can include multiple cameras positioned at various locations on the exterior and / or interior of the vehicle 502. The one or more sensor systems 506 can provide input to the computing device 504.

[0066] The vehicle 502 may also include one or more emitters 508 for emitting light and / or sound. The one or more emitters 508 in this example include interior audio and visual emitters for communicating with passengers of the vehicle 502. By way of example and not limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The one or more emitters 508 in this example also include exterior emitters. By way of example and not limitation, the exterior emitters in this example include lights for signaling direction of travel or other indicators of vehicle operation (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with one or more pedestrians or other nearby vehicles, which may include acoustic beam steering technology.

[0067] The vehicle 502 may also include one or more communication connections 510 that enable communication between the vehicle 502 and one or more other local or remote computing devices (e.g., a remote teleoperated computing device) or a remote service. For example, the one or more communication connections 510 may facilitate communication with other local computing devices on the vehicle 502 and / or one or more drive systems 514. The one or more communication connections 510 may also enable the vehicle 502 to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.).

[0068] The one or more communications connections 510 may include physical and / or logical interfaces for connecting the computing device 504 to another computing device or to one or more external networks 534 (e.g., the Internet). For example, the one or more communications connections 510 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), satellite communications, dedicated short-range communications (DSRC), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices.

[0069] In at least one example, the vehicle 502 can include one or more drive systems 514. In some examples, the vehicle 502 can have a single drive system 514. In at least one example, when the vehicle 502 has multiple drive systems 514, each drive system 514 can be located at either end of the vehicle 502 (e.g., the front and rear, etc.). In at least one example, the drive system 514 can include one or more sensor systems 506 that detect conditions around the drive system 514 and / or the vehicle 502. By way of example and not limitation, the sensor systems 506 can include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive system, cameras or other imaging sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. for acoustically detecting objects around the drive system. Some sensors, such as wheel encoders, can be specific to the drive system 514. In some cases, the sensor systems 506 on the drive system 514 may overlap or complement corresponding systems (e.g., sensor system 506) of the vehicle 502.

[0070] The drive system 514 can include many vehicle systems including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use in other vehicle systems, a steering system (which can be electric) including a steering motor and a steering rack, a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for braking force distribution to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the vehicle's exterior environment), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 514 can include a drive system controller that can receive and pre-process data from the sensor system 506 and control the operation of various vehicle systems. In some examples, the drive system controller can include one or more processors and a memory communicatively coupled to the one or more processors. The memory can store one or more components that perform various functions of the drive system 514. Additionally, drive system 514 also includes one or more communications connections that enable the respective drive system to communicate with one or more other local or remote computing devices.

[0071] The computing device 504 may include one or more processors 516 and a memory 518 communicatively coupled to the one or more processors 516. In the depicted example, the memory 518 of the computing device 504 stores a localization component 520, a perception component 522 with a machine learning component 524, a prediction component 526, a planning component 528, a map component 530, and one or more system controllers 532. While shown as residing within the memory 518 for illustrative purposes, it is contemplated that the localization component 520, the perception component 522, the machine learning component 524, the prediction component 526, the planning component 528, the map component 530, and the one or more system controllers 532 may additionally or alternatively be accessible to the computing device 504 (e.g., stored in a different component of the vehicle 502) and / or accessible to the vehicle 502 (e.g., stored remotely).

[0072] In the memory 518 of the computing device 504, the localization component 520 can include functionality to receive data from the sensor system 506 to determine a position of the vehicle 502. For example, the localization component 520 can include and / or request / receive a three-dimensional map of the environment and can continually determine the location of the autonomous vehicle within the map. In some examples, the localization component 520 can use SLAM (Simultaneous Localization and Mapping) or CLAMS (Calibration, Simultaneous Localization and Mapping) to receive time-of-flight data, image data, lidar data, radar data, sonar data, IMU data, GPS data, wheel encoder data, or any combination thereof, etc. to precisely determine the location of the autonomous vehicle. In some examples, the localization component 520 can provide data to various components of the vehicle 502 to establish an initial position of the autonomous vehicle for generating a trajectory, as described herein.

[0073] As described above, the localization component 520 LearningThe depth data generated by the localization component 524 can be used to perform the operations described above to determine the location of the vehicle 502. The depth data can provide a local map for comparison to a three-dimensional map (e.g., a mesh). In some examples, the localization component 520 can provide functionality to determine errors associated with the local map, the three-dimensional map, and / or one or more sensor systems 506. For example, the localization component 520 can determine a position error (e.g., drift error) associated with the vehicle 502. Over time during operation, the errors can accumulate, resulting in errors in the positioning and / or trajectory data. In some examples, the localization component 520 can determine the error based, for example, on the position error meeting or exceeding a threshold value. In some examples, the localization component 520 can determine a calibration adjustment associated with the one or more sensor systems 506 based on the position error.

[0074] By way of example only, the localization component 520 can determine a location of the vehicle based on GPS data from one or more sensor systems 506. However, a comparison of a local map can indicate that the vehicle is in a different location than the location indicated by the GPS data. Thus, the localization component 520 can indicate that an error exists in the GPS sensor and / or the local map.

[0075] In some examples, the localization component 520 can determine an update associated with the three-dimensional map. For illustrative purposes only, one or more sensor systems 506 can provide location data to the localization component 520. In addition, the localization component 520 can determine a location based on a comparison of the local map and the three-dimensional map. However, the comparison may indicate that one or more points of the local map do not correspond to the three-dimensional map. This may indicate that the three-dimensional map is out of date. The localization component 520 can determine the difference between the local map and the three-dimensional map as an updated map and store the map update or provide the map update to a map data server via the network 534, for example.

[0076] The perception component 522 may include functionality for performing object detection, segmentation, and / or classification. In some examples, the perception component 522 may provide processed sensor data indicative of the presence of an entity proximate to the vehicle 502 and / or the classification of the entity as an entity type (e.g., car, pedestrian, bicyclist, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional and / or alternative examples, the perception component 522 may provide processed sensor data indicative of one or more characteristics associated with the detected entity and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity may include, but are not limited to, x-position (global location), y-position (global location), z-position (global location), orientation, entity type (e.g., classification), velocity of the entity, range (size) of the entity, etc. The characteristics associated with the environment may include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, time of day, day of the week, season, weather conditions, dark / light indication, etc.

[0077] As described above, the perception component 522 can use perception algorithms to determine a perception-based bounding box associated with an object in the environment based on the sensor data. For example, the perception component 522 can receive image data and classify the image data to determine that an object is represented in the image data. Then, using a detection algorithm, the perception component 522 can generate a two-dimensional bounding box and / or a perception-based three-dimensional bounding box associated with the object. The perception component 522 can further generate a three-dimensional bounding box associated with the object. As described above, the three-dimensional bounding box can provide additional information, such as a position, orientation, pose, and / or size (e.g., length, width, height, etc.) associated with the object.

[0078] The perception component 522 may include functionality for storing the perception data generated by the perception component 522. In some examples, the perception component 522 may determine a track corresponding to an object that has been classified as an object type. For illustrative purposes only, the perception component 522 using the sensor system 506 may capture one or more images of an environment. The sensor system 506 may capture an image of the environment including an object, such as a pedestrian. The pedestrian may be in a first location at time T and in a second location at time T+t (e.g., movement during a span of time t after time T). In other words, the pedestrian may move from a first location to a second location during this period. Such movement may be logged, for example, as stored perception data associated with the object.

[0079] The stored sensory data may include, in some examples, fused sensory data captured by the vehicle. The fused sensory data may include a fusion or other combination of sensor data from sensor systems 506, such as image sensors, lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, global positioning system sensors, interior sensors, and / or any combination thereof. The stored sensory data may additionally or alternatively include classification data including a semantic classification of an object (e.g., pedestrian, vehicle, building, road surface, etc.) represented in the sensor data. The stored sensory data may additionally or alternatively include track data (position, orientation, sensor features, etc.) corresponding to the movement of an object classified as a dynamic object through the environment. The track data may include multiple tracks of multiple different objects over time. This track data may be mined to identify images of a particular type of object (e.g., pedestrian, animal, etc.) when the object is stationary (e.g., stationary) or moving (e.g., walking, running, etc.). In this example, the computing device determines a track corresponding to a pedestrian.

[0080] The perception component 522 can include a machine learning component 524, which can be trained according to techniques described herein to determine depth data based on image data. In some examples, the machine learning component 524 can include functionality to receive image data (e.g., a single image frame) and can output depth data associated with each pixel in the image data. In some cases, the machine learning component 524 can function as a backup component in the event that one or more other sensor components are non-functioning, and in some examples, the machine learning component 524 can function as a primary source of depth data for the localization component 520, the perception component 522, the prediction component 526, and / or the planning component 528, as described herein.

[0081] The prediction component 526 can generate one or more probability maps that represent predicted probabilities of possible locations of one or more objects in the environment. For example, the prediction component 526 can generate one or more probability maps for vehicles, pedestrians, animals, etc. within a threshold distance from the vehicle 502. In some examples, the prediction component 526 can measure the tracks of the objects and generate discretized predicted probability maps, heat maps, probability distributions, discretized probability distributions, and / or trajectories for the objects based on the observed and predicted behavior. In some examples, the one or more probability maps can represent the intent of one or more objects in the environment.

[0082] The planning component 528 can determine a path for the vehicle 502 to follow to traverse through an environment. For example, the planning component 528 can determine various routes and paths as well as various levels of detail. In some examples, the planning component 528 can determine a route for traveling from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this description, the route can be a series of waypoints for traveling between the two locations. By way of non-limiting example, the waypoints include roads, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 528 can generate instructions for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 528 can determine how to guide the autonomous vehicle from a first waypoint in the series of waypoints to a second waypoint in the series of waypoints. In some examples, the instructions can be a path or a portion of a path. In some examples, multiple paths can be generated substantially simultaneously (i.e., within engineering tolerances) according to a receding horizon technique. A single path of the multiple paths within the reversal data range that has the highest confidence level may be selected for operating the vehicle.

[0083] In other examples, the planning component 528 can alternatively or additionally use data from the perception component 522 and / or the prediction component 526 to determine a path for the vehicle 502 to follow to traverse through the environment. For example, the planning component 528 can receive data from the perception component 522 and / or the prediction component 526 regarding objects associated with the environment. Using this data, the planning component 528 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location) that avoids objects in the environment. In at least some examples, such a planning component 528 can determine that there is no such collision-free path and then provide a path that leads the vehicle 502 to a safe stop that avoids all collisions and / or otherwise mitigates damage.

[0084] The memory 518 may further include one or more maps 530 that may be used by the vehicle 502 to navigate within the environment. For purposes of this description, a map may be any number of data structures modeled in two, three, or N dimensions that may provide information about the environment, such as, but not limited to, topology (such as intersections), streets, mountain ranges, roads, terrain, and the general environment. In some examples, the map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectance information (e.g., specular reflectance information, retroreflectance information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some examples, the map can be stored in a tiled format, such that individual tiles of the map represent separate portions of the environment, and can be loaded into the working memory as needed, as described herein. In at least one example, the one or more maps 530 can include at least one map (e.g., an image and / or a mesh). In some examples, the vehicle 502 can be controlled at least in part based on the map 530. That is, the map 530 can be coupled to the localization component 520, the perception component 522, the machine learning component 524, the prediction component 526, the localization component 520, the perception component 522, the machine learning component 524, the prediction component 526, the localization component 520, the perception component 522, the machine learning component 524, the prediction component 526, the localization component 520, the perception component 522, the machine learning component 524, the prediction component 526, the localization component 520, the perception component 522, the localization component 520 ... 5 26, and / or in conjunction with the planning component 528 to determine a position of the vehicle 502, identify objects within the environment, generate predicted probabilities associated with the objects and / or the vehicle 502, and / or generate a path and / or trajectory for navigating within the environment.

[0085] In some examples, one or more maps 530 may be stored on a remote computing device (such as computing device 536) accessible via network 534. In some examples, multiple maps 530 may be stored, for example, based on a characteristic (e.g., type of entity, time of day, day of the week, season, etc.). Storing multiple maps 530 may have similar memory requirements, but may increase the speed at which data in the maps can be accessed.

[0086] In at least one example, computing device 504 can include one or more system controllers 532, which can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 502. These system controllers 532 can communicate with and / or control corresponding systems of drive system 514 and / or other components of vehicle 502, which can be configured to operate according to the path provided from planning component 528.

[0087] The vehicle 502 can be connected to a computing device 536 via a network 534 and can include one or more processors 538 and a memory 540 communicatively coupled to the one or more processors 538. In at least one example, the one or more processors 538 can be similar to the processor 516 and the memory 540 can be similar to the memory 518. In the illustrated example, the memory 540 of the computing device 536 stores a training component 542 that includes an image data component 544, a depth data component 546, a machine learning component 548, an unwarping component 550, an upsampling component 552, and a loss component 554. In at least one example, the machine learning component 548 can be similar to the machine learning component 524 after training. While shown for purposes of illustration as residing in memory 540, it is contemplated that the training component 542, the image data component 544, the depth data component 546, the machine learning component 548, the unwarping component 550, the upsampling component 552, and the loss component 554 may additionally or alternatively be accessible to the computing device 536 (e.g., stored in a different component of the computing device 536) and / or be accessible to the computing device 536 (e.g., stored remotely).

[0088] As described herein, the training component 542 can train the machine learning component 548 to generate depth data based on the image data. The training component 542 can determine training data as an input to the machine learning component 548. For example, the training data can include sensor data, such as training image data captured by a vehicle. In some examples, the image data can be corrected image data and / or synthetic image data. Additionally, the training data can include sensor data, such as lidar data, time-of-flight data, and / or radar data, as ground truth data.

[0089] In some examples, the image data component 544 can include functionality for receiving image data captured by a stereo image sensor. In some examples, the image data component 544 can preprocess the image data for training, such as by removing distortion and / or rectifying images relative to one another. Examples of distortions may include lens distortion, color correction, etc. An example of correcting image data may include aligning the image data such that pixels in the first image data are in the same row as corresponding pixels in the second image data.

[0090] In some examples, the depth data component 546 can include functionality to receive depth data captured by one or more lidar sensors, time-of-flight sensors, radar sensors, etc. In some examples, the depth data component 546 can align or register the depth data with respect to the image data such that corresponding points of the depth data are associated with particular pixels in the left image data and / or right image data of the stereo image data.

[0091] The machine learning component 548 can use the training data provided by the training component 542 (e.g., provided by the image data component 544 and / or the depth data component 546) to train a machine learning model to determine depth data associated with the training image data. Once the machine learning component 548 is trained, the machine learning component 548 can be deployed within the vehicle 502 as the (trained) machine learning component 524.

[0092] In some examples, the unwarping component 550 can include functionality to generate or otherwise determine a reconstructed first image based on the estimated depth data (e.g., output by the machine learning component) and based on the second image data. For example, the unwarping component 550 can shift pixels from the second image along a horizontal scan line based at least in part on the predicted depth data to generate a reconstructed first image.

[0093] In some examples, the upsampling component 552 may include functionality for upsampling one or more feature maps output by the machine learning component. In some examples, the upsampling component 552 may be based on local plane guidance (LPG). In some examples, the upsampling component 552 may be included in the decoder portion of the machine learning model or may be a separate component.

[0094] In some examples, the loss component 554 may include functionality for determining self-supervised and / or supervised losses, as described herein. The losses may include, but are not limited to, pixel loss, smoothing loss, structural similarity loss, consistency loss (e.g., left / right consistency loss, depth / disparity consistency loss, edge-based consistency loss, segmentation-based consistency loss, etc.). In some examples, any of the losses may represent L1 loss (mean absolute error), L2 loss (mean squared error), likelihood loss, log loss, cross-entropy loss, hinge loss, Huber loss, etc. Details of the various losses are discussed throughout this disclosure.

[0095] The processor 516 of the computing device 504 and the processor 538 of the computing device 536 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processors 516 and 538 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into registers or other electronic data that can be stored in a memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0096] The memory 518 of the computing device 504 and the memory 540 of the computing device 536 are examples of non-transitory computer-readable media. The memory 518 and 540 can store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functions attributed to the various systems described herein. In various implementations, the memory 518 and 540 can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, and those illustrated in the accompanying drawings are merely examples relevant to the description herein.

[0097] In some examples, some or all aspects of the components described herein may include any model, algorithm, and / or machine learning algorithm. For example, in some instances, the components in memories 518 and 540 may be implemented as neural networks.

[0098] As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer of a neural network may also contain another neural network, or may contain any number of layers (whether convolutional or not). As can be understood in the context of this disclosure, a neural network may utilize machine learning, which may refer to a broad class of algorithms in which an output is generated based on learned parameters.

[0099] Although described in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, machine learning or machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS), instance-based algorithms (e.g., ridge regression, least absolute value shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomy 3 (ID3), chi-squared automated interaction detection (CHAID), decision stump, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, polynomial naive Bayes, average-one dependence estimator (AODE), Bayesian confidence network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backprocessing, etc.), and so on. The following algorithms may be used to train the neural network: permutation, Hopfield network, radial basis function network (RBFN), deep learning algorithms (e.g., deep Boltzmann machine (DBM), deep confidence network (DBN), convolutional neural network (CNN), stacked autoencoder), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), adaboost, hierarchical generalization (blending), gradient boosting machine (GBM), gradient boosted regression tree (GBRT), random forest), support vector machine (SVM), supervised learning, unsupervised learning, semi-supervised learning, and the like.

[0100] Additional examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

[0101] Thus, the techniques described herein provide a robust implementation for determining depth data of an environment based on image data to enable safe operation of an autonomous vehicle.

[0102] 6 and 7 show exemplary processes according to examples of the present disclosure. These processes are illustrated as logical flow graphs, with each operation representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable recording media that, when executed on one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular abstract data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations can be combined in any order and / or in parallel to perform the process.

[0103] 6 is an example process 600 for training a machine learning model to generate image-based depth data. For example, some or all of the process 600 may be performed by one or more components of FIG. 5, as described herein. For example, some or all of the process 600 may be performed by the vehicle computing device 504.

[0104] At operation 602, the process may include receiving stereo image data and depth data associated with the stereo image data. In some examples, operation 602 may include receiving stereo image data and depth data captured by one or more autonomous vehicles as the vehicles traverse an environment. As described above, in some examples, at least a portion of the environment is represented within both pairs of stereo images. Additionally, in some examples, depth data may be associated with at least a portion of the stereo image data to provide a ground truth depth value. As described above, in some examples, the image data may be captured by any multi-view system, including co-vision, and is not limited to a stereo image sensor. In some examples, the depth data may be captured by one or more lidar sensors and / or one or more time-of-flight sensors.

[0105] At operation 604, the process may include inputting the left image data into a machine learning (ML) model. In some examples, the machine learning model may be configured to output predicted depth information associated with the image data. In some examples, weights associated with the ML model may be initialized with random values ​​at a first time and adjusted via backpropagation of losses at one or more second times after the first time, as described herein.

[0106] At operation 606, the process may include receiving predicted depth information associated with the left image data from the ML model. As described above, the predicted depth information may include, but is not limited to, a depth value (e.g., in meters), an inverse depth value (e.g., 1 / meter), or disparity data (e.g., measuring the horizontal displacement of a pixel from one image to another, where the pixel represents the same object or point in both images). In some examples, the ML model may output predicted depth information for some or all pixels of the image data.

[0107] At operation 608, the process may include determining reconstructed left image data based at least in part on the predicted depth information (from operation 606) and the right image data. In some examples, operation 608 may include warping the right image based on the predicted depth information to determine the reconstructed left image data. In some examples, the warping may be performed algorithmically or by a machine learning model.

[0108] At operation 610, the process may include determining a first difference between the left image data and the reconstructed left image data. In some examples, operation 610 may include determining one or more losses, including but not limited to a pixel loss, a smoothing loss (which may be based on a single image, for example), a structural similarity loss, and / or a consistency loss (subject to additional data described herein).

[0109] As can be appreciated, operations 604, 606, 608, and 610 are described in the context of left image data as input. Process 600 can include similar operations for right image data.

[0110] Operation 612 may include inputting the right image data into a machine learning (ML) model. do. Operation 612 may be similar to operation 604 described above, and may be optional, depending on the embodiment.

[0111] Operation 614 may include receiving, from the ML model, predicted depth information associated with the right image data. do. Operation 614 may be similar to operation 606 described above, and may be optional, depending on the embodiment.

[0112] In operation 616, the process may include determining reconstructed right image data based at least in part on the predicted depth information (from operation 614) and the left image data. Operation 616 may be similar to operation 608 described above, and may be optional, depending on the embodiment.

[0113] In operation 618, the process may include determining a second difference between the right image data and the reconstructed right image data. Operation 618 may be similar to operation 610 described above, and may be optional, depending on the embodiment.

[0114] At operation 620, the process may include determining a third difference between the depth data (e.g., the lidar data and / or the time-of-flight data received in operation 602) and the predicted depth information. In some examples, the predicted depth information may be associated with the left image data (e.g., from operation 606) or the predicted depth information may be associated with the right image data (e.g., from operation 614).

[0115] At operation 622, the process may include determining a loss based at least in part on the first difference (and / or the second difference) and the third difference. In some examples, at least one of the first difference, the second difference, or the third difference includes an L2 loss. In some examples, operation 622 may include summing, aggregating, or otherwise determining a loss based on the determined differences based on an embodiment.

[0116] At operation 624, the process may include training a machine learning model based at least in part on the loss to generate a trained machine learning model. For example, the loss may be backpropagated through the machine learning model to adjust one or more weights of various operating functions to minimize the overall loss.

[0117] In some examples, process 600 may further include transmitting the machine learning model as described herein to an autonomous vehicle (or other device or component) to determine the depth data based on the image data. Additionally, the operations may include determining a trajectory based at least in part on the image-based depth data and / or controlling the vehicle to follow the trajectory.

[0118] 7 is another example process 700 for training a machine learning model to generate image-based depth data. For example, some or all of the process 700 may be performed by one or more components of FIG. 5, as described herein. For example, some or all of the process 700 may be performed by the vehicle computing device 504.

[0119] At operation 702, the process may include receiving stereo image data and depth data associated with the stereo image data (e.g., lidar data and / or time-of-flight data). In some examples, operation 702 may substantially correspond to operation 602.

[0120] At operation 704, the process may include determining predicted depth information and predicted disparity information based at least in part on the first image data input to the machine learning model. In some examples, operation 704 may substantially correspond to and / or include aspects of operations 604, 606, 612, and / or 614. In some examples, operation 704 may be based at least in part on local plane guidance upsampling 706. For example, the machine learning model may include a decoder portion that upsamples one or more feature maps output by an encoder portion of the machine learning model. In some examples, the decoder may upsample the feature maps using local plane guidance such that the predicted depth information may be based at least in part on the local plane guidance upsampling.

[0121] At operation 708, the process may include determining reconstructed first image data based at least in part on the predicted disparity information and the second image data. In some examples, operation 708 may substantially correspond to and / or include aspects of operations 608 and / or 616.

[0122] In operation 710, the process may include determining a first difference between the first image data and the reconstructed first image data. In some examples, operation 710 may substantially correspond to and / or include aspects of operation 610 and / or operation 618.

[0123] At operation 712, the process may include determining a second difference between the depth data and the predicted depth information. In some examples, operation 712 may substantially correspond to and / or include aspects of operation 620.

[0124] In operation 714, the process may include determining whether to determine a consistency loss. If YES, the process continues to operation 716. If NO, the process continues to operation 718.

[0125] At operation 716, the process may include determining a third difference between the predicted depth information and the disparity information. In some examples, the third difference may represent a depth / disparity consistency loss, as described herein. For example, the machine learning model may include a first output head that outputs predicted depth information (and / or predicted inverse depth information or disparity information) based on self-supervised training (image-based training) and a second output head that outputs predicted depth information based on supervised training (lidar-based training).

[0126] In operation 718, the process may include determining whether to perform loss balancing to train the machine learning model. If YES, the process continues to operation 720. If NO, the process continues to operation 722.

[0127] At operation 720, the process can include determining weights associated with the differences. For example, operation 720 can include determining (e.g., learning) weights (e.g., scalar values) to weight the various differences (e.g., the first difference, the second difference, and / or the third difference). In some examples, operation 720 can be based at least in part on homoscedasticity and / or uncertainty associated with the various losses.

[0128] At operation 722, the process may include determining a loss based at least in part on the first difference and the second difference (and / or the third difference and / or weighting). In one example, at least one of the first difference, the second difference, and / or the third difference includes an L2 loss, represented as element 724.

[0129] At operation 726, the process may include training a machine learning model based at least in part on the loss to generate a trained machine learning model. In some examples, operation 726 may substantially correspond to and / or include aspects of operation 624.

[0130] [Example of invention content] A: A system including one or more processors and one or more non-transitory computer-readable media storing computer-executable instructions, the computer-executable instructions, when executed, causing the one or more processors to: train a machine learning model to determine depth information, the training including receiving image data captured by a stereo image sensor, the image data including left image data captured by a left image sensor and right image data captured by a right image sensor; receiving lidar data captured by a lidar sensor, the lidar data associated with a portion of the image data; and inputting the left image data into the machine learning model; receiving from the machine learning model predicted depth information associated with the left image data; determining reconstructed left image data based at least in part on the predicted depth information and the right image data; determining a first difference between the left image data and the reconstructed left image data; determining a second difference between at least a portion of the predicted depth information and the lidar data; determining a loss based at least in part on the first difference and the second difference; and training a machine learning model based at least in part on the loss to generate a machine learning model.

[0131] B: The system of paragraph A, wherein the operation further includes transmitting the machine learning model to an autonomous vehicle for controlling the autonomous vehicle.

[0132] C: The system of paragraph A or B, wherein the predicted depth information includes at least one of depth data, inverse depth data, or disparity data.

[0133] D: The system of any of paragraphs A-C, wherein the first difference includes at least one of a pixel loss representing a difference between a first intensity value associated with a first pixel of the left image data and a second intensity value associated with a second pixel in the reconstructed left image data, or a structural similarity loss associated with at least one of an edge or a discontinuity associated with the left image data and the reconstructed left image data.

[0134] E: The system described in any of paragraphs A to D, wherein the predicted depth information is associated with discrete depth values.

[0135] F: A method comprising: receiving first image data captured by a first image sensor including a first field of view; receiving second image data captured by a second image sensor including a second field of view, where at least a portion of the first field of view is associated with at least a portion of the second field of view; receiving depth data captured by a depth sensor, where the depth data is associated with at least a portion of the first image data or the second image data; inputting the first image data into a machine learning model; receiving from the machine learning model predicted depth information associated with the first image data; determining reconstructed first image data based at least in part on the predicted depth information and the second image data; determining a first difference between the first image data and the reconstructed first image data; determining a second difference between the predicted depth information and the depth data; determining a loss based at least in part on the first difference and the second difference; and adjusting parameters associated with the machine learning model based at least in part on the loss to generate a trained machine learning model.

[0136] G. The method of paragraph F, further comprising: transmitting the trained machine learning model to an autonomous vehicle for controlling the autonomous vehicle.

[0137] H: The method of paragraph F or G, wherein the predicted depth information includes at least one of depth data, inverse depth data, or disparity data.

[0138] I: The method of any of paragraphs F-H, wherein the first difference includes a pixel loss representing a difference between a first intensity value associated with a first pixel of the first image data and a second intensity value associated with a second pixel in the reconstructed first image data.

[0139] J: The method of any of paragraphs F-I, further comprising determining a third difference including a smoothing loss based at least in part on the reconstructed first image data, wherein a weighting associated with the smoothing loss is based at least in part on edges represented in the first image data or at least one of the reconstructed first image data.

[0140] K. The method of any of paragraphs F-J, wherein the first difference includes a structural similarity loss based at least in part on at least one of a mean or a covariance associated with a portion of the first image data.

[0141] L: The method of any of paragraphs F-K, wherein the predicted depth information is based at least in part on shape-based upsampling.

[0142] M: The method of any of paragraphs F-L, wherein determining the reconstructed first image data includes warping the second image data based at least in part on the predicted depth information.

[0143] N: The method of any of paragraphs F-M, wherein the predicted depth information is first predicted depth information, the method further including: inputting the second image data into the machine learning model; receiving from the machine learning model second predicted depth information associated with the second image data; determining reconstructed second image data based at least in part on the second predicted depth information and the first image data; determining a third difference between the second image data and the reconstructed second image data; and further determining the loss based at least in part on the third difference.

[0144] O: one or more non-transitory computer-readable media storing instructions executable by one or more processors, the instructions, when executed, causing the one or more processors to: receive first image data captured by a first image sensor of a stereoscopic image sensor; receive second image data captured by a second image sensor of the stereoscopic image sensor; receive depth data captured by a depth sensor, the depth data being associated with a portion of at least one of the first image data or the second image data; input the first image data to a machine learning model; and acquire from the machine learning model the first image data. determining a first difference between the first image data and the reconstructed first image data; determining a second difference between the predicted depth information and the depth data; determining a loss based at least in part on the first difference and the second difference; and adjusting parameters of the machine learning model based at least in part on the loss to generate a trained machine learning model.

[0145] P: The one or more non-transitory computer-readable media of paragraph O, wherein the operations further include transmitting the trained machine learning model to an autonomous vehicle for controlling the autonomous vehicle.

[0146] Q: The one or more non-transitory computer-readable media of paragraphs O or P, wherein the first difference includes at least one of a pixel loss, a smoothing loss, or a structural similarity loss.

[0147] R: One or more non-transitory computer-readable media of any of paragraphs O-Q, wherein the predicted depth information is associated with discrete depth values.

[0148] S: The one or more non-transitory computer-readable media of any of paragraphs O-R, wherein the predicted depth information is first predicted depth information, and the operations further include inputting the second image data to the machine learning model; receiving from the machine learning model second predicted depth information associated with the second image data; determining reconstructed second image data based at least in part on the second predicted depth information and the first image data; determining a third difference between the second image data and the reconstructed second image data; and further determining the loss based at least in part on the third difference.

[0149] T: One or more non-transitory computer-readable media described in any of paragraphs O-S, wherein the operation further includes receiving semantic information associated with an object represented in at least one of the first image data or the second image data, and the loss is based at least in part on the semantic information.

[0150] A system including one or more processors and one or more non-transitory computer-readable media storing computer-executable instructions, the computer-executable instructions, when executed, causing the one or more processors to: receive image data captured by a stereo image sensor, the image data including left image data captured by a left image sensor and right image data captured by a right image sensor; receive lidar data captured by a lidar sensor, the lidar data associated with a portion of the image data; input the left image data to a machine learning model; and receive predicted disparity information and predicted depth information associated with the left image data from the machine learning model. determining a first difference between the left image data and the reconstructed left image data; determining a second difference between the predicted depth information and the lidar data; determining a third difference between the predicted disparity information and the predicted depth information; determining a weight associated with at least one of the first difference, the second difference, or the third difference; determining a loss based at least in part on the first difference, the second difference, the third difference, and the weight; and training a machine learning model based at least in part on the loss to generate a trained machine learning model.

[0151] V: The system described in paragraph U, wherein the operations further include determining an intermediate output of the machine learning model based at least in part on shape-based upsampling, and at least one of the predicted disparity information or the predicted depth information is based at least in part on the intermediate output.

[0152] W: The system of paragraph U or V, wherein the third difference represents a consistency loss.

[0153] X: The system of any of paragraphs U-W, wherein the weights are learned weights based at least in part on uncertainty.

[0154] Y: The system of any of paragraphs U-X, wherein the operation further includes transmitting the trained machine learning model to an autonomous vehicle for controlling the autonomous vehicle.

[0155] Z: A method comprising: receiving first image data captured by a first image sensor including a first field of view; receiving second image data captured by a second image sensor including a second field of view, where at least a portion of the first field of view is associated with at least a portion of the second field of view; receiving depth data captured by a depth sensor, where the depth data is associated with at least a portion of the first image data or the second image data; inputting the first image data into a machine learning model; receiving from the machine learning model predicted disparity information and predicted depth information associated with the first image data; determining reconstructed first image data based at least in part on the predicted disparity information and the second image data; determining a first difference between the first image data and the reconstructed first image data; determining a second difference between the predicted depth information and the depth data; determining a loss based at least in part on the first difference and the second difference; and adjusting parameters of the machine learning model based at least in part on the loss to generate a trained machine learning model.

[0156] AA: The method of paragraph Z, further comprising transmitting the trained machine learning model to an autonomous vehicle for controlling the autonomous vehicle.

[0157] AB: The method of paragraph Z or AA, wherein the predicted depth information includes at least one of depth data, inverse depth data, or disparity data.

[0158] AC: The method of any of paragraphs Z-AB, further comprising determining a third difference between the predicted disparity information and the predicted depth information, and further determining the loss based at least in part on the third difference.

[0159] AD: The method of paragraph AC, wherein the third difference represents a consistency loss.

[0160] AE: The method of any of paragraphs Z-AD, further comprising determining a weight associated with at least one of the first difference or the second difference, and further determining the loss based at least in part on the weight.

[0161] AF: The method of paragraph AE, wherein the weights are learned weights based at least in part on uncertainty.

[0162] AG: The method of any of paragraphs Z through AF, wherein determining the reconstructed first image data includes warping the second image data based at least in part on the predicted disparity information.

[0163] AH: The method of any of paragraphs Z-AG, wherein the first difference includes at least one of a pixel loss or a structural similarity loss.

[0164] AI: one or more non-transitory computer readable media storing instructions executable by one or more processors, which instructions, when executed, cause the one or more processors to: receive first image data captured by a first image sensor including a first field of view; receive second image data captured by a second image sensor including a second field of view, where at least a portion of the first field of view is associated with at least a portion of the second field of view; receive depth data captured by a depth sensor, where the depth data is associated with at least a portion of the first image data or the second image data; input the first image data into the machine learning model; and obtain from the machine learning model predicted disparity information and predicted depth information associated with the first image data. determining a first difference between the first image data and the reconstructed first image data; determining a second difference between the predicted depth information and the depth data; determining a third difference between the predicted disparity information and the predicted depth information; determining a weight associated with at least one of the first difference, the second difference, or the third difference; determining a loss based at least in part on the first difference, the second difference, the third difference, and the weight; and adjusting parameters of the machine learning model based at least in part on the loss to generate a trained machine learning model.

[0165] AJ: The one or more non-transitory computer-readable media of paragraph AI, wherein the predicted depth information is based at least in part on shape-based upsampling.

[0166] AK: The one or more non-transitory computer-readable media of paragraphs AI or AJ, further including: AK: the weight is a first weight associated with the first difference, and the operation is determining a second weight associated with at least one of the second difference or the third difference, the second weight being different from the first weight; and further determining the loss based at least in part on the second weight.

[0167] AL: The one or more non-transitory computer-readable media of paragraph AK, wherein the first weight and the second weight are learned weights.

[0168] AM: One or more non-transitory computer-readable media described in any of paragraphs AI to AL, wherein the operation further includes receiving semantic information associated with an object represented in at least one of the first image data or the second image data, and the loss is based at least in part on the semantic information.

[0169] AN:The one or more non-transitory computer-readable media of paragraph AM, wherein the semantic information includes at least one of a static classification, a dynamic classification, or an object type classification.

[0170] While the exemplary subject matter described above is described with respect to one particular embodiment, it should be understood in the context of this specification that the exemplary subject matter can also be implemented via a method, device, system, computer-readable medium, and / or other embodiment. Furthermore, any of Examples A-AN can be implemented alone or in combination with any other one or more of Examples A-AN.

[0171] [Conclusion] One or more examples of the technology described herein have been described; however, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0172] In the illustrative description, reference is made to the accompanying drawings, which form a part hereof, showing by way of example specific examples of the claimed subject matter. It is to be understood that other examples may be used and modifications or substitutions, such as structural changes, may be made. Such examples, modifications or substitutions do not necessarily depart from the intended scope of the claimed subject matter. Although steps herein may be presented in a particular order, in some cases the order may be changed, such as providing certain inputs at different times or in a different order, without changing the functionality of the described systems and methods. The disclosed procedures may also be performed in different orders. Furthermore, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to reordering, calculations may also be decomposed into sub-calculations that produce the same results.

Claims

1. receiving first image data captured by a first image sensor including a first field of view; receiving second image data captured by a second image sensor including a second field of view, wherein at least a portion of the first field of view is associated with at least a portion of the second field of view; receiving depth data captured by a depth sensor, the depth data being associated with a portion of at least one of the first image data or the second image data; inputting the first image data into a machine learning model; receiving from the machine learning model predicted depth information associated with the first image data; determining reconstructed first image data based at least in part on the predicted depth information and the second image data; determining a first difference between the first image data and the reconstructed first image data, the first difference comprising a structural similarity loss based at least in part on edges or discontinuities associated with the first image data and the reconstructed first image data; determining a second difference between the predicted depth information and the depth data; determining a loss based at least in part on the first difference and the second difference; and adjusting parameters associated with the machine learning model based at least in part on the loss to generate a trained machine learning model; The method includes:

2. 10. The method of claim 1, further comprising: transmitting the trained machine learning model to an autonomous vehicle for controlling the autonomous vehicle.

3. The predicted depth information is Depth data, Inverse depth data, or The method of claim 1 or 2, comprising at least one of disparity data.

4. 4. The method of claim 1, wherein the first difference comprises a pixel loss representing a difference between a first intensity value associated with a first pixel of the first image data and a second intensity value associated with a second pixel in the reconstructed first image data.

5. 5. The method of claim 1, further comprising determining a third difference including a smoothing loss based at least in part on the reconstructed first image data, wherein a weighting associated with the smoothing loss is based at least in part on edges represented in the first image data or at least one of the reconstructed first image data.

6. The method of claim 1, wherein the structural similarity loss is further based at least in part on a mean or covariance associated with a portion of the first image data.

7. The method of claim 1 , wherein the predicted depth information is based at least in part on shape-based upsampling.

8. 8. The method of claim 1, wherein determining the reconstructed first image data comprises warping the second image data based at least in part on the predicted depth information.

9. The predicted depth information is a first predicted depth information, and the method includes: inputting the second image data into the machine learning model; receiving second predicted depth information associated with the second image data from the machine learning model; and determining reconstructed second image data based at least in part on the second predicted depth information and the first image data; determining a third difference between the second image data and the reconstructed second image data; further determining the loss based at least in part on the third difference; The method of claim 1 , further comprising:

10. A computer program comprising coded instructions which, when executed on a computer, performs the method according to any one of claims 1 to 9.

11. One or more non-transitory computer-readable media storing instructions executable by one or more processors, the instructions, when executed, causing the one or more processors to: receiving first image data captured by a first image sensor of a stereoscopic image sensor; receiving second image data captured by a second image sensor of the stereoscopic image sensor; receiving depth data captured by a depth sensor, the depth data being associated with a portion of at least one of the first image data or the second image data; inputting the first image data into a machine learning model; receiving from the machine learning model predicted depth information associated with the first image data; determining reconstructed first image data based at least in part on the predicted depth information and the second image data; determining a first difference between the first image data and the reconstructed first image data, the first difference comprising a structural similarity loss based at least in part on edges or discontinuities associated with the first image data and the reconstructed first image data; determining a second difference between the predicted depth information and the depth data; determining a loss based at least in part on the first difference and the second difference; and adjusting parameters of the machine learning model based at least in part on the loss to generate a trained machine learning model; One or more non-transitory computer-readable media for causing operations to be performed, including:

12. The first difference is Pixel loss, Smoothing loss, or The one or more non-transitory computer-readable media of claim 11 , comprising at least one of a structural similarity loss.

13. 13. The one or more non-transitory computer-readable media of claim 11 or 12, wherein the predicted depth information is associated with discrete depth values.

14. the predicted depth information is a first predicted depth information, and the operation comprises: inputting the second image data into the machine learning model; receiving second predicted depth information associated with the second image data from the machine learning model; and determining reconstructed second image data based at least in part on the second predicted depth information and the first image data; determining a third difference between the second image data and the reconstructed second image data; further determining the loss based at least in part on the third difference; 14. One or more non-transitory computer-readable media according to claim 11, further comprising:

15. The operation, receiving semantic information associated with the object represented in at least one of the first image data or the second image data; 15. The one or more non-transitory computer-readable media of claim 11, wherein the loss is based at least in part on the semantic information.

Images