Monocular depth and optical flow estimation using diffusion model

Abstract

An improved method for generating a depth map or an optical flow map from an input image via a noise diffusion iterative process is provided. Also provided are improved methods for training a machine learning model employed in an iterative process and for enhancing a training data set for training such a model. Improved performance regarding computational cost, training data requirements, model size, and output quality is obtained by transforming a depth or optical flow graph prediction process into a noise diffusion context. In addition, the noise diffusion context allows a trained model as described herein to de novo generate a graph from a target color image and / or start from an initial'guess' graph (e.g., a noisy graph, a hole-containing graph) when generating an improved output graph, thereby native incorporating imperfect priori information represented by such an initial graph.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 481,970 filed on January 27, 2023, U.S. Provisional Patent Application No. 63 / 502,485 filed on May 16, 2023, U.S. Provisional Patent Application No. 63 / 505,476 filed on June 1, 2023, and U.S. Provisional Patent Application No. 63 / 603,306 filed on November 28, 2023, the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] Depth maps representing distances to points in an environment from a reference location (e.g., to a lidar sensor, to a time-of-flight sensor, to a camera) can be beneficial in a variety of mapping, navigation, or other applications. For example, a robot can use such a depth map to navigate an environment and interact with objects therein. Current depth sensors are often expensive and / or generate low-quality depth information (e.g., noisy depth information, depth information containing holes or other artifacts). An image (e.g., a color image) of an environment contains information about the depths of objects therein and can be used to predict depth information and / or enhance depth information generated by a depth sensor associated with (e.g., collocated with and oriented parallel to) a camera used to generate such an image.

[0004] Optical flow maps represent the motion of corresponding pixels between a pair of images. Optical flow maps can be used in a variety of applications (e.g., video compression, object detection and / or avoidance, navigation, or other applications). Optical flow maps can be generated using heuristic algorithms and / or trained machine learning models. However, current methods for optical flow map estimation can exhibit various drawbacks, including high computational cost, difficulty predicting the motion of small features in an image, difficulty resolving ambiguous motion (e.g., the motion of multiple similar objects within a single scene), difficulty predicting the motion of poorly focused features in an image, and other drawbacks. Summary of the Invention

[0005] In a first aspect, a computer-implemented method is provided, comprising: training a machine learning model to predict a depth map of a color image via an iterative diffusion inference process, wherein training the machine learning model comprises: (i) adding noise to a first depth map to generate a second depth map, wherein the first depth map represents a depth across a first scene depicted in a first color image; (ii) applying the first color image and the second depth map to the machine learning model to generate a third depth map; (iii) generating a plurality of loss values ​​based on the third depth map; and (iv) updating the machine learning model based on the plurality of loss values.

[0006] In a second aspect, there is provided a computer-implemented method comprising: predicting, using a machine learning model, a first target depth map of a target color image via an iterative diffusion inference process for a plurality of iterations by: applying the target color image and a respective input depth map estimate to the machine learning model to generate a respective output depth map, wherein an input depth map of a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noisy depth map, wherein an input depth map of each iteration of the plurality of iterations other than the first iteration is the output depth map of the immediately preceding iteration with an amount of noise added, and wherein the first target depth map is the output depth map of a final iteration of the plurality of iterations.

[0007] In a third aspect, there is provided a computer-implemented method comprising: training a machine learning model to predict an optical flow map of a pair of color images via an iterative diffusion inference process, wherein training the machine learning model comprises: (i) adding noise to a first optical flow map to generate a second optical flow map, wherein the first optical flow map represents an optical flow from a first color image of a scene to a second color image of the scene; (ii) applying the first color image, the second color image, and the second optical flow map to the machine learning model to generate a third optical flow map; (iii) generating a plurality of loss values based on the third optical flow map; and (iv) updating the machine learning model based on the plurality of loss values.

[0008] In a fourth aspect, there is provided a computer-implemented method comprising: predicting, using a machine learning model, a first target optical flow map of a target pair of color images via an iterative diffusion inference process for a plurality of iterations by: applying the target pair of color images and a respective input optical flow map estimate to the machine learning model to generate a respective output optical flow map, wherein an input optical flow map of a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noisy optical flow map, wherein an input optical flow map of each iteration of the plurality of iterations other than the first iteration is the output optical flow map of the immediately preceding iteration with an amount of noise added, and wherein the first target optical flow map is the output optical flow map of a final iteration of the plurality of iterations.

[0009] In a fifth aspect, there is provided a non-transitory computer-readable medium having stored therein instructions executable by a computing device to cause the computing device to perform the method of the first, second, third, or fourth aspect.

[0010] In a sixth aspect, there is provided a system comprising: (i) a controller comprising one or more processors; and (ii) a non-transitory computer-readable medium having stored therein instructions executable by the controller device to cause the one or more processors to perform the method of the first, second, third, or fourth aspect.

[0011] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1A Aspects of an example inference process according to example embodiments are shown.

[0013] Figure 1A Aspects of an example training process according to example embodiments are shown.

[0014] Figure 2 Aspects of an example system are shown.

[0015] Figure 3 Aspects of an example process are shown.

[0016] Figure 4A A flow diagram of an example method is shown.

[0017] Figure 4B A flow diagram of an example method is shown.

[0018] Figure 4C A flow diagram of an example method is shown.

[0019] Figure 4D A flow diagram of an example method is shown.

[0020] Figure 5 Aspects of an experimental method are shown.

[0021] Figure 6 Experimental results are shown.

[0022] Figure 7 Experimental results are shown.

[0023] Figure 8 Experimental results are shown.

[0024] Figure 9 Aspects of an experimental method are shown.

[0025] Figure 10 Experimental results are shown.

[0026] Figure 11 Aspects of an experimental method are shown.

[0027] Figure 12 Experimental results are shown.

[0028] Figure 13 Experimental results are shown.

[0029] Figure 14 Experimental results are shown.

[0030] Figure 15 Experimental results are shown.

[0031] Figure 16 Aspects of experimental methods are shown.

[0032] Figure 17 Experimental results are shown.

[0033] Figure 18 Experimental results are shown.

[0034] Figure 19 Experimental results are shown.

[0035] Figure 20 Experimental results are shown.

[0036] Figure 21 Experimental results are shown. DETAILED DESCRIPTION

[0037] The following detailed description describes various features and functions of the disclosed systems and methods with reference to the accompanying figures. The illustrative system and method embodiments described herein are not meant to be limiting. It can be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated herein.

[0038] I. SUMMARY

[0039] In various applications it is desirable to obtain an estimate of depth map information for an environment (e.g., a scene depicted in a monochrome image) and / or optical flow between pairs of images (e.g., to estimate which portions of an image and / or which objects depicted therein are moving, in what direction, and to what extent). However, it can be difficult to obtain such information, or to obtain such information to a desired level of quality. This can be related to the absence of depth sensor elements (e.g., lidar, radar, time-of-flight sensors, ultrasonic sensors), the presence of low-quality depth sensor elements, the scene having been artificially generated (e.g., as an image output from a generative network), the ambiguity or other difficulty of generating an optical flow estimate from a pair of images, or some other factor or condition. In cases where a single image (e.g., a color image) of a scene is available, it is desirable to be able to predict a depth map from the single color image. However, existing models trained to generate such estimates or to estimate optical flow from such pairs of color images are computationally expensive to execute and are expensive to permit. It is also difficult to generate novel predictive models because the training data available (e.g., pairs of depth maps and images depicting the same scene, triples of an image of a scene and an optical flow map therebetween) is limited and is often of low quality (e.g., contains holes, contains noise).

[0040] The machine learning models and methods of training these models provided herein exhibit various improvements over previous models and training methods:

[0041] The ability to achieve improved accuracy when trained on smaller data sets;

[0042] The ability to achieve improved accuracy when trained on lower quality data sets (e.g., noisy depth or optical flow maps, depth or optical flow maps with holes);

[0043] The ability to represent multi-modal / ambiguous depth or optical flow regions (e.g., windows, mirrors) by, for example, generating multiple depth or optical flow maps for an input that can then differ significantly in multi-modal regions;

[0044] The ability to refine noisy or incomplete depth or optical flow maps even without being explicitly trained on such tasks.

[0045] The machine learning models described herein can be applied to a variety of applications:

[0046] Mapping an environment based on a single image of the environment (e.g., in order to navigate the environment, localize a person or object in the environment, manipulate an object in the environment);

[0047] Filling in holes in a depth map using associated images (e.g., to augment depth map information generated by a depth sensor by using images generated by a camera that is collocated with and / or part of the depth sensor and pointed in the same direction as the depth sensor);

[0048] Generating geometric information for a scene represented in an image created by a generative network;

[0049] Generating images / depth maps for fully simulated environments (e.g., environments “hallucinated” by a generative network) in a manner that allows for stable simulation of the geometry of the environment;

[0050] Identifying objects in a scene and / or quantifying the direction and extent of motion of such objects based on a pair of images of the scene.

[0051] The machine learning models described herein are trained for use within a generative noise diffusion context, in which outputs are generated via an iterative diffusion inference process. Such a process produces an output depth map conditioned on an input image or pair of input images (e.g., a color image, such as an RGB image, a YUV image, a hyperspectral image having more or fewer than three color channels). Each iteration of the iterative diffusion inference process includes adding noise to the depth map output of a previous iteration, and applying the previously output with added noise to the input image to the machine learning model to generate a refined depth or optical flow map output. The amount of noise added in each iteration can vary according to a pre-specified noise schedule, in which the amount of noise added generally decreases from an initial iteration to a final iteration.

[0052] Figure 1A depicted using a machine learning model 110 as described herein to generate an output depth or optical flow map 115 therefor via an iterative diffusion inference process from an input color image 101 ( / 103). A single input color image 101 is applied as input to the model 110 to generate an output depth map 115 therefor, representing depth across a scene depicted in the input color image 101, while a pair of input color images (a first color image 101 and a second color image 103) are applied as input to the model 110 to generate an output optical flow map 115 therefor, representing optical flow from the first color image 101 of a scene to the second color image 103 of the scene. For each iteration of the iterative diffusion inference process, an input noisy depth or optical flow map is also applied as input to the model 110. For the first iteration of the process, this input is the initial map 109, and for subsequent iterations, the map input is the map output 115 from the previous iteration, with an amount of noise added thereto. The magnitude of this added noise can be determined according to a noise schedule that varies with iteration of the iterative process (e.g., according to a noise schedule that monotonically decreases for later iterations of the process).

[0053] Additional optional inputs can be provided. For example, a field of view measurement 105 representing an angular field of view of the input color image 101 can be provided, e.g., to facilitate generating a metrologically accurate quantitative output depth map 115 for the input color image 101. The model 110 can be trained to accept such additional field of view measurement 105 input as described elsewhere herein, e.g., to accept a field of view measurement 105 representing a tangent function (e.g., tan / 2) of an angular field of view across at least one dimension of the input color image 101.

[0054] In some examples, a noise estimate 107 representing an expected amplitude of noise present in the input graph 109 / 115 can be provided as input to allow the model 110 to adjust the output graph 115 in a manner sensitive to the "completeness" of the iteration process (e.g., to generate an output graph 115 that is a slightly modified "tuned" version of the input graph for later iterations of the process, while introducing larger modifications to the input graph for earlier iterations). This noise estimate 107 can be determined according to a noise schedule that depends on the iteration of the process (e.g., monotonically decreasing as the number of iterations increases). The model 110 can be trained to accept such additional noise estimate 107 input as described elsewhere herein, e.g., to generate an output depth map 115 that represents a weighted combination of (i) the estimated noise in the input graph and (ii) the estimated "signal" in the input graph, where the proportion of the weighting is determined by the noise estimate 107 input (e.g., the model 110 can be trained to output a "V-parameterized" graph, rather than an "X-parameterized" to output an estimate of the "signal" alone in the input or an "epsilon-parameterized" to output an estimate of the "noise" alone in the input.

[0055] In applications where no corresponding depth or optical flow information or estimate is available for the input image, the initial input graph 109 can be a "pure noise" graph (generated via a random or pseudo-random noise generation process). In cases where depth information is available for the input image (e.g., a noisy depth map, a noisy optical flow map, a low resolution optical flow map, an optical flow map representing only regions of optical flow that have been determined with more than a specified confidence, a depth map containing one or more holes), such an initial estimate graph with noise added to it can be used as the initial input graph 109. This allows the machine learning model as described herein to be used for naive inference (i.e., where no depth or optical flow information is available for the input image or image pair) or to augment or otherwise improve imperfect depth or optical flow information (e.g., hole filling, denoising, upsampling, initial estimate refinement).

[0056] Note that a depth or optical flow map output from a machine learning model as described herein (e.g., map 115 output from model 110) can represent depth or optical flow information in various ways. For example, the output map can represent an estimate of the “true” depth or optical flow map of the input image; a model trained to operate in this way can be described as “X-parametrized.” In another example, the output map can represent an estimate of the “noise” present in the depth or optical flow map input to the model; a model trained to operate in this way can be described as “ε-parametrized.” In yet another example, the output map can represent a weighted combination of (i) an estimate of the “noise” present in the depth or optical flow map input to the model and (ii) an estimate of the “true” depth or optical flow map of the input image, where the weights are set by auxiliary inputs to the model; a model trained to operate in this way can be described as “V-parametrized.” The above-described iterative process can be modified according to the type of output generated by the model. For example, in the case of a model that outputs an estimate of the noise present in the input noisy map (the model is “ε-parametrized”), the n+1 input noisy map to the model can be the n input image, with the n model output subtracted from it (to obtain an estimate of the “true” map of the input color image) and then some amount of noise added to it. Other inter-iteration and terminal computations can be implemented for “X-parametrized” or “V-parametrized” machine learning models to obtain a “final” output depth or optical flow map of the input color image based on the final iteration model output.

[0057] Figure 1B Aspects of a training method for training a machine learning model as described herein (e.g., training model 110) are depicted. The training method includes applying an input color image 121 ( / 123) along with an input noisy depth or optical flow map 129 and any optional additional inputs (e.g., a field of view measurement 125 representing the angular field of view of the input color image 121, a noise estimate 127 representing the magnitude of the noise present in the input map 129) to the model 110 being trained to generate an output depth or optical flow map 135. A loss value is determined 140 by comparing the output map 135 to information about the input map 129 to determine a loss value that is then used (e.g., by backpropagation) to update the model 110.

[0058] The method for determining 140 the loss value can depend on the intended form of the output of the model (e.g., an estimate of a true depth or optical flow map, an estimate of noise in the input image, an input-specified weighted combination of the two). For example, for a model 110 trained to output an estimate of a “true” map of an input image, the pixel-wise difference (e.g., LI difference) between the output map 135 and the “true” map used to determine the input image 129 can be determined. In another example, for a model 110 trained to output an estimate of the “noise” present in an input image 129, the pixel-wise difference (e.g., LI difference) between the output map 135 and the “noise” added to a “true” map in order to determine the input image 129 can be determined. In yet another example, for a model 110 trained to output a map that is a weighted combination of the “noise” present in an input noisy image and the “signal,” the “target” output map (i.e., the map corresponding to the specified weighted combination of the “noise” and “signal” in the input noisy image) can be determined, and the pixel-wise difference (e.g., LI difference) between the output map 135 and the determined target map can be determined. Additionally, in the case where the training data is incomplete (e.g., in the case where the input depth map 129 includes holes or other defects), the loss value (e.g., the pixel-wise difference between the output map 135 and the input image 129 or information related thereto) can be determined based only on the “true” training portion (e.g., based only on the non-hole portions of the input depth map including one or more holes).

[0059] As shown by the experimental results below, training using the LI loss value between the model output map and the corresponding “target” map provides benefits with respect to model accuracy, training speed, and the amount of training data required. Furthermore, the determination of the LI loss value is less computationally expensive than the determination of the L2 loss value.

[0060] The experimental results below also show the benefits of “V-parametrized” models for the applications depicted herein. Such “V-parametrized” models employed in the iterative diffusion inference process described herein result in a significantly fewer number of iterations using the iterative diffusion inference process to determine high-accuracy depth or optical flow map estimates from input color images. This contributes to significant savings with respect to computational cost when such models are used to estimate such depth and / or optical flow maps.

[0061] As noted above, a model of an iterative diffusion inference process as described herein can condition on a field of view measurement that represents an angular field of view of an input color image, the model also conditioning on the input color image in order to estimate a depth map for the input color image. Such a field of view measurement can be obtained based on, for example, a field of view of an image sensor used to generate the color image. Alternatively, the field of view measurement can be determined from the input color image, for example, by applying the input color image to a model trained to estimate a field of view of an input image. Providing such a measurement of a field of view of an input image can allow the model to generate a higher quality (e.g., more accurate) depth map estimate. Additionally, providing a field of view measurement can allow the model to output a metric depth map that quantitatively represents depth across an input color image, thereby allowing such a metric depth map to be directly used for, for example, navigation or other applications that include determining distances to content of an environment from a camera or other image sensor. The field of view measurement can be provided as a tangent function of an angular field of view across at least one dimension of the input color image (e.g., tan / 2 across an angular field of view of the input color image); this can facilitate the model learning how to use additional field of view information with fewer training steps and / or using fewer training data examples, as such a tangent function can more “natively” represent a relationship between a change in an angular field of view of an image and a corresponding change in a metric depth of objects depicted therein.

[0062] Training data sets that include color image field of view annotations are more rare and higher cost relative to color image training data sets that lack such data. Thus, a model as described herein (e.g., to output an estimate of noise and / or signal in an input noisy depth or optical flow map conditioned on an input color image) can be pre-trained using a data set that lacks such field of view annotations but is relatively larger than available training data sets that include such information. Once this pre-training is complete, the pre-trained model can be modified to accept a field of view measurement and further trained using images and associated training data that include field of view measurements for them. In this way, an accurate model can be trained to condition on a field of view measurement while using a significantly reduced number of training examples that include such field of view measurement information. Modifying such a pre-trained model to accept a field of view measurement can include, for example, projecting an input field of view measurement via a sinusoidal position embedding. The result of the sinusoidal position embedding can then be provided as input to various layers of the pre-trained model, for example, via a newly added set of weights or other linear projection of the output of the sinusoidal position embedding to one or more layers of the pre-trained model.

[0063] Additionally or alternatively, model training can be improved by augmenting existing training datasets that include field of view measurements to include more kinds of field of view measurements. This can be done by cropping or uncropping images of the training dataset and modifying the corresponding field of view measurements to account for the degree of cropping / uncropping. In the case of images that are uncropped, various methods can be used to determine the “novel” uncropped portion of the image. This can include, for example, applying the source image to a model trained to uncrop images. In another example, the “novel” portion of the uncropped image can simply be filled with randomly or pseudo-randomly generated noise content (e.g., Gaussian red, green, and blue pixel magnitude data). As shown by the related experimental data below, this “random padding” image uncropping method is able to generate a model of similar quality as using a generative model to uncrop images, and in similar or fewer training steps, thereby saving computational costs significantly. When updating a model using such uncropped data, the loss values determined for it can be limited to the “source” portion of the training image and / or the corresponding portion of the depth or optical flow map (and not based on any “hallucinated” or “noisy” uncropped portion of the input color image).

[0064] The iterative diffusion inference process can be used in a “coarse-to-fine” manner to generate high resolution depth and / or optical flow maps for input color images. For example, the iterative diffusion inference process can be performed based on an entire input color image or pair of color images a first time to generate a first lower resolution estimate of the depth or optical flow map for the input color image or image pair. This low resolution estimate can then be upsampled and divided into patches. The conditioning input color image or image pair can also be divided into a corresponding set of patches, with each patch of the input color image corresponding to a degree to the corresponding patch of the low resolution depth or optical flow map. The iterative diffusion inference process can then be repeated for each associated set of patches of the input image and low resolution map to generate a respective patch of the high resolution estimate of the depth or optical flow map for the original full input color image. The high resolution estimate can then be generated by stitching each high resolution patch together. This can include applying the high resolution patches to a mask such that the regions of the combined high resolution estimate near the boundaries of the patches represent a weighted combination of the corresponding neighboring high resolution patches in order to reduce artifacts at the boundaries between the patches in the combined high resolution estimate depth or optical flow map.

[0065] The machine learning models used to generate the initial low-resolution depth or optical flow maps and the various high-resolution patches used to generate the high-resolution combined depth or optical flow maps can be the same model or different models (e.g., a first model specifically trained to generate low-resolution depth or optical flow maps and a second model specifically trained to generate high-resolution depth or optical flow maps). This two-stage map generation process, first using an iterative diffusion inference process for generating a low-resolution estimate of a depth or optical flow map and then a second set using an iterative diffusion inference process for generating a high-resolution estimate of a respective patch of a depth or optical flow map, starting from a respective initial depth or optical flow estimate that is an upsampled low-resolution estimate of the respective patch, can provide various benefits. For example, the models used to implement this scheme (as they output outputs of relatively small resolution) can be implemented with fewer parameters to achieve a specified accuracy, thereby reducing storage requirements and reducing memory requirements for performing.

[0066] As part of the iterative diffusion inference processes described herein, a machine learning model as described herein can be trained to predict a depth map for an input color image of an indoor and / or outdoor scene. For example, a first model can be trained using a training dataset of images of indoor scenes and depth maps for the same to predict a depth map for a color image of an indoor scene having values between 0 and 10 meters; a second model can be trained using a training dataset of images of outdoor scenes and depth maps for the same to predict a depth map for a color image of an outdoor scene having values greater than 10 meters (e.g., between 10 meters and 80 meters). A target color image can be accompanied by a label indicating whether it depicts an indoor scene or an outdoor scene, and / or the image can be applied to a model trained to identify whether a scene is represented by an indoor scene or an outdoor scene, and the target color image is then applied to the appropriate one of the first or second images to determine a target depth map for the same.

[0067] This two (or more) model approach involves additional costs in terms of training, training data, model storage, and other costs. However, it can be difficult to train a single model to predict depth maps for both indoor and outdoor scenes, or to predict depth maps for some other set of images that differ greatly in terms of the depth of objects depicted therein. The embodiments described herein provide an improved model and improved method of training thereof that can receive an input color image and, via an iterative diffusion inference process described herein, generate an estimated depth map therefor, whether the input image depicts an indoor scene, an outdoor scene, or a scene that includes both indoor and outdoor elements. This is accomplished in part by training the model to output a log-distance depth map that represents the log of the depth across the scene depicted in the input color image. This is accomplished in part by converting the depth information of a training dataset into log space. This can be accomplished by determining the log of the depth information of the training dataset, and then shifting and / or scaling the log of the depth information to correspond to a desired finite range (e.g., a range from -1 to +1 or some other range of values symmetric about 0). A truncation step can also be applied to ensure that the shifted and scaled log data remains within the desired bounded range, discarding high (or low) value depth data that is more likely to represent sensor artifacts or other erroneous or otherwise unwanted data.

[0068] Training the model to predict log depth map outputs using log depth data allows those outputs to be (as demonstrated by the experimental data below) metric log-distance depth maps. That is, the log-distance depth maps generated as described herein can be used to determine (e.g., by scaling, shifting, and exponentiating the output log-distance depth map) the objective depth to objects in the scene depicted by the input color image, not just a depth map that depicts a pattern of relative depths across the scene. Thus, the models as described herein (and the depth maps determined therefrom) can be directly used to map build and / or navigate an environment using only a camera or other low-cost sensor sufficient to generate a color image.

[0069] Using machine learning models in this generative diffusion context provides some of the benefits and advantages of such models as described herein. However, it is difficult to obtain and / or generate sufficient training data (e.g., color images and high-quality depth maps for them, pairs of color images and accurate, unambiguous, high-resolution optical flow maps for them) to train such models. Thus, additional improvements can be provided by pre-training such models using a set of available images (e.g., color images) to perform a pre-training process. The pre-trained model is then trained on a depth and / or optical flow map prediction task using a set of input images and corresponding depth maps and / or pairs of input images and corresponding optical flow maps for them. This pre-training process includes training a machine learning model on an image-to-image prediction task, where a degraded version of a training image is provided to the image input of the model (input for which the output is conditioned), and a noisy version of the training image is provided to what will be the content of the depth or optical flow map input of the model. The model is then trained during this pre-training phase to recover the training image based on the degraded version of the training image as the "input." This allows the model to learn general information about the structure of input images or image pairs based on a much larger training dataset than if the training dataset were available with corresponding available depth or optical flow map information for each image. The set of image degradations applied can include at least one example of cropping, desaturation (i.e., removing color information from a color image, e.g., rendering a color image in black and white), adding JPEG or other compression artifacts, and adding one or more holes by blanking out or otherwise removing pixels of the training image.

[0070] Once pre-trained in this way, the model can then be used as a starting state for training a model to predict depth or optical flow map outputs from input images or image pairs. Since the pre-training process described above includes accepting a multi-channel noisy color image input (e.g., three input channels) and generating a multi-channel color image output (e.g., three output channels) in addition to a multi-channel input of the input image for which the output is conditioned (e.g., three or six input channels pre-trained for depth or optical flow prediction, respectively), modifying the pre-trained model for continued training on a depth or optical flow map prediction task can include removing one or more input kernels for the noisy image input and one or more output kernels for predicting the image output, and replacing them with a single (for depth map prediction) or dual (for optical flow map prediction, e.g., angle and distance of the flow, or first and second distances orthogonal to the flow) channel input and output kernel for receiving input noisy depth or optical flow map information and generating a depth or optical flow map output prediction. This can include initializing the added single or dual channel kernel with random weights.

[0071] In practice, the depth or optical flow map information in a training data set can often include holes, i.e., regions where depth information is not available (e.g., due to limitations of the sensor, due to depth being beyond limits, due to depth measurement noise being too large) or optical flow information is not available (e.g., due to ambiguity or blurring). To provide some improvements described herein, such holes in the training depth map can be filled in. This can include applying the trained model to predict what the depth or optical flow values within the holes should be from the rest of the depth or optical flow map. Additionally or alternatively, a nearest neighbor approach can be used to fill in such holes (e.g., a nearest non-hole pixel or other location of the depth or optical flow map can be assigned a depth value or optical flow angle and distance value for a given pixel or other location within the hole). Additionally or alternatively, an image associated with the depth map (e.g., a color image) can be used to determine a location and extent of sky within the image, and thus within the depth map, and pixels within the determined location and extent of sky can be set to a default “sky” depth value (e.g., a maximum depth value). When training data for such a depth or optical flow map is used (i.e., a depth or optical flow map that has had one or more holes filled in), the loss values or other information used to update the trained model can be determined and / or used only for portions of the depth or optical flow map training data that correspond to the “true” non-filled-in portions of the depth map training data (e.g., only for pixels of the originally set training depth or optical flow map, thus not “filled in” with potentially erroneous depth values).

[0072] This hole filling process is functionally used to attempt to match the distribution of input data at training time (i.e., the distribution of depth or optical flow maps in the training data) to the distribution of input data at inference time. However, at training time, the input data can exhibit longer tails, more “holes,” and other characteristics related to limitations of the depth sensor used to generate the depth map training data, for example. Conversely, at inference time, the input data can be “pure” noise, or generally a short-tailed distribution present from iteration to iteration of model output from the iterative diffusion inference process described herein. To further account for this distribution shift from training to inference, in addition to or as an alternative to the hole filling or other training dataset improvement techniques described herein, another training method can be employed (which can be referred to elsewhere herein as “unbiased conditioning” or “step-unrolled denoising”). This improvement for training a machine learning model includes using the model being trained to provide an initial, improved prediction of an input depth or optical flow map training example. This improved prediction then has noise added to it and is provided again to the machine learning model in order to generate a second output predicted depth map. A loss value or other information used to update the model (e.g., an LI loss value determined between the “true” input map and the output predicted map, or an LI loss value determined between the noise added to the “true” input map to generate the input map and the output predicted map, or an LI loss value determined for training the model according to a V parameterization) can then be determined based on this second output rather than based on the initial output.

[0073] Such a process can be referred to as “unbiased conditioning” or “step-unrolled denoising,” and can provide improved model performance using fewer lower-quality training examples by, for example, reducing the bias of the model being trained between training phase examples and inference phase examples. This “unbiased conditioning” training enhancement can be used only when the model being trained has been updated by a certain amount (e.g., a specified number of training iterations, a specified increase in model performance, model performance exceeding a threshold, etc.). Additionally or alternatively, this “unbiased conditioning” training enhancement can be reserved for specified discrete phases of model training, e.g., only for specific training on task-specific training examples, rather than for prior training on general-purpose training examples.

[0074] The models as described herein can be configured in various ways. For example, such models can be consistent with the Efficient U-Net architecture or some other version of the U-Net architecture. For example, the models as described herein can be the Efficient U-Net architecture with the text cross-attention layers removed and the self-attention layers retained. For depth map prediction tasks, the architecture can be modified to have 4 input channels (three for the three color channels of the conditioned control input color image, and one for the input noisy depth map) and 1 output channel for outputting the depth map prediction. The input and / or output resolution of the model can take various values depending on the application and / or depending on the available training data.

[0075] The trained models and iterative diffusion inference processes described herein can be used to generate depth and / or optical flow maps from input color images or pairs of color images to facilitate various applications, and in ways that improve with respect to the amount of training data required, the size of the model, the computational cost of training or performing inference, or with respect to other factors. Additionally, the use of the iterative diffusion inference process allows prior information about the depth or optical flow map to be predicted (e.g., output from a low-quality or damaged depth sensor, a low resolution, ambiguous, incomplete, or otherwise low-quality estimate of the optical flow map) to be easily injected into the process as the “initial” depth or optical flow map presented to the model in the first iteration of the iterative diffusion inference process.

[0076] This ability to inject prior depth information allows the methods described herein to produce simulated images of an environment in the context of a full generation that depict consistent, fully model-generated internal geometry. For example, an initial image of a scene can be generated by a first generative model (e.g., by a text-to-image model). The methods described herein can then be used to generate an initial depth map for the initial image of the scene. The initial depth map can then be projected onto a point cloud or other persistent representation of the geometry of the scene. That geometry can then be rotated and / or translated to determine a second perspective. The point cloud or other representation of the geometry of the scene can then be used, along with the initial image, to render a second image of the scene from the second perspective. This can include projecting the “visible” aspects of the scene in the initial image onto the rotated and / or translated point cloud, and then generatively (or otherwise) filling in any remaining holes in the projected second image that would correspond to portions of the scene that are “not visible” in the initial image. The methods described herein can then be applied to the second image to generate a second depth map for it. The second depth map can then be used to augment the point cloud or other persistent representation of the internal geometry of the scene, thereby causing the geometry of the scene to “grow” in a way that “preserves” the structure of the geometry that has already been “seen” in the prior image of the scene.

[0077] Another benefit of the iterative diffusion inference process described herein is the ability to run the process multiple times with the same input image to generate corresponding multiple samples of its underlying depth or optical flow distribution. This can allow the variance or confidence of the output depth or optical flow map to be easily calculated, for example, by performing pixel-by-pixel variance across a set of maps generated for the input image. Additionally or alternatively, bimodal (or otherwise multimodal) regions of the distribution can be easily detected by, for example, detecting bimodal regions between two (or more) sets of output depth or flow maps for a particular input image. Such bimodal regions in the depth map can indicate, for example, the presence of a mirror, window, or other reflective, transparent, or translucent object in the scene and / or that the depiction of the scene is inherently ambiguous with respect to depth, and therefore extra care should be taken when navigating or otherwise interacting with such bimodal regions.

[0078] II. Illustrative Systems

[0079] Figure 2 An example computing system 200 is shown that can be used to implement the methods described herein. By way of example and not limitation, computing system 200 can be a cellular mobile phone (e.g., a smartphone), a computer (such as a desktop, laptop, tablet, or handheld computer, a server), an element of a cloud computing system, a robot, a drone, an autonomous vehicle, or some other type of device. It should be understood that computing system 200 can represent a physical computing device, such as a server, a specific physical hardware platform on which a machine learning application (training and / or inference) operates in software, or other combinations of hardware and software configured to perform machine learning or other functions as described herein. Computing system 200 can be a central system (e.g., a server, an element of a cloud computing system) that is configured to receive a color image or other input (e.g., FOV value, incomplete or noisy depth and / or optical flow map) from a remote system and, in response, send to the remote system an output depth and / or optical flow map generated by applying the image and optional additional input to a machine learning model as described herein. Additionally or alternatively, computing system 200 can be a remote system configured to send an image (and optionally additional inputs) to a central system and receive a depth and / or optical flow map in response, and / or receive a representation of a model as described herein from the central system (e.g., having been trained by the central system using the methods described herein) and apply the model locally to the image to generate a depth and / or optical flow map therefrom.

[0080] like Figure 2 As shown, computing system 200 may include a communication interface 202 , a user interface 204 , a processor 206 , and a data storage device 208 , all of which may be communicatively linked together via a system bus, network, or other connection mechanism 210 .

[0081] The communication interface 202 can be used to allow the computing system 200 to communicate with other devices, access networks, and / or transport networks using analog or digital modulations of electrical, magnetic, electromagnetic, optical, or other signals. Thus, the communication interface 202 can facilitate circuit- switched and / or packet-switched communications, such as plain old telephone service (POTS) communications and / or Internet Protocol (IP) or other packetized communications. For example, the communication interface 202 can include a chipset and antenna arranged for wireless communication with a radio access network or access point. Further, the communication interface 202 can take or include the form of a wired interface, such as an Ethernet, Universal Serial Bus (USB), or High-Definition Multimedia Interface (HDMI) port. The communication interface 202 can also take or include the form of a wireless interface, such as a Wifi, BLUETOOTH®, Bluetooth, Global Positioning System (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPP Long-Term Evolution (LTE)). However, other forms of physical layer interface and other types of standard or proprietary communication protocols can be used on the communication interface 202. Moreover, the communication interface 202 can include multiple physical communication interfaces (e.g., a Wifi interface, a BLUETOOTH® interface, and a wide-area wireless interface).

[0082] In some embodiments, the communication interface 202 can be used to allow the computing system 200 to communicate with other devices, remote servers, access networks, and / or transport networks.

[0083] The user interface 204 can be used to allow the computing system 200 to interact with a user or other entity, for example to receive input from and / or provide output to the user. Thus, the user interface 204 can include input components such as a keypad, keyboard, touch- or presence-sensitive panel, computer mouse, trackball, joystick, microphone, etc. The user interface 204 can also include one or more output components, such as a display screen, which can be combined with a touch-sensitive panel, for example. The display screen can be based on CRT, LCD, and / or LED technology, or other technology now known or later developed. The user interface 204 can also be configured to generate audible output via a speaker, speaker jack, audio output port, audio output device, headphones, and / or other similar devices.

[0084] The processor 206 can include one or more general-purpose processors (e.g., microprocessors) and / or one or more special-purpose processors (e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, tensor processing units (TPUs), or application-specific integrated circuits (ASICs)). In some instances, the special-purpose processors can be capable of image processing, image alignment, merging images, transforming images, executing machine learning models, training machine learning models, and other applications or functions. The data storage device 208 can include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and can be integrated in whole or part with the processor 206. The data storage device 208 can include removable and / or non-removable components.

[0085] The processor 206 can be capable of executing program instructions 218 (e.g., compiled or interpreted program logic and / or machine code) stored in the data storage device 208 to perform various functions described herein. Thus, the data storage device 208 can include a non-transitory computer-readable medium having stored thereon program instructions that, when executed by the computing system 200, cause the computing system 200 to perform any of the methods, processes, or functions disclosed in the specification and / or drawings. Execution of the program instructions 218 by the processor 206 can result in the processor 206 using the data 212.

[0086] By way of example, the program instructions 218 can include an operating system 222 (e.g., an operating system kernel, device drivers, and / or other modules) installed on the computing system 200 and one or more application programs 220 (e.g., for performing and / or training machine learning models). The data 212 can include training data (e.g., individual color images and / or cropped, uncropped, or otherwise modified versions thereof, sets of color images and associated field of view values, depths, and / or optical flow maps, etc.) 214 and / or machine learning models 216 that can be determined or otherwise obtained therefrom.

[0087] The application programs 220 can communicate with the operating system 222 through one or more application programming interfaces (APIs). These APIs can facilitate, for example, the application programs 220 sending or receiving information via the communication interface 202, receiving and / or displaying information on the user interface 204, etc.

[0088] The application programs 220 can take the form of “apps” that can be downloadable to the computing system 200 through one or more online application stores or application markets (via, e.g., the communication interface 202). However, the application programs can also be installed on the computing system 200 in other ways, such as via a web browser or through a physical interface (e.g., a USB port) of the computing system 200.

[0089] III. Example Machine Learning Models and Training Thereof

[0090] A machine learning model as described herein can include, without limitation, an artificial neural network (e.g., a convolutional neural network, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system as described herein), a support vector machine, a regression tree, a collection of regression trees (also referred to as a regression forest), a decision tree, a collection of decision trees (also referred to as a decision forest), or some other machine learning model architecture or combination of architectures. Such a machine learning model can include an attention mechanism (e.g., a self-attention matrix or other self-attention feature). Such a machine learning model can include a plurality of elements (e.g., attention and / or neural network layers) arranged according to a multi-layer architecture, e.g., a machine learning model as described herein can be configured as a Transformer.

[0091] An artificial neural network (ANN) can be configured in various ways. For example, an ANN can include two or more layers, can include units having linear, logistic, or otherwise specified output functions, can include neurons that are fully or otherwise connected, can include recurrent and / or feed-forward connections between neurons in different layers, can include filters or other elements to process input information and / or information passed between layers, or can be configured in some other way to facilitate generating a predicted color palette based on an input image.

[0092] An ANN can include one or more filters that can be applied to an input, and then the output of such a filter can be applied to the input of one or more neurons of the ANN. For example, such an ANN can be or can include a convolutional neural network (CNN). A convolutional neural network is a variety of ANN that is configured to facilitate ANN-based classification or other processing based on an image or other large-dimensional input whose elements are organized within two or more dimensions. The organization of the ANN along these dimensions can relate to some structure in the structure of the input (e.g., because relative positions within a two-dimensional space of an image can relate to similarity between pixels of the image).

[0093] In example embodiments, a CNN includes at least one two-dimensional (or higher-dimensional) filter that is applied to an input; the filtered input is then applied to neurons of the CNN (e.g., neurons of a convolutional layer of the CNN). In embodiments in which the input is an image, such a filter and convolution of the input can represent color values from a pixel or a group of pixels of the input. A set of neurons of the CNN can receive respective inputs that are determined by applying the same filter to the input. Additionally or alternatively, a set of neurons of the CNN can be associated with respective different filters, and can receive respective inputs that are determined by applying the respective filters to the input. Such filters can be trained during training of the CNN or can be pre-specified. For example, such filters can represent wavelet filters, center-surround filters, biologically-inspired filter kernels (e.g., from studies of receptive fields in animal visual processing), or some other pre-specified filter pattern.

[0094] A CNN or other various ANN can include multiple convolutional layers (e.g., corresponding to respective different filters and / or features), pooling layers, rectified layers, fully connected layers, or other types of layers. A convolutional layer of a CNN represents a convolution of an input image or some other input (e.g., a filtered, down-sampled, or otherwise processed version of the input image) with a filter. A pooling layer of a CNN applies a non-linear down-sampling to a higher layer of the CNN, e.g., by applying a max, average, L2 norm, or other pooling function to a subset of neurons, outputs, or other features of the higher layer of the CNN. A rectified layer of a CNN applies a rectified non-linear function (e.g., a non-saturating activation function, sigmoid function) to outputs of a higher layer. A fully connected layer of a CNN receives inputs from many or all neurons in one or more higher layers of the CNN. Outputs of neurons of one or more fully connected layers (e.g., a final layer of an ANN or CNN) can be used to determine information about regions of an input image (e.g., for each pixel of the input image) or for the image as a whole.

[0095] Neurons in a CNN can be organized according to corresponding dimensions of input. For example, where input is an image (two-dimensional input, or three-dimensional input where color channels of the image are arranged along a third dimension), neurons of the CNN (e.g., neurons of an input layer of the CNN, neurons of a pooling layer of the CNN) can correspond to locations in the two-dimensional input image. Connections between neurons and / or filters in different layers of the CNN can be related to such locations. For example, a neuron in a convolutional layer of the CNN can receive input based on convolution of a filter with a portion of the input image, or based on convolution of a filter with a portion of some other layer of the CNN, the portion being at a location proximate to the location of the convolutional layer neuron. In another example, a neuron in a pooling layer of the CNN can receive input from neurons in a layer above the pooling layer (e.g., in a convolutional layer, in a higher pooling layer), the neurons having locations proximate to the location of the pooling layer neuron.

[0096] Figure 3 A diagram 300 illustrating a training phase 302 and an inference phase 304 of a trained machine learning model 332 according to example embodiments is shown. Some machine learning techniques involve training one or more machine learning algorithms on a set of inputs of training data to discern patterns in the training data and provide output inferences and / or predictions about the (patterns in the) training data. This output can take the form of a filtered or otherwise modified version of the input, e.g., a noisy depth or optical flow map input can be modified by the machine learning model, conditioned on one or more auxiliary color image inputs thereto, to identify noisy content of the input map, to identify non-noisy content of the input map (i.e., to attempt to predict a ‘true’ noise-free map), and / or to identify a specified weighted combination of noisy and non-noisy content of the input map. The resulting trained machine learning algorithm can be referred to as a trained machine learning model. For example, Figure 3 A training phase 302 is shown in which one or more machine learning algorithms 320 are being trained on training data 310 to become a trained machine learning model 332. Then, during an inference phase 304, the trained machine learning model 332 can receive input data 330 and one or more inference / prediction requests 340 (possibly as part of the input data 330), and responsively provide one or more inferences and / or predictions 350 as output.

[0097] In some examples, such an inference process can include multiple executions of the model. For example, the model can receive a color image and a noisy depth or optical flow image as input, and output a depth or optical flow image and / or a prediction of the noisy content of the input noisy depth or optical flow image based on the image and noisy depth or optical flow image. Such a model can be used to make inferences in a diffusion context by multiple executions of the model, first with pure noise as input at the depth or optical flow input channels, and subsequently with the previous depth or optical flow output corrupted by an additive noise amount (where the additive noise amount is reduced according to a noise schedule with each iteration) as input. Such a diffusion context also has the benefit of adapting to improve the initial estimate of the depth map (e.g., the noisy depth or optical flow image, the depth or optical flow image containing holes) by presenting such an initial estimate (corrupted by an additive noise amount) as an initial input to the depth or optical flow channels of the model for the initial execution of the model.

[0098] As such, the trained machine learning model 332 can include one or more models of one or more machine learning algorithms 320. The machine learning algorithms 320 can include, but are not limited to, an artificial neural network (e.g., a convolutional neural network, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system described herein), a support vector machine, a regression tree, a collection of regression trees (also known as a regression forest), a decision tree, a collection of decision trees (also known as a decision forest), or some other machine learning model architecture or combination of architectures, such as a Transformer composed of multiple sub-models (e.g., ANNs) arranged such that the output from many of the sub-models is provided as input to other sub-models. The machine learning algorithms 320 can be supervised or unsupervised, and can implement any suitable combination of online and offline learning.

[0099] In some examples, on-device co-processors such as graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application-specific integrated circuits (ASICs) can be used to accelerate the machine learning algorithms 320 and / or the trained machine learning model 332. Such on-device co-processors can be used to accelerate the machine learning algorithms 320 and / or the trained machine learning model 332. In some examples, the trained machine learning model 332 can be trained, resident, and executed to provide inferences on a particular computing device, and / or can otherwise be targeted for inferences on a particular computing device.

[0100] During the training phase 302, the machine learning algorithm 320 can be trained by using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques to provide at least training data 310 as training input. Unsupervised learning involves providing a portion (or all) of the training data 310 to the machine learning algorithm 320 and the machine learning algorithm 320 determines one or more output inferences based on the portion (or all) of the training data 310 provided. Supervised learning involves providing a portion of the training data 310 to the machine learning algorithm 320, where the machine learning algorithm 320 determines one or more output inferences based on the portion of the training data 310 provided and the output inferences are accepted or corrected based on correct results associated with the training data 310. In some examples, the supervised learning of the machine learning algorithm 320 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or the set of labels can be used to correct inferences of the machine learning algorithm 320.

[0101] Semi-supervised learning involves having correct results for a portion but not all of the training data 310. During semi-supervised learning, supervised learning is used for a portion of the training data 310 having correct results and unsupervised learning is used for a portion of the training data 310 not having correct results. Reinforcement learning involves the machine learning algorithm 320 receiving a reward signal with respect to a previous inference, where the reward signal can be a numerical value. During reinforcement learning, the machine learning algorithm 320 can output an inference and receive a reward signal in response, where the machine learning algorithm 320 is configured to attempt to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total numerical value provided by the reward signal over time. In some examples, the machine learning algorithm 320 and / or the trained machine learning model 332 can be trained using other machine learning techniques, including but not limited to incremental learning and curriculum learning.

[0102] In some examples, the machine learning algorithm 320 and / or the trained machine learning model 332 can use a transfer learning technique. For example, the transfer learning technique can involve a trained machine learning model 332 that is pre-trained on a set of data and additionally trained using the training data 310. More specifically, the machine learning algorithm 320 can be pre-trained on data from one or more computing devices and the resulting trained machine learning model provided to the computing device CD1, where CD1 is intended to execute the trained machine learning model during the inference phase 304. The pre-trained machine learning model can then be additionally trained during the training phase 302 using the training data 310, which can be derived from the kernel and non-kernel data of the computing device CD1. This further training of the machine learning algorithm 320 and / or the pre-trained machine learning model using the training data 310 of the data of CD1 can be performed using supervised or unsupervised learning. Once the machine learning algorithm 320 and / or the pre-trained machine learning model has been trained on at least the training data 310, the training phase 302 can be completed. The resulting trained machine learning model can be used as at least one of the trained machine learning models 332.

[0103] In particular, once the training phase 302 has been completed, the trained machine learning model 332 can be provided to the computing device, if not already on the computing device. The inference phase 304 can begin after the trained machine learning model 332 is provided to the computing device CD1.

[0104] During the inference phase 304, the trained machine learning model 332 can receive input data 330 and generate and output one or more corresponding inferences and / or predictions 350 with respect to the input data 330. As such, the input data 330 can be used as input to the trained machine learning model 332 for providing corresponding inferences and / or predictions 350 to the kernel components and non-kernel components. For example, the trained machine learning model 332 can generate inferences and / or predictions 350 in response to one or more inference / prediction requests 340. In some examples, the trained machine learning model 332 can be executed by a portion of other software. For example, the trained machine learning model 332 can be executed by an inference or prediction daemon to be readily available to provide inferences and / or predictions upon request. The input data 330 can include data from the computing device CD1 that executes the trained machine learning model 332 and / or input data from one or more computing devices other than CD1.

[0105] The input data 330 can include a set of images provided by one or more sources. The set of images can include video frames, images resident on the computing device CD1, and / or other images. Other types of input data are also possible.

[0106] Inference and / or prediction 350 can include output images, output depth or optical flow maps, numerical values, and / or other output data produced by the trained machine learning model 332 operating on input data 330 (and training data 310). In some examples, the trained machine learning model 332 can use output inferences and / or predictions 350 as input feedback 360. The trained machine learning model 332 can also rely on past inferences as input for generating new inferences.

[0107] IV. Example Methods

[0108] Figure 4A is a flowchart of an example computer-implemented method 400a. The method 400a includes training a machine learning model to predict a depth map of a color image via an iterative diffusion inference process (410a). The training 410a includes adding noise to a first depth map to generate a second depth map, where the first depth map represents depths across a first scene depicted in a first color image (412a). The training 410a also includes applying the first color image and the second depth map to the machine learning model to generate a third depth map (414a). The training 410a also includes generating a plurality of loss values based on the third depth map (416a). The training 410a also includes updating the machine learning model based on the plurality of loss values (428a). The method 400a can include additional or alternative features.

[0109] Figure 4B is a flowchart of an example computer-implemented method 400b. The method 400b includes predicting a first target depth map of a target color image via an iterative diffusion inference process using a machine learning model (410b). The iterative diffusion inference process 410b includes, for a plurality of iterations, applying the target color image and a respective input depth map estimate to the machine learning model to generate a respective output depth map, where an input depth map for a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noise depth map, where an input depth map for each iteration of the plurality of iterations other than the first iteration is the output depth map of an immediately preceding iteration with an amount of noise added, and where the first target depth map is the output depth map of a final iteration of the plurality of iterations (412b). The method 400b can include additional or alternative features.

[0110] Figure 4Cis a flowchart of an example computer-implemented method 400c. The method 400c includes training a machine learning model to predict an optical flow map for a pair of color images via an iterative diffusion inference process (410c). The training 410c includes adding noise to a first optical flow map to generate a second optical flow map, where the first optical flow map represents an optical flow from a first color image of a scene to a second color image of the scene (412c). The training 410c also includes applying the first color image, the second color image, and the second optical flow map to the machine learning model to generate a third optical flow map (414c). The training 410c further includes generating a plurality of loss values based on the third optical flow map (416c). The training 410c also includes updating the machine learning model based on the plurality of loss values (428c). The method 400c can include additional or alternative features.

[0111] Figure 4D is a flowchart of an example computer-implemented method 400d. The method 400d includes predicting a first target optical flow map for a target pair of color images via an iterative diffusion inference process using a machine learning model (410d). The iterative diffusion inference process 410d includes, for a plurality of iterations, applying the target pair of color images and a corresponding input optical flow map estimate to the machine learning model to generate a corresponding output optical flow map, where an input optical flow map for a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noise optical flow map, where an input optical flow map for each iteration of the plurality of iterations other than the first iteration is an output optical flow map of an immediately preceding iteration with an amount of noise added, and where the first target optical flow map is an output optical flow map of a final iteration of the plurality of iterations (412d). The method 400d can include additional or alternative features.

[0112] V. Experimental Results

[0113] The embodiments described herein are implemented in various ways and experimentally evaluated, empirically showing the benefits of the subject matter of the present disclosure.

[0114] First Experiment

[0115] In some experiments, a depth map prediction model (and its training method) is implemented as a "DepthGen" model. DepthGen outperforms state-of-the-art (SOTA) baselines on the indoor NYU dataset and is competitive on the KITTI dataset. Ablation experiments show that unsupervised pre-training, depth infilling, LI loss, and unbiased conditioning all significantly improve performance. As probabilistic models, DepthGen (and similar models as described herein) offer various benefits. With the ability to represent multi-modal distributions, these models can resolve depth ambiguities, e.g., due to reflective or transparent surfaces. Such diffusion models can also be easily modified to impute a map based on an incomplete initial estimate or other priors, allowing the use of DepthGen (and similar models described herein) to infer missing depth values. This property, together with the zero-shot capability of text-to-image models to exist, to build and experimentally evaluate a framework for text-to-3D scene generation and novel view synthesis.

[0116] DepthGen is a diffusion model for monocular depth estimation and includes self-supervised pre-training and supervised fine-tuning. Without specialized loss functions or architectures, it achieves a SOTA relative error of 0.074 on the NYU benchmark.

[0117] To train a diffusion model on noisy incomplete depth data, LI loss, depth infilling, and unbiased conditioning can be used to reduce potential distributional shifts between training and inference.

[0118] DepthGen enables multi-modal depth inference and imputation of missing depth, e.g., for text-to-3D generation and novel view synthesis.

[0119] Figure 5 Aspects of the training architecture that are experimentally evaluated are depicted, as described in more detail below.

[0120] A diffusion model is a latent variable generative model trained to transform samples of Gaussian (or otherwise distributed) noise into samples from a data distribution. Training such a model includes a forward process that gradually annihilates data by adding noise as 'time' t increases from 0 to 1, and a learned generative process that reverses the forward process, starting from a random noise sample at t = 1 and incrementally adding structure (decaying noise) as t decreases to 0. A conditional diffusion model adjusts the steps of the reverse process; in the case of the depth estimation task experimentally evaluated herein, the conditioning signal is an RGB image x, and the target is the conditional distribution p(y | x) on depth maps.

[0121] Central to the model is a denoising network f θwhich is trained to take noisy samples at time step t and predict from them samples with less noise. Using Gaussian noise in the forward pass, the training objective on the sequence of transitions (decreasing slowly with t) can be expressed as the sum of nonlinear regression losses, i.e.,

[0122]

[0123] where N(0, I), t ~ U(0, 1), and where > 0 is computed with a predetermined noise schedule. To infer (e.g., sample), a random noise sample y1is drawn, and then iteratively to estimate the noise in y s where s < t.

[0124] DepthGen training includes self-supervised pre-training on a generic RGD training dataset, followed by supervised training on an RGB-D training dataset. The pre-trained model is a self-supervised multi-task diffusion model. The palette model is trained from scratch on four image translation tasks (colorization, inpainting, uncropping, and JPEG artifact removal).

[0125] Following this pre-training on a generic RGD training dataset, the architecture is modified slightly (including modifications to reduce the dimensionality of the model output channels and the model depth map input channels from three (for color image pre-training) to one (for depth)). Training then continues on paired RGB and depth data. The training dataset available for depth estimation presents substantial challenges for use in such training. The available depth maps are noisy and often contain regions with missing depth values.

[0126] Reasons for such holes include highly reflective surfaces, light-absorbing surfaces, and regions outside the measurement range of the sensor. Holes are largely inconsequential for simple feedforward networks or regression models, as the training process can be modified to backpropagate loss only from a subset of pixels with known depth values, ignoring those with missing depth. On the other hand, for diffusion models, this corruption of the training data is more problematic.

[0127] A diffusion model performs inference by iterative refinement (in the current case, iterative refinement of a depth map y conditioned on an RGB image x). It starts with a sample of Gaussian noise y1and terminates with a sample from the predictive distribution p(y0| x). The refinement steps from time t to s (where s < t) are performed by sampling from the parameterized distribution p θ (y s | y t,x) sampling. Simply put, during inference, each step operates on the output from the previous step. In contrast, during training, the different steps are somewhat decoupled (see Equation 1), where the denoising network operates on a noisy version of the ground truth depth map rather than the output of the previous iteration (reminiscent of the forced teaching in training RNNs). This introduces a distribution shift between training and inference, as the marginal distribution on noisy training depth maps with holes can be significantly different from the distribution of noisy depths at inference, which should ideally (since the model does not learn the distribution of holes in the loss) be a noisy version of the true full depth map (e.g., from a perfect noiseless sensor). This has a significant negative impact on model performance. 156 The presence of structured or heavy-tailed noise in the training depth maps further exacerbates this problem.

[0128] These issues were effectively mitigated by the following modifications during training:

[0129] Deep Interpolation . To reduce the distribution shift between training and inference, missing depth values ​​are estimated in the training dataset. Several ways of achieving this are evaluated, including various interpolation schemes, and including using DepthGen (which fills in the training with nearest neighbor interpolation) to fill in the missing depths. Empirically, two straightforward steps are found to perform roughly as well as more complex methods. In particular, our results show that nearest neighbor interpolation is sufficient to estimate missing depths for indoor training data. For outdoor training data, nearest neighbor interpolation is also used, except for the sky region, which is typically large and much farther from the camera than neighboring objects in the image. Instead, a sky segmenter model is used to identify the "sky" part of the training images, and based on this output, all sky pixels are set to the maximum modeled depth (here, 80m). Although the missing depths are estimated, the training loss is only computed at pixels with known (relative to the filled-in) depth.

[0130]

[0131] Unbiased y t Conditional control. Reduce y between training and inference t Another way to mitigate the bias in the potential marginal distribution of y is to construct y using the model output instead of the true depth. t This is done by modifying the training procedure (Algorithm 1) to run one forward pass of the model, and then construct y by adding noise to the output of the model instead of the original training depth map used to generate this first model output. tis completed. For this forward pass, no gradients are propagated. This slows down training on the TPU v4 by about 15%. The modified training method can be referred to as unbiased y t Conditional control.

[0132] This unbiased conditional control is only performed during fine-tuning, not during supervised deep pre-training. This is because, at early stages of training, the depth predictions are more likely to be less accurate. Therefore, the potential marginals on the noisy training depth maps will be closer to the desired true marginals than the marginals that would be produced by adding noise to the model’s output. Therefore, it is not recommended to perform unbiased conditional control at early stages of supervised pre-training. The curriculum can also be used to gradually introduce unbiased conditional control at later stages of supervised pre-training. This will also introduce an additional hyperparameter, so it is investigated here through experimentation.

[0133] Loss While the L2 loss in Equation 1 is suitable for noiseless training data with additive Gaussian noise, good performance has been reported with L1 loss during training of image-to-image translation models. Given the possibility of large amounts of noise in depth data, especially for large depths and close holes, the robustness provided by L1 loss is also useful in training the RGB-to-depth diffusion models (or RGB-to- optical flow diffusion models) described herein.

[0134] For unsupervised pre-training, the ImageNet-1K and Places365 datasets are used, and training is performed on self-supervised tasks of colorization, inpainting, dehazing, and JPEG decompression.

[0135] Indoor model: For supervised image-to-depth pre-training of the indoor model, the following two datasets are used (with batch-level dataset mixing):

[0136] ScanNet is a dataset of 2.5M images captured using a sensor similar to the Kinect v1. It provides 640x480 depth maps and 1296x968 RGB images.

[0137] SceneNet RGB-D is a synthetic dataset of 5M images generated by rendering ShapeNet objects in scenes from SceneNet at a resolution of 320x240.

[0138] For indoor fine-tuning and evaluation, the NYU Depth v2 dataset is used, which is a commonly used dataset for evaluating indoor depth prediction models. It provides aligned images and depth maps at 640x480 resolution. The official split is used in this paper, which consists of 50k images for training and 654 images for evaluation. Before evaluation, the predicted depth maps from DepthNet are resized to full resolution using bilinear upsampling. Evaluation is performed on the cropped region.

[0139] Outdoor model: For outdoor model training, the Waymo open dataset is used, which is a large-scale driving dataset consisting of approximately 200k frames. Each frame provides RGB images and LiDAR maps from 5 cameras. The RGB images from the front camera, left front camera, and right front camera and the depth map from the top LiDAR are used to construct ~600k aligned RGB depth maps.

[0140] For subsequent fine-tuning and evaluation, KITTI is used, which is an outdoor driving dataset that provides RGB images and LiDAR scans at a resolution close to 1226x370. The training / test split of 26k training images and 652 test images is used. Before evaluation, the predicted depth from DepthGen is upsampled to full resolution using bilinear interpolation. Evaluation is performed on the cropped region.

[0141] Data augmentation and preprocessing. Random horizontal flip data augmentation is used for supervised depth training. Images and depth maps are resized to the resolution of the model for training using bilinear interpolation if needed. The diffusion model expects input and generates output in the range [−1, 1]. For indoor models, a maximum depth of 10 meters is applied, and for outdoor models, the depth maps are normalized to a range with a maximum depth of 80 meters.

[0142]

[0143] Table 1. Performance comparison on the NYU-Depth-v2 dataset. indicates a method using unsupervised pre-training, indicates supervised pre-training, and indicates a method with supervised depth pre-training on auxiliary data. Best / Second Best / Third best results are represented in bold / underlined / italic, respectively. represents lower is better, and represents higher is better.

[0144] The dominant architecture for diffusion models is the U-Net developed for the DDPM model, and subsequently improved in several ways. For DepthGen, the Efficient U-Net architecture developed for Imagen is used. The Efficient U-Net architecture is more efficient than the UNets used in previous work because it has fewer self-attention layers, fewer parameters, and less computation at higher resolutions, along with other adjustments that make it well-suited for training medium-resolution diffusion models.

[0145] Several minor changes were made to the architecture to adapt it for image-to-depth models. The criss-cross attention layer was discarded, but the self-attention layer was retained. The efficient U-Net has six input channels and three output channels because the target is an RGB image (the input consists of a 3-channel source RGB image and a 3-channel noisy target image concatenated along the channel dimension). For deep models, since the output is a scalar output image, the architecture is modified to have four input channels and one output channel. To accomplish this modification, the input and output convolution kernels are reinitialized before the supervised deep pre-training phase.

[0146] Resolution. The retrained palette model was trained on images with a resolution of 256×256. For training the deep model, a resolution close to this was chosen while preserving the aspect ratio of the original deep training dataset. The indoor model was trained at 320×240. For Waymo, 384×256 was used, and for KITTI, 416×128 was used. This model does not contain learned position embeddings, so it is easy to pre-train and fine-tune at different resolutions.

[0147] Self-supervised models are trained for 2.8M steps with L2 loss and a mini-batch size of 512. Other hyperparameters are similar to those in the original palette publication. Deep models are trained with L1 loss. A constant learning rate of 1e-4 is used during supervised deep pre-training, switching to a slightly lower learning rate of 3e-5 during fine-tuning, which has been experimentally observed to achieve slightly better results. A learning rate warmup over 10k steps is used for all models. All deep models are trained with a smaller mini-batch size of 64. Indoor deep models are trained for 2M steps on a mixture of ScanNet and SceneNet RGBD, and then fine-tuned on NYU for 50k steps. Outdoor deep models are trained for 0.9M steps on Waymo and fine-tuned on KITTI for 50k steps. Other details such as the use of optimizers and EMA are similar to those outlined in Saharia et al., 2022.

[0148] A DDPM sampler with 128 denoising steps was used. Further increasing the number of denoising steps did not significantly improve performance.

[0149] For both the NYU Depth v2 and KITTI datasets, the absolute relative error (REL), root mean squared error (RMS), and accuracy metrics are reported (for i e 1,2,3, d i <1.25 i ). For NYU, the absolute error in log depth (log10) is also reported. For KITTI, the squared relative error (Sq-rel) and root mean squared error (RMS log) in log depth are also reported.

[0150] Table 1 shows results on NYU Depth v2. A state-of-the-art absolute relative error of 0.074 is obtained. Table 2 shows results on KITTI, where the embodiments described herein perform comparably to previous work. Results are reported for average depth maps from one or more samples. Note that most previous work reports averaging over two samples obtained by left-right reflection of the input image.

[0151]

[0152] Table 2. Performance comparison on KITTI dataset. indicates a method using unsupervised pre-training, indicates supervised pre-training, and indicates a method with supervised depth pre-training on auxiliary data. Best / Second Best / Third best results are indicated in bold / underlined / italic, respectively. denotes lower is better, and denotes higher is better. E-B5: EfficientNet-B5.

[0153]

[0154] Table 3. Ablations for pre-training (before fine-tuning), showing absolute relative error for single samples on NYU and KITTI.

[0155]

[0156] Table 4. Ablations for handling incomplete depths, showing absolute relative error for single samples on NYU and KITTI

[0157] Ablation experiments show that both pre-training and accounting for missing depths are contributors to model performance. Table 3 shows that both self-supervised pre-training and supervised depth pre-training are important, with supervised depth training having a greater impact. Table 4 shows that depth inpainting is very important for the outdoor KITTI dataset. It has a smaller impact on NYU, which can be related to KITTI having more sparse depth maps. Without inpainting of depth maps, the unbiased conditioning control improves results significantly, especially on KITTI. Even with depth inpainting, the unbiased conditioning control consistently improves performance on both indoor and outdoor datasets. Additionally, the choice of loss function is evaluated in Table 5. The LI loss produces better performance than L2, possibly because LI is more robust to noise at larger depths.

[0158]

[0159] Table 5. Ablation of LI vs. L2 loss, showing absolute relative error on individual samples on NYU and KITTI.

[0160] One advantage of the diffusion models as described herein is their ability to capture complex multi-modal distributions. This can be efficient in representing depth uncertainty, especially in cases where there can be natural depth ambiguities, e.g., in the case of transparency when looking through a window. Figure 6 and Figure 7 Multiple samples on the NYU and KITTI datasets are presented, which show that the models capture multi-modality in the sensor data and provide reasonable samples in cases of ambiguity.

[0161] Figure 6 Examples of multi-modal predictions on the NYU depth v2 val dataset. Rows 1-2 contain a glass door / window, where the model learns to predict the depth of the glass surface or the surface behind it. Row 3 has a dark region next to a refrigerator, for which the depth is unclear from the RGB alone. In row 4, the model hallucinates a reflected door as a bathroom cabinet, which seems reasonable from the RGB image.

[0162] Figure 7 Multi-modal depth predictions on the KITTI val dataset.

[0163] One advantage of the diffusion models is that one can easily estimate a part of an image (or depth map) zero-shot conditioned on the rest of the image (or depth map).

[0164] Here, this property is leveraged to construct a limited but efficient pipeline for text-to-3D scene generation. As Figure 9As depicted, given a text c, the Imagen text-to-image model is used to generate an image, to which DepthGen (zero-shot) is applied to sample the corresponding depth map. The "camera" is then "moved" and an RGBD point cloud is rendered from the new camera pose. Of course, this only provides the RGB and depth values at a subset of the pixels in the new frame, as the field of view is different. The missing pixels are inferred using the diffusion model (i.e., Imagen editor for RGB and DepthGen for depth).

[0165] Let x a and y a be the RGB and depth values at the pixel positions rendered from the new camera pose, respectively, and let x b and y b correspond to the lines of sight that are not visible in the original frame. First, the missing RGB values are inferred using the inpainting / cleanup capabilities of the Imagen editor, i.e., p(x b | x a , c). Then, the missing depth values are estimated using DepthGen, i.e., sampled from p(y b | y a , [x a , x b ]).

[0166] An alternative approach is to sample the unknown depth y a given the existing depth y a and image x = [x b , x b ].

[0167] Figure 8 : Text-to-3D sampling. Given a textual prompt, first an image is generated using Imagen (first row of the first column), after which the depth is estimated (second row of the first column). Subsequently, the "camera" is "moved" to reveal new parts of the scene, which are then filled in using an image completion model and DepthGen (which conditions on both the incomplete depth map and the filled-in image). At each step, the newly generated RGBD points are added to a global point cloud, which is visualized in the rightmost column.

[0168] Figure 9 : Pipeline for iteratively generating a 3D scene conditioned on a text c = bedroom.

[0169] Second Experiment

[0170] Training of useful diffusion models for monocular depth and optical flow inference can be limited by the amount and quality of available training data for those tasks. Given the limited availability of labeled training data, a training pipeline as described herein is implemented and experimentally evaluated. The training pipeline includes multi-task self-supervised pre-training followed by supervised pre-training using a combination of real and synthetic data. The multi-task self-supervised pre-training leverages the strong performance of diffusion models on tasks such as colorization and inpainting. Experimental results provided herein show that supervised (pre)training using a combination of real and large-scale synthetic data significantly improves performance.

[0171] Furthermore, many existing real datasets for depth and optical flow have noisy and incomplete ground truth annotations. This poses challenges to the regular training framework and iterative sampling in diffusion models, resulting in distributional shifts between training and inference. To mitigate these issues, the present disclosure provides a number of improvements, including using L1 loss for robustness, imputing missing depth values during training, and gradual rollout of denoising diffusion. These aspects of the embodiments described herein are shown to be important for both depth and flow estimation through ablation experiments.

[0172] The present disclosure provides, among other things:

[0173] Utilizing generative diffusion models to formulate optical flow and monocular depth estimation as image-to-image translation without specialized loss functions and model architectures.

[0174] A number of solutions to several important issues regarding training data for training such models. For both tasks, to mitigate training and distributional shifts between inference for noisy, incomplete data, we impute, gradually rollout, and L1 loss during training can be used to provide improved results. For flow, to improve generalization, a new dataset mix for pre-training is disclosed herein, resulting in a RAFT baseline that outperforms all published methods in terms of zero-shot performance on Sintel and KITTI training benchmarks.

[0175] For both tasks, the diffusion models described herein are on par or outperform the SOTA. For monocular depth estimation, a SOTA relative error of 0.074 is obtained on the NYU dataset, and comparable performance is obtained on KITTI. For flow, the diffusion outperforms the stronger RAFT baseline in pre-training by a large margin, and the fine-tuned model achieves a 3.26% Fl-all outlier rate on the public KITTI test benchmark, ~25% lower than the best published method.

[0176] The diffusion models described herein are also shown to capture flow and depth uncertainty, and the iterative denoising process enables zero-shot, coarse-to-fine refinement, and estimation.

[0177] Figure 10 : Examples of multimodal predictions on depth (NYU) and optical flow (Sintel and KITTI). Each row shows the input image (or two images superimposed for optical flow), a variance heatmap from 8 samples, and 3 individual samples. The model presented in this paper captures multimodal samples in uncertain / ambiguous situations, such as reflective (e.g., mirrors on NYU), transparent (e.g., car windows on KITTI), and translucent (e.g., fog on Sintel) areas. Due to inaccurate estimates, there is also high variance near object boundaries, which is typically a challenging case for optical flow, and also partially stems from the noisy ground-truth measurement of depth.

[0178] Figure 11 : Training architecture. Given a ground truth stream / depth, missing values ​​are first filled using interpolation. Noise is then added to the label map and used to train a neural network to model the conditional distribution of noise given an RGB image, noisy labels, and timestamps. The denoising step can optionally be unrolled during training (with gradient stopping) to bridge the gap between y t The distribution gap between training and inference.

[0179] Figure 12 The effect of adding synthetic datasets to pre-training. Diffusion models trained with AutoFlow (AF) alone tend to provide very coarse flow estimates and may hallucinate shapes. The addition of FlyingThings (FT), Kubric (KU), and TartanAir (TA) removes the bias towards polygonal regions caused by AF and significantly improves flow quality on fine details such as trees, thin structures, and motion boundaries.

[0180] In contrast to conventional monocular depth and optical flow methods that rely on extensive use of domain expertise in their architectural design, the embodiments described herein include simple, general-purpose architectures and loss functions. In this embodiment, the inductive bias of state-of-the-art architectures and losses is replaced by a powerful generative model combined with a combination of self-supervised pre-training and supervised training on both real and synthetic data.

[0181] Denoising Diffusion Model ( Figure 11 ) takes as input a noisy version of the target map (depth or optical flow map) along with a conditional control signal (one RGB image for depth, two RGB images for flow). The denoiser provides a noise-free estimate of the target map (which can be provided as an estimate of the noise in the input, an estimate of the signal in the input, an estimate of a specified weighted combination of noise and signal in the input, or some other representation of the target map, depending on the specific loss parameterization used). The training loss penalizes residual error in the denoised map, unlike the typical image reconstruction loss used in optical flow estimation.

[0182] Because training data with ground truth annotations is limited for many dense vision tasks, the experiments and embodiments described herein extensively use synthetic data. AutoFlow has recently emerged as a powerful synthetic dataset for training flow models. However, training on AutoFlow alone is not sufficient because diffusion models appear to use a large portion of their representational capacity to represent the shapes of the AutoFlow regions rather than directly solving for the correspondence. Thus, models trained on AutoFlow alone exhibit strong bias to generate flow fields with polygonal regions very similar to those present in the AutoFlow dataset, often ignoring the shape of the boundary in the two-frame RGB input (see, e.g., Figure 12 ).

[0183] To mitigate the bias caused by AutoFlow in training, three additional synthetic datasets, FlyingThings3D, Kubric, and TartanAir, are used during training. For computational efficiency, the model is started from a pre-training on AutoFlow, then a greedy mixing strategy is used where the relative ratio of the previous mix is fixed and the proportion of the newly added dataset is adjusted. Zero-shot testing of the model on Sintel and KITTI (see Table 6 and Figure 12 ) shows significant performance gains for each of the additional synthetic datasets.

[0184] Pre-training is also important for depth estimation (see Table 12). Separate indoor and outdoor models are trained. For the indoor model, a mix of ScanNet and SceneNet RGB-D is used for pre-training. The outdoor model is pre-trained on the Waymo Open Dataset.

[0185]

[0186] Ground truth annotations for real-world depth or flow data are often sparse and noisy due to, e.g., highly reflective surfaces, light-absorbing surfaces, dynamic objects, etc. While regression-based methods can simply compute loss on pixels with valid ground truth, the corruption of the training data is more challenging for diffusion models. A diffusion model performs inference by iterative refinement of a target image y conditioned on RGB image data x. It starts with a sample of Gaussian (or otherwise distributed) noise yi and terminates with a sample from the predicted distribution The refinement step from time t to s, where s < t, is given by sampling from the parameterized distribution Sampling proceeds; each step operates on the output from the previous step. However, during training, the denoising step is decoupled (see Equation 1), where the denoising network operates on a noisy version of the ground truth depth map, not the output of the previous iteration (reminding one of the teacher forcing in RNN training). Thus, there is a distributional shift between the marginals on the noisy target map during training and inference, as the ground truth map has missing annotations and heavy-tailed sensor noise, while the noisy map obtained from the previous time step at inference time should not. This distributional shift can have a negative impact on model performance. However, these issues can be mitigated effectively with the following modifications during training.

[0187] Padding One way to reduce the distributional shift is to estimate the missing ground truth. Several methods were evaluated, including a simple interpolation scheme and inference using the model (trained with nearest neighbor interpolation). Nearest neighbor interpolation was sufficient to estimate the missing values in the ground truth map in the depth and flow training data.

[0188] Despite estimating the missing ground truth depth and flow values, the training loss was only computed and backpropagated from pixels with known (unpadded) ground truth depth. This is referred to as masked denoising loss (see Equation 1). Figure 11 ).

[0189] Stepwise Unfolding Denoising Diffusion Training Mitigating y t A second way to reduce the distributional shift in the marginals is to construct y t from the model output, not the ground truth map. This can be done by slightly modifying the training procedure (see Algorithm 1) to run one forward pass of the model and construct y t by adding noise to the output of the model, not the training map. For this forward pass, gradients are not propagated. This procedure, which can be referred to as stepwise unrolled denoising diffusion, only slightly slows down training (~15% on TPU v4).

[0190] Stepwise unrolled denoising diffusion is only performed during model fine-tuning. At the early stages of training, the denoising predictions are inaccurate, so the potential marginals on the noisy target map will be closer to the desired true marginals than the marginals produced by adding noise to the denoiser network output.

[0191] L1 Denoiser Loss While the L2 loss in Equation 1 is ideal for Gaussian noise and noiseless ground truth maps, in practice, real ground truth depth and flow fields are noisy and heavy-tailed; e.g., for distant objects, near object boundaries, and near pixels with missing annotations. Thus, the robustness provided by the LI loss can be useful when training neural denoising networks.

[0192] Figure 13Visual results comparing RAFT to the methods described herein after pre-training. Note that we do much better on fine details and ambiguous regions

[0193] Figure 14 Visual results comparing RAFT to the methods described herein after fine-tuning. The methods described herein do better on fine details and ambiguous regions.

[0194] Training high-resolution diffusion models is typically slow and memory-intensive, but the estimation accuracy has been shown to improve with resolution. A simple solution is to perform inference in a coarse-to-fine manner, first estimating the flow over the entire field of view at low resolution, and then refining the estimate in a patch-wise manner. To refine, first upsample the low-resolution map to the target resolution using bicubic interpolation. Crop patches from the upsampled map, representing starting at time z, and the corresponding RGB input. Then use the noisy map Performing diffusion model inference. For simplicity, t0 is a fixed hyperparameter set based on the validation set. This process is performed on multiple overlapping patches. The patch estimates are then merged using a weighted mask with lower weights near patch boundaries, as predictions at the boundaries are more prone to error.

[0195] The denoiser backbone described herein uses an efficient UNet architecture, which is pre-trained with palette-wise self-supervised pre-training, and slightly modified to have appropriate input and output channels for each task. Since the diffusion model expects the input and generates an output in the range [−1, 1], the depth is normalized against the maximum depth of 10 meters and 80 meters for indoor and outdoor models, respectively. The optical flow is normalized using the height and width of the ground truth.

[0196] Optical Flow Pre-training is performed at 320x448 resolution, and zero-shot results on the widely used Sintel and KITTI datasets are reported. The model is further fine-tuned on a mix consisting of AutoFlow, FlyingThings, VIPER, HD1K, Sintel, and KITTI at 320x768 resolution. The mean endpoint error (AEPE) metric is used, which computes the L2 distance between ground truth and prediction. On KITTI, the outlier rate, Fl-all, is also computed, which reports the outlier rate in % of all pixels with valid ground truth, where an estimate is considered an outlier if its error exceeds 3 pixels and 5% relative to the ground truth.

[0197] DepthThe indoor and outdoor models are pre-trained on respective pre-training datasets. The indoor depth model is then fine-tuned and evaluated on the NYU Depth v2 dataset, and the outdoor model is fine-tuned and evaluated on the KITTI depth dataset. For both NYU Depth v2 and KITTI, the absolute relative error (REL), root mean squared error (RMS), and accuracy metric (d1<1.25) are reported.

[0198]

[0199] Table 6: Zero-shot optical flow estimation results on Sintel and KITTI. Using the proposed pre-training hybrid provides a new RAFT baseline that significantly improves upon the original. On Sintel.final and KITTI, the diffusion model described herein even outperforms this much stronger baseline and achieves state-of-the-art zero-shot results.

[0200]

[0201] Table 7: Optical flow fine-tuning evaluation on public benchmark datasets (AEPE for Sintel and Fl-all for KITTI). Bold indicates best, Underline second best. Additional datasets (AutoFlow and VIPER) are used on top of the default (FlyingThings, HD1K, KITTI, and Sintel). Warm start

[0202]

[0203] Table 8: Performance comparison on NYU-Depth-v2 and KITTI datasets. indicates a method that uses unsupervised pre-training, indicates supervised pre-training, and indicates the use of auxiliary supervised depth data. Best Second Best results are indicated with bold / underlined, respectively. : lower is better, : higher is better.

[0204] DepthTable 8 reports results on NYU Depth v2 and KITTI. State-of-the-art absolute relative error of 0.074 is achieved on NYU Depth v2. On KITTI, the methods described herein perform on par with previous work. Results are reported with average depth maps from one or more samples. Note that most previous work uses post-processing that averages over two samples, one from the input image and the other based on its reflection about the vertical axis.

[0205] Flow Table 6 reports zero-shot results on Sintel and KITTI training datasets with ground truth provided, for the models described herein. The models are trained on a pre-training mix (AutoFlow (AF), FlyingThings (FT), Kubric (KU), and TartanAir (TA)) described herein. For a fair comparison, RAFT is retrained on this pre-training mix; this new RAFT model significantly outperforms the original RAFT model. The diffusion models described herein outperform this stronger RAFT baseline. It achieves state-of-the-art zero-shot results on both challenging Sintel Final and KITTI datasets.

[0206] Figure 9 A qualitative comparison of pre-trained models is provided. The methods described herein demonstrate finer details about both object and motion boundaries. In particular on KITTI, the models described herein recover fine details very well, e.g., about trees and layered motion between trees and background.

[0207]

[0208] Table 9: Ablation on filling and stepwise rollout. Without either, performance degrades. Without both, the optical flow model cannot train on KITTI

[0209]

[0210] Table 10: Coarse-to-fine refinement improves zero-shot optical flow estimation results on Sintel and KITTI, and Figure 15 qualitative improvements shown in FIG. 7.

[0211]

[0212] Table 11: Addition of optical flow synthetic datasets significantly improves zero-shot results on Sintel and KITTI.

[0213]

[0214] Table 12: The addition of synthetic depth data in pre-training significantly improves the fine-tuning performance on NYU.

[0215] The models described herein are further fine-tuned on a mix of AutoFlow, FlyingThings, HD1K, KITTI, Sintel, and VIPER datasets. Table 7 reports comparisons with state-of-the-art optical flow methods on the public benchmark datasets Sintel and KITTI. On KITTI, the methods described herein outperform all existing optical flow methods (even most scene flow methods that use stereo input) by a significant margin, and set the new state-of-the-art. On the challenging Sintel final, the methods described herein are comparable to other state-of-the-art models. Except for the method that uses a warm-start strategy, the methods described herein are only behind FlowFormer, which employs strong domain knowledge about optical flow (e.g., cost volume, iterative refinement, or attention layers for larger context) unlike the more general models described herein. However, on KITTI, the diffusion models described herein outperform FlowFormer by a large margin (30.34%).

[0216] Ablation studies:

[0217] Padding and gradual unrolling. The effect of padding and gradual unrolling is shown in Table 9. For depth, results of fine-tuning pre-trained models on the NYU and KITTI datasets are reported at the same resolution and augmentations as the best results. For flow, new models are trained on the KITTI training set alone at a resolution of 320x448 and the metrics reported on the KITTI val set (where nearest-neighbor resizing to the target resolution is the only augmentation). Results are reported with a single sample, and without coarse-to-fine refinement. Training on the original sparse data without padding and gradual unrolling leads to poor results, especially on KITTI where the ground truth is very sparse. Gradual unrolling helps stabilize training without requiring any additional data preprocessing. The largest gain comes from interpolating missing values in the sparse labels. Padding and gradual unrolling work well together, as the best results use both; padding (as an approximation) does not completely bridge the training-inference distribution shift of the noisy latent term.

[0218] Coarse to Fine Refinement . Figure 15 Coarse-to-fine refinement is shown to substantially improve the fine-grained details in the estimated optical flow field. It also improves the metrics for zero-shot optical flow estimation on both KITTI and Sintel, as shown in Table 10.

[0219] Figure 15: Visual results with and without coarse-to-fine refinement. For the pre-trained models described herein, refinement helps correct erroneous flows and add details to correct flows.

[0220] Figure 16 : Application of zero-shot depth completion with the models described herein by incorporating them into an iterative 3D scene generation pipeline. Starting from an initial image (optionally generated from a text-to-image model), the image-only conditioned depth map is sampled using the models described herein. The image-depth pair is added to the point cloud. Then images and depth maps (with holes) are iteratively rendered from this point cloud by moving the camera. The image holes are filled using an existing image inpainter (optionally text-conditioned) and then the models described herein are used with replacement guidance to estimate the missing depth (conditioned on the inpainted RGB image and known depth).

[0221] Dataset When pre-trained with different dataset mixtures, diffusion models sometimes capture regional boundaries and shapes at the expense of local texture variations (see, e.g., Figure 12 ). Models trained only on AutoFlow tend to provide very coarse flows and mimic the object shapes found in AutoFlow. The addition of FlyingThings, Kubric, and TartanAir removes this illusion and significantly improves fine details in the flow estimates (e.g., shadows, trees, thin structures, and motion boundaries) as well as a significant boost in accuracy (see Table 11). Similarly, mixing SceneNet RGB-D (a synthetic dataset) along with ScanNet provides performance improvements for the fine-tuning results on NYU Depth v2 as shown in Table 12.

[0222] One advantage of diffusion models is their ability to capture complex multi-modal distributions. This can be effective in representing uncertainty, especially in cases where there can be natural ambiguity and thus multiple predictions, e.g., in the case of transparent, semi-transparent, or reflective situations. Figure 10 Multiple samples on the NYU, KITTI, and Sintel datasets are presented showing that the models described herein capture multi-modality and provide reasonable samples when there is ambiguity.

[0223] Diffusion models trained to model the conditional distribution p(y|x) can be leveraged for zero-shot sampling from p(y|x, y partial ), where y partialis partially known. One approach to do so (referred to as the replacement approach for guidance) is to replace the known part of the latent yt at each inference step with a noisy latent built by applying a forward process to the known labels. Results of deep completion with replacement guidance are investigated and found to be very effective. This is illustrated by building a pipeline for iteratively generating 3D scenes conditioned on text prompts as shown in Figure 16

[0224] Third Experiment

[0225] Prior knowledge of different indoor and outdoor datasets (e.g., NYU for indoor and KITTI for outdoor) is not necessary to generate competent diffusion models to predict depth maps for both indoor and outdoor scenes conditioned on input RGB images. Models as described herein can leverage size-learned priors of objects, for example, to resolve ambiguity for general scenes with large depth variations. These benefits can be obtained by using more diverse input training data, by a combination of augmentations and real training data to allow the model to learn to generalize to a wide range of camera intrinsics (e.g., field of view (FOV)) rather than overfit to the typically fixed intrinsics within public benchmark datasets.

[0226] Provided herein are models that implement metric depth diffusion (DMD) that leverage image-to-image diffusion models to solve for zero-shot metric depth estimation. Such models are effective for several vision tasks without the need for task-specific models. Diffusion models are also shown to capture uncertainty in depth estimation and enable zero-shot depth completion. This motivates their use for metric depth.

[0227] The disclosed subject matter provides many benefits:

[0228] Simple model for joint indoor-outdoor that includes a general-purpose diffusion model rather than task or domain-specific architectures, such as binning or hybrid experts with separate heads for indoor and outdoor.

[0229] Zero-shot model - no need for domain-specific fine-tuning as is common (e.g., for KITTI and NYU).

[0230] Improved training: 1) log-depth to amplify indoor depths and better handle large changes in depth distribution of indoor and outdoor datasets; 2) crop-restore crop-augmentation scheme to augment training data with more diverse fields of view (FoVs) than the original training data with fixed FoVs.

[0231] ​The models as described herein outperform ZoeDepth on OOD datasets experimentally and show high efficiency with few inference steps. Performance is significantly improved with more training data than is typically used.

[0232]

[0233] Table 13. Quantitative results for zero-shot transfer to four unseen indoor datasets. For SUN RGB-D, the evaluation depth upper bound is 8m, for iBims and DIODE Indoor the evaluation depth upper bound is 10m, and for HyperSim the evaluation depth upper bound is 80m. The best and second best results are highlighted, with the former additionally being bolded. After pre-training, DMD-NK is trained on the union of NYUv2 and KITTI, while DMD-MIX adds Taskonomy and NuScenes to the training data. Fine-tuning on the same data, DMD-NK outperforms ZoeD-M12-NK. With more data, DMD-MIX outperforms ZoeD-M12-NK by an even greater margin.

[0234]

[0235] Table 14. Quantitative results for zero-shot transfer to four unseen outdoor datasets. The best and second best results are highlighted, with the former additionally being bolded. After pre-training, DMD-NK is trained on NYUv2 and KITTI, while DMD-MIX adds Taskonomy and NuScenes for training. Using the same fine-tuning data, DMD-NK outperforms ZoeD-M12-NK on all benchmarks (except DIODE Outdoor), with DMD-NK outperforming ZoeD-M12-NK on REL in DIODE Outdoor but falling behind ZoeD-M12-NK on other metrics. With the expanded training mix, DMD-MIX shows further performance improvement over DMD-NK.

[0236] Figure 17 : On eight zero-shot and two in-domain benchmarks, ) the indoor is on the left of the graph and the outdoor is on the right of the graph, the relative depth error (REL ) of the methods described herein compared to ZoeDepth. The methods described herein, DMD, outperform ZoeDepth by a considerable margin on all benchmarks.

[0237] Figure 18 : Qualitative examples of the methods described herein on multiple indoor datasets.

[0238] Figure 19 Qualitative examples of the methods described herein on multiple outdoor datasets.

[0239] As discussed above, key challenges in metric depth estimation stem from depth scale ambiguity and substantial differences between indoor and outdoor data. Prior methods thus either focus on scale and translation invariant depth estimation, or on separate models for indoor and outdoor with fixed camera intrinsics. The present disclosure describes a diffusion model with design decisions aimed at addressing these issues. In particular, monocular depth estimation is cast as a generative RGB-to-depth translation task using denoising diffusion. To this end, several modifications to the regular diffusion model and its training procedure are provided to accommodate zero-shot metric depth.

[0240] In contrast to DDVM, the embodiments described herein model depth on a log scale to adequately allocate modeling capacity for indoor and outdoor scenes. Additionally, v-parametrization is used instead of x- or Parametrization (used in DDVM) results in a more efficient denoiser that can denoise using fewer steps. Field-of-view augmentation, which crops / restores crops and conditionally controls the field-of-view, also allows the model to mitigate scale ambiguity issues. While the models trained herein on NYU and KITTI only already outperform the current SOTA for zero-shot metric depth estimation, the diversified fine-tuning mix further improves performance.

[0241] Diffusion models assume a process that gradually transforms a target distribution into a tractable noisy distribution, for which a neural denoising network is learned to reverse the process, slowly transforming noisy samples into samples from the target distribution. They have been shown to be very effective for images and videos, and they have recently started to be used for dense vision tasks like segmentation, tracking, optical flow, and depth estimation. They are attractive because they exhibit strong performance on regression tasks, capture posterior uncertainty, and do not have task-specific architectures, loss functions, and training procedures.

[0242] V-parametrization is used with a truncated SNR weighted loss with L1 norm, motivated by the finding that training with L1 is generally superior to L2 in the Parametrization case. In particular, the following loss is used:

[0243]

[0244] Training joint indoor-outdoor models can be difficult due to the large disparity in depth distribution found between indoor and outdoor scenes. Many available indoor training data have depths up to 10m, while outdoor scenes include ground truth depths up to 80m. Furthermore, training data often lack camera intrinsics and variation in field of view required for robustness to images from different cameras. Conversely, many datasets are captured with fixed cameras. To mitigate these issues, three improvements are provided: using log depth, field of view augmentation, and field of view conditioning.

[0245] Log Depth

[0246] Diffusion models typically model data distribution in [-1, 1]. The metric depth can be transformed to this range with a linear scaling, e.g.,

[0247] d lin = normalize(d r / d max ) (2)

[0248] where d r is the original depth in meters, d max can be taken as 80m to accommodate the usual outdoor depth range, and normalize(d) = clip(2 d , ,1). However, this allocates little representation capacity for indoor scenes (typically less than 10m).

[0249] Conversely, more representation capacity can be allocated to indoor scenes with a log scaling of depth (d log ) as the target of inference, e.g.,

[0250] (3)

[0251] where d min and d max represent the minimum and maximum supported depth (e.g., 0.5m and 80m). Log scaling was found to be very beneficial empirically.

[0252] Tables 15 and Figure 20 demonstrate this, showing improved quantitative and qualitative performance on indoor scenes with log depth instead of linear depth scaling.

[0253] Field of View AugmentationBecause the datasets used for depth estimation typically have little or no variation in the field of view, the model is prone to overfitting and then generalizes poorly to other camera intrinsics. To encourage the model to generalize well to different fields of view, the training data can be augmented by cropping or de-cropping to simulate different FOVs. While cropping is straightforward, to de-crop, the enlarged image can be padded in various ways. Preliminary experiments used generative de-cropping with a palette, however, it was found through experimentation that simply padding the RGB image with Gaussian noise (white noise, mean of zero, variance of 1) also works while requiring less computation.

[0254] For missing ground truth depth augmented with de-cropping, a combination of nearest neighbor imputation during training and stepwise unfolding denoising is used. This technique is effective in dealing with the inherent distribution shift between training and testing when the ground truth data is noisy or incomplete.

[0255] Field of View Conditioning Due to depth scale ambiguity, other information is needed. This can include conditioning the model on the FOV of the input image. Two scenarios are evaluated, one where prior knowledge of the camera intrinsics is available, as can be the case in autonomous driving or robotics, and one where the image has no available metadata or other associated FOV information, in which case the FOV can be predicted from the image.

[0256] Augmenting the training data FOV can not be enough to make the model robust to changes in FOV; conditioning on the FOV greatly improves the results. tan(θ / 2) is used as the conditional input to the model, where θ is the vertical FOV. Additionally, conditioning on the horizontal FOV is investigated, but this does not further improve the results. is the vertical FOV. Additionally, conditioning on the horizontal FOV is investigated, but this does not further improve the results.

[0257] ​DDVM shows that using a large variety of training data is especially important for general-purpose models without task-specific inductive bias. The models described in this paper are initialized with unsupervised pre-training on ImageNet and Places365 datasets for the various tasks proposed. For the supervised pre-training phase, ScanNet and SceneNet-RGBD are used for indoor training data, and the Waymo Open dataset is used for the outdoor training dataset, with the DIML indoor dataset added for further diversity. Note that FOV augmentation or conditioning is used at this stage. For the final fine-tuning phase, two models are trained. The main model is trained on a mix of NYU, Taskonomy, KITTI, and NuScenes datasets. However, to make a fair comparison with the current SOTAZoeDepth, another model is trained only on NYU and KITTI. FOV augmentation is applied to NYU, KITTI, and NuScenes. Since Taskonomy already has images with different FOVs, no further FOV augmentation is used for it.

[0258] A modified version of the efficient U-Net architecture proposed in DDVM is used as the neural denoiser. The architecture is modified to support FOV conditioning. In particular, the FOV embedding is constructed in a similar way to the time step embedding by first building a sin-cos position embedding and then performing a linear projection. This embedding is added to the time step embedding, and the sum is used to modulate different layers of the denoiser backbone using FiLM layers. Training is performed at a resolution of 384x512. The predicted depth maps are resized to the ground truth resolution for evaluation.

[0259] Figure 20 : Linear scaling depth leads to noisy predictions for images with shallow depth.

[0260]

[0261] Table 15. Ablation shows that log depth improves quantitative performance. This affects datasets with relatively shallow depth, such as indoor datasets and DIML Outdoor, which is an outdoor dataset but has relatively shallow depth, in particular.

[0262]

[0263] Table 16. Ablation shows the relative error of models trained with and without field of view conditioning. Simply augmenting the field of view when training is not enough to make the model robust to changes in the field of view and also requires conditioning on the field of view.

[0264] In addition to FOV augmentation, random horizontal flip augmentation is also used.

[0265] Figure 21 : A plot showing the effect of perturbing the FOV during inference. Best performance is at or near the true FOV. Performance degrades with greater perturbation.

[0266] There were eight denoising steps for the indoor dataset and two denoising steps for the outdoor dataset using the DDPM sampler. Results are reported using the mean of eight samples.

[0267] As shown in Table 21, the samples resulted in a small improvement in performance on average.

[0268] Tables 13 and 14 report zero-shot performance on the eight datasets. To be fair to ZoeDepth, the model was fine-tuned on NYU and KITTI only (referred to as DMD-NK). This model outperformed ZoeDepth on all datasets except DIODE Outdoor, where it outperformed ZoeDepth on relative error but fell behind on RMSE and 1.

[0269] Further improvements in performance are easily obtained by fine-tuning on larger datasets. For example, a variant model trained on a mix of KITTI, NYU, NuScene, and Taskonomy (referred to as DMD-MIX) generalizes better on OOD test data, establishing a new state-of-the-art.

[0270] Tables 19 and 20 show results on the NYU and KITTI datasets, respectively. The models described herein outperform ZoeDepth on relative error and are competitive on other metrics. DMD-MIX also outperforms DMD-NK on in-distribution data.

[0271] For computational efficiency and convenience, only DMD-NK is executed for ablation.

[0272] Log vs Linear Scaled Depth Table 15 shows that modeling depth on a log scale improves quantitative performance, especially on datasets with shallow depths. Furthermore, as shown in Figure 20 , using linear scaling results in noisy artifacts in depth estimates for indoor scenes.

[0273] Field of View Conditioning Conditioning on FOV is necessary to achieve optimal performance. Simply augmenting a dataset (baseline) with FOV augmentation does not result in maximum performance. Figure 21 Perturbing the conditions at inference controls the FOV, and it is found that optimal performance occurs at or near the true FOV.

[0274]

[0275] Table 17. Comparison with ZoeDepth without scene type supervision. ZoeDepth performance degrades significantly when the scene type (indoor or outdoor) is not provided. The model described herein learns well without this supervision.

[0276] No FOV Augmentation or Conditioning ZoeDepth found that without supervision of the scene type by an expert, referred to as Auto Router, their model’s performance degrades even within the domain. To compare with ZoeDepth in this setting, the model was fine-tuned on NYU and KITTI without FOV augmentation or conditional control. The model performs very well in this setting within the domain (Table 17). However, the out-of-domain performance still lags behind the model with FOV augmentation and conditional control.

[0277] Diffusion Parameterization Using v-parametrization drastically reduces the number of denoising steps required to achieve good performance. As shown in Table 18, v-parametrization requires 64 denoising steps to be comparable in performance to the v-parametrization model using a single denoising step.

[0278]

[0279] Table 18. Quantitative comparison on NYU-Depth v2. The reported values for the prior art are from the corresponding original papers. The best result is in bold, and the second best result is underlined. Comparison of the relative error on the NYU and KITTI datasets for the models trained with the - and v-parametrization. Both models were fine-tuned only on NYU and KITTI without FOV augmentation or conditional control.

[0280]

[0281] Table 19. Quantitative comparison on NYU-Depth v2. The reported values for the prior art are from the corresponding original papers. The best result is in bold, and the second best result is underlined.

[0282] While the RGB camera intrinsics are available for most practical uses of monocular depth estimators (e.g. a phone, a robotic platform, or an autonomous car), they can sometimes be unknown (e.g. internet images or generated imagery). For these, a FOV estimator can be used. The naive FOV predictor works well and is comparable to the results using the true FOV.

[0283]

[0284] Table 20. Comparison of performance on the KITTI dataset. > indicates that the method uses unsupervised pre-training, indicates supervised pre-training, and Indicates use of auxiliary supervised depth data. Best Second Best Results are indicated in bold / underlined, respectively. : lower is better, and : higher is better.

[0285]

[0286] Table 21. Averaging over multiple samples leads to small but consistent improvements in both REL and RMS.

[0287] VI. CONCLUSION

[0288] The particular arrangements shown in the Figures should not be viewed as limiting. It should be understood that other embodiments can include more or less of each of the elements shown in a given figure. In addition, some of the illustrated elements can be combined or omitted. Further, exemplary embodiments can include elements that are not illustrated. In addition, while a variety of aspects and embodiments have been disclosed, it will be apparent to those of ordinary skill in the art that many more embodiments can be created without departing from the spirit or scope of the subject matter presented herein. The various aspects and embodiments disclosed herein are for purposes of illustration only and are not intended to be limiting in any way. For example, the various aspects and embodiments disclosed herein can be used in combination with each other. Other embodiments will be apparent to those of ordinary skill in the art from consideration of the specification and practice of the aspects and embodiments presented herein. Other embodiments can be employed without departing from the spirit or scope of the subject matter presented herein. Further, features and elements that are common between the various aspects and embodiments presented herein can and often are combined or interchanged, one with the other, and all are expressly contemplated. Furthermore, it is understood that the aspects and embodiments presented herein are not limited to the particular methodologies, materials and modifications described herein, as these can vary. It is also understood that the terminology used herein is for the purpose of describing only the particular aspects and embodiments, and is not intended to be limiting.

[0289] In addition, while various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those of ordinary skill in the art. The various aspects and embodiments disclosed herein are for purposes of illustration only and are not intended to be limiting in any way. The true scope and spirit of the subject matter presented herein is reflected in the following claims, along with the full scope of equivalents to which such claims are entitled. Other embodiments can be employed without departing from the spirit or scope of the aspects and embodiments presented herein. It will be readily understood to those skilled in the art that aspects and embodiments, as generally described herein, and illustrated in the Figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are implicitly contemplated herein.

[0290] VII. ENUMERATED EXAMPLE EMBODIMENTS

[0291] Accordingly, embodiments of the disclosure can relate to one of the following enumerated example embodiments (EEE). It will be understood that features indicated with respect to one EEE can be combined with other EEEs.

[0292] EEE 1 is a computer-implemented method comprising: training a machine learning model to predict a depth map of a color image via an iterative diffusion inference process, wherein training the machine learning model comprises: (i) adding noise to a first depth map to generate a second depth map, wherein the first depth map represents depths on a first scene depicted in a first color image; (ii) applying the first color image and the second depth map to the machine learning model to generate a third depth map; (iii) generating a plurality of loss values based on the third depth map; and (iv) updating the machine learning model based on the plurality of loss values.

[0293] EEE 2 is the computer-implemented method of EEE 1, further comprising: predicting, using the trained machine learning model via an iterative diffusion inference process for a plurality of iterations, a first target depth map for the target color image by applying the target color image and a corresponding input depth map estimate to the trained machine learning model to generate a corresponding output depth map, wherein an input depth map for a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noisy depth map, wherein an input depth map for each iteration of the plurality of iterations other than the first iteration is the output depth map of an immediately preceding iteration with an amount of noise added, and wherein the first target depth map is the output depth map of a final iteration of the plurality of iterations.

[0294] EEE 3 is the computer-implemented method of EEE 2, further comprising: generating a novel view of a target scene depicted in the target color image by: (i) rendering a substitute color image of the target scene using the first target depth map and the target color image, wherein the substitute color image includes holes representing aspects of the target scene that are occluded from view in the target color image; (ii) rendering a substitute depth map of the target scene as viewed from the perspective of the substitute color image using the first target depth map, wherein the substitute depth map includes holes representing aspects of the target scene that are occluded from view in the first target depth map; (iii) filling in the holes of the substitute color image; and (iv) after filling in the holes of the substitute color image, filling in the holes of the substitute depth map using the trained machine learning model via an iterative diffusion inference process for a plurality of iterations by applying the substitute color image and a corresponding input depth map estimate to generate a corresponding output depth map, wherein an input depth map for a first iteration of the plurality of iterations is the substitute depth map with an amount of noise added, and wherein an input depth map for each iteration of the plurality of iterations other than the first iteration is the output depth map of an immediately preceding iteration with an amount of noise added.

[0295] EEE 4 is the computer-implemented method of any preceding EEE, wherein generating the plurality of loss values comprises: generating a plurality of LI loss values between the third depth map and the first depth map or between the third depth map and the noise added to the first depth map to generate the second depth map.

[0296] EEE 5 is the computer-implemented method of any preceding EEE, further comprising: (1) training a precursor machine learning model to predict, via the iterative diffusion inference process, a restored color image of a degraded color image, wherein training the precursor machine learning model comprises: (i) adding noise to a first source color image to generate a second source color image; (ii) applying the second source color image and a third source color image to the precursor machine learning model to generate a fourth source color image, wherein the third source color image is a degraded version of the first source color image that has been degraded by at least one of: (a) being cropped, (b) being desaturated, (c) having image compression artifacts added thereto, or (d) having one or more missing pixel regions added thereto; and (iii) updating the precursor machine learning model based on a plurality of loss values generated based on a difference between the fourth source color image and the first source color image; and (2) modifying the precursor machine learning model to receive an input color image and an input depth map and generate therefrom an output depth map based on the input depth map and conditioned on the input color image, wherein training the machine learning model to predict, via the iterative diffusion inference process, a depth map of a color image comprises using the modified precursor machine learning model as an initial state of the machine learning model.

[0297] EEE 6 is the computer-implemented method of EEE 5, wherein modifying the precursor machine learning model to receive an input color image and an input depth map and generate therefrom an output depth map based on the input depth map and conditioned on the input color image comprises: (i) dropping, from the precursor machine learning model, at least one three-channel input kernel that receives the second source color image and at least one three-channel output kernel that generates the fourth source color image; and (ii) adding and initializing at least one single-channel input kernel that receives the second depth map and at least one single-channel output kernel that generates the third depth map.

[0298] EEE 7 is the computer-implemented method of any preceding EEE, wherein the machine learning model has a U-Net architecture.

[0299] EEE 8 is the computer-implemented method of any preceding EEE, wherein training the machine learning model further comprises, prior to adding noise to the first depth map to generate the second depth map, inpainting at least one hole in the first depth map.

[0300] EEE 9 is the computer-implemented method of EEE 8, wherein inpainting the at least one hole in the first depth map comprises using a nearest neighbor inpainting method to inpaint pixels of the at least one hole in the first depth map.

[0301] EEE 10 is the computer-implemented method of any of EEEs 8-9, wherein filling in the at least one hole in the first depth map includes (i) identifying a region of the first depth map representing sky based on the first color image, and (ii) filling in pixels of the region of the first depth map representing sky with a specified depth value.

[0302] EEE 11 is the computer-implemented method of any of EEEs 8-10, wherein generating the plurality of loss values based on the third depth map includes: (i) adding noise to the third depth map to generate a fourth depth map; (ii) applying the first color image and the fourth depth map to the machine learning model to generate a fifth depth map; and (iii) generating the plurality of loss values based on a difference between the fifth depth map and the third depth map or a difference between the fifth depth map and the noise added to the third depth map to generate the fourth depth map.

[0303] EEE 12 is the computer-implemented method of any of EEEs 1-7, wherein generating the plurality of loss values based on the third depth map includes: (i) adding noise to the third depth map to generate a fourth depth map; (ii) applying the first color image and the fourth depth map to the machine learning model to generate a fifth depth map; and (iii) generating the plurality of loss values based on a difference between the fifth depth map and the third depth map or a difference between the fifth depth map and the noise added to the third depth map to generate the fourth depth map.

[0304] EEE 13 is the computer-implemented method of any of EEEs 1-12, wherein training the machine learning model to predict a depth map of a color image via an iterative diffusion inference process includes training the machine learning model to predict a log-distance metric depth map of a color image via an iterative diffusion inference process, wherein the first depth map is a first log-distance depth map, wherein the second depth map is a second log-distance depth map, wherein the first log-distance depth map represents a log of a depth across a first scene depicted in the first color image, and wherein the third depth map is a third log-distance depth map.

[0305] EEE 14 is the computer-implemented method of EEE 13, wherein training the machine learning model to predict the log-distance metric depth map of the color image via the iterative diffusion inference process comprises training the machine learning model to predict the log-distance metric depth map of the color image and an associated field of view (FOV) measurement of the color image via the iterative diffusion inference process, wherein applying the first color image and the second log-distance depth map to the machine learning model to generate the third log-distance depth map comprises applying the first color image, the second log-distance depth map, and a first FOV measurement to the machine learning model to generate the third log-distance depth map, and wherein the first FOV measurement represents an angular field of view across at least one dimension of the first color image.

[0306] EEE 15 is the computer-implemented method of EEE 14, further comprising: (i) training a predecessor machine learning model to predict the log-distance metric depth map of the color image via the iterative diffusion inference process; and (ii) augmenting the trained predecessor machine learning model to receive a FOV measurement as input in addition to the color image and the log-distance depth map, wherein training the machine learning model to predict the log-distance metric depth map comprises training the trained, augmented predecessor machine learning model.

[0307] EEE 16 is the computer-implemented method of EEE 15, wherein augmenting the trained predecessor machine learning model to receive the FOV measurement as input in addition to the color image and the log-distance depth map comprises: (i) adding a sinusoidal positional embedding of the FOV measurement input; and (ii) adding a linear projection of an output of the sinusoidal positional embedding to one or more layers of the trained predecessor machine learning model.

[0308] EEE 17 is the computer-implemented method of any of EEEs 14-16, wherein the first FOV measurement represents a tangent function of an angular field of view across at least one dimension of the first color image.

[0309] EEE 18 is the computer-implemented method of any of EEEs 14-17, wherein training the machine learning model to predict, via the iterative diffusion inference process, the log-distance metric depth map of the color image and the associated field of view (FOV) measurement of the color image comprises training the machine learning model to predict, via the iterative diffusion inference process, the log-distance metric depth map of the color image, the associated field of view (FOV) measurement of the color image, and an associated noise input, wherein applying the first color image, the second log-distance depth map, and the first FOV measurement to the machine learning model to generate a third log-distance depth map comprises applying the first color image, the second log-distance depth map, the first FOV measurement, and a first noise estimate to the machine learning model to generate the third log-distance depth map, wherein the first noise estimate represents an amount of noise added to the first log-distance depth map to generate the second log-distance depth map, and wherein generating the plurality of loss values based on the third log-distance depth map comprises (i) determining a weighted combination of (a) the first log-distance depth map and (b) the noise added to the first log-distance depth map to generate the second log-distance depth map, and (ii) determining a pixel-wise difference between the weighted combination and the third log-distance depth map.

[0310] EEE 19 is the computer-implemented method of EEE 13, wherein training the machine learning model to predict, via the iterative diffusion inference process, the log-distance metric depth map of the color image comprises training the machine learning model to predict, via the iterative diffusion inference process, the log-distance metric depth map of the color image and an associated noise input, wherein applying the first color image and the second log-distance depth map to the machine learning model to generate a third log-distance depth map comprises applying the first color image, the second log-distance depth map, and a first noise estimate to the machine learning model to generate the third log-distance depth map, wherein the first noise estimate represents an amount of noise added to the first log-distance depth map to generate the second log-distance depth map, and wherein generating the plurality of loss values based on the third log-distance depth map comprises (i) determining a weighted combination of (a) the first log-distance depth map and (b) the noise added to the first log-distance depth map to generate the second log-distance depth map, and (ii) determining a pixel-wise difference between the weighted combination and the third log-distance depth map.

[0311] EEE 20 is the computer-implemented method of any of EEEs 13-19, further comprising determining the first log-distance depth map from the source distance depth map by: (i) determining a first intermediate depth map by dividing the source distance depth map by a minimum distance of the source distance depth map; (ii) determining a second intermediate depth map by determining a log of the first intermediate depth map; (iii) determining a third intermediate depth map by dividing the second intermediate depth map by a log of a ratio of a maximum distance of the source distance depth map to the minimum distance of the source distance depth map; and (iv) normalizing the third intermediate depth map to values of a finite range centered at zero.

[0312] EEE 21 is the computer-implemented method of any of EEEs 13-20, wherein training the machine learning model to predict a log-distance metric depth map of a color image via an iterative diffusion inference process comprises training the machine learning model using an augmented training dataset, wherein the augmented training dataset includes the first log-distance depth map and the first color image, and wherein the method further comprises augmenting a source training dataset to generate the augmented training dataset by cropping or un-cropping at least one of a source log-distance depth map and a source color image in at least one of a vertical direction or a horizontal direction to generate the first log-distance depth map and the first color image.

[0313] EEE 22 is the computer-implemented method of EEE 21, wherein un-cropping the cropped source log-distance depth map and the source color image in at least one of the vertical direction or the horizontal direction comprises applying a generative machine learning model to un-crop the cropped source log-distance depth map and the source color image.

[0314] EEE 23 is the computer-implemented method of EEE 21, wherein un-cropping the cropped source log-distance depth map and the source color image in at least one of the vertical direction or the horizontal direction comprises padding the source log-distance depth map and the source color image with noise in at least one of the vertical direction or the horizontal direction.

[0315] EEE 24 is the computer-implemented method of any of EEEs 13-23, wherein training the machine learning model to predict a log-distance metric depth map of a color image via an iterative diffusion inference process comprises training the machine learning model using a training dataset, and wherein the training dataset contains examples of images and corresponding depth maps of indoor scenes and outdoor scenes such that the machine learning model is trained to predict a log-distance metric depth map of a color image representing an indoor scene or an outdoor scene.

[0316] EEE 25 is the computer-implemented method of any of EEEs 1-12, wherein training the machine learning model to predict depth maps of color images via an iterative diffusion inference process comprises training the machine learning model to predict metric depth maps and associated field of view (FOV) measurements of color images via the iterative diffusion inference process, wherein applying the first color image and the second depth map to the machine learning model to generate a third depth map comprises applying the first color image, a first FOV measurement, and the second depth map to the machine learning model to generate the third depth map, and wherein the first FOV measurement represents an angular field of view across at least one dimension of the first color image.

[0317] EEE 26 is the computer-implemented method of EEE 25, further comprising: (i) training a predecessor machine learning model to predict metric depth maps of color images via an iterative diffusion inference process; and (ii) augmenting the trained predecessor machine learning model to receive FOV measurements as input in addition to color images and depth maps, wherein training the machine learning model to predict metric depth maps comprises training the trained augmented predecessor machine learning model.

[0318] EEE 27 is the computer-implemented method of EEE 26, wherein augmenting the trained predecessor machine learning model to receive FOV measurements as input in addition to color images and depth maps comprises: (i) adding a sinusoidal positional embedding of the FOV measurement input; and (ii) adding a linear projection of an output of the sinusoidal positional embedding to one or more layers of the trained predecessor machine learning model.

[0319] EEE 28 is the computer-implemented method of any of EEEs 25-27, wherein the first FOV measurement represents a tangent function of an angular field of view across at least one dimension of the first color image.

[0320] EEE 29 is the computer-implemented method of any of EEEs 25-28, wherein training the machine learning model to predict metric depth maps of color images via an iterative diffusion inference process comprises training the machine learning model to predict metric depth maps and associated noise inputs of color images via the iterative diffusion inference process, wherein applying the first color image and the second depth map to the machine learning model to generate a third depth map comprises applying the first color image, the second depth map, and a first noise estimate to the machine learning model to generate the third depth map, wherein the first noise estimate represents an amount of noise added to the first depth map to generate the second depth map, and wherein generating a plurality of loss values based on the third depth map comprises: (i) determining a weighted combination of (a) the first depth map and (b) the noise added to the first depth map to generate the second depth map, and (ii) determining a pixel-wise difference between the weighted combination and the third depth map.

[0321] EEE 30 is a computer-implemented method as in any of EEEs 25-29, wherein training the machine learning model to predict a metric depth map of a color image via an iterative diffusion inference process comprises training the machine learning model using an augmented training dataset, wherein the augmented training dataset comprises the first depth map and the first color image, and wherein the method further comprises augmenting a source training dataset to generate the augmented training dataset by cropping or un-cropping at least one of the source depth map and the source color image in at least one of a vertical direction or a horizontal direction to generate the first depth map and the first color image.

[0322] EEE 31 is a computer-implemented method as in EEE 30, wherein un-cropping the cropped source depth map and the cropped source color image in at least one of a vertical direction or a horizontal direction comprises applying a generative machine learning model to un-crop the cropped source depth map and the cropped source color image.

[0323] EEE 32 is a computer-implemented method as in EEE 30, wherein un-cropping the cropped source depth map and the cropped source color image in at least one of a vertical direction or a horizontal direction comprises padding the source depth map and the source color image with noise in at least one of a vertical direction or a horizontal direction.

[0324] EEE 33 is a computer-implemented method as in any of EEEs 25-32, wherein training the machine learning model to predict a metric depth map of a color image via an iterative diffusion inference process comprises training the machine learning model using a training dataset, and wherein the training dataset contains examples of images and corresponding depth maps of indoor scenes and outdoor scenes such that the machine learning model is trained to predict a metric depth map of a color image representing an indoor scene or an outdoor scene.

[0325] EEE 34 is a computer-implemented method as in any of EEEs 1-12, wherein training the machine learning model to predict a depth map of a color image via an iterative diffusion inference process comprises training the machine learning model to predict a metric depth map and an associated noise input of a color image via an iterative diffusion inference process, wherein applying the first color image and the second depth map to the machine learning model to generate the third depth map comprises applying the first color image, a first noise estimate, and the second depth map to the machine learning model to generate the third depth map, wherein the first noise estimate represents an amount of noise added to the first depth map to generate the second depth map, and wherein the iterative diffusion inference process further comprises determining a weighted combination of (i) the first depth map and (ii) the noise added to the first depth map to generate the second depth map, wherein generating the plurality of loss values comprises determining a pixel-wise difference between the weighted combination and the third depth map to generate the plurality of loss values.

[0326] EEE 35 is the computer-implemented method of EEE 34, wherein training the machine learning model to predict the metric depth map of the color image via the iterative diffusion inference process comprises training the machine learning model using an augmented training dataset, wherein the augmented training dataset includes the first depth map and the first color image, and wherein the method further comprises augmenting a source training dataset to generate the augmented training dataset by cropping or un-cropping at least one of the source depth map and the source color image in at least one of a vertical direction or a horizontal direction to generate the first depth map and the first color image.

[0327] EEE 36 is the computer-implemented method of EEE 35, wherein un-cropping the cropped source depth map and the cropped source color image in at least one of a vertical direction or a horizontal direction comprises applying a generative machine learning model to un-crop the cropped source depth map and the cropped source color image.

[0328] EEE 37 is the computer-implemented method of EEE 35, wherein un-cropping the cropped source depth map and the cropped source color image in at least one of a vertical direction or a horizontal direction comprises padding the source depth map and the source color image with noise in at least one of a vertical direction or a horizontal direction.

[0329] EEE 38 is the computer-implemented method of any of EEEs 34-37, wherein training the machine learning model to predict the metric depth map of the color image via the iterative diffusion inference process comprises training the machine learning model using a training dataset, and wherein the training dataset contains examples of images and corresponding depth maps of indoor scenes and outdoor scenes such that the machine learning model is trained to predict a metric depth map of a color image representing an indoor scene or an outdoor scene.

[0330] EEE 39 is a computer-implemented method comprising predicting, using a machine learning model, a first target depth map of a target color image via an iterative diffusion inference process for a number of iterations by: applying the target color image and a corresponding input depth map estimate to the machine learning model to generate a corresponding output depth map, wherein an input depth map of a first iteration of the number of iterations is a randomly or pseudo-randomly generated noisy depth map, wherein an input depth map of each iteration of the number of iterations other than the first iteration is the output depth map of the immediately preceding iteration with an amount of noise added, and wherein the first target depth map is the output depth map of a final iteration of the number of iterations.

[0331] EEE 40 is the computer-implemented method of EEE 39, further comprising generating a novel view of a target scene depicted in the target color image by: (i) rendering an alternative color image of the target scene using the first target depth map and the target color image, wherein the alternative color image includes holes representing aspects of the target scene that are occluded from view in the target color image; (ii) rendering an alternative depth map of the target scene viewed from the perspective of the alternative color image using the first target depth map, wherein the alternative depth map includes holes representing aspects of the target scene that are occluded from view in the first target depth map; (iii) inpainting the holes of the alternative color image; and (iv) after the holes of the alternative color image are inpainted, filling in the holes of the alternative depth map for a number of iterations via an iterative diffusion inference process using the machine learning model by: applying the alternative color image and a corresponding input depth map estimate to generate a corresponding output depth map, wherein the input depth map for a first iteration of the number of iterations is the alternative depth map with an amount of noise added, and wherein the input depth map for each iteration of the number of iterations other than the first iteration is the output depth map of the immediately preceding iteration with the amount of noise added.

[0332] EEE 41 is the computer-implemented method of any of EEEs 39-40, wherein the machine learning model has a U-Net architecture.

[0333] EEE 42 is the computer-implemented method of any of EEEs 39-41, further comprising generating a second target depth map for the target color image by: (i) upsampling the first target depth map to generate an upsampled depth map; and (ii) predicting a plurality of patches of the second target depth map for corresponding patches of the target color image using the machine learning model via an iterative diffusion inference process, wherein predicting a given patch of the second target depth map for a corresponding patch of the target color image using the machine learning model via the iterative diffusion inference process includes using a corresponding patch of the upsampled depth map as an input depth map for a first iteration of a number of iterations of the process.

[0334] EEE 43 is the computer-implemented method of any of EEEs 39-42, wherein the target color image represents an indoor scene, wherein the method additionally comprises predicting an additional target log distance metric depth map for an additional target color image using the machine learning model via an iterative diffusion inference process, and wherein the additional target color image represents an outdoor scene.

[0335] EEE 44 is the computer-implemented method of any of EEEs 39 to 43, wherein predicting, using the machine-learned model, the first target depth map of the target color image via the iterative diffusion inference process comprises predicting, using the machine-learned model, a first target log-distance metric depth map of the target color image via the iterative diffusion inference process, wherein the respective input depth map estimate is a respective input log-distance depth map, and wherein the respective output depth map is a respective output log-distance depth map.

[0336] EEE 45 is the computer-implemented method of EEE 44: wherein applying the target color image and the respective input log-distance depth map estimate to the machine-learned model to generate the respective output log-distance depth map comprises applying the target color image, a target FOV measurement, and the respective input log-distance depth map estimate to the machine-learned model to generate the respective output log-distance depth map, and wherein the target FOV measurement represents a tangent function of an angular field of view across at least one dimension of the target color image.

[0337] EEE 46 is the computer-implemented method of EEE 45, further comprising determining the target FOV measurement from the target image.

[0338] EEE 47 is the computer-implemented method of EEE 45 or 46, wherein the target FOV measurement represents a tangent function of an angular field of view across at least one dimension of the target color image.

[0339] EEE 48 is the computer-implemented method of any of EEEs 45 to 47, wherein applying the target color image, the target FOV measurement, and the respective input log-distance depth map estimate to the machine-learned model to generate the respective output log-distance depth map comprises applying the target color image, the target FOV measurement, a respective noise estimate from a noise schedule, and the respective input log-distance depth map estimate to the machine-learned model to generate the respective output log-distance depth map.

[0340] EEE 49 is the computer-implemented method of EEE 44, wherein applying the target color image and the respective input log-distance depth map estimate to the machine-learned model to generate the respective output log-distance depth map comprises applying the target color image, a respective noise estimate from a noise schedule, and the respective input log-distance depth map estimate to the machine-learned model to generate the respective output log-distance depth map.

[0341] EEE 50 is the computer-implemented method of any of EEEs 44-49, further comprising determining a linear measure depth map of the target color image by: (i) determining a first intermediate depth map by scaling and shifting the target log distance measure depth map; (ii) determining a second intermediate depth map by taking a power of the first intermediate depth map; and (iii) scaling the second intermediate depth map.

[0342] EEE 51 is the computer-implemented method of any of EEEs 39-43, wherein applying the target color image and the corresponding input depth estimate to the machine learning model to generate the corresponding output depth map comprises applying the target color image, a target FOV measure, and the corresponding input depth estimate to the machine learning model to generate the corresponding output depth map, and wherein the target FOV measure represents a tangent function of an angular field of view across at least one dimension of the target color image.

[0343] EEE 52 is the computer-implemented method of EEE 51, further comprising determining the target FOV measure from the target image.

[0344] EEE 53 is the computer-implemented method of any of EEEs 51 or 52, wherein the target FOV measure represents a tangent function of an angular field of view across at least one dimension of the target color image.

[0345] EEE 54 is the computer-implemented method of any of EEEs 51-53, wherein applying the target color image, the target FOV measure, and the corresponding input depth estimate to the machine learning model to generate the corresponding output depth map comprises applying the target color image, the target FOV measure, a corresponding noise estimate from a noise schedule, and the corresponding input depth estimate to the machine learning model to generate the corresponding output distance depth map.

[0346] EEE 55 is the computer-implemented method of any of EEEs 39-43, wherein applying the target color image and the corresponding input depth estimate to the machine learning model to generate the corresponding output depth map comprises applying the target color image, a corresponding noise estimate from a noise schedule, and the corresponding input depth estimate to the machine learning model to generate the corresponding output distance depth map.

[0347] EEE 56 is the computer-implemented method of any of EEEs 39-55, wherein the machine learning model has been trained as recited in any of EEEs 1-38.

[0348] EEE 57 is a computer-implemented method comprising training a machine learning model to predict an optical flow map for a pair of color images via an iterative diffusion inference process, wherein training the machine learning model comprises: (i) adding noise to a first optical flow map to generate a second optical flow map, wherein the first optical flow map represents an optical flow from a first color image of a scene to a second color image of the scene; (ii) applying the first color image, the second color image, and the second optical flow map to the machine learning model to generate a third optical flow map; (iii) generating a plurality of loss values based on the third optical flow map; and (iv) updating the machine learning model based on the plurality of loss values.

[0349] EEE 58 is the computer-implemented method of EEE 57, further comprising predicting a first target optical flow map for a target pair of color images via an iterative diffusion inference process for a plurality of iterations using the trained machine learning model by: applying the target pair of color images and a respective input optical flow map estimate to the trained machine learning model to generate a respective output optical flow map, wherein an input optical flow map for a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noise optical flow map, wherein an input optical flow map for each iteration of the plurality of iterations other than the first iteration is an output optical flow map of an immediately preceding iteration with an amount of noise added, and wherein the first target optical flow map is an output optical flow map of a final iteration of the plurality of iterations.

[0350] EEE 59 is the computer-implemented method of any of EEEs 57-58, wherein generating the plurality of loss values comprises generating a plurality of LI loss values between the third optical flow map and the first optical flow map or between the third optical flow map and the noise added to the first optical flow map to generate the second optical flow map.

[0351] EEE 60 is the computer-implemented method of any of EEEs 57-59, wherein the machine learning model has a U-Net architecture.

[0352] EEE 61 is a computer-implemented method as described in any one of EEEs 57 to 60, wherein training a machine learning model to predict an optical flow map for a pair of color images via an iterative diffusion inference process includes: training a machine learning model to predict an optical flow map for a pair of color images and an associated noise input via an iterative diffusion inference process, wherein applying a first color image, a second color image, and a second optical flow map to the machine learning model to generate a third optical flow map includes: applying the first color image, the second color image, a first noise estimate, and a second optical flow map to the machine learning model to generate the third optical flow map, wherein the first noise estimate represents an amount of noise added to the first optical flow map to generate the second optical flow map, and wherein the iterative diffusion inference process further includes: determining a weighted combination of (i) the first optical flow map and (ii) the noise added to the first optical flow map to generate the second optical flow map, wherein generating a plurality of loss values ​​includes determining a pixel-by-pixel difference between the weighted combination and the third optical flow map to generate the plurality of loss values.

[0353] EEE 62 is a computer-implemented method comprising: using a machine learning model to predict a first target optical flow map for a target color image pair via an iterative diffusion inference process for a plurality of iterations by the following steps: applying the target color image pair and the corresponding input optical flow map estimate to the machine learning model to generate a corresponding output optical flow map, wherein the input optical flow map for a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noise optical flow map, wherein the input optical flow map for each iteration of the plurality of iterations except the first iteration is the output optical flow map of the immediately preceding iteration to which an amount of noise is added, and wherein the first target optical flow map is the output optical flow map of a final iteration of the plurality of iterations.

[0354] EEE 63 is a computer-implemented method as described in EEE 62, wherein the machine learning model has a U-Net architecture.

[0355] EEE 64 is a computer-implemented method as described in any one of EEEs 62 to 63, further comprising generating a second target optical flow map for the target color image pair by: (i) upsampling the first target optical flow map to generate an upsampled optical flow map; and (ii) predicting multiple patches of the second target optical flow map of corresponding patches of the target color image pair using a machine learning model via an iterative diffusion inference process, wherein predicting a given patch of the second target optical flow map of the corresponding patch of the target color image pair using the machine learning model via an iterative diffusion inference process includes: using the corresponding patch of the upsampled optical flow map as the input optical flow map for the first iteration of multiple iterations of the process.

[0356] EEE 65 is the computer-implemented method of any of EEEs 62-64, wherein applying the target color image pair and the corresponding input optical flow map estimate to the machine learning model to generate a corresponding output optical flow map includes applying the target color image pair, the corresponding noise estimate from the noise schedule, and the corresponding input optical flow map estimate to the machine learning model to generate the corresponding output optical flow map.

[0357] EEE 66 is the computer-implemented method of any of EEEs 62-65, wherein the machine learning model has been trained as recited in any of EEEs 57-61.

[0358] EEE 67 is a computing device comprising one or more processors, wherein the one or more processors are configured to perform a method as recited in any preceding EEE.

[0359] EEE 68 is an article of manufacture including a non-transitory computer-readable medium having stored thereon program instructions that, when executed by a computing device, cause the computing device to perform operations to implement a method as recited in any of EEEs 1-66.

Claims

1. A computer-implemented method comprising: training a machine learning model to predict depth maps of color images via an iterative diffusion inference process, wherein training the machine learning model comprises: adding noise to a first depth map to generate a second depth map, wherein the first depth map represents depths across a first scene depicted in a first color image; applying the first color image and the second depth map to the machine learning model to generate a third depth map; generating a plurality of loss values based on the third depth map; and updating the machine learning model based on the plurality of loss values.

2. The computer-implemented method of claim 1, further comprising: predicting, using the trained machine learning model, a first target depth map of a target color image via the iterative diffusion inference process for a plurality of iterations by: applying the target color image and a corresponding input depth map estimate to the trained machine learning model to generate a corresponding output depth map, wherein an input depth map for a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noisy depth map, wherein an input depth map for each iteration of the plurality of iterations other than the first iteration is an output depth map of an immediately preceding iteration with an amount of noise added, and wherein the first target depth map is an output depth map of a final iteration of the plurality of iterations.

3. The computer-implemented method of claim 2, further comprising: generating a novel view of a target scene depicted in the target color image by: rendering an alternative color image of the target scene using the first target depth map and the target color image, wherein the alternative color image includes holes representing aspects of the target scene that are occluded from view in the target color image; rendering an alternative depth map of the target scene as viewed from a perspective of the alternative color image using the first target depth map, wherein the alternative depth map includes holes representing aspects of the target scene that are occluded from view in the first target depth map; filling in the holes of the alternative color image; and after filling in the holes of the alternative color image, filling in the holes of the alternative depth map using the trained machine learning model via the iterative diffusion inference process for a plurality of iterations by: applying the alternative color image and a corresponding input depth map estimate to generate a corresponding output depth map, wherein an input depth map for a first iteration of the plurality of iterations is the alternative depth map with an amount of noise added, and wherein an input depth map for each iteration of the plurality of iterations other than the first iteration is an output depth map of an immediately preceding iteration with an amount of noise added.

4. The computer-implemented method of claim 1, wherein, generating a plurality of loss values includes generating a plurality of LI loss values between the third depth map and the first depth map or between the third depth map and the noise added to the first depth map to generate the second depth map.

5. The computer-implemented method of claim 1, further comprising: training an ancestral machine learning model to predict a restored color image of a degraded color image via an iterative diffusion inference process, wherein training the ancestral machine learning model comprises: adding noise to a first source color image to generate a second source color image; applying the second source color image and a third source color image to the pre-decade machine learning model to generate a fourth source color image, wherein the third source color image is a degraded version of the first source color image that has been degraded by at least one of: (i) being cropped, (ii) being desaturated, (iii) having image compression artifacts added thereto, or (iv) having one or more missing pixel regions added thereto; and updating the pre-decade machine learning model based on a plurality of loss values generated based on a difference between the fourth source color image and the first source color image; and modifying the pre-decade machine learning model to receive an input color image and an input depth map and generate an output depth map therefrom based on the input depth map and conditioned on the input color image, wherein training the machine learning model to predict a depth map of a color image via an iterative diffusion inference process includes using the modified pre-decade machine learning model as an initial state of the machine learning model.

6. The computer-implemented method of claim 5, wherein, modifying the pre-decade machine learning model to receive an input color image and an input depth map and generate an output depth map therefrom based on the input depth map and conditioned on the input color image includes: dropping, from the pre-decade machine learning model, at least one three-channel input kernel that receives the second source color image and at least one three-channel output kernel that generates the fourth source color image; and adding and initializing at least one single-channel input kernel that receives the second depth map and at least one single-channel output kernel that generates the third depth map.

7. The computer-implemented method of claim 1, wherein, the machine learning model has a U-Net architecture.

8. The computer-implemented method of claim 1, wherein, training the machine learning model further includes: before adding the noise to the first depth map to generate the second depth map, filling in at least one hole in the first depth map.

9. The computer-implemented method of claim 8, wherein, filling in the at least one hole in the first depth map includes filling in pixels of the at least one hole in the first depth map using a nearest neighbor filling method.

10. The computer-implemented method of any one of claims 8 to 9, wherein, filling in the at least one hole in the first depth map includes (i) identifying, based on the first color image, a region of the first depth map that represents a sky, and (ii) filling in pixels of the region of the first depth map that represents the sky with a specified depth value.

11. The computer-implemented method of any one of claims 8 to 9, wherein, generating the plurality of loss values based on the third depth map includes: adding noise to the third depth map to generate a fourth depth map; applying the first color image and the fourth depth map to the machine learning model to generate a fifth depth map; and generating the plurality of loss values based on a difference between the fifth depth map and the third depth map or a difference between the fifth depth map and the noise added to the third depth map to generate the fourth depth map.

12. The computer-implemented method of any one of claims 1 to 7, wherein, generating the plurality of loss values based on the third depth map includes: adding noise to the third depth map to generate a fourth depth map; applying the first color image and the fourth depth map to the machine learning model to generate a fifth depth map; and generating the plurality of loss values based on a difference between the fifth depth map and the third depth map or a difference between the fifth depth map and the noise added to the third depth map to generate the fourth depth map.

13. The computer-implemented method of claim 1 , wherein: training the machine learning model to predict a depth map of a color image via an iterative diffusion inference process includes: training a machine learning model to predict log distance metric depth maps of color images via an iterative diffusion inference process, wherein a first depth map is a first log distance depth map, wherein a second depth map is a second log distance depth map, wherein the first log distance depth map represents a log of a depth across a first scene depicted in a first color image, and wherein a third depth map is a third log distance depth map.

14. The computer-implemented method of claim 13, wherein, training a machine learning model to predict log distance metric depth maps of color images via an iterative diffusion inference process includes training the machine learning model to predict log distance metric depth maps of color images and associated field of view (FOV) measurements of the color images via the iterative diffusion inference process, wherein applying the first color image and the second log distance depth map to the machine learning model to generate the third log distance depth map includes applying the first color image, the second log distance depth map, and a first FOV measurement to the machine learning model to generate the third log distance depth map, and wherein the first FOV measurement represents a tangent function of an angular field of view across at least one dimension of the first color image.

15. The computer-implemented method of claim 14, further comprising: training a predecessor machine learning model to predict log distance metric depth maps of color images via an iterative diffusion inference process; and augmenting the trained predecessor machine learning model to receive a FOV measurement as input in addition to a color image and a log distance depth map, wherein training the machine learning model to predict log distance metric depth maps includes training the trained augmented predecessor machine learning model.

16. The computer-implemented method of claim 15, wherein, augmenting the trained predecessor machine learning model to receive a FOV measurement as input in addition to a color image and a log distance depth map includes: adding a sinusoidal positional embedding of the FOV measurement input; and adding a linear projection of an output of the sinusoidal positional embedding to one or more layers of the trained predecessor machine learning model.

17. The computer-implemented method of any one of claims 14 to 16, wherein, the first FOV measurement represents a tangent function of an angular field of view across at least one dimension of the first color image.

18. The computer-implemented method of any one of claims 14 to 16, wherein, Training a machine learning model to predict a log distance measure depth map of a color image and an associated field of view (FOV) measurement of the color image via an iterative diffusion inference process includes training the machine learning model to predict the log distance measure depth map of the color image, the associated field of view (FOV) measurement of the color image, and an associated noise input via the iterative diffusion inference process, wherein applying the first color image, the second log distance depth map, and the first FOV measurement to the machine learning model to generate a third log distance depth map includes applying the first color image, the second log distance depth map, the first FOV measurement, and a first noise estimate to the machine learning model to generate the third log distance depth map, wherein the first noise estimate represents an amount of noise added to the first log distance depth map to generate the second log distance depth map, and wherein generating a plurality of loss values based on the third log distance depth map includes (i) determining a weighted combination of (a) the first log distance depth map and (b) the noise added to the first log distance depth map to generate the second log distance depth map, and (ii) determining a pixel-wise difference between the weighted combination and the third log distance depth map.

19. The computer-implemented method of claim 13, wherein, Training a machine learning model to predict a log distance measure depth map of a color image via an iterative diffusion inference process includes training the machine learning model to predict the log distance measure depth map of the color image and an associated noise input via the iterative diffusion inference process, wherein applying the first color image and the second log distance depth map to the machine learning model to generate a third log distance depth map includes applying the first color image, the second log distance depth map, and a first noise estimate to the machine learning model to generate the third log distance depth map, wherein the first noise estimate represents an amount of noise added to the first log distance depth map to generate the second log distance depth map, and wherein generating a plurality of loss values based on the third log distance depth map includes (i) determining a weighted combination of (a) the first log distance depth map and (b) the noise added to the first log distance depth map to generate the second log distance depth map, and (ii) determining a pixel-wise difference between the weighted combination and the third log distance depth map.

20. The computer-implemented method of any one of claims 13 to 16 or 19, further comprising determining the first log distance depth map from the source distance depth map by: determining a first intermediate depth map by dividing the source distance depth map by a minimum distance of the source distance depth map; determining a second intermediate depth map by determining a logarithm of the first intermediate depth map; determining a third intermediate depth map by dividing the second intermediate depth map by a logarithm of a ratio of a maximum distance of the source distance depth map to the minimum distance of the source distance depth map; and normalizing the third intermediate depth map to values of a finite range centered at zero.

21. The computer-implemented method of any one of claims 13 to 16 or 19, wherein, Training a machine learning model to predict a log-distance metric depth map of a color image via an iterative diffusion inference process includes training the machine learning model using an augmented training data set, wherein the augmented training data set includes a first log-distance depth map and a first color image, and wherein the method further includes augmenting a source training data set to generate the augmented training data set by cropping or un-cropping at least one of the source log-distance depth map and the source color image in at least one of a vertical direction or a horizontal direction to generate the first log-distance depth map and the first color image.

22. The computer-implemented method of claim 21, wherein, Un-cropping the cropped source log-distance depth map and the source color image in at least one of a vertical direction or a horizontal direction includes applying a generative machine learning model to un-crop the cropped source log-distance depth map and the source color image.

23. The computer-implemented method of claim 21, wherein, Un-cropping the cropped source log-distance depth map and the source color image in at least one of a vertical direction or a horizontal direction includes padding the source log-distance depth map and the source color image with noise in at least one of a vertical direction or a horizontal direction.

24. The computer-implemented method of any one of claims 13 to 16 or 19, wherein, Training a machine learning model to predict a log-distance metric depth map of a color image via an iterative diffusion inference process includes training the machine learning model using a training data set, and wherein the training data set contains examples of images and corresponding depth maps of indoor scenes and outdoor scenes such that the machine learning model is trained to predict a log-distance metric depth map of a color image representing an indoor scene or an outdoor scene.

25. The computer-implemented method of claim 1, wherein, Training a machine learning model to predict a depth map of a color image via an iterative diffusion inference process includes training the machine learning model to predict a metric depth map and an associated field of view (FOV) measurement of a color image via an iterative diffusion inference process, wherein applying a first color image and a second depth map to the machine learning model to generate a third depth map includes applying the first color image, a first FOV measurement, and the second depth map to the machine learning model to generate the third depth map, and wherein the first FOV measurement represents a tangent function of an angular field of view across at least one dimension of the first color image.

26. The computer-implemented method of claim 25, further comprising: training a predecessor machine learning model to predict a metric depth map of a color image via an iterative diffusion inference process; and augmenting the trained predecessor machine learning model to receive a FOV measurement as input in addition to a color image and a depth map, wherein training the machine learning model to predict a metric depth map includes training the trained augmented predecessor machine learning model.

27. The computer-implemented method of claim 26, wherein, augmenting the trained predecessor machine learning model to receive a FOV measurement as input in addition to a color image and a depth map includes: adding a sinusoidal positional embedding of the FOV measurement input; and adding a linear projection of an output of the sinusoidal positional embedding to one or more layers of the trained predecessor machine learning model.

28. The computer-implemented method of claim 25, wherein, the first FOV measurement represents a tangent function of an angular field of view across at least one dimension of the first color image.

29. The computer-implemented method of claim 25, wherein, Training a machine learning model to predict a metric depth map of a color image via an iterative diffusion inference process includes training the machine learning model to predict a metric depth map of a color image via an iterative diffusion inference process and an associated noise input, wherein applying the first color image and the second depth map to the machine learning model to generate a third depth map includes applying the first color image, the second depth map, and a first noise estimate to the machine learning model to generate the third depth map, wherein the first noise estimate represents an amount of noise added to the first depth map to generate the second depth map, and wherein generating a plurality of loss values based on the third depth map includes (i) determining a weighted combination of (a) the first depth map and (b) the noise added to the first depth map to generate the second depth map, and (ii) determining a pixel-wise difference between the weighted combination and the third depth map.

30. The computer-implemented method of claim 25, wherein, Training a machine learning model to predict a metric depth map of a color image via an iterative diffusion inference process includes training the machine learning model using an augmented training dataset, wherein the augmented training dataset includes a first depth map and a first color image, and wherein the method further includes augmenting a source training dataset to generate the augmented training dataset by cropping or un-cropping at least one of a source depth map and a source color image in at least one of a vertical direction or a horizontal direction to generate the first depth map and the first color image.

31. The computer-implemented method of claim 30, wherein, Un-cropping the cropped source depth map and the source color image in at least one of the vertical direction or the horizontal direction includes applying a generative machine learning model to un-crop the cropped source depth map and the source color image.

32. The computer-implemented method of claim 30, wherein, Un-cropping the cropped source depth map and the source color image in at least one of the vertical direction or the horizontal direction includes padding the source depth map and the source color image with noise in at least one of the vertical direction or the horizontal direction.

33. The computer-implemented method of any one of claims 25 to 32, wherein, Training a machine learning model to predict a metric depth map of a color image via an iterative diffusion inference process includes training the machine learning model using a training dataset, and wherein the training dataset contains examples of images and corresponding depth maps of indoor scenes and outdoor scenes such that the machine learning model is trained to predict a metric depth map of a color image representing an indoor scene or an outdoor scene.

34. The computer-implemented method of claim 1, wherein, Training a machine learning model to predict a depth map of a color image via an iterative diffusion inference process includes training the machine learning model to predict a metric depth map of a color image via an iterative diffusion inference process and an associated noise input, wherein applying the first color image and the second depth map to the machine learning model to generate a third depth map includes applying the first color image, a first noise estimate, and the second depth map to the machine learning model to generate the third depth map, wherein the first noise estimate represents an amount of noise added to the first depth map to generate the second depth map, and wherein the iterative diffusion inference process additionally includes: determining a weighted combination of (i) the first depth map and (ii) the noise added to the first depth map to generate the second depth map, wherein generating a plurality of loss values includes determining a pixel-wise difference between the weighted combination and the third depth map to generate the plurality of loss values.

35. The computer-implemented method of claim 34, wherein, Training a machine learning model to predict a metric depth map of a color image via an iterative diffusion inference process includes training the machine learning model using an augmented training data set, wherein the augmented training data set includes a first depth map and a first color image, and wherein the method further includes augmenting a source training data set to generate the augmented training data set by cropping or un-cropping at least one of the source depth map and the source color image in at least one of a vertical direction or a horizontal direction to generate the first depth map and the first color image.

36. The computer-implemented method of claim 35, wherein, Un-cropping the cropped source depth map and the source color image in at least one of the vertical direction or the horizontal direction includes applying a generative machine learning model to un-crop the cropped source depth map and the source color image.

37. The computer-implemented method of claim 35, wherein, Un-cropping the cropped source depth map and the source color image in at least one of the vertical direction or the horizontal direction includes padding the source depth map and the source color image with noise in at least one of the vertical direction or the horizontal direction.

38. The computer-implemented method of any one of claims 34 to 37, wherein, Training a machine learning model to predict a metric depth map of a color image via an iterative diffusion inference process includes training the machine learning model using a training data set, and wherein the training data set contains examples of images and corresponding depth maps of indoor scenes and outdoor scenes, such that the machine learning model is trained to predict a metric depth map of a color image representing an indoor scene or an outdoor scene.

39. A computer-implemented method comprising: predicting, using a machine learning model via an iterative diffusion inference process, a first target depth map of a target color image for a plurality of iterations by: applying the target color image and a corresponding input depth map estimate to the machine learning model to generate a corresponding output depth map, wherein the input depth map for a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noisy depth map, wherein the input depth map for each iteration of the plurality of iterations other than the first iteration is the output depth map of the immediately preceding iteration with an amount of noise added, and wherein the first target depth map is the output depth map of a final iteration of the plurality of iterations.

40. The computer-implemented method of claim 39, further comprising generating a novel view of a target scene depicted in the target color image by: rendering an alternative color image of the target scene using the first target depth map and the target color image, wherein, the substitute color image includes holes representing aspects of the target scene that are occluded from view in the target color image; rendering a substitute depth map of the target scene as viewed from the perspective of the substitute color image using the first target depth map, wherein the substitute depth map includes holes representing aspects of the target scene that are occluded from view in the first target depth map; filling in the holes of the substitute color image; and after filling in the holes of the substitute color image, filling in the holes of the substitute depth map using the machine learning model via an iterative diffusion inference process for a plurality of iterations by: applying the substitute color image and a corresponding input depth map estimate to generate a corresponding output depth map, wherein the input depth map for a first iteration of the plurality of iterations is the substitute depth map with an amount of noise added, and wherein the input depth map for each iteration of the plurality of iterations other than the first iteration is the output depth map of the immediately preceding iteration with an amount of noise added.

41. The computer-implemented method of any one of claims 39-40, wherein, The machine learning model has a U-Net architecture.

42. The computer-implemented method of any one of claims 39-40, further comprising generating, for a target color image, a second target depth map of the target color image by: upsampling the first target depth map to generate an upsampled depth map; and predicting, using a machine learning model, a plurality of patches of a second target depth map for respective patches of a target color image via an iterative diffusion inference process, wherein, predicting, using the machine learning model via the iterative diffusion inference process, a given patch of the second target depth map for a corresponding patch of the target color image comprises using the corresponding patch of the upsampled depth map as an input depth map for a first iteration of a plurality of iterations of the process.

43. The computer-implemented method of any one of claims 39-40, wherein, The target color image represents an indoor scene, wherein the method further comprises predicting, using the machine learning model via the iterative diffusion inference process, an additional target log-distance metric depth map for an additional target color image, and wherein the additional target color image represents an outdoor scene.

44. The computer-implemented method of claim 39, wherein, Predicting, using the machine learning model via the iterative diffusion inference process, the first target depth map for the target color image comprises predicting, using the machine learning model via the iterative diffusion inference process, a first target log-distance metric depth map for the target color image, wherein the respective input depth map estimate is a respective input log-distance depth map, and wherein the respective output depth map is a respective output log-distance depth map.

45. The computer-implemented method of claim 44: wherein Applying the target color image and the respective input log-distance depth map estimate to the machine learning model to generate the respective output log-distance depth map comprises applying the target color image, a target FOV measurement, and the respective input log-distance depth map estimate to the machine learning model to generate the respective output log-distance depth map, and wherein the target FOV measurement represents a tangent function of an angular field of view across at least one dimension of the target color image.

46. The computer-implemented method of claim 45, further comprising determining the target FOV measurement from the target image.

47. The computer-implemented method of claim 45, wherein: The target FOV measurement represents a tangent function of an angular field of view across at least one dimension of the target color image.

48. The computer-implemented method of claim 45, wherein, Applying the target color image, the target FOV measurement, and the respective input log-distance depth map estimate to the machine learning model to generate the respective output log-distance depth map comprises applying the target color image, the target FOV measurement, a respective noise estimate from a noise schedule, and the respective input log-distance depth map estimate to the machine learning model to generate the respective output log-distance depth map.

49. The computer-implemented method of claim 44, wherein, Applying the target color image and the respective input log-distance depth map estimate to the machine learning model to generate the respective output log-distance depth map comprises applying the target color image, a respective noise estimate from a noise schedule, and the respective input log-distance depth map estimate to the machine learning model to generate the respective output log-distance depth map.

50. The computer-implemented method of any one of claims 44-49, further comprising determining a linearity metric depth map for the target color image by: determining a first intermediate depth map by scaling and shifting the target log-distance metric depth map; determining a second intermediate depth map by raising the first intermediate depth map to a power; and scaling the second intermediate depth map.

51. The computer-implemented method of claim 39, wherein, applying the target color image, the target FOV measurement, and the corresponding input depth map estimate to the machine learning model to generate a corresponding output depth map comprises applying the target color image, the target FOV measurement, a corresponding noise estimate from the noise schedule, and the corresponding input depth map estimate to the machine learning model to generate a corresponding output distance depth map.

52. The computer-implemented method of claim 51, further comprising determining the target FOV measurement from the target image.

53. The computer-implemented method of any one of claims 51 or 52, wherein, The target FOV measurement represents a tangent function of an angular field of view across at least one dimension of the target color image.

54. The computer-implemented method of any one of claims 51 to 52, wherein, applying the target color image, the target FOV measurement, and the corresponding input depth map estimate to the machine learning model to generate a corresponding output depth map comprises applying the target color image, the target FOV measurement, a corresponding noise estimate from the noise schedule, and the corresponding input depth map estimate to the machine learning model to generate a corresponding output distance depth map.

55. The computer-implemented method of any one of claims 39-40, wherein, applying the target color image, the target FOV measurement, and the corresponding input depth map estimate to the machine learning model to generate a corresponding output depth map comprises applying the target color image, the target FOV measurement, a corresponding noise estimate from the noise schedule, and the corresponding input depth map estimate to the machine learning model to generate a corresponding output distance depth map.

56. The computer-implemented method of any one of claims 39 to 55, wherein, The machine learning model has been trained as recited in any of claims 1-38.

57. A computer-implemented method comprising: training a machine learning model to predict an optical flow map for a pair of color images via an iterative diffusion inference process, wherein training the machine learning model comprises: adding noise to the first optical flow map to generate a second optical flow map, wherein the first optical flow map represents an optical flow from a first color image of a scene to a second color image of the scene; applying the first color image, the second color image, and the second optical flow map to the machine learning model to generate a third optical flow map; generating a plurality of loss values based on the third optical flow map; and updating the machine learning model based on the plurality of loss values.

58. The computer-implemented method of claim 57, further comprising: using the trained machine learning model to predict a first target optical flow map for a target pair of color images via an iterative diffusion inference process for a plurality of iterations by: applying the target pair of color images and corresponding input optical flow map estimates to the trained machine learning model to generate corresponding output optical flow maps, wherein the input optical flow map for a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noise optical flow map, wherein the input optical flow map for each iteration of the plurality of iterations other than the first iteration is the output optical flow map of the immediately preceding iteration with an amount of noise added, and wherein the first target optical flow map is the output optical flow map of a final iteration of the plurality of iterations.

59. The computer-implemented method of any one of claims 57-58, wherein, generating the plurality of loss values comprises generating a plurality of LI loss values between the third optical flow map and the first optical flow map or between the third optical flow map and the noise added to the first optical flow map to generate the second optical flow map.

60. The computer-implemented method of any one of claims 57-58, wherein, The machine learning model has a U-Net architecture.

61. The computer-implemented method of any one of claims 57-58, wherein, Training a machine learning model to predict an optical flow map for a pair of color images via an iterative diffusion inference process includes training the machine learning model to predict an optical flow map for a pair of color images via an iterative diffusion inference process and an associated noise input, wherein applying the first color image, the second color image, and the second optical flow map to the machine learning model to generate a third optical flow map includes applying the first color image, the second color image, a first noise estimate, and the second optical flow map to the machine learning model to generate the third optical flow map, wherein the first noise estimate represents an amount of noise added to the first optical flow map to generate the second optical flow map, and wherein the iterative diffusion inference process additionally includes: determining a weighted combination of (i) the first optical flow map and (ii) the noise added to the first optical flow map to generate the second optical flow map, wherein generating the plurality of loss values includes determining a pixel-wise difference between the weighted combination and the third optical flow map to generate the plurality of loss values.

62. A computer-implemented method comprising: predicting, using a machine learning model via an iterative diffusion inference process, a first target optical flow map for a target pair of color images for a plurality of iterations by: applying the target pair of color images and respective input optical flow estimates to the machine learning model to generate respective output optical flow maps, wherein an input optical flow map for a first iteration of the plurality of iterations is a randomly or pseudo-randomly generated noise optical flow map, wherein an input optical flow map for each iteration of the plurality of iterations other than the first iteration is an output optical flow map of an immediately preceding iteration with an amount of noise added, and wherein the first target optical flow map is an output optical flow map of a final iteration of the plurality of iterations.

63. The computer-implemented method of claim 62, wherein, the machine learning model has a U-Net architecture.

64. The computer-implemented method of any one of claims 62-63, further comprising generating, for the target pair of color images, a second target optical flow map for the target pair of color images by: upsampling the first target optical flow map to generate an upsampled optical flow map; and predicting, using a machine learning model, a plurality of patches of a second target optical flow map for respective patches of a target color image pair via an iterative diffusion inference process, wherein, predicting, using the machine learning model via the iterative diffusion inference process, a given patch of the second target optical flow map for a corresponding patch of the target pair of color images includes using the corresponding patch of the upsampled optical flow map as an input optical flow map for a first iteration of a plurality of iterations of the process.

65. The computer-implemented method of any one of claims 62-63, wherein, applying the target pair of color images and respective input optical flow estimates to the machine learning model to generate respective output optical flow maps includes applying the target pair of color images, respective noise estimates from a noise schedule, and respective input optical flow estimates to the machine learning model to generate respective output optical flow maps.

66. The computer-implemented method of any one of claims 62-63, wherein, the machine learning model has been trained as recited in any one of claims 57-61.

67. A computing device comprising: one or more processors, wherein the one or more processors are configured to perform the method of any preceding claim.

68. An article of manufacture comprising a non-transitory computer-readable medium having stored thereon program instructions that, when executed by a computing device, cause the computing device to perform operations to implement the method of any one of claims 1-66.

Images