Generating depth images of image data

Abstract

A method, system and apparatus for training a machine learning model configured to generate predicted depth images, including a computer program encoded on a computer storage medium, the method comprising: receiving data representing training samples comprising a plurality of image pairs, each image pair comprising a target image and a reference image, both capturing a particular scene from different perspectives; for each image pair of the plurality of image pairs, generating a compression cost volume for the image pair; using the compression cost volume as an input to the machine learning model; generating output data representing a predicted disparity map for the compression cost volume using the machine learning model; and generating a total loss using the predicted disparity map for the compression cost volume, the total loss comprising a boundary loss, an occlusion loss and a migration loss; and updating a plurality of parameters of the machine learning model by minimizing the total loss for the plurality of image pairs.

Description

Technical Field

[0001] This disclosure relates to image processing, and more particularly to generating depth images of image data. Background Technology

[0002] Generally, a depth image may include pixel-wise information relating to the distance between a first location and a second location (e.g., an observation location), where the first location is the location of a portion of the scene (captured by an image sensor in the original image and represented by corresponding pixels), and the second location is determined by the position of the image sensors, such as a line connecting the focal points of the two image sensors.

[0003] Depth images (e.g., dense depth images that include depth information for each pixel in an image) are widely used in computer graphics and computer vision. Depth image-based techniques can include generating image effects, such as simulating shallow depth of field by blurring pixels in the original image to varying degrees, simulating fog effects on objects in the original image, generating appropriate shadows and brightness for objects in the original image, generating three-dimensional (3D) models from two-dimensional (2D) images, and rendering 3D scenes using Z-buffering and Z-culling, etc. Furthermore, in some implementations, depth images can be used for localization, motion control, or pose control.

[0004] Traditional methods for extracting depth information from a scene to generate a depth image can generally be divided into two categories – active methods and passive methods. Active methods typically involve emitting light signals, such as beams of light or laser beams, towards objects in the scene and measuring the round-trip time of each light signal. Techniques and sensors using active methods include time-of-flight (TOF) cameras and light detection and ranging (LIDAR) methods. Passive methods, on the other hand, typically involve using two or more sensors with known relative displacement and orientation (e.g., stereo cameras) to generate multiple raw images of a given scene (e.g., a reference raw image and one or more target raw images), and generating a depth image of the raw images by minimizing the photometric differences after projection (e.g., rectification).

[0005] Artificial intelligence (AI) is the intelligence exhibited by machines, representing the ability of computer programs or machines to think and learn. One or more computers can be used to train multiple machine learning models based on training samples, and to perform inference operations on each of the trained machine learning models to process model inputs and generate model outputs.

[0006] Neural networks are a subfield of machine learning models. A neural network can use one or more layers of nodes to represent various computations. One or more computers can be configured to perform neural network computations to generate outputs, such as classifying, predicting, or segmenting received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of network parameters. Summary of the Invention

[0007] The techniques described in this disclosure relate to training machine learning models based on training samples. The trained machine learning models can be deployed to generate output predictions of depth images for input image data.

[0008] Existing passive methods can be used to generate depth images. Specifically, a depth image is generated based on carefully designed cost capacity functions and filters to determine dissimilarity values. The cost capacity function can be designed to include different penalty factors to constrain discontinuities between objects in the depth image. For example, one existing passive method generates a depth image by minimizing photometric differences (e.g., costs) using constraints in the pixel matching process and smoothness constraints in the optimization process. More specifically, the smoothness constraint is used to uniformly penalize discontinuities in depth values ​​across all pixels in the depth image. As another example, passive methods can include smoothness constraints based on depth discontinuities based on pixel-wise color information. However, due to “defective” penalty factors, the designed cost capacity and filters can lead to inaccurate depth images. For example, a “defective” penalty factor may be uniform across all pixels or based on their respective pixel-wise color information. Depth images generated based on these penalty factors often fail to properly correlate with the corresponding objects presented in the original image, or may include significant discontinuities of the same objects presented in the original image, etc.

[0009] The techniques described in this disclosure relate to building and training machine learning models on training samples to accurately predict depth images from input image data. Machine learning methods are known to offer superior performance, high accuracy, and robustness for image processing, and can also eliminate inaccuracies caused by the designed cost-capacity function and filters. However, historically, machine learning methods have rarely been used to generate depth images from image data, at least in part due to several training challenges, such as insufficient labels / annotations, mislabeled images (e.g., discrepancies between annotated and true pixels in the input image), and the demands on intensive computational resources and heavy computational requirements. The techniques described in this disclosure can address these issues and alleviate the long-standing need to implement machine learning techniques (or AI methods) for generating depth images.

[0010] As used in this disclosure, the term "sensor" refers to various suitable optical sensors used to acquire scene information. For example, a sensor may include an image sensor (e.g., a digital single-lens reflex (DSLR) camera), a laser sensor, or a video sensor.

[0011] As used in this disclosure, the term "original image" refers to data collected by the aforementioned sensors. For example, an original image may include image data corresponding to photographs or video clips captured by a DSLR camera.

[0012] As used in this disclosure, the term "boundary pixel" refers to a pixel that substantially represents the boundary of an object relative to a background, or the boundary of two or more partially overlapping or intersecting objects captured in the original image. More specifically, the boundary pixel of an object may include the pixel at the outermost edge of the object presented in the original image. Furthermore, the boundary pixel may include pixels of a specific width. For example, a boundary pixel may have a width of one pixel, two pixels, or five pixels from the periphery of the object. The width of the boundary pixel may be determined based on the size of the object, the size of the original image, or the resolution of the original image.

[0013] As used in this disclosure, the term "occlusion area" (also referred to as "occlusions," "occluded area," or "occlusion pixels") means a region or area comprising one or more pixels that represent a portion of one or more objects captured in an image by one of a pair of sensors (e.g., a stereo image sensor) but not by the other. The occlusion area typically represents a portion of a scene where one object is covered or hidden by another object, which is usually closer to the image sensor than the first object.

[0014] As used in this disclosure, the term "cost function" refers to an objective function that minimizes photometric differences in generating a depth image from an input image. The cost function may receive the input location of each pixel in the original image, the direction for generating the cost path, and a specific disparity level, and output a specific value (e.g., cost value) representing a measure of photometric differences. The term "cost values" refers to the values ​​output from the cost function used to process a specific input (e.g., a specific pixel at a specific disparity level). Typically, the cost function can generate a separate cost value for each pixel in the input image (e.g., a reference image in an image pair). The technique relates to optimizing the functional form of the cost function used to generate an accurate depth image, the details of which are described below.

[0015] As used in this disclosure, the term "cost volume" refers to a data structure that stores the cost values ​​generated by various cost functions at different disparity levels for different pixels in an input image. The cost volume can be compressed using the techniques described in this disclosure to generate a "compressed cost volume," which stores less cost value for each pixel in the input image. The cost volume and the corresponding compression / filtering process will be described in more detail below.

[0016] For convenience, in this disclosure, the terms "original image" and "cost" may be simply referred to as image and cost, respectively.

[0017] Specific embodiments of the subject matter described in this disclosure may be implemented to achieve one or more of the following advantages.

[0018] The techniques described below can improve the accuracy of generating depth maps from one or more images representing a scene. More specifically, the techniques described below implement a trained machine learning model to generate depth map predictions without requiring a human-designed penalty factor or cost function. In this way, the modeling of the machine learning technique eliminates ambiguity in determining factors inherent to stereo vision, thereby improving the accuracy of the generated depth images. Furthermore, the machine learning model is more robust to handling diverse input images or training samples, such as images of different scenes with varying quantities, sizes, colors, and object locations, different backgrounds, different lighting or weather conditions, and different image qualities.

[0019] Machine learning models are typically immune to outliers or a limited number of mislabeled samples, or can self-correct, which further improves accuracy. Furthermore, the implemented machine learning model is trained with a total loss function, determined based on considerations of multiple aspects / constraints to enhance accuracy. The total loss function provides semantic-based constraints for stereo vision problems and helps learn to generate high-quality and highly accurate depth maps. Specifically, the total loss is generated based at least on predicted boundary pixels, occlusion regions, and a style transfer based on a baseline depth image. Due to the improved accuracy of the predicted depth images, this technique can further generate better and more engaging depth-of-field (DOF) data that is rendering engine friendly and has clear depth boundaries.

[0020] Furthermore, the techniques described below can further improve computational efficiency. Training machine learning models typically requires dense training samples, thus necessitating high computational resources such as memory size, memory bandwidth, and computing power. However, the techniques described in this disclosure can reduce the size of the input data for the machine learning model by compressing the data structure storing the cost associated with pixels in the input image (e.g., compressing cost capacity). This compression process reduces computational cost while maintaining a high level of accuracy when generating depth images. Moreover, boundary pixel predictions are generated using another pre-trained machine learning model, and the pre-trained machine learning model does not need to recognize the texture and classification information of the input image when processing the input. Therefore, the techniques can be implemented with less computational power and memory size, and the trained machine learning model can be deployed on edge devices such as smartphones, tablets, and smartwatches.

[0021] Furthermore, the techniques described below can be easily scaled and piped to efficiently handle larger or large numbers of input images. More specifically, the training process can be piped and implemented in a distributed manner. For example, the training can be performed by a cloud service in a distributed database. As another example, the training can be performed in a federated manner (e.g., federated learning), which provides sufficient privacy for user data. The training process, or more specifically, the reverse update process, can be distributed across different processors, clusters, or computing units with different memories and bandwidths, and can be executed in parallel.

[0022] Details of one or more embodiments of the subject matter of this disclosure are set forth in the accompanying drawings and the following description. The specification, drawings, and claims will make other features, aspects, and advantages of this disclosure more apparent. Attached Figure Description

[0023] Figure 1 An example system configured to train a machine learning model to generate depth images of training samples is shown;

[0024] Figure 2 An example process for generating compression cost capacity is shown;

[0025] Figure 3 An example boundary prediction engine configured to generate boundary maps of input image data is shown;

[0026] Figure 4 An example occlusion loss engine configured to generate occlusion loss based on an enhanced boundary map with occluded regions is shown.

[0027] Figure 5 An example method is shown, which is configured to determine the migration loss based on a base-truth disparity map.

[0028] Figure 6 This is a flowchart illustrating an example process of training a machine learning model to generate a depth image of the input data;

[0029] The same reference numbers and names in each figure represent the same element. Detailed Implementation

[0030] While depth images have been used in various applications and hardware, such as smartphones, tablets, gaming consoles, robot control, and autonomous vehicle navigation systems, generating depth images using machine learning techniques is challenging. This is partly due to the high demands on computational power, accurate labeling or data labeling techniques (e.g., supervised learning), and large amounts of training samples. For example, most existing trained machine learning models require significant computational resources for inference calculations and are not suitable for deployment on edge devices, as these devices (such as in-situ chips in smartphones or autonomous vehicles) often lack the computational power required. As another example, supervised training methods are generally unsuitable for training machine learning models for predicting depth maps because the labeling process is error-prone, and external measurement tools such as LiDAR and stereo cameras may collect inaccurate data due to ambient lighting conditions (e.g., inclement weather, low infrared reflectivity, or light flares), sparsely distributed data, and limitations in operating conditions.

[0031] Unsupervised learning can be used to eliminate the need for labeling input data or to provide a training process robust to mislabeling. For example, one existing technique uses computer graphics to generate virtual 3D scenes. First, the depth information of the virtual 3D scene is stored, and then rendered from two different viewpoints to generate image pairs (e.g., a pair of images, each capturing the scene from its respective viewpoint). Image pairs generated in this way can avoid labeling errors. However, the process of generating virtual 3D scenes is expensive and time-consuming, and the generated virtual 3D scenes only cover a few scenes of interest. Furthermore, the rendered image pairs differ from real images taken by physical sensors; therefore, machine learning models trained on generated virtual 3D scenes are not easily transferred to handle real image data.

[0032] Other existing techniques, such as self-supervised learning, may struggle to identify textureless regions, repetitive patterns, and intrinsic factors in imaging noise within image data. Furthermore, due to the position of the image sensor (e.g., a stereo camera), each image in a pair of images captured by the sensor may have regions that are at least partially occluded by other images (e.g., part of one or more objects is occluded by one or more other objects). When generating depth images, occluded regions can lead to inaccuracies; therefore, specific processing is often required to mitigate these inaccuracies.

[0033] The techniques described in this disclosure are superior to existing technologies because they at least solve the aforementioned problems. One aspect of this disclosure relates to a method for training a machine learning model configured to generate predictions of a depth map of an input image. The machine learning model may include multiple parameters to be updated / learned during training. The machine learning model may include a neural network comprising multiple network layers, and each network layer may include multiple nodes, each node being associated with its respective node parameters.

[0034] The method includes receiving training samples to train the machine learning model. The training samples consist of multiple image pairs. Each image pair includes a reference image and a target image, both generated by their respective sensors capturing the scene from their respective directions.

[0035] The method further includes generating a compression cost capacity for image pairs, processing the compression cost capacity using the machine learning model to generate a predicted disparity map, generating a total loss for the image pairs based on the predicted disparity map, and updating the model parameters by minimizing the total loss of all training samples. The total loss includes boundary loss, occlusion loss, and migration loss.

[0036] Other embodiments of this disclosure include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the operations described above. For a system comprising one or more computers, being configured to perform a specific operation or action means that software, firmware, hardware, or a combination thereof are installed on the system, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0037] Figure 1 An example system 100 is shown, configured to train a machine learning model 145 to generate depth images for training samples. The example system 100 is implemented on one or more computers at one or more locations, where the systems, components, and techniques described below can be implemented. Some components of the system 100 can be implemented as computer programs configured to run on one or more computers. The machine learning model 145 can be any suitable model configured to generate depth images after training on suitable training samples. The machine learning model can include regression models, multilayer perceptron models, and neural networks. Examples of machine learning models will be described in more detail below.

[0038] like Figure 1As shown, the system 100 may include a training subsystem 120 configured to train a machine learning model 145 using training samples 115. The trained machine learning model 145 may be configured to generate predictions. The predictions may include predictions of depth information (e.g., depth images) for use by other applications or processes. The training samples 115 may include raw images captured by one or more sensors, multiple pairs of images for stereo-based analysis, or filtered images according to specific image processing requirements.

[0039] In some embodiments, the system 100 may include a sensor subsystem 110 having multiple sensors 101a-101z (e.g., cameras) configured to collect image data (e.g., images or videos) of a specific scene and provide the collected image data as training samples 115 to a training subsystem 120. The sensor subsystem 110 may include audio-visual (AV) devices (e.g., an AV streaming mixer including multiple cameras or communicatively connected to multiple cameras), or mobile devices including multiple cameras (e.g., smartphones, tablets, or laptops), etc.

[0040] The sensor subsystem 110 can preprocess the collected image data to generate training samples 115. For example, the sensor subsystem 110 can filter the image data based on criteria such as image quality and the required number of images, or generate multiple image pairs for the training subsystem 120. Each of the multiple image pairs may include a reference image captured by a first sensor for a scene and a target image captured by a second sensor for substantially the same scene, however, the second sensor is located in a different position from the first sensor.

[0041] The system 100 may further include a cost capacity engine 140 configured to process the training samples 115 to generate input data 143 to be provided to the training subsystem 120. The cost capacity engine 140 is configured to generate a cost capacity for each pair of images. The cost capacity of an image pair can be a data structure, typically storing multiple cost values ​​for each pixel in one image of the image pair (e.g., a reference image). The cost capacity engine 140 can compress the cost capacity of the image pairs to generate compressed cost capacity as input data 143, which is also a data structure but stores only a portion of the cost value for each pixel in the image pair. In this way, the input data 143 can have a smaller size than the uncompressed cost capacity and require less storage bandwidth. Details of generating and compressing the cost capacity will be discussed later. Figure 2 Describe it.

[0042] To train the machine learning model 145, the training subsystem 120 can directly use the boundary prediction engine 123 to process the received training samples 115 (e.g., image pairs) to generate a boundary map 125. The training subsystem 120 can also further provide input data 143 (e.g., compression cost capacity) as input to the machine learning model 145 to generate a predicted disparity map 150. The boundary prediction engine 123 may include a pre-trained machine learning model (e.g., a neural network) configured to generate predictions of boundary pixels. For example, as described above, the prediction of the boundary pixels may include data representing the probability that a pixel in the reference image is a boundary pixel of its respective object. The prediction of boundary pixels of the reference image of the image pair is represented as a boundary map 125, which includes data representing the probability that each pixel is a boundary pixel in the image. The probability ranges from 0 to 1, where 0 indicates that the pixel is least likely (or substantially unlikely) to be a boundary pixel, and 1 indicates that the pixel is most likely (or substantially certain) to be a boundary pixel. The process of generating the boundary map 125 will be combined with... Figure 3 Describe it.

[0043] The training subsystem 120 also includes a loss generator 130 configured to generate a total loss 160 for training the machine learning model 145. Specifically, the total loss 160 is used as the objective to be minimized during the reverse operation of the training process. The parameters of the machine learning model 145 (e.g., the node weights of the hidden layers when the machine learning model is a neural network) are updated as a result of minimizing the objective during the reverse operation.

[0044] The loss generator 130 includes a boundary loss engine 133, an occlusion loss engine 135, and a migration loss engine 137, each configured to generate a respective loss based at least on the predicted disparity map 150. More specifically, the boundary loss engine 133 is configured to generate a boundary loss 163 using the boundary map 125 and the predicted disparity map 150 for a given image pair. The occlusion loss engine 135 is configured to generate an occlusion loss 165 using the boundary map 125 and the predicted disparity map 150 corresponding to a given image pair. The migration loss engine 137 is configured to generate a migration loss 167 using the predicted disparity map 150 and the ground truth disparity map 155 corresponding to a given image pair. The ground truth disparity map 155 may be a disparity map generated from a virtual 3D scene using the computer graphics techniques described above, or another disparity map with a ground truth label. The ground truth disparity map 155 may be stored in a storage unit included in the training subsystem 120 or provided to the training subsystem 120 as part of the input data.

[0045] The loss generator 130 can generate the total loss 160 for updating model parameters by combining the boundary loss 163, occlusion loss 165, and transfer loss 167. This combination can be a weighted sum of three different losses. The weight of each of the three losses can be predetermined according to training requirements, as long as the sum of the three weights is 1. For example, the boundary loss 163 can have a weight of 5 / 6, the occlusion loss 165 can have a weight of 1 / 12, and the transfer loss 167 can have a weight of 1 / 12, so that the total weight equals 1. Alternatively, the system 100 can set the weights of one or more losses to zero (e.g., suppress one or more losses) in the first phase of training (e.g., training the machine learning model 145 using a first subset of training samples), and enable the suppressed losses by setting their respective weights to be greater than zero in the second phase of training (e.g., training the machine learning model 145 using a second subset of training samples). For example, in the first training phase, system 100 can set the weight of transfer loss 167 to zero, and assign weights to boundary loss 163 and occlusion loss 165 based on training requirements, as long as the sum of the two weights is 1. In some training scenarios, the system can suppress one or more types of loss in one or more training phases at the cost of longer training cycles to train a machine learning model with higher accuracy.

[0046] For each time step, the training subsystem 120 may adjust one or more parameters of the machine learning model 145 to reduce the total loss 160 in the next time step. Parameter convergence is determined according to predetermined criteria, such as the derived values ​​of one or more parameters satisfying a specific threshold. For example, this is determined in response to determining that the change or rate of change of the parameters between the current time step and the next time step is below a threshold. As another example, it is determined that the change or rate of change of the total loss between the current time step and the next time step is below a threshold.

[0047] When the parameters converge, the training subsystem 120 can provide a machine learning model 145 with converged parameters as a trained machine learning model 180. The trained machine learning model 180 can be used to perform inference operations to process the input image and generate a depth image of the input image. Because the machine learning model 145 is trained using compressed cost capacity, the computational requirements for performing inference operations are also reduced. Therefore, the trained machine learning model 180 can be deployed on edge devices 190, such as smartphones, tablets, and smartwatches, which typically do not have intensive computing capabilities when used as computing workstations. In some embodiments, the edge device is a control unit or control processor installed in an autonomous vehicle. Inference operations are performed using the trained machine learning model 180 with updated or converged parameters, and these parameters are typically not changed or updated during the inference operations. Details of the inference operations are described below.

[0048] Edge device 190 can process input image data using a deployed, trained machine learning model 180 to generate output data with depth information (e.g., depth images or parallax images), and provide the output data to another module, device, system, or subsystem (associated internally or externally with edge device 190) for further analysis or manipulation. For example, a module associated internally with edge device 190 may include an image blurring module configured to process the depth information of the received image and blur corresponding pixels in the image to replicate depth-of-field effects; for example, pixels located further away in the scene or not around the camera's focal length may be automatically blurred by the image blurring module. As another example, a system external to edge device 190 may include a 3D modeling system configured to generate a 3D model of a scene captured from 2D images.

[0049] Continuing with reference to the image pairs in the training sample 115, as described above, the sensor subsystem 110 can generate image pairs from image data collected by sensors 101a-101z (also numbered 101 for multiple sensors). More specifically, the sensor subsystem 110 can position sensors 101 at different locations to capture images presenting substantially the same scene, setting a first image from the first sensor 101a as a reference image and a second image from the second sensor 101b as a target image, generating a first image pair including the first and second images. Generally, the sensor subsystem 110 can set multiple images captured by each sensor as reference images and pair the reference images with one or more target images captured by other sensors to form multiple image pairs. In some embodiments, the training sample 115 only includes the individual images captured by each sensor, and the cost-capacity engine 140 generates image pairs for further operations based on the training sample 115.

[0050] Figure 2 An example process 200 for generating compressed cost capacity 270 is shown. Cost capacity engine 210 can be, for example... Figure 1 The cost-capacity engine 140. For convenience, this process can be performed by a system of one or more computers located in one or more locations. For example, Figure 1 The system 100, with proper programming, can execute this process to generate a compressed cost capacity of 270.

[0051] The cost capacity engine 210 is configured to process training samples 205 (e.g., Figure 1 The training samples 115) are used to generate a compression cost capacity 215 (e.g., as training samples 115) ... Figure 1 Input data 143 is provided. (e.g., ...) Figure 2 As shown, the cost capacity engine 210 can generate a cost capacity 220 for a reference image of an image pair based on the position of a pixel and the corresponding disparity level of the pixel. For example, x represents the position (e.g., coordinate) along the width direction 227, y represents the position along the height direction 225, and each pixel located at the corresponding width x and height y coordinates can have a corresponding cost value at each disparity level along the disparity axis 223 (e.g., z). Example details of generating cost values ​​based on a specific cost function for all potential disparity levels can be found in U.S. Patent Application No. 17 / 376,027, which is incorporated herein by reference.

[0052] The cost capacity engine 210 can further perform morphological analysis on the disparity levels and corresponding candidate generation values ​​of all pixels in the reference image. For example... Figure 2 As shown, the cost capacity engine 210 can generate waveforms 240 for pixels in a reference image. The cost capacity engine 210 can generate these waveforms based on individual candidate cost values ​​and associated disparity levels. As an example, the waveform 240 may include a two-dimensional data structure where values ​​along the vertical direction represent the cost value of a pixel, and values ​​in the horizontal plane represent the disparity level associated with the cost value.

[0053] As another example, the waveform 240 of a pixel could be a discrete-line function representing the various disparity levels and corresponding candidate cost values. For example... Figure 2As shown, without loss of generality, the waveform 240 of a pixel in the reference image can be plotted as a line graph 245 in a 2D coordinate system, where the horizontal axis represents parallax and the vertical axis represents the cost generated at the corresponding parallax level. In the waveform 240 of this pixel, each vertex (e.g., vertex numbered 243) represents a pixel 230 at a possible parallax level and the cost obtained from the corresponding cost capacity generated at said parallax level. Detailed examples of generated waveforms can also be found in U.S. Patent Application No. 17 / 376,027.

[0054] The cost-capacity engine 210 can identify local minima along waveform 240. For example... Figure 2 As shown, the cost capacity engine 210 can identify a first local minimum 247a with value a at index i, a second local minimum 247b with value b at index j, and a third local minimum 247c with value c at index k. Although three local minimums are shown in the waveform, it should be noted that the waveform may include one, two, or more than three local minimums, such as five, ten, twenty, or more.

[0055] Cost capacity engine 210 selects one or more local minima 247a-c as the selected cost values ​​for generating compressed cost capacity 270. For example, cost capacity engine 210 may select the first K local minima in ascending order and store the first K local minima and their corresponding indices in a data structure (e.g., Figure 2 In the vector data structure 250 shown, the first two items in data structure 250 can represent the first local minimum and its index in sequence. Similarly, the second two items can represent the second local minimum and its corresponding index, and the third two items can represent the third local minimum and its corresponding index. Therefore, the generated vector data can have a length of 2K (or 2K channels) for each pixel. For a waveform with n < K local minimums, the cost capacity engine 210 can first select n local minimums, sort the remaining local minimums that have not yet been selected in order (e.g., ascending order), and select the first Kn local minimums to append n local minimums. The value of K can be determined based on specific training requirements predetermined by the user or system; for example, K can have values ​​of 4, 8, 10, 16, 32, or any other suitable value.

[0056] After storing the local minima of all pixels in the reference image in a data structure, the cost capacity engine 210 can generate a compressed cost capacity 270 based on the stored local minima. The compressed cost capacity 270 can be a tensor data structure, similar to the cost capacity 220, but with the disparity axis replaced by coded disparity 273. For each pixel 260 located at specific width and height coordinates, the compressed cost capacity 270 represents a vector data structure 250 storing the selected K local minima and their corresponding indices. The system can at least partially identify the corresponding disparity level of the compressed cost capacity 270 based on the indices.

[0057] The compression process uniquely encodes the cost capacity as a compressed cost capacity, thus reducing the storage size and bandwidth requirements for training machine learning model 145 and for the trained machine learning model 180 to perform inference operations. Although the compressed cost capacity includes less cost than the original cost capacity, it should be noted that the resolution of the image input (i.e., the pixel resolution in the width and height directions) remains unchanged. Therefore, the depth image generated based on the compressed cost capacity does not suffer from quality loss and can provide depth information with the same resolution as the input image.

[0058] Figure 3 An example boundary prediction engine 300 configured to generate a boundary map 340 for input image data 330 is shown. The boundary prediction engine 300 may be, for example... Figure 1 The boundary prediction engine 123. For convenience, this process can be performed by a system of one or more computers located in one or more locations. For example, Figure 1 The system 100, with proper programming, can perform the process of generating the boundary map 340.

[0059] The boundary prediction engine 300 can be configured to include one or more suitable pre-trained machine learning models. For example, the pre-trained machine learning models may include neural networks, such as convolutional neural networks, fully convolutional networks (FCNs), and fully connected layers.

[0060] The machine learning model may further include an encoder 310 and a decoder 320. The encoder 310 may be a neural network comprising one or more neural network layers. For example, the one or more neural network layers may include convolutional layers followed by fully connected layers. The encoder may receive input image data 330 (e.g., image pairs or image frames) and map the input data to a feature map (or embeddings for processing natural language), the feature map being represented by one or more vectors and tensors with real numbers. The encoder 310 may output encoded data 315 representing the aforementioned feature map to the decoder 320. The decoder 320 may also be a neural network comprising one or more neural network layers; however, these layers in the decoder are typically organized in the opposite direction to those in the encoder 310. For example, the decoder 320 may include fully connected layers followed by one or more convolutional transpose layers. The decoder 320 may process the received encoded data 315 and generate a prediction for each pixel in the input image based on the feature map. The generated prediction may be a boundary map 340. The boundary map may include pixel-wise predictions of whether a pixel is a boundary pixel. In some implementations, decoder 320 can generate a reconstructed output image that substantially matches the input image having segmented prediction (e.g., pixel-by-pixel prediction). As a specific example, encoder 310 and decoder 320 can be configured according to the UNet architecture, i.e., a symmetric architecture spanning two parts. Example details of encoder 310 and decoder 320 can be found in U.S. Patent Application No. 17 / 376,027, which is incorporated herein by reference.

[0061] The pre-trained machine learning model in the boundary prediction engine 300 can be trained using specific training samples and following conventional training steps. The training samples may include images containing pixels associated with boundary information, such as images labeled by a computer or user. The training process may be based on a loss function representing the difference between the labels on the input data and the predicted labels on the reconstructed output data, such as the error of predicting boundary pixels given a known label. Generally, any suitable loss function can be used to train the machine learning model, such as hinge loss and log loss.

[0062] The boundary prediction engine 300 ignores texture and classification information used to perform inference operations, so the prediction of boundary pixels (e.g., boundary map 340) generated using a pre-trained machine learning model can be more robust and efficient.

[0063] The boundary prediction engine 300 directs the loss generator (e.g., Figure 1 The loss generator 130 provides a boundary map 340 to generate the boundary loss. (Refer to...) Figure 1The loss generator 130 includes a boundary loss engine 133, which is configured to process the boundary map 125 (e.g., equivalent to) of the input training samples 115 (e.g., image pairs). Figure 3 The boundary map 125 (340) and the predicted disparity map 150 of the image pair are processed. The boundary loss engine 133 can generate a boundary loss 163 after processing the boundary map 125 and the predicted disparity map 150.

[0064] More specifically, the boundary loss engine 133 can normalize the boundary map 125 using predetermined parameters to generate an enhanced boundary confidence map. This is because normalization typically enhances the prediction confidence score associated with the boundary map 125 and predicts as many true boundary pixels as possible. An example normalization process can be represented by the following formula (1):

[0065] P b =min(max(2*boundary_model_output,0),1) Formula (1)

[0066] P in formula (1) b This represents the normalized probability that a pixel is a boundary pixel. In formula (1), boundary_model_output represents the pixel-by-pixel prediction of boundary map 125. In formula (1), Min() is the minimum value function, and max() is the maximum value function.

[0067] The boundary loss engine 133 can generate disparity gradients based on the predicted disparity map 150. For example, the boundary loss engine 133 can apply one or more Sobel filters to the predicted disparity map 150. A Sobel filter is a filter used to determine image edges, typically similar to a method for determining the slope of a function based on two points on the function. Solver filters are typically implemented as activation layers in neural networks and are precursors to image recognition.

[0068] Based on the normalized probability P b With the disparity gradient, the boundary loss engine 133 can generate the boundary loss 163 according to the boundary loss function expressed in the following formula (2).

[0069]

[0070] Loss in Formula (2) b The boundary loss is 163, and the gradient term is... and It is the disparity gradient (or depth gradient) along the x or y direction in the image pair. The loss generator 130 combines the boundary loss 163 with one or more other losses to generate a total loss 160 for training the machine learning model 145.

[0071] Figure 4 An exemplary occlusion loss engine 400 is illustrated, configured to generate an occlusion loss 450 based on an enhanced boundary map 430 having an occlusion region 440. The occlusion loss engine 400 can be, for example... Figure 1 The occlusion loss engine 135. For convenience, this process can be performed by a system of one or more computers located in one or more locations. For example, Figure 1 The system 100, with proper programming, can execute this process to generate an occlusion loss of 450.

[0072] like Figure 4 As shown and combined Figure 1 The occlusion loss engine 400 can process the boundary map 420 (e.g., Figure 1 Boundary map 125) and predicted disparity map 410 (e.g.) Figure 1 The predicted disparity map 150) is used to generate an occlusion loss of 450 (e.g., Figure 1 The shading loss is 165).

[0073] To generate occlusion loss 450, system 100 can be configured to generate an enhanced boundary map 430 with occlusion region 440. System 100 can overlay the generated occlusion map onto boundary map 420 to generate enhanced boundary map 430. In some embodiments, the occlusion loss engine 400 may be configured to generate an occlusion map and overlay the occlusion map onto boundary map 420 to generate enhanced boundary map 430.

[0074] To generate an enhanced boundary map 430 with occluded regions 440, the system 100 may first generate an occlusion map. To generate the occlusion map, the system 100 may perform [operations] on the input image pairs. Figure 1 The machine learning model 145 operates twice to generate two distinct predicted disparity maps 150. Specifically, the system 100 can generate the first of the two predicted disparity maps based on an original image pair (e.g., a reference image and a target image). To generate the second of the two predicted disparity maps, the system 100 can generate a new image pair by swapping or exchanging the reference image and the target image of the original image pair. In this way, the original reference image is now the target image in the new image pair, and the original target image is now the reference image in the new image pair. The system 100 then processes the new image pair using the machine learning model 145 to generate the second predicted disparity map.

[0075] In some implementations, a first predicted disparity map is used to sample a second predicted disparity map. An exemplary sampling process includes, for each pixel in a reference image of the image pair, determining specific coordinates (e.g., (x, y)) of that pixel, obtaining a predicted disparity value d1(x, y) for that pixel in the first predicted disparity map of the reference image, and obtaining a corresponding disparity value for the corresponding pixel in the second predicted disparity map of the target image of the image pair. The first and second predicted disparity maps are generated using the same machine learning model. The corresponding disparity value may be in the form d2(x - d1(x, y), y). After performing sampling on the pixels in the reference image, the system can determine the pixel's discrepancy based on a combination of the two disparity values. For example, the pixel's discrepancy may be the absolute sum of the two disparity values, such as |d1(x, y) + d2(x - d1(x, y), y)|. The discrepancy values ​​of one or more pixels in the reference image can be used to determine occlusion regions.

[0076] The system 100 can compare a first predicted disparity map and a second predicted disparity map, and determine the difference in disparity values ​​of each pixel in the two disparity maps. The system 100 can determine whether each disparity difference of each pixel meets a preset threshold. For example, the system 100 can compare the disparity difference with a preset threshold, such as 1, and include pixels with a difference value less than or equal to the threshold in the occlusion map.

[0077] The system 100 can combine the occlusion map with the boundary map 420. For example, the combination process may involve overlaying the occlusion region 440 of the occlusion map onto the boundary map 420. In some embodiments, the system can determine the enhanced boundary map 430 by expanding the region / pixel predicted as a boundary region / pixel in the boundary map 420 along a direction based on disparity differences, such as a gradient representing the disparity differences in the respective disparity directions. The expansion width is predetermined based on the overall disparity range.

[0078] The occlusion loss is determined based on the combined occlusion map and the enhanced boundary map 430. For example, the occlusion loss is negligible for pixels within the occlusion region 440, and the occlusion loss is considered non-zero for occluded pixels outside the occlusion region 440 in the occlusion map.

[0079] The occlusion loss function can be determined according to the following formulas (3)-(5):

[0080] ΔD(x)=|D l→r (x)+D r→l (xD l→r (x))|×δ(occlusion) Formula (3)

[0081] The term D in formula (3)l→r and D r→l The function represents the disparity value used to retrieve the original image pair and the swapped new image pair. The delta function δ(occlusion) in Equation (3) is used to determine whether a pixel falls outside the extended boundary region (e.g., occlusion region) of the enhanced boundary map 430. For example, if a pixel falls outside the occlusion region, δ(occlusion) can output a zero value; if a pixel falls inside the occlusion region, δ(occlusion) can output a specific non-zero value. The ΔD(x) term in Equation (3) represents the function used to determine the pixel-by-pixel disparity difference of the pixel located at position x (a vector representing the coordinates in the width and height directions, e.g., (x, y)) in the reference image.

[0082] The shading loss is determined according to the following formula (4):

[0083]

[0084] The term Loss in formula (4) o Indicates occlusion loss of 165, item I r Represents the pixel-by-pixel value of the reference image, while I w This represents the pixel-wise value of the new image generated from the target image by distorting it with predicted disparity values. For example, if the image pair includes a color image, then term I... r This can represent the pixel-by-pixel RGB values ​​of a reference image. Assume another term I... t If the target image can be represented by its pixel-by-pixel RGB values, then the system can predict the disparity value (e.g., d) based on the target image. t (x, y) generates a new image (also known as a distorted image). Item I w It can represent the pixel-by-pixel RGB values ​​of a distorted image. This system can generate an item I by sampling the RGB values ​​of the target image for each pixel in the generated disparity map based on the disparity value. w The value of . For example, item I. w It can be based on formula I w (x, y) = I t (xd(x,y), y) is determined.

[0085] The occlusion loss 165 is generated by summing a function f(x) over all image channels (e.g., red, green, and blue channels). The function f(x) is determined based on the following formula (5):

[0086] f(x) = -x γ ×log(1-x) Formula (5)

[0087] In formula (5), the term γ is a preset parameter for the occlusion loss 165.

[0088] Figure 5 An exemplary method 500 is illustrated using an exemplary migration loss engine 515, which is configured to determine a migration loss 520 based on a base-true disparity map 505. The migration loss engine 515 may be, for example... Figure 1 The migration loss engine 137. For convenience, this process can be performed by a system of one or more computers located in one or more locations. For example, Figure 1 System 100, with proper programming, can execute this process to generate migration loss 520.

[0089] Simply performing domain transfer techniques to train a machine learning model 145 is not feasible because the trained machine learning model should be able to generate predictions of disparity maps accurate to any scene, not just scenes aligned with each other using domain transfer techniques. Instead, the technique assumes that the predicted disparity map (or depth map) and the base-true disparity map (or depth map) should generally share the same "disparity pattern" for regions within the boundary. Therefore, the technique redefines the domain transfer problem as a style transfer problem between the predicted disparity map and the base-true disparity map.

[0090] Most existing domain transfer methods aim to align the domains of an input image to transfer data from one domain to another. Using the same idea for stereo learning purposes is not feasible, as the model should generalize well to all input scenes. Therefore, assuming that depth from different scenes shares the same “pattern” with each other in those large, continuous, and distinct regions interwoven with sparse and sharp depth information, the techniques described in this disclosure focus on the output and treat the domain transfer problem as a style transfer between predicted depth and base-true depth. The base-true disparity map 505 is equivalent to... Figure 1 The basic true disparity map 155. The basic true disparity map 505 can be generated using computer graphics techniques. For example, the basic true disparity map 505 can be generated based on Monkaa image frames or other suitable computer-generated images.

[0091] like Figure 5As shown, the transfer loss engine 515 can process the predicted disparity map 510 and the near-true disparity map 505 to generate a transfer loss 520. More specifically, the transfer loss engine 515 may include a feature extraction engine 513. The feature extraction engine 513 may include a machine learning model, such as a neural network, configured to process the disparity map to generate a corresponding feature map. For example, the machine learning model may be a VGG16 neural network, which is a convolutional neural network supporting 16 layers. The VGG16 neural network contains 1x1 convolutional layers to make the decision function more non-linear without changing the receptive field. The VGG16 neural network uses a small receptive field (3x3 pixels, stride 1). The small-sized convolutional filters allow the VGG16 neural network to have a large number of weight layers; of course, the more layers, the better the performance.

[0092] The feature extraction engine 513 can generate a first feature map 530 for processing the predicted disparity map 510, which is equivalent to Figure 1 The predicted disparity map 150. The feature extraction engine 513 can also generate a second feature map 535 for processing the base-truth disparity map 505. Feature maps 530 and 535 can typically be feature representations of data items by an ordered set of numerical values ​​(e.g., representing data items as vectors of points in a multidimensional feature space). In other words, each feature representation can include the numerical value of each of a plurality of features of the input data item (e.g., the input disparity map).

[0093] The migration loss engine 515 may also include a Gram matrix generator 540, configured to process feature maps used to generate the corresponding Gram matrices. A Gram matrix is ​​a matrix generated by multiplying a matrix by its own transpose. Gram matrices are commonly used to analyze the correlation between columns or rows of a matrix. For example, the determinant of a Gram matrix can be used to determine the linear independence between vectors in the matrix.

[0094] like Figure 5 As shown, the Gram matrix generator 540 can process the first feature map 530 to generate a first Gram matrix 550, and process the second feature map 535 to generate a second Gram matrix 555. The generated first Gram matrix 550 and second Gram matrix 555 are provided to the migration loss generator 560 to generate a migration loss 520.

[0095] The migration loss generator 560 in the migration loss engine 515 is configured to generate a migration loss 520 by comparing a first Gram matrix 550 and a second Gram matrix 555. For example, the migration loss generator 560 can be configured to compute the sum of the squared differences between the first Gram matrix 550 and the second Gram matrix 555, also known as the L2 norm or L2 distance. The migration loss 520 can be generated based on the L2 distance.

[0096] As described above and referenced Figure 1 The loss generator 130 is configured to generate a weighted sum of boundary loss 163, occlusion loss 165 and migration loss 167 as the total loss 160 used to train the machine learning model 145.

[0097] Figure 6 This is a flowchart of an example process 600 for training a machine learning model to generate depth images of input data. For convenience, this process can be performed by a system of one or more computers located in one or more locations. For example, Figure 1 System 100, with proper programming, can execute process 600.

[0098] The system is configured to train a machine learning model for generating predictions of depth maps of input images. The machine learning model includes multiple parameters to be updated during training. For example, when the machine learning model is a neural network, the parameters could be node weights for node operations in each of multiple network layers. Detailed examples of the machine learning model will be described in detail at the end of this disclosure.

[0099] The system receives data (610) representing training samples comprising multiple image pairs. Each image pair includes a target image of a specific scene captured by a first image sensor from a first direction, and a reference image of the specific scene captured by a second image sensor from a second direction.

[0100] For each image pair, the system generates a compressed cost capacity (620) for the image pair. The compressed cost capacity includes one or more cost values ​​selected for each pixel of the reference image in the image pair. To generate the compressed cost capacity, the system first generates the cost capacity of the image pair using a specific cost function. The cost capacity includes multiple cost values ​​for each pixel of the reference image at different disparity levels. For each pixel of the reference image, the system generates a waveform of the pixel's cost value along the axis of the disparity level. The system determines multiple local minimum cost values ​​in the waveform and selects one or more local minimum cost values ​​and their corresponding indices to generate the compressed cost capacity. These indices are associated with the corresponding disparity level.

[0101] The system provides the compression cost capacity as input to the machine learning model (630) to generate a predictive disparity map.

[0102] The system uses a machine learning model to generate a predicted disparity map (640) of the compression cost capacity.

[0103] The system generates a total loss (650) using a predicted disparity map for processing the compression cost capacity. The total loss includes boundary loss, occlusion loss, and migration loss. Specifically, the total loss is a weighted sum of the boundary loss, occlusion loss, and migration loss.

[0104] To generate the boundary loss, the system uses a pre-trained machine learning model to generate boundary maps of image pairs. The pre-trained machine learning model can be a semantic boundary machine learning model. The boundary map includes a probability for each pixel in the reference image, indicating the likelihood that the pixel is a boundary pixel. The system generates the boundary loss based on the predicted disparity map and the boundary map. Specifically, it obtains a depth gradient or disparity gradient based on the predicted disparity map, normalizes the boundary map to generate an enhanced boundary confidence map, and multiplies the depth gradient with the enhanced boundary confidence map to generate the boundary loss.

[0105] To generate the occlusion loss, the system first generates two disparity maps using a machine learning model. More specifically, the system generates a first disparity map for the original image pair. Then, the system swaps the target image and the reference image of the image pair to generate a swapped image pair. The system generates a second disparity map for the swapped image pair. The system further determines whether the disparity difference meets a threshold. For example, the system can determine this by comparing the disparity values ​​of each pixel in the first and second disparity maps of the reference image and determining whether the disparity difference is less than or equal to a preset threshold. If the disparity difference is determined to be less than or equal to the threshold, the system calculates the pixel-by-pixel loss for the pixels to be included in the occlusion loss; if the disparity difference is determined to be greater than the threshold, the system sets the pixel-by-pixel loss to zero.

[0106] To generate the transfer loss, the system uses a neural network to process the predicted disparity maps of the image pairs to generate a first feature map, and then uses the same neural network to process the base ground truth disparity map of another image pair to generate a second feature map. The neural network may include a VGG16 neural network. The system generates a first Gram matrix of the first feature map and a second Gram matrix of the second feature map. The system then generates the transfer loss based on the first and second Gram matrices. More specifically, the system may determine the transfer loss as the L2 distance between the first and second Gram matrices.

[0107] The system updates the parameters of the machine learning model by minimizing the total loss (660). At each time step during the reverse operation of the training process, the system updates the parameters of the next time step to reduce the total loss of the current time step.

[0108] After training the machine learning model, the system can deploy the trained machine learning model with updated parameters on an edge device. The edge device may include a smartphone, tablet, smartwatch, and any other suitable edge device. In some embodiments, the edge device may be a control unit or control processor installed in an autonomous vehicle. The edge device may include one or more processors configured to perform inference operations using the trained machine learning model to generate a predicted disparity map of the input image pairs.

[0109] The predicted depth image can also be provided to a rendering engine configured to blur one or more pixels in the first image to replicate a depth-of-field effect. In some implementations, the system can provide depth images to different external devices to perform different operations. For example, the external device may include a 3D renderer configured to render a 3D scene based on a depth map or disparity map of the scene. As another example, the external device may include a shadow generator configured to generate appropriate shadow effects for the scene based on a depth map or disparity map. Optionally, the external device may include a 3D model generator configured to generate 3D models based on a depth map or disparity map. In particular, the 3D model generator may be configured to generate 3D models for components in a dynamic system (e.g., a robotic system), each component having a different pose. The 3D generator may generate 3D models of these components based on depth maps or disparity maps generated from 2D images capturing these components. Optionally, the external device may include one or more machine learning models configured to generate respective predictions based on depth maps or disparity maps associated with additional data. For example, one machine learning model can predict whether an autonomous vehicle traveling along a trajectory will collide with objects in the scene within a specific time period.

[0110] As used in this disclosure, the term "machine learning model" can refer to a neural network model suitable for the task described herein. Depending on the task, the neural network can be configured, i.e., trained to receive any type of numerical data input and generate any type of score, classification, or regression output based on the input.

[0111] In some cases, a neural network is a network configured to perform image processing tasks, i.e., receiving an input image and processing it to generate a network output. In this disclosure, processing an input image refers to using a neural network to process the intensity values ​​of the image's pixels. For example, the task could be image classification, and the output generated by the neural network for a given image could be a score for each of a set of object categories, each score representing the estimated probability that the image contains an object belonging to that category. As another example, the task could be image embedding generation, and the output generated by the neural network could be a numeric embedding of the input image. As yet another example, the task could be object detection, and the output generated by the neural network could identify the location of a specific type of object depicted in the input image. As yet another example, the task could be image segmentation, and the output generated by the neural network could assign each pixel of the input image to one of a set of categories.

[0112] As another example, if the input to the neural network is an internet resource (e.g., a webpage), a file, a portion of a file, or features extracted from an internet resource, file, or portion of a file, then the task could be to classify the resource or file. That is, the output generated by the neural network for a given internet resource, file, or portion of a file could be a score for each of a set of topics, each score representing the estimated probability of the internet resource, file, or portion of a file surrounding that topic.

[0113] As another example, if the input to the neural network is features of the impression context of a particular ad, the output generated by the neural network could be a score representing the estimated probability that the particular ad will be clicked.

[0114] As another example, if the input to the neural network is features for personalized recommendations to a user, such as features characterizing the context of the recommendation, like features characterizing the user's previous actions, then the output generated by the neural network could be a score for each of a set of content items, where each score represents the estimated probability that the user will make a favorable response to the recommended content item.

[0115] As another example, if the input to a neural network is a sequence of texts in one language, the output of the neural network can be a score for each of a set of texts in another language, where each score represents the estimated probability that the text in the other language is correctly translated into the input text.

[0116] As another example, the task could be an audio processing task. For instance, if the input to the neural network is a sequence representing spoken language, the output generated by the neural network could be a score for each segment of a set of texts, with each score representing an estimated probability that the segment of text is a correct transcribed record of the spoken language.

[0117] As another example, the task could be a keyword soptting task, where if the input to the neural network is a sequence representing spoken utterance, the output generated by the neural network can indicate whether a specific word or phrase (“hotword”) was spoken in the spoken utterance. As yet another example, if the input to the neural network is a sequence representing spoken utterance, the output generated by the neural network can identify the natural language used to speak that utterance.

[0118] As another example, the task can be a natural language processing or understanding task, such as an implicit task, a paraphrasing task, a text similarity task, an emotion task, a sentence completion task, a grammatical task, etc., which operate on a text sequence of some natural language.

[0119] As another example, the task could be a text-to-speech task, where the input is natural language text or features of natural language text, and the network output is a spectrogram or other data that defines the audio of the text spoken in natural language.

[0120] As another example, the task could be a health prediction task, where the input is the patient’s electronic health record data and the output is a prediction related to the patient’s future health, such as a predicted treatment to be given to the patient, the likelihood of the patient experiencing an adverse health event, or a predicted diagnosis for the patient.

[0121] As another example, the task could be an agent control task, where the input is an observation characterizing the state of the environment, and the output defines the action the agent should perform in response to the observation. The agent could be, for example, a real-world or simulated robot, a control system of an industrial facility, or a control system that controls different types of agents.

[0122] As another example, the task could be a genomics task, where the input is a sequence representing a DNA sequence or other molecular sequence fragment, and the output is an embedding of the fragment for a downstream task, for example, by using unsupervised learning techniques on a dataset of DNA sequence fragments or the output of a downstream task. Examples of downstream tasks include promoter site prediction, methylation analysis, predicting the functional effects of non-coding variants, etc.

[0123] In some cases, a machine learning task is a combination of multiple individual machine learning tasks; that is, the neural network is configured to perform multiple different individual machine learning tasks, such as the two or more machine learning tasks mentioned above. For example, the neural network may be configured to perform multiple individual natural language understanding tasks. Optionally, the network input may include an identifier of the individual natural language understanding task to be performed on the network input. As another example, the neural network may be configured to perform multiple individual image processing or computer vision tasks, that is, to generate the outputs of multiple different individual image processing tasks in parallel by processing a single input image.

[0124] Embodiments of the subject matter and functional operation described in this disclosure may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this disclosure and their structural equivalents), or combinations thereof. Embodiments of the subject matter described in this disclosure may be implemented as one or more computer programs, for example, modules of one or more computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or combinations thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to transmit encoded information to a suitable receiving device for execution by the data processing device.

[0125] The term "data processing device" refers to data processing hardware, including various means, devices, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The means may also be or further include special-purpose logic circuitry, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Optionally, in addition to hardware, the data processing device may also include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0126] Computer programs (also referred to or described as programs, software, software applications, applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, for example, stored in a markup language file, stored in a single file dedicated to said program, or stored in one or more coordination files, which may be files storing one or more modules, subroutines, or portions of code. Computer programs can be deployed to execute on one or more computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0127] For a system consisting of one or more computers, being configured to perform a specific operation or action means that the system has installed software, firmware, hardware, or a combination thereof that, when running, causes the system to perform the operation or action. For one or more computer programs, being configured to perform a specific operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0128] In this disclosure, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs different from the inputs. An engine can be a coded functional block, such as a library, platform, software development kit (SDK), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet, laptop, music player, e-book reader, laptop or desktop computer, personal digital assistant (PDA), smartphone, or other fixed or portable device including one or more processors and computer-readable media. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0129] The processes and logic flows described in this disclosure can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating outputs. These processes and logic flows can also be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmable computers.

[0130] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for running or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is effectively coupled to one or more mass storage devices to receive data from or transfer data to them, or both includes and is effectively coupled to said mass storage devices. However, a computer does not necessarily include such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, PDA, mobile audio or video player, game console, Global Positioning System (GPS) receiver, or portable storage device, such as a Universal Serial Bus (USB) flash drive, etc.

[0131] Computer-readable media used for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, as examples, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0132] To provide interaction with the user, embodiments of the subject matter described in this disclosure can be implemented on a computer with a display device, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse, trackball, or a presence-sensitive display or other surface through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. Input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving files to and from the device used by the user, for example, sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone, running a messaging application, and receiving response messages from the user.

[0133] Embodiments of the subject matter described in this disclosure can be implemented in a computing system that includes backend components, such as a data server; or middleware components, such as an application server; or frontend components, such as a client computer having a graphical user interface, web browser, or application through which a user can interact with an implementation of the subject matter described in this disclosure; or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0134] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The relationship between clients and servers arises from computer programs running on their respective computers, and they have a client-server relationship. In some embodiments, the server sends data, such as HTML pages, to a user device for example, to display data to a user interacting with the user device and to receive user input from that user, who is the client. Data generated at the user device, such as the results of user interactions, may be received by the server from the user device.

[0135] In addition to the above embodiments, the following embodiments are also innovative.

[0136] Example 1 is a method for training a machine learning model configured to generate predictions of depth maps of an input image, wherein the machine learning model includes multiple parameters to be updated during training. The method includes: receiving data representing training samples comprising multiple image pairs, each image pair comprising a target image of a specific scene captured from a first direction by a first image sensor and a reference image of the specific scene captured from a second direction by a second image sensor; for each of the multiple image pairs, generating a compression cost capacity for the image pair, wherein the compression cost capacity includes one or more cost values ​​selected for each pixel of the reference image in the image pair; providing the compression cost capacity as input to the machine learning model; using the machine learning model to generate output data representing a predicted disparity map of the compression cost capacity; and generating a total loss using the predicted disparity map of the compression cost capacity, the total loss including boundary loss, occlusion loss, and migration loss; and updating multiple parameters of the machine learning model by minimizing the total loss of the multiple image pairs.

[0137] Example 2 is the method of Example 1, wherein generating the compression cost capacity of the image pair includes: generating the cost capacity of the image pair, wherein the cost capacity includes multiple cost values ​​for each pixel of the reference image of the image pair at different disparity levels; for each pixel of the reference image, generating a waveform of the pixel along the disparity level axis based on the multiple cost values ​​of the pixel; determining multiple local minima of the multiple cost values ​​in the waveform; and generating the compression cost capacity of the reference image by selecting one or more local minima and their corresponding disparity levels from the multiple local minima.

[0138] Example 3 is a method of Example 1 or 2, further comprising generating a boundary loss for the image pair, the generation comprising: generating a boundary map of the image pair using a trained machine learning model, wherein the boundary map includes a probability of each pixel of a reference image, the probability indicating the likelihood that the pixel is a boundary pixel; and generating a boundary loss for the image pair based on the predicted disparity map and the boundary map, comprising: generating a depth gradient based on the predicted disparity map, generating an enhanced boundary confidence map based on the boundary map, and multiplying the depth gradient with the enhanced boundary confidence map to generate the boundary loss.

[0139] Example 4 is a method of any one of Examples 1-3, further comprising generating an occlusion loss for an image pair, the occlusion loss for generating the image pair comprising: swapping the target image and the reference image of the image pair to generate a swapped image pair; using the machine learning model to generate data representing a predicted second disparity map of the swapped image pair; and generating an occlusion loss for the image pair based on the predicted disparity map and the predicted second disparity map.

[0140] Example 5 is the method of Example 4. The occlusion loss of the image pair generated based on the predicted disparity map and the predicted second disparity map further includes: for each pixel in the reference image of the image pair, determining the disparity difference by comparing the disparity values ​​of pixels from the predicted disparity map and the predicted second disparity map; determining whether the disparity difference is less than or equal to a threshold; if the disparity difference is determined to be less than or equal to the threshold, calculating the pixel-by-pixel loss of the pixels to be included in the occlusion loss; and if the disparity difference is determined to be greater than the threshold, setting the pixel-by-pixel loss to zero.

[0141] Example 6 is a method of any one of Examples 1-5, further comprising generating a transfer loss for an image pair, the generation comprising: processing a predicted disparity map of the image pair using a neural network to generate a first feature map; processing a base true disparity map of another image pair using the neural network to generate a second feature map; generating data of a first Gram matrix representing the first feature map; generating data of a second Gram matrix representing the second feature map; and generating a transfer loss for the image pair based on the sum of squared distances between the first Gram matrix and the second Gram matrix.

[0142] Example 7 is a method of any one of Examples 1-6, further comprising: deploying a trained machine learning model with updated multiple parameters on an edge device; and generating a predicted disparity map of the input image pair by performing an inference operation on the trained machine learning model used to process the input image pair.

[0143] Example 8 is a system comprising one or more computers and one or more storage devices storing instructions, wherein when executed by one or more computers, the instructions cause the one or more computers to perform respective operations, the operations including the methods of any one of Examples 1-7.

[0144] Example 9 is one or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform respective operations, including the methods of any one of Examples 1-7.

[0145] While this disclosure contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features of particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this disclosure may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although the foregoing features may be described as operating in certain combinations, or even initially claimed, in some cases one or more features from a claimed combination may be removed from said combination, and the claimed combination may be for sub-combinations or variations thereof.

[0146] Similarly, although operations are described in a specific order in the drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all shown operations to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0147] This disclosure has described specific embodiments of the subject matter. Other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in different orders and still achieve the desired results. As an example, the processes shown in the figures do not necessarily require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for training a machine learning model, the method comprising: The machine learning model is configured to predict a depth map of an input image, wherein the machine learning model includes multiple parameters to be updated during training, and the method includes: Receive data representing training samples, the training samples including multiple image pairs, each image pair including a target image of a specific scene captured from a first direction by a first image sensor and a reference image of the specific scene captured from a second direction by a second image sensor; For each of the plurality of image pairs, a compression cost capacity is generated for the image pair, wherein the compression cost capacity includes one or more cost values ​​selected for each pixel of a reference image in the image pair; the compression cost capacity is used as input to the machine learning model; output data representing a predicted disparity map of the compression cost capacity is generated using the machine learning model; and a total loss is generated using the predicted disparity map of the compression cost capacity, the total loss including boundary loss, occlusion loss, and migration loss; and The parameters of the machine learning model are updated by minimizing the total loss of the multiple image pairs; This also includes an occlusion loss for generating the image pairs, the occlusion loss for generating the image pairs including: The target image and the reference image of the image pair are swapped to generate the swapped image pair; Data representing predicted second disparity maps of the exchanged image pairs is generated using the machine learning model; and The occlusion loss of the image pair is generated based on the predicted disparity map and the predicted second disparity map; The occlusion loss for generating the image pair based on the predicted disparity map and the predicted second disparity map further includes: For each pixel in the reference image of the image pair, the disparity difference is determined based on the disparity values ​​of pixels from the predicted disparity map and the predicted second disparity map; Determine whether the disparity difference is less than or equal to a threshold; If the disparity difference is less than or equal to the threshold, calculate the pixel-by-pixel loss of the pixels to be included in the occlusion loss; and If the disparity difference is greater than the threshold, the pixel-by-pixel loss is set to zero.

2. The method of claim 1, wherein, The compression cost capacity for generating the image pairs includes: The cost capacity for generating the image pair includes multiple cost values ​​for each pixel of the reference image of the image pair at different disparity levels. For each pixel of the reference image, a waveform of the pixel is generated along the horizontal disparity axis based on the plurality of cost values ​​of the pixel; and multiple local minima of the plurality of cost values ​​in the waveform are determined; and The compression cost capacity of the reference image is generated by selecting one or more local minima from the plurality of local minima and their corresponding disparity levels.

3. The method of claim 1, wherein, It also includes a boundary loss for generating the image pairs, the boundary loss for generating the image pairs comprising: A boundary map of the image pair is generated using a trained machine learning model, wherein the boundary map includes a probability for each pixel of the reference image, the probability indicating the likelihood that the pixel is a boundary pixel; and Generating a boundary loss for the image pair based on the predicted disparity map and the boundary map includes: generating a depth gradient based on the predicted disparity map; generating an enhanced boundary confidence map based on the boundary map; and generating the boundary loss by multiplying the depth gradient with the enhanced boundary confidence map.

4. The method of claim 1, wherein, It also includes a transfer loss for generating the image pairs, the transfer loss for generating the image pairs comprising: The predicted disparity map of the image pair is processed using a neural network to generate a first feature map; The neural network is used to process the base true disparity map of another image pair to generate a second feature map; Generate data representing the first Gram matrix of the first feature map; Data for generating a second Gram matrix representing the second feature map; and The migration loss for the image pair is generated based on the sum of the squared distances between the first Gram matrix and the second Gram matrix.

5. The method of claim 1, wherein, Also includes: Deploy trained machine learning models with updated parameters on edge devices; as well as A predicted disparity map of the input image pair is generated by performing an inference operation on the trained machine learning model used to process the input image pair.

6. A system comprising one or more computers and one or more storage devices storing instructions, wherein when executed by the one or more computers, the instructions trigger the one or more computers to perform respective operations to train a machine learning model, the machine learning model being configured to predict a depth map of an input image, wherein the machine learning model includes multiple parameters to be updated during training, the operations comprising: Receive data representing training samples, the training samples including multiple image pairs, each image pair including a target image of a specific scene captured from a first direction by a first image sensor and a reference image of the specific scene captured from a second direction by a second image sensor; For each of the plurality of image pairs, a compression cost capacity is generated for the image pair, wherein the compression cost capacity includes one or more cost values ​​selected for each pixel of a reference image in the image pair; the compression cost capacity is used as input to the machine learning model; output data representing a predicted disparity map of the compression cost capacity is generated using the machine learning model; and a total loss is generated using the predicted disparity map of the compression cost capacity, the total loss including boundary loss, occlusion loss, and migration loss; and The parameters of the machine learning model are updated by minimizing the total loss of the multiple image pairs; The operation further includes generating an occlusion loss for the image pair, the occlusion loss for generating the image pair including: The target image and the reference image of the image pair are swapped to generate the swapped image pair; Data representing predicted second disparity maps of the exchanged image pairs is generated using the machine learning model; and The occlusion loss of the image pair is generated based on the predicted disparity map and the predicted second disparity map; The occlusion loss for generating the image pair based on the predicted disparity map and the predicted second disparity map further includes: For each pixel in the reference image of the image pair, the disparity difference is determined based on the disparity values ​​of pixels from the predicted disparity map and the predicted second disparity map; Determine whether the disparity difference is less than or equal to a threshold; If the disparity difference is less than or equal to the threshold, calculate the pixel-by-pixel loss of the pixels to be included in the occlusion loss; and If the disparity difference is greater than the threshold, the pixel-by-pixel loss is set to zero.

7. The system of claim 6, wherein, The compression cost capacity for generating the image pairs includes: The cost capacity for generating image pairs, wherein the cost capacity includes multiple cost values ​​for each pixel of the reference image of the image pair at different disparity levels; For each pixel of the reference image, a waveform of the pixel is generated along the horizontal disparity axis based on the plurality of cost values ​​of the pixel; and multiple local minima of the plurality of cost values ​​in the waveform are determined; and The compression cost capacity of the reference image is generated by selecting one or more local minima from the plurality of local minima and their corresponding disparity levels.

8. The system of claim 6, wherein, The operation further includes a boundary loss for generating the image pairs, the boundary loss for generating the image pairs including: A boundary map of the image pair is generated using a trained machine learning model, wherein the boundary map includes a probability for each pixel of the reference image, the probability indicating the likelihood that the pixel is a boundary pixel; and Generating a boundary loss for the image pair based on the predicted disparity map and the boundary map includes: generating a depth gradient based on the predicted disparity map; generating an enhanced boundary confidence map based on the boundary map; and generating the boundary loss by multiplying the depth gradient with the enhanced boundary confidence map.

9. The system of claim 6, wherein, The operation further includes a transfer loss for generating the image pairs, the transfer loss for generating the image pairs including: The predicted disparity map of the image pair is processed using a neural network to generate a first feature map; The neural network is used to process the base true disparity map of another image pair to generate a second feature map; Generate data representing the first Gram matrix of the first feature map; Data for generating a second Gram matrix representing the second feature map; and The migration loss for the image pair is generated based on the sum of the squared distances between the first Gram matrix and the second Gram matrix.

10. The system according to claim 6, characterized in that, The operation further includes: Deploying trained machine learning models with updated multiple parameters on edge devices; and A predicted disparity map of the input image pair is generated by performing an inference operation on the trained machine learning model used to process the input image pair.

11. One or more computer-readable storage media embodying software that, when executed, is operable to: The computer-readable storage medium stores instructions that, when executed by one or more computers, trigger the computers to perform respective operations to train a machine learning model. The machine learning model is configured to predict a depth map of an input image. The machine learning model includes multiple parameters to be updated during training. The operations include: Receive data representing training samples, the training samples including multiple image pairs, each image pair including a target image of a specific scene captured from a first direction by a first image sensor and a reference image of the specific scene captured from a second direction by a second image sensor; For each of the plurality of image pairs, a compression cost capacity is generated for the image pair, wherein the compression cost capacity includes one or more cost values ​​selected for each pixel of a reference image in the image pair; the compression cost capacity is used as input to the machine learning model; output data representing a predicted disparity map of the compression cost capacity is generated using the machine learning model; and a total loss is generated using the predicted disparity map of the compression cost capacity, the total loss including boundary loss, occlusion loss, and migration loss; and The parameters of the machine learning model are updated by minimizing the total loss of the multiple image pairs; The operation further includes generating an occlusion loss for the image pair, the occlusion loss for generating the image pair including: The target image and the reference image of the image pair are swapped to generate the swapped image pair; Data representing predicted second disparity maps of the exchanged image pairs is generated using the machine learning model; and The occlusion loss of the image pair is generated based on the predicted disparity map and the predicted second disparity map; The occlusion loss for generating the image pair based on the predicted disparity map and the predicted second disparity map further includes: For each pixel in the reference image of the image pair, the disparity difference is determined based on the disparity values ​​of pixels from the predicted disparity map and the predicted second disparity map; Determine whether the disparity difference is less than or equal to a threshold; If the disparity difference is less than or equal to the threshold, calculate the pixel-by-pixel loss of the pixels to be included in the occlusion loss; and If the disparity difference is greater than the threshold, the pixel-by-pixel loss is set to zero.

12. The computer-readable storage medium of claim 11, wherein, The compression cost capacity for generating the image pairs includes: The cost capacity for generating the image pair includes multiple cost values ​​for each pixel of the reference image of the image pair at different disparity levels. For each pixel of the reference image, a waveform of the pixel is generated along the horizontal disparity axis based on the plurality of cost values ​​of the pixel; and multiple local minima of the plurality of cost values ​​in the waveform are determined; and The compression cost capacity of the reference image is generated by selecting one or more local minima from the plurality of local minima and their corresponding disparity levels.

13. The computer-readable storage medium of claim 11, wherein, The operation further includes a boundary loss for generating the image pairs, the boundary loss for generating the image pairs including: A boundary map of the image pair is generated using a trained machine learning model, wherein the boundary map includes a probability for each pixel of the reference image, the probability indicating the likelihood that the pixel is a boundary pixel; and Generating a boundary loss for the image pair based on the predicted disparity map and the boundary map includes: generating a depth gradient based on the predicted disparity map; generating an enhanced boundary confidence map based on the boundary map; and generating the boundary loss by multiplying the depth gradient with the enhanced boundary confidence map.

14. The computer-readable storage medium of claim 11, wherein, The operation further includes a transfer loss for generating the image pairs, the transfer loss for generating the image pairs including: The predicted disparity map of the image pair is processed using a neural network to generate a first feature map; The neural network is used to process the base true disparity map of another image pair to generate a second feature map; Generate data representing the first Gram matrix of the first feature map; Data for generating a second Gram matrix representing the second feature map; and The migration loss for the image pair is generated based on the sum of the squared distances between the first Gram matrix and the second Gram matrix.

Images