Method and system for generating depth images

Abstract

A method, system and computer readable storage medium for generating a depth image are provided, the method comprising: obtaining data representing a first image generated by a first sensor and a second image generated by a second sensor, wherein the first image and the second image each comprise a plurality of pixels; for each pixel of the plurality of pixels comprised by the first image, determining whether the pixel is a boundary pixel associated with a boundary of an object represented by the first image; for each pixel of the first image, determining an optimized penalty value for the pixel from a plurality of candidate penalty values; generating an optimized cost function for the first image based on the optimized penalty values of the plurality of pixels; and generating a depth image for the first image based on the optimized cost function.

Description

Technical Field

[0001] This specification relates to image processing, and more specifically, to stereo-based dense depth images. Background Technology

[0002] Typically, depth images can include pixel-wise information, which relates to the distance (e.g., a line connecting the focal points of two image sensors) of a portion of the scene represented by a corresponding pixel in the raw image acquired by the sensor from the sensor's observation position. Based on the density of the pixel-wise distance information, depth images can be further categorized into sparse depth images and dense depth images. For example, a dense depth image may contain distance information for every pixel in the raw image. For instance, a regular dense depth image can be generated from a corresponding sparse depth image using interpolation or artificial intelligence models.

[0003] Dense depth images are widely used in computer graphics and computer vision. Techniques based on dense depth images can include: generating image effects, such as simulating shallow depth of field by blurring pixels in the original image to varying degrees, simulating fog effects on objects in the original image, and generating appropriate shadows and shades for objects in the original image; generating 3D models from 2D images; and rendering 3D scenes using Z-buffering and Z-culling, to name just a few. Furthermore, in some implementations, dense depth images can be used for localization and movement or pose control.

[0004] Conventional methods for extracting depth information from a scene to generate dense depth images can generally be divided into two categories: active methods and passive methods. Active methods typically involve emitting light signals, such as beams of light or laser beams, towards objects in the scene and measuring the corresponding round-trip time of the light signal. Techniques and sensors employing active methods include time-of-flight (TOF) cameras and light detection and ranking (LIDAR) methods. Passive methods, on the other hand, typically involve generating multiple raw images (e.g., a reference raw image and one or more target raw images) for a specific scene using two or more sensors (e.g., stereo cameras) with known relative displacement and orientation, and generating a depth image for each pixel in the raw images by minimizing the photometric differences after projection (e.g., correction). Summary of the Invention

[0005] The techniques described in the following specification relate to using passive methods to generate dense depth images to extract depth information.

[0006] More specifically, because active methods (e.g., TOF or LiDAR cameras) acquire depth information of objects in a specific scene by measuring the round-trip time of each emitted light signal, their performance is susceptible to weather conditions (e.g., rain and snow), external interference, and the limited power of the emitted light source. Furthermore, active methods typically require high-precision hardware to accurately measure the round-trip time, increasing the cost of acquiring suitable depth images, thus hindering scalability.

[0007] Compared to active methods, passive methods offer the advantage of generating depth images at a lower cost and exhibiting greater robustness to varying weather, interference, and power conditions. One typical passive method generates a depth image by minimizing luminance differences (e.g., cost) through constraints in the pixel matching process and smoothing constraints in the optimization process. More specifically, smoothing constraints penalize discontinuities in depth values ​​across all pixels in the depth image. Another exemplary passive method includes smoothing constraints based on depth discontinuities derived from pixel-wise color information. However, because these typical passive methods apply uniform smoothing constraints to all pixels or individual smoothing constraints based on pixel-wise color information, the output depth image can sometimes be inaccurate. For example, depth images generated by typical passive methods often fail to properly correlate with the individual objects presented in the original image. Furthermore, the depth image may exhibit significant discontinuities within the same object captured in the original image.

[0008] The techniques described in this specification relate to generating accurate depth images from one or more original images by minimizing an optimization cost function. More specifically, the optimization cost function is generated based on an optimization penalty value tailored to pixels located on the boundaries of objects acquired in the original images. The described techniques also involve generating predictions of boundary pixels using a machine learning model, and determining the optimization penalty value for the boundary pixels based on pixel-by-pixel saliency information.

[0009] The term "sensor" as used throughout refers to various suitable optical sensors used to acquire information about a scene. For example, sensors may include image sensors (e.g., digital single-lens reflex (DSLR) cameras), laser sensors, or video sensors.

[0010] Throughout this specification, the term "raw image" refers to data acquired by the aforementioned sensors. For example, a raw image may include image data corresponding to photographs or video clips captured by a DSLR camera.

[0011] The term "boundary pixel" as used throughout refers to a pixel that broadly represents the boundary of an object relative to its background, or the pixel representing the boundary of two or more partially overlapping or intersecting objects captured in the original image. More specifically, the boundary pixels of an object can include the pixels at the outermost edge of the object presented in the original image. Furthermore, boundary pixels can include pixels of a specific width. For example, a boundary pixel can be a single pixel, two pixels, or five pixels away from the edge of the object. The width of the boundary pixel can be determined based on the size of the object, the size of the original image, or the resolution of the original image.

[0012] The term "cost function," used throughout this paper, refers to the objective function that minimizes the photometric difference to generate a depth image from an input image. The cost function can take the input location of each pixel in the original image, the direction of the cost path, and a specific disparity level, and output a specific value (e.g., a cost value) representing a measure of photometric difference. The term "cost value" refers to the value output by the cost function, which is used to process a specific input (e.g., a specific pixel at a specific disparity level). Typically, the cost function for an input map includes multiple pixels, so a single cost function can generate multiple cost values, each corresponding to a specific pixel. The techniques described in this paper involve optimizing the functional form of the cost function to generate accurate depth images, which are described in detail below.

[0013] For the sake of simplicity, in the following instructions, "original image" will also be referred to as image, "dense depth image" will also be referred to as depth image, and "cost value" will also be referred to as cost.

[0014] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages.

[0015] The techniques described below can improve the accuracy of generating depth maps from one or more images representing a scene. More specifically, the techniques described below include a corresponding smoothness penalty for boundary pixels in the image, which relaxes the requirement for depth continuity across object boundaries. Therefore, the accuracy of the depth map is significantly improved compared to conventional techniques. Furthermore, the penalty values ​​for boundary pixels and optionally other pixels are optimized so that the generated depth image can also substantially preserve the geometric details (e.g., the shape and pose of the objects captured in the image). For example, when generating a depth map using the techniques described, details such as human fingers and hair captured in the original image are preserved, whereas in conventional techniques, such geometric details may be sacrificed or even ignored to improve the overall accuracy of the generated depth map.

[0016] Furthermore, the techniques described below can be easily scaled up and pipelined to efficiently process larger or large volumes of input images. More specifically, a system configured to perform the techniques can pipeline the process of obtaining optimized penalty values ​​and optimizing the corresponding cost functions in a distributed manner. For example, the system can first generate multiple candidate cost functions based on multiple candidate penalty values. Then, the system can assign each candidate cost function to a corresponding computing device to compute the corresponding cost value using parallel operations. Subsequently, the system can determine the optimized penalty value and the optimized cost function based on the corresponding cost value.

[0017] Furthermore, the techniques described in this specification can be robustly implemented on input images acquired under different conditions. Smoothness penalty values ​​for boundary values ​​are optimized based on pixel-wise saliency information. Systems employing this technique can generate accurate depth images from input images acquired under various conditions, including images acquired using the same camera under different background lighting or weather conditions, images acquired at the same location but containing objects of different numbers, sizes, colors, and positions, and images of the same scene acquired using different cameras with different light source powers, to name just a few.

[0018] Furthermore, the techniques described below can further improve computational efficiency by leveraging machine learning models to generate predictions of boundary pixels, without needing to identify the texture and classification information of the input image. This improves computational efficiency because for input images of varying sizes, each containing a different number and type of objects, it is typically difficult to apply a general machine learning model to generate texture and classification predictions for every object in such input images.

[0019] One or more embodiments of the subject matter in this specification will be described in detail with reference to the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become more apparent from the following description, drawings, and claims. Attached Figure Description

[0020] Figure 1 An exemplary system for generating optimized depth images by optimizing penalty values ​​is shown;

[0021] Figure 2 An exemplary boundary extraction engine for generating boundary pixel predictions for input data is shown;

[0022] Figure 3 An exemplary method is shown for an exemplary penalty optimization engine configured to determine the optimization penalty value based on morphological analysis;

[0023] Figure 4 It shows the use of Figure 1 A flowchart illustrating an exemplary process by which the system generates optimized depth images;

[0024] Figure 5 An exemplary process for generating significance values ​​based on one or more waveforms is shown;

[0025] The same reference numerals and names in different figures denote the same elements. Detailed Implementation

[0026] Figure 1 An exemplary system 100 is shown that generates an optimized depth image using an optimized penalty value. Typically, system 100 is used to optimize a stereo dense depth image generated from image data.

[0027] like Figure 1 As shown, the system 100 for performing the techniques described herein may include an image processing subsystem 120 for processing input observation data 115 to generate image data with depth information as output data 180. The observation data 115 may be raw images captured by one or more cameras, multiple pairs of images for stereo analysis, or filtered images processed according to specific image processing requirements. The output data 180 may include a depth image (also called a depth map) or a disparity map generated from two or more images acquired for a scene.

[0028] In some embodiments, system 100 may further include a sensor subsystem 110 having multiple sensors 101a-101z (e.g., cameras) configured to acquire image data (e.g., images or videos) of a specific scene and provide the acquired image data as observation data 115 to an image processing subsystem 120. Sensor subsystem 110 may include audiovisual (AV) devices (e.g., an AV stream mixer including multiple cameras or communicatively connected to multiple cameras, or a mobile device including multiple cameras (e.g., a smartphone, tablet, or laptop, to name a few).

[0029] Alternatively, sensor subsystem 110 may also preprocess the acquired image data to generate observation data 115. For example, sensor subsystem 110 may filter the image data based on criteria such as image quality, the number of images required, or generate multiple image pairs for image processing subsystem 120. Each of the multiple image pairs may include a reference image acquired by a first sensor for a particular scene and a target image acquired by a second sensor for the same scene, but the second sensor is located at a different location than the first sensor.

[0030] To generate output data 180, the image processing subsystem 120 can process the received observation data 115 (e.g., image pairs). This processing may include: determining the boundary pixels of at least a portion of the observation data 115 (e.g., a reference image of the image pair), determining an optimized penalty value based on pixel-wise saliency information to generate an optimized cost function, and obtaining depth information as a result of minimizing the cost value based on the optimized cost function. The functionality of the image processing subsystem 120 is described in detail below.

[0031] Image processing subsystem 120 can provide output data 180 with depth information (e.g., depth image or parallax image) to external devices, systems, or subsystems for further analysis or manipulation. For example, the external device, system, or subsystem may include image blurring module 190, configured to process the depth information of the received image and blur corresponding pixels in the image to reproduce a depth-of-field effect; that is, image blurring module 190 can automatically blur pixels located far from the scene or not near the camera's focal length. As another example, the external module may include a 3D modeling system configured to generate a 3D model of a scene acquired from 2D images.

[0032] Returning to the image pairs in observation data 115, as previously described, sensor subsystem 110 can generate image pairs from image data acquired by sensors 101a-101z (or multiple sensors can be simultaneously numbered 101). More specifically, sensor subsystem 110 can position sensors 101 at different locations to acquire images presenting substantially the same scene, using the first image from the first sensor 101a as a reference image and the second image from the second sensor 101b as a target image, generating a first image pair including the first and second images. Typically, the subsystem can use multiple images acquired by each sensor as reference images and pair these reference images with one or more target images acquired by other sensors to form multiple image pairs. In some embodiments, observation data 115 only includes corresponding images acquired by each sensor, and the image processing subsystem 120 generates image pairs based on observation data 115 for subsequent operations.

[0033] Image processing subsystem 120 may include: a boundary extraction engine 130 configured to acquire boundary information (e.g., determine boundary pixels) from input observation data 115; an initial cost generator 140 configured to generate raw cost information when there is pixel correlation between image pairs; a penalty storage unit 160 configured to store and provide penalty values; a penalty optimization engine 170 configured to determine an optimized penalty value based on pixel-wise saliency information; and a cost optimization engine 150 configured to generate an optimized cost function based at least on the optimized penalty value. The components included in image processing subsystem 120 are described in more detail below.

[0034] The boundary extraction engine 130 can receive observation data 115 and extract boundary information from the observation data 115. More specifically, the boundary extraction engine 130 can use a machine learning model to process a reference image of the image pairs in the observation data 115 and generate predictions about which pixels in the reference image are boundary pixels. The image processing subsystem 120 can associate each pixel in the reference image with the predicted boundary information and store the boundary information in a storage device for subsequent operations. The following describes the process in conjunction with... Figure 2 A detailed description of the machine learning model run by the boundary extraction engine 130.

[0035] The initial cost generator 140 can receive observation data 115 to generate an initial cost function for all pixels in the reference image. The initial cost function can be used to generate an optimized cost function in the image processing subsystem 120. More specifically, for each image pair in the observation data 115, the initial cost generator 140 can perform a block matching process for each pixel in the reference image of the image pair with one or more pixels in the target image of that image pair. During the block matching process, the initial cost generator 140 is configured to generate a matching cost function for the image pair, searching for matching pixels (if any) in the target image within a search range of multiple pixels for the corresponding pixels in the reference image. The disparity hierarchy, as input to the matching cost function, can determine the search range.

[0036] The matching process is constrained by a matching cost function, allowing the system to minimize the matching cost value generated by the matching cost function to obtain matching pixels. The matching cost function can receive the input position and disparity level of a specific pixel to determine the search range. The initial cost generator 140 can automatically generate a set of disparity levels for the block matching process. In some implementations, the user can determine the disparity levels through one or more suitable user interfaces and provide them to the initial cost generator engine 140 or the image processing subsystem 120. Furthermore, the initial cost generator 140 can optionally generate an initial path cost function, which includes a uniform smoothness penalty value for each pixel in the reference image. That is, the initial cost function for each pixel can include the corresponding matching cost function and optionally include the corresponding initial path cost function of the reference image.

[0037] Penalty storage 160 is configured to store a plurality of candidate penalty values ​​165 and provide them to cost optimization engine 150, and optionally to penalty optimization engine 170, to determine an optimized penalty value. The candidate penalty values ​​165 stored in penalty storage 160 can be automatically preset by image processing subsystem 120 or preset by user through any suitable user interface. In some embodiments, penalty storage 160 is included in the storage device of image processing subsystem 120 or in a portion of physical memory. The candidate penalty values ​​165 can include three, six, and ten different values, such as 20, 40, 80, 160, 320, to name just a few. Typically, the candidate penalty values ​​165 can be any suitable real number. Specifically, in some cases, each candidate penalty value can be normalized to a real number between 0 and 1.

[0038] The penalty optimization engine 170 is configured to determine an optimized penalty value 175 for at least the boundary pixels of the reference image from candidate penalty values ​​165. More specifically, the penalty optimization engine 170 may receive cost data 155 from the cost optimization engine 150 based on the candidate penalty values ​​and determine the optimized penalty value for the boundary pixels based on pixel-by-pixel saliency information. The determination of the optimized penalty value using saliency information is described in detail below.

[0039] Once the penalty value is determined, the penalty optimization engine 170 can provide it to the cost optimization engine 150 to generate an optimized cost function. Then, the image processing subsystem 120 uses the optimized cost function to output the depth image or disparity image of the input image pair.

[0040] More specifically, the cost optimization engine 150 can receive input boundary data 135 indicating boundary pixels, initial cost data 145 characterizing the matching cost function of the input image, and multiple candidate penalty values ​​165. For each pixel in the boundary pixels of the reference image, the cost optimization engine 150 can generate multiple candidate cost functions based on different candidate penalty values, minimize the cost of each candidate cost function to obtain a candidate disparity level, and then generate a cost amount based on the candidate disparity level. The cost optimization engine 150 can generate cost data 155 characterizing the cost amount and provide it to the penalty optimization engine 170 for determining the optimized penalty value 175 based on the salience information of each cost amount. Figure 3 The generation of each cost quantity and the determination of the optimization penalty value are described in detail.

[0041] As described above, the cost optimization engine 150 can receive an optimization penalty value 175 from the penalty optimization engine 170 and generate an optimization cost function for each pixel in the reference image. In some embodiments, the cost values ​​of the reference image generated according to each cost function (e.g., a matching cost function, a path cost function, or an optimization cost function) are stored in a corresponding tensor data structure, which is described in detail below.

[0042] System 100, more precisely, image processing subsystem 120, can generate a depth image or parallax image based on an optimized cost function of the input image (or input image pair), and provide output data 180 including the depth image or parallax image to external modules, such as image blurring module 190, for other operations.

[0043] Figure 2 An exemplary boundary extraction engine 200 is shown for generating boundary pixel predictions for input data 230. The boundary extraction engine 200 may, for example, be... Figure 1 Boundary extraction engine 130 in the middle.

[0044] The boundary extraction engine 200 can receive input data 230 comprising multiple images, each image including one or more objects for a corresponding scene. The input data 230 can be included in observation data 115, and as... Figure 1 As described, for example, from sensor subsystem 110. Figure 2 As shown, the input data 230 may include images of three objects: a triangle representing a first texture of a first category, an ellipse representing a second texture of a second category, and a rectangle representing a third texture of a third category. For ease of illustration, different textures of different objects are represented by different fill patterns.

[0045] The boundary extraction engine 200 processes the input data 230 and provides output data 240 for generating an optimized cost function. More specifically, the output data 240 includes predictions for each pixel regarding whether it is a boundary pixel of an object. As described above, a boundary pixel is a pixel in an image that is generally located at the outermost edge of an object, represented by multiple pixels in the image. In some implementations, a boundary pixel may be a pixel in a region within the object that is one or more pixels (e.g., one, two, or five pixels) wide from the outermost edge. For ease of illustration, the boundary pixels or boundary information in the output data 240 are represented by the corresponding contours of the objects (triangles, ellipses, and rectangles) in the output data 240. Boundary pixels representing boundary information may also be referred to as semantic boundaries.

[0046] More specifically, the output data may include the likelihood of each pixel being associated with the boundary of an object in the image. For example, for three pixels located inside a first object, the boundary extraction engine 200 may predict that the first pixel has a 99% likelihood of being a boundary pixel of the first object in the image, the second pixel has a 70% likelihood of being a boundary pixel of the first object, and the third pixel has a 1% likelihood of being a boundary pixel of the first object. The boundary extraction engine 200 may then determine that the first pixel is a boundary pixel of the first object, while the second and third pixels are determined to be pixels inside the first object.

[0047] In some implementations, the boundary extraction engine 200 may include a confidence level that indicates the probability or likelihood that a particular likelihood of a pixel being predicted as a boundary pixel of an object is true. Furthermore, the boundary extraction engine 200 may further perform a linear transformation (e.g., linear regression) on the likelihood prediction to determine the confidence level.

[0048] Furthermore, the boundary extraction engine 200 may include one or more criteria based on predicted pixel-by-pixel likelihood to determine whether a pixel is a boundary pixel. For example, the criterion may be a threshold likelihood of 97%, where any pixel with a likelihood higher than this threshold can be identified as a boundary pixel.

[0049] To generate output data 240 representing semantic boundaries, the boundary extraction engine 200 can be configured to include one or more suitable trained machine learning models. For example, the machine learning model can include neural networks, such as convolutional neural networks, fully convolutional networks (FCNs), and fully connected layers. A detailed description of general machine learning models follows.

[0050] The machine learning model included in the boundary extraction engine 200 may further include an encoder 210 and a decoder 220. The encoder 210 may be a neural network comprising one or more neural network layers. For example, the one or more neural network layers may include convolutional layers followed by fully connected layers. The encoder may receive input data 230 (e.g., an image) and map the input data to a feature map represented by one or more vectors and tensors with real numbers (or perform embedding processing on natural language). The encoder 210 may provide encoded data 215 representing the aforementioned feature map to the decoder 220. The decoder 220 may also be a neural network comprising one or more neural network layers, but the layers in the decoder are generally in the opposite direction to those in the encoder 210. For example, the decoder 220 may first include fully connected layers, followed by one or more convolutional transpose layers. The decoder 220 may process the received encoded data 215 and generate a prediction for each pixel in the input image based on the feature map. In some embodiments, the decoder 220 may utilize piecewise prediction (e.g., pixel-by-pixel prediction) to generate a reconstructed output image that substantially matches the input image.

[0051] As a specific example, encoder 210 and decoder 220 can be configured according to the UNet architecture, i.e., a symmetric architecture spanning two parts. The left part can be considered at least a portion of encoder 210, which may include: one or more convolutional layers configured to perform convolutional computations to process input data (e.g., images) into output features (e.g., tensors in feature maps); pooling layers configured to perform max pooling or average pooling operations on the output features; optionally, it may also include dropout layers configured to invalidate certain features of the output features to mitigate overfitting. The right part can be considered at least a portion of decoder 220, which may include one or more transposed convolutional layers configured to unsample the features generated by encoder 210 in the feature maps; optionally, it may also include one or more convolutional layers configured to cascade, merging the unsampled features and the corresponding features generated by encoder 210 to improve prediction accuracy.

[0052] In some implementations, encoder 210 or decoder 220 can reduce the total number of network parameters by including one or more deep convolutional layers configured to perform deep convolution operations on each input channel separately, with the output of the deep convolutional layer having the same number of channels as the input. Optionally, after the deep convolutional layers, encoder 210 and decoder 220 may further include one or more pointwise convolutional layers to merge the output features generated by the deep convolutional layers.

[0053] Furthermore, encoder 210 may include one or more layer blocks, and decoder 220 may include the same number of corresponding layer blocks. Each block of decoder 220 may receive the output features generated by the corresponding block of encoder 210 as input. More specifically, the input features of a decoder block may be an aggregation of the output features of the previous decoder block and the corresponding encoder. For example, the input features of the last decoder block may be the sum of the output features of the previous decoder block and the first encoder block.

[0054] A machine learning model for generating output data 240 can be trained using specific training samples following standard training steps. Training samples can be images including pixels associated with boundary information. The loss function used to train the parameters of the machine learning model using encoder 210 and decoder 220 can be a loss function representing the difference between the input data and the reconstructed output data associated with the prediction. Typically, any suitable loss function can be used to train the machine learning model, such as hinge loss and log loss, to name a few. In some implementations, boundary information of the pixels in the training samples can be imported into system 100 from an external storage device. Alternatively, the user can manually determine the boundary pixels in the training images.

[0055] It is important to note that the output data 240 generated by the machine learning model omits some information from the input data. For example, the output data 240 may omit texture and classification information for each pixel in the input image. Because additional information from the input data (e.g., input image size, or pre-known texture and category information) is required when predicting texture and classification, by omitting texture and classification information during inference computation, the boundary extraction engine 200 can more robustly and efficiently generate predictions of boundary pixels using the trained machine learning model.

[0056] Typically, the machine learning model described above can be used to receive any kind of digital data input and be trained on that input to generate any type of score, classification, or regression output based on that input. Input data items can include image data (including video data here), audio data, or text data, such as words or word fragments (or their representations, such as embeddings) in natural language. Input data items can include sequence data, such as a sequence of data samples representing digitized audio, or an image represented as a sequence of pixels, or a video represented by a sequence of images, or a sequence representing a sequence of words in natural language.

[0057] In some implementations, the trained machine learning model can generate an output that includes feature representations, which can then be further processed to generate a system output. For example, the system output may include a classification output (e.g., data representing the estimated likelihood that an input data item or an object / element of the input data item belongs to a certain category) for classifying input data items into one of several categories such as image, video, or audio categories, or a segmentation output for segmenting regions of the input data item into, for example, objects or actions represented in an image or video. Alternatively, the system output may be an action selection output from a reinforcement learning system.

[0058] In some implementations, the output may include other data items of the same or different types. For example, the input data item may be an image, audio, or text, and the output data item may be a modified version of the image, audio, or text, for example, by changing the style, content, attributes, pose, etc., of the input data item or one or more objects or elements within the input data item; or by filling in (missing) portions of the input data item; or by predicting other versions of the data item or an extension of the video or audio data item; or by providing an upsampled (or downsampled) version of the input data item. For example, the input data item may be text expressed in a first language, and the output data item may be text translated into another language, or a score of the translation of the text into another language. As another example, the input image may be converted into a video, wireframe model, or CAD model, or a 2D input image may be converted into 3D; or vice versa. Alternatively, the input data item may include features obtained from spoken discourse, a sequence of spoken discourse, or features obtained from a sequence of spoken discourse, and the output data item may include a score for each text fragment in a set of text fragments, each score representing the estimated likelihood of the text fragment as a correct copy based on features. For example, the input data item can be an image, audio, or text, and the output data item can be a representation of the input data item in different formats. For instance, a machine learning model can convert text to speech and vice versa (for speech recognition), or convert an image (or video) to text (e.g., for captions). When generating output containing sequential data, a machine learning model can include one or more convolutions, such as dilated convolutional layers.

[0059] In some implementations, a trained machine learning model receives input data items and processes them to generate feature representations of the input data items based on model parameters. Typically, the feature representation of a data item is an ordered set of numerical values; for example, the data item can be represented as a vector of points in a multidimensional feature space. That is, each feature representation can include the numerical value of each of the multiple features of the input data item. As mentioned earlier, the machine learning model can be used to receive any type of numerical data input as input and generate feature representations based on that input. For example, the input data item, also known as the model input, can be an image, a portion of a document, a text sequence, audio data, medical data, and so on.

[0060] Feature representations can be provided as input to another system, for example, to perform machine learning tasks on model inputs. Example tasks may include feature-based retrieval, clustering, near-repetition detection, validation, feature matching, domain adaptation, and video-based weakly supervised learning. For videos, examples include object tracking across video frames and gesture recognition where gestures are performed by entities depicted in the video.

[0061] If the input to a machine learning model is an image or features extracted from an image, the output generated by the machine learning model for a given image can be a score for each object category in a set of object categories, with each score representing the estimated likelihood that the image contains an image of an object belonging to that category.

[0062] Figure 3 An exemplary method 300 is illustrated, employed by an exemplary penalty optimization engine 390 configured to determine an optimization penalty value based on morphological analysis. This exemplary penalty optimization engine 390 may, for example, be... Figure 1 The penalty optimization engine 170 in the middle.

[0063] Before describing how the penalty optimization engine 390 is able to obtain optimization penalty values, a preliminary system setup (e.g., Figure 1 System 100) can aggregate multiple path cost functions L r To generate a cost function S in functional form. Each path cost function L r The cost of pixels along the path traversed along direction r for the pixel located at (x,y) in the 2D coordinate system at disparity level d can be expressed as:

[0064]

[0065] As shown in formula (1), the path cost function L r The input is the position of a specific pixel at disparity level d, where d belongs to a predetermined disparity level set D. The path cost function L... rThe first term of (x,y,d) represents the matching cost function C(x,y,d), which is used to match the pixel located at (x,y) in the reference image with the corresponding pixel in the target image using the search range determined by the disparity d. Return to... Figure 1 In some implementations, the matching cost function C(x,y,d) for each pixel in the reference image is included in the initial cost function generated by the initial cost generator 140.

[0066] Path cost function L r The second term in (x,y,d) represents the accumulation of the lowest path cost function along the direction r of the previous pixel, with smoothness penalty terms P1(x,y) and P2(x,y). It is important to note that penalty terms P1 and P2 are functions of pixel position, meaning that the smoothness penalty values ​​in this specification depend on the characteristics of the pixels.

[0067] For example, the penalty value generated based on the penalty terms P1(x,y) and P2(x,y) of a pixel depends on whether the pixel is a boundary pixel of an object being sampled in the image. It's important to note that the penalty terms can be used as a function of the pixel position. Each penalty term can return a corresponding penalty value based on the input pixel position.

[0068] In some implementations, the penalty terms P1(x,y) and P2(x,y) can be expressed as functions to determine whether an input pixel is predicted as a boundary pixel. For example, for each pixel in the reference image, the penalty terms can include a step function form, as shown below:

[0069]

[0070]

[0071] As described above, the boundary extraction engine 130 can predict the likelihood of a pixel being a boundary pixel by performing inference operations of a machine learning model, and determine whether the input pixel is a boundary pixel when the likelihood exceeds one or more specific criteria.

[0072] System 100 can generate one or more sets of penalty terms by changing the parameters of the function terms. As an example, system 100 can predetermine penalty terms P1(x,y) and P2(x,y) to include a first set of parameters, where... as well as As an example, system 100 can predetermine penalty terms P1(x,y) and P2(x,y) to include a second set of parameters, where as well as

[0073] In some implementations, the likelihood functions of the penalty terms P1(x,y) and P2(x,y) can be respectively... and It is important to note that the function term P b (x,y) is a function representing the likelihood that each pixel in the image becomes a boundary pixel of an object captured in the image; it is also called a boundary likelihood map. b The (x,y) function can be implemented using the machine learning techniques described above, P b (x, y) can be stored in a matrix data structure, where each element of the matrix represents a value derived from the likelihood, determinism, or likelihood of a pixel being predicted as a boundary pixel by a machine learning model. The function term P... r1 and P r2 It could be a constant real value determined for pixels that are more likely to be boundary pixels, while and It can be a constant real value of other pixels. System 100 can select P from multiple candidate penalty values ​​based on, for example, a specific criterion. r1 P r2 , as well as The system 100 can also provide a user interface for selecting the values ​​of these parameters. The details of generating the penalty terms P1(x,y) and P2(x,y) will be described in more detail below.

[0074] System 100 can further aggregate the path cost function L for all directions r. r This generates an aggregate cost function S(x,y,d) for all directions at disparity level d. The accuracy of the depth image increases with the total number of different directions in the aggregate cost function. The total number of different directions can be 4, 8, or 32, to name just a few. The aggregate cost function S(x,y,d) can be expressed as:

[0075]

[0076] System 100 can be based on the formula d0 = argmin d S(x,y,d) selects the disparity level d from the disparity set D as the disparity value of a pixel, i.e., the disparity value d0 of a pixel results in the minimum aggregate cost value of the cost function S(x,y,d) for all disparity levels in the set D.

[0077] In some implementations, the system can be based on the corresponding path cost function L. r The corresponding cost value is generated by the aggregate cost function S and stored as a tensor. For example, the path-specific cost function L... rThe cost value can be stored in a third-order tensor, for example... Where H and W represent the pixel size of the input image in the height and width directions, This represents the size of the disparity hierarchy set. System 100 can further incorporate the path cost function L... r The cost value is stored in multiple directions within a fourth-order tensor, for example... Where R represents the total number of directions. The aggregate cost function S is derived by accumulating the path cost function L over multiple directions. r Therefore, system 100 can store the cost value generated by the aggregation cost function S in a third-order tensor, for example... With specific path cost function L r The storage is similar.

[0078] like Figure 3 As shown, system 100 can generate a cost amount 310 for a reference image based on the position of a pixel and the corresponding parallax level of the pixel. For example, x represents the position (e.g., coordinate) along the width direction 307, y represents the position along the height direction 305, and each pixel located at the corresponding width and height coordinates can have a corresponding cost value along the parallax axis 303 at each parallax level.

[0079] Returning to the aforementioned penalty optimization engine 390, which is configured to select candidate penalty items, marked as and Among the generated candidate penalty values, the optimized penalty term is determined and marked as... and The generated optimized penalty value. The functional forms of the candidate penalty term and the optimized penalty term can be constructed in a similar manner to the penalty term described above (e.g., step function form or likelihood function form). The system can select values ​​of these functional terms from 0.25, 0.5, 1, 2, 4, 8, and 16 to generate the corresponding optimized penalty value and candidate penalty value, to name just a few.

[0080] It is important to note that the candidate penalty values ​​and optimized penalty values ​​are associated with each pixel in the reference image. In some implementations, system 100 may predetermine a common set of penalty values ​​for each pixel in the image (e.g., P1(x,y) = 16, P2(x,y) = 4 for all pixels), and cover one or more constant coefficients for boundary pixels. In some implementations, system 100 may directly determine the aforementioned penalty terms and corresponding penalty values ​​for all pixels in the reference image without covering them.

[0081] In some implementations, the candidate penalty values ​​can differ from each other by a certain constant multiplier; for example, the constant multiplier can be 2, 4, and 8. When the constant multiplier is 2, for example, for and The first pair of candidate penalty values ​​for the boundary pixels can be 1 and 0.25, the second pair of candidate penalty values ​​for the boundary pixels can be 2 and 0.5, and the third pair of candidate penalty values ​​can be 4 and 1.

[0082] To determine the optimal penalty value, the penalty optimization engine 390 can first provide multiple candidate penalty values ​​165 to the cost optimization engine (e.g., cost optimization engine 150), which can then base its decision on a boundary extraction engine (e.g., ...). Figure 1 The boundary extraction engine 130) generates multiple candidate cost functions for each pixel in the prediction generated reference image.

[0083] Multiple candidate cost functions Each candidate cost function may include a pair of candidate penalty values ​​(e.g., a pair of candidate penalty terms corresponding to the candidate cost function). and The value of the candidate path cost function L in direction r r Alternatively or additionally, each candidate cost function It can include an aggregation cost function based on candidate penalty values.

[0084] Then, the cost optimization engine 150 can provide cost data 155, representing multiple candidate cost functions, to the penalty optimization engine 390. The cost optimization engine 150 can generate multiple cost quantities 310 (or candidate cost quantities 310, since they are generated based on candidate penalty values) associated with the corresponding disparity level of each pixel in the image for a given candidate cost function.

[0085] In response, the penalty optimization engine 390 can determine the function term for each boundary pixel based on the saliency value. and The generated optimized penalty value. More specifically, Engine 170 can perform morphological analysis on the candidate cost values ​​and disparity levels associated with their respective candidate penalty values ​​to determine significance values. As mentioned above, the candidate penalty value can be derived from the candidate penalty items. and The generated output can also be a candidate penalty term. and Candidate values ​​for the constant coefficients in the equation.

[0086] To determine the saliency value, the penalty optimization engine 390 is configured to generate multiple waveforms for each pixel based on multiple candidate cost values ​​to determine multiple candidate saliency values ​​for that pixel, and select one candidate saliency value from the multiple candidate saliency values ​​as the saliency value based on one or more criteria. The following section combines... Figure 5 The generation of significance values ​​is described in detail.

[0087] return Figure 3 To generate waveforms for pixels in the reference image, the penalty optimization engine 390 selects a cost quantity from multiple candidate cost quantities, obtains the corresponding cost value of the pixel at all disparity levels from the selected cost quantities, and generates waveforms based on the corresponding candidate cost values ​​and relevant disparity levels. As an example, the waveform may include a two-dimensional data structure where values ​​along the vertical direction represent candidate cost values ​​for the pixel, and values ​​in the horizontal plane represent the disparity levels used to generate the corresponding candidate cost values ​​for the pixel. As another example, the waveform for the pixel may represent each disparity level d and the corresponding candidate penalty value. and The discrete line function corresponding to the candidate cost value.

[0088] like Figure 3 As shown, without loss of generality, the waveform 350 of a pixel in the reference image can be plotted as a line graph 360 in a 2D coordinate system, where the horizontal axis represents disparity d, and the vertical axis represents the candidate cost value generated according to the candidate cost function at the corresponding disparity level, wherein the candidate cost function includes a candidate penalty term. and In waveform 350 of this pixel, each vertex (e.g., vertex numbered 370) represents the candidate cost value of pixel 375 at possible disparity levels, based on the corresponding candidate penalty value. and It is obtained from the corresponding cost amount generated at that parallax level.

[0089] The penalty optimization engine 390 can identify local minimum candidate cost values ​​and global minimum candidate cost values ​​in a line chart 360. For example, as Figure 3 As shown, the penalty optimization engine 390 can determine the global minimum 320 and the second-lowest local minimum 330 in the line graph 360. In response, the engine 170 can calculate a sharpness value, which is the absolute difference between candidate cost values ​​between neighboring vertices and their corresponding minimums (e.g., 320 and 330) in the line graph 360. For example, as Figure 3 As shown, engine 170 can determine difference 1 (labeled 315) and difference 2 (labeled 325) among the candidate cost values ​​between the global minimum 320 and its corresponding neighboring vertices. Similarly, the engine can further determine difference 3 (labeled 340) and difference 4 (labeled 335) among the candidate cost values ​​between the local minimum 330 and its corresponding neighboring vertices.

[0090] The penalty optimization engine 390 can determine candidate saliency values ​​for pixels based on sharpness values. For example, in response to determining that only one minimum candidate cost value exists in waveform 350, engine 170 can generate candidate saliency values ​​for boundary pixels based on the sum of sharpness values ​​associated with the minimum value. As another example, combined with Figure 3 In response to determining that there are two or more minimum values ​​in waveform 350, engine 170 can determine candidate significance values ​​in the following manner:

[0091] Candidate significance value = Difference 1 + Difference 2 - Difference 3 - Difference 4 Equation (3)

[0092] As described above, the penalty optimization engine 390 can generate multiple waveforms for each pixel in the reference image to determine a saliency value. Each waveform is generated from candidate cost quantities based on different candidate penalty values. More specifically, Figure 5 An exemplary process 500 for generating significance values ​​based on one or more waveforms (513a-c) is shown. This process can be executed by an exemplary penalty optimization engine, for example, Figure 1 Penalty optimization engine 170 or Figure 3 390 in the middle.

[0093] like Figure 5 As shown, combined with Figure 3 The penalty optimization engine 390 can generate multiple candidate cost quantities (503a-c) for all pixels in the reference image. Candidate cost quantities 503a, 503b, and 503c are generated by their respective candidate cost functions, each including a corresponding candidate penalty value. As described above, the three dimensions of the cost quantity represent the disparity level, the first coordinate of the pixel in the first direction, and the second coordinate of the pixel in the second direction perpendicular to the first direction, respectively.

[0094] For each pixel in the reference image, the penalty optimization engine 390 can generate a corresponding waveform from the corresponding cost amount among multiple cost amounts. For example, for Figure 3 In the context of pixel 375, the penalty optimization engine 390 can generate a first waveform 513a from the first candidate cost amount 503a, a second waveform 513b from the second candidate cost amount 503b, and a third waveform 513c from the third candidate cost amount 503c. As described above, each waveform (513a-c) can be represented as a line graph.

[0095] Then, as described above, the penalty optimization engine 390 can determine the corresponding candidate significance value for each waveform. Figure 5 As shown, the penalty optimization engine 390 can generate a first candidate significance value 523a from the first waveform 513a, a second candidate significance value 523b from the second waveform 513b, and a third candidate significance value 523c from the third waveform 513c. As described above, for example, the value of each candidate significance value can be determined by equation (3).

[0096] To determine a significance value from candidate significance values ​​523a-523c, the penalty optimization engine 390 may include one or more predetermined criteria and compare each candidate significance value to a predetermined criterion. For example, a criterion may be a threshold for the candidate significance values, or a threshold for the difference between two candidate significance values. The threshold may be a suitable real number, such as 1, 4, 10, 20, and 100.

[0097] In some implementations, the penalty optimization engine 390 can filter out candidate saliency values ​​that do not meet a threshold and select a candidate saliency value as the saliency value of the pixel from the remaining candidate saliency values ​​that meet the threshold. In some implementations, the penalty optimization engine 390 can select the smallest candidate saliency value as the saliency value of the pixel from the remaining candidate saliency values.

[0098] For example, refer to Figure 5 Assuming the first candidate significance value 523a is 1.5, the second candidate significance value 523b is 5, the third candidate significance value 523c is 100, and the threshold for candidate significance values ​​is 3, then the penalty optimization engine 390 is configured to filter out the first candidate significance value 523a because it is less than the threshold, and select the candidate significance value 523b as the significance value because it is the minimum among the remaining candidate significance values.

[0099] After determining the saliency value of a pixel, the penalty optimization engine 390 can determine the candidate cost quantity associated with the pixel's saliency value as the optimization cost quantity 543. For example, as Figure 5 As shown, the penalty optimization engine 390 can determine the candidate cost amount 503b as the optimized cost amount 543.

[0100] After determining the cost of a pixel (543), the penalty optimization engine (390) can select candidate penalty values ​​as the optimized penalty values ​​for the pixels. and This candidate penalty value is used to generate the optimized cost amount 543 from multiple candidate penalty values. Therefore, the penalty optimization engine 390 can generate the optimized penalty value with the lowest cost value using sufficient significance values.

[0101] The penalty optimization engine 390 can repeat the above process to determine the optimized penalty value for all pixels in the reference image. Back to... Figure 1 Then, the penalty optimization engine 390 can provide the determined optimization penalty value 175 to the cost optimization engine 150 to generate output data 180, which includes at least the depth information of each pixel in the reference image (e.g., a disparity map or depth map 553).

[0102] It should be noted that, for ease of explanation, only three candidate cost quantities were generated in process 500. However, the penalty optimization engine 390 can also generate more than three candidate cost functions, such as 5, 10, 20, or 100 candidate cost quantities associated with each candidate penalty value.

[0103] Figure 4 It shows the use of Figure 1 A flowchart of an exemplary process 400 for generating an optimized depth image by system 100 is provided. For convenience, process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, a system for optimizing a stereo dense depth image, such as... Figure 1 System 100, with proper programming, can execute process 400.

[0104] This system acquires data from a sensor subsystem, for example, Figure 1 The sensor subsystem 110 contains multiple sensors that collect observation data. More specifically, the system acquires data representing a first image of a specific scene generated by a first sensor and a second image of the same scene generated by a second sensor (410). The specific scene may include one or more objects, and the first and second images may each include multiple pixels. In some embodiments, the system may combine the first and second images to form an image pair. In the image pair, the first image may be a reference image taken by a reference camera, and the second image may be a target image taken by an auxiliary camera.

[0105] For each of the multiple pixels included in the first image, the system determines whether the pixel is a boundary pixel associated with the boundary of an object detected in the particular scene (420).

[0106] More specifically, when a pixel in the first image is determined to be a boundary pixel, the system can be configured to include a trained machine learning model used to generate pixel predictions when processing the input image. The machine learning model may include an encoder and a decoder, for example, Figure 2 The encoder 210 and decoder 220 are described above. The machine learning model can be trained using training samples (e.g., images) of boundary pixels determined by the user. The output prediction of the trained machine learning model after inference operations on the input image can include the pixel likelihood associated with the boundary of an object represented by multiple pixels in the first image. The system may also include a criterion containing a likelihood threshold, where all pixels with a likelihood higher than the threshold can be identified as boundary pixels.

[0107] Before performing the optimization process, the system can generate an initial cost function for each pixel included in the first image and use this initial cost function to generate initial cost values ​​for all pixels. When the system performs a block matching process to find the corresponding pixel of a reference pixel within a subset of pixels in the second image within a search range, the initial cost value can represent the difference between the reference pixel in the first image and the corresponding pixel in the second image. More specifically, the search range can be determined by the disparity levels in a predetermined set of disparity levels. The initial cost function can include a matching cost function for measuring the performance of the block matching process between pixels in the first image (e.g., the reference image) and one or more pixels within the search range in the second image (e.g., the target image). The matching cost function for each pixel can receive the pixel location and the disparity levels used to determine the search range as input and output a matching cost value that contributes to the aggregate cost value.

[0108] The system determines the corresponding optimized penalty value (430) for all pixels of the reference image from a set of candidate penalty values ​​(e.g., candidate penalty value 165).

[0109] To determine the optimal penalty value, as described above, the system can first retrieve the set of candidate penalty values ​​from the penalty storage (e.g., penalty storage 160). The candidate penalty values ​​can be ordered according to the size of each candidate penalty value.

[0110] Based on this sequence, the system can generate a set of candidate cost functions for each pair of candidate penalty values. Based on the candidate cost functions, the system can generate a corresponding set of cost amounts for each pixel in the first image (e.g., the reference image).

[0111] The system can then use morphological analysis to determine the saliency value of each pixel. This morphological analysis can be performed by generating multiple waveforms for each pixel, each waveform based on the cost value of the corresponding cost amount for the pixel at different disparity values. In response, for each pixel in the first image, the system can select a candidate penalty value from multiple candidate penalty values ​​as the optimized penalty value for that pixel based on its saliency value.

[0112] More specifically, the system can first generate waveforms for the pixels plotted in the line graph. The system can determine one or more minimum values ​​in the waveform (or line graph), such as the global minimum of disparity values ​​or the second-lowest local minimum of disparity values. Then, the system can generate candidate saliency values ​​for the pixels based at least on the minimum disparity value in the waveform. More specifically, the system can generate candidate saliency values ​​based on the minimum disparity value and the absolute differences between corresponding adjacent vertices of each disparity value in the waveform.

[0113] The system can generate candidate saliency values ​​for pixels for all candidate cost values, and as described above, determine the saliency value of a pixel from the candidate saliency values ​​based on one or more criteria.

[0114] The system can then determine the candidate cost quantities representing the significance values ​​among the respective candidate cost quantities as the optimal cost quantities. The system can then determine the candidate penalty values ​​used to generate the optimal cost quantities as the optimal penalty values. The system can repeat the above process to determine the optimal penalty values ​​for all pixels in the reference image.

[0115] After determining the optimized penalty value for each pixel in the reference image, the system can generate an optimized cost function (440) based on the optimized penalty value. As described above, for pixels in the first image (e.g., the reference image), the system can first generate respective optimized path cost functions along multiple directions using the optimized penalty value. Then, the system can generate an optimized cost function for all pixels in the first image by aggregating the respective optimized path cost functions in multiple directions (i.e., generating an optimized aggregated cost function).

[0116] It should be noted that the system can generate optimized penalty values ​​for all pixels in the reference image at the same time. The optimized penalty values ​​for boundary pixels are usually different and are smaller than the optimized penalty values ​​for non-boundary pixels or internal pixels.

[0117] Although the system is configured to determine the optimized penalty value for all pixels in the reference image, in some implementations, the system may first perform the above process to determine the optimized penalty value for boundary pixels. For pixels in the first image that are not boundary pixels (i.e., non-boundary pixels), the system may assign a common penalty value for these pixels to share. In some implementations, after determining the optimized penalty value for boundary pixels, the system may increase the common penalty value for all non-boundary pixels, thereby reducing computational costs.

[0118] Then, the system generates a depth image (450) of the first image based on the cost value derived from the optimized cost function. As described above, the system can select a disparity value that generates the minimum cost value according to the optimized cost function for each pixel as the corresponding optimized disparity value. The system can further generate a depth image of the first image based on the respective optimized disparity values.

[0119] The system can provide a depth image to a rendering engine configured to blur one or more pixels in a first image to reproduce a depth-of-field effect (460). In some implementations, the system can provide depth images to different external devices to perform different operations. In some implementations, the system can directly provide appropriate optimized disparity values ​​to external devices for additional operations. For example, the external device may include a 3D rendering machine configured to render a 3D scene based on a scene depth map or disparity map. Alternatively, the external device may include a shadow generator configured to generate appropriate shadow effects for the scene based on a depth map or disparity map. Alternatively, the external device may include a 3D model generator configured to generate 3D models based on a depth map or disparity map. Specifically, the 3D model generator may be configured to generate 3D models for components in a dynamic system (e.g., a robotic system), each component having a different pose. The 3D model generator may generate a 3D model of the component based on a depth map or disparity map generated from a 2D image of the component. Optionally, the external device may include one or more machine learning models configured to generate corresponding predictions based on depth maps or disparity maps and incorporating additional data. For example, one of the machine learning models could predict whether an autonomous vehicle traveling along a trajectory will collide with objects in the scene within a specific time period.

[0120] The subject matter and functional embodiments described in this specification can be implemented in digital electronic circuits, tangible computer software or firmware, and computer hardware, including the structures disclosed in this specification and their structural equivalents, or one or more combinations thereof. The subject matter embodiments described in this specification can be implemented as one or more computer programs, such as one or more computer program instruction modules encoded on a tangible, non-transitory storage medium, executed by a data processing device or controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device, executed by the data processing device.

[0121] The term "data processing apparatus" refers to data processing hardware, encompassing all types of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be, or further include, dedicated logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or one or more of the following.

[0122] A computer program can also be called or described as a program, software, computer program application, application, module, software module, script, or code, and can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages. A computer program can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program can, but does not necessarily, correspond to a file in a file system. A program can be stored as part of a file containing other programs or data, for example, in one or more scripts in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as a file storing one or more modules, subroutines, or portions of code. A computer program can be deployed to execute on one or more computers located at a single site or distributed across multiple sites interconnected via a data communication network.

[0123] A system configuration of one or more computers to perform a specific operation or action means that the system has software, firmware, hardware, or a combination thereof installed, the operation of which causes the system to perform the operation or action. A system configuration of one or more computer programs to perform a specific operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0124] As used in this specification, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs distinct from its inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet, laptop, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device, including one or more processors and computer-readable media. Furthermore, two or more engines can be implemented on the same or different computing devices.

[0125] The processes and logic described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic can also be executed by dedicated logic circuitry, such as FPGAs or ASICs, or by a combination of dedicated logic circuitry and one or more programmable computers.

[0126] A computer suitable for executing computer programs can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit (CPU). Typically, the CPU receives instructions and data from read-only memory, random access memory, or both. The basic components of a computer are the CPU for executing instructions and one or more memory devices for storing instructions and data. Special-purpose logic circuitry may supplement or be integrated into the CPU and memory. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer may be operatively coupled to one or more mass storage devices to receive data from, transfer data to, or simultaneously receive and transfer data from one or more mass storage devices. However, a computer does not require such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0127] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0128] To interact with the user, the subject embodiments described in this specification can be implemented on a computer with a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, a keyboard, and a pointing device, such as a mouse, trackball, or a display with a sensitive display or other surface through which the user can provide input to the computer. Other types of devices can also be used to interact with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; user input can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents to and from the device used by the user. For example, in response to a request received from a web browser, a webpage is sent to the web browser on the user's device. Simultaneously, the computer can interact with the user by sending text messages or other forms of messages to personal devices, such as smartphones, running messaging applications, and receiving response messages from the user.

[0129] The embodiments of the subject matter described in this specification can be implemented in a computing system including: a back-end component, such as a data server; or a middleware component, such as an application server; or a front-end component, such as a client computer having a graphical user interface, a web browser, or an application through which a user can interact with the implementation of the subject matter described in this specification; or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected in any form, including via digital data communication media, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0130] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is generated by computer programs running on their respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data, such as HTML pages, to a user device, for example, with the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated by the user device, such as the results of user interactions, may be received at the server.

[0131] In addition to the above embodiments, the following embodiments are also innovative:

[0132] Example 1 discloses a method comprising:

[0133] Acquire data representing a first image of a specific scene generated by a first sensor and a second image of a specific scene generated by a second sensor, wherein the specific scene includes one or more objects, and the first image and the second image each include multiple pixels;

[0134] For each of the multiple pixels included in the first image, determine whether the pixel is a boundary pixel associated with the boundary of an object represented by the first image and in a particular scene;

[0135] For each pixel included in the first image, an optimized penalty value for that pixel is determined from a plurality of candidate penalty values;

[0136] The optimization penalty value based on multiple pixels is the first image generation optimization cost function; and

[0137] A depth image is generated for the first image based on an optimized cost function.

[0138] Example 2 is the method according to Example 1, wherein the method further includes: providing a depth image to a rendering engine, the rendering engine being configured to blur one or more pixels in a first image to reproduce a depth-of-field effect.

[0139] Implementation method 3 is the method according to embodiment 1 or 2, wherein generating the optimized cost function further includes:

[0140] For each of the plurality of pixels included in the first image, an initial cost function is generated, which is used to process the pixels based on the disparity levels in a predetermined set of disparity levels.

[0141] Example 4 is the method according to Example 3, wherein the initial cost function includes a matching cost function, which is used to measure the performance of the block matching process between pixels in the first image and one or more pixels in the second image, wherein one or more pixels in the second image are determined based on the pixels in the first image and the disparity hierarchy of the initial cost function.

[0142] Example 5 is the method according to any one of Examples 1-4, wherein determining whether a pixel in the first image is a boundary pixel includes:

[0143] Pixel predictions are generated by a machine learning model trained to process the first image to indicate whether a pixel is a boundary pixel.

[0144] The trained machine learning model includes an encoder module and a decoder module, and pixel prediction includes the likelihood of a pixel being associated with the boundary of an object represented in the first image and in a specific scene.

[0145] Example 6 is the method according to any one of Examples 1-5, wherein determining the optimized penalty value of a pixel includes:

[0146] Obtain multiple candidate penalty values, and arrange the multiple candidate penalty values ​​in order according to the size of each candidate penalty value;

[0147] Based on the order, multiple candidate cost functions are generated for each of the multiple candidate penalty values.

[0148] Based on multiple candidate cost functions, generate a set of candidate cost values ​​for each pixel in the first image;

[0149] Determine the saliency value of each pixel in the first image; and

[0150] For each pixel in the first image, based on the saliency value of that pixel, a candidate penalty value is selected from multiple candidate penalty values ​​as the optimized penalty value for the pixel.

[0151] Example 7 is the method according to Example 6, wherein determining the salience value of the pixel includes:

[0152] Based on the respective candidate cost values ​​of a candidate cost quantity from their respective candidate cost quantity sets and the corresponding disparity levels that generated their respective candidate cost values, waveforms of pixels are generated, wherein the waveforms include one or more minimum values ​​of the respective candidate cost values; and

[0153] Significance values ​​are generated based on one or more minimum values ​​in the waveform.

[0154] Example 8 is a method according to any one of Examples 1-7, wherein generating the optimized cost function for a plurality of pixels in the first image includes:

[0155] Based on the optimized penalty value, optimized path cost functions are generated for multiple directions in the first image for multiple pixels; and

[0156] At least the optimization cost function of the first image is generated based on the respective optimization path cost function.

[0157] Example 9 is the method according to any one of Examples 1-8, wherein generating the optimized cost function for the plurality of pixels in the first image further includes:

[0158] In response to determining an optimized penalty value for a given boundary pixel, a common penalty value shared by all pixels other than the boundary pixel in the first image is added.

[0159] Implementation 10 discloses a system including one or more computers and one or more storage devices storing operable instructions, which, when executed by the one or more computers, cause the one or more computers to perform the method described in any one of Embodiments 1-9.

[0160] Example 11 discloses a computer storage medium encoded with a computer program including operable instructions that, when executed by a data processing device, cause the data processing device to perform the method described in any one of Examples 1-9.

[0161] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or claim, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in this specification within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Moreover, while features described above may function in certain combinations, and even be initially claimed in this way, one or more features of a claimed combination may be removed from the combination in certain circumstances, and the claimed combination may involve sub-combinations or variations thereof.

[0162] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order, or to perform all the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, but rather as meaning that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0163] Specific embodiments of the above-described subject matter have been described. Other embodiments are within the scope of the claims. For example, the actions defined in the claims may be performed in a different order, but the desired result may still be achieved. For example, the processes depicted in the drawings do not necessarily need to be performed in the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for generating a depth image, characterized in that, include: Acquire data, the data representing a first image of a specific scene generated by a first sensor and a second image of the specific scene generated by a second sensor, wherein the specific scene includes one or more objects, and the first image and the second image each include multiple pixels; For each of the plurality of pixels included in the first image, determine whether the pixel is a boundary pixel associated with the boundary of an object represented by the first image and in the particular scene; For the boundary pixels included in the first image, an optimized penalty value for the boundary pixels is determined from multiple candidate penalty values, and for the non-boundary pixels included in the first image, a common penalty value is assigned. The optimized penalty value based on the boundary pixels and the non-boundary pixels is the first image generation optimization cost function; as well as A depth image is generated for the first image based on the optimized cost function.

2. The method according to claim 1, characterized in that, Also includes: The depth image is provided to a rendering engine, which is configured to blur one or more pixels in the first image to reproduce a depth-of-field effect.

3. The method according to claim 1, characterized in that, The generated optimization cost function also includes: For each of the plurality of pixels included in the first image, an initial cost function is generated, the initial cost function being used to process the pixel based on a disparity level in a predetermined set of disparity levels.

4. The method according to claim 3, characterized in that, The initial cost function includes a matching cost function, which measures the performance of a block matching process between pixels in the first image and one or more pixels in the second image, wherein the one or more pixels in the second image are determined based on pixels in the first image and the disparity hierarchy of the initial cost function.

5. The method according to claim 1, characterized in that, Determining whether the pixel is a boundary pixel associated with the boundary of an object represented by the first image in the specific scene includes: A pixel prediction indicating whether a pixel is a boundary pixel is generated by a machine learning model trained for processing the first image. The trained machine learning model includes an encoder module and a decoder module, and the pixel prediction includes the likelihood of the pixel being associated with the boundary of an object represented by the first image in the specific scene.

6. The method according to claim 1, characterized in that, The determination of the optimized penalty value for the boundary pixel includes: Obtain the plurality of candidate penalty values, and arrange the plurality of candidate penalty values ​​in order according to the size of each candidate penalty value; According to the order, multiple candidate cost functions are generated based on each of the multiple candidate penalty values; Based on the plurality of candidate cost functions, a set of candidate cost quantities is generated for each boundary pixel in the first image; Determine the saliency value of each boundary pixel in the first image; and For each boundary pixel in the first image, a candidate penalty value is selected from the plurality of candidate penalty values ​​as the optimized penalty value for the boundary pixel based on the saliency value of the boundary pixel.

7. The method according to claim 6, characterized in that, Determining the saliency value of each boundary pixel in the first image includes: Based on the respective candidate cost value of a candidate cost quantity from the respective candidate cost quantity sets and the corresponding disparity level that generated the respective candidate cost value, a waveform of the boundary pixel is generated, wherein the waveform includes one or more minimum values ​​of the respective candidate cost values; and The significance value is generated based on one or more minimum values ​​in the waveform.

8. The method according to claim 1, characterized in that, Generating an optimized cost function for the first image includes: Based on the optimized penalty value, optimized path cost functions for multiple directions in the first image are generated for the multiple pixels; and The optimization cost function of the first image is generated based at least on the respective optimization path cost functions.

9. A system, characterized in that, It includes one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform operations, including: Acquire data, the data representing a first image of a specific scene generated by a first sensor and a second image of the specific scene generated by a second sensor, wherein the specific scene includes one or more objects, and the first image and the second image each include multiple pixels; For each of the plurality of pixels included in the first image, determine whether the pixel is a boundary pixel associated with the boundary of an object represented by the first image and in the particular scene; For the boundary pixels included in the first image, an optimized penalty value for the boundary pixels is determined from multiple candidate penalty values, and for the non-boundary pixels included in the first image, a common penalty value is assigned. The optimized penalty value based on the boundary pixels and the non-boundary pixels is the first image generation optimization cost function; and A depth image is generated for the first image based on the optimized cost function.

10. The system according to claim 9, characterized in that, The operation also includes: The depth image is provided to a rendering engine, which is configured to blur one or more pixels in the first image to reproduce a depth-of-field effect.

11. The system according to claim 9, characterized in that, Determining whether the pixel is a boundary pixel associated with the boundary of an object represented by the first image in the specific scene includes: A pixel prediction indicating whether a pixel is a boundary pixel is generated by a machine learning model trained for processing the first image. The trained machine learning model includes an encoder module and a decoder module, and the pixel prediction includes the likelihood of the pixel being associated with the boundary of an object represented by the first image in the specific scene.

12. The system according to claim 9, characterized in that, The determination of the optimized penalty value for the boundary pixel includes: Obtain the plurality of candidate penalty values, and arrange the plurality of candidate penalty values ​​in order according to the size of each candidate penalty value; According to the order, multiple candidate cost functions are generated based on each of the multiple candidate penalty values; Based on the plurality of candidate cost functions, a set of candidate cost quantities for each boundary pixel in the first image is generated. Determine the saliency value of each boundary pixel in the first image; and For each boundary pixel in the first image, a candidate penalty value is selected from the plurality of candidate penalty values ​​as the optimized penalty value for the boundary pixel based on the saliency value of the boundary pixel.

13. The system according to claim 12, characterized in that, Determining the saliency value of each boundary pixel in the first image includes: Based on the respective candidate cost value of one of the candidate cost values ​​in the respective candidate cost value sets and the corresponding disparity level that generated the respective candidate cost value, a waveform of the boundary pixel is generated, wherein the waveform includes one or more minimum values ​​of the respective candidate cost values; and The significance value is generated based on one or more minimum values ​​in the waveform.

14. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform their respective operations, characterized in that... The respective operations include: Acquire data, the data representing a first image of a specific scene generated by a first sensor and a second image of the specific scene generated by a second sensor, wherein the specific scene includes one or more objects, and the first image and the second image each include multiple pixels; For each of the plurality of pixels included in the first image, determine whether the pixel is a boundary pixel associated with the boundary of an object represented by the first image and in the particular scene; For the boundary pixels included in the first image, an optimized penalty value for the boundary pixels is determined from multiple candidate penalty values, and for the non-boundary pixels included in the first image, a common penalty value is assigned. The optimized penalty value based on the boundary pixels and the non-boundary pixels is the first image generation optimization cost function; and A depth image is generated for the first image based on the optimized cost function.

15. One or more computer-readable storage media according to claim 14, characterized in that, The operation also includes: The depth image is provided to a rendering engine, which is configured to blur one or more pixels in the first image to reproduce a depth-of-field effect.

16. The one or more computer-readable storage media according to claim 14, characterized in that, Determining whether the pixel is a boundary pixel associated with the boundary of an object represented by the first image in the specific scene includes: A pixel prediction indicating whether a pixel is a boundary pixel is generated by a machine learning model trained for processing the first image. The trained machine learning model includes an encoder module and a decoder module, and the pixel prediction includes the likelihood of the pixel being associated with the boundary of an object represented by the first image in the specific scene.

17. The one or more computer-readable storage media according to claim 14, characterized in that, The determination of the optimized penalty value for the boundary pixel includes: Obtain the plurality of candidate penalty values, and arrange the plurality of candidate penalty values ​​in order according to the size of each candidate penalty value; According to the order, multiple candidate cost functions are generated based on each of the multiple candidate penalty values; Based on the plurality of candidate cost functions, a set of candidate cost quantities for each boundary pixel in the first image is generated. Determine the saliency value of each boundary pixel in the first image; and For each boundary pixel in the first image, a candidate penalty value is selected from the plurality of candidate penalty values ​​as the optimized penalty value for the boundary pixel based on the saliency value of the boundary pixel.

18. One or more computer-readable storage media according to claim 17, characterized in that, Determining the saliency value of each boundary pixel in the first image includes: Based on the respective candidate cost value of a candidate cost quantity from the respective candidate cost quantity sets and the corresponding disparity level that generated the respective candidate cost value, a waveform of the boundary pixel is generated, wherein the waveform includes one or more minimum values ​​of the respective candidate cost values; and The significance value is generated based on one or more minimum values ​​in the waveform.

Images