Estimating color and depth in an image

Abstract

The proposed concepts are therefore directed to providing schemes, solutions, concepts, designs, methods and systems relating to obtaining a prediction of at least one of color and depth of a location in an image of a target view of a scene. This includes generating a plurality of depth values along a ray corresponding to a target position in a target view of the scene based on the obtained depth map of the scene. An input of a neural network may then be generated from the plurality of depth values and provided to a first neural network trained to predict a light reflection characteristic for a position along a ray corresponding to the plurality of depth values. Finally, these predicted light reflection characteristics may be used to predict at least one of color and depth of the target location.

Description

Technical Field

[0001] The present invention relates to the field of estimating at least one of color and depth in an image, and in particular to the field of estimating at least one of color and depth of a location in an image of a scene. Background Art

[0002] Estimating the color and / or depth of locations in an image of a scene is a computationally difficult problem. This is required, for example, when rendering novel views of semi-3D images taken with a smartphone. From as few as two source images of the scene, multiple views can be rendered, each requiring the calculation of the color and / or depth of reflective objects in the scene. Prior art methods are either far from real-time or have low-quality synthesis results. A disadvantage of current real-time methods in the art, compared to slower but higher-quality methods, is that the "corruption" caused by depth map errors occurs before any warping of adjacent source views and therefore cannot be easily fixed by an alignment operation. For example, even after alignment of multiple predictions, correlated depth errors between source view depth maps can cause synthesis errors.

[0003] For example, some current real-time methods attempt to align multiple predicted textures after warping them from adjacent source views via depth image-based rendering. However, other slower but higher quality methods sample many 3D points along each ray in the target view, retrieving features from the corresponding image locations in the adjacent source views. These features are then used to predict the color and density of the 3D point scene location. Volume rendering can then be used to compute the final color and / or depth of the pixel in the target view. Other slower but higher quality methods fit the coefficients of a neural network to predict the density and color along the camera rays in the depth map. After volume rendering (including integration along each ray), the output color can be compared to the ground truth pixel. However, in this approach, the coefficients of the neural network are trained for a given dataset and therefore do not extrapolate to new data.

[0004] What makes the current slower methods more accurate and reliable is that depth estimation is pulled into the neural rendering network as a separate (but implicit) task. The neural network also combines many high-level features from the source view so that view synthesis can use these features to predict at least one of the color and depth of the position. However, these steps are very computationally expensive and therefore make the entire process slower than real-time.

[0005] Therefore, there is a need for a method of estimating at least one of color and depth of a location in an image of a scene that is computationally relatively fast but does not include the drawbacks of current fast prior art methods. Summary of the Invention

[0006] The invention is defined by the claims.

[0007] According to an example of an aspect of the invention, there is provided a method for estimating at least one of color and depth of a location in an image of a scene.

[0008] The method includes obtaining a first depth map and a second depth map of a scene, the first depth map and the second depth map corresponding to a first view and a different second view of the scene, respectively; generating, based on the first depth map and the second depth map, a plurality of depth values ​​along a ray corresponding to a target position in a target view of the scene, the target view being different from the first view and the second view of the scene; generating an input for a first neural network based on the generated plurality of depth values; providing the generated input as input to the first neural network, the first neural network being trained to predict light reflection characteristics for positions along the ray corresponding to the plurality of depth values ​​for the generated input; and predicting at least one of a color and a depth of the target position in an image of the target view of the scene based on the predicted light reflection characteristics.

[0009] Therefore, the concepts presented are directed to providing schemes, solutions, concepts, designs, methods, and systems related to estimating at least one of the color and depth of a location in an image of a scene. Specifically, embodiments are directed to providing a method for deriving a prediction or estimate of at least one of the color and depth of a target location in an image of a target view of a scene based on a first depth map and a second depth map of the scene.

[0010] In other words, it is proposed that the depth of a target position in a target view of a scene can be predicted from only two depth maps of the scene (corresponding to a first view of the scene and a different second view), wherein the target view is different from both the first view and the second view. Alternatively, or alternatively, the color of the target position can also be predicted. The method can be used to render novel views of semi-3D scenes (also called depth images or 2.5D images or range images), and due to the simplicity of the method, these novel views can be rendered quickly and efficiently. For these views, both color and depth are useful for rendering.

[0011] By generating inputs for a first neural network trained to predict light reflection properties for positions along a ray based on a plurality of generated depth values, a computationally simpler method is provided for estimating the depth and / or color of a location in an image of a scene. This is in contrast to alternative methods in which every single possible depth value is input into the first neural network. The depth and / or color of the target location can be predicted based on the predicted light reflection properties. Conventional methods perform a search to analyze all possible depth values ​​for the target location, rather than using a previously estimated depth map from a source view. The method exploits the idea that certain depth values ​​in a measured (sensed) or calculated depth map are more likely to correspond to the true depth of the target location than other depth values. These depth values ​​are based on a previous depth map, and therefore the method only analyzes these possible depth values, resulting in a fast but highly accurate method. The proposed concept can therefore provide an improved method for estimating at least one of the color and depth of a location in an image of a scene.

[0012] For example, by only inputting an input based on a plurality of possible depth values ​​generated (rather than all possible depth values) to the first neural network, the neural network can be smaller and therefore faster. Thus, at least one of the color and depth of a target location in an image of a target view of a scene can be predicted quickly and with relatively little computational effort. In many cases, the prediction of the color or depth of the target location may inform the prediction of the other. For example, where the local background is one color and the local foreground is another color, the color prediction can be used to predict the depth, and vice versa.

[0013] The present invention may be particularly useful in situations where a new view of a depth image is rendered in real time, such as when a user orients the view of the depth image on their smartphone, tablet, virtual reality device, or computer. When rendering a new view, color information is required to correctly render the colors of the new pixels, and depth information can be used to correctly render overlaid 3D graphics.

[0014] Ultimately, the proposed concept(s) may enable improved methods for estimating at least one of color and depth of locations in an image of a scene.

[0015] In some embodiments, the generated multiple depth values ​​may include local foreground depth values ​​and local background depth values. At an object location with uncertain depth, it has been recognized that the local foreground depth and local background depth surrounding the object location can often be used to accurately predict the depth of the object location. This feature can essentially transform the computationally complex depth estimation problem into a simpler foreground / background classification problem, thereby accelerating the estimation of the depth and / or color of the object location.

[0016] In some embodiments, the local foreground depth value and the local background depth value may correspond to the local minimum depth value and the local maximum depth value, respectively, within a spatial window centered at the target location. It was found that the depth map distribution around problematic object transitions (i.e., target locations with uncertain depth) is not completely random. Instead, the depth map distribution is concentrated in a bimodal form, belonging either to the local foreground or the local background. Thus, this feature exploits the bimodality of the possible depths of a target location, thereby allowing for faster prediction of the depth and / or color of the target location.

[0017] In some embodiments, the size of the spatial window can be based on the expected uncertainty value of the depth of the target location in the target view of the scene. This allows for flexibility in the method. Thus, the required data can be adjusted based on the magnitude of the depth and / or color uncertainty of the target location to obtain more reliable local foreground or local background depth values.

[0018] In some embodiments, the light reflection characteristics may include at least one of weight and opacity as well as color. These reflection characteristics may then be integrated and combined along the ray to help predict the color of the target location in the target view of the scene.

[0019] In some embodiments, generating the plurality of depth values ​​includes generating a third depth map corresponding to a target view of the scene based on the first depth map and the second depth map; and generating the plurality of depth values ​​for a target position in the target view of the scene based on the third map. This simplifies the method, allowing the processing of the first depth map and the second depth map to be computationally simpler.

[0020] In some embodiments, generating the third depth map may include forward point rendering. This allows for fast rendering, as most graphics APIs support point rendering, typically taking only 1 to 5 milliseconds.

[0021] In some embodiments, the third depth map may have a lower resolution than the first and second depth maps. This allows for reducing the computational complexity of generating the third depth map.

[0022] In some embodiments, predicting at least one of a color and a depth of a target location in an image of a target view of a scene may include providing the light reflection characteristic as an input to a second neural network, the second neural network being trained to predict at least one of a color and a depth value based on the input. Using a neural network to estimate the depth and / or color of the target location based on the predicted light reflection characteristic has been found to be beneficial, resulting in fast and reliable estimates.

[0023] In some embodiments, the second neural network may include a multilayer perceptron. The use of multilayer perceptrons has been found to be particularly useful in this context because they are relatively simple and fast to compute.

[0024] In some embodiments, generating multiple depth values ​​can be based on local plane fitting of multiple depth surfaces in the scene. This allows the method to consider scenes containing more complex surfaces, such as tilted surfaces or points where two or more surfaces intersect.

[0025] In some embodiments, the generated plurality of depth values ​​may include a local mid-view depth value, which allows the method to be more flexible and accurate in estimating the depth and / or color of the target location.

[0026] In some embodiments, at least one of a color and a depth of a target location in an image of a target view of a scene may be estimated in real time.

[0027] In some embodiments, a computer program is provided comprising code means for implementing any of the above methods when the program is run on a processing system.

[0028] According to another aspect of the present invention, a system for estimating at least one of the color and depth of a location in an image of a scene is provided. The system includes: an input interface configured to obtain a first depth map and a second depth map of the scene, the first depth map and the second depth map corresponding to a first view and a different second view of the scene, respectively; and a depth estimation unit configured to generate a plurality of depth values ​​along a ray corresponding to a target location in a target view of the scene based on the first depth map and the second depth map, the target view being different from the first view and the second view of the scene; generate input for a first neural network based on the generated plurality of depth values; provide the generated input as input to the first neural network, the first neural network being trained to predict light reflection characteristics for the location along the ray corresponding to the plurality of depth values ​​for the generated input; and predict at least one of the color and the depth of the target location in the target view image of the scene based on the predicted light reflection characteristics.

[0029] Therefore, a concept for estimating at least one of color and depth of a location in an image of a scene may be proposed, and this may be done based on generating and analyzing a plurality of depth values ​​obtained based on a first depth map and a second depth map of the scene.

[0030] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] For a better understanding of the invention, and to show more clearly how it may be put into practice, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0032] Figure 1is a simplified flow chart of a method for estimating at least one of color and depth of a location in an image of a scene according to the proposed embodiment;

[0033] Figure 2 is a more in-depth flow chart of a method for estimating at least one of color and depth of a location in an image of a scene according to the proposed embodiment;

[0034] Figure 3A is a simple illustration of how the depth of a target location is estimated according to an example method in the art;

[0035] Figure 3B is a simple illustration of how to estimate the depth of a target location according to the proposed embodiment;

[0036] Figure 4 is a simplified illustration of how local planar fitting of multiple depth surfaces may be employed according to the proposed embodiment;

[0037] Figure 5 is a simplified block diagram of a system for estimating at least one of color and depth of a location in an image of a scene according to the proposed embodiment; and

[0038] Figure 6 Illustrated is an example of a computer in which one or more portions of the embodiments may be employed. DETAILED DESCRIPTION

[0039] The present invention will be described with reference to the accompanying drawings.

[0040] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, system, and method, are intended for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, system, and method of the present invention will be better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are schematic only and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to represent the same or similar parts.

[0041] Those skilled in the art will be able to understand and implement variations to the disclosed embodiments in practicing the claimed invention by studying the drawings, the disclosure, and the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality.

[0042] It should be understood that the drawings are schematic and not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0043] Implementations according to the present disclosure involve various techniques, methods, schemes, and / or solutions related to estimating at least one of the color and depth of a location in an image of a scene. Based on the concepts presented, many possible solutions can be implemented individually or in combination. That is, while these possible solutions may be described individually below, two or more of these possible solutions may be implemented in one or another combination.

[0044] Embodiments of the present invention are directed to a method for estimating at least one of color and depth of a location in an image of a scene, which may be achieved by analyzing a first depth map and a second depth map of the scene, and feeding input generated based on the depth maps to a first neural network.

[0045] Thus, the proposed concept aims to predict at least one of the color and the depth of a location in an image of a scene. This involves generating a plurality of depth values ​​along rays corresponding to a target location in a target view of the scene based on an obtained depth map of the scene. Inputs to a neural network can then be generated based on the plurality of depth values ​​and provided to a first neural network that is trained to predict light reflection properties for locations along rays corresponding to the plurality of depth values. Finally, these predicted light reflection properties can be used to predict at least one of the color and the depth of the target location. In this way, a method is provided that is computationally simple, fast, yet still accurate compared to other methods in the art. In other words, by only feeding input based on the generated plurality of possible depth values ​​(rather than all possible depth values) to the first neural network, the neural network can be smaller and therefore faster than that required by current high-quality methods.

[0046] Thus, instead of using a trained network to directly predict a small set of ray sample positions for each line of sight, at least one of the color and depth of a position in the image of the scene can be predicted based on data from a depth sensor. Using the resulting scene depth map is an integral part of the present invention, and as such, it is possible to estimate at least one of the position and depth in the image of the scene without using a classification network based on logarithmic discretization and spherical warping of depth values ​​to encode surface positions.

[0047] Now refer to Figure 1 , depicting a simplified flow chart of a method 100 for estimating at least one of color and depth of a location in an image of a scene according to the proposed embodiment.

[0048] Method 100 begins at step 110 by obtaining a first depth map and a second depth map of a scene, the first depth map and the second depth map corresponding to a first view and a different second view of the scene, respectively. For example, these may be provided by a camera capable of sensing depth. The first depth map and the second depth map of the scene may include data from a depth sensor. Alternatively, the first depth map and the second depth map of the scene may include only data from a depth sensor.

[0049] In step 120, a plurality of depth values ​​along rays corresponding to target positions in a target view of the scene are generated based on the first depth map and the second depth map. The target view is different from the first view and the second view of the scene.

[0050] The multiple depth values ​​generated include local foreground depth values ​​and local background depth values. At an object location with uncertain depth, it has been recognized that the local foreground depth and local background depth surrounding the object location can often be used to accurately predict the depth of the object location. This feature can essentially transform the computationally complex depth estimation problem into a simpler foreground / background classification problem, thereby accelerating the estimation of the depth of the object location and potentially also accelerating the prediction of the color of the object location.

[0051] The local foreground depth value and the local background depth value may correspond to a local minimum depth value and a local maximum depth value, respectively, within a spatial window centered at the target position. It was found that the depth map distribution around problematic object transitions (i.e., target positions with uncertain depth) is not completely random. Instead, the depth map distribution is concentrated in a bimodal form, belonging either to the local foreground or the local background. Therefore, this feature exploits the bimodality of the possible depths of the target position, thereby allowing for faster prediction of the depth of the target position.

[0052] The size of the spatial window is based on the expected uncertainty value of the depth and / or color of the target location in the target view of the scene. This allows flexibility in method 100. Therefore, the required data can be adjusted according to the magnitude of the depth and / or color uncertainty of the target location to obtain more reliable local foreground or local background depth values.

[0053] Generating the plurality of depth values ​​may be based on local plane fitting of multiple depth surfaces in the scene. This allows the method 100 to consider scenes containing more complex surfaces, such as tilted surfaces or points where two or more surfaces intersect. The generated plurality of depth values ​​may also include local mid-ground depth values, thereby allowing the method to be more flexible and accurate in estimating the depth of the target location. More details about this step are available on Figure 4 supply.

[0054] In step 130, inputs are generated for the first neural network based on the generated plurality of depth values. For example, in one embodiment, the plurality of depth values ​​may be used to compute positions of first and second reference views corresponding to first and second depth maps of the scene, respectively. Features may then be extracted from the first and second depth maps and the computed positions of the first and second reference views. These extracted features may then be used as the generated inputs. The extracted features need not necessarily be the original color and depth provided in the first and second depth maps. The extracted features may be the output of several convolution steps performed on the first and second depth maps, and optionally, the output of several convolution steps performed on first and second reference view images corresponding to the first and second depth maps, respectively. The first and second reference view images may be obtained first. Any operations performed on the reference view image data may be part of the first neural network. This is often referred to as differentiable rendering. This example may be particularly useful when integrating the disclosed method 100 into existing state-of-the-art methods, such as IBRNet.

[0055] In another embodiment, the input may be samples of the first depth map and the second depth map, wherein the positions of the samples are based on multiple depth values. Alternatively, the input may be samples of the first and second reference view images obtained corresponding to the first depth map and the second depth map, respectively.

[0056] In another embodiment, the input may be an input vector formed by combining multiple depth values ​​with image coordinates corresponding to the first depth map and the second depth map. The input vector may contain 3D positions and ray angles. Each 3D position is typically an x, y, z position corresponding to one of the multiple depth values ​​at the image pixel u, v in the target view. The ray angle is typically the 3D angle of a ray corresponding to the image pixel u, v in the target view and passing through the x, y, z position. This vector can typically be represented by a 6D vector. The input vector can then be provided as input to the first neural network. This example may be particularly useful when integrating the disclosed method 100 into an existing prior art method (such as Nerf).

[0057] In step 140, the generated input is provided as an input to a first neural network. The first neural network is trained to predict light reflection characteristics for positions along rays corresponding to multiple depth values ​​based on the generated input. By predicting light reflection characteristics only for positions along rays corresponding to multiple depth values, light reflection characteristics that may be used to predict at least one of the color and depth of the target position can be provided more quickly than predicting light reflection characteristics for all positions along the ray. In other words, light reflection characteristics are only predicted for positions that may correspond to the true depth of the target position, rather than for all positions along the ray, which would take longer and have little effect. In other words, the light reflection characteristics of many positions along the ray are unnecessary for predicting the color and / or depth of the position in the target view image and can therefore be ignored, thereby reducing computational cost.

[0058] The light reflection characteristics include at least one of weight and opacity and color. These light reflection characteristics can then be integrated and combined along the ray to help predict the color of the target position in the target view of the scene. In the semi-volume representation, for a given output pixel in the target image, an output ray in 3D space is first generated. On this ray, the color and opacity or weight are predicted for a given set of points. Near the depth transition, there may be at least two points on the ray with high opacity. The light reflection characteristics can also include more complex volumetric characteristics, such as smoke, fire, water, haze, snow, glass windows, etc. These light reflection characteristics can then be used to predict at least one of the color and depth of the target position in the target view.

[0059] There are several types of neural networks, such as, for example, convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In an embodiment of the present invention, the first neural network may include a learning algorithm based on a CNN.

[0060] CNNs typically contain several layers, including convolutional layers, pooling layers, and fully connected layers. Convolutional layers include a set of learnable filters and extract features from the input. Pooling layers are a form of nonlinear downsampling that reduces data size by combining the outputs of multiple neurons in one layer into a single neuron in the next layer. Fully connected layers connect every neuron in one layer to all neurons in the next layer.

[0061] Methods for training machine learning algorithms are well known. Typically, such methods involve obtaining a training dataset comprising training input data entries and corresponding training output data entries. An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and the corresponding training output data entry is used to modify the machine learning algorithm. This process can be repeated until the error converges and the predicted output data entry is sufficiently similar to the training output data entry (e.g., ±1%). This is generally referred to as a supervised learning technique.

[0062] For example, the weights of the mathematical operations of each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, backpropagation algorithms, etc.

[0063] In step 150, at least one of a color and a depth of a target location in an image of a target view of the scene can be predicted based on the light reflection characteristics. When rendering the target view, color information may be necessary to correctly render the color of the target location, and depth information may be used, for example, to correctly render overlaid 3D graphics. The predicted depth of the target location may also or alternatively be provided directly as an output of method 100. The predicted light reflection characteristics for the locations along the ray may be integrated to predict at least one of the color or depth. Note that this process is differentiable.

[0064] Due to all of these computationally efficient steps, method 100 can predict at least one of the color and depth of a target location in an image of a target view of a scene in real time. This can be particularly useful when rendering novel view images for viewing on a virtual reality headset or smartphone.

[0065] Some slow but high-quality methods in the field for predicting the depth of an object position essentially pull depth estimation entirely into the neural network as an implicit task. However, this aspect, while accurate and reliable, is partly due to the high computational cost. At the other extreme, other relatively fast methods in the field assume that ignoring the previous depth map at most causes simple misalignment errors, which is clearly not the case.

[0066] The truth lies somewhere in between. The estimated or measured prior depth map is neither completely wrong nor completely correct. More precisely, at many target image locations, the prior depth map can be a good predictor, but may need fine-tuning. In particular, at depth transitions, the exact spatial image location of the depth step is often difficult to predict, but local foreground depth or local background depth is likely to be a good predictor. So, in other words, the depth map distribution around the problematic object transition is not uniformly random, but rather bimodally concentrated and can be exploited to speed up the prediction of the depth at the target location. The depth at the target location may also provide information for the prediction of the color at the target location.

[0067] The above implementation provides a method to convert the uncertainty of the depth map into a sampling strategy. Therefore, the present invention proposes to generate a finite set of possible 3D candidate points, i.e. depth values, on the target ray based on the obtained depth map. A plurality of candidates are generated for which we predict with high probability that at least one of them corresponds to the true depth of the target position in the target view. In essence, the proposed embodiment substantially reduces the exhaustive search along the target ray that is usually performed in the prior art methods to only a few evaluations based on a finite set of depth candidates, i.e. depth values. As a result, the size of the neural network may be reduced. This idea may be explained more clearly later with respect to FIG. 3 .

[0068] Now refer to Figure 2 , depicts a more in-depth flow chart of a method 200 for estimating the depth of a location in an image of a scene.

[0069] Steps 110, 130 and 140 are related to Figure 1 The method 100 is basically the same as described in .

[0070] In step 210, a third map corresponding to a target view of the scene is generated based on the first depth map and the second depth map. In step 220, a plurality of depth values ​​for target locations in the target view of the scene are generated based on the third map. This can simplify method 200, thereby allowing computationally simpler processing of the obtained depth map. Generating the third depth map can include forward point rendering. This allows for fast rendering, as most graphics APIs support point rendering, typically requiring only 1 to 5 milliseconds. The resolution of the third depth map can also be lower than the resolution of the first and second depth maps. This allows for reduced computational complexity in generating the third depth map.

[0071] In other words, for a practical implementation of method 200, fast rendering is required. Forward point rendering can then be used to bring the first and second depth maps into the target view image coordinates, i.e., the third depth map. Most graphics APIs support point rendering, and this is one of the fastest methods, resulting in a typical time of 1 to 5 milliseconds. Since the multi-candidate method will efficiently handle depth transitions, the complexity of this step can be further reduced by rendering at a lower depth map resolution. Point rendering will automatically perform depth sorting between the warped source views (i.e., the first and second depth maps). Therefore, while we can keep separate warping results from multiple depth maps, this is not strictly required because the occlusion holes will be automatically filled by the other source views.

[0072] In step 230, the light reflection characteristics are provided as input to a second neural network, which is trained to predict at least one of a color and a depth value based on the output. Using a neural network to estimate the color and / or depth of a target location based on the light reflection characteristics has been found to be beneficial, enabling fast and reliable estimation. The second neural network may comprise a multilayer perceptron. Using multilayer perceptrons has been found to be particularly useful in this context because they are computationally relatively simple and fast. Perceptrons, and particularly multilayer perceptrons, typically output non-binary variables and can also be used to directly predict real numbers, such as color values ​​or depth levels.

[0073] Now refer to Figure 3A , provides a simple explanation of how to estimate the depth of the target position according to the method in the art. Such a method in the art can be, for example, a method similar to that used in IBRNet. The figure shows a method in which sampling is performed along a target ray 350 originating from the target view 330 to collect the source views 310 and 320 along the epipolar line l A and l B In other words, this method requires exhaustive sampling along the target ray and evaluating the n 345. No prior depth information, which may be provided in the form of a source view depth map, is used. Therefore, many feature windows need to be extracted along the epipolar lines in the source views 310 and 320. This makes the method computationally complex and relatively slow.

[0074] Now refer to Figure 3B , provides a simple illustration of how the depth and / or color of a target location can be estimated according to the proposed embodiments. Figure 3ACompared to the method proposed in

[15] , this method uses source view depth maps as prior information, with each source view depth map corresponding to one of source views 310 and 320. To handle the uncertainty of the target position depth, particularly at depth transitions, multiple candidates are generated within a small neighborhood 315 and 325 of each source view pixel. Therefore, a single point in each source view depth map results in two points on target ray 350: one point corresponding to the hypothesis that the target position in target view 330 is at a depth corresponding to foreground 360, and one point corresponding to the hypothesis that the target position is at a depth corresponding to background 365. In other words, this method demonstrates that using depth estimated or measured from the source view depth map can be used to derive a small set of possible depth value candidates on target ray 350. In other words, the multiple depth values ​​in this embodiment only include local foreground depth values ​​and local background depth values. In the uncertain region of the source view, i.e., in the small neighborhood generated around each source view pixel, only two candidate depth values ​​are required: one from the local foreground and one from the local background.

[0075] Once a depth map for the target view 330 is obtained, such as in step 210 of method 200, possible depth candidates can be extracted along the target ray i 350 by extracting the local foreground depth and the local background depth. Thus, a simple algorithm can extract the depth candidates along the target ray i 350 in a spatial neighborhood window W centered at the ray i. i Take the local minimum depth value and the local maximum depth value:

[0076]

[0077] According to the expected uncertainty of the depth transition, a window W around the ray 350 (image point i) in the target view can be i Since the neural network is considered to be responsible for making local decisions between foreground and background hypotheses, the candidate image can be calculated at a lower resolution than the target image in a separate GPU shader process. Typically, a separate GPU shader process might sample the depth map to calculate z fore and z back , both are still at low resolution and can use, for example, 9 points of a 3x3 spatial kernel. It is important to note that today's consumer electronic devices (such as laptops, smartphones, and XR head-mounted kits) all contain GPU hardware. These devices typically support graphics APIs such as OpenGL ES. OpenGL ES supports running fragment shader programs, which can process image pixels in parallel, thereby achieving fast filtering operations. Therefore, fragment shaders can be used to programmatically generate foreground and background depth hypotheses.

[0078] In order for a neural network to predict anything, informative features are needed. If there were resources available for a relatively large neural network containing many coefficients, then all of these informative features could be learned. However, for real-time rendering, a smaller network containing fewer coefficients is required, and therefore a greater reliance on "hand-crafted" informative features is needed to ensure accurate and reliable predictions. For example, a simple but informative feature is the depth step:

[0079] Δ≡z back -z fore (2)

[0080] This depth step feature, by definition, has positive values. The larger its value, the greater the likelihood of a nearby depth transition, and therefore the more likely the neural network will have to make a difficult choice between, for example, a depth value corresponding to the local foreground and a depth value corresponding to the local background. On the other hand, if its value is small, the direct average contribution of source views 310 and 320 will predict the depth and / or color of the target location. The target location is likely the target pixel.

[0081] Other useful informative features for the depth and / or color of the final predicted target location may be, for example, the mean and variance of Δ in the spatial neighborhood of the target view 330. Assuming there are M ≥ 1 depth features, these features can be represented by a vector such as:

[0082]

[0083] Feature maps corresponding to the source views 310 and 320 may preferably be pre-computed and obtained as a video atlas. These source view features may include color, texture, (encoded) depth, or simple "abstract" feature maps, such as those provided by the output of early or intermediate layers of any trained neural network. For example, a useful feature may come from the early layers of a neural network trained for the task of semantic object segmentation.

[0084] The advantage of pre-computing features before encoding is that the renderer only needs to sample / interpolate feature values ​​from the atlas instead of calculating them itself. Alternatively, if the feature map takes up too much atlas space, the features can be partially calculated during or after encoding.

[0085] Given source views A, 310 and B, 320, and a target ray depth candidate z fore 360 and z back 365, the source view image pixel coordinates can be marked as follows:

[0086]

[0087] The source view pixel is calculated by back-projecting the target point from the target view image coordinates to the target view camera coordinates, strictly transforming from the target point to the source view camera coordinates, and computing the source coordinates from which the feature is retrieved. This is expressed as a linear operation in homogeneous coordinates:

[0088]

[0089] In Equation 4, Q is the unprojected matrix for the target view 330; R and t represent the rigid rotation and translation from the target to the source views 310 and 320, and P is the projection matrix of the source view camera. Note that sampling / interpolating the feature maps from the source views is typically done on the GPU using a fragment shader or compute shader. Note that feature maps can be packed into a larger atlas as tiles, and different features from the same source view image coordinate are typically sampled from different atlas locations. A single feature vector in the target view 330 can now be constructed by concatenating the features from the source views 310 and 320 and the foreground / background hypotheses:

[0090]

[0091] In formula 5, the function Represents sampling (multiple) features from x∈{foreground, background} and the source view y∈{A,B}. For example, the sampled features could be the color of a given pixel, a texture measurement derived from the color, or the output of an intermediate layer of a convolutional network with the source view image and / or depth map as input.

[0092] Since rendering is ideally done in real time, it is important to first collect informative features for each target ray 350. Both the target depth and source view features described above may inform the selection and rendering logic that the rest of the neural network must model. Therefore, for each input pixel, we concatenate these two vectors:

[0093]

[0094] The complexity reduction proposed by the present invention comes from the restriction of the depth candidates (i.e., depth values) to be analyzed. In essence, the embodiment involves restricting the full depth search along the target ray 350 to only analyzing a few candidates that may beneficially correspond to local foreground and background estimates. Therefore, the multi-layer perceptron (MLP), i.e., the second neural network, which is capable of predicting at least one of the color and depth of the target position, requires fewer coefficients than are typically used in the prior art because it needs to solve a simpler task. Depth estimation is essentially transformed into a foreground / background classification problem. Therefore, a low-complexity network architecture can be used to predict the output color of the target position. As a first optional step, a 3x3 spatial convolution can combine adjacent feature values ​​in the image coordinates of the target view 330. This 3x3 convolution combines spatial information and therefore can improve spatial consistency, thereby improving the quality of the rendering result.

[0095] A small MLP (e.g., 3 layers deep with 8 nodes per layer) can then be used to predict four blending weights corresponding to the foreground or background color / depth of source view A 310 or source view B 320. The sum of these predicted blending weights can predict the color / depth of the target location. In the simple case, with two constant colors around a clear binary depth transition, the weights will be very close to one or zero, where slight deviations in these weights may be caused by noise in the input.

[0096] As an alternative to this direct blending approach, improved synthesis results may be obtained by predicting opacity and color at positions along ray 350 and then using volume rendering.

[0097] Now refer to Figure 4 , provides a simplified illustration of how local plane fitting of multiple depth surfaces can be employed according to the proposed embodiment. A class boundary 410, a noisy depth signal 420, and two local plane fits 430 and 435 can be seen. Each plane fit corresponds to the depth gradient of one of the two sloped surface intersections.

[0098] In some cases, generating multiple depth values ​​that only include local foreground depth values ​​and local background depth values ​​may lead to unreliable results. For example, when the scene contains tilted surfaces (such as Figure 4 ) or at points where two or more surfaces intersect (called T-junctions). This simplified binarization of the foreground or background can lead to inaccurate depth estimates. In addition, depth measurement noise can significantly affect minimum and maximum operations.

[0099] In order to avoid the above problems, surface fitting can be used to smooth the depth transition, and furthermore, the generated multiple depth values ​​can include more than two values. Figure 4In , six candidate depth values ​​(solid points) are derived based on local plane fitting 430 and 435. The depth surface fitting acts as a filter and suppresses noise. Figure 4 In

[15] , instead of having only one depth value per background or foreground hypothesis, three candidate depth values ​​are provided around each expected local foreground and local background depth value, respectively. This may increase the computational cost of the proposed method, but may also improve the accuracy of depth estimation.

[0100] In some embodiments, a computer program is provided comprising code means for implementing any of the above methods when the program is run on a processing system.

[0101] Now refer to Figure 5 , depicts a simplified block diagram of a system 500 for estimating at least one of color and depth of a location in an image of a scene according to the proposed embodiment.

[0102] The system 500 configured to estimate at least one of color and depth of a location in an image of a scene comprises an input interface 510 and an estimation unit 520. The estimation unit may also be referred to as a processor device.

[0103] System 500 is configured to estimate at least one of the color and depth of a location in an image of a scene by processing first and second depth maps 515 as input. These depth maps correspond to a first view and a different second view of the scene, respectively. A first depth map and a second depth map of the scene are obtained. The system outputs an output result 530 comprising a prediction of the depth and / or color of a target location in an image of a target view of the scene. Output result 530 is predicted based on processing of the first and second depth maps as input to ultimately generate a prediction of at least one of the color and depth of the target location in the image of the target view.

[0104] In more detail, the input interface 510 receives an input first depth map and a second depth map 515. The input interface 510 provides the input first depth map and the second depth map to an estimation unit 520, which is configured to process the first depth map and the second depth map to generate a plurality of depth values ​​along a ray corresponding to a target position in a target view of a scene. The estimation unit 520 includes a first neural network for which inputs are generated based on the generated plurality of depth values. These generated inputs can then be provided as inputs to the first neural network, which is trained to predict light reflection characteristics corresponding to positions along the ray corresponding to the plurality of depth values ​​for the generated inputs. Based on these predicted light reflection characteristics, the estimation unit 520 can then provide as an output 530 a prediction of at least one of a color and a depth of the target position in the image of the target view of the scene based on the predicted light reflection characteristics.

[0105] Figure 6 An example of a computer 600 is shown in which one or more portions of the embodiments may be employed. The various operations discussed above may utilize the functionality of the computer 600. In this regard, it should be understood that the system functional blocks may be executed on a single computer or may be distributed across multiple computers and locations (e.g., connected via the Internet).

[0106] Computer 600 includes, but is not limited to, a PC, a workstation, a laptop computer, a PDA, a handheld device, a server, a memory, and the like. Generally, in terms of hardware architecture, computer 600 may include one or more processors 610, a memory 620, and one or more I / O devices 630 communicatively coupled via a local interface (not shown). The local interface may be, for example, but not limited to, one or more buses or other wired or wireless connections, as known in the art. The local interface may have additional elements, such as controllers, buffers (cache), drivers, repeaters, and receivers, to enable communication. In addition, the local interface may include addressing, control, and / or data connections to enable appropriate communication between the above components.

[0107] Processor 610 is a hardware device for running software that may be stored in memory 620. Processor 610 may be virtually any custom or commercial processor, central processing unit (CPU), digital signal processor (DSP), or auxiliary processor among several processors associated with computer 600, and may be a semiconductor-based microprocessor (in the form of a microchip) or a microprocessor.

[0108] The memory 620 may include any one or a combination of volatile memory elements (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic tape, compact disk read-only memory (CD-ROM), magnetic disk, floppy disk, card, cassette, etc.). In addition, the memory 620 may include electronic, magnetic, optical, and / or other types of storage media. Note that the memory 620 may have a distributed architecture in which various components are remote from each other but can be accessed by the processor 610.

[0109] The software in the memory 620 may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. According to an exemplary embodiment, the software in the memory 620 includes a suitable operating system (O / S) 650, a compiler 660, source code 670, and one or more application programs 680. As shown, the application programs 680 include multiple functional components for implementing the features and operations of the exemplary embodiment. The application programs 680 of the computer 600 may represent various applications, computing units, logic, functional units, processes, operations, virtual entities, and / or modules according to exemplary embodiments, but the application programs 680 are not meant to be limiting.

[0110] The operating system 650 controls the execution of other computer programs and provides scheduling, input-output control, file and data management, memory management, communication control and related services. The inventors contemplate that the application program 680 used to implement the exemplary embodiments can be applied to all commercially available operating systems.

[0111] Application 680 can be a source program, an executable program (object code), a script, or any other entity comprising a set of instructions to be executed. In the case of a source program, the program is typically translated by a compiler (e.g., compiler 660), an assembler, an interpreter, etc., which may or may not be included in memory 620, in order to operate correctly with O / S 650. In addition, application 680 can be written in an object-oriented programming language having classes of data and methods, or a procedural programming language having routines, subroutines, and / or functions, such as, but not limited to, C, C++, C#, Pascal, Python, BASIC, API calls, HTML, XHTML, XML, ASP scripts, JavaScript, FORTRAN, COBOL, Perl, Java, ADA, .NET, etc.

[0112] I / O devices 630 may include input devices such as, but not limited to, a mouse, keyboard, scanner, microphone, camera, etc. In addition, I / O devices 630 may also include output devices such as, but not limited to, a printer, a display, etc. Finally, I / O devices 630 may also include devices that transmit both input and output such as, but not limited to, a NIC or modulator / demodulator (for accessing remote devices, other files, devices, systems, or networks), a radio frequency (RF) or other transceiver, a telephone interface, a bridge, a router, etc. I / O devices 630 also include components for communicating over various networks such as the Internet or an intranet.

[0113] If computer 600 is a PC, workstation, intelligent device, or the like, the software in memory 620 may also include a basic input / output system (BIOS) (omitted for simplicity). The BIOS is a set of basic software routines that initialize and test hardware at startup, start the operating system (OS) 650, and support data transfer between hardware devices. The BIOS is stored in some type of read-only memory, such as ROM, PROM, EPROM, EEPROM, etc., so that the BIOS is executed when computer 800 starts.

[0114] When the computer 600 is in operation, the processor 610 is configured to execute software stored within the memory 620, to transfer data to and from the memory 620, and generally to control the operation of the computer 600 according to the software. The application programs 680 and the O / S 650 are read in whole or in part by the processor 610, possibly buffered within the processor 610, and then executed.

[0115] When application 680 is implemented in software, it should be noted that application 680 can be stored on substantially any computer-readable medium for use by or in conjunction with any computer-related system or method. In the context of this document, a computer-readable medium can be an electronic, magnetic, optical, or other physical device or means that can contain or store a computer program for use by or in conjunction with a computer-related system or method.

[0116] Application 680 can be implemented in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system containing a processor, or other system capable of retrieving instructions from an instruction execution system, apparatus, or device and executing the instructions. In the context of this document, a "computer-readable medium" can be any means that can store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.

[0117] Figure 1-Figure 2 Methods and Figure 5The system can be implemented in hardware or software or a mixture of the two (for example, as firmware running on a hardware device). To the extent that the embodiments are partially or entirely implemented in software, the functional steps shown in the process flow diagram can be performed by a suitably programmed physical computing device (such as one or more central processing units (CPUs) or graphics processing units (GPUs)). Each process (and the individual component steps shown in the flow diagram) can be performed by the same or different computing devices. According to an embodiment, a computer readable storage medium stores a computer program including computer program code, and the computer program code is configured to cause one or more physical computing devices to perform the encoding or decoding method as described above when the program is run on one or more physical computing devices.

[0118] Storage media may include volatile and non-volatile computer memory, such as RAM, PROM, EPROM and EEPROM, optical disks (such as CD, DVD, BD), and magnetic storage media (such as hard disks and magnetic tape). Various storage media may be fixed within the computing device or may be transportable so that one or more programs stored thereon can be loaded into the processor.

[0119] For embodiments implemented partially or entirely in hardware, Figure 4 The blocks shown in the block diagrams may be separate physical components, or logical subdivisions of a single physical component, or may all be implemented in an integrated manner in one physical component. The functionality of a block shown in the diagram may be divided into multiple components in the implementation, or the functionality of multiple blocks shown in the diagram may be combined into a single component in the implementation. Hardware components suitable for use in embodiments of the present invention include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). One or more blocks may be implemented as a combination of dedicated hardware for performing some functions and one or more programmed microprocessors and associated circuits for performing other functions.

[0120] In general, the present invention is defined by the following examples:

[0121] Example

[0122] Embodiment 1, a method for estimating at least one of the color and depth of a position in an image of a scene, the method comprising: obtaining a first depth map and a second depth map of the scene, the first depth map and the second depth map corresponding to a first view and a different second view of the scene, respectively; based on the first depth map and the second depth map, generating a plurality of depth values ​​along a ray corresponding to a target position in a target view of the scene, the target view being different from the first view and the second view of the scene; generating input for a first neural network based on the generated plurality of depth values; providing the generated input as input to the first neural network, the first neural network being trained to predict light reflection characteristics for positions along the ray corresponding to the plurality of depth values ​​for the generated input; and predicting at least one of the color and the depth of the target position in the image of the target view of the scene based on the predicted light reflection characteristics.

[0123] Embodiment 2, according to embodiment 1, wherein the generated multiple depth values ​​include local foreground depth values ​​and local background depth values.

[0124] Embodiment 3, according to embodiment 3, wherein the local foreground depth value and the local background depth value may correspond to a local minimum depth value and a local maximum depth value within a spatial window centered on the target position, respectively.

[0125] Embodiment 4, according to embodiment 3, wherein the size of the spatial window is based on an expected uncertainty value of the depth of the target position in the target view of the scene.

[0126] Embodiment 5, according to any one of embodiments 1-4, wherein the light reflection characteristics include color and at least one of weight and opacity.

[0127] Embodiment 6, according to any one of embodiments 1-5, wherein generating multiple depth values ​​includes: generating a third depth map corresponding to the target view of the scene based on the first depth map and the second depth map; and generating the multiple depth values ​​for the target position in the target view in the scene based on the third depth map.

[0128] Embodiment 7, according to embodiment 6, wherein generating the third depth map comprises forward point rendering.

[0129] Embodiment 8, according to embodiment 6 or 7, wherein the third depth map has a lower resolution than the first depth map and the second depth map.

[0130] Embodiment 9, according to any one of embodiments 1-9, wherein predicting at least one of a color and a depth of a target location in the image of the target view of the scene includes providing the light reflection characteristic as an input to a second neural network, the second neural network being trained to predict at least one of a color and a depth value based on the input.

[0131] Embodiment 10, according to embodiment 9, wherein the second neural network comprises a multilayer perceptron.

[0132] Embodiment 11, according to any one of embodiments 1-10, wherein generating the plurality of depth values ​​is based on local planar fitting of a plurality of depth surfaces in the scene.

[0133] Embodiment 12, according to any one of embodiments 1-11, wherein the generated plurality of depth values ​​includes local mid-ground depth values.

[0134] Embodiment 13, based on any one of embodiments 1-12, wherein the depth of the target position in the image of the target view of the scene is estimated in real time.

[0135] Embodiment 14, a computer program comprising code means, wherein when the program is run on a processing system, the code means is configured to implement the method according to any one of the preceding claims.

[0136] Embodiment 15, a system for estimating at least one of the color and depth of a position in an image of a scene, the method comprising: an input interface, configured to: obtain a first depth map and a second depth map of the scene, the first depth map and the second depth map corresponding to a first view and a different second view of the scene, respectively; and a depth estimation unit, configured to generate a plurality of depth values ​​along a ray corresponding to a target position in a target view of the scene based on the first depth map and the second depth map, the target view being different from the first view and the second view of the scene; generate input for a first neural network based on the generated plurality of depth values; provide the generated input as input to the first neural network, the first neural network being trained to predict light reflection characteristics for the position along the ray corresponding to the plurality of depth values ​​for the generated input; and predict at least one of the color and the depth of the target position in the target view image of the scene based on the predicted light reflection characteristics.

[0137] and are more particularly defined by the appended claims.

[0138] Those skilled in the art can understand and implement variations to the disclosed embodiments when practicing the invention for which protection is sought by studying the drawings, the disclosure, and the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "one" or "an" does not exclude a plurality. A single processor or other unit can perform the functions of several items recited in the claims. Although specific measures are recited in mutually different dependent claims, this does not indicate that a combination of these measures cannot be used advantageously. If a computer program is described above, it can be stored / distributed on a suitable medium such as an optical storage medium or a solid-state medium provided together with other hardware or as part of other hardware, but can also be distributed in other forms such as via the Internet or other wired or wireless telecommunications systems. If the term "suitable for" is used in the claims or description, it should be noted that the term "suitable for" is intended to be equivalent to the term "configured to". Any figure marks in the claims should not be interpreted as limiting the scope.

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of an instruction that includes one or more executable instructions for implementing the specified function(s). In some alternative embodiments, the functions noted in the blocks may occur in the order shown in the diagrams. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or by a combination of dedicated hardware and computer instructions, the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of an instruction that includes one or more executable instructions for implementing the specified function(s). In some alternative embodiments, the functions noted in the blocks may occur in the order shown in the diagrams. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow charts, and combinations of blocks in the block diagrams and / or flow charts, may be implemented by a dedicated hardware system that performs the specified functions or actions or executes a combination of dedicated hardware and computer instructions.

Claims

1. A method for estimating at least one of color and depth of a location in an image of a scene, the method comprising: Obtaining a first depth map and a second depth map of a scene, the first depth map and the second depth map corresponding to a first view and a different second view of the scene, respectively; generating, based on the first depth map and the second depth map, a plurality of depth values ​​along a ray corresponding to a target position in a target view of the scene, the target view being different from the first view and the second view of the scene, wherein the generated plurality of depth values ​​include a local foreground depth value and a local background depth value, wherein the local foreground depth value and the local background depth value correspond to a local minimum depth value and a local maximum depth value, respectively, within a spatial window centered at the target position, and wherein a size of the spatial window is based on an expected uncertainty value of the depth of the target position in the target view of the scene; generating an input to a first neural network based on the generated plurality of depth values; providing the generated input as input to the first neural network, the first neural network being trained to predict light reflection characteristics for positions along the ray corresponding to the plurality of depth values ​​for the generated input; and predicting at least one of a color and a depth of the target location in the image of the target view of the scene based on the predicted light reflection characteristics, wherein predicting at least one of a color and a depth of the target location in the image of the target view of the scene comprises providing the light reflectance characteristic as an input to a second neural network, the second neural network being trained to predict at least one of a color and a depth value based on the input, And wherein the second neural network includes a multilayer perceptron.

2. The method according to any one of the preceding claims, wherein The light reflection characteristics include color and at least one of weight and opacity.

3. The method according to any one of the preceding claims, wherein Generating multiple depth values ​​involves: generating a third depth map corresponding to the target view of the scene based on the first depth map and the second depth map; and A plurality of depth values ​​for the target position in the target view in the scene are generated based on the third depth map.

4. The method according to claim 3, wherein: Generating the third depth map includes forward point rendering.

5. The method according to claim 3 or 4, wherein: The third depth map has a lower resolution than the first depth map and the second depth map.

6. The method according to any one of the preceding claims, wherein Generating the plurality of depth values ​​is based on local planar fitting of a plurality of depth surfaces in the scene.

7. The method according to any one of the preceding claims, wherein The generated plurality of depth values ​​includes a local mid-ground depth value.

8. A method according to any one of the preceding claims, wherein The depth of the target location in an image of the target view of the scene is estimated in real time.

9. A computer program comprising code means for implementing the method of any of the preceding claims when said program is run on a processing system.

10. A system for estimating at least one of color and depth of a location in an image of a scene, the method comprising: Input interface, which is configured as: Obtaining a first depth map and a second depth map of a scene, the first depth map and the second depth map corresponding to a first view and a different second view of the scene, respectively; as well as A depth estimation unit configured to: generating, based on the first depth map and the second depth map, a plurality of depth values ​​along a ray corresponding to a target position in a target view of the scene, the target view being different from the first view and the second view of the scene, wherein the generated plurality of depth values ​​include a local foreground depth value and a local background depth value, wherein the local foreground depth value and the local background depth value correspond to a local minimum depth value and a local maximum depth value, respectively, within a spatial window centered at the target position, and wherein a size of the spatial window is based on an expected uncertainty value of the depth of the target position in the target view of the scene; generating an input to a first neural network based on the generated plurality of depth values; providing the generated input as input to the first neural network, the first neural network being trained to predict light reflection characteristics for positions along the ray corresponding to the plurality of depth values ​​for the generated input; and predicting at least one of a color and a depth of the target location in the image of the target view of the scene based on the predicted light reflection characteristics, wherein predicting at least one of a color and a depth of the target location in the image of the target view of the scene comprises providing the light reflectance characteristic as an input to a second neural network, the second neural network being trained to predict at least one of a color and a depth value based on the input, And wherein the second neural network includes a multilayer perceptron.

Images