Apparatus and method for processing depth maps

Abstract

The processing of the depth map includes, for at least a first pixel in the depth map, determining a set of candidate depth values ​​(105) including other depth values ​​in the depth map, determining a cost value for each of the candidate depth values ​​according to a cost function (107), selecting a first depth value according to the cost values ​​for the set of candidate depth values ​​(109), and determining an updated depth value for the first pixel according to the first depth value (111). The set of candidate depth values ​​includes a first candidate depth value along the first direction that is not included in the set of candidate depth values ​​or that is further away from the first pixel than at least one pixel along the first direction that has a higher cost function than the first candidate depth value.

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for processing depth maps, and more particularly to processing depth maps for multiview depth / disparity estimation, although this is not exclusive. [Background technology]

[0002] Traditionally, the technical processing and use of images were based on two-dimensional imaging, but nowadays, the third dimension is increasingly being explicitly considered in image processing.

[0003] For example, three-dimensional (3D) displays have been developed that add a third dimension to the viewing experience by supplying different views of the scene being viewed to the viewer's two eyes. This can be achieved by having the user wear glasses to separate the two displayed views. However, this is considered inconvenient for the user, so in many scenarios, it is preferable to use automated stereoscopic displays that use means in the display (such as lenticular lenses or barriers) to separate the views and send the views in different directions to reach each of the user's eyes individually. While a stereoscopic display requires two views, automated stereoscopic displays typically require more views (e.g., nine views).

[0004] Another example is the use of free viewpoint, which allows (within limitations) spatial navigation of a scene captured by multiple cameras. This can be done, for example, on a smartphone or tablet, providing a game-like experience. Alternatively, the data can be viewed on an augmented reality (AR) or virtual reality (VR) headset.

[0005] In many applications, it is desirable to generate a view image for a new viewing direction. Various algorithms are known for generating such new view images based on image and depth information, but they tend to be highly dependent on the accuracy of the provided (or derived) depth information.

[0006] In practice, 3D image information is provided by multiple images corresponding to different viewing directions of the scene. Such information can be captured using a dedicated 3D camera system that captures two or more images simultaneously from an offset camera position.

[0007] However, in many applications, the provided images may not directly correspond to the desired orientation, or more images may be required. For example, for automatic upright displays, more than two images are often needed, and in practice, 9 to 26 view images are frequently used.

[0008] Viewpoint shifting processing may be employed to generate images corresponding to different viewing directions. This is typically done by a viewpoint shifting algorithm that uses an image from a single viewing direction along with associated depth information (or possibly multiple images and associated depth information). However, the provided depth information must be sufficiently accurate to generate a new view image without significant artifacts.

[0009] Other exemplary applications include virtual reality experiences in which right-eye and left-eye views are successively generated for a virtual reality headset to match changes in user movement and orientation. Such generation of dynamic virtual reality views may often be based on light intensity images combined with relevant depth maps that provide relevant depth information.

[0010] The quality of one or more three-dimensional images presented from a new view depends on the quality of the received image and depth data; specifically, three-dimensional perception depends on the quality of the received depth information. Other algorithms or processes that depend on the depth information of images are known and tend to be highly sensitive to the accuracy and reliability of the depth information.

[0011] However, in many practical applications and scenarios, the provided depth information tends to be suboptimal. In fact, in many practical applications and usage scenarios, the depth information is not as accurate as desired, which can introduce errors, artifacts, and / or noise into the processing and the resulting images.

[0012] In many applications, depth information describing a real-world scene is estimated from depth indices determined from captured images. For example, depth information may be generated by estimating and extracting depth values ​​by comparing view images for different view positions.

[0013] For example, in many applications, a 3D scene is captured as a stereoscopic image using two cameras positioned slightly differently. A specific depth value is then generated by estimating the parallax between corresponding image objects in the two images. However, such depth extraction and estimation have problems and tend to result in less-than-ideal depth values. This again leads to artifacts and a decrease in 3D image quality.

[0014] To improve depth information, several techniques have been proposed for post-processing and / or improving depth estimation and / or depth maps. However, these tend not to be optimal, tend to lack optimal accuracy and reliability, and / or may be difficult to implement due to the required computational resources, for example. Examples of such algorithms are described in WO2020 / 178289A1 and EP3396949A1.

[0015] A specified approach has been proposed in which the depth map is initialized using a scanning approach where the current pixel depth is updated based on a candidate set of candidate depth values, which are typically the depth values ​​of adjacent pixels, and then updated iteratively. The update of the current pixel depth value depends on a cost function. However, such an approach tends to be suboptimal in all scenarios, including the fact that while it improves the depth map in many scenarios, it does not always produce an optimally accurate depth map. It also tends to be computationally expensive because it must consider a large number of candidate pixels.

[0016] Therefore, improved approaches to generating / processing / modifying depth information are advantageous, in particular, because approaches to processing depth maps enable greater flexibility, easier implementation, reduced complexity, reduced resource requirements, improved depth information, more reliable and / or accurate depth information, an improved 3D experience, improved quality of depth-based rendered images, and / or improved performance. [Overview of the project] [Problems that the invention aims to solve]

[0017] Therefore, the present invention preferably aims to mitigate, reduce, or eliminate one or more of the aforementioned disadvantages, either individually or in any combination. [Means for solving the problem]

[0018] According to an aspect of the present invention, a method for processing a depth map is provided. The method includes receiving a depth map and, for at least a first pixel of the depth map, determining a set of candidate depth values. The set of candidate depth values includes depth values for other pixels of the depth map other than the first pixel. According to a cost function, a cost value is determined for each of the candidate depth values in the set of candidate depth values. According to the cost values of the set of candidate depth values, a first depth value is selected from the set of candidate depth values, and an updated depth value for the first pixel is determined according to the first depth value. The set of candidate depth values includes a first candidate depth value along a first direction from the first pixel. Along the first direction, at least one first intervening pixel set of pixels does not include the candidate depth value for which the cost function for the candidate depth value does not exceed the cost function for the first candidate depth value, and the distance from the first pixel to the first candidate depth value is greater than the distance from the first pixel to the first intervening pixel set.

[0019] The present invention improves depth maps, resulting in improved 3D image processing and perceptual rendering quality. In particular, this approach provides a more consistent and / or accurate depth map in many embodiments and scenarios. This processing provides an improved depth map while maintaining a sufficiently low complexity and / or resource requirement in many embodiments.

[0020] An advantage in many embodiments is that this approach is very suitable for use and integration with depth estimation techniques, such as parallax-based depth estimation using stereo or multi-view images.

[0021] This approach improves depth maps using a relatively low complexity and low resource requirement approach in particular. This approach enables sequential bit scanning and processing with a relatively small number of decisions per pixel that are sufficient to increase overall accuracy, for example. [[ID=IS]]

[0022] The depth map indicates depth values for the pixels of an image. The depth value may be any value indicating depth, including, for example, a disparity value, a z - coordinate, or a distance from a viewpoint value.

[0023] The processing of the first pixel may be repeated with a new pixel of the depth map selected for each iteration. The selection of the first pixel is by a scanning sequence for the depth map. The pixel corresponds to a position / area in the depth map where a depth value is provided. A pixel in the depth map corresponds to one or more pixels in a related image where the depth map indicates depth. The depth map is formed by a two - dimensional component of pixels each having a depth value provided for each pixel. Thus, each pixel / depth value is provided for an (pixel) area of the depth map. A reference to a pixel is a reference to the depth value (for the pixel), and vice versa. A reference to a pixel is a reference to the position in the depth map for the depth value. Each pixel of the depth map is associated with one depth value (and vice versa).

[0024] The cost function is implemented as a merit function, and the cost value is indicated by the merit value. The selection of the first depth value according to the cost value is implemented as the selection of the first depth value according to the merit value determined from the merit function. An increasing merit value / function is a decreasing cost value / function. The selection of the first depth value corresponds to / is equivalent to the selection of the candidate depth value of the set of candidate depth values having the lowest cost value, which is the selection of the candidate depth value of the set of candidate depth values having the highest merit value.

[0025] The updated depth value has a value that depends on the first depth value. This may, in some situations and also, in some cases depending on the pixel, result in an updated depth value that is the same as the depth value of the first pixel before processing. In some embodiments, the updated depth value is determined as a function of the first depth value, specifically, a function that does not depend on other depth values of the depth map than the first depth value. In many embodiments, the updated depth value is set to be equal to the first depth value.

[0026] In some embodiments, the first direction may be the only direction in which the interposed set of pixels as described exists. In some embodiments, the first direction may be the direction of an angular interval from the first pixel in which the interposed set of pixels as described exists. The angular interval is or has a span / width / range not exceeding 1°, 2°, 3°, 5°, 10°, or 15°. In such embodiments, the first direction may be replaced by a reference to a direction within such an angular interval.

[0027] The set of candidate depth values ​​includes a first candidate depth value along the first direction that is further away from the first pixel than at least one pixel along the first direction that is not included in the set of candidate depth values ​​or has a higher cost function than the first candidate depth value.

[0028] According to any feature of the present invention, the cost function along the first direction has a monotonically increasing cost gradient as a function of the distance from the first pixel when the distance is below a distance threshold, and a decreasing cost gradient as a function of the distance from the first pixel when at least one distance from the first pixel is above a threshold.

[0029] This provides improved performance and / or implementation in many embodiments. There exists a first intervening pixel set of at least one pixel that does not contain a candidate depth value among a set of candidate depth values ​​whose cost function for the candidate depth value does not exceed the cost function for the first candidate depth value, and it is ensured that the distance from the first pixel to the first candidate depth value is greater than the distance from the first pixel to the first intervening pixel set.

[0030] A monotonically increasing gradient as a function of distance is a gradient that either always increases as distance increases or remains constant.

[0031] According to any feature of the present invention, the first intervening pixel set is a set of pixels whose depth values ​​are not included in the set of candidate values.

[0032] This provides improved performance and / or implementation in many embodiments. There exists a first intervening pixel set of at least one pixel that does not contain a candidate depth value among a set of candidate depth values ​​whose cost function for the candidate depth value does not exceed the cost function for the first candidate depth value, and it is ensured that the distance from the first pixel to the first candidate depth value is greater than the distance from the first pixel to the first intervening pixel set.

[0033] According to any feature of the present invention, the cost function includes a cost contribution that depends on the difference between image values ​​in a multiview image for pixels offset by disparity that match the candidate depth value to which the cost function is applied.

[0034] This approach, combined with consideration of multiview parallax, provides a favorable depth estimation approach based on multiview images. With this approach, for example, the initial depth map can be iteratively updated based on the matching between different images in the multiview.

[0035] According to any feature of the present invention, the method further comprises determining a first direction as the gravity direction of the depth map, the gravity direction being a direction in the depth map that coincides with the direction of gravity in the scene represented by the depth map.

[0036] This provides particularly efficient performance and improved depth maps, leveraging typical scene characteristics to deliver improved and often more accurate depth maps.

[0037] According to any feature of the present invention, the first direction is the vertical direction in the depth map.

[0038] This provides particularly efficient performance and improved depth maps, leveraging typical scene characteristics to deliver improved and often more accurate depth maps.

[0039] According to any feature of the present invention, the method further comprises determining a depth model for at least a portion of a scene represented by a depth map, wherein the cost function for depth values ​​depends on the difference between the depth values ​​and the model depth values ​​determined from the depth model.

[0040] This provides particularly efficient performance and improved depth maps, leveraging typical properties of scene objects to provide improved and often more accurate depth maps. The cost function increases the cost as the difference between the depth value and the model depth value increases.

[0041] According to any feature of the present invention, the cost function is asymmetric with respect to whether the depth value exceeds or falls below the model depth value.

[0042] This provides a particularly advantageous depth map in many embodiments and scenarios.

[0043] According to any feature of the present invention, the depth model is a background model for the scene.

[0044] This provides a particularly advantageous depth map in many embodiments and scenarios.

[0045] According to any feature of the present invention, the method further includes including candidate depth values ​​in a set of candidate depth values ​​that are not from a depth map, and including at least one depth value among depth values ​​from another depth map in a temporal sequence of depth maps, the sequence including depth maps and depth values ​​independent of the scene represented by the depth maps, and depth values ​​determined according to the offset of the depth value to a first pixel.

[0046] This provides a particularly advantageous depth map in many embodiments and scenarios.

[0047] According to any feature of the present invention, the cost function for a depth value depends on the type of depth value, and the type is one of a group of types that includes at least one of the following: depth values ​​of a depth map, depth values ​​of a depth map closer than a distance threshold, depth values ​​of a depth map further than a distance threshold, depth values ​​from another depth map in a temporal sequence of depth maps including a depth map, depth values ​​having a scene-independent depth value offset with respect to the depth value of a first depth value, depth values ​​independent of the scene represented by the depth map, and depth values ​​determined according to the offset of the depth value with respect to a first pixel.

[0048] This provides a particularly advantageous depth map in many embodiments and scenarios.

[0049] According to any feature of the present invention, the method is configured to process multiple pixels of a depth map by iteratively selecting a new first pixel from a plurality of pixels and performing a step for each new first pixel.

[0050] According to any feature of the present invention, the set of candidate depth values ​​in a second direction from a first pixel does not include a second candidate depth value in which the pixel set of at least one pixel along the second direction does not include that candidate depth value from a set of candidate depth values ​​in which the cost function for the candidate depth value does not exceed the cost function for the second candidate depth value, and the distance from the first pixel to the second candidate depth value is greater than the distance from the first pixel to the pixel set.

[0051] This provides a particularly advantageous depth map in many embodiments and scenarios. In some embodiments, the first direction may be the only direction in which the described set of intervening pixels exists.

[0052] According to an aspect of the present invention, an apparatus for processing a depth map is provided, the apparatus comprising a receiver for receiving a depth map and a processor for processing the depth map, the processing comprising determining a set of candidate depth values ​​for at least a first pixel of the depth map, the set of candidate depth values ​​comprising depth values ​​for other pixels of the depth map other than the first pixel, determining a cost value for each of the candidate depth values ​​in the set of candidate depth values ​​according to a cost function, selecting a first depth value from the set of candidate depth values ​​according to the cost value for the set of candidate depth values, and the first The steps include determining an updated depth value for a first pixel in accordance with the depth value, wherein the set of candidate depth values ​​includes first candidate depth values ​​along a first direction from the first pixel, and along the first direction, a first intervening set of pixels of at least one pixel does not include that candidate depth value from a set of candidate depth values ​​whose cost function for the candidate depth value does not exceed the cost function for the first candidate depth value, and the distance from the first pixel to the first candidate depth value is greater than the distance from the first pixel to the first intervening set of pixels.

[0053] These and other aspects, features and advantages of the present invention will be made clear and explained by reference to the embodiments described below.

[0054] Embodiments of the present invention will be described by reference only to the following drawings. [Brief explanation of the drawing]

[0055] [Figure 1] An example of a method for processing depth maps according to several embodiments of the present invention is shown. [Figure 2] Examples of devices for processing depth maps according to several embodiments of the present invention are shown. [Figure 3] Here are some examples of depth maps. [Figure 4] Here are some examples of pixels in a depth map and their corresponding depth values. [Figure 5] Here are some examples of pixels in a depth map and their corresponding depth values. [Figure 6] Examples of direction-oriented cost functions for methods according to several embodiments of the present invention are shown. [Figure 7] Examples of depth maps generated by different processes are shown. [Modes for carrying out the invention]

[0056] The following description focuses on embodiments of the present invention applicable to the processing of depth maps in images, specifically, processing such depth maps as part of a multiview depth estimation method. However, it will be understood that the present invention is not limited to this application and is applicable to many other scenarios.

[0057] Images representing a scene are now complemented by depth maps that provide information about the depth of image objects in the scene, i.e., provide additional depth data to the image. Such additional information provides many additional services, for example, by enabling viewpoint changes, 3D representation, etc. Depth maps tend to provide depth values ​​for each of several pixels, and pixels are usually arranged in an array with a first number of horizontal and a second number of vertical columns. The depth values ​​provide depth information for the associated image pixels. In many embodiments, the resolution of the depth map is the same as the resolution of the image, so each pixel of the image has a one-to-one connection to one depth value of the depth map. However, in many embodiments, the resolution of the depth map is lower than that of the image, and in some embodiments, the depth values ​​of the depth map are common to multiple pixels of the image (specifically, depth map pixels are larger than image pixels).

[0058] A depth value is any value that indicates depth, specifically including depth coordinate values ​​(e.g., directly providing the z-value of a pixel) or parallax values. In many embodiments, a depth map is a rectangular array (with rows and columns) of pixels, each providing a depth ( / parallax) value.

[0059] The accuracy of scene depth representation is a critical parameter in the quality of the resulting image rendered and perceived by the user. Therefore, generating accurate depth information is crucial. While achieving accurate values ​​is relatively easy for artificial scenes (e.g., computer games), it is extremely difficult for applications requiring the capture of real-world scenes.

[0060] Several different approaches have been proposed for estimating depth. One approach is to estimate the parallax between different images capturing the scene from different viewpoints. However, such parallax estimation is inherently imperfect. Furthermore, this approach requires the scene to be captured from multiple directions, which is not the case for legacy capture, for example. Another option is to perform motion-based depth estimation, which takes advantage of the fact that (in a series of images) the motion of objects in the image tends to be greater for objects closer to the camera than for objects further away (for example, in the case of a translational camera, perhaps after correcting for the actual motion of the corresponding object in the scene). A third approach is to take advantage of predetermined (assumed) information about depth in the scene. For example, in an outdoor scene (and in fact most typical indoor scenes), lower objects in the image tend to be closer than higher objects in the image (for example, the floor or ground increases in distance to the camera as its height increases, and the sky tends to be further behind than, for example, the lower ground). Thus, a predetermined depth profile is used to estimate a suitable depth map value.

[0061] However, most depth estimation techniques tend not to provide perfect depth estimates, although improved and typically more reliable and / or accurate depth values ​​are advantageous for many applications.

[0062] The following describes an approach to processing and updating depth maps. In some embodiments, this approach is used as part of depth estimation, specifically as a depth estimation algorithm that considers parallax between multiple images capturing a scene from different viewpoints. In fact, the process may be an integrated part of a depth estimation algorithm that determines depth from multiview images. However, this is not essential to this approach, and in some embodiments, it is understood that this approach is applied, for example, as post-processing of the estimated depth map.

[0063] This approach improves depth maps and provides more accurate depth information in many scenarios. Furthermore, it is suitable for combining different depth estimation considerations and approaches and can be used to improve depth estimation.

[0064] This approach will be explained with reference to Figure 1, which shows a flowchart of how to process depth maps, and Figure 2, which shows the elements of the corresponding equipment that implements this method.

[0065] The apparatus in Figure 2 may specifically be a processing unit such as a computer or a processing module. Thus, it is implemented by suitable processors such as a CPU, MPU, or DSP. The apparatus may further include volatile and non-volatile memory coupled to the processor, as is known to those skilled in the art. Furthermore, it may include suitable input and output circuits, such as a user interface or network interface.

[0066] The method in Figure 1 begins with step 101, in which receiver 201 receives a depth map. In many embodiments, the depth map is received along with an associated image. Furthermore, in many embodiments, the depth map is received along with an image and one or more images further associated with a separate depth map. For example, other images representing the scene from a different viewpoint are received. In some embodiments, the image and depth map are part of a temporal sequence, such as the image being an image or frame in a video sequence. Thus, in some embodiments, receiver 201 receives the image and / or depth map for other times. The depth map and image being processed will hereafter also be referred to as the first or current depth map and the first or current image, respectively.

[0067] The received first depth map may be the initial first image processed to generate a more accurate depth map, and specifically, in some embodiments, is the initial input to the depth estimation process. The following description focuses on an example in which the method is used as part of a multiview depth estimation process, which involves considering parallax between different images of the same scene from different viewpoints. This process is initialized with an initial depth map that provides a very rough indication of the possible depth of the image.

[0068] For example, an initial depth map can be easily generated by detecting objects in an image and dividing the image into objects and background. A predetermined depth is assigned to the background section, and different predetermined depths are determined for the objects, indicating that they are further ahead, or a rough disparity estimation is performed based, for example, on the search for corresponding objects in another image, and the resulting estimated depth is assigned to the entire object. An example of the resulting depth map is shown in Figure 3(a).

[0069] As another example, a predetermined depth pattern, such as increased depth, is assigned to increase the height in an image. An example of such a depth map is shown in Figure 3(b) and is suitable, for example, for landscape scenes and images. These approaches may be combined, as shown in Figure 3(c).

[0070] In many embodiments, left / right image pairs are used to initialize a block-based 2D disparity vector field, for example, using a pre-fitted 3D depth model. In such embodiments, depth values ​​may be disparity values ​​or depth values ​​that directly indicate the distance from the viewpoint, or may be calculated from these. This approach takes into account knowledge of the geometric structure of the scene, such as the ground and background. For example, a 2D disparity field is generated based on a 3D model and used as an initial depth map.

[0071] Thus, in the approach described, an initial first image with a rough and typically inaccurate initial first depth map is received by receiver 201, in the example, along with at least one other image representing the scene from a different viewpoint. The approach then processes this depth map to generate a more accurate depth map that better reflects the actual depth of different pixels in the depth map.

[0072] One or more depth maps and one or more optional images are supplied from receiver 201 to processor 203, which is configured to perform the remaining steps of the method as described below with reference to Figure 1.

[0073] The following approach can, in principle, be applied only to a subset of depth values / pixels in the depth map, or in fact, only to a single pixel in the depth map, but this process is usually applied to all or nearly all pixels in the depth map. The application is usually sequential. For example, the process scans through the depth map and sequentially selects pixels to process. For example, the method starts from the top left corner and scans horizontally first, then vertically, until the pixel in the bottom right corner is processed; that is, a scan from left to right and top to bottom is applied.

[0074] Furthermore, in many embodiments, the approach may be iterative and applied repeatedly to the same depth map. Thus, the depth map resulting from one process / update is used as the input depth map for subsequent processes / updates. In many embodiments, the scanning of the depth map with updates resulting from the depth values ​​is repeated multiple times, for example, 5 to 10 iterations.

[0075] This method begins in step 101, when the receiver 201 receives the depth map as described above and transfers it to the processor 203.

[0076] In step 103, the next pixel is selected. When the depth map process is started, the next pixel is usually a predetermined pixel, such as the top-right pixel. Otherwise, the next pixel may be the next pixel according to the sequence of the applied process, specifically a predetermined scan sequence / order.

[0077] The identified pixel, hereafter referred to as the first or current pixel, is then used by the method to determine a depth value for this pixel. The depth value of the first or current pixel is also referred to as the first or current depth value, and the terms initial and updated depth value are used to refer to the values ​​of the pixel before and after processing, i.e., the depth values ​​of the depth map before and after the current iteration, respectively.

[0078] In step 105, the processor 203 proceeds to determine / select a set of candidate depth values. The set of candidate depth values ​​is selected to include depth values ​​for a set of candidate pixels. The candidate set of pixels in the current depth map includes several other pixels in the depth map. For example, the candidate set is selected to include depth values ​​for pixels in the neighborhood around the current pixel, such as a set of pixels within a given distance from the current pixel or within a window / kernel around the current pixel.

[0079] In many embodiments, the candidate set of pixels also includes the depth value of the current pixel itself; that is, the first depth value is itself one of the candidate depth values ​​in the set.

[0080] Furthermore, in many embodiments, the set of candidate depth values ​​also includes depth values ​​from other depth maps. For example, in many embodiments where the image is part of a video stream, one or more depth values ​​from previous and / or subsequent frames / images are also included in the set of candidate depth values, or depth values ​​from other views from which the depth map is simultaneously estimated.

[0081] In some embodiments, the set of candidate depth values ​​further includes values ​​that are not directly depth values ​​of the depth map. For example, in some embodiments, the set of candidate depth values ​​includes one or more fixed depth values, or relative offset depth values, such as depth values ​​that are greater or less than the current initial depth value by a fixed offset. In another example, the set of candidate depth values ​​includes one or more random or semi-random depth values.

[0082] Following step 105, in step 107, the cost value is determined for the set of candidate depth values, specifically, the cost value is determined for each candidate depth value in the set of candidate depth values.

[0083] The cost value is determined based on a cost function that depends on several different parameters, as will be described in more detail later. In many embodiments, the cost function of a candidate depth value for a pixel in the current depth map depends on the difference between image values ​​in a multiview image that is offset by the parallax corresponding to the depth value. Thus, for each candidate depth value in a first depth value, or possibly a set of candidate depth values ​​belonging to a set of candidate depth values, the cost function decreases monotonically as a function of the difference between two view images in the multiview image in an image area having parallax between two image views that match the candidate depth value. The image area may specifically be the image area containing the current pixel and / or the pixel of the candidate depth value. The image area is typically relatively small, for example, 1%, 2%, 5%, or 10% or less of the image, and / or less than or equal to, for example, 100, 1000, 2000, 5000, or 10000 pixels.

[0084] In some embodiments, the processor 203 determines the disparity between two images that match a given candidate depth value. Next, this disparity is applied to identify an area in one of the two images that is offset by that disparity to an area in the other image. A measure of the difference between the image signal values ​​in the two areas, e.g., RGB values, is determined. Thus, a measure of difference is determined for the two images / image areas based on the assumption that the candidate depth value is correct. A smaller difference increases the likelihood that the candidate depth value is an accurate reflection of the depth. Thus, a lower measure of difference results in a lower cost function.

[0085] The image area is typically a small area around the first / current pixel, and in fact, in some embodiments, it includes only the first / current pixel.

[0086] For candidate depth values ​​corresponding to the current depth map, the cost function therefore includes a cost contribution that depends on the match between the two multiview images for the disparity corresponding to the candidate depth value.

[0087] In many embodiments, the cost values ​​for candidate depth values ​​of other depth maps associated with an image, such as time-offset depth maps and images, also include the corresponding image matching cost contribution.

[0088] In some embodiments, some of the costs among the candidate depth values ​​do not include the image matching cost contribution. For example, a fixed cost value is assigned to a predetermined fixed depth offset that is not associated with a depth map or image.

[0089] The cost function is typically determined to reflect the likelihood that the depth value will accurately or correctly represent the current pixel's depth value.

[0090] It is understood that determining merit values ​​based on a merit function, and / or selecting candidate depth values ​​based on merit values, is essentially the same as determining cost values ​​based on a cost function, and / or selecting candidate depth values ​​based on cost values. Merit values ​​can be converted to cost functions simply by applying a function to the merit values, where the function is an arbitrarily monotonically decreasing function. The higher the merit value, the lower the cost value; for example, selecting the candidate depth value with the highest merit value is directly equivalent to selecting the lowest cost value.

[0091] Following step 107, which determines a cost value for each candidate depth value in the set of candidate depth values, step 109 selects a depth value from the set of candidate depth values ​​according to the cost value for the set of candidate depth values. The selected candidate depth value will hereafter be referred to as the selected depth value.

[0092] In many embodiments, selection is the selection of candidate depth values ​​from which the lowest cost value is determined. In some embodiments, more complex criteria are evaluated, taking other parameters into consideration as well (equivalently, such considerations are usually considered part of the (modified) cost function).

[0093] Therefore, for the current pixel, this approach selects a candidate depth value that is most likely to reflect the correct depth value for the current pixel, as determined by the cost function.

[0094] Following step 109, step 111 determines an updated depth value for the current pixel based on the selected depth value. The appropriate update depends on the specific requirements and preferences of each embodiment. For example, in many embodiments, the previous depth value for the first pixel may simply be replaced by the selected depth value. In other embodiments, the update may take the initial depth value into consideration, for example, the updated depth value may be determined as a weighted combination of the initial depth value and the selected depth value, for example, the weights depending on the absolute cost value of the selected depth value.

[0095] Therefore, in the subsequent step 111, the updated or modified depth value for the current pixel is determined. In step 113, it is evaluated whether all pixels of the current depth map to be processed have actually been processed. Typically, this corresponds to determining whether all pixels in the image have been processed, specifically, detecting whether the scanning sequence has reached its end.

[0096] Otherwise, the method returns to step 103, where the next pixel is selected, and the process is repeated for this next pixel. Otherwise, the method proceeds to step 115, where it is evaluated whether further iterations should be applied to the depth map. In some embodiments, only one iteration is performed, and step 115 is omitted. In other embodiments, the process is repeatedly applied to the depth map until, for example, a stopping criterion is met (e.g., the total amount of change occurring in previous iterations falls below a threshold), or until a predetermined number of iterations have been performed.

[0097] If further iterations are required, the method returns to step 103, and the next pixel is determined to be the first pixel in the new iteration. Specifically, the next pixel is the first pixel in the scanning sequence. If no further iterations are required, the method proceeds to step 117 and terminates with an updated and usually improved depth map. The depth map can be output to another function, such as a view composite processor, via output circuit 205, for example.

[0098] While the above explanation focuses on application to a single pixel, it is understood that a block process is also applied, for example, where the determined update depth value is applied to all depth values ​​within a block containing the first pixel.

[0099] The specific selection of candidate depth values ​​depends on the desired behavior and performance for a particular application. Typically, the set of candidate depth values ​​includes several pixels in the neighborhood of the current pixel. A kernel, area, or template may cover the current pixel, and pixels within the kernel / area / template may be included in the candidate set. Furthermore, the same pixel at different time frames (typically immediately before or after the current frame in which the depth map is created) may be included, as well as potentially including kernels of neighboring pixels that are typically significantly smaller than the kernel of the current depth map. Typically, at least two offset depth values ​​(corresponding to increased and decreased depths, respectively) are also included.

[0100] However, to reduce computational complexity and resource requirements, the number of depth values ​​included in the candidate set is usually significantly limited. In particular, since all candidate depth values ​​are evaluated for each new pixel and, typically, for all pixels in the depth map per iteration, many additional processing steps are required for each additional candidate value.

[0101] However, the achievable improvements to the depth map tend to depend heavily on the specific selection and weighting of candidates, and in fact, this tends to have a significant impact on the quality of the final depth map. Thus, balancing quality / performance with computational resource requirements is extremely difficult and highly sensitive to the determination of candidate depth values.

[0102] In many typical applications, it is often preferred to have approximately 5 to 20 candidate depth values ​​in the set of candidate depth values ​​per pixel. In many practical scenarios, it is necessary to limit the number of candidate depth values ​​to about 10 in order to achieve real-time processing for a video sequence. However, with a relatively small number of candidate depth values, the decision / selection of which candidate depth values ​​to include in the set becomes extremely critical.

[0103] An intuitive approach to determining the set of candidate depth values ​​for the depth map to consider when updating the current depth value is to include depth values ​​close to the current pixel, and weighting such that the probability of selecting a more distant depth value is low (or at least not higher), i.e., if all other parameters are equal, candidate depth values ​​closer to the current pixel are selected over those further away. Thus, an intuitive approach is to generate a set of candidate depth values ​​as a kernel containing pixels in a (usually very small) neighborhood around the current pixel, and the cost function increases monotonically with distance from the current pixel. For example, the set of candidate depth values ​​may be determined as all pixels within a predetermined distance of 1 or 2 pixels from the current pixel, and a cost function increasing with distance from the first pixel may be applied.

[0104] However, while such an intuitive approach provides advantageous performance in many embodiments, the inventors recognized that in many applications, advantageous performance can be achieved by taking a counterintuitive approach that increases the bias towards selecting pixels that are farther away along a given direction rather than pixels that are closer in that direction. In this approach, the probability of selecting a more distant pixel thus increases with respect to pixels that are closer along that direction.

[0105] In some embodiments, this is achieved by a set of candidate depth values ​​along a direction that is selected / generated / created / determined or included to include one or more further-distant pixels than one or more pixels not included in the set of candidate depth values. For example, the closest one or more pixels along the direction are included in the set of candidate depth values, followed by one or more pixels not included in the set of candidate depth values, and then one or more further-distant pixels included in the set of candidate depth values.

[0106] An example of such an approach is shown in Figure 4. In this example, the set of candidate depth values ​​includes four neighboring depth values / pixels 401 surrounding the current depth value / pixel 403. The set of candidate depth values ​​may further include the current depth value. However, the set of candidate depth values ​​may further include a distant depth value / pixel 405 along a given direction 407. The distant depth value / pixel 405 is Δ pixels away from the current pixel 403, where Δ > 2, and is usually much larger. In this example, the set of candidate depth values ​​further includes the depth value for pixel 409, which is at the same position in the temporal neighbor depth map.

[0107] Therefore, in this example, a set of candidate depth values ​​is created that includes only seven depth values, and the system then determines the cost value by evaluating a cost function for each of these depth values. Next, it updates the current depth value by selecting the depth value with the lowest cost value and setting it, for example, to the value of the selected candidate depth value. Because there are few candidate depth values, processing that requires very fast and / or low resources can be performed.

[0108] Furthermore, despite the constraints on how many candidate depth values ​​can be evaluated, approaches that include not only nearby depth values ​​but also one or more distant depth values ​​have proven to offer particularly advantageous performance, for example, by often providing more consistent and / or accurate updated depth maps. For example, many depth values ​​represent objects present in the scene and at substantially the same distance. Considering even more distant depth values ​​will, in many situations, include depth values ​​that belong to the same object but provide better depth estimates, for example, because they have less local image noise. Considering specific directions reflects likely properties of the object, such as geometric properties or its relationship to the orientation of the capture relative to the image. For example, with respect to the background, depth tends to be relatively constant horizontally and change vertically, so identifying pixels further away horizontally increases the likelihood that both the current pixel and the further away pixels represent the same (background) depth.

[0109] Therefore, in some embodiments, the set of candidate depth values ​​is determined to include a first set of candidate depth values ​​along a direction from the current pixel that is further away than the set of intervening pixels, while along that direction, the intervening set includes one or more pixels that are closer to the current pixel than the first set of candidate depth values, but whose depth values ​​are not included in the set of candidate depth values. In some embodiments, there is therefore a gap along the direction between the pixel location of the first set of candidate depth values ​​and the current pixel where one or more pixels not included in the set of candidate depth values ​​reside. In many embodiments, the gap lies between the first set of candidate depth values ​​and one or more neighboring pixels included in the set of candidate depth values.

[0110] In some embodiments, the candidate set is adapted based on the image object or image object type to which the current pixel belongs. For example, the processor 203 is configured to perform an image object detection process to detect image objects in the depth map (for example, by detecting them in the relevant image). The candidate set is then adjusted by the detected image objects. In some embodiments, the first direction may also be adapted according to the characteristics of the image object to which the current pixel belongs. For example, a pixel is known to belong to a particular image object or object type, and the first direction may be determined according to the characteristics of this object, such as the longest direction relative to the image object. For example, a boat on water tends to have a significantly longer extension horizontally than vertically, so the first direction is determined to be the horizontal direction. Alternatively, a candidate set that extends further horizontally than vertically may be selected.

[0111] In some embodiments, the system detects a certain type of object (such as a car or airplane) and then adjusts the candidate set based on the category of the classified pixels.

[0112] In some embodiments, increasing the weighting of at least one further-away depth value over closer depth values ​​along a direction is not achieved by excluding one or more pixels along the direction from being included in the set of candidate depth values. In some embodiments, all pixels along the first direction may include pixels further away from the current pixel, and are hereafter referred to as the first candidate depth value and the first candidate pixel, respectively.

[0113] In such embodiments, the increase in the bias of the first candidate depth value with respect to depth values ​​belonging to a set of intervening candidate depth values ​​with lower bias is achieved by appropriately designing the cost function. Specifically, the cost function is such that, with respect to the first candidate depth value, the cost function is lower than the cost function for one or more depth values ​​closer to the first pixel.

[0114] In some embodiments, the cost function along the direction has a cost gradient that increases monotonically with respect to distances where the cost function is evaluated for distances below a distance threshold, from the first pixel to the candidate depth value, and a cost gradient that decreases with respect to distances above the threshold for at least one distance from the first pixel. Thus, up to the distance threshold, the cost function increases or remains constant with increasing distance to the first pixel. However, for at least one distance above the distance threshold, the cost function decreases instead.

[0115] For example, the distance threshold corresponds to the distance to the last pixel in the intervening set. Thus, the cost function increases (or remains constant) for the pixels up to and including the furthest pixel in the intervening set. However, the cost gradient decreases with the distance between this furthest pixel and the first candidate pixel. Therefore, the cost function for the first candidate depth value is lower than for at least one pixel along the direction closer to the current pixel.

[0116] A cost function for one pixel being smaller or lower than that for another pixel means that the resulting cost value determined by the cost function is smaller / lower for two pixels where all other parameters are the same, i.e., all parameters except position are the same. Similarly, a cost function for one pixel being larger or higher than that for another pixel means that the resulting cost value determined by the cost function is larger / higher when considering two pixels where all other parameters are the same. Furthermore, a cost function for one pixel exceeding that of another pixel means that the resulting cost value determined by the cost function is higher than that of the other two pixels when considering two pixels where all other parameters are the same.

[0117] For example, a cost function typically takes several different parameters into account, such as the cost value being determined as C=f(d,a,b,c,...), where d represents the pixel's position (e.g., distance) relative to the current pixel, and a,b,c,... reflect other parameters considered, such as the image signal values ​​of the associated image, other depth values, and smoothness parameters.

[0118] The cost function f(d,a,b,c,...) for pixel A is lower for pixel A than for pixel B if the parameters a,b,c,... are the same for both pixels (and so on for the other terms).

[0119] An example of using a cost function to bias pixels further away is shown in Figure 5. This example corresponds to the example in Figure 4, but the set of candidate depth values ​​further includes all pixels along the direction, i.e., pixels 501-507 as well. However, in this example, the cost function is configured to bias a first set of candidate depth values ​​that are higher than some intervening pixels, specifically pixels 501-507.

[0120] An example of a possible cost function and its dependence on the distance d from the first pixel is illustrated in Figure 6. In this example, the cost function is very low for the neighboring pixel 401, and then increases with distance for pixel 501. However, for the first candidate pixel, the cost function decreases with respect to pixel 501, but remains higher for the neighboring pixel 401. Therefore, by applying this cost function, the depth value of the first candidate is biased higher than that of the intervening pixel 501, but not as high as that of the neighboring pixel 401.

[0121] Therefore, in this approach, the set of candidate depth values ​​includes first candidate depth values ​​along a first direction from a first pixel that is at a greater distance from the current pixel than the intervening set of pixels. The intervening set of pixels includes at least one pixel, and all pixels in the intervening set are either not included in the set of candidate depth values ​​or have a cost function higher than the first candidate depth value. Thus, the intervening set of pixels does not include any candidate depth values ​​whose cost function does not exceed the cost function for the first candidate depth value. Therefore, the set of candidate depth values ​​includes first candidate depth values ​​along a first direction that is further away from the first pixel than at least one pixel along that direction that is not included in the set of candidate depth values ​​or has a cost function higher than the first candidate depth value.

[0122] In some embodiments, this directly corresponds to an intervening pixel set, which is a set of pixels whose depth values ​​are not included in the set of candidate values. In some embodiments, this directly corresponds to an intervening set that includes at least one such candidate pixel / depth value whose cost function for that candidate pixel / depth value exceeds the cost function for the first candidate depth value. In some embodiments, this directly corresponds to a cost function along a direction having a cost gradient that increases monotonically with respect to the distance (where the cost value is determined) from the first pixel to the candidate depth value when the distance is below a distance threshold, and a cost gradient that decreases with respect to distance when the distance from at least one pixel above a threshold.

[0123] The appropriate cost function depends on the specific embodiment. In many embodiments, the cost function includes a cost contribution that depends on the difference between image values ​​in the multiview images for pixels offset by a parallax corresponding to the depth value. As mentioned above, the depth map may be a map to an image (or set of images) of a multiview image set capturing the scene from different viewpoints. Thus, there is parallax between the positions of the same object in different images, and this parallax depends on the depth of the object. Therefore, for a given depth value, the parallax between two or more images of the multiview image can be calculated. Thus, in some embodiments, for a given candidate depth value, the parallax to other images is determined so that, assuming the depth value is correct, the position of a first pixel position in other images can be determined. For one or more pixels at each position, image values ​​such as color or luminance values ​​are compared to determine a suitable difference scale. If the depth value is indeed correct, it is more likely that the image values ​​are the same and the difference scale is small than if the depth value is not the correct depth. Therefore, the cost function may include consideration of the differences between image values, specifically, the cost function reflects an increasing cost for an increasing difference scale.

[0124] By including such matching criteria, this approach is used as an essential component of depth estimation based on disparity between images in multiview images. The depth map is initialized and then iterated through, with updates biased toward smaller differences in image values. In this way, this approach effectively performs integrated depth determination and search / matching across different images.

[0125] In this approach, further depth values ​​in the first direction are therefore biased / weighted higher than at least one depth value closer to the first pixel along the first direction. In some embodiments, this may apply to multiple directions, but in many embodiments, it applies (only) to one direction or directions within small intervals, e.g., intervals of 1°, 2°, 3°, 5°, 10°, or 15°. Equivalently, directions can be considered to have degrees of angular intervals of 1°, 2°, 3°, 5°, 10°, or 15° or less.

[0126] Therefore, in many embodiments, the approach is that for any candidate depth value along the second direction from the first pixel, all depth values ​​along the second direction that are shorter in distance to the first pixel belong to the set of candidate depth values, and the cost function along the second direction increases monotonically with distance (assuming all other parameters are the same).

[0127] In most embodiments, the set of candidate depth values ​​for a second direction from a first pixel does not include a second candidate depth value in which the pixel set of at least one pixel along the second direction does not include a candidate depth value in the set of candidate depth values ​​whose cost function does not exceed the cost function for the second candidate depth value, and the distance from the first pixel to the second candidate depth value is greater than the distance from the first pixel to the pixel set.

[0128] In practice, typically only one direction includes candidate depth values ​​that are either not included in the set of candidate depth values, or that are included but have a higher cost function than one or more pixels that are further away.

[0129] By considering that further distanced depth values / pixels are restricted to one direction (including potentially small angular intervals), many embodiments enable particularly advantageous performance. Thus, the approach can be adapted to specific characteristics of the scene that restrict considering further distanced pixels to situations where further distanced pixels are particularly likely to reflect the correct depth.

[0130] In particular, in many embodiments, the first direction corresponds to the direction of gravity in the depth map / image / scene. The inventors have recognized that favorable operation is achieved by considering pixels further away along the direction of gravity, as the likelihood of such depth values ​​being correct is greatly increased.

[0131] In particular, the inventors recognized that in many practical scenarios, an object is positioned or standing on the ground, and in such scenarios, the depth of the entire object is usually comparable to the depth of the furthest part of the object in the direction of gravity. The inventors further recognized that this usually translates the depth value for an object to the depth value in the direction of gravity in a depth map, often to a corresponding relationship in a more similar depth map, rather than to the local neighborhood depth value. For example, the head of a person standing on a flat surface will have a depth approximately the same as the depth of their feet. However, the neighborhood depth around the head is significantly different because it includes pixels corresponding to the distant background. Therefore, depth processing based solely on neighborhood depth may be less reliable and accurate for the head than for the feet. However, with the described approach, the method can include not only neighborhood depth values ​​but also more distant depth values ​​and pixels in the direction of gravity. For example, when processing the depth value for a person's head, the described approach results in one candidate depth value being the depth value from the person's feet (which more accurately reflects the true depth, especially after several iterations).

[0132] Figure 7 illustrates an example of the improvements achieved. This figure illustrates an image, a corresponding depth map after applying the corresponding process that does not consider the candidate far-far gravity direction, and finally, a depth map after applying the process that does not consider the candidate far-far gravity direction. Comparing parts 701 and 703, it can be seen that the head of the athlete on the far left has an incorrect depth value in the first example, but has an correct depth value in the second example.

[0133] In some embodiments, the direction of gravity in the depth map may be predetermined, and the first direction may be predetermined. In particular, for many typical depth maps and images, horizontal capture is performed (or post-processing is performed to align the horizontal direction of the image with that of the scene), and the direction may be predetermined as the vertical direction of the depth map / image. In particular, the direction may be the up and down direction in the image.

[0134] In some embodiments, the processor 203 is configured to determine a first direction as the gravity direction in the depth map, where the gravity direction in the depth map corresponds to the direction of gravity in the scene represented by the depth map.

[0135] In some embodiments, such a decision may be based on an evaluation of an input, such as from a level indicator of a camera capturing an image in which the depth map is updated. For example, if data is received indicating that the stereoscopic camera is at an angle of, for example, 30° with respect to the horizontal, the first direction is determined as the direction offset by 30° with respect to the vertical in the depth map and image.

[0136] In many embodiments, the gravity direction in the depth map may be based on the analysis of the depth map and / or the image. For example, the gravity direction is chosen inversely to a vector pointing from the center of the image to the pixel position in the average weighted image, with a pixel-by-pixel weighting proportional to the amount of blue in the pixel's color. This is a simple way of determining the gravity direction using the blue sky in a photograph. An approach is known in which pairs of stereoscopic images (or multiview images) are rectified so that the so-called epipolar lines are horizontal. It may always be assumed that gravity is orthogonal to the epipolar lines.

[0137] In some embodiments, the cost function may include consideration of a depth model for at least a portion of the scene. In such embodiments, the processor 203 is configured to evaluate the depth model to determine the expected depth model. The cost function then depends on the difference between the depth value and the model depth value determined from the depth model.

[0138] The depth model may be a model that imposes depth constraints on at least some depth values ​​of a depth map, and these depth constraints may be absolute or relative. For example, the depth model may be a 3D model of a scene object that, when projected onto a depth map, gives corresponding depth relationships between depth values ​​for pixels corresponding to image objects. In this way, the depth relationship can be implied if the absolute depth of the scene object is unknown, but the type of object represented by the scene object is known. As another example, the depth model may be a parallax model estimated for a static ground plane and / or a static background, or a parallax model for, for example, a set of dynamically moving planar or cylindrical objects (e.g., representing athletes on a stadium).

[0139] The cost function thus evaluates the model to determine the expected depth values ​​for candidate depth values ​​using the depth model. Next, it compares the actual depth values ​​to the expected values ​​and determines a cost contribution that increases monotonically as the difference increases (for at least some depth values).

[0140] In some embodiments, the cost contribution may be asymmetric and therefore may differ depending on whether the depth value is higher or lower than expected. For example, different functions may be applied so that depth values ​​further away from the model result in a significantly higher cost contribution than depth values ​​closer to the model. This biases updates toward depths further ahead of the model. Such an approach is particularly advantageous when the model is a background model that provides background depth indications / estimations. In such cases, the cost contribution makes it less likely that the depth map will be updated to reflect depth, resulting in perceptually significant artifacts / errors where objects are rendered further behind the depth background.

[0141] In fact, in some cases, the cost contribution for depth values ​​that represent a depth higher than the background depth is high, so the likelihood of this depth value being selected is very low. For example, the cost contribution from model comparisons in such cases is set to a very high value (even infinity in principle).

[0142] For example, the cost contribution to model evaluation is given by the following:

number

[0143] In some embodiments, only candidate depth values ​​from the depth map itself are considered for the set of candidate depth values. However, in some embodiments, the set of candidate depth values ​​is generated to include other candidate depth values. As previously mentioned, the set of candidate depth values ​​includes one or more depth values ​​from other depth maps in a temporal sequence of depth maps that includes the depth map. Specifically, it includes depth values ​​from depth maps of previous and / or subsequent frames in the video sequence. In some embodiments, the set of candidate depth values ​​may include depth values ​​determined according to an offset of the depth value for a first pixel. For example, the set of candidate depth values ​​may include depth values ​​generated by adding a predetermined offset to the depth value for the current pixel, and / or depth values ​​generated by subtracting a predetermined offset from the depth value for the current pixel.

[0144] Including different types of depth values ​​provides improved performance in many applications and scenarios, specifically allowing for more significant updates with fewer constraints. Furthermore, different types of depth values ​​may be included by designing a cost function that takes into account different types of potential possibilities for indicating the correct depth for the current pixel. Specifically, the cost function may be dependent on the type of depth value, and therefore the type of depth value to which the cost function applies may be taken into account. More specifically, the cost function may take into account whether the depth value is a depth value of a depth map, a depth value of a depth map that is closer than a distance threshold (e.g., in the immediate vicinity), a depth value of a depth map that is further away than a distance threshold (e.g., a pixel further away along the direction of gravity), a depth value from another depth map in a temporal sequence of depth maps containing the depth map, a depth value independent of the scene represented by the depth map, or a depth value determined in relation to an offset of the depth value for a first pixel. Naturally, in many embodiments, only a subset of these will be considered.

[0145] As an example, the following cost function may be evaluated for each candidate depth value in the set of candidate depth values. Ctotal = w1C match + w2C smoothness + w3C model + w4C candidate Here, C match is a cost that depends on the matching error between the current view and one or more other views, and C smoothness weights both spatial smoothness and penalizes depth transitions within regions having a constant color intensity. Several different approaches for determining such cost values / contributions are known to those skilled in the art, and for the sake of brevity, these will not be described further. The cost component C model is the described model cost contribution and may reflect the deviation of the disparity from a prior known or estimated disparity model. C candidate may introduce a cost contribution that depends on the type of depth value, such as whether it is from the same depth map, e.g., a temporally adjacent depth map.

[0146] As an example, C candidate is given by the following.

Number

[0147] The cost of local neighbor candidates is usually small because such neighbors are very likely to be good predictors. The same is true for temporal neighbor candidates, although the cost is slightly higher to avoid errors for fast-moving objects. The cost of offset updates must be high to avoid introducing noise. Finally, the cost of far-away (gravitational) candidates is usually higher than that of normal local neighbor candidates because of the large spatial distance. Multiple such candidates (different values ​​for Δ) at different low positions may be used. In this case, the cost can be increased as a function of increasing Δ from the pixel being processed.

[0148] For clarification, the above description will be understood to have illustrated embodiments of the present invention with reference to different functional circuits, units, and processors. However, it will be clear that a preferred distribution of functions between different functional circuits, units, or processors can be used without impairing the present invention. For example, a function exemplified as being performed by a separate processor or controller may be performed by the same processor or controller. For this reason, references to specific functional units or circuits should be seen not as indicating a strict logical or physical structure or organization, but only as a reference to a preferred means for providing the described functionality.

[0149] The present invention can be implemented in any preferred form, including hardware, software, firmware, or any combination thereof. The present invention may optionally be implemented at least partially as computer software running on one or more data processors and / or digital signal processors. Elements and components of embodiments of the present invention may be implemented physically, functionally, and logically in any suitable manner. In fact, functionality may be implemented in a single unit, in multiple units, or as part of other functional units. Therefore, the present invention may be implemented in a single unit, or physically and functionally distributed across different units, circuits, and processors.

[0150] Although the present invention has been described in relation to several embodiments, it is not intended to be limited to any particular form described herein. Rather, the scope of the present invention is limited only by the appended claims. Furthermore, although features appear to be described in relation to the specified embodiments, those skilled in the art will recognize that various features of the described embodiments are combined by the present invention. In the claims, the term "includes" does not preclude the presence of other elements or steps.

[0151] Furthermore, steps of multiple means, elements, circuits, or methods, although individually listed, may be carried out by, for example, a single circuit, unit, or processor. Additionally, individual features may be included in different claims, but they may be advantageously combined, and their inclusion in different claims does not imply that the combination of features is feasible and / or unfavorable. Also, including a feature in one category of claims does not imply limitation to that category, but rather indicates that the feature is equally applicable to other claim categories as needed. Furthermore, the order of features in a claim does not imply a specific order in which the features must operate, and the order of individual steps in a method claim, in particular, does not imply that the steps must be performed in this order. Rather, the steps may be performed in any preferred order. Furthermore, singular references do not exclude plurals. Thus, references to "one," "first," "second," etc., do not exclude plurals. Reference symbols in a claim are provided merely as clear examples and should not be construed as limiting the scope of the claims in any way.

Claims

1. 1. A method for processing a depth map, the method comprising: receiving the depth map; For at least a first pixel of the depth map, determining a set of candidate depth values, the set of candidate depth values ​​including depth values ​​for other pixels of the depth map other than the first pixel; determining a cost value for each of the candidate depth values ​​in the set of candidate depth values ​​in response to a cost function; selecting a first depth value from the set of candidate depth values ​​as a function of the cost value for the set of candidate depth values; determining an updated depth value for the first pixel in response to the first depth value; The set of candidate depth values ​​includes a first candidate depth value along a first direction from the first pixel, a first intervening pixel set of at least one pixel along the first direction does not include a candidate depth value of the set of candidate depth values ​​for which the cost function for a candidate depth value does not exceed the cost function for the first candidate depth value, a distance from the first pixel to the first candidate depth value is greater than a distance from the first pixel to the first intervening pixel set, and the method further comprises determining the first direction as a gravity direction of the depth map, the gravity direction being a direction in the depth map that corresponds to a direction of gravity in a scene represented by the depth map.

2. 2. The method of claim 1 , wherein the cost function along the first direction has a monotonically increasing cost gradient as a function of distance from the first pixel when the distance is below a distance threshold and a decreasing cost gradient as a function of distance from the first pixel when at least one distance from the first pixel is above a threshold.

3. The method of claim 1 , wherein the first set of intervening pixels is a set of pixels whose depth values ​​are not included in the set of candidate depth values.

4. 4. The method of claim 1 , wherein the cost function includes a cost contribution that is dependent on the difference between image values ​​of multi-view images for pixels offset by a disparity that matches the candidate depth value to which the cost function is applied.

5. The method of claim 1 , wherein the first direction is a vertical direction in the depth map.

6. 6. The method according to claim 1, further comprising the step of determining a depth model for at least a part of the scene represented by the depth map, and wherein the cost function for a depth value is dependent on a difference between the depth value and a model depth value determined from the depth model.

7. The method of claim 6 , wherein the cost function is asymmetric with respect to whether the depth value is above or below the model depth value.

8. The method of claim 6 or 7, wherein the depth model is a background model for the scene.

9. including candidate depth values ​​in the set of candidate depth values ​​that are not from the depth map; a depth value from another depth map of a temporal sequence of depth maps, the temporal sequence including the depth map; a scene-independent depth value represented by the depth map; and a depth value determined according to a depth value offset for the first pixel; The method according to claim 1 , further comprising the step of:

10. The cost function for the depth value depends on the type of the depth value, the type being: the depth values ​​of the depth map; depth values ​​in the depth map that are closer than a distance threshold; depth values ​​in the depth map that are farther away than a distance threshold; a depth value from another depth map of a temporal sequence of depth maps that includes the depth map; a depth value having a scene independent depth value offset with respect to a depth value of the first depth value; a scene-independent depth value represented by the depth map; and a depth value determined according to a depth value offset for the first pixel; 10. The method according to claim 1, wherein the method is one of a group of types including at least one of:

11. The method of claim 1 , further comprising processing a plurality of pixels of the depth map by iteratively selecting a new first pixel from the plurality of pixels, performing each step for each new first pixel.

12. 2. The method of claim 1, wherein the set of candidate depth values ​​for a second direction from the first pixel does not include a second candidate depth value for which a pixel set of at least one pixel along the second direction for a second candidate depth value does not include the candidate depth value in the set of candidate depth values ​​for which a cost function for the candidate depth value does not exceed the cost function for the second candidate depth value, and a distance from the first pixel to the second candidate depth value is greater than a distance from the first pixel to the pixel set.

13. 1. An apparatus for processing a depth map, comprising: a receiver for receiving a depth map; a processor for processing the depth map, the processing comprising: For at least a first pixel of the depth map, determining a set of candidate depth values, the set of candidate depth values ​​including depth values ​​for other pixels of the depth map other than the first pixel; determining a cost value for each of the candidate depth values ​​in the set of candidate depth values ​​in response to a cost function; selecting a first depth value from the set of candidate depth values ​​as a function of the cost value for the set of candidate depth values; determining an updated depth value for the first pixel as a function of the first depth value; The set of candidate depth values ​​includes a first candidate depth value along a first direction from the first pixel, a first intervening pixel set of at least one pixel along the first direction does not include a candidate depth value in the set of candidate depth values ​​for which the cost function for the candidate depth value does not exceed the cost function for the first candidate depth value, a distance from the first pixel to the first candidate depth value is greater than a distance from the first pixel to the first intervening pixel set, and the process further includes determining the first direction as a gravity direction of the depth map, the gravity direction being a direction in the depth map that matches a direction of gravity in a scene represented by the depth map.

14. A computer program comprising computer program code means adapted to perform all the steps of the apparatus according to claim 13 when said computer program is executed on a computer.

Images