Generation of depth maps for images

Abstract

An apparatus generates a depth map for capture locations of an image and a depth map from at least two depth sensing locations. Some depth image pixels are designated as uncertain. A view shift processor (305) applies a view shift from the depth sensing position to the capture position to the depth map and designates pixels of the view shifted depth map as uncertain if it is not determined that the pixels are shifted to the pixel position. A combiner (307) generates a combined depth map for the capture position by combining the view shifted depth maps, and designates pixels of the combined depth map as depth-uncertain pixels if any pixels used in the combination are designated as uncertain pixels. A depth map generator (309) generates an output depth map by determining depth values for uncertain pixels based on the image values or determined pixels of the combined depth map.

Description

Technical Field

[0001] The present invention relates to the generation of depth maps, and in particular, but not exclusively, to the generation of view synthesis / shift enabled depth maps based on scene capture images. Background Art

[0002] In recent years, with the continuous development and introduction of new services and ways to utilize and consume video, the types and scope of image applications and video applications have increased significantly.

[0003] For example, one increasingly popular service is to provide image sequences in a way that enables the viewer to actively and dynamically interact with the system to change the parameters of the presentation. In many applications, a very attractive feature is the ability to change the viewer's effective viewing position and viewing direction, for example allowing the viewer to move around and look around the scene being presented.

[0004] An example of a video service or application that has been proposed is immersive video, in which video is played back on, for example, a VR head mounted device to provide a three-dimensional experience. For immersive video, the viewer can look and move freely in the presented scene, so that it can feel like watching from different viewpoints. However, in many typical approaches, the amount of movement is limited, for example to a relatively small area around a nominal viewpoint, which typically corresponds to the viewpoint from which video capture of the scene has been performed. In such applications, three-dimensional scene information is often provided, which allows high-quality view image synthesis for viewpoints that are relatively close to (one or more) reference viewpoints, but if the viewpoint deviates too much from the reference viewpoint, such view synthesis will degrade.

[0005] Immersive video is often also referred to as 6 degrees of freedom (6DoF) or 3DoF+ video.MPEG Immersive Video (MIV) is an emerging standard where metadata is used on top of existing video codecs to enable and standardize immersive video.

[0006] In many applications, the capture of a real-world scene is based on multiple spatially distinct cameras, for example, cameras arranged in a row. This provides images of the scene from different viewing positions. In addition, the captured images may be provided together with a depth map, which provides information about the distance from the camera to objects of the scene being captured. For example, immersive video data may be provided in the form of multiple views, possibly accompanied by a depth data (MVD) representation of the scene.

[0007] In some applications, depth information can be estimated based on information in the captured images (e.g., by estimating disparity values ​​for corresponding objects in different images). However, because such estimates are often suboptimal and can lead to inaccuracies and errors, many capture systems include dedicated depth sensors that can capture the distance to the object. For example, indirect time-of-flight (iToF)-based depth sensors have become popular, and many cameras that combine optical sensors and depth sensors have been developed. In many capture systems, multiple such cameras can be used to provide a set of images and depths for different viewing postures.

[0008] However, while including one or more depth sensors into the capture arrangement tends to increase robustness and more reliable depth information, depth sensors may also suffer from errors or inaccuracies. Most notably, accurate depth measurements near depth transitions (steps) are often difficult due to the blurred / mixed signals received by the depth sensor. It should be noted that similar problems are known to exist in stereo or multi-view depth estimation based on image matching (correlation) because pixels cannot be matched due to occlusion.

[0009] Another problem is that due to practical implementation constraints, depth sensors and optical sensors are usually physically offset relative to each other. As a result, the measured depth information will be used for a pose that is offset relative to the captured image, so the image and depth map will not be perfectly aligned. This can lead to errors and inaccuracies when, for example, performing view shifting and rendering based on the depth map.

[0010] Therefore, improved methods would be advantageous. In particular, methods that allow improved operation, increased flexibility, reduced complexity, facilitated implementation, improved synthesized image quality, improved rendering, improved depth information, improved consistency between depth maps and associated images, improved and / or facilitated view synthesis for different viewing postures, reduced data requirements and / or improved performance and / or operation would be advantageous. Summary of the invention

[0011] Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

[0012] According to one aspect of the present invention, there is provided an apparatus for generating a depth map for an image of a view representing a scene, the apparatus comprising: a position receiver arranged to receive a capture position for the image, the capture position being a position at which the image is captured; a depth receiver arranged to receive a first depth map providing depth values ​​from a first position and a second depth map providing depth values ​​from a second position, the first depth map including at least some pixels designated as depth uncertain pixels, and the second depth map including at least some pixels designated as depth uncertain pixels; a view shift processor arranged to: perform a first view shift of the first depth map from the first position to the capture position to generate a first view shifted depth map, and perform a second view shift of the second depth map from the second position to the capture position to generate a second view shifted depth map, the view shift processor being further arranged to designate the following pixels as uncertain: a first view shifted depth map to which the shift does not project depth values ​​of pixels not designated as uncertain pixels in the first depth map, and a second view shifted depth map to which the second view shift does not project depth values ​​of pixels not designated as uncertain pixels in the second depth map; a combiner arranged to generate a combined depth map for the capture location by combining depth values ​​of pixels at same positions of the first view shifted depth map and the second view shifted depth map, the combiner being arranged to: designate pixels of the combined depth map as depth uncertain pixels if any pixels at same positions are designated as uncertain pixels; and a depth map generator arranged to generate an output depth map for the capture location by: using at least one of image values ​​of the image and depth values ​​of pixels not designated as uncertain in the combined depth map to determine depth values ​​designated for pixels designated as uncertain pixels in the combined depth map.

[0013] The method may allow for the generation of an improved depth map for an image based on depth maps from depth sensors at other locations. The method may allow for improved image synthesis for different viewing postures. In many embodiments, depth data may be generated that allows for efficient and high quality view shifting and synthesis.

[0014] The terms "pose" or "placement" are commonly used in the art to refer to position and / or orientation.

[0015] The method may allow efficient combination of depth information from different depth sensors and depth maps. In particular, the method may prevent artifacts, errors and inaccuracies caused by using wrong or inaccurate depth values ​​in many scenarios. It may provide improved consistency in many scenarios.

[0016] This approach may allow for efficient and convenient implementation, and may provide reduced computing resource usage and complexity in many scenarios.

[0017] The position may be a scene position. The position may be referenced to a scene coordinate system. Pixels at the same position in different depth maps (representing depth from the same position) are pixels that represent the same viewing direction from a given position.

[0018] In many cases, uncertain pixels may be referred to as unknown, unreliable, invalid, and / or untrustworthy pixels.

[0019] In many embodiments, the apparatus may comprise a view synthesizer arranged to generate a view image for a viewing pose having a different location than the capture location from the image and the output depth map.

[0020] According to an optional feature of the invention, an angle between a direction from the capture position to the first position and a direction from the capture position to the second position is not less than 90°.

[0021] This may provide particularly advantageous operation and / or implementation in many embodiments.In many embodiments, the angle may be no less than 100°, 120°, 140° or 160°.

[0022] In some embodiments, the angle between a line connecting the capture position to the first position and a line connecting the capture position to the second position is no less than 90°, 100°, 120°, 140°, 160°.

[0023] In some embodiments, the angle between a line passing through the capture position and the first position and a line passing through the capture position and the second position is no less than 90°, 100°, 120°, 140°, 160°.

[0024] According to an optional feature of the invention, the first position, the second position and the capture position are arranged in a linear configuration, wherein the capture position is located between the first position and the second position.

[0025] This may provide particularly advantageous operation and / or implementation in many embodiments.

[0026] According to an optional feature of the invention, the first position, the second position and the capture position are arranged such that, for the first view shift and the second view shift, movement directions of pixels at the same depth are opposite directions.

[0027] This may provide particularly advantageous operation and / or implementation in many embodiments.

[0028] In some embodiments, the first position, the second position and the capture position are arranged such that the directions of movement of image coordinates for a given depth are in opposite directions for the first view shift and the second view shift.

[0029] According to an optional feature of the invention, the combiner is arranged to determine a depth value for a given pixel in the combined depth map as a weighted combination of a first depth value for a pixel that is not designated as a depth uncertain pixel in the first view shift map and has the same position as the given pixel, and a second depth value for a pixel that is not designated as a depth uncertain pixel in the second view shift map and has the same position as the given pixel.

[0030] This may provide particularly advantageous operation and / or implementation in many embodiments. It may generally provide suppression of deep noise or errors.

[0031] In some embodiments, if the first depth value is for a pixel in the first depth map that is not designated as a depth-uncertain pixel, and the second depth value is for a pixel in the second depth map that is not designated as a depth-uncertain pixel, the combiner may be arranged to determine the depth value for a given view direction in the combined depth map as a weighted combination of the first depth value for the given view direction in the first view shift map and the second depth value for the given view direction in the second view shift map.

[0032] According to an optional feature of the invention, the depth map generator is arranged to generate depth values ​​for pixels designated as uncertain in the combined depth map by estimating from depth values ​​of pixels in the combined depth map that are not designated as uncertain.

[0033] This may provide particularly advantageous operation and / or implementation in many embodiments.

[0034] According to one aspect of the present invention, an image capture system is provided, which includes the device described previously and also includes: a first image camera, which is located at the capture position and is arranged to provide the image to the image receiver; a first depth sensor, which is located at the first position and is arranged to provide depth data for the first depth map to the depth receiver; and a second depth sensor, which is located at the second position and is arranged to provide depth data for the second depth map to the depth receiver.

[0035] The method may provide an image capture system that may generate particularly accurate and / or reliable 3D information including both an image and depth information.

[0036] According to an optional feature of the present invention, the capture system further comprises: at least one additional image camera located at an additional capture position, the angle between the direction from the additional capture position to the first position and the direction from the additional capture position to the second position being not less than 90°.

[0037] This may provide particularly advantageous operation and / or implementation in many embodiments.

[0038] The method described with reference to the first camera can also be applied to each of the at least one further image camera.

[0039] In many embodiments, the angle may be no less than 100°, 120°, 140°, or 160°.

[0040] According to an optional feature of the present invention, the capture system further comprises a plurality of image cameras including the first image camera, and a plurality of depth sensors including the first depth sensor and the second depth sensor, the plurality of depth sensors being arranged in a configuration such that: for all image cameras of the plurality of image cameras, two of the plurality of depth sensors are arranged such that the angle between the position of the image camera and the direction of the two depth sensors is not less than 120°. This may provide particularly advantageous operation and / or implementation in many embodiments. In many embodiments, the angle may be not less than 140° or 160°.

[0041] According to an optional feature of the invention, the plurality of image cameras and the plurality of depth sensors are arranged in a linear configuration.

[0042] This may provide particularly advantageous operation and / or implementation in many embodiments.

[0043] According to an optional feature of the invention, at least one depth sensor of the plurality of depth sensors is located between each pair of adjacent image cameras of the plurality of image cameras.

[0044] This may provide particularly advantageous operation and / or implementation in many embodiments.

[0045] According to an optional feature of the invention, the plurality of image cameras and the plurality of depth sensors are arranged in a two-dimensional configuration.

[0046] This may provide particularly advantageous operation and / or implementation in many embodiments.

[0047] According to an optional feature of the invention, a number of depth sensors in the plurality of depth sensors exceeds a number of image cameras in the plurality of image cameras.

[0048] This may provide particularly advantageous operation and / or implementation in many embodiments.

[0049] According to an optional feature of the invention, a number of image cameras in the plurality of image cameras exceeds a number of depth sensors in the plurality of depth sensors.

[0050] This may provide particularly advantageous operation and / or implementation in many embodiments.

[0051] According to one aspect of the present invention, a method for generating a depth map for an image of a view representing a scene is provided, the method comprising: receiving a capture position for the image, the capture position being a position at which the image is captured; receiving a first depth map providing depth values ​​from a first position and a second depth map providing depth values ​​from a second position, the first depth map including at least some pixels designated as depth uncertain pixels, and the second depth map including at least some pixels designated as depth uncertain pixels; performing a first view shift of the first depth map from the first position to the capture position to generate a first view shifted depth map, performing a second view shift of the second depth map from the second position to the capture position to generate a second view shifted depth map, designating the following pixels as uncertain: the first view shift does not designate the following pixels as uncertain a pixel in the first view shift depth map to which depth values ​​of pixels not designated as uncertain pixels in the first depth map are projected, and a pixel in the second view shift depth map to which the second view shift does not project depth values ​​of pixels not designated as uncertain pixels in the second depth map; generating a combined depth map for the capture location by combining depth values ​​of pixels at the same position of the first view shift depth map and the second view shift depth map, the combiner being arranged to: designate pixels of the combined depth map as depth uncertain pixels if any pixels at the same position are designated as uncertain pixels; and generating an output depth map for the capture location by: determining depth values ​​designated as uncertain depth values ​​in the combined depth map using at least one of an image value of the image and a depth value not designated as uncertain.

[0052] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0054] Figure 1 An example of elements of an image distribution system is illustrated;

[0055] Figure 2An example of an image capture scenario is illustrated;

[0056] Figure 3 illustrates examples of elements of an apparatus according to some embodiments of the present invention;

[0057] Figure 4 An example of a scene capture situation is illustrated;

[0058] Figure 5 An example of a scene capture situation is illustrated;

[0059] Figure 6 illustrates an example of a depth map for a scene capture scenario;

[0060] Figure 7 An example of processing of a depth map for a scene capture context is illustrated;

[0061] Figure 8 illustrates examples of elements of depth and image capture scenarios;

[0062] Fig. 9 illustrates examples of elements of depth and image capture scenarios;

[0063] Fig.10 illustrates examples of elements of depth and image capture scenarios;

[0064] Fig.11 illustrates examples of elements of depth and image capture scenarios;

[0065] Fig.12 illustrates examples of elements of depth and image capture scenarios; and

[0066] Fig.13 Some elements of a possible arrangement of a processor for implementing elements of an apparatus according to some embodiments of the invention are illustrated. DETAILED DESCRIPTION

[0067] In some applications and services, three-dimensional image and video capture, distribution and presentation are becoming more and more popular and welcomed. A specific method is called immersive video, and generally includes providing a view of a real-world scene (and often a real-time event) that allows small viewer movements (e.g., relatively small head movements and rotations). For example, a real-time video broadcast such as a sporting event (which allows the view to be generated based on a local client after a small head movement of the viewer) can provide the user with the impression of sitting in the stands to watch the sporting event. The user can, for example, look around and will have a natural experience similar to that of the viewer appearing in the stands. In recent years, display devices with position tracking and 3D interaction support applications based on 3D capture of real-world scenes have become increasingly popular. Such display devices are very suitable for immersive video applications that provide an enhanced three-dimensional user experience.

[0068] The following description will focus on immersive video applications, but it will be appreciated that the principles and concepts described may be used in many other applications and embodiments, including embodiments that capture a single image.

[0069] In many approaches, immersive video can be provided locally to a viewer by, for example, a standalone device that does not use or even have access to any remote video server. However, in other applications, immersive applications can be based on data received from a remote or central server. For example, video data can be provided to a video rendering device from a remote central server, and the video data can be processed locally to generate a desired immersive video experience.

[0070] Figure 1 An example of an immersive video system is illustrated, wherein a video presentation client 101 is in contact with a remote immersive video server 103, for example via a network 105 (eg, the Internet). The server 103 may be arranged to support a potentially large number of client devices 101 simultaneously.

[0071] The server 103 may support an immersive video experience, for example, by transmitting three-dimensional video data describing a real-world scene. The data may specifically describe the visual features and geometric properties of the scene, which is generated by real-time capture of the real world by a set of (possibly 3D) cameras.

[0072] In order to provide such services for real-world scenes, scenes are usually captured from different locations and different camera capture poses are used. As a result, the relevance and importance of multi-camera capture and, for example, 6DoF (6 degrees of freedom) processing are increasing rapidly. Applications include live concerts, live sports, and telepresence. The freedom to choose your own viewpoint improves the sense of presence compared to conventional videos, thereby enriching these applications. In addition, immersive scenarios can be envisioned in which viewers can navigate and interact with the live captured scenes. For broadcast applications, this may require real-time view synthesis at the client device. View synthesis introduces errors, and these errors depend on the implementation details of the algorithm.

[0073] To provide a suitable scene representation, it is common to capture images / videos with depth Figure 1 In some embodiments, the depth map may be provided together with a depth map indicating the depth of / distance to an object represented by pixels of the image. In many embodiments, the depth map linked to the image may include a depth value and depth pixel for each pixel of the image, so the depth map(s) and the image(s) may have the same resolution. In other embodiments, the depth map may have a lower resolution, and the depth map may provide some depth values ​​that are common to multiple image pixels.

[0074] In some capture systems, a depth map can be estimated and derived from the captured image. However, improved depth information can often be achieved by using a dedicated depth sensor. For example, a depth sensor has been developed that transmits a signal and estimates the time delay of the echo arriving from different directions. Then, a depth map can be generated using depth values ​​reflecting this measured flight time in different directions. A 3D camera has been developed that includes an optical camera and one or more depth sensors. In a suitable capture configuration, multiple such cameras are often used together. For example, multiple 3D cameras can be arranged in a row to provide an MVD representation of images for different capture poses together with the associated depth camera.

[0075] However, while this approach can provide excellent results in many scenarios, it does not operate optimally in all situations. In particular, in some cases, it may be difficult for the depth sensor to accurately determine the time of flight and therefore the distance to an object. In general, near depth transitions (e.g., at the edge of an object), depth estimation is often difficult or inaccurate.

[0076] It has been found that in many applications, it is advantageous to process depth values ​​that are not considered sufficiently reliable (rather than depth values ​​that are considered reliable). Therefore, in many capture systems, the generated depth map can be provided together with data indicating whether the pixel / depth value is considered to be an uncertain pixel or a determined pixel. For example, a binary map can be provided, which indicates for each pixel and depth value of the depth map whether the corresponding pixel of the depth map is a depth uncertain pixel, that is, whether the depth value provided by the depth map for the pixel is designated as an uncertain pixel.

[0077] It should be understood that any suitable method or algorithm for determining which depths and depth pixels (pixels of the depth map) are designated as uncertain pixels and which depths and depth pixels (pixels of the depth map) are not designated as uncertain pixels can be used, and different methods can be used in different embodiments. Typically, the depth sensor may include a built-in function to generate a reliability estimate or confidence level for each depth value, and if the reliability estimate or confidence level indicates that the reliability is below a given threshold, the corresponding pixel can be designated as uncertain. For example, if the echo in a given direction extends over a relatively long time interval, or if, for example, more than one significant peak is detected, the depth sensor may deem the reliability low and may designate the pixel as any uncertain pixel.

[0078] It should be understood that an uncertain pixel may correspond to an unreliable / untrusted pixel or an unknown depth pixel, and in many embodiments, the term may be replaced by any such term. Depth pixels or values ​​that are not designated as uncertain / unreliable / unknown depth pixels may be referred to as determined / reliable / known depths.

[0079] Thus, the output from the depth sensor may be a depth map comprising a plurality of pixels of depth values ​​and pixels that are considered to be uncertain (particularly unknown). Processing of such a depth map may include addressing such pixels and using other means to generate depth values ​​for uncertain and unknown pixels. For example, depth values ​​for uncertain pixels may be determined by disparity estimates between different corresponding pixels in different images for different views. Alternatively or additionally, depth values ​​for depth uncertain pixels may be determined by extrapolation of depth pixels that are considered to be certain or known depth pixels.

[0080] Another problem for many capture arrangements is that the depth sensor is not fully co-located with the optical camera sensor, i.e. the depth data of the depth map is provided for a different location than the capture location of the image, i.e. the depth data is for a different location than the location of the image. While the capture device may be designed to try to reduce this positional offset as much as possible, this may still lead to inaccuracies and errors when, for example, performing view shifting and image synthesis based on the image and depth information.

[0081] In the following, a method and apparatus are described that can generate improved depth maps for many depth maps in many scenarios. The method can specifically consider depth values ​​for at least two depth maps from different locations to generate a depth map for the capture location by warping the depth maps to the capture location and using a specific method for combining the shifted depth maps, wherein the combination includes specific consideration of whether a pixel is designated as certain or uncertain.

[0082] This method will refer to Figure 2 1. The capture device shown in FIG. 2 is described. In this example, the optical camera is positioned at the image capture position 201, the first depth sensor is positioned at the first depth sensing position 203, and the second depth sensor is positioned at the second depth sensing position 205. The three positions are arranged in a row, wherein the first depth sensing position 203 and the second depth sensing position 205 are on both sides of the capture position 201, respectively.

[0083] In some embodiments, the optical camera and each depth sensor may, for example, be separate devices / items. In other embodiments, the camera and the depth sensor may, for example, be implemented in the same device (e.g., they may all be implemented in a single 3D camera device). In other embodiments, the depth sensor may, for example, be part of a different device or camera. For example, multiple 3D cameras may each include an optical camera and a depth sensor arranged adjacent to each other. Placing these cameras adjacent to each other may produce Figure 2 The arrangement shown in FIG. 1 is a diagram in which the two depth sensors are the depth sensors of adjacent 3D cameras. Therefore, in such a scenario, each camera can provide an image for the capture position 201 and a depth map for the depth sensing positions 201 and 203. Using the depth maps from the two devices, depth maps for the first depth sensing position 203 and the second depth sensing position 205 can be provided accordingly.

[0084] Figure 3 Components of an apparatus for generating a depth map for a capture location are illustrated. Figure 2 The operation of the device will be described with reference to an arrangement of FIG. 1 , but it will be appreciated that the principles and methods described may be used with many different capture arrangements.

[0085] Figure 3 The apparatus comprises a position receiver 301, which is arranged to receive a capture position for an image, wherein the capture position is the (scene) position of the capture of the image. The position receiver 301 may be specifically coupled to one or more cameras and may receive a position indication from these cameras (e.g., the camera may include GPS functionality for determining the position and may provide it to the position receiver 301). In other embodiments, the position receiver 301 may receive the capture position from other units (including from a separate device, from a dedicated position measurement unit, or for example by direct user input of the capture position).

[0086] In a specific example, the position receiver 301 is coupled to Figure 2 And it receives data describing the capture position 201 directly from the camera.

[0087] The apparatus further comprises a depth receiver 303, which is arranged to receive a depth map representing the scene. Specifically, the depth receiver 303 may be arranged to receive the depth map from a depth sensor of a capture configuration. In addition, the depth receiver 303 may receive position information indicating a location where the depth map has been captured. Specifically, the depth receiver 303 may receive a first depth map providing depth values ​​from a first location (also referred to as a first depth sensing location) (e.g., specifically, from a depth sensor of a scene). Figure 2 The depth receiver 303 may also receive a second depth map (e.g., specifically, a depth map from a sensor at the first depth sensing location 203) providing depth values ​​from a second location (also referred to as a second depth sensing location). Figure 2 The following description will focus on embodiments in which depth maps are received from only two locations, but it should be understood that in other embodiments, depth maps from more locations may be received.

[0088] The depth receiver 303 also receives data indicating whether pixels of the first depth map and the second depth map are designated as uncertain pixels or as certain pixels. The indication of whether a pixel is considered uncertain reflects the degree of certainty / reliability / confidence / validity (or even existence) that the depth value of the pixel is correct / an accurate indication of the depth of the object represented by the pixel.

[0089] In some embodiments, the depth receiver 303 may, for example, receive data including a list or other indication of all pixels in the first depth map and / or the second depth map that are considered uncertain, or an indication of all pixels that are considered certain. In other embodiments, the depth receiver 303 may, for example, receive a confidence map for each depth map, wherein, for example, each pixel of the confidence map indicates a confidence level for the depth value of the pixel in the depth map. In some embodiments, an indication of whether a pixel is certain may be included in the depth map itself. For example, a depth value may be assigned to each pixel, but wherein, for example, an odd value indicates a certain value, while an even value indicates an uncertain pixel. As another example, an uncertain value may be indicated by a specific depth value, for example, a zero depth value is reserved to indicate that the depth value is uncertain.

[0090] It should also be understood that, although the indication of uncertain / determined pixels can be provided by binary values, in other embodiments, they can also be provided by non-binary values. For example, the indication of the confidence value can be indicated by a range including multiple possible values. For example, the confidence value can be given by an 8-bit value. In this case, if the confidence value meets the uncertainty criterion, the pixel can be designated as an uncertain value, otherwise it is a determined value. Specifically, if the confidence value does not exceed a threshold value, the pixel is designated as uncertain, otherwise it is designated as determined.

[0091] It will be appreciated that any suitable methods and criteria for designating depth values / pixels of depth values ​​as determinate / uncertain may be used, depending on the particular preferences and requirements of individual embodiments.

[0092] The depth receiver 303 and the position receiver 301 are coupled to a view shift processor 305, which is arranged to perform a view shift of the depth map. The view shift of the depth map shifts the depth map from representing the depth from the depth sensing position to representing the depth from the capture position of the image. Therefore, the view shift processor 305 is arranged to view shift the first depth map from the first depth sensing position 203 to the capture position 201, and view shift the second depth map from the second depth sensing position 205 to the capture position 201.

[0093] Therefore, the view shift processor 305 is arranged to generate a view shifted depth map for a capture position from the depth maps of the positions of the different depth sensors. Hence, after the view shift by the view shift processor 305, all view shifted depth maps represent depth from the same position, and indeed all are from the same position of the captured image.

[0094] It should be understood that many different algorithms for view shifting images and depth maps are known to those skilled in the art, and for the sake of brevity and clarity, these algorithms will not be described herein. It should be understood that any suitable view shifting methods and algorithms may be used without departing from the invention (and it should be noted that view shifting of images can generally be directly applied to view shifting of depth maps).

[0095] The view shift processor (305) is further arranged to designate pixels of the view shifted depth map as indeterminate or determinate depending on the designation result of the view shifted pixels in the original depth map.

[0096] The view shift processor 305 is specifically arranged to designate pixels whose depth values ​​of pixels in the original depth map which are not designated as uncertain pixels and to which the view shift is not projected as uncertain pixels of the view shifted depth map(s).

[0097] The view shift moves the position of a pixel in the original depth map to a modified position in the view shifted depth map, where this change depends on the view shift. As such, each pixel in the view shifted depth map can receive contributions from zero, one, or more pixels in the original depth map. If a given pixel in the view shifted depth map does not receive any contribution from pixels that are considered to be certain pixels in the original depth map, the pixel is designated as an uncertain pixel. If the pixel does not receive contributions from any pixels, it is designated as an uncertain pixel. If the pixel receives contributions from one or more original pixels, but these original pixels are all designated as uncertain pixels, the pixel is also designated as an uncertain pixel. This contribution can, for example, come from a depth value for an uncertain pixel (e.g., a default value, or an estimated value that is however considered unreliable). However, in many embodiments, no depth value can be considered to be associated with an uncertain pixel, so view shifting is not performed for the uncertain pixel. Therefore, in this scenario, the uncertain pixel does not provide a contribution to another pixel due to the view shift.

[0098] However, if a pixel receives a contribution from one or more pixels designated as deterministic in the original depth map, then the pixel of the view-shifted depth map is designated as a deterministic pixel. In the event that more than one deterministic pixel is shifted to a given pixel due to view shifting, the depth value may generally be selected as the most forward one.

[0099] Therefore, in many embodiments, if no determined pixel is view-shifted projected to a given pixel, the pixel may be determined to be uncertain. In some embodiments, all other pixels may be designated as determined (not uncertain) pixels.

[0100] In some embodiments, if a pixel of the original depth map is projected to a pixel due to view shift, the pixel of the view shift depth map may be considered an uncertain pixel. Thus, in some embodiments, a pixel may be designated as a certain pixel if at least one certain pixel of the original depth map is projected to the pixel, and in some embodiments, subject to the requirement that no uncertain pixels are projected to the pixel. In some embodiments, all other pixels may be designated as uncertain pixels.

[0101] In the case where multiple depth values ​​are projected to a pixel due to view shifting, the depth value for the view shifted pixel may be determined according to a suitable combination criterion. In many embodiments, any depth value from an uncertain pixel may be discarded, and the combination may include only depth values ​​from determined pixels. Typically, the combination of multiple depth values ​​from determined pixels may be a selection combination, such as specifically selecting the most forward pixel. In other embodiments, other methods (including, for example, a weighted combination or a combination or selection based on a confidence level) may be used.

[0102] The view shift processor 305 is coupled to a combiner 307 which generates a combined depth map for a capture location by combining depth values ​​of co-located pixels of the view shift depth maps. Co-located pixels (from the capture location) in different depth maps are pixels that represent the same view direction from the capture location. Co-located pixels may be pixels in the view shift depth map that represent the same view direction as the capture location. For a given pixel in the combined depth map, depth values ​​for the same pixel (at the same position) in the view shift depth map are combined according to a suitable formula to generate a combined depth value for the pixel. This process may be repeated for all pixels of the combined depth map.

[0103] The combiner 307 is also arranged to designate pixels of the combined depth map as uncertain or determined, depending on the designation results of the pixels of the view shift depth map being combined. The combiner 307 is specifically arranged to designate a pixel as uncertain if any pixel at the same position from the view shift depth map is designated as uncertain. Therefore, if any pixel used in the combination to determine the combined depth value of the combined depth map is designated as an uncertain pixel, the corresponding pixel of the combined depth map is also designated as an uncertain pixel (even if one or more pixels are designated as determined pixels). Specifically, the combiner 307 may designate a pixel of the combined depth map as determined only if all pixels of the view shift depth map at the same position are designated as determined, and if any pixel among all pixels of the view shift depth map at the same position is designated as uncertain, the combined pixel will be designated as uncertain.

[0104] In many embodiments, the combiner 307 may be arranged to generate the depth value of the combined depth map as a weighted combination of the depth values ​​in the view shifted depth map. The combination of a given pixel representing a given view direction may be a weighted combination of the depth values ​​of the pixels in the view shifted depth map representing the same direction and not designated as depth uncertain.

[0105] As a specific example, each view-shifted depth map may be initialized with a value · indicating that the pixel value / depth value is uncertain. For 16-bit integer depth encoding, a value of 0 may typically be used. The depth map pixel value ·(·,·) for depth map / sensor / position i is view-shifted to capture position ·. Let ·(·,·) denote this view-shifted depth value. Note that due to the initialization, ·(·,·) will also contain uncertain pixels indicated by the reserved value ·.

[0106] If we now have two predictions coming from opposite directions, the combined depth value ·(·,·) can be determined as:

[0107]

[0108] The above formula provides two functions: (1) handling of uncertain pixels; (2) if both predictions are certain, they are averaged to suppress the remaining depth measurement noise. This weighting may be more accurate in the presence of depth noise, depth quantization, and / or camera / sensor calibration errors.

[0109] The apparatus further comprises a depth map generator 309, which is arranged to generate an output depth map for the capture location by determining depth values ​​for pixels in the combined depth map that are designated as uncertain. The depth map generator 309 can thus generate (new) depth values ​​for the uncertain pixels. Generating these new values ​​is typically based on at least one of the image values ​​of the image and the depth values ​​in the combined depth map that are not designated as uncertain.

[0110] In many embodiments, the depth map generator 309 can perform an image-based extrapolation of the determined depth value. For example, the depth value of the uncertain pixel can be determined by a weighted combination of the depth values ​​of the determined pixels within the kernel / neighborhood of the uncertain pixel. The weight of the combination can depend on the image similarity between the corresponding pixels in the image. This method can generally provide very efficient depth value generation. As an alternative, the depth of the uncertain pixel can be estimated using matching-based multi-view correspondence estimation. Determining the pixel can help to locally constrain the solution.

[0111] In many embodiments, the depth map generator 309 may be arranged to determine the depth value of the uncertain pixel based on one or more depth values ​​of the determined pixels within the region determined relative to the position of the uncertain pixel. Such a mechanism may rely on the image values ​​of the position of the uncertain pixel and the position of (one or more) determined pixels, and in particular may rely on the difference between the image values.

[0112] In many embodiments, the depth map generator 309 may be arranged to determine the depth value of the uncertain pixel as a weighted combination (specifically, in many cases a weighted sum) of the depth values ​​of a set of determined pixels (pixels that are not designated as uncertain pixels). The set of determined pixels may be pixels having positions that satisfy geometric criteria (e.g., within a given distance) in terms of the positions of the uncertain pixels. In many embodiments, the weights may depend on the image values ​​of the determined pixels, and typically also on the image values ​​of the uncertain pixels. For example, the weight of the determined pixel may be a monotonically decreasing function of the difference between the image value of the pixel position of the determined pixel and the image value of the pixel position of the uncertain pixel.

[0113] In many embodiments, the depth map generator 309 may be arranged to determine the depth values ​​of uncertain pixels by extrapolating the depth values ​​of determined pixels, which may typically be or at least include neighboring pixels.

[0114] In some embodiments, the depth map generator 309 may be arranged to determine the depth value of the uncertain pixel based solely on the depth value of one or more determined pixels. For example, in some cases, the depth map generator 309 may simply set the depth value of the uncertain pixel equal to the depth value of the nearest determined pixel, or, for example, to the average value (or other weighted sum / kernel) of a group of determined pixels having positions that satisfy a criterion in terms of the position of the uncertain pixel.

[0115] In fact, in some cases, the depth value of the uncertain pixel can even be determined by considering only the image value without considering the depth value of the determined pixel. For example, the depth map generator 309 can determine the dominant color for the determined pixel and consider that this corresponds to the background. If the difference in the image value of the uncertain pixel is less than a threshold, the depth map generator 309 can assign the nominal depth value of the background to the uncertain pixel, otherwise the depth map generator 309 can assign the nominal foreground depth value (e.g., a value corresponding to the display plane (often considered to be zero depth)) to the uncertain pixel.

[0116] The described apparatus may allow for the generation of an improved depth map for an image based on depth information provided by a (shifted) depth sensor. In many embodiments, it may provide more accurate depth values, particularly near depth transitions in the scene when viewed from a capture position. The improved depth map may also allow for improved view shifting and image synthesis based on the image and the combined depth map, thereby allowing for improved services (e.g., improved immersive video, extended reality applications, etc.).

[0117] The method is based inter alia on the recognition by the inventors that conventional methods may lead to artifacts and errors.

[0118] Figure 4 An example of a scene being captured is illustrated, the scene including a background object B and a foreground object F. The example also shows how the scene is viewed / captured from a first depth sensing position 203. In this example, an uncertain depth sensing region U is captured near a depth transition. Due to the depth transition, the depth sensor cannot provide an accurate depth measurement, so the depth values ​​near the depth transition are considered uncertain, and pixels near this region are therefore designated as uncertain.

[0119] If this first depth map is view-shifted to the capture position 201 (the position of the image camera), the foreground depth values ​​and the background depth values ​​will move differently, such as Figure 4 As indicated, the view of the depth transition and the area corresponding to the uncertain depth transition change. Specifically, two different uncertain pixel areas U appear. In addition, a gap G is introduced for which the depth value cannot be determined based on the first depth map (i.e., deocclusion occurs). In such an example, the view shift processor 305 can set the entire area of ​​uncertain and deoccluded pixels (U+G) to uncertain pixels U'.

[0120] Figure 5 The corresponding situation captured from the second depth sensing position 205 is shown. In this case, there is no gap because no de-occlusion occurs. In addition, the uncertainty region of the foreground object overlaps not only with the uncertainty region U' of the background object, but also with some of the uncertainty regions E of the background object. When considering the depth map, the corresponding situation is as follows Figure 6 As shown, Figure 6 The depth map captured at the second depth sensing position 205 and the resulting view-shifted depth map for the capture position 201 are shown respectively. The figure also illustrates the location of the real depth step 601 for reference. As shown, region 603 corresponds to the overlap of the uncertainty region of the background object B.

[0121] Therefore, warping (view shifting) the depth image to a new viewpoint may result in valid / deterministic depth pixels of the background covering new uncertain areas (of foreground objects). Therefore, special care must be taken when view shifting and combining depth maps in order not to cause undesired effects.

[0122] The above method is arranged to mitigate or even prevent this situation from occurring. It reflects the inventors' recognition that this problem specifically occurs when a foreground object occludes a background object, rather than when a foreground object de-occludes a background object. In addition, the inventors have recognized that an improved method can be achieved by positioning the depth sensor in different directions relative to the capture position and view shift and combining the view shifted images as described above. In particular, if a pixel is uncertain due to a view shift from a depth sensor in different directions, the method can designate the pixel as uncertain, and this can prevent any pixel from being designated as certain unless it is ensured that no occlusion by a background object occurs. This method typically results in a larger uncertainty zone, however, which ensures that a depth value is not designated as certain, which is caused by occlusion of a background object by a foreground object. Therefore, the combined consideration of more depth maps from different directions can reduce the risk of designating an erroneous depth value as certain due to the warping operation.

[0123] In the above method, the apparatus is arranged to designate a pixel as uncertain if there is any view-shifted depth map that has a corresponding pixel designated as uncertain. Figure 7 This is illustrated in the diagram. Figures 4 to 6 scenarios, and specifically, Figure 7 The results of specifying horizontal lines applied to the first view shifted depth map 701, the second view shifted depth map 703 and the resulting combined depth map 705, respectively, are illustrated.

[0124] The method can automatically adapt to scene properties, etc. and can generally adaptively determine uncertainty zones for pixels that reflect the actual scene context. Thus, for different capture contexts, the method can automatically adjust the size of the uncertainty zone to correspond to the actual scene properties. For example, the method can automatically adapt to depth transitions in opposite directions and does not require any information about the direction (or even the presence) of the depth step.

[0125] While in some scenarios this approach may result in larger regions of pixels being designated as uncertain rather than strictly necessary, this is generally a perfectly acceptable compromise since depth values ​​for such regions may be determined based on image-based extrapolation, multi-view matching, or the like.

[0126] In some embodiments, the apparatus may further comprise a view synthesis processor (not shown) arranged to generate a view image for a viewing posture (at a position different from the capture position) from the image and the combined depth map. It should be understood that many algorithms for such view synthesis are known to the skilled person.

[0127] The described method is particularly suitable for some capture configurations comprising one or more image cameras and two or more depth sensors. In particular, the capture arrangement can advantageously be arranged such that occlusion caused by a foreground object does not (and ideally cannot) occur simultaneously for all depth sensors.

[0128] In many embodiments, for example, Figure 2 In the example of , the image camera(s) and the depth sensor(s) may be advantageously arranged in a linear configuration, and advantageously, wherein the depth sensor(s) are located on opposite sides of the image camera(s). Thus, in many embodiments, the capture locations may be advantageously arranged in a linear configuration, wherein the capture locations are located between the depth sensing locations. This configuration may be extended to multiple image cameras and more depth sensors, such as Figure 8 As indicated (wherein the solid triangles indicate the depth sensor and the hollow triangles indicate the image camera).

[0129] In these examples, the depth sensor and the image camera are positioned along a straight line, and the direction from the capture position 201 to at least two different depth sensors is 180°. Therefore, in this example, the direction from the capture position to one depth sensing position and the direction from the capture position to another depth sensing position form an angle of 180°. However, it is not strictly required that the angle be 180°, but in some embodiments the angle may be smaller. In many embodiments, the closer the angle is to 180°, the more advantageous it is, but in many embodiments, depending on the specific requirements of the individual embodiments, the angle is not less than 90°, 100°, 120°, 140°, 160° is sufficient. Therefore, in many embodiments, the angle between the line connecting the capture position to the first position and the line connecting the capture position to the second position is not less than 90°, 100°, 120°, 140°, 160°. In many embodiments, the angle between the line passing through the capture position and the first position and the line passing through the capture position and the second position is not less than 90°, 100°, 120°, 140°, 160°.

[0130] The arrangement is typically such that at least two depth sensing positions are arranged such that: when a view shift to the capture position is performed, the direction of movement of pixels at the same depth caused by the view shift is in opposite directions. For a given pixel in the combined depth map, a contribution may be provided from a pixel to the right of the given pixel, and another contribution may be provided from a pixel to the left of the given pixel. In some embodiments, the first position, the second position, and the capture position are arranged such that: for view shifts of at least two different depth maps / for at least two depth sensing positions, the direction of movement of image coordinates for a given depth is in opposite directions.

[0131] This effect is typically achieved, for example, by construction as described previously (and as described below), and may provide particularly advantageous performance by ensuring that potentially occluded pixels are appropriately designated as uncertain.

[0132] In many embodiments, the capture arrangement may include multiple image cameras positioned at different locations. This approach may allow for improved scene capture, particularly 3D processing and view synthesis for different viewing postures (e.g., reducing the risk of portions of the scene being occluded in all captured images).

[0133] In many cases, the arrangement of at least the second image camera will be such that the second image camera / capture position also satisfies the properties of the first image camera / capture position described above. In particular, the angle between the direction / line from the second capture position to / through the first depth sensing position and the direction / line from / through the other capture position to the second position is not less than 90°, 100°, 120°, 140°, 160°. In such an arrangement, the operations described with respect to the first image camera / capture position may also be performed on the second image camera / capture position based on the same depth sensor and the same depth map. It should be understood that the method may be repeated for more cameras. For example, Fig. 9 and Fig.10 An example of a linear configuration is illustrated, where multiple cameras can be processed as described based on a common depth sensor.

[0134] In many embodiments, the following capture configuration can be adopted: multiple image cameras and multiple depth sensors are adopted in such a configuration, wherein, for all image cameras, the angle between the direction / line from the corresponding capture position to / through at least one depth sensing position and the direction / line from / through the corresponding capture position to at least another depth sensing position is not less than 90°, 100°, 120°, 140°, 160°.

[0135] In many such embodiments, the image camera and image sensor are advantageously arranged in a linear configuration. Figure 8 , Fig. 9 and Fig.10An example of a suitable configuration is shown.

[0136] In some such systems, at least one depth sensor may be advantageously positioned between each pair of adjacent image cameras of the plurality of image cameras. In particular, in some embodiments, such a configuration may include an alternating arrangement of depth sensors and image cameras. Such an approach may, for example, allow improved performance, and may, for example, allow a reduction in the area of ​​uncertain pixels, since the view shift required from the closest depth sensor in either direction may be shorter. In many embodiments, it may also allow for convenient implementation. For example, it may allow the configuration to be implemented using a variety of combination cameras including one image camera and one depth sensor. Typically, such combination cameras may be simply positioned in a straight line.

[0137] In the previous description, the capture configurations considered were primarily one-dimensional configurations of cameras and depth sensors in a linear arrangement. However, in many embodiments, the image cameras and depth sensors may be advantageously arranged in a two-dimensional configuration. In such examples, the depth sensor may not be located directly to one side of the depth camera(s), but may also be offset in different directions, for example. Fig.11 Such an example is illustrated.

[0138] The previously described properties and relationships may also generally apply to two-dimensional configurations and configurations with a relatively large number of image cameras and / or depth sensors. The method may specifically include: selecting an appropriate set (or subset) of depth sensors for each camera, and then performing the described operations on these depth sensors. In many embodiments, the subset of depth sensors for a given depth camera may include two depth sensors.

[0139] In many embodiments, (at least some) depth sensors may be arranged asymmetrically near the image cameras. For example, in many embodiments, the distance to the next closest depth sensor / depth sensor position may be no less than 2 times (or in some cases 5 times or even 10 times) the distance to the closest depth sensor / depth sensing position. For example, in many embodiments, the depth sensors may be placed very close to one, some, or often all image cameras. Such close sensors can reduce the view shift required to change from the depth sensing position to the camera position, and may tend to provide a more accurate depth map for the image. Another depth sensor may be positioned substantially further away (e.g., on the opposite side of the image camera), and may be particularly suitable for helping to determine uncertain pixels as previously described. Due to the greater distance, the depth values ​​may not be as accurate, so these may be based primarily or exclusively on the nearside depth sensors, while the remote depth sensors may be used primarily or exclusively to determine the uncertainty zone. This approach may be very advantageous in many embodiments because it may allow or facilitate the use of some depth sensors for multiple image cameras. For example, in Fig.12 In the example of , each camera may be positioned approximately at the same location as an image sensor (and indeed they may be part of the same combined camera device). The depth sensor may be primarily the source of depth values ​​for generating a depth map for the image camera. However, in addition, a single depth sensor may be positioned centrally for multiple image cameras such that it is opposite the local depth sensor. This single depth sensor may allow the described method to be used for all image cameras, thus requiring only a single central depth sensor.

[0140] In some capture arrangements, the number of depth sensors may exceed the number of image cameras. In many embodiments, this may facilitate operation and / or improve performance. For example, it may be well suited for scenarios where depth sensors may be much cheaper than image cameras. A greater number of depth sensors may generally allow for a more flexible arrangement, which may provide accurate depth information for an image. It may also generally reduce the size of the uncertain pixel region. The described method may provide very advantageous operation in scenarios where the number of depth sensors exceeds the number of image cameras.

[0141] In some capture arrangements, the number of image cameras may exceed the number of depth sensors. In many embodiments, this may facilitate operation and / or improve performance. For example, it may be well suited for scenarios where image cameras may be cheaper or more readily available than depth sensors. The described method may provide highly advantageous operation in scenarios where the number of image cameras exceeds the number of depth sensors.

[0142] Fig.131 is a block diagram illustrating an example processor 1300 according to an embodiment of the present disclosure. The processor 1300 may be used to implement one or more processors that implement an apparatus or element of an apparatus as described above. The processor 1300 may be any suitable processor type, including but not limited to a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable array (FPGA) (where the FPGA has been programmed to form a processor), a graphics processing unit (GPU), an application specific integrated circuit (ASIC) (where the ASIC has been designed to form a processor), or a combination thereof.

[0143] Processor 1300 may include one or more cores 1302. Core 1302 may include one or more arithmetic logic units (ALUs) 1304. In some embodiments, core 1302 may include a floating point logic unit (FPLU) 1306 and / or a digital signal processing unit (DSPU) 1308 in addition to or instead of ALU 1304.

[0144] The processor 1300 may include one or more registers 1312 that are communicatively coupled to the core 1302. The registers 1312 may be implemented using dedicated logic gates (e.g., flip-flops) and / or any memory technology. In some embodiments, the registers 1312 may be implemented using static memory. The registers may provide data, instructions, and addresses to the core 1302.

[0145] In some embodiments, the processor 1300 may include one or more levels of cache memory 1310 communicatively coupled to the core 1302. The cache memory 1310 may provide computer readable instructions to the core 1302 for execution. The cache memory 1310 may provide data processed by the core 1302. In some embodiments, the computer readable instructions may have been provided to the cache memory 1310 by a local memory (e.g., a local memory attached to the external bus 1316). The cache memory 1310 may be implemented with any suitable cache memory type (e.g., a metal oxide semiconductor (MOS) memory, such as a static random access memory (SRAM), a dynamic random access memory (DRAM), and / or any other suitable memory technology).

[0146] The processor 1300 may include a controller 1314 that may control inputs to the processor 1300 from other processors and / or components included in the system and / or outputs from the processor 1300 to other processors and / or components included in the system. The controller 1314 may control data paths in the ALU 1304, FPLU 1306, and / or DSPU 1308. The controller 1314 may be implemented as one or more state machines, data paths, and / or dedicated control logic units. The gates of the controller 1314 may be implemented as independent gates, FPGAs, ASICs, or any other suitable technology.

[0147] Registers 1312 and cache 1310 may communicate with controller 1314 and core 1302 via internal connections 1320A, 1320B, 1320C, and 1320D. The internal connections may be implemented as a bus, multiplexer, crossbar switch, and / or any other suitable connection technology.

[0148] Input and output for the processor 1300 may be provided via a bus 1316, which may include one or more conductors. The bus 1316 may be communicatively coupled to one or more components of the processor 1300, such as the controller 1314, the cache 1310, and / or the registers 1312. The bus 1316 may be coupled to one or more components of the system.

[0149] Bus 1316 may be coupled to one or more external memories. External memory may include read-only memory (ROM) 1332. ROM 1332 may be a mask ROM, an electrically programmable read-only memory (EPROM), or any other suitable technology. External memory may include random access memory (RAM) 1333. RAM 1333 may be static RAM, battery-backed static RAM, dynamic RAM (DRAM), or any other suitable technology. External memory may include electrically erasable programmable read-only memory (EEPROM) 1335. External memory may include flash memory 1334. External memory may include a magnetic storage device (e.g., disk 1336). In some embodiments, external memory may be included in the system.

[0150] All positions can be referenced to the scene coordinate system.

[0151] The present invention can be implemented in any suitable form (comprising hardware, software, firmware or any combination thereof).Optionally, the present invention can be implemented at least in part as the computer software running on one or more data processors and / or digital signal processors.The elements and parts of embodiments of the present invention can be implemented physically, functionally and logically in any suitable manner.In fact, the function can be implemented in a single unit, in multiple units or be implemented as the part of other functional units.Just for this reason, the present invention can be implemented in a single unit, or can be distributed between different units, circuits and processors physically and functionally.

[0152] According to standard terminology in the art, the term "pixel" may be used to refer to attributes associated with a pixel, such as light intensity, depth, position of the portion / element of a scene represented by the pixel. For example, the depth of a pixel or pixel depth may be understood to refer to the depth of an object represented by the pixel. Similarly, the brightness of a pixel or pixel brightness may be understood to refer to the brightness of an object represented by the pixel.

[0153] Although the present invention has been described in conjunction with some embodiments, the present invention is not limited to the specific forms set forth herein. On the contrary, the scope of the present invention is limited only by the appended claims. In addition, although features may appear to be described in conjunction with specific embodiments, those skilled in the art will recognize that the various features of the described embodiments may be combined in accordance with the present invention. In the claims, the term "comprising" does not exclude the presence of other elements or steps.

[0154] In addition, although multiple units, elements, circuits or method steps are listed separately, these multiple units, elements, circuits or method steps can be implemented by, for example, a single circuit, unit or processor. In addition, although various features may be included in different claims, these features may be advantageously combined, and the fact that these features are included in different claims does not mean that the combination of features is not feasible and / or disadvantageous. Moreover, including a feature in a claim category does not mean that this feature is limited to this category, but indicates that this feature is also applicable to other appropriate claim categories. In addition, the order of features in the claims does not mean that the features must work in any specific order, especially the order of the various steps in the method claims does not mean that these steps must be performed in this order. On the contrary, these steps can be performed in any suitable order. In addition, singular references do not exclude pluralities. Therefore, references to "one", "one", "first", "second", etc. do not exclude multiples. The figure marks in the claims are provided only for illustrative examples and should not be interpreted as limiting the scope of the claims in any way.

Claims

1. An apparatus for generating a depth map for an image representing a view of a scene, the apparatus comprising: a position receiver (301) arranged to receive a capture position for the image, the capture position being a position at which the image is captured; a depth receiver (303) arranged to receive a first depth map from a first location providing depth values ​​and a second depth map from a second location providing depth values, the first depth map comprising at least some pixels designated as depth-uncertain pixels and the second depth map comprising at least some pixels designated as depth-uncertain pixels; a view shift processor (305) arranged to perform a first view shift of the first depth map from the first position to the captured position to generate a first view shifted depth map, and to perform a second view shift of the second depth map from the second position to the captured position to generate a second view shifted depth map, the view shift processor (305) being further arranged to designate the following pixels as uncertain: The first view shift does not project depth values ​​of pixels in the first depth map that are not designated as uncertain pixels, and The second view shift does not project depth values ​​of pixels in the second depth map that are not designated as uncertain pixels, to pixels in the second view shifted depth map; a combiner (307) arranged to generate a combined depth map for the capture location by combining depth values ​​of co-located pixels of the first view shifted depth map and the second view shifted depth map, the combiner (307) being arranged to: designate a pixel of the combined depth map as a depth uncertain pixel if any co-located pixel is designated as an uncertain pixel; as well as A depth map generator (309) is arranged to generate an output depth map for the capture location by determining depth values ​​assigned to pixels designated as uncertain pixels in the combined depth map using at least one of image values ​​of the image and depth values ​​of pixels not designated as uncertain in the combined depth map.

2. The device according to claim 1, wherein: An angle between a direction from the capture position to the first position and a direction from the capture position to the second position is not less than 90°.

3. The device according to claim 1 or 2, wherein: The first position, the second position, and the capture position are arranged in a linear configuration, wherein the capture position is located between the first position and the second position.

4. An apparatus according to any preceding claim, wherein: The first position, the second position and the capture position are arranged such that, for the first view shift and the second view shift, movement directions of pixels at the same depth are opposite directions.

5. An apparatus according to any preceding claim, wherein: The combiner (307) is arranged to determine a depth value for a given pixel in the combined depth map as a weighted combination of a first depth value for a pixel that is not designated as a depth uncertain pixel in the first view shift map and has the same position as the given pixel, and a second depth value for a pixel that is not designated as a depth uncertain pixel in the second view shift map and has the same position as the given pixel.

6. An apparatus according to any preceding claim, wherein: The depth map generator (309) is arranged to generate depth values ​​for pixels designated as uncertain in the combined depth map by estimating from depth values ​​of pixels in the combined depth map that are not designated as uncertain.

7. A capture system comprising the apparatus of any preceding claim, and further comprising: a first image camera located at the capture location and arranged to provide the image to the image receiver (301); a first depth sensor located at the first location and arranged to provide depth data for the first depth map to the depth receiver (303); as well as A second depth sensor is located at the second position and is arranged to provide depth data for the second depth map to the depth receiver (303).

8. The capture system of claim 7, further comprising: At least one further image camera is located at a further capture position, the angle between the direction from the further capture position to the first position and the direction from the further capture position to the second position being no less than 90°.

9. The capture system according to any one of claims 7 and 8, comprises a plurality of image cameras including the first image camera, and a plurality of depth sensors including the first depth sensor and the second depth sensor, wherein the plurality of depth sensors are arranged in a configuration such that: for all image cameras of the plurality of image cameras, two depth sensors among the plurality of depth sensors are arranged such that the angle between the position of the image camera and the direction of the two depth sensors is not less than 120°.

10. The capture system of claim 9, wherein: The plurality of image cameras and the plurality of depth sensors are arranged in a linear configuration.

11. The capture system according to claim 9 or 10, wherein: At least one depth sensor of the plurality of depth sensors is located between each pair of adjacent image cameras of the plurality of image cameras.

12. The capture system of claim 11, wherein: The plurality of image cameras and the plurality of depth sensors are arranged in a two-dimensional configuration.

13. The capture system according to any one of claims 9 to 12, wherein: A number of depth sensors in the plurality of depth sensors exceeds a number of image cameras in the plurality of image cameras.

14. The capture system according to any one of claims 9 to 12, wherein: The number of image cameras in the plurality of image cameras exceeds the number of depth sensors in the plurality of depth sensors.

15. A method of generating a depth map for an image representing a view of a scene, the method comprising: receiving a capture position for the image, the capture position being a position at which the image is captured; receiving a first depth map from a first location providing depth values ​​and a second depth map from a second location providing depth values, the first depth map including at least some pixels designated as depth-uncertain pixels and the second depth map including at least some pixels designated as depth-uncertain pixels; performing a first view shift of the first depth map from the first position to the capture position to generate a first view shifted depth map, performing a second view shift of the second depth map from the second position to the capture position to generate a second view shifted depth map, Designate the following pixels as uncertain: The first view shift does not project depth values ​​of pixels in the first depth map that are not designated as uncertain pixels, and The second view shift does not project depth values ​​of pixels in the second depth map that are not designated as uncertain pixels, to pixels in the second view shifted depth map; generating a combined depth map for the capture location by combining depth values ​​of co-located pixels of the first view shifted depth map and the second view shifted depth map, the combiner being arranged to: designate a pixel of the combined depth map as a depth uncertain pixel if any co-located pixel is designated as an uncertain pixel; and An output depth map for the capture location is generated by determining depth values ​​in the combined depth map that are designated as uncertain depth values ​​using at least one of image values ​​of the image and depth values ​​that are not designated as uncertain.

16. A computer program product comprising computer program code means adapted to perform all the steps of claim 15 when said program is run on a computer.

Images