Method for generating layered depth data of a scene - Patents.com

Abstract

A computer-implemented method and computer program product are provided for generating layered depth data for a scene from light field content, whether video or pictures. [Solution] The method includes retrieving 401 light field content of a scene captured and provided by a light field camera or stored in a storage unit of a computing device, computing 402 a depth map for at least one viewpoint of the retrieved light field content, computing 403 a first set of occlusion information associated with the image in a first direction different from the view direction of the image of the light field content, computing 404 a second set of occlusion information associated with the image in a second direction different from the view direction of the image and the first direction, generating 405 stacked depth data of the scene, and transmitting the stacked depth data to a rendering or processing device.

Description

[Technical field]

[0001] Technical Field The present invention relates to stacked depth data, and more precisely to a stacked format that exploits properties of light field contents, whether those light field contents are videos or pictures. [Background technology]

[0002] background In stereo or multi-view systems, only the horizontal dimension of the scene is taken into account when formatting the data. For example, in the case of an acquisition system consisting of a camera rig where the cameras are aligned horizontally, only the horizontal disparity between the 3D views can be extracted. Depth image based rendering techniques are well known for interpolating intermediate views between the captured ones, but always in the horizontal direction.

[0003] In multi-view images, there is a large amount of redundancy between images. The stacked depth video format or LDV is a well-known formatting solution for formatting multi-view images, which reduces the amount of redundant information between images. In LDV, a reference center image is selected, and information provided by other images of the multi-view image where the area of ​​the center image is mainly occluded is provided. The LDV format is then composed of four layers that represent the information required to process the multi-view image. - Center image selected - The depth map associated with the selected central image - Occlusion images - Depth Occlusion Map

[0004] Therefore, only non-redundant information is sent to the rendering device, which is included in the occlusion mask generated from the depth occlusion map.

[0005] As is true for other formats used in multi-view contexts, the LDV format contains a single horizontal occlusion layer and therefore fails to render viewpoints that reveal the multi-layer disocclusion that can occur in complex scenes viewed at wide inter-axis distances.

[0006] The present invention has been devised with the above in mind. Summary of the Invention

[0007] Summary of the Invention According to a first aspect of the invention there is provided a computer implemented method for generating layered depth data for a scene, comprising the steps of: - computing a depth map of an image from a light field content representative of a scene, said image being viewed according to a predefined view direction; - computing a first set of occlusion information associated with the image from the light field content in a first direction different from a view direction of the image from the light field content; - computing at least a second set of occlusion information associated with the image from the light field content in at least a second direction different from the view direction of the image from the light field content and the first direction; aggregating the images from the light field content, the depth map, the first set of occlusion information, and the second set of occlusion information to generate stacked depth data for the scene; A method is provided, comprising:

[0008] The method according to the embodiment of the present invention is not limited to light field contents directly acquired by optical devices. These contents can be computer graphic images (CGI) fully or partially simulated by a computer for a given scene description. Another source of light field contents can be post-film data that is modified (e.g. color graded) light field contents obtained from optical devices or CGI. Also, it is now common in the movie industry to have a mix of both data acquired using optical acquisition devices and CGI data.

[0009] The method according to an embodiment of the invention relies on the use of light field content, which provides parallax in all directions and allows for viewpoint changes in many directions different from the view direction of the image considered.

[0010] Such methods, which rely on the use of light field content, enable the rendering of viewpoints that reveal the multi-layer disocclusions that can occur in complex scenes viewed at wide axial distances.

[0011] The layered depth data generated according to the method referred to above includes at least an image from a light field content, a depth map associated with the image, a first set of occlusion information associated with the image, and a second set of occlusion information.

[0012] In an embodiment of the present invention, the optically applied content may be video content.

[0013] According to an embodiment of the method for generating layered depth data, the first occlusion information set and the second occlusion information set are merged together to generate a third occlusion information set, which is aggregated with images and depth maps from the light field content to generate layered depth data for the scene.

[0014] Merging the first and second occlusion information sets can reduce the amount of transmission data and the amount of processing data at the receiver side. Merging the first and second occlusion information sets generates a third occlusion information set, which may take the form of unique occlusion information and represents occlusion information in two considered directions.

[0015] According to an embodiment of the method for generating stacked depth data, computing the first and second occlusion information sets consists in comparing an image of the content light field with another adjacent image from the light field content in each of the first and second directions.

[0016] For example, considering that the first direction is horizontal with respect to the view direction of the image from the light field content, the first occlusion information set is obtained by comparing the image of the light filed content with adjacent images from the light field content in the horizontal direction.

[0017] For example, considering that the second direction is perpendicular to the view direction of the image from the light field content, the second occlusion information set is obtained by comparing the image of the light filed content with an adjacent image from the light field content in the perpendicular direction.

[0018] Another object of the invention is to provide an apparatus for generating layered depth data of a scene, comprising: - computing a depth map of an image from a light field content representative of a scene, said image being viewed according to a predefined view direction; - computing a first set of occlusion information associated with the image from the light field content in a first direction different from a view direction of the image from the light field content; - computing at least a second set of occlusion information associated with the image from the light field content in at least a second direction different from the view direction of the image from the light field content and the first direction; aggregating the images from the light field content, the depth map, the first set of occlusion information, and the second set of occlusion information to generate stacked depth data for the scene; The present invention relates to an apparatus including a processor configured to perform the steps of:

[0019] According to an embodiment of the apparatus for generating layered depth data of a scene, the first occlusion information set and the second occlusion information set are merged together to generate a third occlusion information set, which is aggregated with images and a depth map from the light field content to generate layered depth data of the scene.

[0020] According to an embodiment of an apparatus for generating stacked depth data of a scene, computing the first and second occlusion information sets consists in comparing an image of the content light field with another adjacent image from the light field content in each of the first and second directions.

[0021] Another object of the invention relates to a method for processing light field content representing a scene, comprising processing said light field content based on stacked depth data of the scene associated with the light field content of the scene, wherein the stacked depth data comprises a depth map of an image from the light field content, a first set of occlusion information associated with the image from the light field content, the first set of occlusion information calculated in a first direction different from a view direction of the image from the light field content, and at least a second set of occlusion information associated with the image from the light field content, the first set of occlusion information calculated in a second direction different from a view direction of the image from the light field content.

[0022] Another object of the invention relates to a signal transmitted by a first device capable of generating stacked depth data of a scene to a second device capable of processing said stacked depth data of the scene, the signal conveying a message comprising stacked depth data of the scene, the stacked depth data comprising a depth map of an image from a light field content of the scene, a first set of occlusion information associated with the image from the light field content, the first set of occlusion information calculated in a first direction different from the view direction of the image from the light field content, and at least a second set of occlusion information associated with the image from the light field content, the first set of occlusion information calculated in a second direction different from the view direction of the image from the light field content, and processing of the captured image by the second device relates to a signal based on said stacked depth data.

[0023] Some processes implemented by elements of the present invention may be computer-implemented. Accordingly, such elements may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Moreover, such elements may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.

[0024] Since elements of the invention can be implemented in software, the invention can be embodied as computer readable code for provision to a programmable apparatus on any suitable carrier medium. Tangible carrier media can include storage media such as floppy disks, CD-ROMs, hard disk drives, magnetic tape devices, or solid-state memory devices and the like. Transient carrier media can include signals such as electrical, electronic, optical, acoustic, magnetic, or electromagnetic signals (e.g., microwave or RF signals).

[0025] BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention will now be described, by way of example only, with reference to the following drawings: [Brief description of the drawings]

[0026] [Figure 1A] 1 illustrates a schematic representation of a plenoptic camera. [Figure 1B] 1 shows a schematic diagram of a camera rig. [Figure 2A] It represents three horizontally aligned cameras C1, C2, C3 of a camera array containing at least five cameras and the portion of the space captured by these cameras. [Figure 2B] Shown are three vertically aligned cameras C4, C2, C5 of a camera array containing at least five cameras and the portion of space captured by these cameras. [Diagram 3] FIG. 1 is a schematic block diagram illustrating an example of an apparatus for obtaining layered depth data of a scene, according to an embodiment of the present disclosure. [Figure 4] 4 is a flowchart illustrating a process for generating layered depth data for a scene according to an embodiment of the present invention. [Figure 5A] 4 represents stacked depth data for a scene generated according to a first embodiment of the present invention. [Figure 5B] 4 represents stacked depth data for a scene generated according to a second embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Detailed Description As will be appreciated by one of ordinary skill in the art, aspects of the present principles may be embodied as a system, method, or computer readable medium. Accordingly, aspects of the present principles may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system." Moreover, aspects of the present principles may take the form of a computer readable storage medium. Any combination of one or more computer readable storage media may be utilized.

[0028] Plenoptic cameras can measure the amount of light traveling along each bundle of rays that intersect with the sensor by placing a microlens array between the main lens and the sensor. The data acquired by such cameras is called light field data or light field content. These light field data can be post-processed to reconstruct images of a scene from different viewpoints. The light field data can be used to generate a focal stack, which includes a collection of images, each with a different refocus depth. As a result, a user can change the focus of the images. Compared to conventional cameras, plenoptic cameras can obtain additional information to achieve reconstruction of images of a scene from different viewpoints and refocus depths through post-processing.

[0029] Thus, use of these specificities of the light field data in the context of stacked depth imaging is possible.

[0030] 1A is a schematic diagram of a plenoptic camera 100. A light field camera is capable of recording four-dimensional (or 4D) light field data. The plenoptic camera 100 includes a main lens 101, a microlens array 102, and an image sensor 104.

[0031] 1B depicts a schematic diagram of a camera rig 110. The camera rig 110 includes a number of sensors 114, each of which is associated with a lens 112.

[0032] In the example plenoptic camera 100 shown in Figure 1A, a main lens 101 receives light from an object (not shown) in an object field of the main lens 101 and passes the light to an image field of the main lens 101. A microlens array 102 includes a number of microlenses 103 arranged in a two-dimensional array.

[0033] The data captured by a light field camera can be post-processed to reconstruct an image of a scene from different viewpoints. Light field cameras are capable of capturing a collection of partial views of the same scene from slightly altered viewpoints, allowing the different partial views to be combined to create an image with a customized focal plane.

[0034] 2A and 2B represent an array of cameras including three horizontally aligned cameras C1, C2, C3 and three vertically aligned cameras C4, C2, C5, and the portion of space captured by these cameras. Of course, the number of cameras is not limited to five, and there may be less than five cameras or more than five cameras embedded in the array of cameras.

[0035] In Figure 2A, cameras C1, C2, C3 are aligned along the horizontal axis: a first area 200 of screen 20 is visible to camera C1 but not to cameras C2, C3, and a second area 201 of screen 20 is visible to camera C3 but not to cameras C2, C1.

[0036] Reference 202 in Figure 2A is an image of the scene as seen by camera C1. A first portion 2020 of image 202 is seen by both camera C1 and camera C2. A second portion 2021 of image 202 is seen by camera C1 and occluded by camera C2.

[0037] Reference 203 in FIG. 2A is an image of the scene as seen by camera C2.

[0038] Reference 204 in Figure 2A is an image of the scene as seen by camera C3. A first portion 2040 of image 204 is seen by both camera C3 and camera C2. A second portion 2041 of image 204 is seen by camera C3 and occluded by camera C2.

[0039] In Figure 2B, cameras C4, C2, C5 are aligned along the horizontal axis: a first area 210 of screen 20 is visible to camera C4 but not to camera C2, and a second area 211 of screen 20 is visible to camera C5 but not to camera C2.

[0040] Reference 212 in Figure 2B is an image of the scene as seen by camera C4. A first portion 2120 of image 212 is seen by both camera C4 and camera C2. A second portion 2121 of image 212 is seen by camera C4 and occluded by camera C2.

[0041] Reference 203 in FIG. 2B is an image of the scene as seen by camera C2.

[0042] Reference 214 in Figure 2B is an image of the scene as seen by camera C5. A first portion 2140 of image 214 is seen by both camera C5 and camera C2. A second portion 2141 of image 214 is seen by camera C5 and occluded by camera C2.

[0043] FIG. 3 is a schematic block diagram illustrating an example of an apparatus for generating layered depth data for a scene, according to an embodiment of the present disclosure.

[0044] The device 300 includes a processor 301, a storage unit 302, an input device 303, a display device 304 and an interface unit 305, which are connected by a bus 306. Of course, the components of the computing device 300 can be connected by connections other than a bus connection.

[0045] The processor 301 controls the operation of the device 300. The storage unit 302 stores at least one program executed by the processor 301, various data including data of 4D light field images captured and provided by the light field camera, parameters used by the calculations performed by the processor 301, intermediate data of the calculations performed by the processor 301, etc. The processor 301 can be formed by any known suitable hardware or software, or a combination of hardware and software. For example, the processor 301 can be formed by dedicated hardware such as a processing circuit or by a programmable processing unit such as a CPU (Central Processing Unit) that executes a program stored in its memory.

[0046] The storage unit 302 can be formed by any suitable storage device or means capable of storing programs, data or the like in a computer readable manner. Examples of the storage unit 302 include semiconductor memory devices and non-transitory computer readable storage media such as magnetic, optical or magneto-optical recording media loaded into a read and write unit. The program causes the processor 301 to execute a process for obtaining a registration error map representing the blurriness of an image according to an embodiment of the present disclosure, as described below with reference to FIG. 5.

[0047] The input device 303 may be formed by a keyboard, a pointing device such as a mouse, or the like, for use by a user to input commands and make user selections on a three-dimensional (or 3D) model of the target object used to define the refocus surface. The output device 304 may be formed by, for example, a display device for displaying images generated according to the embodiments of the present disclosure, such as a graphical user interface (GUI). The input device 303 and the output device 304 may be integrally formed by, for example, a touch screen panel.

[0048] The interface unit 305 provides an interface between the device 300 and an external device. The interface unit 305 may be capable of communicating with the external device via cable or wireless communication. In an embodiment, the external device may be a light field camera. In this case, data of a 4D light field image captured by the light field camera may be input from the light field camera to the device 300 through the interface unit 305 and then stored in the storage unit 302.

[0049] It should be noted that, although in this embodiment the device 300 is exemplarily discussed as being separate from the light field camera and capable of communicating with each other via cable or wireless communication, the device 300 can be integrated with such a light field camera, in this latter case the device 300 can be, for example, a portable device such as a tablet or smartphone that embeds a light field camera.

[0050] FIG. 4 is a flow chart illustrating a process for generating layered depth data for a scene according to an embodiment of the present disclosure.

[0051] In step 401, the processor 301 of the device 300 retrieves light field content of a scene captured and provided by a light field camera or stored in the storage unit 302 of the device 300. In this latter case, the light field content is for example a computer graphics image (CGI) that is fully or partially simulated by a computer for a given scene description.

[0052] In step 402, the processor 301 of the device 300 computes a depth map for at least one viewpoint of the retrieved light field content. The considered viewpoint from the light field content corresponds to a given view direction of the scene. For a given image, the information about the depth is related to the inter-view disparity. The inter-view disparity is an inverse function of the depth versus the magnification depending on the focal length and axial distance of the optical system of the real or virtual light field camera used to acquire the light field content. The inter-view disparity is estimated hierarchically based on the number of pixels, for example by performing a correspondence analysis as described in "A precise real-time stereo algorithm", V. Drazic, N. Sabater, Proceedings of the 27th Conference on Image and Vision Computing New Zealand. In order to present sharp edges with smooth depth variations in the computed depth map, regularization can be performed during the computation of the depth map using an appropriate regularization cost or as a post-processing, for example using bilateral filtering.

[0053] Depth maps are computed for all available view directions. For example, as represented in Figures 2A and 2B, when the light field content is acquired by an array of cameras, the image considered is the image 202 acquired by camera C2. A left-to-right disparity estimation is performed to obtain a depth map corresponding to the image 202 acquired by camera C1 located to the left of camera C2. A right-to-left disparity estimation is performed to obtain a depth map corresponding to the image 204 acquired by camera C3 located to the right of camera C2.

[0054] A top-to-bottom disparity estimation is then performed to obtain a depth map corresponding to image 212 acquired by camera C4 located above camera C2, and a bottom-to-top disparity estimation is performed to obtain a depth map corresponding to image 214 acquired by camera C5 located below camera C2.

[0055] In step 403 , the processor 301 computes a first set of occlusion information associated with the image 203 in a first direction different from the view direction of the image 203 .

[0056] Occlusions are detected by comparing depth maps associated with two adjacent images, e.g. images 202 and 203. Occlusions occur in areas where the depth maps associated with the two adjacent images 203 and 202 do not match. These correspond to a second portion 2021 of the image 202 that is visible from camera C1 and occluded from camera C2. Such portions of the depth map that correspond to portion 2021 of image 202 are labeled as empty, since the depth estimated by correspondence analysis is unreliable, and are then filled in with a conventional method, e.g. based on background propagation that preserves depth gradients and curvature.

[0057] During step 403, another set of occlusion information can be computed by comparing the depth maps associated with two adjacent images, such as images 203 and 204.

[0058] In step 404, the processor 301 computes a second set of occlusion information associated with the image 203 in a view direction of the image 203 and in a second direction different from the first direction.

[0059] For example, depth maps associated with two adjacent images, such as images 212 and 203, are computed. Occlusions occur in areas where the depth maps associated with the two adjacent images 203 and 212 do not match. These correspond to a second part 2121 of the image 212 that is visible from camera C4 and occluded from camera C2.

[0060] During step 404, another set of occlusion information can be computed by comparing the depth maps associated with two adjacent images, such as images 213 and 214.

[0061] The processor 301 may compute more than two occlusion information sets associated with the image 203 in other directions different from the view direction of the image 203 and the first direction. In step 405, the processor 301 generates stacked depth data of the scene. The stacked depth data generated according to the method mentioned above includes at least an image from the light field content, a depth map associated with said image, a first occlusion information set and a second occlusion information set associated with said image.

[0062] In an embodiment of the present invention, the optically applied content may be video content.

[0063] In a first embodiment, represented in Figure 5A, the layered depth data of the scene is generated by aggregating the image 203, the depth map 50 associated with the image 203, a first set of occlusion information in the form of an occlusion mask 51 and a second set of occlusion information in the form of an occlusion mask 52. In a second embodiment, represented in Figure 5B, the layered depth data of the scene is generated by aggregating the image 203, the depth map 50 associated with the image 203 and a third set of occlusion information in the form of an occlusion mask 53.

[0064] This third set of occlusion information is computed by merging the first and second sets of occlusion information.

[0065] For example, the third occlusion information set may include an average value of the first and second occlusion information. In cases where more than two occlusion information sets are available, the processor 301 may select one of them as the occlusion information set used to generate the stacked depth data, for example based on an associated confidence criterion.

[0066] Then, in step 406, the stacked depth data is sent to a rendering or processing device.

[0067] Although the present invention is described above with reference to specific embodiments, the present invention is not limited to the specific embodiments, and modifications will be apparent to those skilled in the art and are within the scope of the present invention.

[0068] Many further modifications and variations will suggest themselves to those skilled in the art upon reference to the foregoing exemplary embodiments, which are provided merely by way of example and are not intended to limit the scope of the invention, which is determined solely by the appended claims. In particular, different features from different embodiments may, where appropriate, be interchangeable.

Claims

1. 1. An apparatus including a circuit with a processor, the circuit comprising: receiving light field content including a reference view of a scene, a first set of additional views that differ in viewpoint from the reference view along a first direction, and at least a second set of additional views that differ in viewpoint from the reference view along at least a second direction, the second direction being different from the first direction; generating a depth map of an image from the light field content of the reference view; generating a first set of occlusion information in the first direction; generating at least a second set of occlusion information in said at least a second direction; merging the first occlusion information set and the at least second occlusion information set into a third occlusion information set; generating layered depth data for the scene comprising the image from the light field content of the reference view, the depth map, and the third set of occlusion information; An apparatus configured to:

2. The apparatus of claim 1 , further comprising a capture device for capturing the light field content, the capture device comprising a light field camera.

3. 2. The apparatus of claim 1, wherein the light field content comprises any of light field content acquired by an optical device, a computer graphics image at least partially simulated by a computer, post-generation data of the light field content acquired from the optical device, post-generation data of the computer graphics image at least partially simulated by the computer, and a combination of at least two of these.

4. Generating the first occlusion information set includes: generating a first set of depth maps using a disparity analysis between the reference view and each view of the first set of views; comparing depth maps of the first set of depth maps to detect corresponding areas of inconsistency; The apparatus of claim 1 , comprising:

5. Generating the at least second occlusion information set includes: generating a second set of depth maps using a disparity analysis between the reference view and each view of the at least a second set of views; comparing depth maps of the second set of depth maps to detect corresponding areas of inconsistency; The apparatus of claim 1 , comprising:

6. The apparatus of claim 1 , wherein the reference view is a central view relative to the first set of views and the at least second set of views.

7. The device of claim 1 , wherein the device is one of a portable device, a tablet, and a smartphone.

8. A non-transitory computer readable medium having instructions stored thereon, the instructions comprising: receiving light field content including a reference view of a scene, a first set of additional views that differ in viewpoint from the reference view along a first direction, and at least a second set of additional views that differ in viewpoint from the reference view along at least a second direction, the second direction being different from the first direction; generating a depth map of an image from the light field content of the reference view; generating a first set of occlusion information in the first direction; generating at least a second set of occlusion information in said at least a second direction; merging the first occlusion information set and the at least second occlusion information set into a third occlusion information set; generating layered depth data for the scene comprising the image from the light field content of the reference view, the depth map, and the third set of occlusion information; A non-transitory computer-readable medium that causes a processor to:

9. 9. The non-transitory computer-readable medium of claim 8, wherein the light field content includes any of light field content acquired by an optical device, a computer graphics image at least partially simulated by a computer, post-generation data of the light field content acquired from the optical device, post-generation data of the computer graphics image at least partially simulated by the computer, and a combination of at least two of these.

10. Generating the first occlusion information set includes: generating a first set of depth maps using a disparity analysis between the reference view and each view of the first set of views; comparing depth maps of the first set of depth maps to detect corresponding areas of inconsistency; 9. The non-transitory computer readable medium of claim 8, comprising:

11. Generating the at least second occlusion information set includes: generating a second set of depth maps using a disparity analysis between the reference view and each view of the at least a second set of views; comparing depth maps of the second set of depth maps to detect corresponding areas of inconsistency; 9. The non-transitory computer readable medium of claim 8, comprising:

12. The non-transitory computer-readable medium of claim 8 , wherein the reference view is a central view relative to the first set of views and the at least second set of views.

13. receiving light field content including a reference view of a scene, a first set of additional views that differ in viewpoint from the reference view along a first direction, and at least a second set of additional views that differ in viewpoint from the reference view along at least a second direction, the second direction being different from the first direction; generating a depth map of an image from the light field content of the reference view; generating a first set of occlusion information in the first direction; generating at least a second set of occlusion information in said at least a second direction; merging the first occlusion information set and the at least second occlusion information set into a third occlusion information set; generating layered depth data for the scene comprising the image from the light field content of the reference view, the depth map, and the third set of occlusion information; A method comprising:

14. 14. The method of claim 13, wherein the light field content comprises any of light field content acquired by an optical device, a computer graphics image at least partially simulated by a computer, post-generation data of the light field content acquired from the optical device, post-generation data of the computer graphics image at least partially simulated by the computer, and a combination of at least two of these.

15. Generating the first occlusion information set includes: generating a first set of depth maps using a disparity analysis between the reference view and each view of the first set of views; comparing depth maps of the first set of depth maps to detect corresponding areas of inconsistency; The method of claim 13, comprising:

16. Generating the at least second occlusion information set includes: generating a second set of depth maps using a disparity analysis between the reference view and each view of the at least a second set of views; comparing depth maps of the second set of depth maps to detect corresponding areas of inconsistency; The method of claim 13, comprising:

17. The method of claim 13 , wherein the reference view is a central view relative to the first set of views and the at least second set of views.

Images