Depth Segmentation in Multiview Video

Abstract

A method of depth segmentation for generating multi-view video data, the method including obtaining a plurality of source-view images and a source-view depth map representing a 3D scene from a plurality of sensors, a foreground object in the 3D scene being segmented from the source-view images and / or the source-view depth map, and one or more patches being generated for each source-view image and source-view depth map including at least one foreground object, each patch corresponding to one foreground object, and generating the patches including generating a patch texture image, a patch depth map, and a patch transparency map based on the source-view images and the source-view depth map.

Description

[Technical field]

[0001] The present invention relates to the field of multiview video, in particular to depth segmentation for multiview video generation and rendering of multiview video. [Background technology]

[0002] Existing approaches to rendering from multi-view images with depth use blending to combine warped textures from multiple source-view (capture) cameras. The blending operation may depend on variables such as source and target camera position / orientation (e.g., ray angle difference), depth magnitude, depth variation, deocclusion, transparency and color. More advanced techniques may use trained convolutional neural networks to align textures at the target viewpoint. There are several formats for storing multi-view images.

[0003] Layered Depth Images (LDI) store a set of depth pixels along a single line of sight (rather than just one). As the virtual viewpoint moves away from the LDI stored viewpoint, occluded surfaces become visible.

[0004] Multi-plane imaging (MPI) and multi-sphere imaging (MSI) techniques compose color and transparency for a given set of planes or spheres in 3D space, and for new virtual viewpoints, images are constructed using back-to-front layer compositing.

[0005] A Layered Mesh (LM) can be constructed from MPI and MSI and represents a traditional graphics mesh with texture, and is therefore suitable for atlas construction and transmission using existing video codecs. Summary of the Invention [Problem to be solved by the invention]

[0006] Layered formats (LDI, MPI, MSI, LM) can potentially yield larger viewing zones due to explicit occlusion handling, but these formats are difficult to generate, especially in real time, from multi-camera systems.

[0007] Loghman Maziar et al. "Segmentation-based view synthesis for multi-view video plus depth", Multimedia Tools and Applications, Kluwer Academy Publishers Boston vol. 74, no. 5, 8 November 2013, discloses a method for image synthesis by segmenting objects from source images and warping the segmented objects individually. [Means for solving the problem]

[0008] The invention is defined by the claims.

[0009] According to an example embodiment of the present invention, there is provided a depth segmentation method for generating multi-view video data, the method comprising: The method includes steps of obtaining a plurality of source-view images and a source-view depth map representing a 3D scene from a plurality of sensors, segmenting foreground objects in the 3D scene from the source-view images and / or the source-view depth map, and generating one or more patches for each source-view image and source-view depth map including at least one foreground object, each patch corresponding to a foreground object, and the generating patches includes generating a patch texture image, a patch depth map and a patch transparency map based on the source-view images and the source-view depth map.

[0010] Typical formats for multi-view video require significant processing power to generate (e.g., layered depth images, multi-plane images, etc.) due to the complex analysis typically required to estimate the depth value of each pixel. For example, the inability to find robust algorithms to do this has led to the increased use of data-driven approaches based on deep learning. This problem is particularly present in the broadcast of multi-view videos, such as live sporting events, since the creation of depth and texture atlas data for data reduction must be done in real time for each frame.

[0011] Therefore, the inventors proposed to "segment" patches from source views (i.e., images and depth maps) that contain foreground objects. Thus, the atlas contains data only from patches (and background), rather than data for all source views. Each patch is based on a section of the source view image that is smaller than the source view image itself, and the corresponding depth and transparency data for that section. In other words, instead of using a source view that corresponds to an arbitrary portion of the scene, each patch acts as a partial source view with texture, transparency, and depth data corresponding to the foreground objects used to render the scene. Patches can overlap each other. Respective patches may have the same size, or may have different sizes (e.g., depending on their corresponding foreground objects). In some cases, for example, if they are based on the same section of the source view image (e.g., if a particular section contains two or more foreground objects), different patches may have identical patch texture images and patch depth maps.

[0012] There are various methods for segmenting foreground objects from either the source view image or the source view depth map. For example, a segmentation algorithm or an object detection algorithm can be used on the source view image to detect / segment the foreground object. Alternatively, a depth difference (above a threshold) in the source view depth map can be calculated so that the edge of the foreground object can be defined by a large depth difference. In the case of multi-view video, the difference between frames (of either the source view image or the source view depth map) can be used to detect the motion of the foreground object to detect the foreground object.

[0013] Thus, patches can be generated for foreground objects in each source view. Each patch corresponds to a single foreground object. In particular, the extent of a patch may be determined, at least in part, by the extent of the object (or part of the object). However, a foreground object may have more than one corresponding patch. Alternatively, each foreground object may be associated with a single patch. A patch has a patch texture image that contains the texture / color data of the patch, and a patch depth map that contains the depth data of the patch. A patch also has a patch transparency map (also called a patch alpha map) that contains a transparency value for each pixel in the patch.

[0014] The method includes obtaining a plurality of background depth maps of the 3D scene representing a background of the 3D scene, the background depth maps including depth data of the background from a particular orientation, and segmenting the foreground object based on a difference between the background depth map and a corresponding source view depth map.

[0015] For example, if segmenting is based on the source view image itself, the generated patches are likely to bleed across depth boundaries, which can cause problems in encoding or later rendering. Therefore, by using the background depth map, foreground objects can be robustly segmented and these problems can be avoided.

[0016] The background depth map of each source view can be obtained by fitting a predefined geometric scene model to a subset of the source views. For example, assuming the background consists of a horizontal ground plane and a vertical background plane, these planes can be first positioned and shifted / rotated relative to each other and to the camera such that an image-based multiview matching criterion is minimized. After fitting the predefined geometric scene to the subset of the source views, a background depth map can be rendered for each source view.

[0017] A background depth map includes depth data of the background of a 3D scene. For example, a background depth map may be generated based on the views of multiple cameras capturing the 3D scene from different angles. The background depth map may be generated from a set of cameras different from the cameras used to capture the source views. For example, if the 3D scene is a soccer field, a camera on the side of the soccer field may be used to capture the foreground objects (i.e., players and ball), and a camera looking down on the soccer field (e.g., a top-down camera) may be used to generate the background depth map.

[0018] Foreground objects can be segmented by thresholding the difference between the source view depth map and the background depth map of each source view. After this global threshold, a second local thresholding can be applied to separate connected foreground objects based on their relative depth steps.

[0019] A trained human detection algorithm can be used to detect foreground objects. A ball detector can be used to detect the ball in a sports game. Motion estimation or time frame differencing can be used to further improve foreground object detection.

[0020] The method may further include obtaining a background model comprising a background depth map and background texture data.

[0021] The method can further include generating an atlas based on the patch texture image, the patch depth map, the patch transparency map, and the background model. For example, the atlas can include the patch texture image, the patch depth map, the patch transparency map, and the background model.

[0022] An atlas is essentially a data matrix that contains various images and / or maps (e.g., texture, depth, and transparency data). To find an image or map in the atlas, the "coordinates" of each image (i.e., the column and row values ​​of the matrix) are specified. Thus, an atlas contains data from multiple source views.

[0023] Typically, all patch data is included separately in the atlas, but it is also possible, for example, to define patch transparency maps in binary (i.e., transparency values ​​of 0 or 1) and encode them via reserved values ​​in the depth map.

[0024] Detecting foreground objects may include subtracting a respective background depth map from the source view depth map to generate a difference image and thresholding the difference image, where the thresholding may include comparing pixel values ​​of the difference image to a threshold to generate a threshold map, thereby distinguishing between background and foreground objects. Pixels in the source view depth map corresponding to a depth step are identified, where the depth step is defined by a difference between adjacent depth values ​​in the source view depth map that is greater than a depth threshold. All depth values ​​corresponding to the depth step are adjusted in the threshold map, thereby distinguishing foreground objects from each other, and a bounding box for the foreground object is generated based on the adjusted threshold map.

[0025] Thresholding the difference image results in a binary map where pixel value "1" means foreground and "0" means background. To identify foreground objects, connected components are identified via an algorithm that labels 4-connected or 8-connected components. If this is done immediately after the first thresholding operation, multiple foreground objects will be erroneously identified as one object. To avoid this, for example, the spatial derivative of the original source view depth map is analyzed. If a depth step exceeds a "depth threshold", the binary map is set to "0" (i.e., background) on the far side of the step. When the resulting binary map is input to a connected component labeling algorithm, foreground objects can receive different labels.

[0026] The size and position of the patch may be based on the size and position of the bounding box.

[0027] The depth values ​​of the background depth map are subtracted from the source view depth map to bring all values ​​to zero (or close to zero) away from foreground objects present in the source view depth map, and the subtracted map is then thresholded based on a "threshold" to, for example, set all of the depth values ​​corresponding to the background to zero (or black) and set all of the depth values ​​corresponding to the foreground objects to one (or white).

[0028] Depth steps are also identified in the source view depth map. Depth steps correspond to large changes in depth of adjacent / neighboring pixels and indicate the edge of a foreground object. Depth steps can be identified by differences between adjacent depth values ​​that are greater than a depth threshold (e.g., greater than 0.1 in the normalized depth map).

[0029] The depth values ​​of the threshold map may then be adjusted, for example to zero (or black), at a depth step to accentuate and differentiate the edges of each foreground object, and a bounding box is generated for each foreground object based on the adjusted threshold map (e.g., to segment the foreground object in the adjusted threshold map).

[0030] The size and location of the patch may be the size and location of the bounding box, or multiple patches may be generated for one bounding box. For example, the number of patches per bounding box may depend on the size of the bounding box, the type of foreground object, the location of the foreground object, etc.

[0031] Detecting the foreground object may further include expanding a bounding box to include an area of ​​the foreground object in the subtracted map that is below the threshold.

[0032] For example, expanding the bounding boxes may be based on the difference between the source view depth map and the background depth map being less than a threshold for regions containing foreground objects, and the bounding boxes are expanded such that each foreground object is enclosed by the bounding box.

[0033] In some cases, a foreground object may have parts with depth values ​​close to the depth values ​​of the background, and therefore during thresholding, the foreground object may appear small and the bounding box may not completely surround the foreground object in the source view.

[0034] For example, the feet of a soccer player have a depth that is close to the soccer field they are standing on. In these cases, the bounding boxes are expanded (e.g., expanded downwards) so that the bounding box corresponding to the foreground object completely surrounds the foreground object.

[0035] Generating the patch texture image and the patch transparency map may be based on alpha matting the source view image. Alpha matting is based on extracting the foreground from the image. Thus, alpha matting can be used to estimate the texture and transparency (alpha value) of each pixel of the patch.

[0036] The method may further include adapting pixel depth values ​​of the patch depth map such that all of the pixel depth values ​​of the patch depth map are of values ​​equal to or lower than the depth values ​​of the corresponding foreground object.

[0037] For consistency and clarity, any depth map defined in this application is constructed such that a maximum value (e.g., 255) represents the closest distance to the viewpoint (i.e., the smallest depth value) and a minimum value (e.g., 0) represents the furthest distance (i.e., the largest depth value). Any references in this application to "smaller" or "greater" with respect to the values ​​of pixels in a depth map should be interpreted with respect to the foregoing definitions. However, it should be noted that any other format for representing a depth map may also be used and is known to one of ordinary skill in the art. For example, a "0" pixel may represent the closest distance and a "1" value may represent the furthest distance.

[0038] Some patch depth maps may contain depth data from other foreground objects that occlude the corresponding foreground object. The "unwanted" (or leftover) depth data may cause artifacts when rendering the foreground object. Therefore, it may be beneficial to adapt (i.e., change pixel values) the pixel depth values ​​of the patch depth map such that all of the pixel depth values ​​are greater than or equal to the depth value of the target foreground object (i.e., the foreground object that corresponds to the patch depth map in question).

[0039] Additionally, the method may further include identifying a plurality of patches from different source views that correspond to the first foreground object based on identifying patch depth maps of the identified patches within the object depth range and correcting the identified patch depth maps so that they correspond to object positions in the 3D scene.

[0040] For example, the patch depth maps can be corrected (e.g., filtered) by projecting the centroid positions of the patch depth maps of all views into a common world-space coordinate system. Patches from different source views that map to similar world-space coordinates (i.e., within a given inter-object distance) likely originate from one and the same physical foreground object. Thus, the patch depth maps can be corrected (i.e., to have more similar world-space coordinates). After correction, back projection onto the source view results in a filtered depth map for each patch.

[0041] The method may further include pruning patches based on measuring consistency between patches in multiple source views. For example, the method may include filtering a particular patch to remove it if it does not have enough corresponding patches in other source views, indicating that the patch is an isolated error that may be a result of estimation noise.

[0042] This can help to identify foreground patches that are incorrectly detected. For example, after projecting the patch onto a common world coordinate system, one can calculate the number of patches from other source views that are closer than the patch's minimum world-space (Euclidean) distance. If this number is smaller than a patch count threshold (e.g., a given percentage of the number of source views), the patch is discarded. For example, if a "foreground object" is identified in fewer than three of the eight source views, the patch for that particular foreground object is discarded. When a patch is discarded, it is not used in the atlas.

[0043] Generating the one or more patches may include identifying a subregion in the source view depth map, determining a number of depth surfaces of different depths present in the subregion, and generating a patch for each depth surface in the subregion, each patch having a different patch transparency map.

[0044] Alternatively or additionally, generating the one or more patches may include identifying a sub-region in the source-view image.

[0045] The present invention further provides a system having one or more processors comprising computer program code that, when executed on a computing device, causes the computing system to perform the aforementioned method, and a number of sensors configured to acquire source view images and a source view depth map.

[0046] The present invention provides a method for rendering multi-view video, the method comprising: receiving an atlas having a plurality of patches and a background model of the 3D scene, each patch corresponding to a foreground object, each patch having a patch texture image, a patch depth map, and a patch transparency map derived from a source view image and the source view depth map; receiving a virtual viewpoint within a 3D scene; The method includes the steps of sorting the patches based on a difference between a position of a virtual viewpoint and a position of a foreground object corresponding to each patch, and rendering a background model and the sorted patches.

[0047] The rendering method may further include grouping the patches based on a position of the corresponding foreground object relative to the virtual viewpoint.

[0048] The step of rendering the background model and the sorted patches includes the steps of rendering the background model, warping and / or blending the first patch group, compositing the warped and / or blended first patch group onto the rendered background model, warping and / or blending the second patch group, where a position of a foreground object corresponding to the second patch group relative to the virtual viewpoint is closer to the virtual viewpoint than a position of a foreground object corresponding to the first patch group, and compositing the warped and / or blended second patch group onto the warped and / or blended first patch group.

[0049] The method may further include receiving metadata including a position and geometry of each patch in the atlas and a position and geometry of each patch in the source view image and / or source view depth map, and rendering the patches is based on both the position and the geometry.

[0050] The present invention also provides a computer program product comprising computer program code which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of a depth segmentation method for generating multi-view video data and / or a multi-view video rendering method, and a processor configured to execute said computer program code.

[0051] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]

[0052] For a better understanding of the present invention and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Figure 1]1 illustrates a method for depth segmentation for multiview video. [Diagram 2] FIG. 2 illustrates a source view image and a source view depth map. [Diagram 3] FIG. 3 shows patches corresponding to the triangular object of FIG. 2. [Figure 4] FIG. 2 illustrates a first process for generating a patch. [Diagram 5] FIG. 1 shows a source view depth map with three foreground objects. [Figure 6] A diagram showing four patch transparency maps generated for a single region. [Figure 7] Diagram showing steps in warping and blending patch groups from an atlas. [Figure 8] FIG. 1 illustrates a 3D scene with two source views and two target views. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0053] The present invention will now be described with reference to the drawings.

[0054] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems and methods, are for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems and methods of the present invention will become better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.

[0055] The present invention provides a method of depth segmentation for generating multi-view video data. The method includes obtaining a plurality of source-view images and a source-view depth map representing a 3D scene from a plurality of sensors. Foreground objects in the 3D scene are segmented from the source-view images (102) and / or the source-view depth map (104). Then, one or more patches are generated for each source-view image and source-view depth map that includes at least one foreground object, each patch corresponding to one foreground object, and generating the patches includes generating a patch texture image, a patch depth map, and a patch transparency map based on the source-view images and the source-view depth map.

[0056] FIG. 1 shows a method of depth segmentation for multi-view video. The method is based on creating an atlas 112 that contains all the data required to render a multi-view video. Typically, multiple source views are needed to render a multi-view image frame. Each source view typically has a source view image 102 and a source view depth map 104. The depth map 104 can be derived using multi-view image-based matching, or alternatively, by adding one or more depth sensors (e.g., laser depth sensors) to a multi-camera system setup (used to acquire source view data) and warping the measured depth at each source view, after which filtering / filling can be used to make the depth map complete for each source view.

[0057] We propose to segment all source view depth maps 104 based on their difference from a globally determined background model 106. The background model 106 is used to generate a background depth map for each source view. Then, the foreground objects 108 are further segmented based on the relative depth difference between pixels in the background depth map and the source view depth map 104. Instead of generating a single hierarchical representation, segmented patches 110 of the foreground objects 108 are kept for all source views and packed together in an atlas 112.

[0058] The client device can sort the patches 110 along the z-axis of the new virtual viewpoint, and the view synthesis algorithm can then visit the patches 110 in this order and alternate between blending patches 110 from different source views if they have similar depths, and compositing the blended view onto a previously synthesized output.

[0059] FIG. 2 shows a source-view image 102 and a source-view depth map 104. FIG. 2(a) shows the source-view image 102, and FIG. 2(b) shows the source-view depth map 104. The source-view image 102 and the source-view depth map 104 are of a 3D scene that includes a background 206 and two foreground objects (i.e., a rectangle 202 and a triangle 204). As can be seen in FIG. 2(b), the depth of the background 206 varies, while the depths of the two objects 202 and 204 remain constant. The lower section of the rectangular object 202 has a similar depth as the nearest section of the background 206.

[0060] 3 shows a patch 110 (as shown in FIG. 1) corresponding to a triangular object 204 (as shown in FIG. 1). For each patch 110, a patch texture image 302, a patch depth map 306, and a patch transparency map 304 (e.g., alpha channel) are generated. FIG. 3(a) shows the patch texture image 302 of the triangular object 204. FIG. 3(b) shows the patch transparency map 304 corresponding to the triangular object 204 of FIG. 2, where black parts of the patch transparency map 304 indicate fully transparent areas and white parts indicate opaque areas. FIG. 3(c) shows the patch depth map 306 of the triangular object 204.

[0061] The patch texture image 302 and patch depth map 306 may be generated by directly copying data from the source view image 102 (FIG. 2) and source view depth map 104 (FIG. 2) in the sections corresponding to the patch 110. The transparency value may be set to 1 where the triangle object 204 is present and 0 where it is not.

[0062] Alternatively, a more accurate algorithm can be used to estimate the foreground color and alpha (transparency) using so-called alpha matting for each patch 110. In that case, the color of pixel i may be a linear combination of the local foreground color F and background color B based on the pixel's transparency value α, as follows:

number

[0063] A trimap can be constructed based on the object component label map for each pixel in each patch 110. The trimap consists of the classes "definitely foreground" (α=1), "definitely background" (α=0) and "uncertain" (α needs to be estimated). An alpha matting algorithm then mattes α for the uncertain pixels. i and F i Both are estimated.

[0064] When the patch depth map 306 is used for depth image-based rendering, the triangular object 204 may be obscured behind a rectangular "rest" (e.g., the bottom left rectangular portion of FIG. 3(c)). This is because the pixels of this rest area are closer to the camera than the pixels of the triangular object 204. This is related to the view synthesis logic, which renders the patch 110 using a triangular mesh whose each vertex depends on the patch depth map 306.

[0065] To solve this problem, it is advantageous to process the patch depth map 306 such that the "left" regions of the other patches 110 are modified to a depth value that is at least further away than the local foreground of the corresponding foreground object of the patch 110. The modified patch depth map 308 of FIG. 3(c) shows a simple method of simply setting all pixels outside the triangular object 204 to a depth value equal to the minimum value taken across all valid object pixels. In this example, the entire modified patch depth map 308 receives the constant depth that the triangular object 204 has. Alternatively, all pixels that do not correspond to the triangular object 204 may be set to a depth value that is lower than any depth value that corresponds to the triangular object 204.

[0066] In this example, the approach of modifying depth pixels outside the object region but inside the rectangle arises from the design of the client rendering system: client rendering applications typically warp each rectangle as a single entire regular mesh (ignoring which pixels correspond to the background). Pixels outside the object region that are close to the camera result in the mesh being wrapped onto the object itself.

[0067] Alternatively, so-called geometry shaders can be used to select / crop triangles during rendering, so that only pixels with an object label are warped. However, this is more complicated for implementation in a real-time renderer.

[0068] If the depth of the triangular object 204 does not change, or only changes slightly, then the patch depth map 306 does not need to be stored as a map, and a scalar depth value can be indicated for the entire patch 110 .

[0069] FIG. 4 shows a first process for generating patches 110 (FIG. 1). Each source view is assumed to have an associated background depth map 106 (FIG. 1). In the case of a sports stadium, this may be a depth map rendering of a ground surface model combined with a geometric model of the stadium. FIG. 4(a) shows a source view depth map 104 with two foreground objects 202 and 204 (e.g., athletes). The foreground objects 202 and 204 can be detected by subtracting the depth of the background model 106 from the estimated source view depth map 104 and thresholding the result. FIG. 4(b) shows the thresholded map 402. As can be seen in FIG. 4(b), the foreground objects 202 and 204 that occlude each other are still joined together in the (binary) threshold map 402.

[0070] To separate the joined objects, the depth step edge pixels are detected and set, for example, to zero. FIG. 4(c) shows a threshold map 404 with the depth step pixels set to zero. Note that the binary mask of local background pixels is set to zero at the boundaries of the depth step, and not the pixels corresponding to the local foreground. This is done to prevent foreground pixels from being lost. The labeling of continuously connected components separates the foreground objects 202 and 204 into segments from the background 206 and from each other. FIG. 4(d) shows an image 406 with segments labeled for each object (the background 206 labeled as 0, the triangular object 204 labeled as 1, and the rectangular object 202 labeled as 2).

[0071] A bounding box 408 can then be detected for each segment. FIG. 4(e) shows the source view depth map 104 with bounding boxes 408 for the foreground objects 202 and 204. Each bounding box 408 can then be further extended vertically downward to include sections of the rectangular object 202 that touch (or are near) the ground depth of the background 206. FIG. 4(f) shows the source view depth map 104 with extended bounding boxes 410. Patches 110 can then be determined based on the bounding boxes 408 and / or the extended bounding boxes 410. By extending the bounding boxes 408, the bounding boxes 408 can further enclose any portions of the foreground objects 202 and 204 that were cut off in the thresholding step (e.g., due to portions of the foreground objects 202 and 204 being close to the background 206).

[0072] 5 shows a source view depth map 104 with three foreground objects 202, 204 and 502. A rectangular grid 504 is shown that defines a sub-region 506 of the source view depth map 104 that contains all three foreground objects.

[0073] An alternative approach for generating patches 110 (FIG. 1) uses sub-regions 506, such as the sub-region 506 shown in FIG. 5. Depending on the number of depth planes present in the sub-region 506, one or more patches 110 are constructed corresponding to the sub-region 506. FIG. 5 shows an example sub-region 506 in which four patches are required to model all depth planes in the sub-region 506 (i.e., the background 206 and the three foreground objects 202, 204 and 502).

[0074] Having multiple patches per subregion 506 allows multiple patches to share the same patch texture image 302, and potentially the same patch depth map 304, thus reducing the overall amount of data that needs to be broadcast. Additionally, the spatial relationship between patches can also be defined by a grid (of subregions 506), instead of having to define the position of each patch.

[0075] A background depth map generated from the background model 106 (FIG. 1) can be, but is not required to be, used to segment the foreground objects 108. In the example of FIG. 5, instead of a background model, classification, clustering and / or binning of pixels based on their depth values ​​in the source view depth map 104 can be used to identify respective segments, thereby generating multiple patches for each sub-region 506.

[0076] The number and positions of sub-regions 506 may depend, for example, on an object detection algorithm (or the like) that detects foreground objects in source-view image 102 (FIG. 1) or source-view depth map 104. Alternatively, the number and positions of sub-regions 506 may be fixed based on known positions of foreground objects 202, 204, and 502.

[0077] Figure 6 shows four patch transparency maps generated for a single subregion 506 (Figure 5). All four patch transparency maps 602, 604, 606 and 608 are for the subregion 506 shown in Figure 5 and each correspond to a different patch 110 (Figure 1). In this example, the four patches 110 (and thus the four patch transparency maps) are generated for the single subregion 506 because of the presence of four depth planes in the subregion 506 (the background and three objects).

[0078] Figure 6(a) shows a first patch transparency map 602 for a thin object 502 (Figure 5). Figure 6(b) shows a second patch transparency map 604 for a rectangular object 202 (Figure 5). Figure 6(c) shows a third patch transparency map 606 for a triangular object 204 (Figure 5). Figure 6(d) shows a fourth patch transparency map 608 for a background 206 (Figure 5).

[0079] Thus, when a region of the source view depth map 104 contains multiple depth planes, multiple patches 110 may be generated, each plane resulting in a separate patch 110. Although only a transparency (alpha) map is shown in Figure 6, the patch texture image 302 (Figure 3) only needs to be stored once, since it is the same for all four patches 110. Furthermore, for some patches 110 a scalar depth map may be sufficient, while for other patches a (difference) patch depth map may be required.

[0080] A single background model 106 (video sprite texture and depth) can be constructed from the source view by taking into account the fact that it is known where foreground patches 110 have been removed and using this to fill in any gaps. For example, when multiple cameras film a hockey game, a single background sprite image can be generated that includes only the ground and spectators but not the players. This single background sprite can be modeled with a perspective projection that has a wider field of view than the source view. The background sprite and depth can be packed into a single atlas 112 along with the source view patches 110 (FIG. 1).

[0081] View Synthesis View synthesis of a patch 110 begins after decoding atlas 112 data, which includes patch texture images 302, patch transparency maps 304 (FIG. 3) and patch depth maps 306 (FIG. 3) of the same foreground object 108 for multiple source views. For example, a foreground object 108 may be visible in only five of the total eight source views. The background model 106 is known to always be the farthest object in a 3D scene, and is therefore rendered in front of all other patches 110.

[0082] Given a target view matrix (which defines the position and orientation from which the 3D scene is viewed), the patches 110 are first sorted in decreasing order based on their distance from the virtual viewpoint (z-axis). The sorted patches then form patch groups, where the variation in z-coordinates within a group is typically smaller than the variation in z-coordinates between patch groups. Note that patches 110 from multiple source views will end up in the same group depending on the virtual viewpoint.

[0083] View synthesis then alternates between warping and / or blending groups of patches and compositing the blended result onto the previous synthesis result.

[0084] Figure 7 shows the steps of warping and blending a patch group from the atlas 112. After fetching a patch 110 shown in Figure 1 in a patch group from the atlas 112 (i.e., patch texture image 302, patch transparency map 304 (Figure 3), and patch depth map 306), the patch 110 is warped to its associated source view buffer using the target view matrix. The result is then expressed directly in target view image coordinates.

[0085] For purposes of illustration, only the patch texture image 302 and the patch depth map 306 are shown in the atlas 112. The patch transparency map 304 may be included in the atlas 112 as well, or may be embedded in the patch depth map 306, for example.

[0086] Each patch 110 is warped to its associated source view buffer 702, and all (or some) of the source view buffer 702 is used to composite the foreground object 108 corresponding to the patch group onto the background model 106 (FIG. 1) if no other patch groups have been composited, or onto a previously composited image 704 (if the patch group has already been composited).

[0087] The number of source view buffers 702 used to composite foreground objects 108 (FIG. 1) may be fixed to maintain constant memory usage during compositing. For example, only eight of the source view images may be selected to perform compositing (based on the proximity of the source view to the target view).

[0088] Figure 8 shows a 3D scene with two source views 800 and two target views 808 and 810. Figure 8(a) shows a first target viewpoint 808 and Figure 8(b) shows a second viewpoint 810. Figure 8 illustrates the need to sort patches 110 (Figure 1) on the client side based on the coordinates of the target views 808 or 810. The target views 808 and 810 are sometimes referred to as virtual viewpoints or virtual views.

[0089] For the first target view 808 shown in Figure 8(a), objects 804 and 806 fall into the same patch group that is warped and blended first. Object 802 is closest to the coordinates of the first target view 808 and is therefore warped, blended, and composited last.

[0090] However, this is different for the second target view 810 shown in Figure 8(b) . For the second target view 810, because object 806 is closer to the second target view 810 than objects 802 and 804, objects 802 and 804 are warped, blended, and composited first, and object 806 is warped and composited last.

[0091] Metadata Metadata can also be stored for each patch 110. For example, for each patch 110, a source view identifier, a source view position and size (u 0,v , v 0,v , w v , h v ), as well as the location and size (u 0,a , v 0,a , w a , h a ) may be stored.

[0092] (u 0,a , v 0, a Let (u, v) represent the bottom left corner of the rectangular patch 110 in atlas coordinates. Therefore, we only need to sample the atlas coordinates that lie within the warped rectangle for a given patch size. Let the normalized coordinates (u, v) be in the domain [0,1], then the normalized atlas coordinates (u a , v a ) can be calculated as follows:

number

[0093] Atlas coordinates (u a , v a ) is used to access depth values ​​in the patch depth map 306 (FIG. 3) during the vertex shader stage and to interpolate color and transparency by passing the atlas coordinates to the fragment shader.

[0094] However, to warp the patch 110 to the output view, we may need to know the coordinates of the rectangular source-view image 102 (FIG. 1). Given that the normalized coordinates (u, v) are in the domain [0,1], we can find the normalized atlas coordinates (u v , v v ) can be calculated as follows:

number

[0095] Note that because normalized coordinates are used, the source-view image 102 and / or depth map may have fewer or more pixels than those stored in the atlas 112. With a fixed pixel budget in the atlas 112, the patches 110 can always be scaled to fit.

[0096] Those skilled in the art can readily develop a processor to perform any of the methods described herein. Thus, each step of the flowchart may represent a different action performed by a processor and may be performed by a respective module of the processing processor.

[0097] As mentioned above, the system utilizes a processor to process data. The processor may be implemented in various ways using software and / or hardware to perform the various functions required. The processor typically uses one or more microprocessors that are programmed using software (e.g., microcode) to perform the required functions. The processor may also be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

[0098] Examples of circuitry that may be used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0099] In various implementations, the processor may be associated with one or more storage media, e.g., volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when executed on the one or more processors and / or controllers, perform the required functions. The various storage media may be attached within the processor or controller, or may be transportable such that one or more programs stored on the storage media can be loaded into the processor.

[0100] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0101] A single processor or other unit may fulfill the functions of several items recited in the claims. The computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, provided together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0102] When the term "adapted for" appears in the claims or the specification, it is meant to be synonymous with the term "configured to."

[0103] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A method of depth segmentation for generation of multi-view video data, comprising: acquiring a plurality of source-view images and a source-view depth map representing a 3D scene from a plurality of sensors; segmenting foreground objects in the 3D scene from the source-view image and / or the source-view depth map; generating one or more patches for each source-view image and source-view depth map including at least one foreground object, each patch corresponding to one foreground object, generating a patch texture image, a patch depth map, and a patch transparency map based on the source-view image and the source-view depth map, each patch being based on a section of a corresponding source-view image that is smaller than the source-view image; obtaining a background model representing the background of the 3D scene, the background model including at least one background depth map and background texture data; generating an atlas based on the patch texture image, the patch depth map, the patch transparency map and the background model; The method according to claim 1,

2. 2. The method of claim 1, further comprising obtaining a plurality of background depth maps of the 3D scene representing a background of the 3D scene, each background depth map including depth data of the background from a particular orientation, and wherein segmentation of foreground objects is based on a difference between a background depth map and a corresponding source view depth map.

3. subtracting the background depth map from each corresponding source view depth map to generate respective difference images; thresholding the difference image, where the thresholding compares pixel values ​​of the difference image to a threshold to generate a threshold map that distinguishes between background and foreground objects; identifying a pixel in the source view depth map that corresponds to a depth step, the depth step being defined by a difference between adjacent depth values ​​in the source view depth map that is greater than a depth threshold; marking the depth steps in the depth map as background to distinguish the foreground objects from each other; generating a bounding box around the foreground object based on the adjusted threshold map; The method according to claim 1 or 2, further comprising detecting foreground objects by

4. The method of claim 3 , wherein detecting a foreground object further comprises expanding the bounding box to include an area of ​​the foreground object in the difference map that is below the threshold.

5. The method of claim 1 or 2, further comprising the step of adapting pixel depth values ​​of a patch depth map such that all pixel depth values ​​of the patch depth map are of values ​​less than or equal to a depth value of a corresponding foreground object.

6. identifying a plurality of patches from the patches corresponding to a first foreground object based on identifying the patch depth map of the patches within an object depth range; and modifying the identified patch depth map to correspond to object positions in the 3D scene.

7. The method of claim 1 or 2, further comprising pruning the patches based on a measure of consistency between the patches in multiple source views.

8. 3. The method of claim 1 or 2, wherein generating one or more patches comprises identifying a subregion in a source view depth map, determining a number of depth planes of different depths that exist in the subregion, and generating a patch for each depth plane in the subregion, each patch having a different patch transparency map.

9. one or more processors comprising computer program code that, when executed on a computing device, causes said computing device to perform all the steps of the method according to claim 1 or 2; and a plurality of sensors configured to acquire the source-view image and the source-view depth map.

10. 1. A method for rendering multi-view video, comprising: receiving an atlas including a plurality of patches and a background model of the 3D scene; Each patch corresponds to one foreground object, each patch having a patch texture image, a patch depth map and a patch transparency map derived from a source view image and a source view depth map; each patch being based on a section of a corresponding source-view image that is smaller than the source-view image; receiving a virtual viewpoint within the 3D scene; sorting the patches based on a difference between a position of the virtual viewpoint and a position of a foreground object corresponding to each patch; and rendering the background model and the sorted patches.

11. The method of claim 10 , further comprising grouping the patches based on a position of a corresponding foreground object relative to the virtual viewpoint.

12. Rendering the background model and the sorted patches comprises: rendering the background model; warping and / or blending a first group of patches; compositing the warped and / or blended first group of patches onto the rendered background model; warping and / or blending a second patch group such that a position of a foreground object corresponding to the second patch group relative to the virtual viewpoint is closer to the virtual viewpoint than a position of a foreground object corresponding to the first patch group; compositing the warped and / or blended second patch group onto the warped and / or blended first patch group; 12. The method of claim 11, having the following structure:

13. A computer program comprising computer program code when executed on a computing device having a processing system, the computer program code causing the processing system to perform the method according to claim 1 or 2 and / or the method according to any one of claims 10 to 12.

14. A processor configured to execute the computer program of claim 13.

Images