Method for processing depth map

Abstract

A method for processing a depth map for subsequent transmission is provided. Depth regions in the depth map are identified, each of the depth regions defining a portion of the depth map. A range of depth values is determined for the depth region and further transmitted with the depth map. The range of depth values provides segmentation information for a depth region in the depth map.

Description

Technical Field

[0001] The present invention relates to processing depth maps. In particular, the present invention relates to processing depth maps for subsequent transmission and use of the depth maps. Background Art

[0002] Multi-view imaging generally refers to the imaging of a scene and obtaining the geometry of the scene. Typically, a depth map or similar is used to capture the geometry of the scene. The depth of the scene together with the image of the scene can be used to synthesize a new image at a new viewpoint. In other words, multi-view imaging can be used to view a scene from a selected viewpoint, independent of the camera's viewpoint.

[0003] Multi-view imaging may include segmenting an image with depth into separate tiles. The image and depth tiles are packed in atlases. On the client side, the tiles are first sorted in order of decreasing value of the z-axis of the virtual viewpoint (i.e., the viewpoint to be synthesized). The view synthesis algorithm then accesses the tiles in that order and alternates between the mixed tiles from different source views when they have similar depths, and synthesizes the mixed view on the previously synthesized output.

[0004] However, placing a variable number of tiles per frame with irregular shapes in an atlas destroys spatial and temporal consistency (thus increasing bitrate) and increases the number of pixels required to store pixel data related to the segmentation. An atlas usually consists of a background sprite texture with a deep model and a variable number of rectangles, where transparency is used to encode the irregular tile shapes within the rectangles, or a separate segmentation map is stored in the atlas.

[0005] Additionally, actually placing tiles for texture, depth, and segmentation is like a puzzle, which can be computationally demanding due to bitrate and pixel area constraints. While this may not be a problem for professional broadcast applications with sufficient computing resources (e.g., using cloud computing), it may be a problem for remote presentation applications with real-time, low latency, and low computational complexity requirements.

[0006] Finally, most existing video standards do not support alpha channels by default. Alpha channels provide transparency values ​​for pixels. Therefore, segmenting shapes will consume considerable pixel space. If full view textures and full view images are placed directly into image frames, the total number of views that can be transmitted will be limited. For example, encoders typically support 4K at 60Hz or 8K at 30Hz.

[0007] Therefore, there is a need to improve the segmentation methods used in multi-view imaging.

[0008] US 10855965 B1 discloses a segmented 3D multi-view image generator which generates fewer multi-view images for partitions having less salient features.

[0009] US 2019 / 158838 A1 discloses efficient wedgelet-based coding for encoding modules of different sizes. Summary of the invention

[0010] The invention is defined by the claims.

[0011] According to an example of one aspect of the present invention, there is provided a method for processing a depth map for subsequent transmission, the method comprising:

[0012] Obtaining a depth map of a scene comprising a plurality of depth values;

[0013] identifying one or more depth regions in the depth map, wherein each of the one or more depth regions defines a portion of the depth map;

[0014] determining one or more depth value ranges for the one or more depth regions; and

[0015] The depth map and the one or more depth value ranges are transmitted.

[0016] Improved multi-view frames can be rendered by using depth maps with segmented objects. However, segmentation is usually done at the encoder before transmitting / broadcasting multi-view information (e.g., texture and depth), so the multi-view information must include segmentation information so that the decoder can use the segmentation during rendering.

[0017] It has been found that providing a depth range for each object essentially provides depth segmentation for depth regions with much lower bit rate requirements than, for example, providing a segmentation mask. This is because a segmentation mask (e.g., a segmentation map) does not need to be transmitted to provide the segmentation information. The depth value ranges enable segmentation to be efficiently implemented at the decoder by comparing the depth values ​​in the depth map with the transmitted depth value ranges. When the depth value falls within one of the ranges, it can be assumed that the depth value corresponds to the object from which the depth value range was obtained.

[0018] Additionally, this provides multi-view information that can be encoded at relatively low computational cost.

[0019] The identified depth region is a continuous portion of the depth map and may include a portion / part of the entire object.

[0020] The depth region may include a region in the depth map having a relatively smooth depth gradient. Preferably, the depth region consists of an object or a part thereof.

[0021] The depth value range may be transmitted as two values ​​indicating the maximum and minimum depth values. Of course, other low bit rate methods of transmitting depth value ranges will be known. In particular, transmitting a range of depth values ​​for a depth region (rather than identifying which pixels have which depths within the range) provides a low data way to implement the segmentation function. Each depth value range may correspond to a single depth region, or the same range may be shared between different depth regions. An indication of which depth region corresponds to which depth value range may also be transmitted.

[0022] The method also includes defining a plurality of cells in the depth map, each cell corresponding to a portion of the depth map, wherein identifying one or more depth regions in the depth map includes identifying one or more depth regions in each of the cells.

[0023] Additionally, cells defined in the depth map that correspond to only a portion of the depth map can encode different portions of the depth map separately. This can reduce the processing power required to render multi-view frames using depth maps by rendering each frame based on a corresponding number of depth value ranges in each cell. For example, cells containing only background depth information (i.e., a single range of the entire cell) can be accurately rendered without using back-to-front synthesis, whereas cells containing depth regions in more than one range are preferably rendered using back-to-front synthesis.

[0024] Preferably, the depth map is divided into more than 15 cells.

[0025] Defining a plurality of cells in the depth map may include dividing the depth map into a grid of cells.

[0026] The cell grid includes a plurality of cells, wherein the size of each cell is known. For example, the grid may include a plurality of cells of equal shape.

[0027] Knowing the size of each cell in the grid enables more efficient encoding and rendering of multi-view frames using depth maps, since the position of each cell is known and consistent between frames.

[0028] The method also includes transmitting a number of different depth value ranges in each cell.

[0029] 10. The depth interval data should be compressed and decompressed with low computational complexity (and latency), but should not take up too much space either. In a binary data format, this can be achieved by indicating where the depth range for one cell ends and where the depth range for another cell begins. The way to achieve this is to have a 4-bit number for each cell (maximum 16 intervals per cell) that indicates how many intervals there are in the cell.

[0030] Only 16 or fewer different depth value ranges may be transmitted.

[0031] This enables all indications of the range to be encoded in four bits. For example, a depth value range may be encoded using 16 bits each.

[0032] It seems realistic to assume a limited number of depth ranges occurring within a cell, e.g., a maximum of 4, 10, 12, or 16. Using this information will help reduce the bitrate with limited encoding / decoding complexity, thus also adding only a relatively small amount of latency.

[0033] The depth range can vary between cells. For example, on one side of the depth map there may be close objects, and on the other side there may be objects further away. Therefore, it may be advantageous to have a specific depth range for each cell. Having many cells per depth map and / or many depth intervals per cell may also introduce greater overhead for the depth range.

[0034] Identifying one or more depth regions may be based on determining depth smoothness between depth values ​​of the depth map.

[0035] For example, first and / or second order derivatives can be used to determine how quickly depth changes as a function of image coordinates. Depth variations can be allowed within a fragment only if the local variations are smooth. As the variation in depth from one pixel to the next increases, this variation will result in an increased chance of coverage / occlusion or exposure / unocclusion during rendering. Fundamentally, depth smoothness is used to predict from a depth map in sensor space (i.e., at the time the depth map is acquired) where the corresponding real-world 3D object has a continuous surface.

[0036] The method may also include determining a grid resolution based on the depth smoothness and transmitting the grid resolution.

[0037] The present invention also provides a method for synthesizing an image at a target viewpoint, the method comprising:

[0038] receiving at least one or more images and one or more depth maps of a scene;

[0039] receiving one or more depth value ranges for at least one of the depth maps, wherein each depth value range corresponds to one or more depth regions of one of the depth maps; and

[0040] At least one of the depth maps is processed using the range of depth values ​​along with at least one of the images to synthesize an image at the target viewpoint.

[0041] Processing the depth map based on its depth value range enables the rendering step to provide more accurate results, since the depth regions now correspond to the depth value ranges. Thus, any errors in the depth values ​​that may occur during rendering, for example due to warping and / or blending steps, can be corrected by covering the depth value range.

[0042] Additionally, receiving depth value ranges for various depth regions enables the decoder side to process depth data more efficiently. For example, the value ranges can be used to segment the received depth map. Having depth segmentation for the depth map can provide more accurate and efficient rendering of multi-view frames. Vertex shaders and fragment shaders typically used for rendering can also utilize depth value ranges, as will be explained below.

[0043] For example, processing at least one of the depth maps using the depth value range may include modifying positions of vertices in one or more meshes generated based on the depth map to fit within the corresponding depth value range, and / or modifying transparency values ​​in a transparency map generated based on the depth map based on the one or more depth value ranges.

[0044] Processing at least one depth map may include presenting depth regions in one or more depth maps in a back-to-front depth order.

[0045] Processing at least one of the depth maps may include: selecting a mesh for each depth value range; applying depth values ​​of one or more depth regions corresponding to each depth value range to the corresponding mesh based on the depth map; modifying positions of vertices in each mesh to fit within the corresponding depth value range; and synthesizing an image at a target viewpoint using the image and the modified one or more meshes.

[0046] The depth variation within a region (i.e., the depth range) can be used to anticipate what mesh resolution will be used. In practice, generating a mesh means allocating memory for a given mesh size. Applying depth values ​​to a mesh means computing 3D (x, y, z) mesh coordinates based on the incoming depth map (e.g., unprojecting the depth to x, y, z coordinates).

[0047] For example, the mesh may be modified by a vertex shader during rendering. The depth value range provides constraints on the positions of vertices in the mesh. This means that erroneous depth values ​​that are outside the depth value range are modified to be within the range. As a result, the accuracy of the mesh is increased.

[0048] This is advantageous because the mesh can be used for warping operations in multi-view imaging. Therefore, having accurate vertex coordinates in the mesh provides more accurate warping for the target viewpoint. Of course, the depth value range can also be used by other geometry-based shaders / operations in the graphics pipeline to modify the coordinates / geometry of the mesh, etc.

[0049] Assuming a cell size of 128x128 pixels, various grid sizes may be generated (e.g., 128x128 grids, 64x64 grids, 32x32 grids, 16x16 grids, and 8x8 grids are stored in memory). Depending on the smoothness within the depth region, the depth region may use the lowest resolution grid possible.

[0050] Depth map smoothness is a relative concept. Consider the depth map of a soccer ball. If the mesh is too coarse, texture rendering errors may become visible. So, although the soccer ball surface is locally quite smooth, the curvature is still considerable and could benefit from a higher resolution mesh. In another example, a book is oriented with the cover parallel to the depth image sensor. In this case, the entire cover can be perfectly represented by a single quadrilateral (or two triangles) in the mesh.

[0051] A grid resolution to be used for a particular depth region may be received along with a range of depth values ​​for the depth region.

[0052] The method may also include obtaining a grid resolution based on a depth smoothness of each of the one or more depth maps, wherein selecting the grid for each depth value range is based on the grid resolution.

[0053] A depth map with high depth smoothness (ie, not much variation between pixels) may not require as many vertices in the mesh, since gradual depth variations can be accurately depicted with a small number of vertices.

[0054] Preferably, the depth map is divided into various cells and depth segmentation is performed for each cell at the encoder. The fragments obtained from the segmentation are encoded using multiple intervals (ie depth value ranges) for each cell. Each interval isolates a region of the cell.

[0055] Each region consisting of multiple depth pixels has certain smoothness properties. For example, a given region may consist of constant depth, so a mesh consisting of a single quadrilateral (two triangles) is sufficient. However, another region may contain ridges or curved surfaces. In order to represent this interval / region, a mesh with a higher spatial resolution will be required. This enables parts of the depth map with high depth smoothness (depth regions inside cells) to use meshes with fewer vertices, thereby reducing computational complexity, while parts of the depth map with lower depth smoothness (higher variability in depth values) can use meshes with more vertices to improve accuracy.

[0056] Processing at least one of the depth maps may include warping the one or more depth maps to a target viewpoint, generating a transparency map using the one or more warped depth maps, and modifying transparency values ​​in the transparency map based on one or more depth value ranges, wherein synthesizing an image at the target viewpoint is based on using the one or more warped depth maps and the transparency map.

[0057] Transparency maps are typically used to set different pixels to be transparent (or partially transparent) for different layers in a multi-view frame based on a layered format. The transparency value can be modified so that when the depth value in the depth map is outside the corresponding depth value range, the corresponding transparency value is set to 100% transparent. This makes erroneous depth values ​​(e.g., caused by warping and / or blending steps) invisible in the multi-view frame.

[0058] Additionally, modification of the transparency map substantially segments out foreground edges for objects in each layer, since objects in a different layer will likely have depth values ​​outside the range of depth values ​​corresponding to that layer.

[0059] For example, this may be implemented at the fragment shader of a graphics pipeline.Of course, other pixel-based operations are suitable for being based on depth value ranges.

[0060] The method may also include receiving an indication of a number and shape of cells in a grid of cells defined for each of the one or more depth maps, wherein rendering the depth region includes rendering the depth region for each cell separately.

[0061] The present invention also provides a computer program comprising computer program code, which, when running on a processor, causes the processor to perform all the steps of any (or all) of the aforementioned methods.

[0062] The present invention also provides a processor configured to run the above-mentioned computer program code.

[0063] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] For a better understanding of the invention, and in order to more clearly show how the invention may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0065] Figure 1 illustrates an example partition with a grid drawn on a depth map;

[0066] Figure 2 A single grid cell of the grid is shown;

[0067] Figure 3Shown are depth regions A to C with markings Figure 2 The grid cells;

[0068] Figure 4 Pictured Figure 2 and Figure 3 The one-dimensional profile of the cell shown has a corresponding range of depth values;

[0069] Figure 5 Shown in Figure 2 and 3 A low-resolution grid drawn on the cells of

[0070] Figure 6 The figure shows a grid drawn for a depth region A;

[0071] Figure 7 The diagram shows a grid drawn for depth region B;

[0072] Figure 8 The figure shows a grid drawn for a depth region C;

[0073] Fig. 9 A method for processing a depth map at an encoder is shown; and

[0074] Fig.10 A method for synthesizing an image at a target viewpoint is shown. DETAILED DESCRIPTION

[0075] The present invention will be described with reference to the accompanying drawings.

[0076] It should be understood that the detailed description and specific examples, although indicating exemplary embodiments of the device, system and method, are intended for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the device, system and method of the present invention will be better understood from the following description, the appended claims and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar components.

[0077] The present invention provides a method for processing a depth map for subsequent transmission. A depth region is identified in the depth map, each of the depth regions defining a portion of the depth map. A depth value range is determined for the depth region and further associated with the depth region. Figure 1 The depth value range provides segmentation information of the depth region in the depth map.

[0078] In order to avoid the additional pixel space requirements of segmentation and avoid the computational complexity associated with data packets, it is proposed to send the segmentation information as a set of thresholds for the depth map rather than as a spatial map. In particular, the segmentation information is sent as depth value ranges, where each depth value range contains one or more objects in the scene. Preferably, the threshold set is specified for a predefined area geometry (e.g., a rectangle). An approximate segmentation map can then be reconstructed by applying the depth value range to the depth map after decoding by the client. The depth value range can be sent as metadata. Note that the depth value range can be referred to as a threshold or a depth interval.

[0079] The first step can be to create an upper bound on the segmentation size and shape by cutting the image domain into patches of a regular grid. For example, assuming the image size is 1920x1080 pixels, a grid cell of 128x128 pixels can be used.

[0080] In fact, the image and the corresponding depth Figure 2 The texture and depth can thus be defined for each grid cell and then used on the client side to synthesize a new image at the target viewpoint. However, in order to define the segmentation information in this case, processing of the depth map is sufficient.

[0081] Figure 1 An example partition with a grid 102 drawn on a depth map 100 is illustrated. The grid 102 defines a plurality of grid cells. It should be noted that the depth map 100 may have a lower resolution than the corresponding image, for example, 4 times lower. Thus, the depth map may have a grid cell size of 32x32 pixels.

[0082] Figure 1 Person 104 and table 106 are shown, wherein the shadow of table 106, compared to the shadow of person 104, indicates that table 106 is closer to the viewpoint of depth map 100 than person 104. Background 108 is also shown without any shadows. This may indicate that the depth value of background 108 is uncertain, or that the depth value is outside a predefined depth domain.

[0083] Most grid cells will not contain occluding edges at all, and depth segmentation and a back-to-front rendering approach may not be needed. However, some grid cells do contain occluding edges. For such cases, during rendering, the more distant objects (e.g., background 108) need to be drawn first, and in a later rendering step, the closer objects (e.g., person 104 and table 106) need to be synthesized over the drawn results. Therefore, in such cases, having segmentation information during rendering is significantly advantageous because it enables more accurate rendering of edges between objects at different depths.

[0084] There are also computational complexity benefits to using depth segmentation information in combination with back-to-front view synthesis with compositing. At the client device, the depth map is typically converted to a regular mesh in the vertex shader. For occluding edges, the mesh needs to have many vertices, and since the mesh is regular, the vertex shader becomes computationally expensive. For each view, millions of vertices may need to be rendered for each frame. In contrast, depth segmentation reduces the depth variations (inside the grid cells) and therefore reduces the number of vertices required for the occlusion depth step. The foreground / background transition (i.e., the depth step) can be efficiently transferred to the transparency map of the grid cells. Figure 1 For many cells shown, four mesh vertices (one at each corner of the mesh cell) will be sufficient. This is especially true for cells that contain only one object.

[0085] Figure 2 A single grid cell 200 of the grid 102 is shown. In this case, three objects are shown at different depths. The closest object is the rod 202, followed by the circle 204, and finally the background 206. The depth of the circle 204 varies from left to right. Using depth smoothness, connected regions can be marked in the cell 200. In this case, the rod 202 divides the background 206 into connected regions 1 and 5, and divides the circle 204 into connected regions 2 and 4. The portion of the rod 202 within the cell 200 is marked as connected region 3.

[0086] It is important to realize that for occlusion-aware rendering on the client side, the regions do not necessarily need to be connected. In other words, it is important that pixels with the same or similar depths are blended across views, but each depth layer is synthesized from back to front. Therefore, connected regions 1 to 5 can be converted to depth regions A, B, and C based on the depth value range.

[0087] Figure 3 Shown are depth regions A to C with markings Figure 2 Grid cell 200. As can be seen, the original connected region pair [1, 5] and pair [2, 4] can be represented by individual depth regions A and B, respectively. Connected region 3 can be represented by depth region C. This results in each pixel in cell 200 receiving one of three depth class labels (A, B, or C), where each class corresponds to a single depth range (minimum and maximum depth values).

[0088] Figure 4 Pictured Figure 2 and Figure 3The one-dimensional profile of the cell 200 is shown with a corresponding range of depth values. The range of depth values ​​may be referred to as a depth interval. In this case, depth region A (i.e., background 206) corresponds to depth interval 402, depth region B (i.e., circle 204) corresponds to depth interval 404, and depth region C (i.e., rod 406) corresponds to depth interval 406. The depth may be calculated by accessing all pixels in each depth region and keeping track of the minimum and maximum depth values ​​in the depth map.

[0089] The depth intervals (i.e., depth value ranges) encode the approximate segmentation as they can be used to reconstruct the class labels after decoding. This will be discussed below.

[0090] One way to encode the depth intervals is to assume a predefined scanning order over the grid cells, and each grid cell stores the number of depth classes and the corresponding depth intervals.

[0091] Note that depth intervals are allowed to overlap. During rendering, this may mean that the same pixel is rendered multiple times. However, since this will not happen very often, the encoder overhead of removing overlapping depth intervals will probably not be worth the limited reduction in computational rendering cost.

[0092] A maximum limit may be placed on the number of depth value ranges represented for a grid cell. For example, with 4 bits, 16 depth categories may be encoded. Note that it may be necessary to be able to represent the case of no depth category (i.e., the entire grid cell is transparent), and so, in this case, 4 bits may be used to represent no depth category + 15 depth categories. Using 4-bit encoding, 128x128 pixel grid cells, and 1920x1080 image resolution, 15x8x4=480 bits per frame would be required to encode the number of depth categories.

[0093] The depth interval itself may be encoded with, for example, 16-bit precision.Of course, the depth value range may also be differentially encoded in space and / or time.

[0094] In summary, including depth value ranges in metadata for each unit will add low bitrate overhead and can be encoded at relatively low computational cost.

[0095] At the client device, the multiview texture is received along with the corresponding depth map and the depth interval for each grid cell (as metadata). The decoding itself usually occurs during rendering on the graphics processing unit (GPU). The depth interval can be used in the vertex and fragment shaders.

[0096] In the vertex shader, the depth interval can be used to clip the depth of mesh vertices within a range of depth values, so that pixels within the depth region are warped appropriately, and pixels outside the depth region but within the same cell are not drawn over the depth region.

[0097] In the fragment shader, the depth interval can be used to determine the alpha channel (ie, the transparency value of the texture). If the warp depth is outside the depth interval, the corresponding pixel can be set to transparent.

[0098] Each time a grid cell for a given depth layer and a given view is rendered, the corresponding depth interval may be passed to the GPU as a shader uniform parameter.

[0099] Start with the vertex shader, which draws the low-resolution mesh vertices on the depth map pattern. Figure 5 Shown in Figure 2 and Figure 3 500. It is important to realize that major computational savings are achieved at this point in the rendering. As long as the thin rod 204 is "hit" by at least one mesh vertex 502, the mesh resolution of each mesh cell can be further reduced. Figure 5 A thin object is shown with at least three vertices 502 hitting the illustrated low-resolution mesh.

[0100] Note that rendering occurs in order from back to front. Therefore, the first depth region drawn is depth region A, as shown in Figure 3 and 4 As shown. The vertex shader receives a depth value range for depth region A and, for all vertices in parallel, clips the sampled vertex depth value within the depth value range for depth region A. Since there is not much depth variation in depth region A, all vertices will receive almost the same depth value.

[0101] Figure 6 Illustrated is a mesh 600 drawn for depth region A. For context, objects 202 and 204 are shown. The coordinates / depth values ​​of vertex 602 are clipped to fall within the depth value range of depth region A.

[0102] The second depth region drawn has label B. Figure 7 Grid 700 is shown drawn for depth region B. Now, due to the depth distribution of the circles, there are more depth variations (e.g. Figure 4 ). However, since the depth variation on the circle varies relatively smoothly in spatial coordinates, a low-resolution grid is still sufficient.

[0103] In this case, vertex 702 drawn on circle 202 is given the corresponding depth value of the circle. Vertex 704 drawn on depth region B (i.e., on the background) is clipped to have the maximum depth value within the range of depth values ​​corresponding to depth region B. Similarly, vertex 706 drawn on depth region C (i.e., on pole 204) is clipped to have the minimum depth value within the range of depth values ​​corresponding to depth region B.

[0104] Finally, the third category C is drawn. Figure 8 Grid 800 is shown plotted for depth region B. It can be seen that the three samples that "hit" rod 204 set the depth value range to be relatively small (e.g. Figure 4 4), which effectively clips all vertices to the same depth value corresponding to the depth of pole 204.

[0105] At the vertex shader, the ray angle difference between the source and destination rays can be used to precompute weights that are passed to the fragment shader. This weight can be encoded as transparency and later used to blend the source view contribution.

[0106] Computing the transparency (α) of the texture on each mesh can be performed using the depth value range corresponding to that mesh in conjunction with the (now warped) higher resolution depth map. Note that inside the cell, α is set as a ray angle weight. In a later blending step on the view, α (if above a threshold) is interpreted as a blending weight for the blended source view contribution, and (if below or equal to a threshold) is interpreted as α and used for compositing on the previously rendered segmentation.

[0107] However, the alpha value may be modified based on the depth value of the warped depth map compared to the corresponding depth value range. In particular, within a cell, if the depth value of the warped depth map is outside the corresponding depth value range for that pixel, the alpha value may be set to 0 (i.e., transparent). This means that any pixel outside the depth value range is set to transparent. Back Figure 8 , the texture of mesh 800 will be transparent at all locations except where the mesh is drawn on top of rod 204.

[0108] In summary, the vertex shader works with a low-resolution mesh for each grid cell, decodes the segmentation information using a range of depth values, and warps the texture patch to the target viewpoint.

[0109] This is not the case for the fragment shader. In this case, new foreground edges are synthesized with pixel accuracy over the already drawn scene content. Pixels that are not in the depth interval are set to transparent, with the result that the foreground edges are well segmented. To do this, the high-resolution depth map is warped towards the target viewpoint. The fragment shader thresholds the warped high-resolution depth map using a range of depth values ​​and sets all pixels outside the depth interval to transparent.

[0110] Note that in the above example, the mesh resolution is assumed to be constant and has a lower resolution than the depth map in order to save a lot of vertex shader calculations. For example, with a quad size of four pixels, four vertices can represent an image block of 4x4=16 pixels, and two triangles can be used to warp the entire patch. This significantly reduces complexity compared to the two triangles per pixel required for traditional depth image-based rendering.

[0111] However, the computational complexity can be further reduced by making the grid resolution of each grid cell variable. Figure 1 The grid cells of the table in will only need a single quad to perform accurate viewpoint shifting (since the table is flat). This will mean that many grid cells can use only four vertices to warp the entire 128x128 image patch. Grid cells whose magnitude of depth smoothness (e.g., first and / or second spatial derivatives of the depth map) exceeds a threshold may need more vertices for accurate warping. Therefore, the grid resolution can be calculated on the production side (i.e., at the encoder before transmission / broadcast) and transmitted as metadata.

[0112] This can be achieved, for example, by encoding the quadrilateral size (in image pixels) of each grid cell as an 8-bit number to represent a quadrilateral size of 2, 4, 8, 16, 32, etc. pixels. Temporal differential encoding of this quadrilateral size parameter can further reduce the bit rate because, on average, the depth values ​​in a grid cell do not change very quickly.

[0113] Fig. 9 A method for processing a depth map at an encoder is shown. A depth map of a scene is obtained at step 902. For example, a depth sensor may obtain the depth map. Alternatively, the depth map may be generated by a calculated disparity vector between two images of the scene taken from different viewpoints.

[0114] In step 904, depth regions are identified in the depth map. Depth regions may be identified using regions of similar and / or smooth depths. For example, depth values ​​may be clustered at a particular depth, indicating the presence of an object. A depth region generally indicates one or more objects in a scene. For example, a depth region may contain pixels having similar depth values ​​and / or having relatively smooth depth variations (e.g., using the first / second spatial derivatives of the depth map) in the depth map. Of course, conventional segmentation may also be used to identify depth regions. In this case, there is no need to transmit a segmentation mask obtained by segmentation.

[0115] At step 906, a depth value range is determined for the depth region. For example, the maximum and minimum depth values ​​in the depth region may be used to define a depth value range for the depth region. Of course, other methods may be used to determine the depth value range (e.g., the 5th and 95th percentiles of the depth value distribution).

[0116] Therefore, the depth map and the depth value range may be transmitted in steps 908 and 910, respectively. The depth value range provides low bit rate segmentation information for the depth map. The depth value range may be transmitted together with an encoding identifier that provides which pixels correspond to which depth value ranges.

[0117] Fig.10 A method for synthesizing an image at a target viewpoint is shown. Steps 1002, 1004, and 1006 include receiving an image (texture tile) of a scene, a depth map of the scene, and a depth value range for the depth map, respectively. For example, a client decoder may receive these components from an encoder. The depth map is then processed using the depth value range and the image.

[0118] In a first example, the depth map and depth value range are used in a vertex shader and / or a fragment shader as discussed above.

[0119] In a second example, the depth map may be warped to the target viewpoint and blended. The resulting depth map may then be modified based on the depth value range (eg, to avoid any errors due to the warping and blending).

[0120] In a third example, the decoder can identify which depth value range the pixel corresponds to by using the depth value at each pixel before the warped depth map. Thus, a pseudo segmentation map can be generated for each depth region, thereby providing segmentation of the depth region without having to perform any segmentation at the decoder, and the encoder does not have to transmit a complete segmentation map.

[0121] More generally, the depth value range may be used as depth segmentation information in a graphics pipeline used when synthesizing a new image at a target viewpoint.

[0122] Therefore, the processed depth map together with the image can be used to synthesize a new image at the target viewpoint.

[0123] A skilled person will be easily able to develop a processor for executing any of the methods described herein. Therefore, each step of the flowchart may represent a different action performed by the processor and may be performed by a corresponding module of the processor.

[0124] As described above, the system utilizes a processor to perform data processing. The processor can be implemented in a variety of ways using software and / or hardware to perform the various functions required. The processor typically employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. The processor can be implemented as a combination of dedicated hardware for performing some functions and one or more programmed microprocessors and associated circuits for performing other functions.

[0125] Examples of circuits that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0126] In various embodiments, the processor may be associated with one or more storage media, such as volatile and non-volatile computer memory, such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that perform the desired functions when run on one or more processors and / or controllers. The various storage media may be fixed within the processor or controller, or may be transportable so that one or more programs stored thereon can be loaded into the processor.

[0127] Variations to the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0128] The functions implemented by a processor may be implemented by a single processor or by multiple individual processing units that may be considered together to constitute a “processor.” In some cases, such processing units may be remote from each other and communicate with each other in a wired or wireless manner.

[0129] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0130] The computer program may be stored / distributed on suitable media such as optical storage media or solid-state media provided together with or as part of other hardware, but may also be distributed in other forms such as via the Internet or other wired or wireless telecommunication systems.

[0131] If the term "suitable for" is used in the claims or the description, it should be noted that the term "suitable for" is intended to be equivalent to the term "configured to". If the term "arranged" is used in the claims or the description, it should be noted that the term "arranged" is intended to be equivalent to the term "system" and vice versa.

[0132] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A method for processing a depth map for subsequent transmission, the method comprising: Obtaining (902) a depth map of a scene comprising a plurality of depth values; defining a plurality of cells in the depth map, each cell corresponding to a portion of the depth map; identifying (904) one or more depth regions in each of the cells; determining (906) one or more depth value ranges for the one or more depth regions; as well as The depth map and the one or more depth value ranges are transmitted (908, 910).

2. The method according to claim 1, wherein: Defining a plurality of cells in the depth map includes dividing the depth map into a grid of cells.

3. The method according to claim 1 or 2, further comprising transmitting the number of different depth value ranges in each unit.

4. The method according to claim 3, wherein: Only transmit 16 or fewer distinct depth value ranges.

5. The method according to any one of claims 1 to 4, wherein: Identifying one or more depth regions is based on determining depth smoothness between the depth values ​​of the depth map.

6. The method according to claim 5, further comprising: determining a grid resolution based on the depth smoothness; as well as Transmits the grid resolution.

7. A method for synthesizing an image at a target viewpoint, the method comprising: Receiving (1002, 1004) at least one or more images and one or more depth maps of a scene; receiving (1006) one or more depth value ranges for a plurality of cells in at least one of the depth maps, wherein each depth value range corresponds to one or more depth regions in one of the depth maps, and wherein each cell corresponds to a portion of the depth map; and At least one of the depth maps is processed (1008) using the range of depth values ​​for each cell along with at least one of the images to synthesize (1010) an image at the target viewpoint.

8. The method according to claim 7, wherein: Processing at least one depth map includes presenting the depth regions in the one or more depth maps in a back-to-front depth order.

9. The method according to claim 7 or 8, wherein: Processing at least one of the depth maps comprises: Select a grid for each range of depth values; applying the depth values ​​of the one or more depth regions corresponding to each depth value range to a corresponding grid according to the depth map; modifying the positions of vertices in each mesh to fit within a corresponding range of depth values; and The image is synthesized at the target viewpoint using the image and the modified one or more meshes.

10. The method of claim 9, further comprising obtaining a grid resolution based on a depth smoothness of each of the one or more depth maps, wherein: The grid selected for each depth value range is based on the grid resolution.

11. The method according to any one of claims 7 to 9, wherein: Processing at least one of the depth maps comprises: warping the one or more depth maps to a target viewpoint; generating a transparency map using the one or more warped depth maps; and modifying transparency values ​​in the transparency map based on the one or more depth value ranges, Wherein, synthesizing the image at the target viewpoint is based on using the one or more warped depth maps and the transparency map.

12. The method of any one of claims 7 to 10, further comprising receiving an indication of the number and shape of cells in a cell grid defined for each of the one or more depth maps, wherein Rendering the depth region includes rendering the depth region separately for each unit.

13. A computer program comprising computer program code which, when run on a processor, causes the processor to perform the method according to any one of claims 1 to 6 and / or all the steps of the method according to any one of claims 7 to 12.

14. A processor configured to execute the computer program code according to claim 13.

Images