Compression of depth maps

By segmenting the depth map and generating the vertex position map, an efficient video bitstream is formed, which solves the problems of degraded view synthesis quality and high computing resource consumption in immersive video technology, and achieves high-quality and low-complexity rendering effects.

CN118975254APending Publication Date: 2024-11-15KONINKLIJKE PHILIPS NV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380031288.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-30
Filing Date
2023-03-20
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

When the existing immersive video technology moves outside the viewing space, the image quality of the synthetic view is degraded, errors and artifacts occur, resulting in poor user experience, high computing resource consumption, excessive data rate, and difficult to achieve efficient rendering.

Method used

By receiving the depth map of the video frame, segmenting it into two-dimensional geometric primitives, generating a vertex position map and a new depth map to form an efficient video bitstream to reduce data rates and improve rendering performance.

Benefits of technology

High-quality image synthesis in different view poses is achieved, reducing rendering complexity and data rate, improving computing efficiency, improving user experience, and reducing artifacts and inaccuracies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118975254A_ABST
    Figure CN118975254A_ABST
Patent Text Reader

Abstract

An apparatus includes a receiver (201) that receives a first depth map for a first video frame of a plurality of video frames. A divider (203) divides the first depth map into two-dimensional geometric primitives / polygons greater than two pixels. A map generator (207) generates a vertex position map indicating vertex positions of geometric primitives in the first depth map. A depth map generator (205) generates a second depth map including depth values of the first depth map at vertex positions, and a bitstream generator (209) generates a video bitstream to include the second depth map and a vertex position map for the first video frame. The segmentation depends on depth values of the depth map. The complementary device renders the image using the mesh generated from the bitstream by determining original vertex positions based on the received vertex position map and projecting these into the 3D space.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to video signals and their processing, and specifically, but not exclusively, to the generation and rendering of images from bitstreams based on a three-dimensional grid. Background Technology

[0002] In recent years, with the continuous development and introduction of new services and methods for utilizing and consuming video, the types and scope of image and video applications have increased significantly.

[0003] For example, an increasingly popular service is providing image sequences that allow viewers to actively and dynamically interact with the system to change rendering parameters. In many applications, a particularly appealing feature is the ability to alter the viewer's effective viewing position and orientation, such as allowing the viewer to move and look around within the presented scene.

[0004] Such features specifically allow for the provision of virtual reality experiences to users. This allows users, for example, to move freely within a virtual environment and dynamically change their position and what they are looking at. Typically, such extended reality (XR) applications are based on a 3D model of the scene, which is dynamically evaluated to provide a specific requested view. This approach is well-known in applications such as gaming, for example in the first-person shooter genre for computers and consoles. Extended reality (XR) applications include virtual reality (VR), augmented reality (AR), and mixed reality (MR) applications.

[0005] An example of a proposed video service or application is immersive video, where video is played on, for example, a VR headset to provide a three-dimensional experience. With immersive video, viewers are free to watch and move within the presented scene, thus allowing them to perceive it as being viewed from different perspectives. However, in many typical approaches, the amount of movement is limited, for example, to a relatively small area around a nominal viewpoint, which typically corresponds to the viewpoint from which scene video capture is performed. In such applications, 3D scene information is usually provided, which allows for high-quality view image synthesis for viewpoints relatively close to a reference viewpoint, but this deteriorates if the viewpoint deviates too much from the reference viewpoint.

[0006] Immersive video is often also referred to as 6 degrees of freedom (6DoF) or 3DoF+ video. MPEG Immersive Video (MIV) is an emerging standard that uses metadata on top of existing video codecs to implement and standardize immersive video.

[0007] One problem with immersive video is the limited viewing space (a 3D space in which viewers can have a sufficiently high-quality 6DoF experience). As viewers move out of the viewing space, the quality degradation and errors caused by the synthesized view images become increasingly severe and can lead to an unacceptable user experience. Because the provided 3D video data does not provide sufficient information for view synthesis (e.g., deocclusion data), errors, artifacts, and inaccuracies may appear in the generated view images.

[0008] For example, immersive video data can be provided in a multi-view format, possibly accompanied by a depth data (MVD) representation of the scene. The scene can be captured by multiple spatially differentiable cameras, and the captured images can be compared with depth data. Figure 1 Provided by [the relevant authority].

[0009] View composition in applications such as immersive video is often based on depth maps that provide depth information to the source views. Depth maps may be accompanied by texture maps, which may directly correspond to captured images from these views, or they may describe textures to map onto 3D objects and features in the scene.

[0010] Many view compositing algorithms are based on constructing a mesh model from a depth map, which can be, for example, packed into pixel data from a video stream. Then, during rendering, multiple source view textures are distorted to the target viewpoint based on the mesh model and blended and / or composited to generate an image of that target viewpoint.

[0011] Such operations require significant processing power, including tasks such as warping numerous vertices of a mesh model to appropriate 2D positions for a specific target viewpoint. This process is well-suited for parallel processing, and efficient implementations using massively parallel processing architectures are common. For example, efficient graphics pipelines have been developed and implemented in the graphics cards of standard or gaming PCs.

[0012] In traditional graphics processing, such as for games and many fully virtual reality experiences, rendering may be based on a relatively static mesh representation of the scene, which is updated only at a relatively slow rate to reflect dynamic changes in the scene. However, in contrast, for immersive video applications, a new depth map is typically provided for each new frame, and these depth maps need to be (in some cases implicitly) converted into a mesh representation as part of the rendering.

[0013] Therefore, rendering the received video for immersive video typically requires very high computational resources. Furthermore, the data rate of the video signal, including the 3D information used for rendering, is often excessively high. Attempts to mitigate these problems often result in performance degradation, poor user experience, and / or image quality deterioration. Thus, traditional methods for 3D / immersive video are often not optimal.

[0014] Therefore, improved methods would be advantageous. In particular, a method that allows for improved operation, increased flexibility, improved immersive user experience, reduced complexity, facilitated implementation, improved composite image quality, improved rendering, improved and / or simplified view composition for different view poses, improved computational efficiency, improved quality, reduced data rate, more accurate representation of the scene, and / or improved performance and / or operation would be advantageous. Summary of the Invention

[0015] Therefore, the present invention seeks to mitigate, alleviate or eliminate one or more of the above-mentioned disadvantages, preferably alone or in any combination.

[0016] According to one aspect of the present invention, an apparatus for generating a video signal including a bitstream is provided, the apparatus comprising: a receiver arranged to receive a first depth map for at least a first video frame of a plurality of video frames; a segmenter arranged to segment at least a portion of the first depth map into two-dimensional geometric primitives, the two-dimensional geometric primitives having an average area more than twice the pixel area of ​​the first depth map, and the segmentation comprising determining vertex positions of vertices of the first depth map for the geometric primitives; a graph generator arranged to generate a vertex position map indicating the vertex positions in the first depth map; a depth map generator arranged to generate a second depth map, the second depth map including the depth values ​​of the first depth map at the vertex positions in the first depth map; a bitstream generator arranged to generate the video bitstream including the second depth map and the vertex position map for the first video frame; and wherein the segmenter is arranged to generate the vertex positions based on the depth values ​​of the first depth map.

[0017] This invention allows for improved data characterizing the scene being generated. The method allows for improved image synthesis for different view poses based on the generated bitstream. In many embodiments, data that allows for efficient and high-quality view shifting and synthesis can be generated.

[0018] This invention can provide an improved user experience in many embodiments and scenarios. The method can, for example, allow for improved immersive video or XR / AR / VR / MR applications. In many embodiments, the method can support an improved user experience with the potential for rendering high-quality view images from video signals and bitstreams.

[0019] In many embodiments, this method can reduce the data rate of the bitstream. This method can allow for improved spatial information for a given data rate. In many embodiments, this method can provide improved support for rendering and can allow for reduced complexity and / or higher computational efficiency for the generation and / or rendering ends. This method can, for example, provide improved applicability through highly efficient and optimized features, such as the use of highly parallelized graphics pipelines.

[0020] In some embodiments, the receiver may further receive a texture map linked to the depth map, and the bitstream generator may be arranged to include the texture map in the bitstream.

[0021] Segmentation can be surface subdivision in many embodiments. Two-dimensional geometric primitives can be polygons in many embodiments, and are typically triangular, quadrilateral, or hexagonal. Vertex locations can be the locations of the vertices / corners of the geometric primitives / polygons in the first depth map, and are typically pixel locations.

[0022] In many embodiments, the two-dimensional geometric primitives may have an average area that is not less than 3, 4, 8, or 16 times the pixel area of ​​the first depth map. The second depth map includes fewer depth values ​​per pixel than the first depth map, and is typically a factor of not less than 2, 3, 4, 8, or 16.

[0023] The segmenter can be arranged to generate vertex positions to reduce depth variation within the geometric primitive, for example, to reduce the reduced average depth variation within the geometric primitive.

[0024] According to an optional feature of the invention, the segmenter is arranged to determine the vertex position by offsetting the reference vertex position of the reference segment of the depth map; the offset for the first vertex depends on the value of the first depth map of the pixel within the geometric primitive for the first vertex.

[0025] In many embodiments, this can provide improved and / or enhanced operation and / or performance. It can facilitate the determination of vertex positions / 2D primitives and can generally allow for more efficient representation.

[0026] A reference segment can correspond to / be equivalent to a reference pattern of a vertex position / reference pattern of a reference vertex position / reference set of a reference pattern of a geometric primitive.

[0027] The offset of the reference vertex position can keep the average area of ​​the geometric primitive unchanged.

[0028] The number of vertices represented by a vertex position graph may not exceed the number of reference vertex positions. The number of vertices represented by a vertex position graph may be the same as the number of reference vertex positions.

[0029] According to an optional feature of the invention, the reference vertex positions of the reference segmentation form a regular mesh.

[0030] In many embodiments, this can provide improved and / or enhanced operation and / or performance. In many embodiments, the reference mesh can provide an improved or optimal starting point for adapting to a specific depth.

[0031] In many embodiments, a new vertex position can be determined by a relative shift of the vertex position relative to a reference vertex position of the relative vertex position pattern.

[0032] According to an optional feature of the invention, the vertex position map indicates the vertex positions as relative to the reference vertex positions of the reference segment.

[0033] This can provide advantageous operation in many embodiments. It typically allows for substantially reduced data rates and allows for the generation of vertex location maps that are highly suitable for encoding and compression (including using existing video coding techniques).

[0034] According to an optional feature of the invention, the offset includes a first offset along a first predetermined direction, and the first offset is determined to be an offset in which the change in depth map value along the first predetermined direction satisfies a criterion.

[0035] This can provide particularly advantageous operation in many embodiments. In many embodiments, it can allow for lower complexity and / or improved computational efficiency. The method can provide efficient and high-performance adaptation to many different depth maps. In some embodiments, the segmenter can determine the step size change for different offsets and determine the offset based on the depth change for different offsets.

[0036] The first predetermined direction can be horizontal and / or vertical.

[0037] According to an optional feature of the invention, the offset includes a second offset along a second predetermined direction, and the second offset is determined to be an offset in which the change of depth map value along the second predetermined direction satisfies a criterion; the graph generator is arranged to generate a vertex position map to indicate the offset along a first predetermined direction, and further generate an additional vertex position map to indicate the offset along the second predetermined direction; and the bitstream generator is arranged to include the additional vertex position map in the bitstream.

[0038] This can provide particularly advantageous operation in many embodiments. In many embodiments, it can allow for lower complexity and / or improved computational efficiency. The method can provide efficient and high-performance adaptation to many different depth maps. In some embodiments, the segmenter can determine the step size change for different offsets and determine the offset based on the depth change for different offsets.

[0039] The first predetermined direction and the second predetermined direction can be horizontal and vertical directions, respectively.

[0040] According to an optional feature of the invention, the first predetermined direction is a horizontal direction.

[0041] According to an optional feature of the invention, the first predetermined direction is a vertical direction.

[0042] According to an optional feature of the invention, the offset is constrained to be below an offset threshold.

[0043] According to an optional feature of the invention, the bitstream generator includes a video encoder arranged to encode the vertex location map using video encoding.

[0044] This can provide more efficient coding in many embodiments and / or allow the reuse of existing functionality that is also used for other purposes.

[0045] According to an optional feature of the invention, the segmenter is arranged to determine vertex positions to align the geometric primitives with the depth transformation in the depth map.

[0046] This can provide particularly advantageous operation in many implementations and scenarios. In particular, it can reduce artifacts and inaccuracies that may be caused by depth transformation.

[0047] According to another aspect of the present invention, an apparatus for rendering a bitstream of a video signal is provided, the apparatus comprising: a receiver for receiving a video bitstream, the video bitstream including, for at least a first frame, a first depth map and a vertex position map, the vertex position map indicating vertex positions of depth values ​​in a second map of the first depth map, the second map having a higher resolution than the first depth map, the vertex positions defining segmented two-dimensional geometric primitives forming the second map; a renderer arranged to render a video image from the video bitstream, the renderer comprising: a mesh generator arranged to generate a mesh representation of a scene, the mesh being formed by projections of the two-dimensional geometric primitives onto a three-dimensional space, the projection of a first vertex of the two-dimensional geometric primitive depending on the vertex position in the second depth map and the depth value in the first depth map for the first vertex; and a rendering unit arranged to render an image based on the mesh representation.

[0048] This invention allows for the generation of improved images in numerous embodiments and scenarios. The method enables improved image synthesis for different view poses.

[0049] This invention can provide an improved user experience in many embodiments and scenarios. The method can, for example, allow for improved immersive video or XR / AR / VR / MR applications. In many embodiments, the method can support an improved user experience with the potential for rendering high-quality view images from video signals and bitstreams.

[0050] In many embodiments, this method can reduce the data rate of the bitstream. This method can generally reduce rendering artifacts and inaccuracies, especially when combined with depth steps / transformations. In many embodiments, this method can allow for reduced complexity and / or higher computational efficiency. This method can, for example, provide improved applicability through efficient and optimized features, such as the use of highly parallelized graphics pipelines.

[0051] In some embodiments, the bitstream may further include a texture map linked to a depth map, and the rendering units may be arranged to render images based on the texture map.

[0052] Rendering units can be arranged to render images corresponding to view images from view poses. Rendering can include evaluating a mesh representation of the view pose.

[0053] According to an optional feature of the invention, the vertex position map indicates vertex positions as relative vertex positions relative to reference vertex positions of a reference segment; and the mesh generator is arranged to determine absolute vertex positions in a second depth map in response to the relative vertex positions and the reference vertex positions.

[0054] In many embodiments, this can provide improved and / or enhanced operation and / or performance.

[0055] According to an optional feature of the invention, the bitstream includes an indication of a reference segmentation; and a grid generator is arranged to determine the reference vertex position in response to the indication of the reference segmentation.

[0056] In many embodiments, this can provide improved and / or enhanced operation and / or performance.

[0057] According to another aspect of the present invention, a method for generating a video signal including a bitstream is provided, the method comprising: receiving a first depth map for at least a first video frame among a plurality of video frames; segmenting at least a portion of the first depth map into two-dimensional geometric primitives, the two-dimensional geometric primitives having an average area more than twice that of a pixel region of the first depth map, and the segmentation comprising determining vertex positions in the first depth map for vertices of the geometric primitives; generating a vertex position map indicating the vertex positions in the first depth map; generating a second depth map, the second depth map including the depth values ​​of the first depth map at the vertex positions in the first depth map; generating the video bitstream to include the second depth map and the vertex position map for the first video frame; and wherein the generation of the vertex positions depends on the depth values ​​of the depth map.

[0058] According to another aspect of the present invention, a method for rendering a bitstream of a video signal is provided, the method comprising: receiving a video bitstream, the video bitstream including, for at least a first frame, a first depth map and a vertex position map, the vertex position map indicating vertex positions of depth values ​​in a second map of the first depth map, the second map having a higher resolution than the first depth map, the vertex positions defining segmented two-dimensional geometric primitives forming the second map; rendering a video image from the video bitstream, the rendering comprising: generating a mesh representation of a scene, the mesh being formed by projecting the two-dimensional geometric primitives into a three-dimensional space, the projection of a first vertex of the two-dimensional geometric primitives depending on the vertex position of the first vertex in the second depth map and the depth value in the first depth map for the first vertex; and rendering the image based on the mesh representation.

[0059] These and other aspects, features and advantages of the present invention will become apparent from the embodiments described below, and will be explained with reference thereto. Attached Figure Description

[0060] Embodiments of the invention will be described by way of example only with reference to the accompanying drawings, wherein...

[0061] Figure 1 An example of the components of a server-client system is shown;

[0062] Figure 2 Examples of elements of a device according to some embodiments of the present invention are shown;

[0063] Figure 3 Examples of elements of a device according to some embodiments of the present invention are shown;

[0064] Figure 4 An example of a depth map representation of a scene view is shown;

[0065] Figure 5 An example of a depth map representation of a scene view is shown;

[0066] Figure 6 An example of a depth map representation of a scene view is shown;

[0067] Figure 7 An example of a depth map representation of a scene's view is shown; and

[0068] Figure 8 An example of a depth map representation of a scene's view is shown. Detailed Implementation

[0069] The following description will focus on immersive video applications; however, it should be understood that the concepts, implementations, methods, and principles described in other embodiments can be applied to other video applications.

[0070] 3D video capture, distribution, and presentation are becoming increasingly popular and desirable in various applications and services. A particular approach is known as immersive video and typically involves providing a view of a real-world scene, often including real-time events that allow for small viewer movements (such as relatively small head movements and rotations). For example, real-time video broadcasting of a sporting event, allowing for a view generated based on a local client after a small head movement of the viewer, can provide the impression of a user sitting in the stands watching the event. The user can, for example, view and have a natural experience similar to that of a spectator present in that location in the stands. Recently, there has been a growing popularity of display devices based on 3D capture of real-world scenes, featuring positional tracking and 3D interaction capabilities that support applications. Such display devices are well-suited for immersive video applications that provide an enhanced 3D user experience.

[0071] The following description will focus on immersive video applications, but it should be understood that the principles and concepts described can be used in many other applications and implementations.

[0072] In many approaches, immersive video can be provided locally to viewers via, for example, a standalone device that uses no data or even has any access to any remote video server. However, in other applications, immersive applications can be based on data received from a remote or central server. For example, video data can be provided to the video presentation device from a remote central server and processed locally to generate the desired immersive video experience.

[0073] Figure 1An example of such an immersive video system is shown, in which a video rendering client 101 is connected to a remote immersive video server 103, for example, via a network 105 such as the Internet. The server 103 can be configured to simultaneously support a potentially large number of client devices 101.

[0074] Server 103 can support immersive video experiences, for example, by sending video signals that include 3D video data describing a real-world scene. The data can specifically describe the visual features and geometric properties of a scene generated by a set of (potentially 3D) cameras capturing real-world scenes in real time.

[0075] To provide such services for real-world scenes, scenes are typically captured from different locations, and poses are captured using different cameras. As a result, the relevance and importance of multi-camera capture and, for example, 6DoF (6 degrees of freedom) processing have increased rapidly. Applications include live concerts, live sports, and telepresence. The degree of freedom to choose one's own viewpoint enriches these applications by adding a sense of presence to regular video. Furthermore, immersive scenes can be envisioned where observers can navigate and interact with the live-captured scene. For broadcast applications, this may require real-time view compositing at the client device. View compositing introduces errors, and these errors depend on the implementation details of the algorithm.

[0076] A popular approach to providing relevant information to the rendering side is to provide a video signal, for which a depth map and a texture map can be provided for each video frame. In such an embodiment, the renderer can generate a mesh representation based on the depth map, and then generate a view image by projecting the mesh onto the appropriate location in a view image / port for a given view pose. The renderer can distort the texture map associated with the mesh to synthesize a visible image for a given view pose.

[0077] Figure 1 Server 103 is configured to generate a video signal comprising data describing the scene via a sequence of frames, wherein depth maps and texture data are provided for at least some frames. Client 101 is configured to receive and process this video signal to provide a view in a given view pose. Client 101 may generate an output video stream or display drive signal that dynamically reflects changes in view pose, thereby providing an immersive video experience in which the presented view adapts to changes in viewing / user pose / placement.

[0078] In this field, the terms placement and pose are used as common terms for position and / or orientation / direction. For example, a combination of position and orientation / direction of an object, camera, head, or view can be referred to as a pose or placement. Therefore, a placement or pose indication may include six values / components / degrees of freedom, where each value / component typically describes a separate property of the corresponding object's position / location or orientation / direction. Of course, in many cases, placement or pose can be considered or represented with fewer components, for example, if one or more components are considered fixed or unrelated (e.g., if all objects are considered to be at the same height and have a horizontal orientation, four components can provide a complete representation of the object's pose). In the following text, the term pose is used to refer to position and / or orientation that can be represented by one to six values ​​(corresponding to the maximum possible degrees of freedom). The term pose can be replaced by the term placement. The term pose can be replaced by the terms position and / or orientation. The term pose can be replaced by the terms position and orientation (if the pose provides information on both position and orientation), the term position (if the pose provides information on position (possibly only)), or by orientation (if the pose provides information on orientation (possibly only)).

[0079] Typically, rendering a video image of a given view pose based on a depth map and texture map involves the following steps: decoding a still image or video frame into GPU memory, and while the vertex shader retrieves / samples image space vertex data to form the still image or video frame, constructing a homogeneous coordinate vector in image space for each vertex, and transforming / distorting each vertex to the image space of the target view. Vertices involve a predefined mesh topology typically created during the initialization phase. The mesh topology (e.g., triangles / quadrilaterals or others) determines how the fragment shader needs to distort the associated texture to the target view after the vertex shader stage. Since the image or video frame is already in GPU memory, this fragment shader samples directly from it to generate the resulting view.

[0080] Typically, depth maps are required to have high resolution in order to obtain a fine mesh that accurately represents the geometric properties. Typically, depth maps have at least the same resolution as texture maps. This provides an accurate representation of the spatial properties of the scene through a detailed, high-resolution mesh model. However, this method requires high computational resources, and in practice, such high accuracy often does not allow for real-time rendering due to the large number of mesh triangles (e.g., two triangles per pixel).

[0081] To address this issue, a coarse mesh model with larger triangles can be used, and these larger triangles can be generated either by subsampling a high-resolution depth map or simply by encapsulating a lower-resolution depth map within a video frame that uses a lower-resolution mesh for prediction. Triangles from a finer mesh generated from the depth map can then be combined. However, such conventional methods can lead to a reduction in rendering quality (e.g., near-depth transformation).

[0082] The following sections describe methods that can provide improved operation and / or performance in many scenarios. Improved rendering quality can be achieved in many cases.

[0083] Figure 2 Elements of an exemplary apparatus for generating a video signal comprising a bitstream representing a scene are shown. Figure 2 The device can specifically correspond to Figure 1 Server 103, and will be described in that context. Figure 3 The video bitstream used to render the video signal is shown, and specifically from... Figure 2 The apparatus is an exemplary component of a device for rendering video bitstreams. Specifically, this apparatus may be an end-user device or client that receives audio-visual data streams from a remote device, such as from... Figure 2 The device. Specifically, Figure 3 The device can be Figure 1 The following description will focus on the context of the components of client 101.

[0084] Figure 2 The device, and therefore in a specific example, server 102 includes receiver 201, which receives data including depth maps and linked texture maps for one or more video frames. In some embodiments, multiple depth maps may be linked to the same texture map, and for example, texture maps may not be provided for all frames and / or depth maps may not be provided for all frames.

[0085] The server can specifically receive an input video signal having temporally ordered video frames represented by depth maps and texture maps. Such an input video signal can be received, for example, from a processing unit that generates depth maps and texture maps from live captures, such as multiple images of a scene captured from different view poses. In other embodiments, input data can be received, for example, from an external or internal source arranged to generate depth maps and texture maps from a model of the scene. It should be understood that the generation and use of depth maps and linked texture maps are known to those skilled in the art of 3D video, and therefore will not be described in detail for clarity and brevity.

[0086] The depth map can be provided at a relatively high resolution, and specifically, in many embodiments, the depth map and texture map can be provided at the same resolution. Hereinafter referred to as the encoding device. Figure 2The apparatus is arranged to generate output video signals, where one, more, and typically all video frames are represented by depth maps and texture maps. Furthermore, one, more, or all of the output depth maps are generated at a lower resolution than the corresponding depth maps. The encoding apparatus uses specific methods for reducing the resolution of the depth maps, enabling improved representations that allow for enhanced rendering in many scenes.

[0087] A depth map is fed from receiver 201 to segmenter 203, which is arranged to segment at least a portion of the depth map into two-dimensional geometric primitives. These two-dimensional geometric primitives are typically polygons, such as specific triangles or squares. In most embodiments, but not necessarily all, the segmentation is into the same geometric primitive. However, in some embodiments, the segmentation may be into different geometric primitives.

[0088] In some embodiments, a segmentation may be applied to a portion of the depth map, but typically it will be applied to the entire depth map. In some embodiments, a segmentation may allow for gaps or holes between defined geometric primitives (e.g., a segmentation may use polygons or sets of polygons that are not perfectly aligned and combined to cover the entire depth map region but may include small gaps / holes). However, these are typically so small that they have no significant impact on performance.

[0089] Segmentation can correspond to overlapping spatial 2D image regions, such that when rendered from a given view pose, the resulting distorted meshes overlap. Each individual spatial 2D image region can have different primitive parameters. For example, the size of the mesh primitives can differ between regions.

[0090] In many embodiments, a segmentation is a surface subdivision of at least a portion of the depth map, and typically a surface subdivision of the entire depth map. Surface subdivisions can be, for example, triangular, square, or hexagonal. Surface subdivisions are typically regular surface subdivisions, but in many embodiments can be irregular surface subdivisions.

[0091] Segmenter 203 is typically arranged to perform segmentation (and specifically, surface subdivision) on all polygons having the same number of sides (e.g., triangles or squares), but with variations in the shape of each polygon. In practice, in many embodiments, segmenter 203 may be arranged to perform segmentation / surface subdivision by first performing standard rule segmentation / surface subdivision, which is then modified to provide the final segmentation / surface subdivision.

[0092] The segmenter 203 can be arranged to perform segmentation / surface subdivision by determining the vertices in the depth map of the geometric primitives and the specific (new) vertex positions in the depth map to define the geometric primitives.

[0093] Geometric primitives are typically polygons, and usually triangles or squares. Vertices are typically corners of polygons, such as the corners of triangles or squares. Vertices are shared among multiple polygons to provide surface subdivision. For example, for a triangular geometric primitive, most vertices can be shared among six or eight triangles, and for a quadrilateral geometric primitive, most vertices can be shared among four quadrilaterals. Therefore, changing the position of a vertex can modify multiple geometric primitives simultaneously.

[0094] In many embodiments, vertex positions are pixel positions in the depth map, and therefore, each vertex position is determined as the position of a pixel in the depth map. The vertex position resolution can be equal to the pixel resolution of the depth map.

[0095] The segmenter 203 is arranged to determine geometric primitives / polygons such that these are on average greater than pixels, and in fact greater than two pixel regions. Therefore, for at least one side of the polygon, the vertex positions are on average more than one pixel, and are typically at least twice the pixel distance for at least one side. Generally, the number of geometric primitives determined for a portion of the segmented / subdivided depth map is less than the number of pixels in the segmented portion, and is typically a factor of not less than 2, 3, 5, or even 10. In many embodiments, the number of geometric primitives generated by the segmentation is less than the number of pixels in the input depth map by a factor of not less than 2, 3, 5, or even 10. The number of distinct vertices of the geometric primitives in many embodiments is less than the number of pixels in the input depth map (a portion) by a factor of not less than 2, 3, 5, or even 10 for the segmentation / subdivision.

[0096] Compared to segmentation, which is generated using geometric primitives that use pixels as vertices, segmentation therefore provides a reduction in extraction / resolution.

[0097] Segmenter 203 is arranged to generate vertex positions that depend on depth values ​​in the depth map. Therefore, instead of generating predetermined or fixed segmentations into polygons / primitives, an adaptive and flexible approach is applied, where segmentation into polygons / primitives is performed to reflect depth variations in the depth map. In many embodiments, segmenter 203 is arranged to determine vertex positions to align the polygons with depth transformations in the depth map. In some embodiments, depth transformation alignment can be accomplished with subpixel accuracy. Segmenter 203 may be arranged to determine a fixed number of polygons and vertices, but the size and position of the polygons vary to align the polygons with the depth transformations.

[0098] Segmenter 203 is coupled to depth map generator 205, which receives data from segmenter 203 describing the determined vertex positions in the input depth map. Depth map generator 205 is also coupled to receiver 201, from which it receives the input depth map.

[0099] The depth map generator 205 is configured to generate an output depth map that includes the depth values ​​of the input depth map at the determined vertex locations in the first depth map. Therefore, the depth values ​​at the selected vertex locations are included in the output depth map, excluding the depth values ​​of other locations / pixels in the input depth map. The output depth map thus includes a subset of the depth values ​​of the input depth map, and therefore, spatial decimation / resolution reduction is achieved.

[0100] Depth map generator 205 is also coupled to position map generator 207, which further feeds indications of the determined new vertex positions. Position map generator 207 is arranged to generate a vertex position map indicating the vertex positions in the input depth map. In many embodiments, the vertex position map may include position indications for each vertex position. Thus, in many embodiments, the output depth map and the vertex position map may have the same resolution, and in fact, each depth value in the output depth map may be linked to a position value in the vertex position map, and vice versa. In some embodiments, the vertex position map may have vertex positions with a higher resolution than the original vertex positions (which typically tend to correspond to discrete depth map pixel positions).

[0101] Figure 2 The apparatus further includes a bitstream generator 209, which is arranged to generate a video bitstream for a video signal.

[0102] Bitstream generator 209 may be specifically arranged to generate a video stream for a given frame including a texture map received from receiver 201 (typically the same as the texture map received by receiver 201), an output depth map received from depth map generator 205, and a vertex position map received from position map generator 207. In some embodiments, all three types of data may be provided for all frames of the video signal. In other embodiments, only one, more, or all of the three types of data may be provided for some frames. In such examples, data for frames that do not include explicit data may be generated on the rendering side from other frames for which data is provided. For example, in some embodiments, values ​​from other frames may be used directly or may be used for interpolation or extrapolation.

[0103] Bitstream generator 209 can be configured to generate bitstreams that conform to any suitable structure, arrangement, and / or standard. For example, a bitstream can be generated based on a known video standard (e.g., a vertex location map is included in the metadata section). Furthermore, bitstream generator 209 can implement suitable encoding algorithms. Specifically, texture maps and depth maps can be encoded using known video coding algorithms. In many embodiments, the vertex location map can also be encoded using video coding schemes (including potentially lossy video coding schemes).

[0104] The following description will focus on an embodiment in which the segmenter 203 is arranged to perform surface subdivision of the entire input depth map into triangles, and thus each geometric primitive / polygon has three vertices. However, it should be understood that this is merely exemplary and other polygons / primitives may be used, and only a portion of the depth map may be segmented / subdivided, and / or in some cases, the segmentation may leave gaps or holes.

[0105] In many embodiments, the determination of vertex positions, and therefore the segmentation / surface subdivision, is based on a reference segmentation of the depth map. The reference segmentation is given by a reference pattern of vertex positions, and therefore, equivalently, the determination of vertices is based on a reference pattern of vertex positions in the input depth map.

[0106] In many embodiments, the reference segmentation / position pattern can be a regular grid. For example, the reference vertex position can be given as a position with a predetermined distance between its nearest neighbors in both the horizontal and vertical directions. The distance can be given based on the pixel position.

[0107] Figure 4 The example shown is a reference segmentation / pattern based on a regular grid. In this example, the grid is formed by the vertex positions at every fourth pixel of the input depth map in both the horizontal and vertical directions. In this example, the reference geometric primitives are triangles, and two triangles are formed for each 4x4 region. Figure 4 In the example, the reference segmentation therefore corresponds to a regular grid with two triangles per 4x4 pixel block. Figure 4 In the example, the depth map contains values ​​that reflect the depth steps / transitions that occur between gray-shaded pixels and non-gray-shaded pixels.

[0108] Therefore, reference segmentation and meshing provide undersampling of the input depth map, thus providing spatial extraction of the depth map. Consequently, the reference pattern can provide a significant reduction in the number of vertices required to represent the corresponding mesh compared to using all depth values ​​to form a depth map. This allows for more efficient representation of the underlying scene. However, coarse segmentation and meshing also lead to a less accurate representation of spatial properties. Figure 2 In this device, the determined reference vertex position can be offset relative to a given reference vertex position. The vertex position offset is determined based on the depth values ​​of the input depth map. This method can mitigate the effects of resolution reduction in many scenarios.

[0109] In many embodiments, a new vertex position is determined by a relative shift of the vertex position relative to a reference vertex position in a reference pattern / segmentation. Segmenter 203 may be arranged to determine the offset of a given vertex based on the depth values ​​of an input depth map of pixels within at least one of the geometric primitives that include the given vertex. In many embodiments, segmenter 203 may consider depth values ​​in more than one of the primitives that include the given vertex, and in practice, it may typically consider at least one of the values ​​of at least two primitives (partially) defined by the given vertex. In some embodiments, the offset may also consider depth values ​​belonging to primitives not formed by the offset given vertex. For example, if the offset of a vertex causes other vertices to shift, this may affect depth changes within another primitive, and these changes may be taken into account.

[0110] As yet another example, in some embodiments, vertex localization can also be based on texture variations within a region. For instance, regions that are reasonably flat (according to any suitable criterion) and each have similar textures, but different regions with varying degrees of texture, can be identified. Thus, regions without large depth transformations but with different textures can be identified. Vertices can be arranged such that this produces individual vertices. This will become visible after view composition, as more textured regions are generally more susceptible to depth artifacts.

[0111] It should be understood that the specific algorithms and criteria used to determine the output depth map and the specific offset vertex positions will vary depending on the requirements and preferences of each embodiment.

[0112] In many embodiments, the segmenter 203 is arranged to determine offset vertex positions to align primitives with depth transformations in the depth map. For example, lines corresponding to depth transformations can be determined in the depth map, such as by determining lines whose depth gradients exceed a given threshold or contour lines of regions with large depth gradients (it will be understood that many different techniques for determining depth transformation lines in the depth map will be known to those skilled in the art). Vertex offsets can then be determined based on the determined lines of depth transformations. For example, a given vertex can be offset to increase the size of the primitive while adhering to the criterion that the primitive may not extend across the depth transformation line and suffers maximum offset.

[0113] As another example, the offset can be determined based on the depth variation of the depth values ​​within a primitive. For instance, a depth variation metric can be determined for a given primitive for different possible offsets. The offset that results in the largest primitive having a depth variation below a given threshold can then be selected. As another example, a depth variation metric can be determined for all primitives affected by the offset. A cost metric can then be determined for each primitive as a monotonically increasing function of the primitive's size and the depth variation (e.g., for a given offset, the cost metric can be determined for each primitive by multiplying the primitive's size by the primitive's depth variation metric). The cost metric can then be determined by combining the cost values ​​of all primitives for a given offset, and the offset with the minimum cost value can be selected. In many scenarios, this approach results in offsets being selected where depth transformations and variations are concentrated in smaller primitives, while larger primitives tend to be relatively flat.

[0114] It should be understood that different measures of depth variation can be used, such as depth variance, maximum depth difference, difference between different percentiles, etc.

[0115] Figure 5 It shows how it can be offset. Figure 4 The example references the location to generate primitives that are more closely aligned with depth variations and depth transitions in the image. In this example, two different depth levels are indicated by the gray shading of pixels. It can be seen that smaller primitives are generated around the depth transition, and larger primitives are generated further away from the depth transition.

[0116] In many embodiments, the position map generator 207 may be arranged to indicate the vertex positions in the input depth map as relative positions to reference vertex positions of a reference segment. For example, the vertex position map may include offset indications indicating the magnitude and possible orientation of the determined offset.

[0117] In many embodiments, vertex positions in the input depth map are indicated by pixel positions, and offsets can be simply given as integer numbers of pixels or, in some cases, subpixels. Furthermore, the maximum offset is typically a relatively small number of pixels, and the vertex position map can be correspondingly given as a map of relatively small integers. In many embodiments, relatively large offsets can be zero (e.g., in large, relatively flat regions where the benefit of offsetting vertex positions is very low). In many embodiments, the vertex position map can be in the form of data highly suitable for efficient image or video compression as part of encoding and included in the output bitstream.

[0118] In some embodiments, the depth map generator 205 may be arranged to perform an offset in a predetermined direction. For example, the depth map generator 205 may be arranged to determine an offset in a horizontal or vertical direction. In some embodiments, the offset may be in the horizontal direction of the input depth map. In some embodiments, the offset may be in the vertical direction of the input depth map. For example, Figure 5 By only in the horizontal direction Figure 4 The regular grid is generated by introducing an offset.

[0119] Constraining the offset to a predetermined direction results in an offset value that is simply one-dimensional, and therefore can be represented in the vertex position graph by a simple scalar value. Furthermore, implementing the offset, particularly in the horizontal direction, significantly reduces complexity while achieving substantial advantages. Since humans typically move primarily in the horizontal plane, the benefit of spatial rendering lies in having better depth step alignment, especially in the horizontal direction.

[0120] In many embodiments where the offset is along a predetermined direction, the offset can be determined as an offset that satisfies a criterion for changes in depth map values ​​for different offsets along a first predetermined direction. For example, depth map generator 205 can be used to increase the range of offsets to determine a measure of the depth change of the current offset relative to the minimum offset. For example, to increase (or decrease) the offset, a maximum depth step can be determined, for example, as the difference between the maximum and minimum depth values ​​of all offsets smaller than the current offset. The maximum offset with a depth step below a threshold can then be selected. This range can be a range that includes only negative offsets, positive offsets, or both negative and positive offsets.

[0121] Figure 6 This demonstrates how this method can produce... Figure 5 Here's an example of vertex positions. In this example, vertex position A is the initial vertex position, unchanged. The new vertex position B is determined by offsetting the next reference vertex position horizontally. This horizontal offset is increased until a depth step above a given threshold is detected. The new vertex position is then determined as the last offset before the depth step. For the next horizontal position, the offset increases in the opposite direction, resulting in the determination of a new vertex position C. D is the last position in the row and therefore does not offset. The corresponding method can be applied to the remaining vertex positions, and specifically to other rows of the reference pattern.

[0122] Figure 6 An example of the resulting extracted output depth map (b) is also shown, which includes depth values ​​for the 12 vertex locations AL determined in the 9x13 exemplary depth map (it should be understood that typical depth maps tend to be much larger, and the figures are intended only as illustrative examples). Furthermore, Figure 6An example of the corresponding vertex position graph (c) is shown, which indicates the horizontal shift of each vertex position using simple signed integers.

[0123] therefore, Figure 6 The diagram shows vertices from A to L, with arrows indicating how a vertex moves horizontally from its original position. It demonstrates how a low-resolution depth map can be constructed by storing only the depth values ​​of the deformed vertex positions from the original high-resolution depth map. A vertex position map / deformation map with the same resolution as the output depth map is generated, storing the deformations / offsets of the vertices applied to the rectangular mesh. Since the resulting maps are 2D arrays, they can be efficiently encoded, encapsulated, and compressed using various video coding algorithms.

[0124] As a specific example, the segmenter 203 can be arranged to determine the offset vertex positions using an algorithm that horizontally shifts the vertices based on the initial grid. For example, the following method can be used (where Q indicates the grid distance between vertices, i.e., Q = 4 for a grid with a 4x4 pixel grid, and D indicates the depth value of the input depth map):

[0125] 1. For horizontal terrain For each vertex i searched between

[0126] 2. Determine the position x that maximizes the value. step : Δ=max(|D x -D x-1 |,|D x -D x+1 |)

[0127] 3. If Δ exceeds the given threshold

[0128] 4. Move the vertex position to x step .

[0129] This method attempts to horizontally shift pixels until a new vertex position is found after the boundary depth transformation.

[0130] In some embodiments, the depth map generator 205 may be further arranged to determine offsets (components) in different predetermined directions. Thus, in many embodiments, offsets can be determined in two different predetermined directions. In many embodiments, the determination can be sequential and / or independent. The depth map generator 205 may be arranged to first determine an offset (component) along a first predetermined direction, for example, as previously described. Subsequently, and typically starting from the positional offset in the first direction, a second offset (component) along a second predetermined direction is determined. This determination may, for example, use the same method as for the first direction, but the offset is now along the second direction. The result can be considered as a single offset with two components or equivalently as two offsets in two corresponding directions.

[0131] The two predetermined directions can typically be horizontal and vertical. For example, the offset along the row direction can be determined first, followed by the offset along the column direction.

[0132] exist Figure 7 An example of applying the described method in the horizontal direction is shown. In this example, the depth map generator 205 has already... Figure 4 The rule reference mesh begins. Then, as previously described, the horizontal offset for each vertex is determined, thereby generating... Figure 5 The modified vertex positions. Then, the same method is used to determine the positions along the vertical direction. Figure 5 The offset of the vertex. This results in, for example... Figure 7 The vertex position pattern is shown. It can be seen that the triangular primitives have been adapted to depth variations in the depth map, particularly improvements to the depth step size. The triangles around the depth step are smaller and closer to the depth transformation. This is compensated for by a significant increase in triangle size in flatter regions. In fact, the resulting mesh pattern uses the same number of primitives and vertices as the reference pattern / segmentation to cover the entire depth map. Therefore, the average size of the primitives is changed by the deformation of the regular segmentation / subdivision.

[0133] In many embodiments, the segmenter 203 will be arranged to maintain the same number of primitive / vertex positions for the output depth map / vertex position map for the reference segmentation / vertex position pattern.

[0134] The position map generator 207 is arranged to generate information reflecting two-dimensional offsets. In some embodiments, a vertex position map may be generated, which includes two offset components for each pixel along two predetermined directions.

[0135] However, in many embodiments, the position map generator 207 can be arranged to generate separate vertex position maps for two predetermined directions. Therefore, in addition to the vertex position map indicating an offset in a first direction (e.g., in the horizontal direction), the position map generator 207 can also generate an additional vertex position map indicating an offset in a second direction, such as the vertical direction. Thus, two separate vertex position maps can be generated instead of a single combined vertex position map. Each of these can include a scalar value, and typically the offset can be indicated as an integer representing the number of pixels in which the offset occurs. Figure 8 Showing the corresponding Figure 7 The example shows an offset relative to the original reference pattern, with a first vertex position map (b) indicating the offset in the horizontal direction and a second vertex position map (c) indicating the offset in the vertical direction. As shown in the example, the vertex position maps can typically include small integers and, in many scenarios, can include a large number of zero values.

[0136] The position map generator 207 can feed two vertex position maps to the bitstream generator 209, which is arranged to include both maps in the generated bitstream. A particular advantage of this approach is that it allows for efficient encoding of offset information. For example, each of the vertex position maps can be encoded using an efficient video coding algorithm or other compression techniques optimized for efficient compression of small integer values ​​(with, for example, a large number of identical values). Encoding two separate such two-dimensional maps can be more efficient than encoding the combined vertex position map.

[0137] Therefore, in many embodiments, the bitstream generator 209 can be arranged to encode the output depth map and / or (multiple) vertex position maps using a video coding algorithm. In practice, encoding can be based on lossy video coding techniques. More specifically, for example, assume a common YUV4:2:0 video frame format typically input to a video compression algorithm. The YUV4:2:0 video format has a full-resolution luminance channel but two chrominance channels downsampled by a factor of 4. In this case, the depth map can be placed in the Y component, and the horizontal and vertical displacement maps can be placed in the U and V components. The video encoder can now be configured to correspond to a lossy compression but still high-quality level (quality = 18). Due to lossy coding, some permanent errors may exist in both the depth map and the displacement map. However, this may not have a significant impact on the rendering quality, as it will only mean a small shift in one or more vertices. It should be noted that the video coding utilizes the spatial and temporal redundancy likely present in both the depth map and the displacement map using block-based spatial frequency transformation, followed by quantization and motion prediction.

[0138] Figure 3 The video bitstream used to render the video signal is shown, and specifically from... Figure 2 The device generates a video bitstream, which is an exemplary component of the device for rendering video bitstreams.

[0139] The device includes a receiver 301, which is arranged to receive from Figure 2 The server receives video signals. Therefore, receiver 301 receives video signals comprising a bitstream, which for one or more, and typically all, frames includes a first depth map (the server's output depth map) and a vertex position map indicating the vertex positions of the first depth map in a second depth map (the server's input depth map) having a higher resolution than the first depth map. The vertex positions of the vertex position map define the two-dimensional geometric primitives that form the segments of the second depth map.

[0140] In the example where the receiver is coupled to renderer 303, renderer 303 is arranged to render video images from the received data. Renderer 303 is arranged to generate a view image of the scene based on the viewer's pose input, thereby providing an immersive video experience.

[0141] The renderer 303 includes a mesh generator 305, which is arranged to generate a 3D mesh representation of the scene based on the received information and specifically based on a first depth map and a vertex position map.

[0142] Mesh generator 305 is arranged to generate a mesh to be formed by three-dimensional geometric primitives formed by projecting two-dimensional geometric primitives of a second depth map onto a 3D representation.

[0143] Mesh generator 305 may be specifically arranged to generate a mesh by first determining the vertex positions in a second map (i.e., a map with a higher spatial resolution than the received first depth map). This determination is based on the vertex position map.

[0144] As an example, as previously described, the vertex position map may include vertex position information provided as a position offset relative to a reference pattern of vertex positions. Mesh generator 305 may determine the vertex position corresponding to the depth value of the first depth map as the position in the higher-resolution map corresponding to the reference position, offset by the value indicated in the vertex position map.

[0145] For example, mesh generator 305 can be derived from... Figure 4 Starting from the reference position indicated in the diagram, each position is offset by the offset indicated by the vertex position map. Therefore, in this example, where the vertex position map is used, the vertex position map is used as follows: Figure 6 The vertex position maps shown in (c) and 8(b)(c) are used to recreate the vertex positions determined at the encoder, as follows: Figure 6 As shown in (a) and 8(a).

[0146] The mesh generator 305 is then arranged to project vertex positions from 2D locations in a higher-resolution depth map onto their positions in the 3D space of the mesh. The projection is given based on the 2D positions of the vertex locations and the depth map values. The projection of the 2D vertex locations into 3D space also corresponds to the projection of 2D primitives into 3D primitives, and therefore to the 3D mesh being generated.

[0147] It should be understood that generating a mesh from the depth map and (vertices) positions in the depth map is known to those skilled in the art, and will not be described further for the sake of brevity and clarity.

[0148] The renderer 303 also includes a rendering unit arranged to generate an output image based on the generated mesh and texture map. The rendering unit 307 may specifically receive the view pose of a view image for its desired scene. The rendering unit 307 may then perform a warping of the mesh to a 2D viewport / image for the viewer's pose, and then generate an image from the texture map.

[0149] The rendering unit 307 can, for example, implement a graphics pipeline including vertex shaders, geometry shaders, pixel shaders, etc., as known to those skilled in the art. The renderer 303 can, for example, use a rendering implementation where the mapping is sampled in the vertex shader of the graphics pipeline. First, vertex coordinates can be shifted (referenced) according to the vertex position map in the vertex shader. The shifted coordinates can then be passed to the fragment shader and used to calculate the 3D position of the vertex (which is also the output passed by the vertex shader).

[0150] The rendering unit 307 may use any suitable algorithm or method for generating output images based on a grid, and it should be understood that those skilled in the art will know many such methods.

[0151] In many embodiments, the reference pattern of the vertices, and therefore the reference segmentation or surface subdivision, is a predetermined fixed pattern known at both the transmitting / encoding / server side and the receiving / decoding / client side. For example, the reference pattern may be a standardized pattern that can be applied at both ends.

[0152] However, in some embodiments, more than one option of reference pattern may exist, and the bitstream may include indications of the reference pattern for which a vertex position map is generated. For example, in some embodiments, the segmenter 203 may select among a predetermined set of possible reference patterns and may include an indication of that pattern in the bitstream. For example, an identifier may be assigned to each possible pattern (e.g., by a criterion), and the bitstream may be generated to include the identifier of the given selected pattern. As another example, the reference pattern may be predetermined to be a regular mesh, but the segmenter 203 may dynamically select the size of the mesh. For example, the bitstream may be generated to include variables indicating the distances (in pixels of the input depth map) between reference vertex positions.

[0153] In embodiments where a video bitstream is generated to include an indication of a selected reference segmentation / pattern, renderer 303 may be configured to generate a local copy of the reference pattern based on that indication. For example, it may retrieve a pattern corresponding to a received identifier, or it may continue to generate the position of the pattern by applying a function or rule (such as the distance between points in a regular grid) indicated in the bitstream.

[0154] In some embodiments, a reference pattern can be selected / indicated for the entire video signal, but in other embodiments, the reference pattern can be dynamically adjusted and changed during the video signal. In fact, in some embodiments, an indication of the reference pattern can be provided for each new vertex position map being transmitted / received.

[0155] In many embodiments, this adaptive use of the reference location can provide improved performance. For example, it can allow for better adaptation to specific properties of the depth map and / or video signal. In some cases, the reference pattern can be selected, for example, based on the desired / required maximum bit rate or a given quality level.

[0156] The described method can provide improved operation in many scenarios and embodiments. It can allow for a substantial reduction in the typically substantial resolution of the depth map included in the bitstream / video signal. This method can allow for closer adaptation to the specific scene being represented and can provide improved information to the rendering side of spatial properties. Primitive adaptation to depth variations allows for improved scene representation. It can generally reduce and mitigate errors, artifacts, and inaccuracies. In particular, it can generally reduce or mitigate errors, artifacts, and inaccuracies around depth transformation or step size.

[0157] This method offers improved computational efficiency. In many scenarios, the number of vertices that need to be considered and processed can be significantly reduced. This method can, for example, enable real-time processing and adaptive 3D video rendering on devices with relatively low computational resources in many scenarios.

[0158] This method could, for example, allow for improved immersive video, where the presented / rendered view image adapts to the viewer's posture and specifically to reflect the user's head movement.

[0159] This approach can often allow for an improved user experience and, for example, can frequently allow users to have a more realistic 3D perception.

[0160] It should be understood that although the preceding description focuses on the generated bitstream including a texture map, and on rendering based on that texture map, this is not necessary or essential for the operation, and particularly not necessary for communication of mesh-related information or for creating such information on the client side. For example, in other embodiments, the mesh can be generated without considering the texture map. For example, in some embodiments, the client application can assign material properties to the mesh, introduce light sources into the scene, and render shadowed meshes.

[0161] This invention can be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. The invention can optionally be implemented, at least in part, as computer software running on one or more data processors and / or digital signal processors. The elements and components of embodiments of the invention can be implemented physically, functionally, and logically in any suitable manner. In practice, functionality can be implemented in a single unit, in multiple units, or as part of other functional units. Thus, the invention can be implemented in a single unit or can be physically and functionally distributed among different units, circuits, and processors.

[0162] According to standard terminology in this field, the term pixel can be used to refer to attributes associated with a pixel, such as the light intensity, depth, and position of a portion / element of the scene represented by the pixel. For example, the depth of a pixel or pixel depth can be understood as referring to the depth of the object represented by that pixel. Similarly, the brightness of a pixel or pixel brightness can be understood as referring to the brightness of the object represented by that pixel.

[0163] Although the invention has been described in conjunction with some embodiments, it is not intended to be limited to the specific forms set forth herein. Rather, the scope of the invention is limited only by the appended claims. Furthermore, although features may appear to be described in conjunction with specific embodiments, those skilled in the art will recognize that various features of the described embodiments can be combined according to the invention. In the claims, terms include those that do not exclude the presence of other elements or steps.

[0164] Furthermore, although listed separately, multiple units, elements, circuits, or method steps may be implemented by, for example, a single circuit, unit, or processor. Additionally, although individual features may be included in different claims, these features may be advantageously combined, and inclusion in different claims does not imply that such a combination of features is impractical and / or advantageous. Furthermore, including a feature in one claim class does not imply a limitation on that class, but rather indicates that the feature is equally applicable to other claim classes. Moreover, the order of features in a claim does not imply any particular order in which the features must operate, and in particular, the order of steps in a method claim does not imply that the steps must be performed in such an order. Rather, the steps may be performed in any suitable order. Furthermore, singular references do not exclude plural. Therefore, references to “a(a),” “an,” “first,” “second,” etc., do not exclude plural. The reference numerals in the claims are provided only as clarifying examples and should not be construed as limiting the scope of the claims in any way.

Claims

1. A device for generating a video signal comprising a bit stream, the device include: a receiver (201) arranged to receive a first depth map for at least a first video frame of a plurality of video frames; a segmenter (203) arranged to divide at least a portion of the first depth map into two-dimensional geometric primitives having an average area exceeding twice the area of ​​pixels for the first depth map, and the segmentation comprises determining, for vertices of the geometric primitives, vertex positions in the first depth map; a map generator (207) arranged to generate a vertex position map indicating the positions of the vertices in the first depth map; a depth map generator (205) arranged to generate a second depth map comprising said depth values ​​of said first depth map at said vertex positions in said first depth map; a bitstream generator (209) arranged to generate the video bitstream to include the second depth map and the vertex position map for the first video frame; and in The segmenter (203) is arranged to generate the vertex positions based on the depth values ​​of the first depth map.

2. The device according to claim 1, in, The segmenter (203) is arranged to determine the vertex positions by offsetting reference vertex positions of a reference segmentation of the first depth map; The offset performed for a first vertex depends on values ​​of the depth map for pixels within a geometric primitive of the first vertex.

3. The device according to claim 2, in, The reference vertex positions of the reference segmentation form a regular grid.

4. The device according to any one of claims 2 and 3, in, The vertex position map indicates the vertex positions as relative positions relative to the reference vertex positions of the reference partition.

5. The device according to any one of claims 2 to 4, in, The offset includes a first offset along a first predetermined direction, and the first offset is determined as an offset for which a change in a depth map value along the first predetermined direction satisfies a criterion.

6. The device according to any one of claims 2 to 5, in, the offset comprises a second offset along a second predetermined direction, and the second offset is determined as an offset for which a change in the depth map value along the second predetermined direction satisfies a criterion; The map generator (207) is arranged to generate the vertex position map to indicate the deviation along the first predetermined direction, and also to generate an additional vertex position map to indicate the deviation along the second predetermined direction; as well as The bitstream generator (207) is arranged to include the additional vertex position map in the bitstream.

7. The device according to any one of claims 2 to 6, in, The first predetermined direction is a horizontal direction.

8. The device according to any one of claims 2 to 6, in, The first predetermined direction is a vertical direction.

9. The device according to any one of claims 2 to 8, in, The excursion is constrained to be below an excursion threshold.

10. The device according to any one of the preceding claims, in, The bitstream generator (207) comprises a video encoder arranged to encode the vertex position map using video coding.

11. The device according to any one of the preceding claims, in, The segmenter (203) is arranged to determine the vertex positions to align the geometric primitives with the depth transform in the depth map.

12. A device for rendering a bit stream of a video signal, the device include: A receiver (301) for receiving a video bitstream, the video bitstream comprising, for at least a first frame, a first depth map and a vertex position map indicating vertex positions of depth values ​​of the first depth map in a second map having a higher resolution than the first depth map, the vertex positions defining two-dimensional geometric primitives forming a segmentation of the second map; A renderer (303) arranged to render a video image from the video bitstream, the renderer comprising: a mesh generator (305) arranged to generate a mesh representation of a scene, the mesh being formed of three-dimensional geometric primitives formed by projecting the two-dimensional geometric primitives into a three-dimensional space, the projection for a first vertex of the two-dimensional geometric primitive being dependent on a vertex position of the first vertex in the second depth map and a depth value for the first vertex in the first depth map; as well as A rendering unit (307) is arranged to render an image based on the grid representation.

13. The device according to claim 12, in, The vertex position map indicates the vertex positions as relative vertex positions relative to reference vertex positions of a reference segmentation; and the mesh generator (305) is arranged to determine absolute vertex positions in the second depth map in response to the relative vertex positions and the reference vertex positions.

14. The device according to claim 13, in, The bitstream comprises an indication of the reference partition; and the mesh generator (305) is arranged to determine the reference vertex positions in response to the indication of the reference partition.

15. A method of generating a video signal comprising a bit stream, the method include: receiving a first depth map for at least a first video frame of a plurality of video frames; segmenting at least a portion of the first depth map into two-dimensional geometric primitives having an average area greater than twice an area of ​​pixels for the first depth map, and the segmenting includes determining vertex positions in the first depth map for vertices of the geometric primitives; generating a vertex position map indicating the positions of the vertices in the first depth map; generating a second depth map, the second depth map comprising the depth values ​​of the first depth map at the vertex positions in the first depth map; generating the video bitstream to include the second depth map and the vertex position map for the first video frame; and in Generating the vertex positions depends on the depth values ​​of the depth map.

16. A method for rendering a bit stream of a video signal, the method include: receiving a video bitstream, the video bitstream comprising, for at least a first frame, a first depth map and a vertex position map indicating vertex positions of depth values ​​of the first depth map in a second map having a higher resolution than the first depth map, the vertex positions defining two-dimensional geometric primitives forming a segmentation of the second map; Rendering a video image from the video bitstream, the rendering comprising: generating a mesh representation of a scene, the mesh being formed of three-dimensional geometric primitives, the three-dimensional geometric primitives being formed by projecting the two-dimensional geometric primitives into three-dimensional space, the projection for a first vertex of the two-dimensional geometric primitive being dependent on a vertex position of the first vertex in the second depth map and a depth value for the first vertex in the first depth map; as well as An image is rendered based on the mesh representation.

17. A computer program product comprising computer program code means adapted to perform all the steps of claim 15 or 16 when said program is run on a computer.

18. A video signal comprising a video bitstream, the video bitstream comprising, for at least a first frame, a first depth map and a vertex position map, the vertex position map indicating vertex positions of depth values ​​of the first depth map in a second depth map having a higher resolution than the first depth map, the vertex positions defining segmented two-dimensional geometric primitives forming the second depth map.