Volume video with auxiliary partitioning

Abstract

Methods and apparatuses for encoding and decoding data representing a 3D scene are disclosed. A first set of first tiles is generated from first MVD content acquired from a first region of the 3D scene. A tile is a portion of one of the views of the MVD content. A second set of second tiles is generated from second MVD content acquired from a second region of the 3D scene. An atlas packing the first tiles and the second tiles is generated and the atlas is associated with metadata indicating for a tile of the atlas whether the tile is a first tile or a second tile at a decoding side, the first tiles being used for rendering a viewport image and the second tiles being used for pre-processing or post-processing the viewport image.

Description

1. Technical Field

[0001] The principles of this invention generally relate to the domain of three-dimensional (3D) scenes and volumetric video content. This document is also understood in the context of encoding, formatting, and decoding data representing the textures and geometry of 3D scenes for rendering volumetric content on end-user devices such as mobile devices or head-mounted displays (HMDs). 2. Background Technology

[0002] This section aims to introduce the reader to various aspects of the art that may relate to the aspects of the inventive principles described and / or claimed below. This discussion is believed to help provide the reader with background information to facilitate a better understanding of the various aspects of the inventive principles. Therefore, it should be understood that these statements should be interpreted in this light, rather than as an admission of prior art.

[0003] Recently, the availability of wide field-of-view content (up to 360°) has increased. Users viewing content on immersive display devices (such as head-mounted displays, smart glasses, PC screens, tablets, smartphones, etc.) may not be able to see the entire content. This means that at any given moment, a user can only view a portion of the content. However, users can typically navigate within the content using various means such as head movement, mouse movement, touchscreens, voice, and the like. Encoding and decoding of this content is generally required.

[0004] Immersive video (also known as 360° planar video) allows users to see everything around them by rotating their heads around a stationary viewpoint. Rotation only allows for a 3-degree-of-freedom (3DoF) experience. Even if 3DoF video is sufficient for first-time omnidirectional video experiences (e.g., using a head-mounted display (HMD device)), it can quickly become frustrating for viewers expecting more freedom (e.g., by experiencing parallax). Furthermore, 3DoF can cause dizziness because users never just rotate their heads, but also translate them in three directions, movements that are not reproduced in a 3DoF video experience.

[0005] In this context, large field-of-view content can be three-dimensional computer graphics scenes (3D CGI scenes), point clouds, or immersive videos. Many terms can be used to design such immersive videos: for example, virtual reality (VR), 360, panoramic, 4π spherical, immersive, omnidirectional, or large field of view.

[0006] Volumetric video (also known as 6DoF video) is an alternative to 3DoF video. When watching 6DoF video, in addition to rotation, users can pan their head and even their body within the content being viewed, experiencing parallax and even volume. This type of video significantly increases immersion and perception of scene depth, and prevents motion sickness by providing consistent visual feedback during head panning. The content is created using dedicated sensors, allowing for the simultaneous recording of color and depth of the scene of interest. Even though technical challenges remain, using color camera equipment incorporating photogrammetry is another way to perform this recording.

[0007] While 3DoF video comprises a sequence of images derived from the demapping of textured images (e.g., spherical images encoded according to latitude / longitude projection maps or isometric projection maps), 6DoF video frames embed information from multiple viewpoints. They can be viewed as a temporal series of point clouds generated by 3D capture. Two types of volumetric video can be considered depending on the viewing conditions. The first (i.e., full 6DoF) allows for completely free navigation within the video content, while the second (aka 3DoF+) restricts the user's viewing space to a finite volume called the viewing bounding box, thus allowing for limited head translation and parallax experience. This second case represents a valuable trade-off between free navigation and the passive viewing conditions of a seated audience.

[0008] In 3DoF+ scenarios, one approach involves sending only the information needed to view the 3D scene from any point within the viewing bounding box. Another approach considers sending additional geometric and / or color information, invisible from the viewing bounding box but used for other processes on the decoder side, such as relighting, collision detection, or haptic interaction. This additional information can be delivered in the same format as the visible points. However, a format and method are needed to indicate to the decoder that a portion of the information will be used for rendering and another portion for other processing. 3. Summary of the Invention

[0009] The following is a simplified overview of the principles of the invention to provide a basic understanding of some aspects of these principles. This summary is not a broad overview of the principles of the invention and is not intended to identify key or essential elements of the invention. The following summary presents only some aspects of the principles of the invention in a simplified form as a preface to the more detailed description that follows.

[0010] The present invention relates to a method for encoding data representing a 3D scene in a data stream. The method includes:

[0011] - Generate a set of first chunks based on the first multi-view depth (MVD) content obtained for rendering the 3D scene. Obtain the first MVD from the first region of the 3D scene. A chunk is a portion of a view within the view of the MVD content.

[0012] - Generate a set of second chunks based on the second MVD content obtained for preprocessing or postprocessing purposes; obtain the second MVD from a second region of the 3D scene. The second region may overlap with or be separated from the first region.

[0013] - Generate an atlas using the first and second chunks. The atlas is created by packaging images into chunks according to the atlas layout and associating them with metadata indicating whether a chunk is the first or second chunk; and

[0014] - Encode the atlas in the data stream.

[0015] The principles of this invention also relate to a method for decoding data representing a 3D scene from a data stream.

[0016] The method includes:

[0017] - Decode the data stream to retrieve the atlas and associated metadata. An atlas is an image packaged into chunks according to an atlas layout. A chunk is a portion of a view of MVD content obtained from a region of the 3D scene. Metadata includes data indicating whether a chunk in the atlas is a first chunk or a second chunk; a first chunk is a portion of MVD content obtained from a first region of the 3D scene, and a second chunk is a portion of MVD content obtained from a second region of the 3D scene. The first and second regions may overlap or be separated.

[0018] - Render the viewport image from the viewpoint within the 3D scene using a chunk indicated as the first chunk in the metadata; and

[0019] - The viewport image is preprocessed and / or postprocessed using a block indicated as the second block in the metadata.

[0020] The principles of the present invention also relate to an apparatus including a processor configured to implement the above-described encoding method, and to an apparatus including a processor configured to implement the above-described decoding method.

[0021] The principles of this invention also relate to data streams and / or non-transitory media carrying data representing a 3D scene. Data streams or non-transitory media include:

[0022] - Package the atlas images of the first and second chunks according to the atlas layout. The first chunk is a portion of a view of MVD content acquired for rendering a 3D scene, and the second chunk is a portion of a view of MVD content acquired for preprocessing or post-processing purposes.

[0023] - Metadata associated with the atlas, including data indicating whether a chunk is the first chunk or the second chunk for the atlas. 4. Description of the attached drawings

[0024] This disclosure will be better understood, and further specific features and advantages will emerge after reading the following description with reference to the accompanying drawings, in which:

[0025] - Figure 1 A three-dimensional (3D) model of an object according to a non-limiting embodiment of the principles of the present invention and points of a point cloud corresponding to the 3D model are shown;

[0026] - Figure 2 Non-limiting examples of encoding, transmitting, and decoding data representing a sequence of 3D scenes according to a non-limiting embodiment of the principles of the present invention are shown;

[0027] - Figure 3 The illustration shows a non-limiting embodiment of the invention that can be configured to achieve the following: Figure 8 and Figure 9 An exemplary architecture of the device for the described method;

[0028] - Figure 4 Examples of embodiments of the syntax of a stream when transmitting data via a packet-based transport protocol are shown, according to a non-limiting embodiment of the principles of the present invention;

[0029] - Figure 5 A spherical projection from a central viewpoint is shown as a non-limiting embodiment according to the principles of the present invention;

[0030] - Figure 6 Examples of atlas 60 and atlas 61 generated by an encoder according to a non-limiting embodiment of the principles of the present invention are shown.

[0031] - Figure 7 The diagram illustrates a 3DoF+ rendered view and the acquisition of additional views for auxiliary tiled representations according to a non-limiting embodiment of the principles of the present invention.

[0032] - Figure 8 A method 80 for encoding volumetric video content including auxiliary information, according to a non-limiting embodiment of the present invention, is shown;

[0033] - Figure 9 A method 90 for decoding volumetric video content including auxiliary information, according to a non-limiting embodiment of the principles of the present invention, is shown. 5. Detailed Implementation

[0034] The principles of the invention will be described more fully below with reference to the accompanying drawings, in which examples of the principles of the invention are shown. However, the principles of the invention may be embodied in many alternative forms and should not be construed as limited to the examples set forth herein. Therefore, while the principles of the invention are susceptible to various modifications and alternatives, specific examples are shown by way of example in the drawings and will be described in detail herein. However, it should be understood that there is no intention to limit the principles of the invention to the specific forms disclosed, but rather, this disclosure is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the principles of the invention as defined by the claims.

[0035] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit the principles of the invention. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the stated feature, integer, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as “responding” or “connected” to another element, it may directly respond to or be connected to the other element, or there may be intermediate elements present. Conversely, when an element is referred to as “directly responding” or “directly connected” to another element, there are no intermediate elements present. As used herein, the term “and / or” includes any and all combinations of one or more of the listed related items and may be abbreviated to “ / ”.

[0036] It should be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the teachings of the principles of the invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0037] Although some illustrations include arrows along the communication path to show the main communication direction, it should be understood that communication can occur in the opposite direction to the arrows depicted.

[0038] Examples of block diagrams and operation flowcharts are described where each box represents a circuit element, module, or code section, the code section including one or more executable instructions for implementing a specified logical function. It should also be noted that in other specific implementations, the functions marked in the boxes may not appear in the order they are marked. For example, two boxes shown consecutively may actually be executed substantially simultaneously, or these boxes may sometimes be executed in reverse order depending on the functions involved.

[0039] The references to "according to an example" or "in an example" in this document mean that a particular feature, structure, or characteristic described in connection with the example may be included in at least one specific embodiment of the principles of the invention. The appearance of the phrases "according to an example" or "in an example" in various places in the specification does not necessarily refer to the same example in all instances, nor is it necessarily a separate or alternative example that is mutually exclusive with other examples.

[0040] The reference numerals appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. Although not explicitly described, these examples and variations may be employed in any combination or sub-combination.

[0041] Figure 1 A three-dimensional (3D) model 10 of an object and points corresponding to a point cloud 11 of the 3D model 10 are shown. The 3D model 10 and point cloud 11 may, for example, correspond to possible 3D representations of objects in a 3D scene including other objects. Model 10 may be a 3D mesh representation, and the points of point cloud 11 may be vertices of the mesh. The points of point cloud 11 may also be points distributed on the surface of the mesh face. Model 10 may also be represented as a sputtered version of point cloud 11, the surface of which is created by sputtering the points of point cloud 11. Model 10 may be represented by many different representations such as voxels or splines. Figure 1 This demonstrates that a point cloud can be defined using a surface representation of a 3D object, and that a surface representation of a 3D object can be generated from cloud points. As used herein, projecting points of a 3D object (and by extension, points of a 3D scene) onto an image is equivalent to projecting any representation of that 3D object, such as a point cloud, mesh, spline model, or voxel model.

[0042] Point clouds can be represented in memory as, for example, a vector-based structure, where each point has its own coordinates (e.g., 3D coordinates XYZ, or solid angle and distance from / to the viewpoint (also called depth)) and one or more attributes, also called components, in the viewpoint's frame of reference. An example of components is color components, which can be represented in various color spaces, such as RGB (red, green, and blue) or YUV (Y is the luminance component and UV are the two chrominance components). A point cloud is a representation of a 3D scene including objects. The 3D scene can be viewed from a given viewpoint or viewpoint range. Point clouds can be obtained in various ways, such as:

[0043] • Capture of real objects from camera equipment, optionally supplemented by active depth sensing devices;

[0044] • Capture of virtual / composite objects taken by a virtual camera setup within a modeling tool;

[0045] • A mixture of real and virtual objects.

[0046] Figure 2A non-limiting example of encoding, transmitting, and decoding data representing a sequence of 3D scenes is shown. The encoding format may, for example, be compatible with 3DoF, 3DoF+, and 6DoF decoding simultaneously.

[0047] Obtain 20 3D scene sequences. Just as frame sequences are 2D video, 3D scene sequences are 3D (also known as volumetric) video. These 3D scene sequences can be provided to volumetric video rendering devices for 3DoF, 3DoF+, or 6DoF rendering and display.

[0048] A 3D scene sequence 20 can be provided to an encoder 21. The encoder 21 takes a 3D scene or a sequence of 3D scenes as input and provides a bitstream representing that input. The bitstream can be stored in a memory 22 and / or on an electronic data medium, and can be transmitted via a network 22. The bitstream representing the 3D scene sequence can be read from the memory 22 and / or received from the network 22 by a decoder 23. The decoder 23 takes the bitstream input and provides a 3D scene sequence, for example, in point cloud format.

[0049] Encoder 21 may include several circuits implementing several steps. In a first step, encoder 21 projects each 3D scene onto at least one 2D frame. 3D projection is any method of mapping three-dimensional points onto a two-dimensional plane. This type of projection is widely used, especially in computer graphics, engineering, and drafting, because most current methods for displaying graphics data are based on a planar (pixel information from several bit planes) two-dimensional medium. Projection circuitry 211 provides at least one two-dimensional frame 2111 for the sequence of 3D scenes 20. Frame 2111 includes color information and depth information representing the 3D scene projected onto frame 2111. In a variant, the color and depth information are encoded in two separate frames 2111 and 2112.

[0050] Metadata 212 is used and updated by projection circuitry 211. Metadata 212 includes information about projection operations (e.g., projection parameters) and information about how color and depth information is organized within frames 2111 and 2112, such as... Figures 5 to 7 As stated above.

[0051] The video encoding circuit 213 encodes the sequence of frames 2111 and 2112 into video. The frames 2111 and 2112 of the 3D scene (or the sequence of frames of the 3D scene) are encoded in the stream by the video encoder 213. Then, the video data and metadata 212 are encapsulated in the data stream by the data encapsulation circuit 214.

[0052] Encoder 213 is compatible with, for example, encoders such as:

[0053] -JPEG, specification ISO / CEI 10918-1UIT-T Recommendation T.81, https: / / www.itu.int / rec / T-REC-T.81 / en;

[0054] -AVC, also known as MPEG-4 AVC or h264. It is specified in both UIT-T H.264 and ISO / CEI MPEG-4 Part 10 (ISO / CEI 14496-10), http: / / www.itu.int / rec / T-REC-H.264 / en, HEVC (its specification can be found on the ITU website, T recommendation, H series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en);

[0055] -3D-HEVC (an extension of HEVC, the specification of which can be found on the ITU website, T recommendation, H series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en annex G and I);

[0056] - VP9 developed by Google; or

[0057] - AV1 (AOMedia Video 1) was developed by Alliance for Open Media.

[0058] The data stream is stored in a memory accessible by the decoder 23, for example, via network 22. The decoder 23 includes different circuitry implementing various decoding steps. The decoder 23 takes the data stream generated by the encoder 21 as input and provides a sequence 24 of 3D scenes to be rendered and displayed by a volumetric video display device, such as a head-mounted display (HMD). The decoder 23 obtains the stream from the source 22. For example, the source 22 belongs to a group that includes:

[0059] - Local storage, such as video storage or RAM (or random access memory), flash memory, ROM (or read-only memory), hard disk;

[0060] - Storage interfaces, such as interfaces for mass storage devices, RAM, flash memory, ROM, optical discs, or magnetic media;

[0061] - Communication interfaces, such as wired interfaces (e.g., bus interfaces, WAN interfaces, LAN interfaces) or wireless interfaces (e.g., IEEE 802.11 interfaces or...). Interface); and

[0062] - User interfaces that enable users to input data, such as graphical user interfaces.

[0063] Decoder 23 includes circuitry 234 for extracting data encoded in the data stream. Circuitry 234 takes the data stream as input and provides metadata 232 corresponding to metadata 212 encoded in the stream and two-dimensional video. The video is decoded by video decoder 233, which provides a sequence of frames. The decoded frames include color and depth information. In a variant, video decoder 233 provides two frame sequences, one containing color information and the other containing depth information. Circuitry 231 uses metadata 232 to deproject the color and depth information from the decoded frames to provide a 3D scene sequence 24. 3D scene sequence 24 corresponds to 3D scene sequence 20, potentially resulting in a loss of accuracy associated with encoding and video compression as 2D video.

[0064] For example, additional circuitry and functionality can be added before or in post-processing steps following the non-projection step of circuit 231. For instance, circuitry can be added to re-illuminate the scene from another light located anywhere in the scene. Collision detection can be performed on depth compositing, for example, to add new objects to the 3DoF+ scene in a consistent, realistic manner or for path planning. Such circuitry may require geometric and / or color information about the 3D scene that is not used for the 3DoF+ rendering itself. The semantics of these different kinds of information must be indicated through a bitstream representing the 3DoF+ scene.

[0065] Figure 3 It shows that it can be configured to implement about Figure 8 and Figure 9 An exemplary architecture of device 30 for the described method. Figure 2 The encoder 21 and / or decoder 23 can implement this architecture. Alternatively, each circuit in the encoder 21 and / or decoder 23 can be based on... Figure 3 Devices with an architecture that are linked together, for example, via their bus 31 and / or via I / O interface 36.

[0066] Device 30 includes the following components connected together via data and address bus 31:

[0067] - Microprocessor 32 (or CPU), which is, for example, a DSP (or digital signal processor);

[0068] -ROM (or read-only memory) 33;

[0069] -RAM (or random access memory) 34;

[0070] - Storage interface 35;

[0071] -I / O interface 36, which is used to receive data to be transmitted from the application; and

[0072] - Power source, such as a battery.

[0073] According to one example, the power supply is external to the device. In each mentioned memory, the term "register" used in the specification can correspond to a small area (a few bits) or a very large area (e.g., the entire program or a large amount of received or decoded data). ROM 33 includes at least the program and parameters. ROM 33 can store algorithms and instructions for executing the technology according to the principles of the invention. When powered on, CPU 32 loads the program from RAM and executes the corresponding instructions.

[0074] RAM 34 includes a program executed by CPU 32 and uploaded after device 30 is turned on, input data in the register, intermediate data in different states of the method in the register, and other variables used to execute the method in the register.

[0075] The specific embodiments described herein may be implemented, for example, in methods or processes, apparatus, computer program products, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method or apparatus), the specific implementation of the discussed features may be implemented in other forms (e.g., programs). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented in apparatus (such as, for example, a processor) that generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0076] According to the example, device 30 is configured to implement about Figure 8 and Figure 9 The described method belongs to a set that includes the following items:

[0077] -mobile device;

[0078] - Communication equipment;

[0079] -Gaming devices;

[0080] - Tablet PC (or tablet computer);

[0081] - Laptop;

[0082] - Still image camera;

[0083] -Camera;

[0084] - Encoding chip;

[0085] - Servers (such as broadcast servers, video-on-demand servers, or web servers).

[0086] Figure 4 An example of an implementation of the syntax for streams is shown when data is transmitted via a packet-based transport protocol. Figure 4 An exemplary structure 4 for a volumetric video stream is shown. This structure is contained within a container that organizes the stream by syntactic, independent elements. The structure may include a header section 41, which is a set of data common to each syntactic element of the stream. For example, the header section includes metadata about the syntactic elements, describing the properties and roles of each of them. The header section may also include... Figure 2 This is part of the metadata 212, such as the coordinates of the center viewpoint used to project points of the 3D scene onto frames 2111 and 2112. The structure includes a payload comprising syntax element 42 and at least one syntax element 43. Syntax element 42 includes data representing color and depth frames. The image may have been compressed according to a video compression method.

[0087] Syntax element 43 is part of the payload of the data stream and may include metadata about how the frames of syntax element 42 are encoded, such as parameters for projecting points of the 3D scene onto the frames. Such metadata may be associated with each frame or group of frames of the video (also known as a group of frames (GoP) in video compression standards).

[0088] Figure 5 A tiled atlas method is illustrated using four projection centers as an example. The 3D scene 50 includes characters. For example, projection center 51 is a perspective camera, and camera 53 is an orthophoto camera. The camera can also be an omnidirectional camera with, for example, a spherical mapping (e.g., an isorectangular mapping) or a cubic mapping. Based on the projection operations described in the projection data of the metadata, 3D points of the 3D scene are projected onto a 2D plane associated with a virtual camera located at the projection center. Figure 5 In the example, the projection of the points captured by camera 51 is mapped onto block 52 according to perspective mapping, and the projection of the points captured by camera 53 is mapped onto block 54 according to orthophoto mapping.

[0089] Clustering of projected pixels produces multiple 2D tiles, which are packed into a rectangular atlas 55. The organization of tiles within the atlas defines the atlas layout. In an embodiment, two atlases have the same layout: one for texture (i.e., color) information and one for depth information. Two tiles captured by the same camera or by two different cameras may include information representing the same portion of the 3D scene, such as, for example, tiles 54 and 56.

[0090] The packing operation generates chunk data for each generated chunk. Chunk data includes references to the projection data (e.g., an index in the projection data table or a pointer to the projection data (i.e., an address in memory or the data stream)) and information describing the location and size of the chunk within the atlas (e.g., top-left corner coordinates, size, and width in pixels). Chunk data items are added to metadata to be encapsulated in the data stream in association with the compressed data of one or two atlases.

[0091] Figure 6 Examples of atlases 60 and 61 generated by an encoder are shown. According to a non-limiting embodiment of the invention, atlases 60 and 61 include texture information (e.g., RGB or YUV data) of points in a 3D scene. (See also: ...) Figure 5 The atlas is described as a packaged, chunked collection of images. For example, in... Figure 6 In the example, the encoder takes a multi-view plus depth video consisting of three views, 62, 63, and 64, as input. The encoder removes redundancy between views (a pruning step) and packs the selected texture and depth chunks into one or more atlases. Thus, the bitstream consists of multiple video streams (e.g., HEVC video streams) carrying atlases with texture and depth chunks, along with metadata describing the camera parameters and atlas layout of the input views.

[0092] A tiled atlas consists of texture and depth atlas component pairs, with the same image size and layout (same wrapping) for both texture and depth. In one approach, the atlas carries only the information needed for 3DoF+ rendering of the scene from any point within the viewing bounding box. In another approach, the atlas may carry additional geometric and / or color information for other processing, such as scene relighting or collision detection. For example, this additional information could be the geometry of the back faces of objects in the 3D scene. Such tiles are called auxiliary tiles. These tiles are not intended to be rendered by the decoder but are used by the decoder's preprocessing or post-processing circuitry.

[0093] Figure 7This illustrates the acquisition of the 3DoF+ rendered view and additional views of auxiliary tiles. On the encoder side, the generation of auxiliary tiles can be performed using different methods. For example, to acquire scene 70, a first set 71 of actual or virtual cameras can be placed pointing towards the front of scene 70. A second set 72 of actual or virtual cameras is placed to view the back and sides of the volumetric scene. In this implementation, the view captured by camera 72 has a lower resolution than that of camera 71. Camera 72 acquires the geometry and / or color of hidden parts of objects. Tiles obtained from the view captured by camera 71 are tiles used for 3DoF+ rendering, while tiles obtained from the view captured by camera 72 are auxiliary tiles used to describe geometric and / or color information for preprocessing or post-processing purposes. Metadata associated with the tiles of the atlas can be formatted to signal the semantics of each tile. On the decoder side, the viewport renderer must skip tiles that are invalid for rendering. The metadata also indicates which modules of the decoder can use these invalid rendering tiles. For example, the relighting circuitry will use this auxiliary information to update its geometry from the light's viewpoint and accordingly change the lighting texture of the entire scene to generate appropriate shadows.

[0094] A method for generating auxiliary blocks, typically captured at lower resolution from the back and sides, describing the geometry of the rear portion of an object and associated with camera 72, is added to the encoder. First, additional depth views of the back and sides must be obtained, which can be done in various ways. In the case of synthetically generated objects, depth images associated with a virtual camera placed anywhere are obtained directly from the 3D model. For native 3D capture, additional color and / or active depth cameras can be added during the shooting phase: the depth camera directly provides the depth view, while the photogrammetric algorithm estimates the depth based on the color view. When neither the 3D model nor the additional capture is available, a convex shape completion algorithm can be used to generate seemingly realistic closed shapes based on open-form geometry recovered from a front camera. Then, redundancy between views is removed by trimming, similar to the method performed on the view of camera 71. In this implementation, trimming is performed independently on both sets of views. Therefore, potential redundancy between regular blocks and additional blocks is not removed. The resulting auxiliary depth blocks are packaged together with the regular blocks within the depth block atlas.

[0095] In another implementation, if auxiliary tiles are defined only for depth, the texture portions of the atlas will remain blank if the same layout is used for both the texture atlas and the depth atlas. Even though these auxiliary tiles should be defined at a lower resolution, this results in a loss of space in the texture atlas. In such implementations, different layouts can be used for both the depth atlas and the texture atlas, and this difference is indicated in the metadata associated with the atlas.

[0096] Possible syntax for describing the metadata of an atlas may include a high-level concept called 'entity_id': this entity_id allows chunk groups to be attached to indexes for advanced semantic processing, such as object filtering or composition. The following table illustrates possible syntax a for atlas parameter metadata.

[0097]

[0098]

[0099] According to an embodiment of the principles of the present invention, auxiliary blocks are identified as specific entities, referred to as auxiliary entities. The number of entities and their functions (i.e., whether they are auxiliary entities) are then described in the metadata as shown in the following table:

[0100]

[0101] An auxiliary_flag value of 1 indicates that each entity structure has an auxiliary description.

[0102] The value of auxiliary_entity_flag[e] equal to 1 indicates that the chunks associated with entity e are not used for viewport rendering.

[0103] In another embodiment of the present invention, auxiliary blocks are signaled at the block level by modifying the atlas parameter syntax as shown in the table below:

[0104]

[0105]

[0106] An auxiliary_flag value of 1 indicates that each block structure has an auxiliary flag.

[0107] The value of auxiliary_patch_flag[a][p] equal to 1 indicates that the atlas patch p is not used for viewport rendering.

[0108] In another implementation, the chunking information data syntax defines auxiliary chunking tags, as shown in the table below:

[0109]

[0110] On the decoding side, auxiliary_patch_flag is used to determine whether the patch includes information for rendering and / or information from another module.

[0111] Figure 8A method 80 for encoding volumetric video content including auxiliary information, according to a non-limiting embodiment of the invention, is illustrated. In step 81, chunks intended for 3DoF+ rendering are generated, for example, by trimming redundant information from multi-view depth content acquired by a first set of cameras. In step 82, auxiliary chunks are generated from views captured by cameras that capture portions of the scene not intended for rendering. Steps 81 and 82 can be performed in parallel or sequentially. The views used to generate the auxiliary chunks are captured by a second set of cameras, for example, located at the back and sides of the 3D scene. For example, auxiliary chunks are generated by trimming redundant information included in the views captured by the first and second sets of cameras. In another embodiment, for example, auxiliary chunks are generated by trimming redundant information included in the views captured only by the second set of cameras. In this embodiment, redundancy may exist between the 3DoF+ chunks and the auxiliary chunks. In step 83, an atlas is generated by packing the 3DoF+ chunks and the auxiliary chunks into the same image. In this embodiment, the packing layout is different for the depth and color components of the atlas. Metadata describing the atlas parameters and tile parameters is generated according to the syntax described in the table above. The metadata includes information indicating for each tile whether it is a 3DoF+ tile intended for rendering or an auxiliary tile intended for preprocessing and / or post-processing. In step 84, the generated atlas and associated metadata are encoded in a data stream.

[0112] Figure 9 A method 90 for decoding volumetric video content including auxiliary information, according to a non-limiting embodiment of the principles of the present invention, is illustrated. At step 91, a data stream representing the volumetric content is obtained from a stream. The data stream is decoded to retrieve an atlas and associated metadata. The atlas is an image that packages at least one chunk according to a packing layout. A chunk is an image that includes depth and / or color information representing a portion of a 3D scene. The metadata includes information for back-projecting the chunks and retrieving the 3D scene. At step 92, the chunks are unpacked from the atlas, and properties are attributed to each chunk based on information included in the metadata. The chunks may be 3DoF+ chunks intended for rendering a viewport image at step 93, or auxiliary chunks intended for preprocessing or post-processing operations at step 94. Steps 93 and 94 may be performed in parallel or sequentially.

[0113] The specific implementations described herein may be implemented, for example, in methods or processes, apparatus, computer program products, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method or apparatus), the specific implementations of the discussed features may also be implemented in other forms (e.g., programs). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented in apparatus (such as, for example, a processor) that generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, smartphones, tablets, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0114] Specific implementations of the various processes and features described herein can be found in a wide variety of devices or applications, particularly those associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture and / or depth information. Examples of such devices include encoders, decoders, post-processors that process the output from decoders, pre-processors that provide input to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, cellular phones, PDAs, and other communication devices. It should be understood that the devices can be mobile, even mounted in mobile vehicles.

[0115] Additionally, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) can be stored on a processor-readable medium, such as, for example, an integrated circuit, a software carrier, or other storage device, such as, for example, a hard disk, a compact disk (“CD”), an optical disk (such as, for example, a DVD, commonly referred to as a digital versatile optical disk or digital video optical disk), random access memory (“RAM”), or read-only memory (“ROM”). Instructions can form an application program tangibly embodied on the processor-readable medium. Instructions can be, for example, hardware, firmware, software, or a combination thereof. Instructions can be found, for example, in an operating system, a standalone application, or a combination of both. Thus, a processor can be characterized, for example, as a device configured to execute a process and a device comprising a processor-readable medium (such as a storage device) having instructions for executing the process. Furthermore, in addition to or instead of instructions, the processor-readable medium can store data values ​​generated by the implementation.

[0116] It will be apparent to those skilled in the art that the embodiments may produce various signals formatted to carry, for example, storable or transmissible information. The information may include, for example, instructions for performing a method or data generated by one of the embodiments. For example, the signal may be formatted as data carrying rules for writing or reading the syntax of the described embodiment, or as data carrying actual syntax values ​​written by the described embodiment. Such signals may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and using a modulated carrier for the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0117] Several specific embodiments have been described. However, it should be understood that many modifications can be made. For example, elements of different embodiments can be combined, supplemented, modified, or removed to produce other embodiments. Furthermore, those skilled in the art will understand that other structures and processes can be replaced with those disclosed, and the resulting embodiments will perform at least substantially the same function in at least substantially the same manner to achieve at least substantially the same results as the disclosed embodiments. Therefore, this application considers these and other embodiments.

Claims

1. A method for encoding a 3D scene in a data stream, the method comprising: - Generate a set of first chunks based on the first multi-view depth-plus-depth (MVD) content obtained for rendering the 3D scene, wherein the first chunk is a portion of a view of the first MVD content used to render the viewport image at the decoder side; - Generate a set of second chunks based on the second MVD content, the second chunks being a portion of a view of the second MVD content that is not used to render the viewport image at the decoder side; - Generate at least one atlas that packages the first chunk and the second chunk, wherein the atlas is an image of the chunk packaged according to the atlas layout; as well as - The at least one atlas and metadata are encoded in the data stream, the metadata indicating whether a block in one or more chunk groups within the at least one atlas is a first chunk or a second chunk.

2. The method according to claim 1, wherein, The second block is used on the decoder side for processing other than rendering.

3. The method according to claim 2, wherein, The other processing is at least one of collision detection and scene relighting.

4. The method according to claim 1 or 2, wherein, The second MVD is obtained at a resolution lower than that of the first MVD.

5. The method according to claim 1 or 2, wherein, Chunking is a portion of a view in an MVD obtained by removing information redundancy between views in the MVD.

6. A method for decoding a 3D scene from a data stream, the method comprising: - Decode from the data stream at least one atlas that packages a first chunk and a second chunk according to the atlas layout, and metadata indicating for one or more chunk groups within the at least one atlas whether the chunk in the group is a first chunk or a second chunk, the first chunk being a portion of a view for rendering a first MVD content of a viewport image, and the second chunk being a portion of a view for not rendering a second MVD content of the viewport image. as well as - The viewport image is rendered from a viewpoint within the 3D scene using the first block.

7. The method according to claim 6, wherein, The second block is used for processing other than rendering.

8. The method according to claim 7, wherein, The other processing is at least one of collision detection and scene relighting.

9. The method according to claim 6 or 7, wherein, The resolution of the second MVD is lower than that of the first MVD.

10. An apparatus for encoding a 3D scene in a data stream, the apparatus including a memory associated with a processor configured to: - Generate a set of first chunks based on the first multi-view depth-plus-depth (MVD) content obtained for rendering the 3D scene, wherein the first chunk is a portion of a view of the first MVD content used to render the viewport image at the decoder side; - Generate a set of second chunks based on the second MVD content, the second chunks being a portion of a view of the second MVD content that is not used to render the viewport image at the decoder side; - Generate at least one atlas that packages the first chunk and the second chunk, wherein the atlas is an image of the chunk packaged according to the atlas layout; as well as - The at least one atlas and metadata are encoded in the data stream, the metadata indicating whether a block in one or more chunk groups within the at least one atlas is a first chunk or a second chunk.

11. The device according to claim 10, wherein, The second block is used on the decoder side for processing other than rendering.

12. The device according to claim 11, wherein, The other processing is at least one of collision detection and scene relighting.

13. The device according to claim 10 or 11, wherein, The second MVD is obtained at a resolution lower than that of the first MVD.

14. The device according to claim 10 or 11, wherein, Chunking is a portion of a view in an MVD obtained by removing information redundancy between views in the MVD.

15. An apparatus for decoding a 3D scene from a data stream, the apparatus comprising a processor configured to: - Decode from the data stream at least one atlas, packaged according to the atlas layout, including a first chunk and a second chunk, and metadata indicating, for one or more chunk groups within the at least one atlas, whether a chunk in the group is a first chunk or a second chunk, the first chunk being a portion of a view used to render first MVD content of a viewport image, and the second chunk being a portion of a view not used to render second MVD content of the viewport image; and - The viewport image is rendered from a viewpoint within the 3D scene using the first block.

16. The device according to claim 15, wherein, The second block is used for processing other than rendering.

17. The device according to claim 16, wherein, The other processing is at least one of collision detection and scene relighting.

18. The device according to claim 15 or 16, wherein, The resolution of the second MVD is lower than that of the first MVD.

Images