Method and apparatus for encoding / decoding 3D scene

By decoding volume frame sequences from the data stream and using 3D motion information for resampling and time alignment, the artifact problem caused by frame rate misalignment in immersive videos is solved, and the immersion and rendering stability are improved.

CN119072718APending Publication Date: 2024-12-03INTERDIGITAL CE PATENT HOLDINGS SAS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380035458.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-21
Filing Date
2023-04-06
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing immersive video technologies have artifact problems when rendering 3D scenes, especially when frame rates are aligned or multiple object substreams at different frame rates, resulting in dizziness and reduced immersion.

Method used

By decoding the volume frame sequence from the data stream, 3D motion information is obtained and resampled or time-aligned, the points of the volume frame are shifted to the combined time frame using the 3D motion information to achieve accurate reconstruction and rendering of the frame sequence.

Benefits of technology

It effectively solves the artifact problem caused by frame rate misalignment, improves immersion and scene depth perception, prevents dizziness, and ensures stable rendering effect at different frame rates.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072718A_ABST
    Figure CN119072718A_ABST
Patent Text Reader

Abstract

A method and apparatus for reconstructing at least one sequence of volumetric frames from a data stream is provided in which the sequence of volumetric frames is temporally resampled and / or temporally aligned with a combined temporal frame using 3D motion information obtained for the volumetric frames. The method comprises: decoding at least one sequence of volumetric frames from a data stream; obtaining 3D motion information representative of a displacement in 3D space of a point of a volumetric frame of the at least one volumetric frame sequence; and shifting points of at least one volumetric frame of the sequence to a combined temporal frame using 3D motion information.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to European application number 22305591.4 filed on April 21, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The principles of the present invention generally relate to the field of three-dimensional (3D) scenes and volumetric video content. This document is also understood in the context of encoding, formatting and decoding data representing the texture and geometry of a 3D scene in order to render volumetric content on an end-user device, such as a mobile device or a head mounted display (HMD). Background Art

[0004] This section is intended to introduce the reader to various areas of the art that may be related to various aspects of the principles of the present invention described and / or claimed below. This discussion is considered to be helpful in providing the reader with background information to promote a better understanding of various aspects of the principles of the present invention. Therefore, it should be understood that these statements will be interpreted from this perspective and not as an admission of prior art.

[0005] Recently, there has been a growth in the availability of large field of view content (up to 360°). Users viewing such content on an immersive display device (such as a head mounted display, smart glasses, PC screen, tablet, smartphone, etc.) may not be able to see the entire content. This means that at a given moment, the user may only be viewing a portion of the content. However, the user can typically navigate through the content by various means such as head movement, mouse movement, touch screen, voice, etc. It is often desirable to encode and decode this content.

[0006] Immersive video (also called 360° planar video) allows the user to see everything around him or her by rotating his or her head around a stationary viewpoint. Rotation only allows an experience with 3 degrees of freedom (3DoF). Even though 3DoF video is sufficient for a first omnidirectional video experience, e.g. using a head mounted display (HMD), it can quickly become frustrating for viewers who expect more degrees of freedom, e.g. by experiencing parallax. Additionally, 3DoF can also cause dizziness, because the user never just rotates his or her head, but also translates it in three directions, and translation cannot be reproduced in a 3DoF video experience.

[0007] Large field of view content may include, among other things, three-dimensional computer graphics scenes (3D CGI scenes), point clouds, or immersive videos. Many terms may be used to design such immersive videos: for example, virtual reality (VR), 360, panoramic, 4π steradian, immersive, omnidirectional, or large field of view.

[0008] Volumetric video, also known as 6 degrees of freedom (6DoF) video, is an alternative to 3DoF video. When watching 6DoF video, the user can translate his head and even his body within the viewed content, in addition to rotation, and experience parallax and even volume. This type of video greatly increases immersion and scene depth perception and prevents dizziness by providing consistent visual feedback during head translation. The content is created with the help of dedicated sensors, allowing the simultaneous recording of color and depth of the scene of interest. Using a color camera setup in combination with photogrammetry techniques is a way to perform this type of recording, even if there are still technical difficulties.

[0009] While 3DoF videos consist of a sequence of images resulting from unmapping of texture images (e.g., spherical images encoded according to latitude / longitude projection mapping or equirectangular projection mapping), 6DoF video frames embed information from multiple viewpoints. They can be viewed as a temporal sequence of point clouds resulting from three-dimensional capture. Depending on the viewing conditions, two types of volumetric videos can be considered. The first (i.e., full 6DoF) allows completely free navigation within the video content, while the second (also known as 3DoF+) restricts the user viewing space to a limited volume (called the viewing bounding box), allowing limited head translation and parallax experience. The second case is a valuable trade-off between free navigation for seated audience members and passive viewing conditions.

[0010] When a 3D scene contains multiple objects encoded in separate substreams that may not be temporally aligned or encoded at different frame rates, artifacts may appear when rendering the reconstructed 3D scene from the substreams. In a similar manner, when a sequence of volumetric frames is to be rendered at a frame rate different from the frame rate at which the sequence was encoded, some artifacts may appear in the reconstructed sequence of volumetric frames rendered at the desired frame rate. Summary of the invention

[0011] The following presents a simplified overview of the principles of the present invention, so as to provide a basic understanding of some aspects of the principles of the present invention. Summary of the invention is not an exhaustive summary of the principles of the present invention. It is not intended to point out the important or key elements of the principles of the present invention. The following summary of the invention only presents some aspects of the principles of the present invention in a simplified form, as a preface to the more detailed description provided below.

[0012] According to one aspect of the principles of the present invention, a method and apparatus are provided for reconstructing at least one sequence of volumetric frames from a data stream, wherein the sequence of volumetric frames is temporally resampled and / or temporally aligned with a combined time frame using 3D motion information obtained for the volumetric frames.

[0013] According to one aspect of the principles of the present invention, a method for reconstructing at least one volumetric frame sequence from a data stream is provided, wherein the method comprises: decoding the at least one volumetric frame sequence from the data stream; obtaining 3D motion information representing the displacement of points of the volumetric frames of the at least one volumetric frame sequence in 3D space; and using the 3D motion information to shift the points of at least one volumetric frame of the sequence to a combined time frame.

[0014] According to another aspect of the principles of the present invention, a device is provided comprising one or more processors configured to reconstruct at least one volumetric frame sequence from a data stream, wherein the one or more processors are configured to: decode the at least one volumetric frame sequence from the data stream; obtain 3D motion information representing the displacement of points of the volumetric frames of the at least one volumetric frame sequence in 3D space; and use the 3D motion information to shift the points of at least one volumetric frame of the sequence to a combined time frame.

[0015] According to another aspect, a method or device configured to reconstruct at least one volumetric frame sequence is provided, wherein the reconstruction comprises: decoding the at least one volumetric frame sequence from the data stream; obtaining 3D motion information representing the displacement of points of a volumetric frame of the at least one volumetric frame sequence in a 3D space; and resampling the at least one volumetric frame sequence at a frame rate different from a frame rate used during encoding, wherein the resampling uses the 3D motion information.

[0016] According to another aspect, a method for encoding at least one volume frame sequence representing a 3D scene is provided. The method comprises: obtaining a patch-atlas based representation for at least one volume frame of the sequence; obtaining 3D motion information representing the displacement of deprojected samples of the patch-atlas based representation in 3D space between two volume frames of the sequence; encoding the patch-atlas based representation and the 3D motion information in a data stream.

[0017] According to another aspect, a device includes one or more processors configured to encode at least one volumetric frame sequence, wherein the one or more processors are further configured to: obtain a patch-atlas-based representation for at least one volumetric frame of the sequence; obtain 3D motion information representing the displacement of de-projected samples of the patch-atlas-based representation in 3D space between two volumetric frames of the sequence; and encode the patch-atlas-based representation and the 3D motion information in a data stream.

[0018] According to an embodiment, the combined time frame is different from the time frame encoded in the data stream. According to an embodiment, the at least one volumetric frame sequence represents a 3D scene including at least two objects encoded in separate substreams of the data stream. In yet another embodiment, shifting points of the at least one volumetric frame comprises: using the 3D motion information to motion compensate the decoded volumetric frame. In another embodiment, each of the at least two objects of the 3D scene is encoded as a sequence of volumetric frames.

[0019] In some embodiments, the 3D motion information is encoded as a 3D motion properties video substream. In other embodiments, the 3D motion information is encoded as metadata associated with at least one patch of the patch-atlas based representation of the at least one volumetric frame sequence.

[0020] In some embodiments, at least one syntax element indicating the presence of 3D motion information is encoded in said data stream.

[0021] One or more embodiments also provide a computer program, the computer program including instructions, which, when executed by one or more processors, cause the one or more processors to perform an encoding method or a decoding method according to any one of the embodiments described herein. One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding a sequence of volume frames according to the method described herein. One or more embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the method described herein. One or more embodiments also provide a method and apparatus for transmitting or receiving a bitstream generated according to the method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present disclosure will be better understood, and other specific features and advantages will appear, upon reading the following description, which refers to the accompanying drawings, in which:

[0023] Figure 1 A three-dimensional (3D) model of an object and points of a point cloud corresponding to the 3D model are shown in accordance with a non-limiting embodiment of the principles of the present invention;

[0024] Figure 2 A non-limiting example of encoding, transmission and decoding of data representing a sequence of 3D scenes according to a non-limiting embodiment of the principles of the present invention is shown;

[0025] Figure 3 Non-limiting embodiments showing principles according to the present invention may be configured to implement Figures 8 to 18 Example architecture of apparatus for the described method;

[0026] Figure 4 An example of an embodiment of the syntax of a data stream transmitted over a packet-based transport protocol according to a non-limiting embodiment of the principles of the present invention is shown;

[0027] Figure 5 shows a spherical projection from a central viewpoint in accordance with a non-limiting embodiment of the principles of the present invention;

[0028] Figure 6 An example of an atlas including texture information of points of a 3D scene according to a non-limiting embodiment of the principles of the present invention is shown;

[0029] Figure 7 A non-limiting embodiment according to the principles of the present invention is shown including Figure 6 An example of an atlas of depth information of points of a 3D scene;

[0030] Figure 8 An example of a segmentation map of an object provided for one of the MIV test sequences is shown;

[0031] Fig. 9 An example of a 3D scene consisting of 4 objects is shown;

[0032] Fig.10 An example of object-based MIV substreams that are not aligned in time is shown;

[0033] Fig.11 An example of a method for encoding a 3D scene according to an embodiment is shown;

[0034] Fig.12 An example of a method for decoding a 3D scene according to an embodiment is shown;

[0035] Fig.13 An example of 3D motion estimation of point cloud samples is shown;

[0036] Fig.14 An example of a method for reconstructing a 3D scene with multiple 3D objects encoded at different frame rates according to an embodiment is shown;

[0037] Fig.15 An example of a MIV bitstream with 3D motion attribute video according to an embodiment is shown;

[0038] Fig.16 Examples of point-based (a) and block-based (b) 3D motion compensation are shown;

[0039] Fig.17 An example of an object-based encoding method according to an embodiment is shown;

[0040] Fig.18 An example of an object-based decoding method according to an embodiment is shown. DETAILED DESCRIPTION

[0041] The principles of the present invention will now be described more fully below with reference to the accompanying drawings, in which examples of the principles of the present invention are shown. However, the principles of the present invention can be embodied in many different forms and should not be construed as being limited to the examples set forth herein. Therefore, although the principles of the present invention are susceptible to various modifications and alternative forms, specific examples of the principles of the present invention are still shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that the principles of the present invention are not intended to be limited to the specific forms disclosed, on the contrary, the present disclosure will cover all modifications, equivalents and alternatives within the spirit and scope of the principles of the present invention as defined in the appended claims.

[0042] The terms used herein are only used for the purpose of describing specific examples and are not intended to limit the principles of the present invention. As used herein, the singular forms "one", "an" and "said" are also intended to include plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises", "comprising", "includes" and / or "including" when used in this specification stipulate that there are stated features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups thereof. In addition, when an element is referred to as "responsive to" or "connected to" another element, it can directly "respond to" or "connect to" another element, or there can be an intermediate element. On the contrary, when an element is referred to as "directly responsive to" or "directly connected to" another element, there is no intermediate element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and can be abbreviated as " / ".

[0043] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the teaching of the principles of the present invention.

[0044] Although some diagrams include arrows on communication paths to illustrate a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0045] Some examples are described with respect to block diagrams and operational flow charts, wherein each box represents a circuit element, module or code portion including one or more executable instructions for implementing a specified logical function. It should also be noted that in other implementations, the functions mentioned in the box may not occur in the order mentioned. For example, depending on the functions involved, two boxes shown in succession may actually be executed substantially simultaneously, or boxes may sometimes be executed in reverse order.

[0046] References herein to "according to an example" or "in an example" mean that a particular feature, structure, or characteristic described in conjunction with the example may be included in at least one implementation of the principles of the present invention. The phrases "according to an example" or "in an example" appearing in different places in the specification do not necessarily all refer to the same example, nor are separate or alternative examples necessarily mutually exclusive of other examples.

[0047] Reference numerals appearing in the claims are for illustration purposes only and shall not have any limiting effect on the scope of the claims.The examples and variations of the invention may be used in any combination or sub-combination even if not explicitly described.

[0048] The MIV standard (ISO / IEC 23090-12:2022 MPEG Immersive Video) allows volumetric information to be carried as a combination of 2D patches stored in atlas frames, encoded using a conventional 2D video codec (usually the HEVC or VVC standards). Each patch corresponds to a projection of a sub-portion of the 3D input scene, with its geometry, various properties and associated camera parameters. So far, according to the MIV profile, the following atlas components can be present in the bitstream: geometry (depth map), occupancy (binary map of pixel significance), color properties and transparency properties.

[0049] Figure 1 A three-dimensional (3D) model 10 of an object and the points of a point cloud 11 corresponding to the 3D model 10 are shown. The 3D model 10 and the point cloud 11 may correspond, for example, to a possible 3D representation of an object of a 3D scene including other objects. The model 10 may be a 3D mesh representation and the points of the point cloud 11 may be vertices of the mesh. The points of the point cloud 11 may also be points distributed on the surface of a mesh face. The model 10 may also be represented as a stitched version of the point cloud 11, the surface of the model 10 being created by stitching the points of the point cloud 11. The model 10 may be represented by many different representations, such as voxels or splines. Figure 1The following fact is shown: a point cloud can be defined with a surface representation of a 3D object and a surface representation of a 3D object can be generated from the points of the cloud. As used herein, projecting the points of a 3D object (by extension of the 3D scene) onto an image is equivalent to projecting any representation of the 3D object (e.g., a point cloud, a mesh, a spline model, or a voxel model).

[0050] A point cloud can be represented in memory as a vector-based structure where each point has its own coordinates in the viewpoint's reference frame (e.g., three-dimensional coordinates XYZ, or solid angles and distances from / to the viewpoint (also called depth)) and one or more attributes (also called components). Examples of components are color components that can be expressed in various color spaces, such as RGB (red, green, and blue) or YUV (Y is the luminance component and UV are two chrominance components). A point cloud is a representation of a 3D scene including objects. The 3D scene can be seen from a given viewpoint or range of viewpoints. Point clouds can be obtained in a variety of ways, such as: by the capture of real objects taken by a camera device optionally compensated by a depth active sensing device; by the capture of virtual / synthetic objects taken by a virtual camera device in a modeling tool; or by a mixture of both real and virtual objects.

[0051] Figure 2 A non-limiting example of encoding, transmission and decoding of data representing a 3D scene sequence is shown. The encoding format may for example be compatible with 3DoF, 3DoF+ and 6DoF decoding simultaneously.

[0052] A 3D scene sequence is obtained 20. Since the picture sequence is a 2D video, the 3D scene sequence is a 3D (also called volumetric) video. The 3D scene sequence may be provided to a volumetric video rendering device for 3DoF, 3DoF+ or 6DoF rendering and display.

[0053] The 3D scene sequence 20 is provided to an encoder 21. The encoder 21 takes as input a 3D scene or a sequence of 3D scenes and provides a bitstream representing the input. The bitstream may be stored in a memory 22 and / or on an electronic data medium and may be transmitted over a network 22. The bitstream representing the 3D scene sequence may be read from the memory 22 and / or received from the network 22 by a decoder 23. The decoder 23 is input with the bitstream and provides the 3D scene sequence, for example in a point cloud format.

[0054] The encoder 21 may include several circuits that implement several steps. In a first step, the encoder 21 projects each 3D scene onto at least one 2D picture. 3D projection is any method of mapping three-dimensional points onto a two-dimensional plane. Since most current methods for displaying graphic data are based on planar (pixel information from several bit planes) two-dimensional media, this type of projection is widely used, especially in computer graphics, engineering, and drawing. The projection circuit 211 provides at least one two-dimensional frame 2111 for the 3D scene of the sequence 20. The frame 2111 includes color information and depth information representing the 3D scene projected onto the frame 2111. In a variant, the color information and the depth information are encoded in two separate frames 2111 and 2112.

[0055] The metadata 212 is used and updated by the projection circuit 211. The metadata 212 includes information about the projection operation (e.g., projection parameters) and about the way in which color and depth information is organized within the frames 2111 and 2112, such as information about the Figures 5 to 7 described.

[0056] The video encoding circuit 213 encodes the sequence of frames 2111 and 2112 into a video. The pictures of the 3D scenes 2111 and 2112 (or a sequence of pictures of the 3D scenes) are encoded in a stream by the video encoder 213. The video data and metadata 212 are encapsulated in a data stream by the data encapsulation circuit 214.

[0057] The encoder 213 conforms to, for example, an encoder such as a HEVC or VVC encoder.

[0058] The data stream is stored in a memory accessible by a decoder 23 via, for example, a network 22. The decoder 23 comprises different circuits implementing the different steps of decoding. The decoder 23 takes as input the data stream generated by the encoder 21 and provides a 3D scene sequence 24 for rendering and display by a volumetric video display device such as a head mounted device (HMD). The decoder 23 obtains the stream from a source 22. For example, the source 22 belongs to the set of: a local memory such as a video memory or a RAM (or random access memory), a flash memory, a ROM (or read-only memory), a hard disk; a storage interface such as an interface with a mass storage device, a RAM, a flash memory, a ROM, an optical disk or a magnetic support; a communication interface such as a wired interface (e.g. a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or interface); and a user interface, such as a graphical user interface that enables a user to input data.

[0059] The decoder 23 comprises a circuit 234 for extracting data encoded in the data stream. The circuit 234 takes the data stream as input and provides metadata 232 corresponding to the metadata 212 and the two-dimensional video encoded in the stream. The video is decoded by a video decoder 233, which provides a sequence of frames. The decoded frames include color and depth information. In a variant, the video decoder 233 provides two sequences of frames, one including color information and the other including depth information. The circuit 231 uses the metadata 232 to unproject the color and depth information from the decoded frames to provide a 3D scene sequence 24. The 3D scene sequence 24 corresponds to the 3D scene sequence 20, where there may be a loss of precision associated with encoding as 2D video and associated with video compression.

[0060] Figure 3 An example architecture of an apparatus 30 is shown, which may be configured to implement a method according to any of the embodiments described below. Figure 2 The encoder 21 and / or decoder 23 of can implement this architecture. Alternatively, each circuit of the encoder 21 and / or decoder 23 can be based on Figure 3 The architecture includes devices connected together, for example, via its bus 31 and / or via an I / O interface 36 .

[0061] The device 30 includes the following elements connected together by a data and address bus 31: a microprocessor 32 (or CPU), which is, for example, a DSP (or digital signal processor); a ROM (or read-only memory) 33; a RAM (or random access memory) 34; a storage interface 35; an I / O interface 36 for receiving data to be transmitted from an application; and a power source, for example, a battery.

[0062] According to an example, the power supply is external to the device. In each of the memories mentioned, the word "register" used in the specification may correspond to an area of ​​small capacity (a few bits) or a very large area (e.g., an entire program or a large amount of received or decoded data). ROM 33 includes at least programs and parameters. ROM 33 can store algorithms and instructions to perform the technology according to the principles of the present invention. When turned on, CPU 32 uploads the program to RAM and executes the corresponding instructions.

[0063] RAM 34 includes in registers the program executed by CPU 32 and uploaded after device 30 is turned on, includes in registers input data, includes in registers intermediate data at different states of the method, and includes in registers other variables used to execute the method.

[0064] The implementations described herein may be implemented in, for example, a method or process, an apparatus, a computer program product, a data stream, or a signal. Even if discussed only in the context of a single implementation (e.g., discussed only as a method or apparatus), the implementations of the features discussed may also be implemented in other forms (e.g., programs). The apparatus may be implemented, for example, with appropriate hardware, software, and firmware. The method may be implemented in, for example, an apparatus, for example, such as a processor, which is generally referred to as a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, for example, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0065] According to the example, device 30 is configured to implement a method according to any one of the embodiments described below and belongs to a set including at least one of the following: a mobile device; a communication device; a gaming device; a tablet computer (or tablet computer); a laptop computer; a still picture camera; a video camera; an encoding chip; or a server (e.g., a broadcast server, a video on demand server, or a web server).

[0066] Figure 4 An example of an embodiment of the syntax of a stream when data is transmitted over a packet-based transport protocol is shown. Figure 4 An example structure 4 of a volumetric video stream is shown. The structure consists in a container that organizes the stream into separate elements of syntax. The structure may include a header portion 41, which is a set of data common to each syntax element of the stream. For example, the header portion includes some of the metadata about the syntax elements, which describes the properties and role of each of the syntax elements. The header portion may also include Figure 2 The structure includes a portion of metadata 212, for example, coordinates of a central viewpoint for projecting points of a 3D scene onto frames 2111 and 2112. The structure includes a payload, the payload including a syntax element 42 and at least one syntax element 43. The syntax element 42 includes data representing a color and depth frame. The image may have been compressed according to a video compression method.

[0067] Syntax element 43 is part of the payload of the data stream and may include metadata about how to encode the frames of syntax element 42, for example, parameters for projecting and packing points of a 3D scene onto frames. Such metadata may be associated with each frame of the video or with a group of frames (also called a group of pictures (GoP) in video compression standards).

[0068] Figure 5The patch-atlas method is illustrated with four projection centers as an example. The 3D scene 50 includes characters. For example, the projection center 51 is a perspective camera, and the camera 53 is an orthographic camera. The camera can also be an omnidirectional camera with spherical mapping (e.g., equirectangular mapping) or cube mapping, for example. According to the projection operation described in the projection data of the metadata, the 3D points of the 3D scene are projected onto a 2D plane associated with the virtual camera located at the projection center. Figure 5 In the example of , the projection 51 of the point captured by the camera is mapped onto the patch 52 according to the perspective mapping, and the projection of the point captured by the camera 53 is mapped onto the patch 54 according to the orthogonal projection.

[0069] The aggregation of the projected pixels produces a plurality of 2D patches, which are packed in a rectangular atlas 55. The organization of the patches within the atlas defines the atlas layout. In an embodiment, the two atlases have the same layout: one for texture (i.e., color) information and one for depth information. Two patches captured by the same camera or by two different cameras may include information representing the same portion of a 3D scene, e.g., patches 54 and 56.

[0070] The packing operation produces patch data for each generated patch. The patch data includes a reference to the projection data (e.g., an index into a projection data table or a pointer to the projection data (i.e., an address in memory or in a data stream)) and information describing the location and size of the patch within the atlas (e.g., upper left corner coordinates, size, and width in pixels). The patch data item is added to the metadata to be packaged in the data stream in association with the compressed data of one or two atlases.

[0071] Figure 6 An example of an atlas 60 including texture information (eg, RGB data or YUV data) of points of a 3D scene according to a non-limiting embodiment of the principles of the present invention is shown. Figure 5 As explained, an atlas is an image packing patch, which is a picture obtained by projecting a portion of the points of a 3D scene.

[0072] exist Figure 6 In the example of , the atlas 60 comprises a first part 61 and one or more second parts 62, the first part comprising texture information of points of the 3D scene visible from the viewpoint. The texture information of the first part 61 can be obtained, for example, according to equirectangular projection mapping, which is an example of spherical projection mapping. Figure 6In the example of , the second part 62 is arranged at the left and right boundaries of the first part 61, but the second part can be arranged differently. The second part 62 includes texture information of a portion of the 3D scene that is complementary to the portion visible from the viewpoint. The second part can be obtained by removing points visible from the first viewpoint from the 3D scene (whose texture is stored in the first part) and by projecting the remaining points according to the same viewpoint. The latter process can be iteratively repeated to obtain the hidden part of the 3D scene each time. According to a variation, the second part can be obtained by removing points visible from a viewpoint (e.g., a central viewpoint) from the 3D scene (whose texture is stored in the first part) and by projecting the remaining points according to a viewpoint different from the first viewpoint (e.g., from one or more second viewpoints of a visual space centered on the central viewpoint (e.g., a viewing space of 3DoF rendering)).

[0073] The first part 61 can be seen as a first large texture patch (corresponding to a first part of the 3D scene), and the second part 62 includes smaller texture patches (corresponding to a second part of the 3D scene complementary to the first part). Such an atlas has the advantage of being compatible with both 3DoF rendering (when only the first part 61 is rendered) and 3DoF+ / 6DoF rendering.

[0074] Figure 7 A non-limiting embodiment according to the principles of the present invention is shown including Figure 6 An example of an atlas 70 of depth information of points of a 3D scene. The atlas 70 can be viewed as Figure 6 The depth image corresponding to the texture image 60.

[0075] The atlas 70 comprises a first part 71 comprising depth information of points of the 3D scene visible from the central viewpoint and one or more second parts 72. The atlas 70 may be obtained in the same manner as the atlas 60, but contains depth information associated with the points of the 3D scene instead of texture information.

[0076] For 3DoF rendering of a 3D scene, only one viewpoint is considered, usually the central viewpoint. The user can rotate his head with three degrees of freedom around the first viewpoint to see parts of the 3D scene, but the user cannot move the only viewpoint. The points of the scene to be encoded are those visible from the only viewpoint, and only texture information needs to be encoded / decoded for 3DoF rendering. For 3DoF rendering, points of the scene that are not visible from the only viewpoint do not need to be encoded because the user cannot access them.

[0077] Regarding 6DoF rendering, the user can move the viewpoint anywhere in the scene. In this case, every point of the scene (depth and texture) needs to be encoded in the bitstream, because each point is accessible to the user who can move his / her viewpoint. At the encoding stage, it is not possible to know a priori from which viewpoint the user will observe the 3D scene.

[0078] With respect to 3DoF+ rendering, the user can move the viewpoint within a limited space around the central viewpoint. This enables the experience of parallax. Data representing the portion of the scene visible from any point in the view space will be encoded into the stream, including data representing the 3D scene visible from the central viewpoint (i.e., the first parts 61 and 71). The size and shape of the view space can be determined and determined, for example, at the encoding step and encoded in the bitstream. The decoder can obtain this information from the bitstream, and the renderer limits the view space to the space determined by the obtained information. According to another example, the renderer determines the view space according to hardware constraints, such as those related to the ability of the sensor to detect the user's movement. In this case, if a point visible from a point within the view space of the renderer has not been encoded in the bitstream at the encoding stage, the point will not be rendered. According to yet another example, data (e.g., textures and / or geometry) representing each point of the 3D scene is encoded in the stream without considering the rendering view space. In order to optimize the size of the stream, only a subset of the points of the scene can be encoded, for example, a subset of the points that can be seen according to the rendering view space.

[0079] The MIV standard provides great flexibility in organizing patches into multiple atlases and / or multiple tiles in each atlas. In addition, there is a signaling mechanism for assigning each patch to a unique object ID (i.e., the so-called entity ID) and thus enabling filtering of selected objects for partial reconstruction at the decoder side. Figure 8 Depicted is a segmentation into 25 objects provided on one of the MIV test sequences (W. Cochran, B. Salahieh, J. Boyce, F. Yeung, “Object Maps for Technicolor Museum Sequence,” ISO / IEC JTC1 / SC29 / WG11 m52474, January 2020).

[0080] Therefore, patches belonging to a given object (identified by its ID) can be put into a separate atlas or separate tiles within an atlas. The geometry / occupancy / attribute patch-atlas components corresponding to each object can then be encoded independently in a separate video bitstream or video tile sub-bitstream (HEVC MCTS or VVC sub-picture) and streamed in separate video tracks.

[0081] Time (mis)alignment

[0082] The first version of MIV required that all metadata sub-bitstreams and video sub-bitstreams that make up the MIV bitstream be temporally aligned. However, the use cases and requirements for MIV.Edition-2 ("Use cases and requirements for MIV.Edition-2", ISO / IEC JTC 1 / SC 29 / WG 2N00157, January 2022) opened the door to object-based coding, where each object has a different temporal sampling (frame rate) and may not be aligned:

[0083] 7. Version 2 of the specification shall enable handling of heterogeneous object-specific parameters (e.g., temporal sampling, duration, atlas size, and non-Lambertian) at the MIV bitstream level.

[0084] 18. Version 2 of the specification shall enable combining multiple MIV bitstreams into a single MIV bitstream.

[0085] 19. Version 2 of the specification should maintain / enhance support for independent object-level encoding and decoding in the bitstream.

[0086] In this paper, it is considered that the stereoscopic 3D scene has been generated by combining multiple 3D sources captured by a camera array or computer-generated, and is represented and compressed by a MIV-compliant encoder on a per-object basis, i.e., each object has an independently decodable sub-bitstream, each sub-bitstream consisting of its own atlas metadata, camera parameters, and video sub-bitstream. Fig. 9 An example of such a 3D scene is depicted in , where 4 objects are shown.

[0087] Assume that the constraint of aligning all object sub-bitstreams in time is relaxed, since the different objects originate from different acquisition devices that are not synchronized and run at different frame rates. An example of the result in terms of (mis)alignment of bitstream "assembly units" can be given by Fig.10 Shown.

[0088] exist Fig.10 In the example shown, the portion of the 3D scene corresponding to each four objects can be reconstructed, but such scene portions are not synchronized and it is not easy to recover the complete 3D scene representation at a given common output time. Fig.10 , depicts the decomposition of an object bitstream into partitions associated with a given output time - so-called "combination units" - in order to reconstruct a volumetric frame at a given time instance. For a given output time t3 in the bitstream of object #3, the three nearest combination units of three other objects are highlighted (dark grey areas).

[0089] For time alignment, the timeline of the object with the highest frame rate can be selected as the reference timeline ( Fig.10 In the example above, the output time of the video image is t3, which is at 70fps in the video image, and the combined units in each object bitstream that are closest to the selected output time (here t3) can be collected. However, this will produce shaking artifacts in the reconstructed 3D scene, which are more annoying when the object or camera moves faster.

[0090] According to one aspect of the principles presented herein, a 3D scene reconstruction method is provided in which accurate temporal alignment of various object components of a 3D scene is provided.

[0091] In an embodiment, a scenario consisting of object sub-bitstreams that are time aligned but sampled at different frame rates may also be considered. The problem that arises is to perform accurate temporal upsampling of the objects from the lowest frame rate to the highest frame rate on the reconstruction side, since it is well known that simple frame repetition can produce shaking artifacts on moving objects.

[0092] One aspect of the present principles relates to the transmission of a 3D scene consisting of multiple objects captured by a volumetric video encoder (e.g., MIV encoder) as a multi-view plus depth (MVD) representation and transmitted by temporally non-aligned independently encoded sub-bitstreams. In accordance with the present principles, additional 3D motion information is transmitted with the sub-bitstreams of the objects, thereby enabling accurate 3D motion compensation when all object scene parts are temporally aligned at the reconstruction side.

[0093] In an embodiment, the 3D motion information may be transmitted in the form of an additional V3C "attribute" video stream.

[0094] Fig.11 A method 1100 for encoding a 3D scene according to an embodiment is provided. In an embodiment, the 3D scene includes a plurality of 3D objects encoded in different substreams. In a variant, each object is encoded as a sequence of volume frames. In this embodiment, the following steps are performed on the encoder side for each object or sequence of volume frames. In another embodiment, the entire 3D scene is encoded as a sequence of volume frames (without considering the object segmentation of the scene).

[0095] At 1110, a patch-atlas based representation is obtained (ie, generated) for the object or volume frame sequence under consideration, for example, as described above with respect to Figures 5 to 7 A patch-atlas based representation is generated for each frame of a volumetric frame sequence representing an object or a 3D scene.

[0096] At 1120, 3D motion information is obtained for at least one volume frame of the sequence. The 3D motion information represents the displacement of deprojected samples in 3D space based on the patch-atlas representation between two volume frames of the sequence. Therefore, the atlas samples of the volume frame are deprojected to their 3D positions in 3D space, and a 3D motion vector is estimated for each deprojected 3D sample. In a variant, a parameterized motion model for each patch can also be estimated based on the estimated 3D motion vector.

[0097] At 1130, the patch-atlas data and 3D motion information are encoded for each sequence, for example, in a MIV substream. The 3D motion information can be encoded as a 3D motion attribute video stream or 3D motion parameters for each patch. Thus, the MIV bitstream consists of the MIV object sub-bitstream with additional 3D motion information.

[0098] Fig.12 A method 1200 for decoding a 3D scene according to an embodiment is provided. For example, when a 3D scene includes multiple 3D objects encoded in different substreams, the following steps are performed on the decoder and reconstruction side for each object. In an embodiment, each object is represented as a sequence of volume frames. Therefore, the 3D scene is encoded with one or more volume frame sequences. In another embodiment, the entire 3D scene is encoded as a sequence of volume frames. At 1210, the volume frame sequence is decoded from the data stream, for example, using the MIV decoding process when the data stream conforms to the MIV standard to reconstruct a portion of the 3D scene point cloud. Therefore, in this embodiment, the patch-atlas based representation is decoded for each volume frame of the sequence.

[0099] At 1220, 3D motion information is obtained, wherein the 3D motion information represents the displacement of the 3D points of the reconstructed point cloud in the 3D space. In an embodiment, the 3D motion information is decoded from the substream. In another embodiment, the 3D motion information can be estimated from the reconstructed 3D scene.

[0100] At 1230, the 3D scene is reconstructed using the 3D motion information at the combined time frame. According to the principles of the present invention, when a 3D scene is encoded using multiple substreams that are respectively encoded on objects of the 3D scene, and when the substreams are not aligned in time, the principles of the present invention allow the reconstructed point cloud sequence to be aligned to a common timeline using 3D motion compensated interpolation techniques. This allows the reconstructed volume frames to be aligned at time frames where no data is encoded in the substreams. In other words, the combined time frame is different from the time frame encoded in the data stream.

[0101] According to the principles of the invention, and according to another embodiment of the volumetric scene (the entire scene is a single object) without considering object segmentation, the volumetric sequence is temporally resampled at a different frame rate by high-quality - 3D motion compensated - frame rate conversion. Thus, the volumetric sequence is temporally resampled at a frame rate different from the frame rate used when encoding.

[0102] 3D Motion Estimation

[0103] Fig.13 The 3D motion field of an object point cloud is shown in . If a point cloud sample is at position (X, Y, Z) at time t and at position (X+δX, Y+δY, Z+δZ) at time t+δt in the scene reference coordinate system, then its 3D motion vector is equal to (δX / t, δY / δt, δZ / δt) expressed in scene units / time units. State-of-the-art 3D techniques for estimating the 3D motion of point clouds can be used here. The size of the search neighborhood, the similarity measure, the search strategy may differ, but all algorithms share the same paradigm: for each point in the source point cloud, find the best (most similar) matching point in the target neighboring point cloud. This is shown in . Fig.13 , which depicts, from left to right: a) a given 3D sample of the point cloud at time t, b) the search for the best matching point within the bounding box at time t+δt, and c) the 3D motion field of the entire object, one forearm and one foot of which are moving.

[0104] In an embodiment, the 3D motion field at time t is estimated between the current point cloud frame and the next point cloud frame. In another embodiment, the 3D motion field at time t is estimated between the current point cloud frame and the previous point cloud frame. In yet another embodiment, the 3D motion field at time t is the average of the two previous estimates. In all embodiments, the 3D motion values ​​are normalized in scene units (e.g., meters) / time units (e.g., seconds) that are independent of the spatial and temporal resolution of the current object.

[0105] Fig.14 An example of a method 1400 for reconstructing a 3D scene with multiple 3D objects encoded at different frame rates in separate substreams according to an embodiment is shown. At 1410, a common timeline is determined for the 3D scene. For example, an object encoded at the highest frame rate is selected, or a desired frame rate is selected based on the application. At 1420, each object sub-bitstream is decoded. The sub-bitstream includes the encoded data of the object and the 3D motion information of the object.

[0106] For each time sample T of the common timeline, and for each decoded object sub-bitstream, the following steps are performed:

[0107] - At 1430, the closest decoded frame (time T±δt) is determined.

[0108] - at 1440, the determined decoded frame (at time T±δt) is then used to reconstruct a corresponding point cloud of the 3D scene portion,

[0109] - At 1450, the 3D point of the corresponding reconstructed point cloud is shifted to its position at time T along its 3D motion direction using the 3D motion information decoded from the sub-bitstream.

[0110] The time-aligned point cloud sequences are combined into a single 3D scene point cloud at 1460. The combined 3D scene can then be used for any further application processing, such as virtual viewport rendering for 6DoF navigation.

[0111] 3D motion properties

[0112] In an embodiment, the 3D motion information is encoded as a 3D motion attribute video substream. To this end, new attributes are specified in V3C to carry the 3D motion information. The following Table 1 in the V3C specification is modified, for example, as follows (the modification is shown as Underlined ):

[0113] Table 1-V3C attribute types

[0114] ai_attribute_type_id[j][I] Identifier Property Type 0 ATTR_TEXTURE Texture 1 ATTR_MATERIAL_ID Material ID 2 ATTR_TRANSPARENCY transparency 3 ATTR_REFLECTANCE Reflectivity 4 ATTR_NORMAL Norm <![CDATA[ 5 ]]> <![CDATA[ ATTR_3D_MOTION ]]> <![CDATA[ 3D Motion ]]> 6..14 ATTR_RESERVED reserve 15 ATTR_UNSPECIFIED not specified

[0115] ATTR_3D_MOTION indicates an attribute containing three-dimensional vector information associated with each point in a volumetric frame. The vector specifies the 3D motion of the point expressed in scene units / time units. Attribute frames with this attribute type shall have ai_attribute_dimension_minus1 equal to 2.

[0116] MIV bitstream with 3D motion attributes

[0117] Each object that makes up the scene is encoded independently in a MIV sub-bitstream consisting of depth, color, and 3D motion video bitstreams associated with the atlas metadata, as shown in Fig.15 shown.

[0118] 3D Motion Modeling

[0119] In an embodiment, the 3D motion information is encoded as per-patch motion model parameters of an atlas-patch based representation.

[0120] If a 3D patch can be approximated by rigid 3D surface elements, a 6-dimensional motion model consisting of a 3D translation and a 3D rotation (with the origin being the patch center) can be used to describe the 3D motion of all samples of the patch. The 3 coefficients of translation and the 3 coefficients of rotation are then estimated from the 3D displacement vectors of the points belonging to the patch.

[0121] If the 3D motion attributes of the required bitrate video bitstream are replaced by 3D motion parameters attached to the patches, then this motion model should be valid for all patches and patch lifecycles (typically the 32-frame patch duration corresponding to the intra-video period implemented in the reference MIV encoder "TestModel 11for MPEG Immersive Video", ISO / IEC JTC 1 / SC 29 / WG 4N00142, October 2021, may be too long). However, if the rigid motion model assumption is valid, the 3D motion attributes can be replaced by additional syntax elements of the MIV extension of the atlas sequence parameter set and patch data unit (as shown in the following table as Underlined ).

[0122]

[0123]

[0124] asme_3d_motion_present_flag equal to 1 specifies that the 3D motion parameters syntax element is present in the pdu_miv_extension() syntax structure. asme_3d_motion_present_flag equal to 0 specifies that the 3D motion parameters syntax element is not present in the pdu_miv_extension() syntax structure. When not present, the value of asme_3d_motion_present_flag is inferred to be equal to 0.

[0125]

[0126] pdu_trans_x specifies the x component of the translation part of the patch's 3D motion.

[0127] pdu_quat_x specifies the x component of the rotational portion of the patch's 3D motion using a quaternion representation.

[0128] And the same goes for the y and z components.

[0129] 3D Motion Compensation

[0130] Fig.16On the left (a) point-based motion compensation is depicted, where each point of a 3D object has a motion vector encoded in a 3D motion attribute, and each point is displaced along the direction of its 3D motion attribute. Fig.16 The right side (b) depicts patch-based motion compensation, where all points of the same patch follow the same motion of the patch (the 3D rotation and translation of the patch encoded in the patch's metadata).

[0131] Encode-Decode-Align Workflow

[0132] For the 3D motion attribute embodiment (point-based), Fig.17 and Fig.18 The encoding and decoding-reconstruction workflows are shown in , respectively. For the 3D motion model embodiment, there is no 3D motion properties video bitstream, but rather patch 3D motion parameters carried in the atlas data bitstream.

[0133] exist Fig.17 In the embodiment of the present invention, each object in the scene (MVD object #1, ..., #N) is provided to a MIV encoder, which performs 3D motion estimation on the 3D points of the object. In a variant, after encoding-decoding the patch-atlas based representation, motion estimation is performed on the deprojected samples of the patch-atlas based representation. The attributes of each object (atlas metadata, depth component, color attributes, 3D motion attributes) are provided to a multiplexer, which outputs a MIV bitstream including a substream encoding a sequence of volumetric frames for each object.

[0134] exist Fig.18 In the process, the MIV bitstream is transmitted to a decoder, where each MIV substream of an object is provided to a MIV decoder. The MIV decoder outputs atlas metadata, depth components, color attributes, and 3D motion attributes of the object. These data are provided to a module that performs 3D deprojection (de-projection) on the patch-atlas based representation and outputs a reconstructed point cloud of the object.

[0135] The reconstructed point clouds of the objects may not be aligned on the same timeline or may not be at the same frame rate. Therefore, the temporal alignment module takes the reconstructed point clouds of the objects and the 3D motion information of each object as input and performs temporal alignment on the objects in the 3D scene. The temporal alignment module performs 3D motion compensated interpolation on each object using the 3D motion information so that a volume frame at the same combined time frame is obtained for each object. The temporally aligned point clouds are then output.

[0136] After reconstructing the time-aligned point cloud sequence, a virtual viewport can be rendered for 6DoF navigation, such as Fig.18As shown. It is also possible to resynthesize the complete original MVD sequence and sample it at the new output frame rate. In the latter case, a 3D motion compensated volumetric frame rate conversion is performed.

[0137] If the original volume sequence is temporally upsampled and then played at normal speed, 3D motion compensated slow motion is performed.

[0138] According to the principles of the present invention, a 3D scene with all object parts synchronized in time can be reconstructed without judder artifacts from a compressed representation composed of object sub-bitstreams that are not aligned in time or composed of sub-bitstreams that are aligned in time but sampled at different frame rates. High-quality volumetric video frame rate conversion and high-quality volumetric video slow motion are provided.

[0139] The implementations described herein may be implemented in, for example, a method or process, a device, a computer program product, a data stream, or a signal. Even if discussed only in the context of a single implementation (e.g., discussed only as a method or device), the implementation of the features discussed may also be implemented in other forms (e.g., a program). The device may be implemented, for example, with appropriate hardware, software, and firmware. The method may be implemented in, for example, a device, for example, such as a processor, which is generally referred to as a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, for example, such as a smart phone, a tablet computer, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0140] The implementation of the various processes and features described herein can be embodied in a variety of different devices or applications, particularly, for example, devices or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and / or depth information. Examples of such devices include encoders, decoders, post-processors for processing outputs from decoders, pre-processors for providing inputs to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, mobile phones, PDAs, and other communication devices. It should be clear that the device can be mobile and even installed in a mobile vehicle.

[0141] Additionally, the method may be implemented by a processor executing instructions, and such instructions (and / or data values ​​produced by the implementation) may be stored on a processor-readable medium (e.g., such as an integrated circuit, a software carrier), or other storage device (e.g., such as a hard disk, a compact disk ("CD"), an optical disk (e.g., such as a DVD, commonly referred to as a digital versatile disk or digital video disk), a random access memory ("RAM"), or a read-only memory ("ROM")). The instructions may form an application tangibly embodied on a processor-readable medium. The instructions may be in, for example, hardware, firmware, software, or a combination. The instructions may be found in, for example, an operating system, a separate application, or a combination of both. Thus, a processor may be characterized as both, for example, a device configured to perform a process and a device including a processor-readable medium (such as a storage device) having instructions for performing the process. Furthermore, in addition to or in lieu of instructions, the processor-readable medium may store data values ​​produced by the implementation.

[0142] It will be apparent to those skilled in the art that implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the syntax of the described embodiment, or to carry as data the actual syntax values ​​written by the described embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0143] Many implementations have been described. However, it should be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Additionally, one of ordinary skill in the art will appreciate that other structures and processes may replace those disclosed, and that the resulting implementations will perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results as the disclosed implementations. Therefore, the present application contemplates these and other implementations.

Claims

1. A method comprising reconstructing at least one sequence of volumetric frames from a data stream, comprising: decoding the at least one sequence of volumetric frames from the data stream, obtaining 3D motion information representing displacement of points of a volume frame of the at least one volume frame sequence in a 3D space; Points of at least one volumetric frame of the sequence are shifted to a combined temporal frame using the 3D motion information.

2. An apparatus comprising one or more processors configured to reconstruct at least one sequence of volumetric frames from a data stream, comprising: decoding the at least one sequence of volumetric frames from the data stream, obtaining 3D motion information representing displacement of points of a volume frame of the at least one volume frame sequence in a 3D space; Points of at least one volumetric frame of the sequence are shifted to a combined temporal frame using the 3D motion information.

3. The method according to claim 1 or the apparatus according to claim 2, wherein: Shifting the points of the at least one volumetric frame is part of resampling the at least one sequence of volumetric frames at a frame rate different from the frame rate used in encoding.

4. The method according to claim 1 or 3 or the apparatus according to claim 2 or 3, wherein: The combined time frame is different from the time of the volumetric frame encoded in the data stream.

5. A method according to any one of claims 1 or 3 to 4 or an apparatus according to any one of claims 2 to 4, wherein: The at least one sequence of volumetric frames represents a 3D scene comprising at least two objects encoded in separate sub-streams of the data stream.

6. A method according to any one of claims 1 or 3 to 5 or an apparatus according to any one of claims 2 to 5, wherein: Shifting points of the at least one volumetric frame includes motion compensating the decoded volumetric frame using the 3D motion information.

7. The method or apparatus according to claim 5, wherein: Each of the at least two objects of the 3D scene is encoded as a sequence of volumetric frames.

8. The method according to any one of claims 1, 3 to 7 or the apparatus according to any one of claims 2 to 7, wherein: Decoding the at least one sequence of volumetric frames includes decoding a patch-atlas based representation of the volumetric frames.

9. The method or apparatus according to claim 5, wherein: Reconstructing the at least one volume frame sequence further includes: for each of the at least two objects, determining a time frame of a corresponding substream closest to the combined time frame, and decoding and reconstructing the object of the at least two objects at the determined time frame.

10. The method according to any one of claims 1, 3 to 9 or the apparatus according to any one of claims 2 to 9, wherein: Obtaining the 3D motion information comprises: decoding a 3D motion attribute video substream for the at least one volume frame sequence.

11. The method or apparatus according to claim 10, wherein: The 3D motion properties video substream includes 3D motion vector information associated with at least one point of the volumetric frame.

12. The method according to any one of claims 1, 3 to 9 or the apparatus according to any one of claims 2 to 9, wherein: Obtaining the 3D motion information includes: determining a 3D motion vector for at least one point of the volume frame.

13. The method according to any one of claims 1, 3 to 9 or the apparatus according to any one of claims 2 to 9, wherein: The 3D motion information is obtained by decoding metadata associated with at least one patch of the patch-atlas based representation of the at least one volumetric frame sequence.

14. The method or apparatus according to claim 13, wherein: The metadata includes: a 3D motion determined based on deprojected samples for the at least one patch, parameters of a motion model determined for the at least one patch.

15. The method according to any one of claims 1, 3 to 14 or the apparatus according to any one of claims 2 to 14, wherein: At least one syntax element indicating the presence of 3D motion information is encoded in the data stream.

16. A method comprising encoding at least one sequence of volumetric frames representing a 3D scene, comprising: for at least one volume frame of the sequence, obtaining a patch-atlas based representation, obtaining 3D motion information representing a displacement in 3D space of a deprojected sample of the patch-atlas based representation between two volumetric frames of the sequence, The patch-atlas based representation and the 3D motion information are encoded in a data stream.

17. An apparatus comprising one or more processors configured to encode at least one volumetric frame sequence, wherein: The one or more processors are further configured to: for at least one volume frame of the sequence, obtaining a patch-atlas based representation, obtaining 3D motion information representing a displacement in 3D space of a deprojected sample of the patch-atlas based representation between two volumetric frames of the sequence, The patch-atlas based representation and the 3D motion information are encoded in a data stream.

18. The method of claim 16 or the apparatus of claim 17, wherein: The 3D motion information is encoded as a 3D motion properties video substream.

19. The method or apparatus of claim 18, wherein: The 3D motion properties video substream comprises, for at least one volumetric frame of the sequence: 3D motion vector information associated with at least one point in the at least one volumetric frame.

20. A method according to any one of claims 16, 18 to 19 or an apparatus according to any one of claims 17 to 19, wherein: The 3D motion information is obtained by determining a 3D motion vector for at least one deprojected sample.

21. The method of claim 16 or the apparatus of claim 17, wherein: The 3D motion information is encoded as metadata associated with at least one patch of the patch-atlas based representation.

22. The method or apparatus of claim 21, wherein: The metadata includes: a 3D motion determined based on deprojected samples for the at least one patch, parameters of a motion model determined for the at least one patch.

23. A method according to any one of claims 16 or 18 to 22 or an apparatus according to any one of claims 17 to 22, wherein: At least one syntax element indicating the presence of 3D motion information is encoded in the data stream.

24. A method according to any one of claims 16 or 18 to 23 or an apparatus according to any one of claims 17 to 23, wherein At least one 3D object is represented in the at least one sequence of volumetric frames, and wherein properties associated with at least one point of one volumetric frame are encoded in a separate substream for each of the at least one 3D object.

25. A bitstream comprising encoded data representing at least one sequence of volumetric frames representing a 3D scene, the encoded data comprising: Properties of a patch-atlas based representation are obtained for the at least one sequence of volumetric frames and 3D motion information representing a displacement in 3D space of deprojected samples of the patch-atlas based representation between two volumetric frames in the sequence.

26. The bitstream of claim 25, further comprising at least one syntax element indicating the presence of 3D motion information, the at least one syntax element being encoded in the data stream.

27. A computer readable medium comprising a bit stream according to claim 25 or 26.

28. A computer-readable storage medium storing instructions for causing one or more processors to perform the method of any one of claims 1, 3 to 16, or 18 to 24.

29. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1, 3 to 16 or 18 to 24.

30. An apparatus comprising: The apparatus according to any one of claims 2 to 15; as well as At least one of: (i) an antenna configured to receive a signal, the antenna comprising data representing at least one volumetric frame sequence, (ii) a frequency band limiter configured to limit the received signal to a frequency band comprising the data representing the at least one volumetric frame sequence, or (iii) a display configured to display the at least one volumetric frame sequence.

31. An apparatus comprising: an access unit configured to access data comprising a signal storing a bit stream according to claim 25 or 26, A transmitter is configured to transmit the accessed data.

32. A method comprising: accessing data comprising a signal storing a bit stream according to claim 25 or 26; and transfer of accessed data.