Methods and apparatus for encoding, decoding, and rendering 6DOF content from 3DOF+ components.

Abstract

This invention discloses a method where volumetric content is encoded into a cluster by an encoder and transmitted to a decoder that retrieves the volumetric content. Clusters shared across different viewpoints are obtained and collectively managed. These clusters are then projected onto a 2D image and encoded into independent video streams. This achieves reduction in visual artifacts, as well as reduction in storage and streaming data.

Description

1. TECHNICAL FIELD

[0001] The present principles generally relate to the domain of three-dimensional (3D) scene and volumetric video content. The present document is also understood in the context of encoding, formatting and decoding data representative of textures and geometry of a 3D scene, to render volumetric content on an end-user device such as a mobile device or a Head-Mounted Display (HMD). 2. BACKGROUND

[0002] This section is intended to introduce the reader to various aspects of art that can be related to various aspects of the present principles that are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present principles. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] Recently, there has been a growth in available large field of view content (up to 360°). A user watching content on an immersive display device (such as a head-mounted display, smart glasses, PC screen, tablet, smartphone, etc.) can not be able to see the whole of such content. This means that at a given moment, the user can only watch a part of the content. However, the user can typically navigate within the content through various means such as head movement, mouse movement, touch screen, voice, and the like. It is generally desirable to encode and decode such content.

[0004] Immersive video (also called 360° planar video) allows a user to watch everything around him by rotating his head around a static viewing point. The rotation only allows a 3 degrees of freedom (3DoF) experience. Even if 3DoF video is sufficient to meet the requirements of a first omnidirectional video experience (e.g. using a head-mounted display (HMD device)), 3DoF video can quickly become frustrating for a viewer expecting more freedom (e.g. by experiencing parallax). Moreover, 3DoF can also cause dizziness because a user never only rotates his head but also translates his head in three directions, which are not reproduced in a 3DoF video experience.

[0005] Among others, the large field of view content can be a three-dimensional computer graphics image scene (3D CGI scene), a point cloud or an immersive video. Many terms can be used to design such immersive video: for example, Virtual Reality (VR), 360, panoramic, 4π steradians, immersive, omnidirectional or large field of view.

[0006] Volumetric video (also known as 6DoF video) is an alternative to 3DoF video. When watching 6DoF video, in addition to rotation, users can pan their head and even their body within the content being viewed, experiencing parallax and even volume. This type of video significantly increases immersion and perception of scene depth, and prevents motion sickness by providing consistent visual feedback during head panning. The content is created using dedicated sensors, allowing for the simultaneous recording of color and depth of the scene of interest. Even though technical challenges remain, using color camera equipment incorporating photogrammetry is another way to perform this recording.

[0007] While 3DoF video comprises a sequence of images derived from the demapping of textured images (e.g., spherical images encoded according to latitude / longitude projection maps or isometric projection maps), 6DoF video frames embed information from multiple viewing points. They can be viewed as a temporal series of point clouds generated by 3D capture. Two types of volumetric video can be considered depending on the viewing conditions. The first (i.e., full 6DoF) allows for completely free navigation within the video content, while the second (referred to as 3DoF+) restricts the user's viewing space to a finite volume called the viewing bounding box, thus allowing for limited head translation and parallax experience. This second case represents a valuable trade-off between free navigation and the passive viewing conditions of a seated audience.

[0008] However, rendering artifacts may occur during 3DoF+ volumetric rendering, such as areas with missing information. Reducing rendering artifacts is necessary.

[0009] In the 3DoF+ rendering experience, users can move the viewpoint within a viewing bounding box. This is achieved by encoding the 3D scene from multiple viewpoints within the viewing bounding box. For each viewpoint within the viewing bounding box, points visible in 360° from these viewpoints are projected to obtain 2D projections of the 3D scene. These 2D projections are encoded using well-known video coding techniques such as HEVC (High Efficiency Video Coding) and transmitted over the network.

[0010] The quality of the user experience depends on the number of viewpoints considered when encoding a 3D scene for a given viewing bounding box. Increasing the number of viewpoints allows for a reduction in artifacts.

[0011] However, increasing the number of viewpoints increases the volume of the data payload corresponding to the volumetric video, which affects storage and transmission.

[0012] Additionally, when making large movements from the view bounding box to an adjacent view bounding box, data associated with that adjacent view bounding box needs to be retrieved for rendering. If the data load is high, there is a risk that the latency in retrieving and rendering content may be perceptible to the user.

[0013] The goal is to minimize the data load corresponding to 3DoF+ volumetric video while providing users with a seamless navigation experience. 3. Summary of the Invention

[0014] The following is a simplified overview of the principles of the invention to provide a basic understanding of some aspects of these principles. This summary is not a broad overview of the principles of the invention and is not intended to identify key or essential elements of the invention. The following summary presents only some aspects of the principles of the invention in a simplified form as a preface to the more detailed description that follows.

[0015] According to one or more embodiments, a method and apparatus for encoding volumetric content involving a 3D scene are provided. The method includes:

[0016] The points in the 3D scene are clustered into multiple clusters based on at least one clustering criterion;

[0017] The cluster is projected according to the projection parameters to obtain a set of 2D images;

[0018] Encode the set of 2D images and the projection parameters in a set of data streams.

[0019] According to one implementation, each of the 2D images being encoded is a different data stream. In another implementation, a viewing frame is defined in the 3D scene, and 2D images obtained by projection from a cluster visible to two viewing points within the viewing frame are encoded in the same data stream. In yet another implementation, two viewing frames are defined in the 3D scene, and 2D images obtained by projection from a cluster visible to two viewing points located within each of the two viewing frames are encoded in the same data stream.

[0020] This disclosure also relates to a method and apparatus for decoding a 3D scene. The method includes:

[0021] At least one 2D image is obtained from a set of data streams, the 2D image representing a projection of at least one cluster of points in the 3D scene according to projection parameters, the points in the cluster satisfying at least one clustering criterion;

[0022] The pixels of the at least 2D image are projected based on the projection parameters in the 3D scene and the viewing point.

[0023] In the implementation scheme, the method further includes

[0024] Obtain metadata, which includes:

[0025] The list of viewing frames defined within the 3D scene; and

[0026] A description of the data stream of an encoded 2D image of a viewing frame, which represents a cluster of 3D points visible from the viewing point of the viewing frame;

[0027] Decode a 2D image from a data stream that includes a cluster of 3D points visible from the viewing point.

[0028] This disclosure also relates to a medium in which instructions are stored, the instructions being used to cause at least one processor to perform at least the steps of the encoding method and / or decoding method and / or rendering method and / or receiving method as described above. 4. Description of the attached drawings

[0029] This disclosure will be better understood, and further specific features and advantages will emerge after reading the following description and referring to the accompanying drawings, in which:

[0030] - Figure 1 A three-dimensional (3D) model of an object according to a non-limiting embodiment of the principles of the present invention and points of a point cloud corresponding to the 3D model are shown;

[0031] - Figure 2 Examples of encoding devices, transmission media, and decoding devices for encoding, transmitting, and decoding data representing a sequence of 3D scenes according to a non-limiting embodiment of the principles of the present invention are shown;

[0032] - Figure 3 The illustration shows a non-limiting embodiment of the invention that can be configured to achieve the following: Figure 14 and Figure 15 An exemplary architecture of the encoding and / or decoding device for the described encoding and / or decoding method;

[0033] - Figure 4 Examples of embodiments of the syntax of a stream when transmitting data via a packet-based transport protocol are shown, according to a non-limiting embodiment of the principles of the present invention;

[0034] - Figure 5 A 3D scene including several objects is shown;

[0035] - Figure 6 This illustrates the concept of three DoF+ viewing bounding boxes in the three-dimensional space that generates the 3D scene, in relation to 3DoF+ rendering.

[0036] - Figure 7 This demonstrates the parallax experience allowed by volumetric rendering;

[0037] - Figure 8 This demonstrates the parallax experience and the demasking effect;

[0038] - Figure 9A method for structuring volume information according to a non-limiting embodiment of the principles of the present invention is shown;

[0039] - Figure 10 An example of a method for clustering a 3D scene into multiple point clusters according to a non-limiting embodiment of the principles of the present invention is shown;

[0040] - Figure 11 The 2D parameterization of a 3D scene according to a non-constraint embodiment based on the principles of the present invention is illustrated.

[0041] - Figure 12 and Figure 13 An example top view of a 3D scene with clusters, illustrating a non-limiting embodiment of the principles of the present invention.

[0042] - Figure 14 A method for encoding volumetric content involving a 3D scene is shown according to a non-constraint embodiment of the principles of the present invention;

[0043] - Figure 15 A method for decoding volumetric content involving a 3D scene is shown according to a non-constraint embodiment of the principles of the present invention;

[0044] - Figure 16 A method for rendering volumetric content involving a 3D scene is shown according to a non-constraint embodiment of the principles of the present invention;

[0045] - Figure 17 A method for receiving volumetric content relating to a 3D scene is shown according to a non-constraint embodiment of the principles of the present invention. 5. Detailed Implementation

[0046] The principles of the invention will be described more fully below with reference to the accompanying drawings, in which examples of the principles of the invention are shown. However, the principles of the invention may be embodied in many alternative forms and should not be construed as limited to the examples set forth herein. Therefore, while the principles of the invention are susceptible to various modifications and alternatives, specific examples are shown by way of example in the drawings and will be described in detail herein. However, it should be understood that there is no intention to limit the principles of the invention to the specific forms disclosed, but rather, this disclosure is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the principles of the invention as defined by the claims.

[0047] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit the principles of the invention. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the stated feature, integer, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as “responding” or “connected” to another element, it may directly respond to or be connected to the other element, or there may be intermediate elements present. Conversely, when an element is referred to as “directly responding” or “directly connected” to another element, there are no intermediate elements present. As used herein, the term “and / or” includes any and all combinations of one or more of the listed related items and may be abbreviated to “ / ”.

[0048] It should be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the teachings of the principles of the invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0049] Although some illustrations include arrows along the communication path to show the main communication direction, it should be understood that communication can occur in the opposite direction to the arrows depicted.

[0050] Examples of block diagrams and operation flowcharts are described where each box represents a circuit element, module, or code section, the code section including one or more executable instructions for implementing a specified logical function. It should also be noted that in other specific implementations, the functions marked in the boxes may not appear in the order they are marked. For example, two boxes shown consecutively may actually be executed substantially simultaneously, or these boxes may sometimes be executed in reverse order depending on the functions involved.

[0051] The references to "according to an example" or "in an example" in this document mean that a particular feature, structure, or characteristic described in connection with the example may be included in at least one specific embodiment of the principles of the invention. The appearance of the phrases "according to an example" or "in an example" in various places in the specification does not necessarily refer to the same example in all instances, nor is it necessarily a separate or alternative example that is mutually exclusive with other examples.

[0052] The reference numerals appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. Although not explicitly described, these examples and variations may be employed in any combination or sub-combination.

[0053] The principles of this invention will be described with reference to specific embodiments of methods for encoding volumetric content involving 3D scenes in a stream, methods for decoding such volumetric content from a stream, and methods for volumetric rendering of volumetric content decoded according to the aforementioned decoding methods.

[0054] According to a non-limiting embodiment, a method is disclosed for structuring, and / or transmitting (e.g., streaming), and / or decoding, and / or rendering volumetric information of the 3D scene to be encoded, based on a point cluster of the 3D scene. To capture the 3D scene, the 3D space is organized within a view bounding box, referred to as a 3DoF+ view bounding box. A cluster shared by different 3DoF+ view bounding boxes is obtained. The volumetric content of the 3DoF+ view bounding box is encoded using this cluster. A 6DoF volumetric rendering experience is achieved through a continuous 3DoF+ volumetric rendering experience.

[0055] The advantages of the principles of the invention for encoding, transmitting, receiving, and rendering will be presented in the following description with reference to the accompanying drawings.

[0056] Figure 1 A three-dimensional (3D) model 10 of an object and points corresponding to a point cloud 11 of the 3D model 10 are shown. The 3D model 10 and point cloud 11 may, for example, correspond to possible 3D representations of objects in a 3D scene including other objects. Model 10 may be a 3D mesh representation, and the points of point cloud 11 may be vertices of the mesh. The points of point cloud 11 may also be points distributed on the surface of the mesh face. Model 10 may also be represented as a sputtered version of point cloud 11, the surface of which is created by sputtering the points of point cloud 11. Model 10 may be represented by many different representations such as voxels or splines. Figure 1 This demonstrates that a point cloud can be defined using a surface representation of a 3D object, and that a surface representation of a 3D object can be generated from cloud points. As used herein, projecting points of a 3D object (and by extension, points of a 3D scene) onto an image is equivalent to projecting any representation of that 3D object, such as a point cloud, mesh, spline model, or voxel model.

[0057] Point clouds can be represented in memory as, for example, a vector-based structure, where each point has its own coordinates (e.g., 3D coordinates XYZ, or solid angle and distance from / to the viewpoint (also called depth)) and one or more attributes, also called components, in the viewpoint's frame of reference. An example of components is color components, which can be represented in various color spaces, such as RGB (red, green, and blue) or YUV (Y is the luminance component and UV are the two chrominance components). A point cloud is a representation of a 3D scene including objects. The 3D scene can be viewed from a given viewpoint or viewpoint range. Point clouds can be obtained in various ways, such as:

[0058] • Capture of real objects from camera equipment, optionally supplemented by active depth sensing devices;

[0059] • Capture of virtual / composite objects taken by a virtual camera setup within a modeling tool;

[0060] • A mixture of real and virtual objects.

[0061] Figure 2 A non-limiting example of encoding, transmitting, and decoding data representing a sequence of 3D scenes is shown. The encoding format may, for example, be compatible with 3DoF, 3DoF+, and 6DoF decoding simultaneously.

[0062] Obtain a 3D scene sequence. Just as a frame sequence is 2D video, a 3D scene sequence is 3D (also known as volumetric) video. The 3D scene sequence can be provided to a volumetric video rendering device for 3DoF, 3DoF+, or 6DoF rendering and display.

[0063] A sequence of 3D scenes 20 can be provided to encoder 21. Encoder 21 takes a 3D scene or a sequence of 3D scenes as input and provides a bitstream representing that input. The bitstream can be stored in memory and / or on an electronic data medium and can be transmitted via network 22. The bitstream representing the sequence of 3D scenes can be read from memory and / or received from network 22 by decoder 23. Decoder 23 takes the bitstream input and provides a sequence of 3D scenes in, for example, point cloud format.

[0064] Encoder 21 may include several circuits implementing several steps. In a first step, encoder 21 projects each 3D scene onto at least one 2D image. 3D projection is any method of mapping three-dimensional points onto a two-dimensional plane. This type of projection is widely used, especially in computer graphics, engineering, and drafting, because most current methods for displaying graphic data are based on a planar (pixel information from several bit planes) two-dimensional medium. The selected and used projection method can be represented and encoded as a set or list of projection parameters. Projection circuitry 211 provides at least one two-dimensional image 2111 for the 3D scene of sequence 20. Image 2111 includes color information and depth information representing the 3D scene projected onto image 2111. In a variant, the color and depth information are encoded in two separate images 2111 and 2112.

[0065] Metadata 212 is used and updated by projection circuit 211. Metadata 212 includes information about projection operations (e.g., projection parameters) and information about how color and depth information is organized within images 2111 and 2112, such as... Figures 5 to 7 As stated above.

[0066] The video encoding circuit 213 encodes the sequence of images 2111 and 2112 into video. Images 2111 and 2112 of the 3D scene (or a sequence of images of the 3D scene) are encoded in the stream by the video encoder 213. Then, the video data and metadata 212 are encapsulated in the data stream by the data encapsulation circuit 214.

[0067] Encoder 213 is compatible with, for example, encoders such as:

[0068] -JPEG, specification ISO / CEI 10918-1UIT-T Recommendation T.81, https: / / www.itu.int / rec / T-REC-T.81 / en;

[0069] -AVC, also known as MPEG-4 AVC or h264. It is specified in both UIT-T H.264 and ISO / CEI MPEG-4 Part 10 (ISO / CEI 14496-10), http: / / www.itu.int / rec / T-REC-H.264 / en, HEVC (its specification can be found on the ITU website, T recommendation, H series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en);

[0070] -3D-HEVC (an extension of HEVC, the specification of which can be found on the ITU website, T recommendation, H series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en annex G and I);

[0071] - VP9 developed by Google; or

[0072] - AV1 (AOMedia Video 1) was developed by Alliance for Open Media.

[0073] The data stream is stored in a memory accessible by the decoder 23, for example, via network 22. The decoder 23 includes different circuitry implementing various decoding steps. The decoder 23 takes the data stream generated by the encoder 21 as input and provides a sequence of 3D scenes 24 to be rendered and displayed by a volumetric video display device, such as a head-mounted display (HMD). The decoder 23 obtains the stream from the source 22. For example, the source 22 belongs to a group that includes:

[0074] - Local storage, such as video storage or RAM (or random access memory), flash memory, ROM (or read-only memory), hard disk;

[0075] - Storage interfaces, such as interfaces for mass storage devices, RAM, flash memory, ROM, optical discs, or magnetic media;

[0076] - Communication interfaces, such as wired interfaces (e.g., bus interfaces, WAN interfaces, LAN interfaces) or wireless interfaces (e.g., IEEE 802.11 interfaces or...). interface);

[0077] as well as

[0078] - User interfaces that enable users to input data, such as graphical user interfaces.

[0079] Decoder 23 includes circuitry 234 for extracting encoded data from the data stream. Circuitry 234 takes the data stream as input and provides metadata 232 corresponding to metadata 212 encoded in the stream and two-dimensional video. This video is decoded by video decoder 233, which provides a sequence of images. The decoded images include color and depth information. In a variant, video decoder 233 provides two image sequences, one containing color information and the other containing depth information. Circuitry 231 uses metadata 232 to deproject the color and depth information from the decoded images to provide a 3D scene sequence 24. The 3D scene sequence 24 corresponds to the 3D scene sequence 20, potentially resulting in a loss of accuracy associated with encoding and video compression as 2D video.

[0080] The principles disclosed herein relate to encoder 21, and more specifically to projection circuitry 211 and metadata 212. They also relate to decoder 23, and more specifically to deprojection circuitry 231 and metadata 232.

[0081] Figure 3 This shows that it can be configured to implement combination. Figure 14 and Figure 15 An exemplary architecture of device 30 for the described method. Figure 2 The encoder 21 and / or decoder 23 can implement this architecture. Alternatively, each circuit in the encoder 21 and / or decoder 23 can be based on... Figure 3 Devices with an architecture that are linked together, for example, via their bus 31 and / or via I / O interface 36.

[0082] Device 30 includes the following components connected together via data and address bus 31:

[0083] - Microprocessor 32 (or CPU), which is, for example, a DSP (or digital signal processor);

[0084] -ROM (or read-only memory) 33;

[0085] -RAM (or random access memory) 34;

[0086] - Storage interface 35;

[0087] -I / O interface 36, which is used to receive data to be transmitted from the application; and

[0088] - Power source, such as a battery.

[0089] According to one example, the power supply is external to the device. In each mentioned memory, the term "register" used in the specification can correspond to a small area (a few bits) or a very large area (e.g., the entire program or a large amount of received or decoded data). ROM 33 includes at least the program and parameters. ROM 33 can store algorithms and instructions for executing the technology according to the principles of the invention. When powered on, CPU 32 loads the program from RAM and executes the corresponding instructions.

[0090] RAM 34 contains the program executed by CPU 32 and uploaded after device 30 is turned on, input data in the register, intermediate data in different states of the method in the register, and other variables used to execute the method in the register.

[0091] The specific embodiments described herein may be implemented, for example, in methods or processes, apparatus, computer program products, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method or apparatus), the specific implementation of the discussed features may be implemented in other forms (e.g., programs). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented in apparatus (such as, for example, a processor) that generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0092] According to the example, the device 30 is configured to achieve bonding. Figure 14 and Figure 15 The described method, and belongs to a group that includes:

[0093] -mobile device;

[0094] - Communication equipment;

[0095] -Gaming devices;

[0096] - Tablet PC (or tablet computer);

[0097] - Laptop;

[0098] - Still image camera;

[0099] -Camera;

[0100] - Encoding chip;

[0101] - Servers (such as broadcast servers, video-on-demand servers, or web servers).

[0102] Figure 4 An example of an implementation of the syntax for streams is shown when data is transmitted via a packet-based transport protocol. Figure 4 An exemplary structure 4 for a volumetric video stream for a viewing bounding box is shown. Structure 4 organizes the stream in independent elements of a syntax. In this example, structure 4 includes three syntax elements 41, 42, and 43. Element 41 of the syntax is a header containing data common to all elements of structure 4. For example, this header 41 contains metadata describing the properties and function of each element of the syntax of structure 4. This header portion 41 also contains... Figure 2 This is part of the metadata 212, such as information related to the location of the viewing bounding box (e.g., the center point of the viewing bounding box).

[0103] The structure 4 includes a payload containing a syntax element 42 and at least one syntax element 43. The syntax element 42 includes encoded video data, such as data representing color and depth images 2111 and 2112.

[0104] Element 43 of the syntax includes metadata about how images 2111 and 2113 are encoded, specifically parameters used for projecting and encapsulating points of the 3D scene onto the images. This metadata can be associated with each image or group of images in the video (also known as a group of pictures (GoP) in video compression standards).

[0105] As mentioned earlier, rendering artifacts can occur during volumetric rendering experiences, such as regions with missing information. An example of missing information is parallax information. For instance, in the case of 3DoF+ volumetric rendering, the viewing space is limited to a finite volume called the viewing bounding box. A central focal point is attached to each viewing bounding box. When a user moves from the central focal point of the viewing bounding box and translates within the viewing bounding box, portions of the initially occluded 3D scene become visible. This is called the parallax effect, and the data associated with the occluded portions is called parallax data. This parallax data must be encoded and transmitted to render these occluded portions as the user moves. Depending on how the data is encoded, some parallax data may be lost, resulting in a degraded rendering experience. The parallax effect will then combine with... Figure 5 , Figure 6 andFigure 7 To provide a more detailed description.

[0106] Figure 5 An image representing a 3D scene is shown. 3D scenes can be captured using any suitable technology. Figure 5 The exemplary 3D scene shown includes several objects: houses 51 and 52, figures 54 and 55, and a well 56. Figure 5 The image depicts a cube 53 to illustrate the user's view bounding box from which the 3D scene is easily observed. The central point of the view bounding box 53 is referred to as 50.

[0107] Figure 6 Rendering on an immersive rendering device (e.g., a concave or head-mounted display device (HMD)) is shown in more detail. Figure 5 The concept of a viewing bounding box in a 3D scene. Scene point 64a in the 3D scene corresponds to the elbow of character 54. It is visible from viewpoint 50 because no occluding object is located between viewpoint 50 and scene point 64a. Conversely, scene point 65a, corresponding to the elbow of character 55, is not visible from viewpoint 50 because it is occluded by the point of character 54. In 3DoF+ rendering, the user can change the viewpoint within the 3DoF+ viewing bounding box, as previously described. For example, as in combination Figure 7 As shown, the user can move his viewpoint within the viewing bounding box 53 and experience parallax.

[0108] Figure 7 It shows Figure 5 The parallax experience allowed by the volumetric rendering of 3D scenes. Figure 7 B shows a portion of the 3D scene as seen by the user from the central point 50. From this viewpoint, characters 54 and 55 are in a given spatial configuration; for example, character 55's left elbow is hidden by character 54's body, while his head is visible. This configuration does not change as the user rotates his head in three degrees of freedom around the central point 50. If the viewpoint is fixed, character 55's left elbow (in...) Figure 6 It is referred to as 65a) invisible. Figure 7 A shows the first peripheral viewpoint from the left side of the viewing bounding box 53 (in Figure 6 The same 3D scene is referred to as 67. From viewpoint 67, point 65a is visible due to the parallax effect. This is called the deocclusion effect. For example, by moving from viewpoint 50 to viewpoint 67, point 65a is deoccluded. Figure 7 C shows the second peripheral viewpoint from the right side of the viewing bounding box 53 (in Figure 6 The same 3D scene observed from viewpoint 68 (referred to as 68). From viewpoint 68, character 55 is almost completely hidden by character 54, while from viewpoint 50 he is still visible. (Reference) Figure 6It is understandable that point 65b is occluded by moving from viewpoint 50 to viewpoint 68.

[0109] Very typically, the demasked data corresponds to small blocks of data. Figure 8 The demasking data required for volumetric rendering is shown. Figure 8 A is a top-down view of a 3D scene that includes two objects, P1 and P2, captured by three virtual cameras; a first peripheral camera C1, a central camera C2, and a second peripheral camera C3 associated with a viewing bounding box V. The center of the viewing bounding box V is located at the position of the central camera C2. Points visible from virtual cameras C1, C2, and C3 are represented by lines 81, 82, and 83, respectively. Figure 8 B. Figure 8 C and Figure 8 D shows the combination Figure 8 The rendering of the captured 3D scene described in A. Figure 8 B and Figure 8 In C, the cone F defines the field of view and portions of the 3D scene visible from viewpoints O0 and O1, respectively. O0 and O1 are viewpoints included within the viewing bounding box V. The user experiences parallax by moving from viewpoint O0 to viewpoint O1. Demasking points represent small patches within background objects.

[0110] exist Figure 8 In segment D, O2 represents a viewpoint outside the viewing bounding box V. From viewpoint O2, the new data represented by segment D, which was invisible from the viewing bounding box V, is now visible and unoccluded. This is the de-occlusion effect. Segment D does not belong to the volumetric content associated with the viewing bounding box V. When the user moves significantly, such as from viewpoint O0 to viewpoint O2 and outside the viewing bounding box V, it can result in uncompensated de-occlusion in different areas of the 3D scene. Unoccluded parts can represent large areas of missing information that are highly visible on the rendering device, leading to a poor immersive experience.

[0111] The way information in the volume content to be encoded is structured affects encoding efficiency, as shown below.

[0112] for Figure 8 3D scenes, Figure 9 A illustrates a first method for structuring volumetric information representing a 3D scene, and Figure 9 B illustrates a method for structuring identical volume information according to a non-limiting embodiment of the principles of the present invention.

[0113] According to this first method, the only element contained in the closed dashed line 910 is captured from the viewpoint O0. In fact, the only accessible data is represented by the thick lines 911, 912, and 913. It can be observed that the region of object P2 occluded by object P1 is inaccessible; that is, the region of object P2 is missing.

[0114] In the principle of this invention, points in a 3D scene are clustered according to clustering criteria. Figure 9 In the implementation described in B, the clustering criterion involves the depth range of points in a 3D scene, thus separating the 3D scene into multiple depth layers. This allows for, for example, the creation of background and foreground clusters, which respectively include portions of physical objects participating in the scene's background and foreground. Alternatively or in combination, the clustering is based, for example, on semantic classification and / or motion classification and / or color segmentation of points. All points in a cluster share the same properties. Figure 9 In section B, two clusters are obtained, enclosed by closed dashed lines 921 and 922 respectively. The accessible data represented by thick lines 923 and 924 differs from that used in... Figure 9 The data obtained using the first method shown in Figure A. Figure 9 In B, all information relating to P2 is available, even information located behind object P1 viewed from viewpoint O0. It is not a combination. Figure 9 The method described in section A. By structurally representing the volumetric information of a 3D scene using clustered points according to the principles of the present invention, it is possible to increase the information available for rendering the 3D scene. Returning to the previously described parallax experience, one advantage of the clustering method described above is that data related to occlusion areas is accessible from any viewpoint.

[0115] Figure 10 The example illustrates how clusters 921 and 922 are obtained. This example refers to a case where the clustering criterion is a depth-filtering criterion. One method to obtain this cluster is by using virtual cameras with different positions, orientations, and fields of view to capture points. Each virtual camera is optimized to capture as many points as possible for a given cluster. For example, in Figure 10 In this context, a virtual camera (C) can be used. A_0 Capture cluster 921. Virtual camera C A_0 Capture all pixels within the near depth range, and clip object P2 that is not within the near depth range. Cluster 922 can use a virtual camera C. B_0 Capture. Virtual camera C B_0 Capture all pixels within the far depth range, and clip objects P1 that do not belong to the far depth range. Advantageously, use a virtual camera positioned at a distance, independent of that viewpoint and viewing bounding box, to acquire the background clusters, while use virtual cameras positioned at different viewpoints within that viewing bounding box to acquire the foreground clusters. Typically, virtual cameras positioned at a smaller number of viewpoints within the viewing bounding box compared to the foreground clusters are used to capture medium depth clusters.

[0116] We will now explain how the volumetric information of a 3D scene, structured by the point clustering method described above, is encoded in the video stream.

[0117] Figure 11 A 2D atlas method is shown for encoding the volumetric content representing a 3D scene from a given viewpoint 116. Figure 11 The image shows a top view 100 of a 3D scene. The 3D scene includes a person 111, a flowerpot 112, a tree 113, and a wall 114. Image 117 is an image representing the 3D scene as observed from viewpoint 116. Using a point clustering method, clusters represented by dashed ellipses 111c, 112c, 113c, and 114c are obtained from the volumetric content and projected in the direction of viewpoint 116 to create a set of 2D images. This set of 2D images is then encapsulated to form an atlas 115 (an atlas is an aggregation of 2D images). The organization of the 2D images within the atlas defines the atlas layout. In this embodiment, two atlases have the same layout: one for color (or texture) information and one for depth information.

[0118] A time series of 2D maps is generated for consecutive points in time. Typically, the time series of 2D images is transmitted as a set of coded videos, where each video corresponds to a specific cluster: each image in the video corresponds to a 2D image obtained by projecting this specific cluster onto the viewpoint 116 at a given time. A series of 2D images of this specific cluster constitutes an independent video.

[0119] The point clustering method based on the principles of this invention aims to structurally represent the volumetric information of a 3D scene in a way that allows this volumetric information to be encoded as a set of independent videos.

[0120] Utilizing the principles of this invention, the 3D scene is not transmitted as a single video stream corresponding to a series of images 117 acquired at different points in time, but rather as a set of smaller, independent videos corresponding to a sequential time series of 2D images within a 2D atlas. Each video can be transmitted independently of the others. For example, different videos can be acquired by using virtual cameras with different fields of view. In another example, different videos can be encoded with different image rates or different quality levels.

[0121] For example, frequent configurations are 3D scenes where animated foreground objects move more than the background of the scene. These animated objects have their own lifecycles and can be advantageously encoded with a higher image rate than the background.

[0122] Furthermore, when volumetric content is streamed, the quality of the video can be adjusted for each video stream to adapt to changes in the streaming environment. For example, the video stream corresponding to the foreground can be encoded at a higher quality than the video stream corresponding to the background of the scene.

[0123] Another advantage is that it allows for scalable personalization of the 3D scene, such as by inserting specific objects like advertisements. This customization is optimized compared to volumetric content that would be encoded in a uniform manner.

[0124] To decode, the 3D scene is obtained by combining the independent video streams. 2D images corresponding to different clusters in the 2D atlas are recombined to form an image representing the 3D scene as seen from viewpoint 116. This image undergoes a 2D-to-3D deprojection step to obtain volumetric data. This volumetric data is rendered from the viewpoint corresponding to viewpoint 116 in the 3D rendering space during the volumetric rendering experience.

[0125] We will now explain how a 6DoF volumetric rendering experience, based on a continuous 3DoF+ volumetric rendering experience, can benefit from using point clustering methods as described previously.

[0126] 3D scenes can be rendered by continuously rendering the volumetric content associated with the view bounding box and moving from one view bounding box to another in the 3D rendering space. The benefits in terms of data storage and transport, for example, will be highlighted below.

[0127] Figure 12 It shows Figure 11 A top-down view of the 3D scene, where the viewing bounding box is represented by a dashed ellipse 121. Two dashed lines 122 and 123 represent the field of view visible from the viewing bounding box 121. This field of view includes the view through... Figure 11 The four clusters obtained from clustering points in the 3D scene are: cluster 120a associated with flowerpot 112, cluster 120b associated with task 111, cluster 120c associated with tree 113, and cluster 120d associated with wall 114.

[0128] Two viewpoints 124 and 125, contained within the viewing bounding box 121, and their respective fields of view are represented (by two cones 126 and 127). It can be observed that some clusters, or portions of some clusters, are shared by viewpoints 124 and 125. Figure 12 In the example, these shared clusters are clusters 120c and 120d. In this particular example, they correspond to portions of the 3D scene at distances from viewpoints 124 and 125. The 2D image produced by the 3D-to-2D projection step of these shared clusters is called the 2D shared image. The 2D image produced by the 3D-to-2D projection step of clusters other than the shared clusters is called the 2D patch.

[0129] 2D shared images typically contain a majority of non-empty pixels. For example, when using depth standards, shared clusters often correspond to background points within the volumetric content and contain multiple points. Typically, 2D patches are small regions distinct from the areas surrounding them. 2D patches generally contain less information than 2D shared images and therefore have a smaller size in terms of, for example, the number of pixels. For example, clusters corresponding to foreground points within the volumetric content typically contain a limited number of points, such as those representing a person or object placed in front of a large background feature.

[0130] The two maps, comprising a set of 2D images generated by 3D-to-2D projections of clusters associated with viewpoints 124 and 125 respectively, share a common 2D shared image. Therefore, when moving from viewpoint 124 to viewpoint 125 within the viewing bounding box 121, or vice versa, the data corresponding to the 2D shared image is already available for rendering. This results in an improved parallax experience for the user. The latency that would otherwise be introduced to retrieve and render this data is eliminated. Another advantage is a reduction in the amount of data to be transmitted.

[0131] Returning to the reference 2D atlas method, the 2D shared image is transmitted as a shared video, with each 2D patch transmitted as a separate video. The shared information previously embedded in each image 117 is shared and transmitted separately in the shared video. When this depth standard is used, the shared video typically corresponds to a cluster representing the background portion of a 3D scene. The shared video will be very stable or show almost no change over time, such as... Figure 11 Wall 114. Therefore, for example, by means of time prediction, a very efficient codec can be used to encode shared video.

[0132] Figure 13 It shows Figure 11 A top-down view of a 3D scene is shown, in which two viewing bounding boxes 131 and 138 are represented. A viewpoint 134 within viewing bounding box 131 and a viewpoint 135 within viewing bounding box 138 are shown. The first viewpoint 134 is located within viewing bounding box 131, and the second viewpoint 135 is located within viewing bounding box 138. The fields of view from viewpoints 134 and 135 are referenced to 136 and 137, respectively. It can be seen that clusters or portions of clusters are common to both fields of view 136 and 137. Therefore, viewing bounding boxes 131 and 138 share common clusters or portions of clusters.

[0133] The shared 2D images corresponding to these common clusters can be shared among several view bounding boxes. For each view bounding box, they can be stored, encoded, transmitted, and rendered once. This results in a further reduction in the data load used for storage and transmission. Another advantage is reduced latency artifacts when the user makes extensive movements in the render space (from the first view bounding box to the second view bounding box).

[0134] Figure 14 A method for encoding volumetric content involving a 3D scene, according to a non-constraint embodiment of the principles of the present invention, is illustrated. This method is intended for use in... Figure 2 In encoder 21.

[0135] In step 1400, the 3D scene is obtained from the source.

[0136] In step 1401, points in the 3D scene are clustered into multiple clusters according to at least one clustering criterion. In an implementation, the clustering criterion relates to the depth range of points in the 3D scene, thus separating the 3D scene into multiple depth layers. This allows, for example, the creation of background clusters and foreground clusters, which include portions of physical objects that participate in the background and foreground of the scene, respectively. Alternatively or in combination, the clustering is, for example, based on semantic classification and / or motion classification and / or color segmentation of points. For a given viewpoint, the 3D scene is described as a set of clusters.

[0137] In step 1402, the clusters of the group are projected according to the projection parameters to obtain a set of 2D images. These 2D images are encapsulated within a map, or within two maps with the same layout. For example, one map contains color data, and the other maps contain depth data.

[0138] In step 1403, volumetric content carrying data representing the 3D scene is generated. The data representing the 3D scene is a map or a pair of maps obtained in step 1402.

[0139] In the implementation, the 3D rendering space is organized into viewing bounding boxes, each including a central viewpoint and, in a preferred implementation, peripheral viewpoints. In step 1401', a common cluster of different viewing bounding boxes is obtained.

[0140] When performing step 1401', step 1402 includes two sub-steps 1402A and 1402B. In sub-step 1402A, a cluster common to different viewing bounding boxes is projected according to projection parameters to obtain a 2D shared image. In sub-step 1402B, clusters other than those common to different viewing bounding boxes are projected to obtain 2D patches. This is done frame-by-frame. For each viewing bounding box, the cluster is projected in the direction of the center point of that viewing bounding box to create a set of 2D patches. Preferably, the cluster is also projected in the direction of one or more peripheral viewpoints, thereby creating additional sets of 2D patches (one for each peripheral viewpoint). As a result, each viewing bounding box is associated with several sets of 2D shared images and 2D patches.

[0141] In step 1402', metadata is generated, which includes a list of view bounding boxes contained in the 3D rendering space of the 3D scene, and a list of sets of applied 2D shared images and 2D patches used for the view bounding boxes in the 3D rendering space. The metadata generated at step 1402' is included in the volume content generated in step 1403. For example, as Figure 4 Structure 4, as described in the document, is used to encapsulate information related to the viewing bounding box, and all structures 4 of the 3D scene are encapsulated together in a superstructure that includes a header containing metadata generated at step 1402'.

[0142] For example, the metadata generated at step 1402' includes:

[0143] - A list of bounding boxes to view in 3D rendering space

[0144] - A list of common clusters for this 3D rendering space, each common cluster being characterized by a common cluster identifier and associated with a uniform source identifier used to retrieve the corresponding video stream from that source.

[0145] - For each view bounding box: a list of groups representing clusters in the 3D scene;

[0146] - For each cluster associated with the viewing bounding box:

[0147] The identifier of the common cluster, and

[0148] ο A list of clusters other than the common cluster that has a uniform source identifier for retrieving the corresponding video stream from the source.

[0149] In an advantageous implementation, the 2D image is encoded with different quality levels or different image rates, resulting in several sets of 2D images for the same viewpoint. This allows, for example, adjustments to the video quality or rate to accommodate streaming environments.

[0150] Figure 15 A method for decoding volumetric content involving a 3D scene, according to a non-constraint embodiment of the principles of the present invention, is illustrated. This method is intended to use... Figure 2 Decoder 23.

[0151] In step 1500, the volumetric content is obtained from the source. The volumetric content contains at least one 2D image representing at least one cluster of points in a 3D scene. The points in the cluster satisfy a clustering criterion. In an embodiment, the clustering criterion relates to the depth range of points in the 3D scene. Alternatively or in combination, the clustering criterion relates to, for example, semantic classification and / or motion classification and / or color segmentation of points.

[0152] In step 1501, the at least one 2D image is projected according to the projection parameters.

[0153] In step 1502, a 3D point cloud representing the 3D scene is obtained from the deprojected 2D image.

[0154] Figure 16 A non-limiting embodiment of the principles of the present invention is shown, a method for rendering volumetric content involving a 3D scene in a device configured to be used as a volumetric display device or a rendering device.

[0155] In step 1600, a first viewpoint in the 3D rendering space is obtained. This first viewpoint relates to a first viewing bounding box in the 3D rendering space. When the rendering device is an HMD, the first viewpoint is the end-user's position, obtained, for example, using the HMD's IMU (Inertial Measurement Unit). The HMD includes one or more displays (e.g., LCD (Liquid Crystal Display), OLED (Organic Light Emitting Diode), or LCOS (Liquid Crystal on Silicon)) and sensors, such as gyroscopes or IMUs, configured to measure changes in the HMD's position according to one, two, or three axes (pitch, yaw, and / or roll) in the real world.

[0156] In step 1601, the rendering device receives a first volumetric content relating to the 3D scene. This first volumetric content includes metadata associated with the 3D scene, as described above with respect to step 1402' (a list of view bounding boxes contained in the 3D rendering space and a list of groups of 2D patches for a 2D shared image for each view bounding box) and video data and metadata associated with the first view bounding box.

[0157] In step 1602, the first volume content is decoded using the above-described decoding method to obtain a first 3D point cloud representing the 3D scene. Based on the metadata received in step 1601, a set of shared 2D images and 2D patches corresponding to the first viewpoint are selected. The 2D image is projected according to the projection parameters transmitted in the stream. As a result, the first 3D point cloud is obtained.

[0158] In step 1603, the first 3D point cloud is rendered from the first viewpoint and displayed based on volume rendering.

[0159] As mentioned earlier, 6DoF rendering can be enabled by rendering several volumes of content consecutively in 3DoF+. To achieve this, the rendering method according to the principles of the present invention includes the following additional steps.

[0160] In step 1604, the user moves from the first viewpoint to the second viewpoint in the rendered 3D space.

[0161] In step 1605, the set of 2D images to be used for rendering from the second viewpoint is obtained based on the metadata retrieved in step 1601. 2D images that are not yet available for rendering are retrieved from the source. Previously retrieved shared 2D images do not need to be retrieved again.

[0162] In step 1606, the 2D image retrieved from the source is deprojected to create a second 3D point cloud. This second 3D point cloud is combined with points from the first 3D point cloud corresponding to the common 2D image between the first and second viewing bounding boxes.

[0163] In step 1607, the result of this combination is rendered and displayed from the second viewpoint according to the 3DoF+ volumetric rendering technique.

[0164] When the user moves from one viewpoint to another viewpoint within the 3D scene, steps 1604 to 1607 can be repeated.

[0165] The previously described rendering method illustrates the principles of this invention: how to use clustered volumetric element groups based on multi-view 3DoF+ rendering to allow 6DoF volumetric rendering.

[0166] Figure 17 A method for receiving volumetric content relating to a 3D scene in 3D rendering space in a device configured as a receiver, according to a non-constraintive embodiment of the principles of the present invention, is illustrated. Figure 17 In the example, the volumetric rendering experience occurs in an adaptive streaming environment. The video stream is encoded with different quality levels or different image rates. And this receiver includes an adaptive streaming player that detects the conditions of the adaptive streaming environment and selects the video stream to be transmitted.

[0167] In step 1700, the receiver receives metadata associated with the 3D scene. For example, when using the DASH streaming protocol, this metadata, also known as representation, is transmitted using a Media Rendering Description (MPD). As previously described, this metadata includes a list of view bounding boxes contained in the 3D rendering space, and information about the clusters of view bounding boxes / viewpoints to be used for rendering (identifiers of the clusters to be used and information for retrieving the clusters from the source).

[0168] In step 1701, the adaptive streaming player detects the conditions of the streaming environment, such as available bandwidth.

[0169] In step 1702, a specific viewing bounding box / viewpoint in the 3D rendering space is considered. Using the conditions of the streaming environment, the adaptive streaming player selects a list of groups of at least one 2D shared image and at least one 2D patch. For example, priority is given to the foreground cluster, such that higher-quality 2D patches with lower-quality 2D shared images are selected.

[0170] In step 1703, the adaptive streaming player sends a request for the selected group to the server.

[0171] In step 1704, the receiver receives the selected group. The group is then decoded and rendered according to one of the previously presented methods.

[0172] Besides depth, or as an alternative, other criteria, such as motion, can be used. Bandwidth prioritization is typically used to select 2D patches for encoding fast-moving clusters, still compared to clustering. In practice, a portion of the 3D scene can be static, while other objects can move at various speeds. This is particularly evident for small animated objects (often in the foreground) that can have their own lifecycles (position, color) different from other elements in the scene (often in the background). Clustering such objects relative to their motion speed, for example, allows them to be transmitted according to different transmission parameters (such as frequency rate). One advantage is the reduced streaming cost due to content heterogeneity.

[0173] In another specific embodiment of the invention, the receiver includes a prediction module to predict the user's next position in the 3D rendering space. A corresponding group is selected based on this metadata. When several clusters are available, one of them is selected as described above. Finally, the receiver sends a request to retrieve the corresponding video stream.

[0174] Using the current principle, it is likely that some video stream, such as a more stable background video stream, will be needed. Advantageously, the receiver considers the probability of occurrence and triggers the retrieval of the most likely video stream first. Foreground clusters are more general and the transmission is also lighter. The receiver can postpone prediction and retrieval until the last receivable moment. Therefore, the cost of erroneous predictions is reduced.

[0175] The specific embodiments described herein may be implemented, for example, in methods or processes, apparatus, computer program products, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method or apparatus), the specific implementation of the discussed features may be implemented in other forms (e.g., programs). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented in apparatus (such as, for example, a processor) that generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, smartphones, tablets, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0176] Specific implementations of the various processes and features described herein can be found in a wide variety of devices or applications, particularly those associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture and / or depth information. Examples of such devices include encoders, decoders, post-processors that process the output from decoders, pre-processors that provide input to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, cellular phones, PDAs, and other communication devices. It should be understood that the devices can be mobile, even mounted in mobile vehicles.

[0177] Additionally, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) can be stored on a processor-readable medium, such as, for example, an integrated circuit, a software carrier, or other storage device, such as, for example, a hard disk, a compact disk (“CD”), an optical disk (such as, for example, a DVD, commonly referred to as a digital versatile optical disk or digital video optical disk), random access memory (“RAM”), or read-only memory (“ROM”). Instructions can form an application program tangibly embodied on the processor-readable medium. Instructions can be, for example, hardware, firmware, software, or a combination thereof. Instructions can be found, for example, in an operating system, a standalone application, or a combination of both. Thus, a processor can be characterized, for example, as a device configured to execute a process and a device comprising a processor-readable medium (such as a storage device) having instructions for executing the process. Furthermore, in addition to or instead of instructions, the processor-readable medium can store data values ​​generated by the implementation.

[0178] It will be apparent to those skilled in the art that the embodiments may produce various signals formatted to carry, for example, storable or transmissible information. The information may include, for example, instructions for performing a method or data generated by one of the embodiments. For example, the signal may be formatted as data carrying rules for writing or reading the syntax of the described embodiment, or as data carrying actual syntax values ​​written by the described embodiment. Such signals may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and using a modulated carrier for the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0179] Several specific embodiments have been described. However, it should be understood that many modifications can be made. For example, elements of different embodiments can be combined, supplemented, modified, or removed to produce other embodiments. Furthermore, those skilled in the art will understand that other structures and processes can be replaced with those disclosed, and the resulting embodiments will perform at least substantially the same function in at least substantially the same manner to achieve at least substantially the same results as the disclosed embodiments. Therefore, this application considers these and other embodiments.

Claims

1. A method for encoding a 3D scene, the method comprising: The points in the 3D scene are clustered into multiple clusters based on the depth range of the points in the 3D scene; A first set of 2D images is obtained by projecting a cluster visible from a first set of viewing points, including at least two viewing points, according to a first projection parameter, wherein the first set of viewing points surrounds a first viewing frame defined in the 3D scene. A second set of 2D images is obtained by projecting a cluster visible from a second set of viewing points according to a second set of projection parameters, wherein the second set of viewing points is surrounded by a second viewing frame defined in the 3D scene that is different from the first viewing frame. Identify the 2D shared image corresponding to the shared cluster of the first viewing frame and the second viewing frame; as well as The first set of 2D images and the first projection parameter, excluding the 2D shared image, are encoded in a first data stream, and each 2D image and the second projection parameter in the second set of 2D images, excluding the 2D shared image, are encoded in a different set of data streams, and the 2D shared image is encoded only once in the shared data stream.

2. The method of claim 1, wherein the clustering is further based on semantics associated with points in the 3D scene, color of the points in the 3D scene, or motion of the points in the 3D scene.

3. The method according to any one of claims 1 to 2, wherein the method further comprises encoding metadata, the metadata including: The list of viewing frames defined in the 3D scene and the list of shared clusters in the 3D scene; and For a view frame, there is a list of multiple clusters representing the 3D scene of the view frame, and for each cluster associated with the view frame, an identifier for the shared cluster and a list of clusters other than the shared cluster.

4. A method for decoding a 3D scene, the method comprising: Decode at least two first 2D images from a first data stream, excluding a shared 2D image, and a second 2D image from each of a different set of data streams, excluding the shared 2D image, and decode the shared 2D image from the shared data stream only once. The first 2D image represents a projection of at least one cluster of points in the 3D scene visible from a first set of viewing points, according to a first projection parameter, the first set of viewing points enclosing a first viewing frame defined in the 3D scene. The second 2D image represents a projection of at least one cluster of points in the 3D scene visible from a second set of viewing points, according to a second projection parameter, the second set of viewing points enclosing a second viewing frame defined in the 3D scene, different from the first viewing frame. The shared 2D image corresponds to a shared cluster of the first and second viewing frames, and the points in the 3D scene are clustered into multiple clusters based on the depth range of the points in the 3D scene. as well as The pixels of the first 2D image are projected according to the first projection parameters and the first set of viewing points, and the pixels of the second 2D image are projected according to the second projection parameters and the second set of viewing points, and the pixels of the 2D shared image are also projected.

5. The method according to claim 4, further comprising: Obtain metadata, which includes: The list of viewing frames defined in the 3D scene and the list of shared clusters in the 3D scene; and For a view frame, there is a list of multiple clusters representing the 3D scene of the view frame, and for each cluster associated with the view frame, an identifier for the common cluster and a list of clusters other than the common cluster. Decode a 2D image from a data stream, which includes a cluster of 3D points visible from the current viewing point.

6. An apparatus for encoding a 3D scene, the apparatus including a memory associated with a processor configured to perform the following operations: The points in the 3D scene are clustered into multiple clusters based on the depth range of the points in the 3D scene; A first set of 2D images is obtained by projecting a cluster visible from a first set of viewing points, including at least two viewing points, according to a first projection parameter, wherein the first set of viewing points surrounds a first viewing frame defined in the 3D scene. A second set of 2D images is obtained by projecting a cluster visible from a second set of viewing points according to a second set of projection parameters, wherein the second set of viewing points is surrounded by a second viewing frame defined in the 3D scene that is different from the first viewing frame. as well as Identify the 2D shared image corresponding to the shared cluster of the first viewing frame and the second viewing frame; as well as The first set of 2D images and the first projection parameter, excluding the 2D shared image, are encoded in a first data stream, and each 2D image and the second projection parameter in the second set of 2D images, excluding the 2D shared image, are encoded in a set of different data streams, and the 2D shared image is encoded only once in the shared data stream.

7. The device of claim 6, wherein the clustering is further based on semantics associated with points in the 3D scene, color of the points in the 3D scene, or motion of the points in the 3D scene.

8. The device of claim 6 or 7, wherein the processor is further configured to encode metadata, the metadata comprising: The list of viewing frames defined in the 3D scene and the list of shared clusters in the 3D scene; and For a view frame, there is a list of multiple clusters representing the 3D scene of the view frame, and for each cluster associated with the view frame, an identifier for the shared cluster and a list of clusters other than the shared cluster.

9. A device for decoding a 3D scene, the device including a memory associated with a processor configured to perform the following operations: Decode at least two first 2D images from a first data stream, excluding a shared 2D image, and second 2D images from each of a set of different data streams, excluding the shared 2D image, and decode the shared 2D image from the shared data stream only once. The first 2D image represents a projection of at least one cluster of points in the 3D scene visible from a first set of viewing points, according to a first projection parameter, the first set of viewing points enclosing a first viewing frame defined in the 3D scene. The second 2D image represents a projection of at least one cluster of points in the 3D scene visible from a second set of viewing points, according to a second projection parameter, the second set of viewing points enclosing a second viewing frame defined in the 3D scene, different from the first viewing frame. The shared 2D image corresponds to a shared cluster of the first and second viewing frames, and the points in the 3D scene are clustered into multiple clusters based on the depth range of the points in the 3D scene. The pixels of the first 2D image are projected according to the first projection parameters and the first set of viewing points, and the pixels of the second 2D image are projected according to the second projection parameters and the second set of viewing points, and the pixels of the 2D shared image are also projected.

10. The device of claim 9, further wherein the processor is further configured to perform the following operations: Obtain metadata, which includes: The list of viewing frames defined in the 3D scene and the list of shared clusters in the 3D scene; and For a view frame, there is a list of multiple clusters representing the 3D scene of the view frame, and for each cluster associated with the view frame, an identifier for the common cluster and a list of clusters other than the common cluster. Decode a 2D image from a data stream, which includes a cluster of 3D points visible from the current viewing point.

Images