Hybrid volumetric video encoding with implicit neural video representations to handle non-lambertian surfaces

Abstract

The present principles relate to methods, devices and data stream for encoding, decoding and rendering a 3D scene. The encoding method comprises segmenting diffuse parts and non-diffuse parts of the 3D scene. Then, the diffuse parts are encoded as a patch atlas in a data stream and, for the non-diffuse parts, their locations are encoded in the data stream and an implicit neural network is trained. Weights of the implicit neural network are then encoded in the data stream. In an embodiment, the 3D scene is obtained as a multi-view plus depth information and the method comprises transforming the 3D scene in a 3D point cloud. In another embodiment, the implicit neural network is a Neural Radiance Fields network. In yet another embodiment, the architecture of the implicit neural network is encoded in the data stream.

Description

[0001] HYBRID VOLUMETRIC VIDEO ENCODING WITH IMPLICIT NEURAL VIDEO REPRESENTATIONS TO HANDLE NON-LAMBERTIAN SURFACES

[0002] 1. Technical Field

[0003] The present principles generally relate to the domain of encoding, decoding and rendering volumetric videos comprising non-Lambertian elements. In particular, the present principles relate to formatting Lambertian surfaces as image patches while non-Lambertian surfaces are encoded as implicit neural networks. The present document is also understood in the context of the playing of volumetric videos and / or virtual or extended reality applications when rendered on end-user devices such as mobile devices or Head- Mounted Displays (HMD) like see-through glasses.

[0004] 2. Background

[0005] The present section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present principles that are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present principles. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0006] Advances in 3D capturing and rendering technologies makes nowadays volumetric video an integral part of Virtual / Augmented / Mixed reality (VR / AR / MR) applications. Volumetric video can be defined as a sequence of 3D frames, that can be in various 3D formats such as point clouds, meshes or multi-view plus depth videos. An efficient approach to encode volumetric frames is by converting the 3D volumetric information into a collection of 2D images (commonly called “patches” because they are packed in atlas images) and associated data (describing how the patches have been obtained). The converted 2D images may be coded using legacy 2D video encoders, based on specifications such as AVC, HEVC or VVC and the associated data can be coded in an additional metadata stream. The coded images and the associated metadata can then be decoded and used to reconstruct the 3D volumetric information. MPEG Immersive Video (MIV) standard developed by MPEG belongs to this 2D- video-compatible approach and is dedicated to multi-view plus depth inputs. However, such an approach does not consider non-Lambertian characteristics (like specular surfaces, transparent objects, non-diffuse reflection, etc.). In such cases, a same (for example reflected) object has different aspects according to the view (that is according to the location of the camera which has captured it). At the rendering, difference appear as a function of the pose of the virtual camera used to synthetize a view. So, there is a lack of encoding, decoding and rendering methods for handling realistic volumetric videos.

[0007] 3. Summary

[0008] The following presents a simplified summary of the present principles to provide a basic understanding of some aspects of the present principles. This summary is not an extensive overview of the present principles. It is not intended to identify key or critical elements of the present principles. The following summary merely presents some aspects of the present principles in a simplified form as a prelude to the more detailed description provided below.

[0009] The present principles relate to a method for encoding a 3D scene. The method comprises segmenting diffuse parts and non-diffuse parts of the 3D scene. Then, the diffuse parts are encoded as a patch atlas in a data stream and, for the non-diffuse parts, their locations are encoded in the data stream and an implicit neural network is trained. Weights of the implicit neural network are then encoded in the data stream. In an embodiment, the 3D scene is obtained as a multi-view plus depth information and the method comprises transforming the 3D scene in a 3D point cloud. In another embodiment, the implicit neural network is a Neural Radiance Fields network. In yet another embodiment, the architecture of the implicit neural network is encoded in the data stream.

[0010] The present principles also relate to a device comprising a processor and a memory associated with the processor that is configured to implement the method above.

[0011] The present principles also relate to a method for decoding a 3D scene. The method comprises decoding a patch atlas from a data stream. The patches of the patch atlas correspond to diffuse parts of the 3D scene. Locations of non-diffuse parts of the 3D stream are decoded from the data stream as well as weights of an implicit neural network. Then, a view is synthetized by using the patches for pixels of the view relative to diffuse parts of the 3D scene and by inputting parameters relative to pixels of non-diffuse parts of the view in the implicit neural network.

[0012] The present principles also relate to a device comprising a processor and a memory associated with the processor that is configured to implement the method above.

[0013] The present principles also relate to a data stream representative of a 3D scene and comprising a patch atlas for diffuse parts of the 3D scene and weights of an implicit neural network for the non-diffuse parts of the 3D scene. 4. Brief Description of Drawings

[0014] The present disclosure will be better understood, and other specific features and advantages will emerge upon reading the following description, the description making reference to the annexed drawings wherein:

[0015] - Figure 1 shows an example of a 3D scene having non- Lambertian surfaces captured from two points of view;

[0016] - Figure 2 illustrates a method for encoding a 3D scene according to the present principles;

[0017] - Figure 3 shows an example architecture of a device which may be configured to implement encoding, decoding and / or rendering methods according to Figure 2 or 6;

[0018] - Figure 4 shows an example of an embodiment of the syntax of a data stream encoding a 3D scene comprising non-Lambertian parts according to the present principles;

[0019] - Figure 5 illustrates segmenting step of Figure 2 when the 3D scene is captured as a MVD image; and

[0020] - Figure 6 diagrammatically shows a method 60 for decoding and for rendering a 3D scene comprising non-Lambertian surfaces according to the present principles.

[0021] 5. Detailed description of embodiments

[0022] The present principles will be described more fully hereinafter with reference to the accompanying figures, in which examples of the present principles are shown. The present principles may, however, be embodied in many alternate forms and should not be construed as limited to the examples set forth herein. Accordingly, while the present principles are susceptible to various modifications and alternative forms, specific examples thereof are shown by way of examples in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the present principles to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present principles as defined by the claims.

[0023] The terminology used herein is for the purpose of describing particular examples only and is not intended to be limiting of the present principles. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises", "comprising," "includes" and / or "including" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Moreover, when an element is referred to as being "responsive" or "connected" to another element, it can be directly responsive or connected to the other element, or intervening elements may be present. In contrast, when an element is referred to as being "directly responsive" or "directly connected" to other element, there are no intervening elements present. As used herein the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as" / ".

[0024] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the teachings of the present principles.

[0025] Although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0026] Some examples are described with regard to block diagrams and operational flowcharts in which each block represents a circuit element, module, or portion of code which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in other implementations, the function(s) noted in the blocks may occur out of the order noted. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0027] Reference herein to “in accordance with an example” or “in an example” means that a particular feature, structure, or characteristic described in connection with the example can be included in at least one implementation of the present principles. The appearances of the phrase in accordance with an example” or “in an example” in various places in the specification are not necessarily all referring to the same example, nor are separate or alternative examples necessarily mutually exclusive of other examples.

[0028] Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims. While not explicitly described, the present examples and variants may be employed in any combination or sub-combination.

[0029] Figure 1 shows an example of a 3D scene having non- Lambertian surfaces captured from two points of view 10 and 11. The 3D scene comprises transparent objects, like glasses in the example of Figure 1 and reflecting objects, like the oven glass. Objects of the scene are not seen the same way according to the view they are captured or synthetized from. These light effects are difficult to handle. For example, MIV format does not handle specular reflections and other complex light effects and assumes that the 3D scene is composed of Lambertian surfaces, i.e. with diffuse reflection only. For realistic rendering, if a specular reflection is captured by one camera of an acquisition rig, as observed from the point of view of this camera, rendering the 3D scene from a different virtual viewpoint requires modifying the position and appearance of the reflected content according to the new point of view. There is no information transmitted in an MIV stream to perform such an operation. Moreover, as a rendered virtual view is typically generated by blending patches originating from several input views, each one having captured a given reflection at different positions in the frame, replications of the reflected objects may be observed. Though, as reflecting materials are commonly observed in 3D scenes.

[0030] A possible method consists in “baking” the light effects. This method selects one of the input views (for instance, a central view covering the major part of the 3D scene) to convey the viewport dependent light effect. Instead of blending patches originating from different views at rendering stage, only patches from the selected view are used, when available. Doing so, the light effects are baked and do not change whatever the viewing position, as expected. The result is a visually stable rendered content but is not realistic with respect to specular light effects.

[0031] Another method for handling non-diffuse reflecting surfaces is to transmit multiple, view-dependent, textures per surface patch. For example, 3 or 5 or 7 representative textures per patch are selected among, for instance a greater number of captures. The drawback of such a method is the increase of pixel rate and bit rate, which rapidly becomes prohibitive.

[0032] Al-based rendering technologies introduced Neural Radiance Fields (NeRF). NeRF is a method that synthesizes new views of complex 3D scenes by leveraging fully connected neural networks. In a nutshell, an Implicit Neural network Representation (INR) (for example, a fully connected multilayer perceptron (MLP)) is learned from a multi-view capture of the 3D scene. This MLP can be queried with 3D spatial coordinates (x,y,z) and 2D viewing direction d=(q,f) and outputs the volume density s and a RGB color at that point if the scene. At rendering stage, a 2D view of the 3D scene can be generated by evaluating the color C(u,v,d) of a pixel at location (u,v) as a camera ray going through the 3D scene. The color of a ray a function of a start point, an end point and a direction given by equations Eq1 , where co denote the parameters of the NeRF: In practice, a 2D view is rendered by approximating these integrals. An INR is a neural network, composed of multiple neural layers, such as fully connected layers. Each neural layer can be described as a function that first multiply the input by a tensor, add a vector called the bias and then apply a nonlinear function on the resulting values. The shape (and other characteristics) of the tensor and the type of non-linear functions are called the architecture of the network. Herein, the values of the tensor and the bias are denoted by the term “weights”. The weights and, if applicable, the parameters of the non-linear functions, are called the parameters o) of the network. The architecture and the parameters define a “model”. A model trained on a 3D scene can be saved in a memory and transmitted in a data stream and be used to synthetize new views of the 3D scene.

[0033] The 2D-patch based representations of volumetric videos, such as specified by the MIV standard, have proven to be an efficient compression method, in particular compared with previous multi-view encoding techniques such as MV-HEVC. The current version of MIV does not properly handle non-Lambertian surfaces, that is object parts which appearance changes depending on the viewpoint, due to light effects. On the other hand, implicit representations with Neural Radiance Fields (NeRF) offer an attractive solution to render 3D points with viewpoint dependent appearance. Such representations have to be learned and transmitted, however, which creates computational demand at the encoder and bandwidth requirement. On yet another hand, non-Lambertian surfaces typically represents a small proportion of the complete 3D scene, as illustrated in Figure 1. According to the present principles, a volumetric video encoding scheme combining a patch-based representation of the scene, with local NeRFs targeting non-Lambertian scene parts is proposed.

[0034] Figure 2 illustrates a method 20 for encoding a 3D scene according to the present principles. At a step 21 , a 3D point cloud of the 3D scene to encode is obtained. For example, if the 3D scene is captured as a multi-view plus depth representation, the MVD is de-projected into a 3D point cloud. In this case, the input format of the volumetric video content comprises a set of N input views, with colour Ci(u,v) and depth Dj(u,v) image components, associated with intrinsic and extrinsic camera parameters 0. If the 3D scene is a mesh, a 3D point cloud is constructed according to vertices and faces of the mesh. At a step 22, the 3D point cloud is segmented into diffuse and non-diffuse 3D surface patches. At a step 23, the diffuse part is encoded as a video-based patch atlas in a data stream. At a step 24, that may be performed in parallel to step 23, the locations of the non-diffuse parts are encoded as metadata in the data stream. At a step 25, an INR (for instance a NeRF) is learnt for the non-diffuse part of the 3D scene only and a representation of the INR is encoded in the data stream. Figure 5 illustrates segmenting step 22 of Figure 2 when the 3D scene is captured as a MVD image. According to the present principles, step 22 consists in separating the diffuse (i.e. Lambertian) surface parts of the 3D scene from the non-diffuse parts. In Figure 5, an example of a 3D scene 51 with a non-Lambertian surface patch 52 is depicted. The scene is captured by 4 cameras 53 to 56 and the appearance of the non-Lambertian patch differs on each view depending on the camera, contrary to the rest of the scene. An example embodiment for the segmentation of the 3D scene comprises the following steps. First, se- projecting each pixel (u,v) of each input view C, to its corresponding 3D point (X, Y,Z) coordinate, according to its depth value Dj(u,v) and camera parameters 0, making thus a point cloud representing the 3D scene. Second, calculating for each 3D point P=(X, Y,Z) of the scene point cloud, an “appearance variability” feature measuring how much the color of this point varies in the different 2D views which capture it. An example embodiment of this feature is the variance ay (P) of the luminance component Y, which is computed according to equations Eq2, if point P projects in each view C / onto pixel p, with luminance Y(p,).

[0035] Then third, thresholding the “appearance variability” feature yielding a two-class segmentation of the point cloud into low-variability and high-variability points. And, fourth, applying a 3D clustering algorithm to segment the points with “high variability” into connected parts, which correspond to the non-Lambertian surface parts.

[0036] At step 23, the 3D points belonging to the segmented non-Lambertian parts are back- projected into the input 2D views and the corresponding pixels are labelled accordingly “non- Lambertian”. A 2D video-based patch-atlas encoding of the input volumetric content in multiview plus depth input is performed, for example by using a MIV compliant encoder, but excluding the pixels labelled “non-Lambertian”. In another embodiment, the non-Lambertian parts is encoded in the patch atlas, to ensure retro-compatibility (that is using a MIV representation for the entire 3D scene). In this embodiment, corresponding patches are labelled, so as not to be decoded from the data stream when the hybrid codec scheme with INR representation of non-Lambertian parts according to the present principles is used. In the MIV standard, ancillary patches are not intended to be used for view rendering. They are grouped into a dedicated atlas with asme_ancillary_atlas_flag set to 1 in the atlas sequence parameter set. Such an ancillary atlas can be used to transport non-Lambertian patches not intended to be used when a INR representation is combined with MIV to render the non- Lambertian scene parts. At step 24, locations of the pixels labelled “non-Lambertian” are encoded. Some embodiments may rely on ancillary patches to encode these locations. For example, ancillary patches of a specific type could be used to encode a bounding box around the 3D points that have been labeled non-Lambertian. If multiple areas of the 3D scenes are labeled as non- Lambertian, multiple bounding boxes might be used. In a variant, pixels labelled “non- Lambertian” through the back-projection of the non-Lambertian parts have their locations in 2D views encoded by ancillary patches of a specific type. In the retro-compatibility embodiment, the ancillary patches used for retro-compatibility are used to recover the locations of non-Lambertian parts of the scene. In that case, no additional information is necessary.

[0037] At step 25, the INR (e.g. NeRF) encoding reuses the 2D views with the pixels labelled “non-Lambertian” through the back-projection of the non-Lambertian parts. An example embodiment for encoding these pixels is to train the parameters o) of the INR network by minimizing the loss function according to Eq3: where R is the bitrate of the encoded parameters, CLand where \C^L| respectively denotes the non-Lambertian pixels in view Q and the number of such pixels and where A is a trade-off parameter between D and R. The optimization of the weights is typically performed by a machine learning approach such as a batch gradient descent method but any optimization algorithm can be used. Different losses may also be used, for example without the rate, with a different distortion metric or with additional terms to induce some properties in the parameters such as sparsity. Similarly, any variant of a NeRF network can be used according to the present principles. When the parameters have been optimized, they are encoded in the data stream. The parameters (and optionally the network architecture) is encoded for transmission or for later use. This may involve the use of entropy coders. The INR parameters and architecture can be encoded using any off the shelf method. For example, they may be encoded using existing codec such as MPEG-NNC or Open Neural Network Exchange format.

[0038] Figure 6 diagrammatically shows a method 60 for decoding and for rendering a 3D scene comprising non-Lambertian surfaces according to the present principles. At a step 61 , a data stream encoded according method described in relation to Figure 2 is obtained from a memory and / or from a network. At a step 62, 2D video-based patch-atlas patches are decoded. At a step 63, which may be performed in parallel to step 62, locations of the non-Lambertian parts are decoded from the data stream, for example in metadata part of the data stream. At a step 64, weights and parameters of an INR are decoded from the data stream. At a step 65 a 2D view of the 3D scene is generated (i.e. synthetized) according to the present principles. For parts of the view which do not concern the non-Lambertian locations, image patches and corresponding metadata are used to de-project pixels with a color at a depth encoded in patches. For parts corresponding to the non-Lambertian locations, starting and end points (and / or direction) and depth are inputted in the decoded I NR and a color is retrieved for pixels of these locations. In some embodiments, this may involve ancillary patches described in the encoding of the locations of the non-Lambertian parts. In another embodiment, it involves comparing the shape, size and depth of ancillary patches to non-ancillary patches to detect whether these ancillary patches are occluded and, if not, which parts are visible. Color Cw(u, v, d) of non-Lambertian pixels is evaluated as a camera ray going through the 3D scene. The color of a ray with start and end points tsand teand direction d is evaluated according to equations Eq4.

[0039] Figure 3 shows an example architecture of a device 30 which may be configured to implement encoding, decoding and / or rendering methods according to Figure 2 or 6. The device is linked with other devices via their bus 31 and / or via I / O interface 36.

[0040] Device 30 comprises following elements that are linked together by a data and address bus 31 :D

[0041] - a processor 32 (or CPU), which is, for example, a DSP (or Digital Signal Processor);

[0042] - a ROM (or Read Only Memory) 33;

[0043] - a RAM (or Random Access Memory) 34;

[0044] - a storage interface 35;

[0045] - an I / O interface 36 for reception of data to transmit, from an application; and

[0046] - a power supply (not represented in Figure 2), e.g. a battery.

[0047] In accordance with an example, the power supply is external to the device. In each of mentioned memory, the word « register » used in the specification may correspond to area of small capacity (some bits) or to very large area (e.g. a whole program or large amount of received or decoded data). The ROM 33 comprises at least a program and parameters. The ROM 33 may store algorithms and instructions to perform techniques in accordance with present principles. When switched on, the CPU 32 uploads the program in the RAM and executes the corresponding instructions. The RAM 34 comprises, in a register, the program executed by the CPU 32 and uploaded after switch-on of the device 30, input data in a register, intermediate data in different states of the method in a register, and other variables used for the execution of the method in a register.

[0048] The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0049] Device 30 is linked, for example via bus 31 to a set of sensors 37 and to a set of rendering devices 38. Sensors 37 may be, for example, cameras, microphones, temperature sensors, Inertial Measurement Units, GPS, hygrometry sensors, I R or UV light sensors or wind sensors. Rendering devices 38 may be, for example, displays, speakers, vibrators, heat, fan, etc.

[0050] In accordance with examples, the device 30 is configured to implement a method according to the present principles of encoding, decoding and rendering a 3D scene or a volumetric video, and belongs to a set comprising:

[0051] - a mobile device;

[0052] - a communication device;

[0053] - a game device;

[0054] - a tablet (or tablet computer);

[0055] - a laptop;

[0056] - a still picture camera;

[0057] - a video camera.

[0058] Figure 4 shows an example of an embodiment of the syntax of a data stream encoding a 3D scene comprising non-Lambertian parts according to the present principles. Figure 4 shows an example structure 4 of a 3D scene. The structure consists in a container which organizes the stream in independent elements of syntax. The structure may comprise a header part 41 which is a set of data common to every syntax element of the stream. For example, the header part comprises some of metadata about syntax elements, describing the nature and the role of each of them. The structure also comprises a payload comprising an element of syntax 42 and an element of syntax 43. Syntax element 42 comprises data representative of the media content items, comprising at least a patch atlas for the diffuse parts of the 3D scene and weights of an I NR for the non-diffuse parts of the 3D scene. Element of syntax 43 is a part of the payload of the data stream and comprises data encoding the 3D scene description as described according to the present principles.

[0059] The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, Smartphones, tablets, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0060] Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and / or depth information. Examples of such equipment include an encoder, a decoder, a postprocessor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices. As should be clear, the equipment may be mobile and even installed in a mobile vehicle.

[0061] Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions (and / or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD”), an optical disc (such as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory (“RAM”), or a read-only memory (“ROM”). The instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination. Instructions may be found in, for example, an operating system, a separate application, or a combination of the two. A processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.

[0062] As will be evident to one of skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntax-values written by a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

[0063] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Additionally, one of ordinary skill will understand that other structures and processes may be substituted for those disclosed and the resulting implementations will perform at least substantially the same function(s), in at least substantially the same way(s), to achieve at least substantially the same result(s) as the implementations disclosed. Accordingly, these and other implementations are contemplated by this application.

Claims

CLAIMS1. A method for encoding a 3D scene, the method comprising:- segmenting diffuse parts and non-diffuse parts of the 3D scene;- encoding diffuse parts of the 3D scene as a patch atlas in a data stream;- encoding locations of the non-diffuse parts in the data stream;- training an implicit neural network for non-diffuse parts; and- encoding weights of the implicit neural network in the data stream.

2. The method of claim 1 , wherein the 3D scene is obtained as a multi-view plus depth information and wherein the method comprises transforming the 3D scene in a 3D point cloud.

3. The method of claim 1 or 2, wherein the implicit neural network is a Neural Radiance Fields network.

4. The method of one of claims 1 to 3, wherein an architecture of the implicit neural network is encoded in the data stream.

5. A device for encoding a 3D scene, the device comprising a processor configured for:- segmenting diffuse parts and non-diffuse parts of the 3D scene;- encoding diffuse parts of the 3D scene as a patch atlas in a data stream;- encoding locations of the non-diffuse parts in the data stream;- training an implicit neural network for non-diffuse parts; and- encoding weights of the implicit neural network in the data stream.

6. The device of claim 5, wherein the 3D scene is obtained as a multi-view plus depth information and wherein the processor is configured for transforming the 3D scene in a 3D point cloud.

7. The device of claim 5 or 6, wherein the implicit neural network is a Neural Radiance Fields network.

8. The device of one of claims 5 to 7, wherein an architecture of the implicit neural network is encoded in the data stream.

9. A method for decoding a 3D scene, the method comprising:- decoding a patch atlas from a data stream, wherein patches of the patch atlas correspond to diffuse parts of the 3D scene;- decoding locations of non-diffuse parts of the 3D scene from the data stream;- decoding weights of an implicit neural network from the data stream; and- synthetizing a view by using the patches for pixels of the view relative to diffuse parts of the 3D scene and by inputting parameters relative to pixels of non-diffuse parts of the view in the implicit neural network.

10. A device for decoding a 3D scene, the device comprising a processor configured for:- decoding a patch atlas from a data stream, whertein patches of the patch atlas correspond to diffuse parts of the 3D scene;- decoding locations of non-diffuse parts of the 3D scene from the data stream; - decoding weights of an implicit neural network from the data stream; and- synthetizing a view by using the patches for pixels of the view relative to diffuse parts of the 3D scene and by inputting parameters relative to pixels of non-diffuse parts of the view in the implicit neural network.

11. A data stream representative of a 3D scene and comprising: - a patch atlas for diffuse parts of the 3D scene; and- weights of an implicit neural network for non-diffuse parts of the 3D scene.

Images