Scalable 3D Scene Representation Using Neural Field Modeling

Abstract

A method, system, and bitstream syntax for scalable 3D scene representation are described. The general framework presents a two-layer architecture in which a base layer provides a baseline scene representation and an enhancement layer provides enhancement information under various scalability criteria. The enhancement information is coded using trained neural fields. An example system is provided using PSNR criteria and baseline multiplane images (MPI). An example bitstream syntax for metadata information is also provided.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority from U.S. Provisional Application No. 63 / 404,885, filed September 8, 2022, which is incorporated herein by reference in its entirety.

[0002] technology This specification relates generally to images. More particularly, embodiments of the present invention relate to scalable 3D scene representations using a dual-layer approach in which information in the enhancement layer is modeled using neural fields. [Background technology]

[0003] In recent years, there has been growing interest in efficient modeling and representation of 3D scenes. 3D scenes may be used in a variety of applications, including volumetric imaging, virtual reality, or augmented reality. While deep learning techniques have shown promising results in 3D scene representation and reconstruction, not all devices can handle the computational load associated with such approaches. As recognized by the inventors, it is desirable to provide scalable 3D scene representations under various scalability criteria, and thus, improved techniques for 3D scene representation are described herein.

[0004] The term "metadata" herein relates to any auxiliary information transmitted as part of an encoded bitstream that assists a decoder in rendering a decoded image or 3D scene. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, camera parameters, neural network parameters, etc.

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Accordingly, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, it should not be assumed, based on this section, that issues identified with one or more approaches were also recognized in the prior art unless otherwise indicated. [Brief explanation of the drawings]

[0006] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference numerals refer to similar elements and in which:

[0007] [Figure 1A] FIG. 1 illustrates an example of an encoder for scalable 3D scene representation under a general scalability framework, according to one embodiment of the present invention.

[0008] [Figure 1B] FIG. 1 illustrates an example of a decoder for scalable 3D scene representation under a general scalability framework, according to an embodiment of the present invention.

[0009] [Figure 1C] FIG. 1 illustrates an example of an encoder for scalable 3D scene representation under a PSNR criterion, according to an embodiment of the present invention.

[0010] [Figure 1D] FIG. 2 illustrates an example of a decoder for scalable 3D scene representation under a PSNR criterion according to an embodiment of the present invention.

[0011] [Figure 2A]FIG. 1 illustrates an example of an encoder for scalable 3D scene representation and multi-plane image (MPI) representation under a PSNR criterion, according to an embodiment of the present invention.

[0012] [Figure 2B] 1 illustrates an example of a decoder for scalable 3D scene representation and MPI representation under a PSNR criterion according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] Exemplary embodiments relating to scalable 3D scene representation are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. However, it will be apparent that various embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail in order to avoid unnecessarily obscuring, obscuring, or obfuscating embodiments of the present invention.

[0014] overview FIELD OF THE INVENTION In one embodiment, in an encoder, to generate a scalable 3D scene representation, a processor: accessing a first set of images (102) in a first format of a scene; generating a first 3D scene representation (107) of the scene based on the first set of images; accessing a second set of images (104) of the scene in a second format; generating a second 3D scene representation (112) of the scene based on the second set of images, the second 3D representation being better than the first 3D scene representation according to one or more quality criteria; generating output image residuals (122) based on the first 3D scene representation and the second 3D scene representation using an original set of viewing positions and a new set of viewing positions; training a residual neural field network (125) using the output image residual to generate a predicted residual image that approximates the output image residual; transmitting the first 3D scene representation (107) of the scene as a base layer; transmitting information about the trained residual neural field network as an enhancement layer; Execute.

[0015] In an embodiment, in the decoder, to generate the output 3D scene, the processor: receiving a base layer bitstream (107) including a first 3D scene representation (107) of a scene; receiving an enhancement layer bitstream (127) containing information for reconstructing a trained residual neural field network; Given the observer position, generating a first 3D output (132) of the scene based on the first 3D scene representation; generating image residuals (145) using the observer position and the trained residual neural field network; Combining the primary 3D output of the scene with the image residual to generate an enhanced 3D output of the scene. Stages and Execute.

[0016] 3D Scene Representation and Neural Fields There are several 3D scene representation models, including the multi-view plus depth (MVD) method (Reference [6]), multi-plane imaging (MPI) (Reference [5]), and neural radiance field (NeRF) (Reference [2]) representation. Among all of these methods, there are three main evaluation criteria: (1) computational complexity during training and testing, (2) the bit size (bandwidth requirement) of the scene representation and model size, and (3) the 3D scene reconstruction quality. In practice, there are multiple end devices, and applications must address the end application's computational power and the required 3D reconstruction quality while maintaining communication bandwidth. Some devices can only provide a low computational load, but their users can accept lower quality. For high-end devices, it is feasible to add more computation to achieve better quality. To cover a wide range of needs and requirements, embodiments herein propose a dual-layer system with a base layer (BL) to meet a baseline set of requirements and an enhancement layer (EL) to enhance the user experience. The proposed framework can also incorporate various scalability criteria based on peak signal-to-noise ratio (PSNR), dynamic range, color gamut, spatial resolution, temporal frame rate, etc.

[0017] As an example, the MPI representation may be adopted for the base layer due to its ultra-low decoding computation. Such a base layer ensures wide deployment of the encoded bitstream to multiple devices while maintaining baseline quality. However, MPI lacks the ability to provide many specular highlights (non-Lambertian; for example, transparent materials belong to the non-Lambertian family). To provide these specular highlights, the difference between a 3D scene with specular highlights and the MPI can be encoded in the enhancement layer using neural field coding. The base layer can be coded (compressed) using traditional codec techniques such as AVC, HEVC, VVC, and AV1, while the enhancement layer can carry neural network coefficients representing neural fields. As devices have more computing power, they can decode the enhancement and add it on top of the base layer to provide better rendering quality.

[0018] Some other advantages compared to using neural networks of single-layer solutions such as NeRF to directly code 3D scenes include: Neural network solutions such as NeRF, or more generally MLP (Multilayer Perceptron), require scene-specific training, which can be problematic for some applications. In contrast, MPI can use pre-trained networks. If MLP is used only for the residuals, the neural network (NN) can be greatly simplified, and training time should be dramatically reduced.

[0019] For MLPs like NeRF, the model size is approximately 5 megabytes per image scene. Direct transmission of such models for video sequences can be a significant burden on the network. Furthermore, the compressibility of such NN representations is still under investigation. If MLPs are used for the residual layer, the transmission bitrate can be dramatically reduced.

[0020] In certain embodiments, the enhancement layer is outside the coding loop. Therefore, it can provide quality enhancement by simply adding NN residual information to a scene rendered using only the base layer. Furthermore, operations outside the coding loop do not require bit-accurate processing. Platforms can choose either floating-point or fixed-point arithmetic to suit their computing environment.

[0021] In one embodiment, but not limited to, the NN coefficients can be carried within the bitstream or downloaded from external means, for example, using the syntax defined in Reference

[13] (see also Reference [4]).

[0022] Scalability allows for application of a wide variety of quality criteria for generating enhancement layers, including: · PSNR: 3D rendering quality can be improved by adding an enhancement layer residual on top of a lower quality base layer. Dynamic Range: You can improve the dynamic range of a rendered image by adding an enhancement layer residual on top of a standard dynamic range (SDR) 3D scene to produce a high dynamic range (HDR) 3D scene. Color gamut: Color gamut can be improved by adding enhancement layer residuals on top of a 3D scene with a narrower SDR color gamut to produce a 3D scene with a wider color gamut. Spatial resolution: Using an upscaled version of the base layer, enhancement layer information can be added to enhance the details of the final scene at a higher resolution than the base layer resolution. Temporal frame rate: Frame rate interpolation can be applied to the base layer and then neural fields residuals added to produce output at a higher frame rate. · Any combination of the above scalability criteria.

[0023] Neural Fields The term "neural field" refers to a coordinate-based fully connected neural network (Reference [1]). A neural network connects many layers of artificial neurons to learn to nonlinearly map fixed-size inputs to fixed-size outputs. Multilayer perceptron (MLP) neural networks can approximate arbitrary functions through learned parameters. Thus, neural fields can be constructed from multilayer perceptrons (MLPs). For the remainder of this discussion, the terms MLP and neural field will be used interchangeably.

[0024] The end-to-end MLP network has weights {W k} and bias {b k} parameters. These parameters Φ={{W k},{b k This MLP network takes an input x and an output ^y (for convenience, y is sometimes written this way with a ^), where:

number

number

[0025] In some 3D scene representations, such as NeRf (see reference [2]), the input x consists of a spatial position (x,y,z) and a viewing direction (θ,φ), and outputs the volume density (σ) and view-dependent emitted radiance (r,g,b) at those coordinates.

[0026] Position Encoding Neural fields suffer from a loss of frequency details. To address this issue, applying positional encoding is a common solution. In positional encoding, the network input is mapped to a higher dimensional space. This is because the neural network is more biased to learn lower frequency functions. Thus, typical neural networks cannot represent high frequency variations in color and geometry. For neural scene representation, the performance of a neural network is determined by the transformation of a position coordinate p from R to R. L where L is the number of frequencies. A typical mapping γ acting on a coordinate p can be expressed as:

number

[0027] Alternatively, one may apply parametric encoding, i.e., place additional trainable parameters (beyond weights and biases) in an auxiliary data structure such as a grid or tree, and look up and (optionally) interpolate these parameters depending on the input vector. An additional solution to help mitigate high frequency modeling involves using a periodic function as the activation function (see SIREN in reference [7]).

[0028] Forward Mapping In some applications, the output from the MLP is not the directly required result, but requires another mapping. For example, in NeRF, the output from the MLP is (σ, r, g, b) at the coordinate query point (x, y, z, θ, φ). Volume rendering is required by querying all particles along each ray and calculating the final rendered RGB values ​​to construct the projected 2D image. In the proposed embodiment, the output from the neural residual network is already a rendered RGB residual, which can be directly added on top of the rendered new view from the base layer. Next, we will describe various architecture designs.

[0029] A general framework for scalable 3D representation. {t g A set of images capturing the same scene from several different observation positions, denoted as}

number

number

number

[0030] A second 3D scene representation algorithm can then be used that can provide an increased level of quality beyond the basic level. As discussed previously and in more detail below, such increased levels of quality may include improved PSNR, higher bit depth, wider color gamut, etc. Depending on the scalability criteria, the same or a different training dataset may be applied to achieve the desired scalability (parameter Φ s (with

number

number

[0031] In one embodiment, the residual image is a representation of a first base 3D scene representation R b Φb and a second 3D scene representation R s Φs It can be generated by taking the rendering difference between the original observation position {t g} and the new observation position {t n}, it becomes:

number

[0032] In some embodiments, both sets of residual images

number

number

[0033] Unlike NeRF, which requires volume rendering, neural residual does not require forward mapping to obtain a rendered 2D image. The output from the MLP is already in the RGB domain. The main goal of a neural residual network is to take an arbitrary observation location {t} and generate a predicted residual image

number

number

[0034] FIG. 1A illustrates an exemplary processing pipeline for encoding a scalable 3D representation using a generic framework that supports a variety of scalability criteria, such as: a. PSNR scalability b. Dynamic range scalability c. Color Gamut Scalability d. Spatial resolution scalability e. Temporal Frame Rate Scalability

[0035] As shown in Figure 1A, the base layer includes a first unit (105) that generates a first baseline 3D scene representation (107). The input to this unit is a first set of reference input images (102) of the scene in a first format. This 3D scene representation may be further compressed using either traditional image and video coding tools or alternative neural network representation coding tools (not shown).

[0036] To generate the enhancement layer (127), a second set of reference input images (104) of the same scene, but in a second format, is fed to a second unit (110) that generates a second 3D representation (112). For example, depending on the scalability criteria, and without loss of generality, the two reference image sets (102, 104) may represent: a. PSNR scalability: the first image set is the same as the second image set; b. Dynamic range scalability: the first set of images is SDR and the second set of images is HDR; c. Color gamut scalability: the first image set is R.709 and the second image set is R.2020; d. Spatial resolution scalability: the first set of images is 1080p, and the second set of images is 2160p; e. Temporal frame rate scalability: the first set of images is at 24 fps, the second set is at 48 fps. In unit 110, for the second 3D scene representation, a 1:1 bypass may be used if the rendered scene is at the original camera position for which a ground truth image is available. In other words, using the 1:1 bypass, the pose {t g} can take advantage of having a ground truth image.

[0037] As shown in FIG. 1A, in some embodiments, a reformatter (115) may be required when there is a spatial and / or temporal mismatch between the base layer output and the enhancement layer output (107 and 112) (e.g., in cases d) and e) above). For spatial resolution scalability, the reformatter may perform spatial upscaling or downscaling. For temporal frame rate scalability, the reformatter may drop frames or perform inter-frame interpolation. This reformatter is used in both the encoder and decoder (see FIG. 1B). In some embodiments, the reformatter may be used in the enhancement layer, after the second / enhancement layer representation unit (110).

[0038] Given two scene representations, a residual (122) representing their difference is generated by the residual generator (120). All residuals from different views are encoded by a neural field (125). The neural network representation of the residual neural field (125) is compressed and transmitted as the neural network residual bitstream output (127).

[0039] As shown in FIG. 1B, at the decoder side, the decoder receives bitstreams (107) and (127) representing baseline and enhancement information. Note that if the bitstream (107) was compressed before transmission, it should also be appropriately decompressed at the decoder (not shown). Some decoders may simply use the baseline information and ignore any enhancement information. As shown in FIG. 1B, given a user's specified observation position to render a scene, the base layer unit (130) reconstructs a rendered baseline view (132). Depending on the scalability criteria, as described above, if the decoder uses residual information, the baseline view (132) may need to be processed by a reformatter (115). The enhancement layer bitstream (127), along with the user's observer position input, is decoded to render a residual (145) generated using a neural field (140). The output from the reformatter is added to the residual to generate a refined new view (150).

[0040] In some embodiments, it may be desirable to reduce the computational complexity of generating the neural field (125) and / or reduce the neural field model size, for example, by training the neural field 125 using an input residual (122) with a lower spatial resolution. This step of reducing the spatial resolution of the residual may be a separate processing unit (not shown) located after the residual generator (120) but before the residual neural field (125), or may be absorbed by the structure of the residual field (125). In the decoder, a spatial upscaling unit can be added after the neural field 140. Alternatively, because the neural field is a continuous function block, during inference, higher-resolution outputs can be queried even if the neural residual network was trained using lower-resolution grid data. In this way, the residual decoding neural field (140) can absorb the spatial interpolation operation, eliminating the need for a separate spatial / temporal interpolation module.

[0041] Scalability using PSNR criteria Figures 1C and 1D show simplified versions of Figures 1A and 1B when the scalability criterion is PSNR. As shown in Figures 1C and 1D, the reformatter (115) is removed and the encoder is trained based on a single set of reference views and scenes (108).

[0042] In Figure 1D, given a new observation position t, the base 3D scene representation R b Φb The base image (132)

number

number

number

[0043] In some embodiments, the baseline representation may be based on the Multiview and Depth (MVD) format. In such a scenario, the original view image is the encoded image of the input, and the new view image is a Depth Image Based Rendering (DIBR) generated image (see [6]).

[0044] In another embodiment, the baseline representation may be based on the MPI representation (Reference [5]). Note that the term "MPI" is typically used to deal with face-forward scenes. When dealing with 360° video, the term MSI (Multi-sphere Imaging) is used (Reference [8]). However, the concept is the same. Next, additional details about the MPI representation are given, assuming, by way of example and not limitation, a face-forward scene where all camera poses are in the same plane. As an example, only image scenes are considered, but similar concepts may also be applied to video scenes.

[0045] Figure 2A shows an example embodiment of an encoder for scalable 3D scene representation under the PSNR criterion and MPI representation. Given a reference multi-camera captured image (202), the base layer (207) contains an MPI bitstream. For each camera position (camera pose or view) t ∈ {t g}, in order to reduce the load on the decoder, a pre-trained NN is used in unit (205) to

number

number

number

number

number

[0001] . Ta is the depth σd i This is the distance to the front-to-parallel plane relative to the source camera at

[0046] Rendered image I from s to t (s→t) can be calculated as a warped texture and alpha channel:

number

number

number

number

[0047] In one embodiment, the enhancement layer (227) includes a neural network-encoded bitstream that includes NN MLP model parameters (e.g., for model 225). The input to the NN MLP (225) is (x, y, m, n), where (x, y) are pixel locations in the image and (m, n) are pose coordinates. The output of the NN MLP is the RGB value for any given (x, y, m, n). On the encoder side, an MLP needs to be trained for each NN residual scene. The training residual images (220) may be generated (e.g., in unit 215) using the reference view and the new view as follows: First, (parameter Φ s Model R s Φs Apply a second 3D scene representation algorithm (e.g., NeRF 210) using the same dataset to obtain R s Φs Using the original observation position {t g} and the new observation position {t n} to render the image. g} and {t n The rendered images (212) in} are expressed as follows, respectively:

number

number

number

number

number

number

[0048] This residual image generation does not take compression artifacts into account. If compression is taken into account, one more parameter can be added to the NN MLP input function, e.g., (x, y, m, n, Qp), where Qp is the average quantization parameter used to encode the base layer. An additional parameter can also be used to indicate the quality level. When generating training data, MPI-rendered images can be replaced by compressed MPI-rendered images.

[0049] Figure 2B shows an example embodiment of the corresponding decoder. Given a baseline input 207, for any given new pose (m, n), the base layer NN (230) decodes the required multiple (e.g., four) camera pose MPIs and renders a base layer image (232). For the enhancement layer, given the input 227, a residual image (245) is generated using the trained MLP (240). The base layer (232) and enhancement layer (245) are then added to generate a final image (250).

[0050] An example MLP function is shown below using PyTorch code: [Table 1]

[0051] During training, the loss function may be defined using the normalized root mean squared error, although other loss functions may also be applied. [Table 2] As an example, in one embodiment, for the training parameters, the learning rate is 1e-3 and Adam optimization (a permutation optimization algorithm for stochastic gradient descent to train deep learning models) is used.

[0052] In an alternative embodiment, the base layer 3D representation can be a baseline NeRF with a smaller model size, and the enhancement layers can be created via a more advanced (or higher accuracy) NeRF.

[0053] Each NeRF's input,x,consists of a spatial location (x,y,z) and a viewing direction (θ,φ), and outputs the volume density (σ) and view-dependent radiance (r,g,b) at these coordinates.

[0054] For each observation direction, the parameter Φ b A smaller model NeRF with

number

number

number

number

[0055] Neural Residual Network MPI Φrcan be trained using a similar method as mentioned for the MPI base layer.

[0056] In another embodiment, the base 3D representation can be generated using a scene-independent NeRF, such as PixelNeRF (Reference [3]). The training procedure for such a scenario is the same as for the scene-dependent NeRF discussed above.

[0057] Messaging Considerations The proposed approach is outside the encoding loop. In one embodiment, syntax related to system parameters may be communicated using metadata such as the Supplemental Enhancement Information (SEI) used in the MPEG video coding standard. Syntax may also be carried as part of a Video Program Sequence (VPS), Slice Program Sequence (SPS), Picture Program Sequence (PPS), picture header, slice header, etc.

[0058] For example, an SEI message can carry information for the camera, base layer, and enhancement layer. The camera information should include camera parameters and camera position. For the base layer, since the base layer bitstream is a codec bitstream, it is only necessary to signal additional information that the codec bitstream does not carry. Such information may include the base layer representation, such as MPI, MVD, etc. For each input representation, some additional information may be required. For example, for MPI, it may be necessary to communicate the number of cameras, the number of MPI layers, and the tiles assembly. For the enhancement layer, syntax elements need to specify the NN MLP parameters. The NN parameters can be carried by external means or by using a neural network represented by an ISO / IEC 15938-17 bitstream. As in the NNPFC SEI (Reference

[10] ), the enhancement layer information can include input and output formatting information.

[0059] Table 2 provides an example SEI message for communicating syntax parameters related to neural fields used in scalable 3D scene representations. To avoid duplication, only additional information is listed; that is, information not carried in the NNPFC SEI. Descriptor information (e.g., ue(v), u(n), etc.) is defined similarly to the NNPFC SEI. [Table 3] TIFF2025529350000040.tif152170

[0060] For camera viewpoint information, depending on the setup, for 1D setups, the multiview acquisition information SEI and multiview view position SEI in VSEI, HEVC, and AVC can be used. For general setups, the viewport camera parameters SEI and viewport position SEI in Visual Volumetric Video-based Coding (V3C) (Reference

[11] ) and MPEG Immersive video (MIV) (Reference

[12] ) can be used. An example is shown in Table 3 below. Note that the following SEIs, i.e., multiview_acquisition_info(), multiview_view_position(), viewport_camera_parameters(), and viewport_position(), do not need to be included in the camera_viewport_info() SEI message shown in Table 3. They can also be sent outside the camera_viewport_info() SEI message. In another embodiment, one could simply always use the case of a 6DoF (six degrees of freedom) setup, since it includes a 1D setup, but that may require more bits. [Table 4] cv_idc specifies the setup of the camera viewpoint information as shown in the table below. [Table 5] nnr_purpose_idc specifies the purpose of the neural network residual output. It is a 5-bit on-off signaling, where bit 0 signals PSNR improvement, bit 1 signals dynamic range improvement, bit 2 signals color gamut improvement, bit 3 signals spatial resolution improvement, and bit 4 signals temporal frame rate. Each bit can be 0 or 1, for a total of 2 5 There are a combination of streets. nnr_output_pic_width_in_luma_samples specifies the width of the output picture referring to the SEI, in units of luma samples. nnr_output_pic_height_in_luma_samples specifies the height of the output picture, in units of luma samples, that references the SEI. Note: The resolution of the decoded picture of the base layer may be different from the final output resolution using this SEI.

[0061] The following syntax elements are for color spaces where a 3D neural network residual layer system may be applied in a different color space than the coded layer video sequence (CLVS) layer. For example, the bitstream may be in YCbCr color space and the two-layer system may operate on RGB color space. nnr_output_colour_description_present_flag equal to 1 indicates that the unique combination of primaries, transfer characteristics, and matrix coefficients for the output picture resulting from the SEI is specified in the SEI message syntax structure. nnr_output_colour_description_present_flag equal to 0 indicates that the combination of primaries, transfer characteristics, and matrix coefficients for the output picture resulting from the SEI is the same as that indicated in the VUI parameters for CLVS. nnr_colour_primaries has the same semantics as specified in Section 7.3 for the vui_colour_primaries syntax element, except for the following: · nnr_colour_primaries specifies the picture primaries that result from applying the SEI message, not the primaries used for CLVS. When nnr_colour_primaries is not present in the SEI message, the value of nnr_colour_primaries is inferred to be equal to vui_colour_primaries. nnr_transfer_characteristics has the same semantics as specified in Section 7.3 for the vui_transfer_characteristics syntax element, except that: nnr_transfer_characteristics specifies the transfer characteristics of the picture resulting from applying the SEI message, not the transfer characteristics used for CLVS. When nnr_transfer_characteristics is not present in the SEI message, the value of nnr_transfer_characteristics is inferred to be equal to vui_transfer_characteristics. nnr_matrix_coeffs has the same semantics as specified in Section 7.3 for the vui_matrix_coeffs syntax element, except that: nnr_matrix_coeffs specifies the matrix coefficients of the picture resulting from applying the SEI message, not the matrix coefficients used for CLVS. When nnr_matrix_coeffs is not present in the SEI message, the value of nnr_matrix_coeffs is inferred to be equal to vui_matrix_coeffs. · The allowed values ​​for nnr_matrix_coeffs are not constrained by the chroma format of the decoded video picture indicated by the value of ChromaFormatIdc for the VUI parameter semantics. nnr_num_cameras_minus1 plus 1 specifies the number of viewport cameras.

[0062] The following are the semantics for base layer related information: nnr_bl_idc specifies the base layer signal input format for 3D scene representation as follows: [Table 6] nnr_mpi_layer_minus1 plus 1 specifies the number of MPI layers in the MPI representation. nnr_sf_value specifies the scaling factor to use for view rendering, in units of 0.001. Note that depth_representation_info() does not need to be signaled within the proposed SEI; it can be signaled outside the current SEI.

[0063] The following are the semantics used for the enhancement layer NNR: nnr_mode_idc equal to 0 specifies that the neural network residual is determined by external means not specified in this specification. nnr_mode_idc equal to 1 specifies that the neural network residual is the neural network represented by the ISO / IEC 15938-17 bitstream contained in this SEI message. nnr_mode_idc equal to 2 specifies that the neural network residual is the neural network identified by the specified tag, uniform resource identifier (URI) (nnr_uri_tag[i]), and neural network information URI (nnr_uri[i]). The value of nnr_mode_idc shall be in the range 0 to 255, inclusive. Values ​​of nnr_mode_idc greater than 2 are reserved for future specification by ITU-T / ISO / IEC and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​of nnr_mode_idc. nnr_input_dimension_minus3 plus 3 specifies the input signal dimension. [Table 7] nnr_position_encoding_freq[i] specifies the frequency of the input position encoding for the i-th dimension. nnr_normalized_weight specifies the weight value in units of 0.001. nnr_abs_normalized_offset specifies the absolute offset value in units of 0.001. nnr_sign_normalized_offset specifies the sign of the offset value. nnr_normalized_offset=((nnr_sign_normalized_offset>=0)? 1:-1)*nnr_abs_normalized_offset The residual input x to the neural network is scaled to y in 0001: y = (nnr_normalized_weight*x+nnr_normalized_offset)*0.001

[0064] For unbounded scenes, it may also be necessary to specify minimum and maximum values ​​for the pose coordinator and / or depth information (for zoom-in / out effects) to avoid over-rendering. In one example, the reference camera pose can be expressed as (x si ,y si ,z si ), where i is the index of the reference camera. The reference camera poses form a 3D volume in 3D space, and the minimum and maximum values ​​are x_min=min(x si );x_max=max(x si ); y_min=min(y si );y_max=max(y si ); z_min=min(z si );z_max=max(z si ) The new view (x t ,y t ,z t ) is rendered, the camera pose of the new view needs to be bounded within the 3D volume by the minimum and maximum values ​​of the reference pose. x tb =min(x_max,max(x_min,x t )); y tb =min(y_max,max(y_min,y t )); z tb =min(z_max,max(z_min,z t )) In another example, a scaling factor may be further applied to the minimum and maximum values ​​of the reference camera pose to adjust the bounded area.

[0065] References Each of the references (“non-patent literature”) listed herein is incorporated by reference in its entirety. [Non-Patent Document 1] Yiheng Xie et al., "Neural Fields in Visual Computing and Beyond," Eurographics 2022 / CGF, State-of-the-Art Report, Volume 41 (2022), No. 2, 2022. [Non-patent document 2] Ben Mildenhall et al., "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis," ECCV 2020, also in arXiv:2003.08934v2, 5 April 2022. [Non-patent document 3] Alex Yu et al., "pixelNerF: Neural Radiance Fields from One or Few Images," CVPR 2021, also in arXiv:2012.02190v3, 30 May 2021. [Non-patent document 4] Heiner Kirchhoffer et al, "Overview of the Neural Network Compression and Representation (NNR) Standard," IEEE Trans. on Circuits and Systems for Video Technology, Vol. 32, No. 5, May 2022, pp.3203~3216. [Non-Patent Document 5] Richard Tucker and Noah Snavely, "Single-view view synthesis with multiplane images," CVPR 2020. [Non-patent document 6] “Test Model 11 of 3D-HEVC and MV-HEVC,” JCT3V-K1003, Geneva, CH, Feb. 2015. [Non-Patent Document 7] Vincent Sitzmann et al., "Implicit Neural Representations with Periodic Activation Functions," NeurIPS 2020, also in arXiv:2006.09661v1, 17 June, 2020. [Non-patent document 8] Benjamin Attal et al., "MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images," (ECCV) 2020. [Non-Patent Document 9] Ben Mildenhall et al., "Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines", ACM Transactions on Graphics, Vol. 38, No. 4, Article 29, July 2019. [Non-Patent Document 10] Sean McCarthy et al., “Additional SEI messages for VSEI (Draft 2)”, JVET-AA2006v2, JVET 27th meeting, 13-22 July 2022. [Non-Patent Document 11] ISO / IEC 23090-5, Information technology―Coded Representation of Immersive Media-Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC). [Non-Patent Document 12] ISO / IEC 23090-12, Information technology―Coded representation of immersive media-Part 12: MPEG Immersive video. [Non-Patent Document 13] ISO / IEC 15938-17:2022, “MPEG NNC specification: Information technology―Multimedia content description interface-Part 17: Compression of neural networks for multimedia content description and analysis.”

[0066] Embodiments of the present invention may be implemented using a computer system, a system comprised of electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete-time or digital signal processor (DSP), an application-specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to the scalable 3D scene representation as described herein. The computer and / or IC may calculate any of various parameters or values ​​related to the scalable 3D scene representation as described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0067] Certain implementations of the present invention include a computer processor executing software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. may implement methods related to scalable 3D scene representation, such as those described above, by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention may be provided in the form of a program product. A program product may include any non-transitory tangible medium bearing a set of computer-readable signals containing instructions that, when executed by a data processor, cause the data processor to perform the methods of the present invention. A program product according to the present invention may be in any of a wide variety of non-transitory tangible forms. A program product may include physical media, such as magnetic data storage media including floppy disks and hard disk drives, optical data storage media including CD-ROMs and DVDs, ROMs, and electronic data storage media including flash RAM. The computer-readable signals on the program product may optionally be compressed or encrypted. Where a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, the reference to that component (including the reference to "means") should be interpreted as including any component that performs the function of the described component (e.g., is functionally equivalent) as an equivalent of that component, including components that are not structurally equivalent to the disclosed structures that perform that function in the illustrated exemplary embodiments of the present invention.

[0068] Equivalents, Extensions, Substitutions and Others Exemplary embodiments of scalable 3D scene representation are thus described. In the foregoing specification, embodiments of the invention are described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what the invention is and what Applicant intends to be the invention is the set of claims issuing from this application, in the specific form in which such claims are issued, including any subsequent amendments. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Accordingly, no limitations, elements, properties, features, advantages, or attributes not expressly recited in a claim should in any way limit the scope of such claims. Accordingly, the specification and drawings are to be regarded in an illustrative and not a restrictive sense.

Claims

1. 1. A method of generating a scalable 3D scene representation in an encoder, the method comprising: accessing a first set of images (102) of a scene in a first format; generating a first 3D scene representation (107) of the scene based on the first set of images; accessing a second set of images (104) of the scene in a second format; generating a second 3D scene representation (112) of the scene based on the second set of images, the second 3D representation being better than the first 3D scene representation according to one or more quality criteria; generating output image residuals (122) based on the first 3D scene representation and the second 3D scene representation using an original set of viewing positions and a new set of viewing positions; training a residual neural field network (125) using the output image residual to generate a predicted residual image that approximates the output image residual; transmitting the first 3D scene representation (107) of the scene as a base layer; transmitting information about the trained residual neural field network as an enhancement layer; A method comprising:

2. The method of claim 1 , further comprising reformatting an output of the first 3D scene representation or the second 3D scene representation before generating the image residual.

3. The method of claim 2 , wherein the reformatting comprises image upscaling, image downscaling, frame dropping, frame interpolation, or dynamic range / color gamut expansion.

4. The method of claim 1 , wherein the one or more quality criteria include PSNR scalability, dynamic range scalability, color gamut scalability, spatial resolution scalability, and temporal frame rate scalability.

5. The method of claim 1 , wherein the first set of images is the same as the second set of images.

6. The method of claim 1 , wherein the first set of images differs from the second set of images in dynamic range or bit depth, color gamut, spatial resolution, or frame rate.

7. The method of claim 1 , wherein the 3D scene representation can be one of a multi-view plus depth (MVD) representation, a multi-plane imaging (MPI) representation, or a neural radiation field (NeRF) neural network representation.

8. The first 3D scene representation includes a first NeRF model, the second 3D scene representation includes a second NeRF model, the second NeRF model renders a better quality image than the first NeRF model, and generating the output image residual comprises: First Image Residual [Equation 1] Calculate Second Image Residual [Equation 2] Calculating t g represents the original camera pose, and t n represents a new camera pose, [Equation 3] represent the image rendered based on the first NeRF model and the second NeRF model for a spatial location (x, y, z) and a viewing direction (θ, φ), respectively; [Equation 4] represents an image in the first image set, The method of claim 5.

9. During training, the parameters of the residual neural field network are optimized. [Equation 5] is generated by Φ r * represents the optimal set of parameters of the residual field network, [Equation 6] represents the output of the trained residual field network at view t, [Equation 7] denotes the image residual at view t, and D() denotes the loss function to be minimized during training. The method of claim 8.

10. 1. A method of generating an output 3D scene in a decoder, the method comprising: receiving a base layer bitstream (107) including a first 3D scene representation (107) of a scene; receiving an enhancement layer bitstream (127) containing information for reconstructing a trained residual neural field network; Given the observer position, generating a first 3D output (132) of the scene based on the first 3D scene representation; generating an image residual (145) using the observer position and the trained residual neural field network; combining the first 3D output of the scene with the image residual to generate an enhanced 3D output of the scene. Stages and Including, a method.

11. The method of claim 10 , further comprising reformatting the first 3D output of the scene or the image residuals before combining.

12. The method of claim 11 , wherein the reformatting comprises image upscaling, image downscaling, frame dropping, or frame interpolation.

13. The information about the trained residual neural field network is: a quality parameter (nnr_purpose_idc) specifying said one or more quality criteria; Camera viewport information (viewport_camera_info_present_flag parameters), first model parameters (nnr_bl_idc) for said first 3D representation model; the number of hidden layers in the residual neural field (this is related to the NN topology and can be carried in the NN bitstream or externally, depending on nnr_mode_idc); the input position encoding method (nnr_position_encoding_freq[i]), the activation function (which is related to the NN topology and can be carried in the NN bitstream or externally, depending on nnr_mode_idc. This can be ReLU in Table 1 or SIREN in [7], etc.); parameters related to residual rescaling (nnr_normalized_weight / offset and nnr_sign_normalized_offset), The input coordinate parameter descriptor (nnr_input_dimension_minus3), and Descriptors of output parameters (nnr_colour_primaries, nnr_output_pic_width_in_luma_samples, nnr_output_pic_height_in_luma_samples, etc.), Residual rescaling parameters (nnr_normalized_weight, nnr_abs_normalized_offset, nnr_sign_normalized_offset) including one or more of The method of claim 1.

14. The method of claim 13 , wherein the information is transmitted as part of supplemental enhancement information messaging.

15. Training the residual neural field network (125) with the output image residual is at a first spatial resolution, and further comprising: training the residual neural field network using the output image residual at a second spatial resolution lower than the first spatial resolution as an input and the predicted residual image at the first spatial resolution as an output. The method of claim 1.

16. A non-transitory computer-readable storage medium storing computer-executable instructions for performing the method of any one of claims 1 to 15 by one or more processors.

17. 16. An apparatus having a processor configured to perform the method of any one of claims 1 to 15.

Images