Scalable 3D scene representation using neural field modeling

Abstract

A method, system, and bitstream syntax for scalable 3D scene representation are described. The general framework presents a two-layer architecture in which a base layer provides a baseline scene representation and an enhancement layer provides enhancement information under various scalability criteria. The enhancement information is coded using trained neural fields. An example system is provided using PSNR criteria and baseline multiplane images (MPI). An example bitstream syntax for metadata information is also provided.

Description

[Technical Field]

[0001] Cross-references to related applications This application claims the benefit of priority from U.S. Provisional Application No. 63 / 404,885, filed on 8 September 2022, which is incorporated herein by reference in its entirety.

[0002] technology This specification relates in general to images. More specifically, embodiments of the present invention relate to scalable 3D scene representations using a dual-layer approach in which information in the upper layer is modeled using neural fields. [Background technology]

[0003] In recent years, there has been growing interest in the efficient modeling and representation of 3D scenes. 3D scenes can be used in a variety of applications, including volumetric imaging, virtual reality, or augmented reality. Deep learning techniques have shown promising results in 3D scene representation and reconstruction, but not all devices can handle the computational load associated with such approaches. As understood by the inventors, it is desirable to provide scalable 3D scene representations under a variety of scalability criteria, and therefore, improved techniques for 3D scene representation are described herein.

[0004] The term “metadata” as used herein relates to any auxiliary information transmitted as part of an encoded bitstream that assists the decoder in rendering a decoded image or 3D scene. Such metadata may include, but is not limited to, color space or color gamut information, reference display parameters, camera parameters, neural network parameters, and the like.

[0005] The approaches described in this section are approaches that could have been pursued, but not necessarily approaches that were previously conceived or pursued. Therefore, unless otherwise indicated, none of the approaches described in this section should be assumed to qualify as prior art simply by being included in this section. Similarly, any issues identified with respect to one or more approaches should not be assumed, unless otherwise indicated, to have been recognized in any prior art based on this section. [Brief explanation of the drawing]

[0006] Embodiments of the present invention are shown in the accompanying drawings as examples, not as limitations. Similar reference numerals refer to similar elements.

[0007] [Figure 1A] This figure shows an example of an encoder for scalable 3D scene representation under a general scalability framework, according to one embodiment of the present invention.

[0008] [Figure 1B] This figure shows an example of a decoder for scalable 3D scene representation under a general scalability framework, according to one embodiment of the present invention.

[0009] [Figure 1C] This figure shows an example of an encoder for scalable 3D scene representation under PSNR criteria, according to one embodiment of the present invention.

[0010] [Figure 1D] This figure shows an example of a decoder for scalable 3D scene representation under PSNR criteria, according to one embodiment of the present invention.

[0011] [Figure 2A]This figure shows an example of an encoder for scalable 3D scene representation and multiplane image (MPI) representation under PSNR criteria, according to one embodiment of the present invention.

[0012] [Figure 2B] An example of a decoder for scalable 3D scene representation and MPI representation under PSNR criteria, according to one embodiment of the present invention, is shown. [Modes for carrying out the invention]

[0013] Exemplary embodiments relating to scalable 3D scene representation are described herein. The following description includes numerous specific details to provide a full understanding of the various embodiments of the invention for illustrative purposes. However, it will be apparent that various embodiments of the invention can be carried out without these specific details. On the other hand, well-known structures and devices are not described in exhaustive detail to avoid unnecessarily obscuring, ambiguizing, or confusing the embodiments of the invention.

[0014] overview The exemplary embodiments described herein relate to scalable 3D scene representations. In one embodiment, in an encoder, to generate a scalable 3D scene representation, the processor: The step involves accessing a first set of images (102) in a first format for the scene; A step of generating a first 3D scene representation (107) of the scene based on a first set of images; The step involves accessing a second set of images (104) in a second format for the scene; A step of generating a second 3D scene representation (112) of the scene based on a second set of images, wherein the second 3D representation is better than the first 3D scene representation according to one or more quality criteria; A step of generating output image residuals (122) based on the first 3D scene representation and the second 3D scene representation using the original set of observation positions and the new set of observation positions; The steps include: training a residual neural field network (125) using the output image residuals to generate a predicted residual image that approximates the output image residuals; The step of transmitting the first 3D scene representation (107) for the aforementioned scene as the base layer; The step of transmitting information about the trained residual neural field network as an improvement layer. Execute this.

[0015] In one embodiment, in a decoder, to generate an output 3D scene, the processor: The process involves receiving a base layer bitstream (107) containing a first 3D scene representation (107) of the scene; The step involves receiving an improved layer bitstream (127) containing information for reconstructing the trained residual neural field network; Given the observer's position, Based on the first 3D scene representation, a first 3D output (132) of the scene is generated. The observer position and the trained residual neural field network are used to generate image residuals (145). Combine the first 3D output of the scene with the image residuals to generate an improved 3D output of the scene. stages and Execute this.

[0016] 3D scene representation and neural fields There are several 3D scene representation models, including multi-view plus depth (MVD) methods (Reference [6]), multi-plane imaging (MPI) (Reference [5]), and neural radiance field (NeRF) (Reference [2]) representations. Of all these methods, there are three main evaluation criteria: (1) computational complexity during training and testing, (2) bit size (bandwidth requirements) of the scene representation and model size, and (3) 3D scene reconstruction quality. In practice, multiple end devices exist, and applications must address the computational power of the end application and the required 3D reconstruction quality while maintaining communication bandwidth. Some devices can only provide a low computational load, but their users may accept lower quality. For high-end devices, it is feasible to add more computation to achieve better quality. To cover a wide range of needs and requirements, embodiments of this specification propose a dual-layer system having a base layer (BL) to meet a baseline set of requirements and an enhancement layer (EL) to improve the user experience. The proposed framework can also incorporate a variety of scalability criteria based on peak signal-to-noise ratio (PSNR), dynamic range, color gamut, spatial resolution, and temporal frame rate.

[0017] As an example, the base layer may employ an MPI representation due to its extremely low decoding computation. Such a base layer ensures widespread deployment of the encoded bitstream to multiple devices while maintaining baseline quality. However, MPI lacks the ability to provide many specular highlights (non-Lambertian; for example, transparent materials belong to the non-Lambertian family). To provide those specular highlights, the difference between a 3D scene with specular highlights and MPI can be encoded in the enhancement layer using neural field coding. The base layer can be coded (compressed) using conventional codec techniques such as AVC, HEVC, VVC, and AV1, while the enhancement layer can carry neural network coefficients representing the neural field. As devices have more computing power, this enhancement can be decoded and added on top of the base layer to provide better rendering quality.

[0018] Some other advantages of using MPI compared to using a single-layer neural network solution such as NeRF for directly coding 3D scenes include: NeRF, or more generally, a neural network solution such as MLP (Multilayer Perceptron), requires scene-specific training, which can be problematic for some applications. In contrast, MPI can use a pre-trained network. If MLP is used only for residuals, the neural network (NN) can be greatly simplified, and training time should be dramatically reduced.

[0019] For MLPs like NeRF, the model size is approximately 5 megabytes per image scene. Direct transmission of such models for video sequences can be a significant burden on the network. Furthermore, the compressibility of such NN representations is still under investigation. When MLPs are used in the residual layer, the transmission bitrate can be dramatically reduced.

[0020] In certain embodiments, the enhancement layer resides outside the coding loop. Therefore, quality improvements can be provided simply by adding NN residual information to a scene rendered using only the base layer. Furthermore, operations outside the coding loop do not require bit-precise processing. The platform can choose either floating-point or fixed-point arithmetic to suit its computing environment.

[0021] In some embodiments, but not limited to them, the NN coefficients may be transported within the bitstream or downloaded from external means using, for example, the syntax defined in Reference

[13] (see also Reference [4]).

[0022] Scalability allows for the application of a wide range of diverse quality standards for generating improved layers, including: • PSNR: 3D rendering quality can be improved by adding improved layer residuals on top of a lower quality base layer. • Dynamic Range: By adding enhanced residuals to a standard dynamic range (SDR) 3D scene, the dynamic range of the rendered image can be improved to generate a high dynamic range (HDR) 3D scene. • Color Gamut: The color gamut can be improved by adding enhanced residuals to a 3D scene with a narrower SDR color gamut, thereby generating a 3D scene with a wider color gamut. • Spatial Resolution: By using an upscaled version of the base layer and adding information from the upgraded layer, it is possible to enhance the detail of the final scene at a resolution higher than the base layer resolution. • Temporal frame rate: By applying frame rate interpolation to the base layer and then adding neural field residuals, it is possible to generate output at a higher frame rate. Any combination of the above scalability criteria.

[0023] Neural Field The term “neural field” refers to a coordinate-based fully connected neural network (reference [1]). A neural network connects many layers of artificial neurons to learn to non-linearly map fixed-size inputs to fixed-size outputs. A multilayer perceptron (MLP) neural network can approximate any function through its learned parameters. Thus, a neural field can be constructed from a multilayer perceptron (MLP). For the remainder of this discussion, the terms MLP and neural field will be used interchangeably.

[0024] An end-to-end MLP network uses weights {W k} and bias {b k It consists of K layers of parameters Φ = {{W k},{b k This is shown as}}. This MLP network takes input x and output ^y (for convenience, y with a ^ is sometimes written this way), where,

number

number

[0025] In some 3D scene representations, such as NeRf (see reference [2]), the input x consists of a spatial position (x, y, z) and an observation direction (θ, φ), and outputs the volume density (σ) and view-dependent emitted radiance (r, g, b) at those coordinates.

[0026] Location encoding Neural fields suffer from a loss of frequency detail. A common solution to address this problem is to apply position encoding. In position encoding, the network input is mapped to a higher-dimensional space. This is because the neural network is more biased to learn lower frequency functions. Therefore, a typical neural network cannot represent high-frequency variations in color and geometry. For neural scene representation, the performance of the neural network is measured by mapping position coordinates p from R to R. L This is significantly improved by mapping to . Here, L is the number of frequencies. A typical mapping γ acting on coordinate p can be expressed as follows:

number

[0027] Alternatively, parametric encoding may be applied, that is, additional trainable parameters (other than weights and biases) may be placed in an auxiliary data structure such as a grid or tree, and these parameters may be looked up and (optionally) interpolated depending on the input vector. An additional solution to help mitigate high-frequency modeling involves using a periodic function as the activation function (see SIREN in reference [7]).

[0028] Forward mapping In some applications, the output from an MLP requires a different mapping than the directly desired result. For example, in NeRF, the output from the MLP is (σ,r,g,b) at the coordinate query point (x,y,z,θ,φ). Volume rendering is required to construct a projected 2D image by querying all particles along each ray and calculating the final rendered RGB values. In the proposed embodiment, the output from the neural residual network is already the rendered RGB residual. The RGB residual can be added directly onto the rendered new view from the base layer. Next, various architecture designs will be described.

[0029] A general framework for scalable 3D representation {t g} A set of images capturing the same scene from several different observation positions represented as

Number

Number

Number

[0030] Next, a second 3D scene representation algorithm can be used to provide an improvement in the quality level beyond the basic level. As discussed previously and in more detail later, such an improvement in the quality level can include an improvement in PSNR, a higher bit depth, a wider color gamut, etc. Depending on the scalability criteria, the same training dataset or a different training dataset can be applied to train a model (with parameters Φ s ),

number

number

[0031] In one embodiment, the residual image is a first base 3D scene representation R b Φb and the second 3D scene representation R s Φs It can be generated by taking the rendering difference with the original observation position {t g} and new observation position {t n In}, it will be as follows:

number

[0032] In one embodiment, both sets of residual images

number

number

[0033] Unlike NeRF, which requires volume rendering, neural residuals do not require forward mapping to obtain a rendered 2D image. The output from MLP is already in the RGB domain. The main goal of the neural residual network is to take an arbitrary observation position {t} and predict the residual image.

number

number

[0034] Figure 1A shows an exemplary processing pipeline for encoding a scalable 3D representation using a general-purpose framework that supports a variety of scalability criteria, as follows: a. PSNR scalability b. Dynamic range and scalability c. Color Gamut Scalability d. Spatial resolution scalability e. Temporal frame rate scalability

[0035] As shown in Figure 1A, the base layer includes a first unit (105) that generates a first baseline 3D scene representation (107). The input to this unit is a first set of reference input images (102) about the scene in a first format. This 3D scene representation may be further compressed using either traditional image and video coding tools or alternative NN representation coding tools (not shown).

[0036] To generate the improved layer (127), a second set of reference input images (104) of the same scene, but in a second format, is supplied to a second unit (110) that generates a second 3D representation (112). For example, depending on the scalability criteria and without loss of generality, the two sets of reference images (102, 104) could represent the following: a. PSNR scalability: The first image set is identical to the second image set; b. Dynamic range scalability: The first image set is SDR, and the second image set is HDR; c. Color gamut scalability: The first image set is R.709, and the second image set is R.2020; d. Spatial resolution scalability: The first image set is 1080p, and the second image set is 2160p; e. Temporal frame rate scalability: The first image set is 24fps, and the second image set is 48fps. In unit 110, for the second 3D scene representation, a 1:1 bypass may be used if the rendered scene is at the original camera position where the ground truth image is available. In other words, by using the 1:1 bypass, instead of using the second model (110), where the output may still contain artifacts / distortions, the ground truth image is used directly to generate the residuals, thereby correcting the pose {t g This allows us to take advantage of having a correct image in}.

[0037] As shown in Figure 1A, in some embodiments, a reformatter (115) may be required when there is a spatial and / or temporal mismatch between the base layer output and the up-layer output (107 and 112) (for example, in cases d) and e) above). For spatial resolution scalability, the reformatter may perform spatial upscaling or downscaling. For temporal frame rate scalability, the reformatter may drop frames or perform inter-frame interpolation. This reformatter is used in both the encoder and decoder (see Figure 1B). In some embodiments, the reformatter may be used in the up-layer, after the second / up-layer representation unit (110).

[0038] Given two scene representations, a residual generator (120) generates residuals (122) representing the difference between them. All residuals from the different views are encoded by a neural field (125). The neural network representation (125) of the residual neural field is compressed and transmitted as a neural network residual bitstream output (127).

[0039] As shown in Figure 1B, on the decoder side, the decoder receives bitstreams (107) and (127) representing baseline and enhancement information. Note that if bitstream (107) is compressed before transmission, it should also be appropriately decompressed in the decoder (not shown). Some decoders may use only baseline information and ignore any enhancement information. As shown in Figure 1B, given a user-specified observation position to render the scene, the base layer unit (130) reconstructs the rendered baseline view (132). Depending on the scalability criteria, if the decoder uses residual information as described above, the baseline view (132) may need to be processed by a reformatter (115). The enhancement layer bitstream (127) is decoded along with the user's observer position input to render the residual (145) generated using the neural field (140). The output from the reformatter is added to the residual to produce a refined new view (150).

[0040] In one embodiment, it may be desirable to reduce the computational complexity of generating the neural field (125) and / or the size of the neural field model by training the neural field 125 with input residuals (122) with lower spatial resolution. This step of reducing the spatial resolution of the residuals may be performed by a separate processing unit (not shown) located after the residual generator (120) and before the residual neural field (125), or it may be absorbed by the structure of the residual field (125). In the decoder, a spatial upscaling unit may be added after the neural field 140. Alternatively, since the neural field is a continuous function block, during inference, a higher resolution output can be queried even if the neural residual network was trained with lower resolution grid data. Thus, the residual decoded neural field (140) can absorb the spatial interpolation operation, eliminating the need for a separate spatial / time interpolation module.

[0041] Scalability using PSNR standards Figures 1C and 1D show simplified versions of Figures 1A and 1B when the scalability criterion is PSNR. As shown in Figures 1C and 1D, the reformatter (115) is removed, and the encoder is trained based on a single set of reference views and scenes (108).

[0042] In Figure 1D, given a new observation position t, the basic 3D scene representation R b Φb The basic image (132)

number

number

number

[0043] In one embodiment, the baseline representation may be based on a multiview and depth (MVD) format. In such a scenario, the original view image is the encoded input image, and the new view image is a depth image based rendering (DIBR) generated image (reference [6]).

[0044] In another embodiment, the baseline representation may be based on the MPI representation (Reference [5]). Note that the term "MPI" is typically used when dealing with face-forward scenes. When dealing with 360° video, the term MSI (Multi-sphere Imaging) is used (Reference [8]). However, the concept is the same. Additional details about the MPI representation are then given, for example, assuming a face-forward scene where all camera poses are in the same plane, though not limited to this example. For example, only image scenes are considered, but similar concepts can be applied to video scenes as well.

[0045] Figure 2A shows an exemplary embodiment of an encoder for scalable 3D scene representation under PSNR criteria and MPI representation. Given a reference multi-camera captured image (202), the base layer (207) contains an MPI bitstream. Each camera position (camera pose or view) t ∈ {t g Regarding}, in order to reduce the burden on the decoder, in unit (205), a pre-trained NN processes the image

number

number

number

number

number

[0001] TTherefore, a is the depth σd i This is the distance to the front-to-parallel plane relative to the source camera.

[0046] Rendered image I from s to t (s→t) This can be calculated as a warped texture and alpha channel:

number

number

number

number

[0047] In one embodiment, the improvement layer (227) includes a neural network encoded bitstream containing NN MLP model parameters (for example, for model 225). The input to NN MLP(225) is (x,y,m,n), where (x,y) is the pixel position of the image and (m,n) is the pose coordinate. The output of NN MLP is the RGB values ​​for any given (x,y,m,n). On the encoder side, the MLP needs to be trained for each NN residual scene. The training residual image (220) can be generated using a reference view and a new view (for example, in unit 215) as follows: First, (parameter Φ) s Model R (with) s Φs To obtain the result, apply a second 3D scene representation algorithm (e.g., NeRF 210) using the same dataset. s Φs Using the original observation position {t g} and new observation position {t n Images can be rendered using {t g} and {t n The rendered images (212) in} are represented as follows:

number

number

number

number

number

number

[0048] This residual image generation does not take compression artifacts into account. If compression is considered, another parameter can be added to the NN MLP input function, for example, (x, y, m, n, Qp), where Qp is, for example, the mean quantization parameter used to encode the basic layer. Additional parameters may also be used to indicate quality levels. When generating training data, MPI-rendered images can be replaced with compressed MPI-rendered images.

[0049] Figure 2B shows an exemplary embodiment of the corresponding decoder. Given a baseline input 207, for any given novel pose (m,n), the base layer NN(230) decodes the required multiple (e.g., four) camera pose MPIs and renders the base layer image(232). For the improvement layer, given input 227, a residual image(245) is generated using the trained MLP(240). The base layer(232) and improvement layer(245) are then added to generate the final image(250).

[0050] An example of an MLP function is shown below using PyTorch code. [Table 1]

[0051] During training, the loss function can be defined using the normalized root mean square error. Other loss functions may also be applied. [Table 2] As an example, in one embodiment, the learning rate for the training parameters is 1e-3, and Adam optimization (a permutation optimization algorithm for stochastic gradient descent to train a deep learning model) is used.

[0052] In an alternative embodiment, the base layer 3D representation may be a baseline NeRF with a smaller model size, and the improved layers may be created via a more advanced (or higher-precision) NeRF.

[0053] Each NeRF input x consists of a spatial position (x, y, z) and an observation direction (θ, φ), and outputs the volume density (σ) and view-dependent radiance (r, g, b) at these coordinates.

[0054] For each observation direction, parameter Φ b A smaller model, NeRF, is used to generate the rendered base image.

number

number

number

number

[0055] Neural Residual Network MPI ΦrIt can be trained using a method similar to that mentioned for the MPI base layer.

[0056] In another embodiment, the basic 3D representation may be generated using a scene-independent NeRF such as PixelNeRF (reference [3]). The training procedure for such a scenario is the same as that for the scene-dependent NeRF discussed earlier.

[0057] Messaging considerations The proposed approach lies outside the encoding loop. In one embodiment, syntax related to system parameters may be communicated using metadata such as Supplemental Enhancement Information (SEI) used in the MPEG video encoding standard. Syntax may also be carried as part of a video program sequence (VPS), slice program sequence (SPS), picture program sequence (PPS), picture header, slice header, etc.

[0058] For example, an SEI message can carry information for the camera, base layer, and advanced layer. Camera information should include camera parameters and camera position. For the base layer, since the base layer bitstream is a codec bitstream, only additional information not carried by the codec bitstream needs to be signaled. Such information may include base layer representations such as MPI and MVD. For each input representation, some additional information may be required. For example, for MPI, the number of cameras, the number of MPI layers, and the tile assembly may need to be communicated. For the advanced layer, the syntax element needs to specify NN MLP parameters. NN parameters can be carried by external means or by using a neural network represented by an ISO / IEC 15938-17 bitstream. As with NNPFC SEI (reference

[10] ), advanced layer information may include input and output formatting information.

[0059] Table 2 provides exemplary SEI messages for communicating syntax parameters related to neural fields used in scalable 3D scene representations. To avoid duplication, only additional information is listed; i.e., information not carried in NNPFC SEI. Descriptor information (e.g., ue(v), u(n), etc.) is defined similarly to NNPFC SEI. [Table 3] TIFF0007855795000039.tif152170

[0060] For camera viewpoint information, depending on the setup, for 1D setups, the multiview acquisition information SEI and multiview view position SEI can be used in VSEI, HEVC, and AVC. For general setups, the viewport camera parameter SEI and viewport position SEI can be used in Visual Volumetric Video-based Coding (V3C) (Reference

[11] ) and MPEG Immersive Video (MIV) (Reference

[12] ). An example is shown in Table 3 below. Note that the following SEIs, namely multiview_acquisition_info(), multiview_view_position(), viewport_camera_parameters(), and viewport_position(), do not need to be included in the camera_viewport_info() SEI message shown in Table 3. They can also be sent outside of the camera_viewport_info() SEI message. In another embodiment, a 6DoF (6 degrees of freedom) setup includes a 1D setup, so one could simply always use the 6DoF setup case, although that might require more bits. [Table 4] cv_idc specifies the setup of camera viewpoint information as shown in the table below. [Table 5] nnr_purpose_idc specifies the purpose of the neural network residual output. It is a 5-bit on-off signaling, where the 0th bit signals PSNR image quality improvement, the 1st bit signals dynamic range improvement, the 2nd bit signals color gamut improvement, the 3rd bit signals spatial resolution improvement, and the 4th bit signals temporal frame rate. Each bit is either 0 or 1, totaling 2 5 There are combinations of streets. nnr_output_pic_width_in_luma_samples specifies the width of the output picture, which references SEI, in units of luma samples. nnr_output_pic_height_in_luma_samples specifies the height of the output picture, which references SEI, in units of luma samples. Note: The resolution of the decoded picture in the base layer may differ from the final output resolution using this SEI.

[0061] The following syntax elements are for color spaces in which a 3D neural network residual layer system can be applied, in a different color space than the coded layer video sequence (CLVS) layer. For example, if the bitstream is in the YCbCr color space, the dual-layer system can operate in the RGB color space. A value of 1 for nnr_output_colour_description_present_flag indicates that a unique combination of primary colors, transfer characteristics, and matrix coefficients for the output picture resulting from SEI is specified in the SEI message syntax structure. A value of 0 for nnr_output_colour_description_present_flag indicates that the combination of primary colors, transfer characteristics, and matrix coefficients for the output picture resulting from SEI is the same as that shown in the VUI parameters for CLVS. nnr_colour_primaries has the same semantics as specified in Section 7.3 for the vui_colour_primaries syntax element, except as follows: nnr_colour_primaries specifies the picture's primary colors, which result from applying SEI messages, rather than the primary colors used for CLVS. When nnr_colour_primaries is not present in the SEI message, its value is presumed to be equal to vui_colour_primaries. nnr_transfer_characteristics has the same semantics as specified in Section 7.3 for the vui_transfer_characteristics syntax element, except for the following: nnr_transfer_characteristics specifies the picture transfer characteristics resulting from applying SEI messages, rather than the transfer characteristics used for CLVS. If nnr_transfer_characteristics is not present in the SEI message, its value is presumed to be equal to vui_transfer_characteristics. nnr_matrix_coeffs has the same semantics as specified in Section 7.3 for the vui_matrix_coeffs syntax element, except for the following: nnr_matrix_coeffs specifies the matrix coefficients of the picture resulting from applying SEI messages, rather than the matrix coefficients used for CLVS. If nnr_matrix_coeffs is not present in the SEI message, its value is presumed to be equal to vui_matrix_coeffs. The acceptable values ​​for nnr_matrix_coeffs are not constrained by the chroma format of the decoded video picture, as indicated by the value of ChromaFormatIdc for the semantics of the VUI parameter. Adding 1 to nnr_num_cameras_minus1 specifies the number of viewport cameras.

[0062] The following is the semantics for information related to the basic layer. nnr_bl_idc specifies the basic layer signal input format for 3D scene representation as follows: [Table 6] Adding 1 to nnr_mpi_layer_minus1 specifies the number of MPI layers in the MPI representation. nnr_sf_value specifies the scaling factor used for view rendering in units of 0.001. Please note that `depth_representation_info()` does not need to be signaled within the proposed SEI. It can be signaled outside of the current SEI.

[0063] The following is the semantics used for improved layer NNR. A value of nnr_mode_idc equal to 0 indicates that the neural network residual is determined by an external means not specified herein. A value of nnr_mode_idc equal to 1 indicates that the neural network residual is the neural network represented by the ISO / IEC 15938-17 bitstream included in this SEI message. A value of nnr_mode_idc equal to 2 indicates that the neural network residual is the neural network identified by the specified tag, uniform resource identifier (URI) (nnr_uri_tag[i]) and neural network information URI (nnr_uri[i]). The value of nnr_mode_idc shall be within the range of 0 to 255, including both extreme values. A value of nnr_mode_idc greater than 2 is reserved for future specifications by ITU-T|ISO / IEC and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​for nnr_mode_idc. Adding 3 to nnr_input_dimension_minus3 specifies the input signal dimension. [Table 7] nnr_position_encoding_freq[i] specifies the frequency of the input position encoding for the i-th dimension. nnr_normalized_weight specifies the weight value in units of 0.001. nnr_abs_normalized_offset specifies the absolute offset value in units of 0.001. nnr_sign_normalized_offset specifies the sign of the offset value. nnr_normalized_offset=((nnr_sign_normalized_offset>=0)? 1:-1)*nnr_abs_normalized_offset The residual input x to the neural network is scaled to y within

[0001] : y = (nnr_normalized_weight * x + nnr_normalized_offset) * 0.001 (Residual rescaling)

[0064] For an unbounded scene, it may also be necessary to specify the minimum and maximum values of the pose coordinator and / or depth information (for the zoom-in / zoom-out effect) to avoid over-rendering. In one example, the reference camera pose is represented as (x si , y si , z si ) in the camera-world coordinate system, where i is the index of the reference camera. The reference camera pose forms a 3D volume in 3D space, and the minimum and maximum values are x_min = min(x si ); x_max = max(x si ); y_min = min(y si ); y_max = max(y si ); z_min = min(z si ); z_max = max(z si ) which can be calculated by. When a new view (x t , y t , z t ) is rendered, the camera pose of the new view needs to be bounded within the 3D volume by the minimum and maximum values of the reference pose. That is, x tb = min(x_max, max(x_min, x t )); y tb = min(y_max, max(y_min, y t ));[[ID=..]] z tb = min(z_max, max(z_min, z t )) In another example, scaling factors may be further applied to the minimum and maximum values ​​of the reference camera pose to adjust the defined area.

[0065] References Each of the references listed herein ["non-patent literature"] is incorporated in its entirety by reference. [Non-Patent Document 1] Yiheng Xie et al., "Neural Fields in Visual Computing and Beyond," Eurographics 2022 / CGF, State-of-the-Art Report, Volume 41 (2022), No. 2, 2022. [Non-Patent Document 2] Ben Mildenhall et al., "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis," ECCV 2020, also in arXiv:2003.08934v2, 5 April 2022. [Non-Patent Document 3] Alex Yu et al., "pixelNerF: Neural Radiance Fields from One or Few Images," CVPR 2021, also in arXiv:2012.02190v3, 30 May 2021. [Non-Patent Document 4] Heiner Kirchhoffer et al, "Overview of the Neural Network Compression and Representation (NNR) Standard," IEEE Trans. on Circuits and Systems for Video Technology, Vol. 32, No. 5, May 2022, pp.3203~3216. [Non-Patent Document 5] Richard Tucker and Noah Snavely, "Single-view view synthesis with multiplane images," CVPR 2020. [Non-Patent Document 6] “Test Model 11 of 3D-HEVC and MV-HEVC,” JCT3V-K1003, Geneva, CH, Feb. 2015. [Non-Patent Document 7] Vincent Sitzmann et al., "Implicit Neural Representations with Periodic Activation Functions," NeurIPS 2020, also in arXiv:2006.09661v1, 17 June, 2020. [Non-Patent Document 8] Benjamin Attal et al., "MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images," (ECCV) 2020. [Non-Patent Document 9] Ben Mildenhall et al., "Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines", ACM Transactions on Graphics, Vol. 38, No. 4, Article 29, July 2019. [Non-Patent Document 10] Sean McCarthy et al., “Additional SEI messages for VSEI (Draft 2)”, JVET-AA2006v2, JVET 27th meeting, 13-22 July 2022. [Non-Patent Document 11] ISO / IEC 23090-5, Information technology―Coded Representation of Immersive Media-Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC). [Non-Patent Document 12] ISO / IEC 23090-12, Information technology―Coded representation of immersive media-Part 12: MPEG Immersive video. [Non-Patent Document 13] ISO / IEC 15938-17:2022, “MPEG NNC specification: Information technology―Multimedia content description interface-Part 17: Compression of neural networks for multimedia content description and analysis.”

[0066] Embodiments of the present invention may be implemented using computer systems, systems composed of electronic circuits and components, integrated circuit (IC) devices such as microcontrollers, field-programmable gate arrays (FPGAs), or other configurable or programmable logic devices (PLDs), discrete-time or digital signal processors (DSPs), application-specific ICs (ASICs), and / or apparatus including one or more such systems, devices, or components. Computers and / or ICs can execute, control, or implement instructions relating to scalable 3D scene representations as described herein. Computers and / or ICs can calculate any of the various parameters or values ​​relating to scalable 3D scene representations as described herein. Embodiments of images and videos may be implemented in hardware, software, firmware, and various combinations thereof.

[0067] Certain implementations of the present invention include a computer processor that executes software instructions causing the processor to perform the method of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc., may implement the method relating to the scalable 3D scene representation described above by executing software instructions in program memory accessible to the processor. Embodiments of the present invention may be provided in the form of a program product. The program product may include any non-temporary tangible medium that, when executed by a data processor, carries a set of computer-readable signals including instructions causing the data processor to perform the method of the present invention. The program product according to the present invention may be any of a wide variety of non-temporary tangible forms. The program product may include physical media such as magnetic data storage media including floppy disks and hard disk drives, optical data storage media including CD-ROMs and DVDs, and electronic data storage media including ROMs and flash RAM. The computer-readable signals on the program product may optionally be compressed or encrypted. Where components (e.g., software modules, processors, assemblies, devices, circuits, etc.) are referred to above, unless otherwise indicated, references to such components (including references to “means”) should be interpreted as including any components that perform the function of the described component (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structure performing the function of the illustrated exemplary embodiment of the Invention.

[0068] Equivalents, extensions, substitutes and others Exemplary embodiments relating to scalable 3D scene representation are described herein. In the above specification, embodiments of the invention are described with reference to a number of specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what the invention is and what the applicant intends to be the invention is the set of claims issued herein, including any subsequent amendments, in the particular form in which such claims are issued. Any definitions expressly provided herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Thus, no limitations, elements, characteristics, features, advantages, or attributes not expressly provided in the claims shall in any way limit the scope of such claims. Thus, this specification and the drawings should be considered illustrative and not restrictive.

Claims

1. An encoder provides a method for generating a scalable 3D scene representation, the method being: The step involves accessing a first set of images (102) in a first format for the scene; A step of generating a first 3D scene representation (107) of the scene based on the first set of images; The step of accessing a second set of images (104) in a second format for the aforementioned scene; A step of generating a second 3D scene representation (112) of the scene based on the second set of images, wherein the second 3D representation is better than the first 3D scene representation according to one or more quality criteria; The steps include: generating output image residuals (122) based on the first 3D scene representation and the second 3D scene representation using the original set of observation positions and the new set of observation positions; The process involves training a residual neural field network (125) using the output image residuals to generate a predicted residual image that approximates the output image residuals; The step of transmitting the first 3D scene representation (107) for the aforementioned scene as the base layer; The step of transmitting information about the trained residual neural field network as an improvement layer. Methods that include...

2. The method according to claim 1, further comprising reformatting the output of the first 3D scene representation or the second 3D scene representation before generating the output image residual.

3. The method according to claim 2, wherein reformatting includes image upscaling, image downscaling, frame dropping, frame interpolation, or dynamic range / color gamut expansion.

4. The method according to claim 1, wherein one or more of the quality criteria include PSNR scalability, dynamic range scalability, color gamut scalability, spatial resolution scalability, and temporal frame rate scalability.

5. The method according to claim 1, wherein the first set of images is identical to the second set of images.

6. The method according to claim 1, wherein the first image set differs from the second image set in terms of dynamic range or bit depth, color gamut, spatial resolution or frame rate.

7. The method according to claim 1, wherein the 3D scene representation may be one of the following: multiview plus depth (MVD) representation, multiplane imaging (MPI) representation, or neural radiation field (NeRF) neural network representation.

8. The first 3D scene representation includes a first NeRF model, the second 3D scene representation includes a second NeRF model, the second NeRF model renders images of better quality than the first NeRF model, and generates the output image residuals: Image residuals of the first image [Math 1] Calculate, Second image residual [Math 2] This includes calculating, t g This represents the original camera pose, t n This represents a new camera pose, [Math 3] These represent images rendered based on the first NeRF model and the second NeRF model, respectively, for spatial position (x, y, z) and observation direction (θ, φ). [Math 4] represents an image in the aforementioned first image set, The method according to claim 5.

9. During training, the parameters of the residual neural field network are optimized. [Math 5] Generated by, Φ r * This represents the optimal set of parameters for the residual neural field network, [Math 6] This represents the output of the trained residual neural field network in view t. [Number 7] D() represents the image residual at view t, and D() represents the loss function that should be minimized during training. The method according to claim 8.

10. A decoder, a method for generating an output 3D scene, wherein the method is: The process involves receiving a base layer bitstream (107) containing a first 3D scene representation (107) of the scene; The step involves receiving an improved layer bitstream (127) containing information for reconstructing the trained residual neural field network; Given the observer's position, Based on the first 3D scene representation, a first 3D output (132) of the scene is generated. Using the observer position and the trained residual neural field network, image residuals (145) are generated. The first 3D output of the scene and the image residual are combined to generate an improved 3D output of the scene. stages and Includes, method.

11. The method according to claim 10, further comprising the step of reformatting the first 3D output or the image residual of the scene before combining them.

12. The method according to claim 11, wherein reformatting includes image upscaling, image downscaling, frame dropping, or frame interpolation.

13. Information about the aforementioned trained residual neural field network: A quality parameter (nnr_purpose_idc) that specifies one or more of the aforementioned quality criteria, Camera viewport information (viewport_camera_info_present_flag parameters), The first model parameter (nnr_bl_idc) for the first 3D representation model, The number of hidden layers in the residual neural field (which is related to the NN topology and can be carried in an NNR bitstream or external means based on nnr_mode_idc), Input position encoding method (nnr_position_encoding_freq[i]), Activation function (which is related to the NN topology and can be carried in an NNR bitstream or external means based on nnr_mode_idc), Parameters related to residual rescaling (nnr_normalized_weight / nnr_abs_normalized_offset and nnr_sign_normalized_offset), The descriptor for the input coordinate parameter (nnr_input_dimension_minus3), and Output parameter descriptors (nnr_colour_primaries, nnr_output_pic_width_in_luma_samples, nnr_output_pic_height_in_luma_samples) Including one or more of the following: The method according to claim 1.

14. The method according to claim 13, wherein the aforementioned information is transmitted as part of supplemental enhancement information messaging.

15. Training the residual neural field network (125) using the output image residuals is performed at a first spatial resolution, and further: This includes training the residual neural field network using the output image residual with a second spatial resolution lower than the first spatial resolution as input, and the predicted residual image with the first spatial resolution as output. The method according to claim 1.

16. A non-temporary computer-readable storage medium storing computer-executable instructions for performing the method according to any one of claims 1 to 15 by one or more processors.

17. A device having a processor configured to perform the method described in any one of claims 1 to 15.

Images