Neural residual radiation field for streaming media free viewpoint video

Abstract

A residual radiation field (ReRF) as a highly compact neural representation to enable real-time FVV rendering of long-term dynamic scenes. ReRF explicitly models residual information between adjacent timestamps in a spatiotemporal feature space, using a compact MLP based on overall coordinates as a feature decoder. ReRF employs a compact motion grid and a residual feature grid to utilize inter-frame feature similarity. A sequential training scheme for maintaining the smoothness and sparsity of the motion / residual mesh is also provided. Based on ReRF, a special FVV codec that implements a three-order compression ratio can be designed, and a mating ReRF player can be implemented to support long-term FVV online streaming of dynamic scenarios.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present invention claims the benefit of and priority to PCT / CN2023 / 088271, entitled “Neural Residual Radiance Fields for Streaming Free-Viewpoint Video”, filed on April 14, 2023. The above-mentioned application is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention generally relates to apparatus and methods for neural network-based compact encoding of radiance fields to facilitate streaming free-viewpoint video. Background Art

[0004] The success of Neural Radiance Fields (NeRF) in modeling and rendering free-viewpoint videos of static objects has inspired numerous attempts to render dynamic scenes. Current techniques for facilitating free-viewpoint video (FVV) using neural rendering are limited to offline rendering or can only process short sequences with minimal motion. This paper describes a new technique, Residual Radiance Fields or ReRF, as a highly compact neural representation to enable real-time FVV rendering of long-duration dynamic scenes. . Summary of the invention

[0005] Various embodiments of the specification include, but are not limited to, systems, methods, and non-transitory computer-readable media for compact encoding of radiance fields using neural networks to facilitate streaming free-viewpoint video.

[0006] In some aspects, the technology described herein relates to a computer-implemented method comprising: training a multi-layer perceptron (MLP) feature decoder for a dynamic radiation field (e.g., trained from a start frame of the dynamic radiation field); capturing a first frame in the dynamic radiation field (e.g., the first frame is after the start frame); estimating a high-resolution motion field from the first frame to a second frame, wherein the second frame immediately precedes the first frame in the dynamic radiation field and the high-resolution motion field includes motion vectors for voxels in the first frame pointing to corresponding voxels in the second frame; constructing a low-resolution motion mesh for the first frame by performing motion pooling on the high-resolution motion field; constructing a residual mesh for the first frame by warping features of the second frame and the low-resolution motion mesh using the trained MLP decoder; and encoding the dynamic radiation field into a low-resolution representation based on a feature mesh representation of a frame in the dynamic radiation field (e.g., the start frame) and a series of low-resolution motion meshes and residual meshes constructed at each time step of the dynamic radiation field.

[0007] In some aspects, the technology described herein is directed to a method in which a dynamic radiation field includes a long-term RGB video input.

[0008] In some aspects, the techniques described herein are directed to a method in which a low-resolution motion mesh for a first frame includes voxel offsets between a plurality of voxels in the first frame and corresponding voxels in a second frame.

[0009] In some aspects, the techniques described herein relate to a method in which a residual grid includes sparse compensation for neighboring warp errors in a first frame and newly observed regions.

[0010] In some aspects, the technology described herein relates to a method in which constructing a low-resolution motion mesh includes: segmenting a high-resolution motion field into a plurality of three-dimensional cubes; for each of the plurality of three-dimensional cubes, applying average pooling to force each cube to share the same motion vector; and downsampling the high-resolution motion field into a low-resolution motion mesh.

[0011] In some aspects, the technology described herein is directed to a method in which applying average pooling includes selecting a voxel pointed to by an average motion vector in a second frame as a voxel motion vector of a three-dimensional cube.

[0012] In some aspects, the technology described herein relates to a method in which constructing a residual mesh for a first frame includes: obtaining a feature mesh for a second frame; warping the feature mesh and a low-resolution motion mesh of the second frame into a base mesh for the first frame, wherein the base mesh roughly compensates for feature differences caused by inter-frame motion; and learning the residual mesh for the first frame by fixing the base mesh of the first frame and the trained MLP and back-propagating gradients to update only parameters of the residual mesh.

[0013] In some aspects, the technology described herein is directed to a method that also includes decoding the low-resolution representation, wherein the decoding includes generating a feature grid for the first frame based on the feature grid of the second frame, the low-resolution motion grid for the first frame, and the residual grid for the first frame.

[0014] In some aspects, the technology described herein relates to a method in which generating a feature mesh for a first frame includes: applying a low-resolution motion mesh for the first frame to a feature mesh for a second frame to extract inter-frame redundancy and obtain a base feature mesh for the first frame; and adding a residual compensation in the residual mesh to the base feature mesh for the first frame to obtain a feature mesh for the first frame.

[0015] In some aspects, the technology described herein is directed to a method that also includes transmitting a low-resolution representation of the dynamic radiation field to a player for free-viewpoint viewing.

[0016] In some aspects, the technology described herein relates to a system comprising one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions, the instructions executable by the one or more processors to cause the system to perform operations, the operations comprising: training a multi-layer perceptron (MLP) feature decoder for a dynamic radiation field (e.g., trained from a start frame of the dynamic radiation field); capturing a first frame in the dynamic radiation field (e.g., the first frame is after the start frame); estimating a high-resolution motion field from the first frame to a second frame, wherein the second frame immediately precedes the first frame in the dynamic radiation field and the high-resolution motion field includes motion vectors for voxels in the first frame pointing to corresponding voxels in the second frame; constructing a low-resolution motion mesh for the first frame by performing motion pooling on the high-resolution motion field; constructing a residual mesh for the first frame by warping features of the second frame and the low-resolution motion mesh using the trained MLP decoder; and encoding the dynamic radiation field into a low-resolution representation based on a feature mesh representation of a frame in the dynamic radiation field (e.g., the start frame) and a series of low-resolution motion meshes and residual meshes constructed at each time step of the dynamic radiation field.

[0017] In some aspects, the technology described herein relates to a system in which a dynamic radiation field includes a long-term RGB video input.

[0018] In some aspects, the techniques described herein relate to a system in which a low-resolution motion mesh for a first frame includes voxel offsets between a plurality of voxels in the first frame and corresponding voxels in a second frame.

[0019] In some aspects, the techniques described herein relate to a system in which a low-resolution motion mesh for a first frame includes voxel offsets between a plurality of voxels in the first frame and corresponding voxels in a second frame.

[0020] In some aspects, the techniques described herein relate to a system in which a residual grid includes sparse compensations for neighboring warp errors in a first frame and newly observed regions.

[0021] In some aspects, the technology described herein relates to a system in which constructing a low-resolution motion mesh includes: segmenting a high-resolution motion field into a plurality of three-dimensional cubes; for each of the plurality of three-dimensional cubes, applying average pooling to force each cube to share the same motion vector; and downsampling the high-resolution motion field into a low-resolution motion mesh.

[0022] In some aspects, the technology described herein relates to a non-transitory computer-readable storage medium configured with instructions that are executable by one or more processors to cause the one or more processors to perform operations, the operations including: training a multi-layer perceptron (MLP) feature decoder for a dynamic radiation field (e.g., trained from a starting frame of the dynamic radiation field); capturing a first frame in the dynamic radiation field (e.g., the first frame is after the starting frame); estimating a high-resolution motion field from the first frame to a second frame, wherein the second frame immediately precedes the first frame in the dynamic radiation field and the high-resolution motion field includes motion vectors for voxels in the first frame pointing to corresponding voxels in the second frame; constructing a low-resolution motion mesh for the first frame by performing motion pooling on the high-resolution motion field; constructing a residual mesh for the first frame by warping features of the second frame and the low-resolution motion mesh using the trained MLP decoder; and encoding the dynamic radiation field into a low-resolution representation based on a feature mesh representation of a frame in the dynamic radiation field (e.g., the starting frame) and a series of low-resolution motion meshes and residual meshes constructed at each time step of the dynamic radiation field.

[0023] In some aspects, the technology described herein relates to a non-transitory computer-readable storage medium, wherein constructing a residual mesh for a first frame includes: obtaining a feature mesh for a second frame; warping the feature mesh and a low-resolution motion mesh of the second frame into a base mesh for the first frame, wherein the base mesh roughly compensates for feature differences caused by inter-frame motion; and learning the residual mesh for the first frame by fixing the base mesh of the first frame and the trained MLP and back-propagating gradients to update only parameters of the residual mesh.

[0024] In some aspects, the technology described herein relates to a non-transitory computer-readable storage medium, wherein the operations also include decoding a low-resolution representation, wherein the decoding includes: generating a feature mesh for a first frame based on a feature mesh of a second frame, a low-resolution motion mesh for the first frame, and a residual mesh for the first frame.

[0025] In some aspects, the technology described herein relates to a non-transitory computer-readable storage medium, wherein generating a feature mesh for a first frame includes: applying a low-resolution motion mesh for the first frame to a feature mesh of a second frame to extract inter-frame redundancy and obtain a base feature mesh for the first frame; and temporally adding a residual compensation in a residual mesh to the base feature mesh for the frame to obtain a feature mesh for the first frame.

[0026] These and other features of the systems, methods and non-transitory computer readable media disclosed herein, as well as methods of operation and functions of the related elements of the structures and combinations of parts and manufacturing economies, will become more apparent after considering the following description and appended claims with reference to the accompanying drawings, all of which constitute a part of this specification, wherein like reference numerals represent corresponding parts in the various drawings. It should be expressly understood, however, that the drawings are for illustration and description only and are not intended as a definition of the limits of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Preferred and non-limiting embodiments of the present invention may be more readily understood by reference to the accompanying drawings, in which:

[0028] Figure 1 An example ReRF is shown that utilizes residual radiation fields and a global MLP to achieve highly compressible and streaming radiation field modeling in accordance with some embodiments.

[0029] Figure 2 An example method for implementing ReRF according to some embodiments is shown.

[0030] Figure 3 An exemplary neural residual radiated field (ReRF) is shown in accordance with some embodiments.

[0031] Figure 4 An exemplary flow diagram of a ReRF-based codec and player according to some embodiments is shown.

[0032] Figure 5 An exemplary GOF structure is shown in accordance with some embodiments.

[0033] Figure 6 Shown are rendering appearance results of an exemplary ReRF method for a long inward 360° sequence with large motion, according to some embodiments.

[0034] Figure 7-10 Example experimental results of the ReRF method described in the present invention are shown.

[0035] Fig.11 An example overview for training a ReRF according to some embodiments is shown.

[0036] Fig.12 An overview of a ReRF based codec and player is shown according to some embodiments.

[0037] Fig.13 An example GOF structure is shown in accordance with some embodiments.

[0038] Fig.14Rendering results for a forward scene in a Neural 3D dataset are shown according to some embodiments.

[0039] Fig.15 Quantization results on a synthetic NeRF dataset are shown according to some embodiments.

[0040] Fig.16 A qualitative comparison with several state-of-the-art works according to some embodiments is shown.

[0041] Fig.17 Performance on long sequences (3000 frames) according to some embodiments is shown. DETAILED DESCRIPTION

[0042] Specific non-limiting embodiments of the present invention will now be described with reference to the accompanying drawings. It should be understood that the specific features and aspects of any embodiment disclosed herein can be used and / or combined with the specific features and aspects of any other embodiment disclosed herein. It should also be understood that these embodiments are only examples and only illustrate a few embodiments within the scope of the present invention. Various changes and modifications that are obvious to those skilled in the art to which the present invention belongs are considered to be within the spirit, scope and expectation of the present invention as further defined in the appended claims.

[0043] The residual radiance field (ReRF) described in this paper models the residual information between adjacent timestamps in the spatiotemporal feature space, using a small multi-layer perceptron (MLP) based on the global coordinates as a feature decoder. Specifically, ReRF adopts a compact motion grid and a residual feature grid to exploit the feature similarity between frames. We show that this strategy can handle large motions without sacrificing quality. This application also describes a sequential training scheme for maintaining the smoothness and sparsity of the motion / residual grids. Based on ReRF, this application also describes a special FVV codec that achieves a compression rate of three orders of magnitude, and provides a supporting ReRF player to support online streaming of long-term FVV of dynamic scenes. Extensive experiments demonstrate the effectiveness of ReRF in compactly representing dynamic radiance fields, achieving an unprecedented free-viewpoint viewing experience in terms of speed and quality.

[0044] 1. Introduction

[0045] Photorealistic free viewpoint videos (FVV) of dynamic scenes, especially human performances, close the gap between performers and viewers. But the goal of making FVV as easy to produce and watch as clicking and watching regular 2D videos on streaming platforms remains far-reaching. There are many challenges ranging from data processing and compression to streaming and rendering.

[0046] Geometry-based solutions reconstruct dynamic 3D meshes or points, while image-based solutions interpolate new views over densely transmitted sequential footage. Both techniques rely on high-quality reconstructions that are often susceptible to occlusions and textureless regions. Recent advances in neural technology have brought an alternative route that bypasses explicit geometric reconstruction. The pioneering work of Neural Radiance Fields (NeRF) concisely represents a static scene in a coordinate-based multi-layer perceptron (MLP) for photorealistic volume rendering. The MLP can be viewed as an implicit feature decoder from a spatially continuous feature space to a radiance output with RGB and density. However, even moderately deep MLPs can be too expensive for real-time rendering. Therefore, various extensions focus on using intelligent representations to "carve" the feature space to achieve a complex balance between computational speed and accuracy. Recent examples include explicit feature volumes, multi-scale hashing, codebooks, tri-planes, tensors, etc.

[0047] While effective, almost all existing solutions so far are tailored to handle static scenes. In contrast, streaming dynamic radiance fields require decoding features from a spatio-temporal continuous feature space into a radiance output using a global coordinate-based MLP. A naive per-frame solution is to apply a static method on a sequence of independent spatial feature spaces. This scheme discards important temporal coherence, resulting in low quality and inefficiency for long sequences. Recent methods attempt to maintain the canonical feature space by warping the features back into the canonical space in time to reproduce the features in each real-time frame. Various schemes have been proposed to compensate for temporal motion by adopting implicit matching or data-driven priors such as depth, Fourier features, optical flow, or skeleton / facial motion priors. However, the heavy reliance on a global canonical space makes them vulnerable to large motions or topological changes. The training overhead also increases significantly depending on the sequence length. Recent work has begun to explore feature redundancy between adjacent frames, but it has failed to maintain a coherent spatio-temporal feature space.

[0048] Figure 1 An example ReRF is shown, which utilizes residual radiation field and global MLP to achieve highly compressible and streaming radiation field modeling. The ReRF-based codec solution and streaming media player bring rich interactive experience to users.

[0049] like Figure 1As shown, the example ReRF models the residual of the radiation field between adjacent timestamps in the spatiotemporal feature space. Specifically, the global tiny MLP is configured to roughly estimate the radiation output of the dynamic scene in a sequential manner. In order to maintain high efficiency of training and inference, ReRF uses an explicit grid representation to model the feature space. However, ReRF only trains the first frame (also referred to as the key frame in the following description) to obtain the grid volume and MLP decoder for the entire sequence, and uses the obtained grid volume as the initial feature volume. For each subsequent frame, ReRF uses a compact (or low-resolution) motion grid and a sparse residual feature grid: the low-resolution motion grid represents the position offset from the current frame to the previous frame, while the sparse residual grid is used to compensate for errors in the current frame and newly observed areas. Here, a key frame refers to a full frame of an image in a video, from which a feature volume grid is learned, without using a motion grid and a residual grid. The main benefit of this design is that ReRF makes full use of the feature similarity between adjacent frames, where the complete feature grid of the current frame can be simply obtained from two adjacent frames, while avoiding the use of a global canonical space. In addition, both the motion mesh and the residual mesh are suitable for compression, especially for long-term dynamic scenes.

[0050] In some embodiments, a two-stage scheme can be used to efficiently obtain ReRF from RGB videos through sequential training. For example, a novel motion pooling strategy can be used to maintain the smoothness and compactness of the inter-frame motion grid and a sparse regularization term to improve the compactness of ReRF. In some embodiments, in order to make ReRF practical for users, a ReRF-based codec can be developed based on the traditional keyframe-based strategy to achieve a compression rate of three orders of magnitude compared to the per-frame neural representation. In some embodiments, a matching ReRF player can be developed for long-term FVV online streaming of dynamic scenes. With ReRF, users can pause, play, fast forward / rewind, and seek on dynamic radiation fields for the first time, just like watching 2D videos, and get an unprecedented high-quality free viewpoint viewing experience.

[0051] Figure 2 An example method for implementing ReRF according to some embodiments is shown. In some embodiments, the method may start with a sequential training scheme (see Section 3 for more details) to generate a mesh M with motion for each frame i (based on frame i and frame i-1) i and the residual grid r i Next, ReRF-based codec schemes and players (see Section 4 for more details) can compress the ReRF representation for fast data transmission and online playback.

[0052] The ReRF-based solution described in the rest of this invention brings at least the following technical improvements:

[0053] Introducing Residual Radiance Field (ReRF), a novel neural representation to support free-viewpoint viewing of streaming media with dynamic radiance fields.

[0054] Designing custom motion meshes and residual meshes to support sequential training without using a global canonical space that is notorious for large motions. In particular, designing multiple training strategies to achieve high compression rates while maintaining high rendering quality.

[0055] A complete description of a ReRF-based codec and a companion FVV player (enabling development by those skilled in the art) to stream long sequences of dynamic radiation fields, with extensive control capabilities.

[0056] 2. Related Work

[0057] Novel view synthesis for static scenes. Novel view synthesis, the problem of synthesizing new viewpoints given a set of 2D images, has attracted considerable attention recently. Light field representation formulates the problem via a two-plane parameterization. Early methods generated rays for new viewpoints by interpolation, which enables real-time rendering but requires caching all rays. Recent work uses neural networks for compact storage. Mesh-based representations allow efficient storage and can record view-dependent textures. However, optimizing meshes for scenes with complex topology remains a challenge. Multi-plane images show the ability to handle complex scenes due to their topology-free properties. Recently, the breakthrough method NeRF has greatly improved the realism of renderings and inspired many subsequent works, including multi-scale, relighting, editing, 3D-aware generation, etc. However, it assumes static scenes and cannot handle scene changes over time.

[0058] Novel view synthesis for dynamic scenes. Dynamic scenes are more complex due to lighting changes and object movement. One approach is to reconstruct the dynamic scene and render the geometry from a new view. RGB or RGB-D solutions have been widely explored. Other approaches model dynamic scenes via neural networks for view synthesis. For example, some methods use neural networks to regress each image from all other images to achieve viewpoint, temporal, or light interpolation. Some methods use encoder-decoder networks to transfer 2D images into 3D volumes and use volume rendering for end-to-end training. Some methods combine point features with multi-view images for dynamic human rendering. Using motion advection to convey feature vectors for still image animation is also an interesting direction.

[0059] Recently, NeRF has been extended to dynamic settings. Some works directly condition the neural radiance fields in time to handle spatial variations. Other works learn a spatial offset from the current scene to the learned canonical radiance field at each timestamp. For example, some methods condition NeRF on additional high-dimensional coordinates to account for discontinuous topological changes in addition to continuous deformation fields. Some methods handle dynamic changes in the scene by modeling the trajectory of each point in the scene. Some methods use explicit voxels to model the canonical space and deformation fields for dynamic scenes. Some methods model time-varying density and color via Fourier coefficients to extend octree-based radiance fields to dynamic scenes. Compared to existing methods, our method uses three orders of magnitude less storage and enables long sequences with large motion.

[0060] NeRF acceleration and compression. NeRF shows remarkable results in free-view rendering, but it is slow to train and render. Recent approaches reduce the complex MLP computation by decomposing NeRF into explicit 3D feature encodings using a shallow MLP decoder. Methods involving voxel grids, octrees, triplanes, multi-scale hashing, codebooks, tensor decomposition, and textured polygons have been explored.

[0061] Using explicit encodings greatly reduces training and inference time, but the additional storage consumption associated with these 3D structures is a problem. Some methods achieve high compression ratios through CP decomposition, rank reduction, or vector quantization, but are limited to static scenes. Recent dynamic methods use narrowband tuning on sparse voxel grids for video sequences, which is efficient to train but still has a size of MB per frame. Some methods decompose the 4D space into static, deforming, and new regions for efficient dynamic scene training and rendering, but are limited by the length of video sequences. In contrast, we adopt a residual radiance field and a ReRF-based codec scheme that achieves high compression and streaming of long sequences with large motion.

[0062] 3. Neural residual radiation field

[0063] In this section, an example ReRF representation for dynamic scenes is described (Section 3.1), followed by a companion training scheme for generating ReRF from RGB video input (Section 3.2). The steps described in this section are for illustrative purposes. Depending on the implementation, the method may include fewer, more, or alternative steps to achieve different technical advantages.

[0064] 3.1 Motion-aware residual field

[0065] Recall that radiance with color and density (c,σ) in NeRF is formulated as c,σ = Ψ(x,d), and an MLP (denoted as Ψ) is used as the decoder, given a 3D position x and a viewing direction d. Then, volume rendering is used to perform photorealistic novel view synthesis based on the radiance field. To maintain efficient training and inference, ReRF can use an explicit mesh representation. Specifically, for an explicit density mesh V σ and color feature grid V c , the radiation field of the static scene is:

[0066]

[0067] where interp(·) denotes the trilinear interpolation function on the grid, and Φ is a relatively shallow MLP for acceleration. For simplicity, we can append an additional channel to V c Come and V σ and V c are united into a common feature grid f. For this purpose, an explicit grid representation for the static radiation field is constructed from the feature grid f and a tiny MLP Φ as an implicit feature decoder.

[0068] Figure 3 An exemplary neural residual radiance field (ReRF) is shown. In some embodiments, first, a dense (also called high-resolution) motion field D is estimated. t Next, a compact motion mesh M is generated by motion pooling. t Finally, we can t-1 (For example, the feature grid for frame at time t-1 is the feature grid for frame f at time t. t Before) distorted into base mesh To learn the residual grid r t , thereby improving feature sparsity and facilitating compression.

[0069] To further characterize the dynamic radiation field, a coordinate-based tiny MLPΦ can be employed as a global feature decoder for the spatiotemporal feature space. A naive solution is to utilize a per-frame feature grid for a dynamic scene with N frames. But it discards important temporal coherence. Recent work DeVRF maintains the dense motion field (high-resolution motion field) The canonical feature grid f1 of t is used to reproduce the features in each real-time frame, but it is vulnerable to large motions or topological changes due to its reliance on the canonical space. In sharp contrast, the ReRF described in this paper can explicitly explore the feature similarities between adjacent timestamps in the spatiotemporal feature space. Here, a compact (also called low-resolution) motion grid M is introduced for the current frame t. t and the residual feature grid r t . Low-resolution motion mesh Mt Represents the voxel offset to indicate the corresponding voxel index in the previous frame for the voxel in the current frame. t represents the sparse compensation for both the neighboring warp error and the newly observed region in the current frame. In addition, for the first frame, the full explicit feature grid representation f1 with the accompanying global MLP Φ is adopted. Finally, ReRF sequentially characterizes the dynamic radiation field with N frames as Φ,f1, and like Figure 2 shown.

[0070] Note that ReRF implements efficient sequential feature modeling. Given the previous f t-1 , can be obtained from M t and r t Simply get the current feature grid f t , while avoiding the use of a global canonical space. Specifically, M t First, apply to f t-1 To extract inter-frame redundancy and obtain the basic feature grid for the current frame Let p denote the index of the explicit grid. Then, the per-voxel basis feature grid is formulated as:

[0071]

[0072] This in turn tries to explore the feature similarity between frames as much as possible. The entire feature grid can then be recovered by adding residual compensation: By using Equation 1 at f t The global MLPΦ is applied to reconstruct the current radiation field. t}, the motion-aware residual represents {M t ,r t} is compact and compression-friendly, which naturally models feature variations in a coherent spatiotemporal feature space.

[0073] 3.2 Sequential residual field generation

[0074] In some embodiments, a two-stage sequential training scheme can be used to obtain the long-term RGB video input including Φ, f1 and This naturally enhances the compactness of both the residual mesh and the motion mesh for streaming applications in Section 4. At the beginning, the complete explicit feature mesh f1 for the first frame (also called the start frame of the dynamic radiation field) can be obtained using off-the-shelf methods, accompanied by a global MLPΦ as a feature decoder. Then, given the previous frame (i.e., the frame immediately following f t The feature grid f of the previous frame) t-1 and the input image for the current frame, the motion mesh M can be compactly generated in the following two stages:t and the residual grid r t .

[0075] Motion Mesh Estimation. First, a dense motion field D from the current frame to the previous frame can be generated by using the previous frame as the canonical space. t In order to maintain a smooth and compact moving mesh M t , applying motion pooling strategy. Voxel p t The motion vector in can point to a different voxel p in the previous frame. t-1 . Thus, similar to the standard average pooling operation, the voxel pointed to by the average vector Selected as Voxel Sports Specifically, firstly D t The D is divided into cubes (e.g., 3D cubes), where each cube contains 8×8×8 consecutive voxels. Then, for each cube, D is t Average pooling is applied to force each cube to share the same motion vector. Afterwards, it is downsampled to generate a low-resolution motion mesh M t Note that the compact motion grid M t is compression-friendly, as its size is 512 times smaller than the original dense grid. In this way, some feature cubes from the previous frame can be tracked through the motion field, which can further reduce the entropy of the residual voxels. For this purpose, a low-resolution M is generated. t to compactly represent smooth motion over frames.

[0076] Residual mesh optimization. On the compact moving mesh M t With the help of t-1 Warp to current base mesh It roughly compensates for the feature differences caused by inter-frame motion. To optimize the residual grid, and Φ are fixed, and the gradient can be back-propagated to the residual grid r t To update only r t In addition to the photometric loss, L1 loss can be used to enhance its sparsity to improve compactness to make r t regularization. This sparse formulation also forces r t Only sparse information of inter-frame residuals or newly observed regions is compensated. t The total loss function The formula is:

[0077]

[0078] in is a set of training pixel rays; c(1) and are the ground truth color and predicted color of light 1, respectively; λ=0.01 is the weight of the regularization term.

[0079] Once you get M t ,r t , the explicit feature grid f of the current frame can be recovered as shown in Section 3.1 t , which also enables the continuation training of the next frame. Note that due to the compact representation and sparse characteristics, M t and r t The design and generation mechanism of make them compression-friendly, implementing the following ReRF codec and streaming. Section 7 below provides additional exemplary training details of ReRF.

[0080] 4. ReRF codec and streaming applications

[0081] 4.1 Feature-level residual compression.

[0082] Figure 4 An exemplary flow chart of a ReRF-based codec and player according to some embodiments is shown. In some embodiments, the encoder compresses the input signal by using PCA, 3D-DCT, quantization and entropy coding to produce a bitstream. The decoder receives the compressed bitstream, decodes each syntax element and reverses the encoding process. In addition, the decoded motion field (motion grid) M is given. t And the previously reconstructed feature grid The predicted feature grid can be obtained by deformation

[0083] In some embodiments, both the motion mesh and the residual mesh are suitable for compression, especially for long-duration dynamic scenes. In order to make ReRF practical for users, a ReRF-based codec and a matching FVV player can be developed for online streaming of long-duration dynamic scenes, such as Figure 4 As shown. Figure 4 In the exemplary flow chart shown in FIG. 1 , the feature grid sequence is first divided into several consecutive feature grid groups (GOFs), each GOF being as follows: Figure 5 (showing an exemplary GOF structure according to some embodiments) shows a collection of continuous grids. A GOF includes an I feature grid (keyframe) and a P feature grid. Each GOF starts with an I feature grid that is encoded independently of all other feature grids. The P feature grid contains a residual grid that is deformably compensated relative to the previous feature grid. Let {f1, r2, …, r t-1 ,r t ,…} represents GOF, where f1 is the characteristic grid and r t is the residual grid.

[0084] In some embodiments, Figure 4 The codec in (left part of the flowchart) can first convert f1 and r t Transformed into f1(m,n) and r t (m,n), an m×n feature matrix, where m and n are the number of non-empty feature voxels and the number of feature channels, respectively. Then, r t (m,n) performs linear principal component analysis (PCA) to obtain the main direction V. Finally, it can be obtained by r′ t =r t ·V will r t Projected to the main direction. Grids f1 and r′ t Each channel of is divided into a cube of 8×8×8 voxels and each cube is transformed individually using 3D DCT. Afterwards, the transform coefficients are quantized using a quantization matrix.

[0085] The quantized transform coefficients are entropy coded and combined with auxiliary information such as the motion field M t , frame type, etc. Specifically, the DC coefficient is encoded using the Differential Pulse Code Modulation (DPCM) method.

[0086] AC coefficient encoding involves arranging the quantized DCT coefficients in a "3D zigzag" order, grouping similar frequencies together using a run-length encoding (RLE) algorithm, and inserting length-encoded zeros. Finally, the DPCM-encoded DC coefficients and the RLE-encoded AC coefficients can be further compressed using Huffman coding. The advantage of the above compression method is that it can achieve variable bit rates by adjusting the quantization parameters, thereby achieving dynamic adaptive streaming of ReRF according to the available bandwidth.

[0087] 4.2 Network Streaming ReRF Player

[0088] A companion ReRF player for online streaming of long sequences of dynamic radiation fields with extensive control capabilities can thus be implemented. When a bitstream is received, the I feature grid is first reconstructed by performing inverse quantization and inverse transform on the quantized transform coefficients

[0089] After reconstructing the I feature grid, the subsequently received P feature grid is reconstructed. Specifically, the initially reconstructed residual grid is generated by inverse quantization and inverse transformation of the quantized transform coefficients. Then Back-project to the origin space In addition, the decoded motion field M is given t And the previously reconstructed feature grid The predicted feature grid can be obtained by deformation Finally, add and To generate the final reconstructed feature grid Will Output to a renderer to generate photorealistic FVV of dynamic scenes.

[0090] Benefiting from the design of the GOF structure, the ReRF player allows fast seeking to new positions during playback. Encountering a new GOF in the compressed bitstream means that the decoder can decode the compressed feature grid without reconstructing any previous feature grids. With the ReRF player, users can pause, play, fast forward / rewind, and search dynamic radiation fields for the first time, just like watching 2D video, bringing an unprecedented high-quality free-viewpoint viewing experience.

[0091] 5. Experimental results

[0092] In this section, ReRF is evaluated on various challenging scenarios. The captured dynamic dataset contains approximately 74 views with a resolution of 1920×1080 and a frame rate of 25fps. The proposed network is trained using the PyTorch framework on a single NVIDIA GeForce RTX3090. A companion ReRF player is also implemented for online streaming dynamic fields with long sequences. To verify the effectiveness of the proposed ReRF player, a PC with an Intel(R) Core(TM) i9-11900 CPU@2.5GHz and an NVIDIAGeForce RTX3090 GPU is used as the test platform. In the experiments, the length of each GOF is set to 20. As Figure 6 and Fig.14 As shown, high-quality appearance results are generated in both inward 360° scenes and forward scenes with long sequences and challenging large motions. Figure 6 The quantization factors shown in the third row are used to flexibly adjust the storage.

[0093] Figure 6 The rendering appearance results of an exemplary ReRF method for a long inward 360° sequence with large motion are shown. The last row shows that ReRF can achieve variable bitrates.

[0094] 5.1 Comparison

[0095] Figure 7 : Qualitative comparison with dynamic scene reconstruction methods and per-frame static reconstruction methods.

[0096] Comparison for dynamic scenarios. Experimental results are provided to demonstrate the effectiveness of the proposed ReRF method. ReRF is qualitatively and quantitatively compared with other state-of-the-art methods for dynamic scenarios, including DeVRF, DVGO, INGP, INGPT, and TiNeuVox. INGP-T is an improved temporally conditioned version of NGP. It uses the normalized 4D input [x, y, z, t] as the hash table input. Figure 7 The visual quality results of different methods are shown when compared with the ReRF compression method for both short and long sequences. Specifically, the ReRF method can achieve realistic free viewpoint rendering comparable to per-frame reconstruction DVGO and INGP, but with much smaller storage overload. Compared to dynamic reconstruction methods (DeVRF, INGP-T, TiNeuVox), ReRF achieves the most realistic rendering results in terms of realism and clarity, without losing performance in long sequences. DeVRF learns an explicit deformation field from the real-time frame to the first frame. When the motion is large, especially in long sequences, it is difficult to warp directly from the first frame. As the number of frames increases, INGPT and TiNeuVox encounter severe blurring effects. Note that no matter how the number of frames increases (even to 4000 frames), ReRF always maintains high realism and clarity, as shown in Figure 8 shown.

[0097] Figure 8 : Quantitative comparison in terms of the number of frames, which shows that the performance of our method does not degrade with the increase of the number of frames.

[0098] For quantitative comparison, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are used as metrics to evaluate ReRF rendering accuracy. 70 captured views are selected as the training set and the other 4 views are used as the test set. Table 1 shows that we can effectively use small storage to achieve high-quality results. In long sequences with large motion, the ReRF method outperforms other dynamic methods in terms of appearance.

[0099] Also, note that the ReRF approach achieves fast training (on the order of 10 minutes per frame) and fast rendering (20fps), significantly faster than NeRF and many previous approaches.

[0100]

[0101] Table 1: Qualitative comparison with dynamic scene reconstruction methods and per-frame static reconstruction methods. Average storage between frames and average PSNR between frames and views are calculated. Compared to the original DVGO, ReRF model size is three orders of magnitude smaller and preserves visual quality.

[0102] 5.2 Evaluation

[0103] Fig. 9: Rate-distortion curve. This figure shows the rate-distortion of different components for different configurations. Fig. 9 denoted as “our full” in , which is the most compact and can dynamically scale bitrate to varying storage requirements.

[0104] Fig.10 : Qualitative evaluation of different variations in ReRF-based methods.

[0105] Ablation study. Analysis of motion-aware residual modules and PCA modules. For non-motion-aware residuals, the method trains each frame independently and directly encodes the residuals of 2 frames. Fig. 9 It is highlighted that motion awareness can significantly improve compactness. Similarly, the PCA module can further improve compactness. Fig.10 Results are shown under a 700KB storage limit. In contrast, the full ReRF-based model produces photorealistic results with minimal noise caused by compression.

[0106] Storage Analysis. Table 2 shows the storage of each component in the high-quality version based on ReRF. It reports the compressed residual features, voxel motion field, PCA back-projection matrix V T and the average bitrate of other items including masks indicating empty space and header information. Note that the total average model size is 793KB, which is three orders of magnitude more compact.

[0107] Runtime Analysis. As shown in the runtime breakdown analysis in Table 3, our ReRF player supports real-time decoding and rendering of on-demand ReRF streams. The average time for decoding and rendering one frame is approximately 47.03ms and 44.62ms, respectively. In addition, the decoding time and rendering time are close to each other, which is more friendly to parallel processing. With parallel decoding and rendering, the total processing time of the player is approximately 50ms. Users can experience high frame rate free-viewpoint videos in an immersive way, as smooth as watching 2D videos on YouTube.

[0108]

[0109] Table 2: Quantitative evaluation of the storage of different components. It shows that the ReRF-based method is 1000 times smaller than the original model size without compression.

[0110] 6. Discussion

[0111] Conclusion. This paper proposes a novel residual radiance field (ReRF) technique for compact modeling of long-duration dynamic scenes. The novel motion / residual grid in ReRF is compression-friendly to sequentially model the spatiotemporal feature space of dynamic scenes. The ReRF-based codec scheme achieves three orders of magnitude compression improvement, while the ReRF player further enables online dynamic radiance field streaming and free viewing. Experimental results demonstrate the effectiveness of ReRF in highly compact and efficient dynamic scene modeling. Leveraging the unique streaming capability for long-duration dynamic scenes, the described ReRF-based approach is a key step in neural scene modeling with various potential immersive applications in VR / AR.

[0112]

[0113] Table 3: Breakdown of the processing time per frame in each stage of the ReRF player. Results are averaged over the entire sequence.

[0114] 7. Supplementary Materials

[0115] Fig.11 An example overview for training a ReRF according to some embodiments is shown.

[0116] Details of training neural residual fields.

[0117] This section provides a detailed example of the training scheme used to generate ReRF from RGB video input. Fig.11 Specifically, an explicit feature grid can be constructed for the first frame. Then, the previous feature grid f is given in sequence. t-1 and the RGB image at the current timestamp, the model optimizes the motion mesh M for the current frame t and the residual grid r t , to generate compact neural representations.

[0118] A.1 Feature mesh optimization at the first frame.

[0119] In some embodiments, given a long multi-view sequence, the first step is to learn an explicit feature grid and a global MLP Φ from the first frame. The explicit density grid V can be used σ and color feature grid V c Characterize the first frame. To render the view, a ray can be cast through the pixels and sample points along the ray. For a sampled point p, scene properties (density and color features) can be efficiently queried from the grid via trilinear interpolation:

[0120]

[0121] Where C is the number of color feature dimensions, N x 、Ny and N z Yes V σ and V c voxel resolution. For example, C=12 is chosen in the above experiments. Softplus and post-activation are used to obtain the density properties of the sampled points, and a global MLPΦ is applied to the color features for view-dependent rendering. This shallow MLP contains two hidden layers, and each layer has 128 channels. During training, the density grid V can be gradually enlarged σ and color feature grid V c The initial number of voxels is 125×125×125. After training steps 1000, 2000, and 4000, the final resolution will be enlarged to 150×150×150, 200×200×200, and 250×250×250, respectively.

[0122] During training of this explicit feature grid, a photometric MSE loss can be used and V v The total change loss is:

[0123]

[0124] where λ TV =0.000016; is a set of training pixel rays; c(l) and are the ground truth color and predicted color of ray 1, respectively. represents the squared difference between density values ​​in a voxel. The total variation loss is only activated during training iterations 1000 to 12000. We train with the Adam optimizer for 5000 iterations in the coarse phase and 16000 iterations in the fine phase with a batch size of 10192 rays. σ ,V c The learning rates of the and global MLP are 0.1, 0.11, and 0.002, respectively.

[0125] A.2 Motion mesh optimization.

[0126] This section provides the information needed to generate our compact low-resolution motion mesh M t , which represents the position offset from the current frame to the previous frame to explore feature similarity. We propose to estimate the motion field D from the dense t Generate M t , which is the shape of 3×N x ×N y ×N z and contains the distortion information from frame t to frame t-1.

[0127] For dense motion field estimation, we can first map the pixel rays of frame t to the point pt Then the sampled point p t The 3D motion Δp will be queried by trilinear interpolation t→t-1 =D t (p t ):

[0128]

[0129] Find the corresponding point p t-1 =p t +Δp t→t-1 After that, you can use it from t The previous feature grid f t-1 The features are obtained. Then, the global MLPΦ decodes the color features into RGB space. Finally, the pixel colors can be calculated by volume rendering. During this estimation, we also gradually expand the deformation field D from (125×125×125)→(150×150×150)→(200×200×200)→(250×250×250) after reaching training steps 1000, 2000, and 4000, respectively. t The following photometric MSE loss and total variation loss 25 are used to estimate D t :

[0130]

[0131] where the total variation loss enforces the smoothness of dense motion fields, and λ TV The Adam optimizer is used to train for 3000 iterations in the coarse phase and 16000 iterations in the fine phase, with a batch size of 10192 rays and a learning rate of 10 -4 .

[0132] Then, the motion pooling strategy from D can be used as described in the main manuscript. t Generate a smooth and compact motion mesh M t .

[0133] A.3 Residual grid optimization.

[0134] This section provides the sparse residual grid r that generates the current frame t The sparse residual grid is used to compensate for the distortion error and the newly observed area. Specifically, the embodiment can first use the compact motion field M t Distort the previous frame feature f t-1 , to generate the basic feature mesh Then, during optimization, the ray can be computed from image pixels and sample points p along the ray. t Both the basic feature grid and the residual grid are queried by trilinear interpolation to obtain and r t (p t ):

[0135]

[0136] Note that C = 13, this is due to the fact that we jointly represent density and color features in our feature grid f. Then, the final scene attributes of the current frame can be obtained by summing: Finally, a global MLPΦ decodes it into a radiance field with volume rendering to compute pixel colors.

[0137] In some embodiments, after reaching training steps 1000, 2000, and 4000, respectively, the same step-by-step training scheme can be used for the residual grid, starting from (125×125×125)→(150×150×150)→(200×200×200)→(250×250×250). In addition to the photometric MSE loss and the total variation loss on the density residual, we also use an additional L1 loss to encourage residual sparsity:

[0138]

[0139] in Represents the feature grid f t The density of TV = 0.000016 and λ 残差 =0.01.

[0140] Similar to our first frame explicit mesh optimization, the total change loss of the density residual is only activated during training iterations 1000 to 12000. We use the Adam optimizer to train for 5000 iterations in the coarse phase and 16000 iterations in the fine phase with a batch size of 10192 rays. t The density and residual grid r t The learning rates of the color features are 0.1 and 0.11 respectively. Note that the basic feature grid is fixed, and we initialize the residual grid with zero values ​​during our residual grid optimization.

[0141] B ReRF codec and streaming applications

[0142] Over the past few decades, image and video compression standards such as JPEG, JPEG2000, H.264 / AVC, and H.265 / HEVC have been proposed and widely used in many practical applications. Most of these video compression methods follow a hybrid coding structure, where motion compensation and residual coding are used to reduce spatial and temporal redundancy. Recent works have also attempted to use neural networks for video compression and have shown considerable performance. Inspired by these compression methods, we propose a ReRF-based codec and a companion FVV player for online streaming of long-duration dynamic scenes while still ensuring an immersive exploration experience on existing networks. Fig.12 Shows the overall pipeline of our framework.

[0143] Fig.12 An overview of a ReRF-based codec and player is shown (the modeling elements of the encoder and decoder are colored in light green and pink, respectively). The encoder compresses the input signal to produce a bitstream by using PCA, 3D-DCT, quantization, and entropy coding. The decoder receives the compressed bitstream, decodes each syntactic element, and reverses the encoding process. In addition, the decoded motion field M is given. t And the previously reconstructed feature grid We can obtain the predicted feature grid by deformation

[0144] B.1 Feature-level residual compression.

[0145] Fig.13 An example GOF structure is shown.

[0146] Both moving meshes and residual meshes are suitable for compression, especially for long-term dynamic scenes. To make ReRF practical for users, we design a ReRF-based codec that follows the traditional keyframe-based strategy. We first divide the feature mesh sequence into several consecutive groups of feature meshes (GOFs), such as Fig.13 As shown, it is a collection of continuous grids. GOF includes two frame types, namely I feature grid (key frame) and P feature grid. Each GOF starts with an I feature grid, which is encoded independently of all other feature grids and contains most of the important information of the following P feature grid sequence. The p feature grid contains a deformable compensated residual grid relative to the previous feature grid. Let {f1,r2,…,r t-1 ,r t ,…} represents GOF, where f1 is the initial feature grid and r t is the residual mesh at the current time step. Our goal is to generate a high-quality reconstructed feature mesh at any given bit rate

[0147] PCA. We first transform f1 and r t Transformed into f t (m,n) and r t (m,n), an m×n feature matrix, where m and n are the number of non-empty feature voxels and feature channels, respectively. Then, we t (m,n) performs a linear principal component analysis (PCA) to obtain a named tuple (U,S,V) which is r t An almost optimal approximation to the singular value decomposition of (m,n):

[0148] r t (m,n)=U·diag(S)·V T (10)

[0149] Among them, V is an n×q matrix representing the main direction, S is the q vector, and U is an m×q matrix. Finally, we will r t Projected into the main direction, as follows:

[0150] r′ t =r t ·V (11)

[0151] 3D DCT. Grids f1 and r′ t Each channel of is divided into a cube of 8×8×8 voxels, and each cube is transformed individually using 3D DCT [11, 1]. The residual voxel of each cube is represented by r(i,j,k), and the DCT coefficients R(u,v,w) can be calculated as:

[0152]

[0153] The transformed value at the coordinate origin R(0,0,0) is the DC coefficient, which is the most important value of the transformed coefficients. The DC coefficient has a larger amplitude and contains more RL energy. Although the remaining coefficients are the AC of the x coefficients, they contain less energy in the entire 3D-DCT used, and most of their energy is concentrated on the major axes of the cube.

[0154] Quantization. After that, the transform coefficients are quantized using a quantization matrix. The quantization matrix of the coefficient cube should have an entry for each coefficient. The values ​​in the quantization matrix depend on whether the corresponding coefficient is valid and the base quality factor adopted. We perform scalar quantization on the 3D DCT coefficients. Each quantized transform coefficient is given by

[0155]

[0156] Where S qis the scaling factor, and Q(y,v,w) is the quantization matrix entry. In this work, we construct the quantization matrix based on the psycho-visual experiment. The values ​​of the quantization matrix are provided in the supplementary material as quant.npy.

[0157] Entropy coding. The quantized transform coefficients are entropy coded and compared with the motion field M t , frame type and other auxiliary information. Entropy coding involves arranging the quantized DCT coefficients in a "3D zigzag" order, grouping similar frequencies together using a run-length encoding (RLE) algorithm, inserting length-coded zeros, and then using Huffman coding on the remainder. The DC coefficient is encoded separately from the AC coefficients. Specifically, the DC coefficient is encoded using the differential pulse code modulation (DPCM) method: except for the first DC coefficient, we encode the difference between the current DC coefficient and the previous DC coefficient. The AC coefficient is encoded using the RLC method. In order to make it most likely to reach long periods of zero, a "3D zigzag" scan is used. Finally, we use Huffman coding to further compress the DPCM-encoded DC coefficient and the RLE-encoded AC coefficient.

[0158] Experimental results show that our ReRF-based codec achieves three orders of magnitude compression compared to per-frame neural representations. Another advantage of our compression method is the ability to adjust the scaling factor S during quantization. q to achieve variable bit rate, thus enabling dynamic adaptability of ReRF streaming according to the available bandwidth.

[0159] B.2 Network Streaming ReRF Player

[0160] We also implement a companion ReRF player for online streaming of long sequences of dynamic radiance fields with extensive control capabilities. Our ReRF player supports downloading encoded bitstreams from a streaming server. When the bitstream is received, the I feature grid is first reconstructed by performing inverse quantization and inverse transform on the quantized transform coefficients.

[0161] After reconstructing the I feature grid, the subsequently received P feature grid is reconstructed. Specifically, the initially reconstructed residual grid is generated by inverse quantization and inverse transformation of the quantized transform coefficients. Then, the following formula Back-project to the origin space

[0162]

[0163] In addition, the decoded motion field M is given t And the previously reconstructed feature grid We can obtain the predicted feature grid by deformation Let p denote the index of our explicit grid. Then, the predicted feature grid is formulated as:

[0164]

[0165] Add predicted feature grid and the initial reconstructed residual grid To generate the final reconstructed feature grid as follows:

[0166]

[0167] Finally, the feature mesh will be reconstructed Output to a renderer to generate photorealistic FVV of dynamic scenes. Since our ReRF player can efficiently reconstruct and render dynamic scenes, users are free to choose their perspectives as if they were in the target scene.

[0168] Fig.14 Rendering results for the forward scene in the Neural 3D dataset are shown.

[0169] Benefiting from the design of the GOF structure, our ReRF player allows fast seeking to new positions during playback. The reason is that the encoded bitstream consists of continuous GOFs. The first frame in a GOF is an I feature grid (keyframe) containing independently encoded features. Encountering a new GOF in the compressed bitstream means that the decoder can decode the compressed feature grid without reconstructing any previous feature grids. With the ReRF player, users can pause, play, fast forward / rewind, and search dynamic radiation fields for the first time, just like watching 2D videos, bringing an unprecedented high-quality free-viewpoint viewing experience.

[0170] Note that the I feature grid (keyframe) is different from the explicit feature grid used for the first frame. In the first frame training, we only use the explicit feature grid for representation. All other sequential frame features are represented using the residual grid r t trained, and can generate feature grid f t The GOF structure is used to achieve fast search. It will select key frames for each GOF size. For I feature grid, for the complete feature grid f t Encoding (generated from the residual grid). For P feature grids, for the residual grid r t coding.

[0171]

[0172]

[0173] Table 4: Comparison with state-of-the-art methods on static scenes. We compare our method with previous and contemporary novel view synthesis methods on both datasets. All scores for baselines are taken directly from their papers (whenever available). Our method uses minimal storage while maintaining high PSNR.

[0174] C Experiment

[0175] C.1 Dataset Details

[0176] Our captured dynamic dataset contains about 74 views with a resolution of 1920×1080 and a frame rate of 25fps. The camera looks towards the center in a cylindrical distribution. The longest sequence is over 1000 frames and the longest sequence contains 4000 frames. We use five real-world captured data and two synthetic data for experiments.

[0177] C.2 Other experimental results

[0178] Static Scene Comparison. To demonstrate the performance of our I-feature mesh (keyframe) compression, we also compare it with existing static scene novel view synthesis methods on the synthetic NeRF dataset and TanksTemples dataset in Table 1.

[0179] Compared to the original DVGO, we achieve an order of magnitude smaller bitrate without significantly sacrificing quality. We choose 2 different quantization factors to show our high-quality compression and low-quality compression results. Note that our high-quality version achieves the most compact modeling, with rendering quality roughly the same as the original DVGO, and outperforming vanilla NeRF and many other methods. Our low-quality version also shows that we can achieve high PSNR (>30) with more compact storage (<1MB).

[0180] Fig.15 Quantitative results on the synthetic NeRF dataset are shown.

[0181] Fig.16 A qualitative comparison with some recent work is shown.

[0182] Fig.15 and Fig.16 It is also shown that our method is the most compact and maintains high rendering quality for static scenes compared to previous and contemporary methods. Seven different quantization scaling factors are adopted to achieve variable bitrate. Our method also achieves fast reconstruction and rendering simultaneously.

[0183] Fig.17 The performance on long sequences (3000 frames) is shown.

Claims

1. A computer-implemented method, characterized in that: include: Training a multi-layer perceptron (MLP) feature decoder for dynamic radiation fields; capturing a first frame of the dynamic radiation field; estimating a high-resolution motion field from the first frame to a second frame, wherein the second frame immediately precedes the first frame in the dynamic radiation field and the high-resolution motion field includes motion vectors for voxels in the first frame pointing to corresponding voxels in the second frame; constructing a low-resolution motion mesh for the first frame by performing motion pooling on the high-resolution motion field; constructing a residual mesh for the first frame by warping features of the second frame and the low-resolution motion mesh using the trained MLP decoder; as well as The dynamic radiation field is encoded into a low-resolution representation based on a feature grid representation of a frame in the dynamic radiation field and a series of low-resolution motion grids and residual grids constructed at each time step of the dynamic radiation field.

2. The computer-implemented method of claim 1, wherein: The dynamic radiation field includes a long-term RGB video input.

3. The computer-implemented method of claim 1 , wherein: The low-resolution motion mesh for the first frame includes voxel offsets between a plurality of voxels in the first frame and corresponding voxels in the second frame.

4. The computer-implemented method of claim 1, wherein: The residual grid includes sparse compensations for neighboring warp errors and newly observed regions in the first frame.

5. The computer-implemented method of claim 1, wherein: Constructing the low-resolution motion mesh includes: dividing the high-resolution motion field into a plurality of three-dimensional cubes; For each of the plurality of three-dimensional cubes, applying average pooling to force each cube to share the same motion vector; and The high-resolution motion field is down-sampled to a low-resolution motion mesh.

6. The computer-implemented method of claim 5, wherein: Applying average pooling includes: The voxel pointed to by the average motion vector in the second frame is selected as the voxel motion vector of the three-dimensional cube.

7. The computer-implemented method of claim 1, wherein: Constructing the residual grid for the first frame comprises: Obtaining a feature grid of the second frame; warping the feature mesh and the low-resolution motion mesh of the second frame into a base mesh of the first frame, wherein the base mesh roughly compensates for feature differences caused by inter-frame motion; and The residual grid for the first frame is learned by fixing the base grid and the trained MLP of the first frame and back-propagating gradients to update only parameters of the residual grid.

8. The computer-implemented method of claim 1, wherein: Also included is decoding the low-resolution representation, wherein the decoding comprises: A feature grid for the first frame is generated based on the feature grid of the second frame, the low-resolution motion grid for the first frame, and the residual grid for the first frame.

9. The computer-implemented method of claim 8, wherein: Generating the feature grid for the first frame comprises: applying the low-resolution motion mesh for the first frame to the feature mesh of the second frame to extract inter-frame redundancy and obtain a base feature mesh for the first frame; and The residual compensation in the residual grid is added to the base feature grid for the first frame to obtain the feature grid for the first frame.

10. The computer-implemented method of claim 1, wherein: Also included is transmitting the low resolution representation of the dynamic radiation field to a player for free viewpoint viewing.

11. A system, characterized in that: The system includes one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions, the instructions being executable by the one or more processors to cause the system to perform operations, the operations comprising: Training a multi-layer perceptron (MLP) feature decoder for dynamic radiation fields; capturing a first frame of the dynamic radiation field; estimating a high-resolution motion field from the first frame to a second frame, wherein the second frame immediately precedes the first frame in the dynamic radiation field and the high-resolution motion field includes motion vectors for voxels in the first frame pointing to corresponding voxels in the second frame; constructing a low-resolution motion mesh for the first frame by performing motion pooling on the high-resolution motion field; constructing a residual mesh for the first frame by warping features of the second frame and the low-resolution motion mesh using the trained MLP decoder; and The dynamic radiation field is encoded into a low-resolution representation based on a feature grid representation of a frame in the dynamic radiation field and a series of low-resolution motion grids and residual grids constructed at each time step of the dynamic radiation field.

12. The system according to claim 11, characterized in that The dynamic radiation field includes a long-term RGB video input.

13. The system according to claim 11, characterized in that The low-resolution motion mesh for the first frame includes voxel offsets between a plurality of voxels in the first frame and corresponding voxels in the second frame.

14. The system according to claim 11, characterized in that The low-resolution motion mesh for the first frame includes voxel offsets between a plurality of voxels in the first frame and corresponding voxels in the second frame.

15. The system according to claim 11, characterized in that The residual grid includes sparse compensations for neighboring warp errors and newly observed regions in the first frame.

16. The system according to claim 11, characterized in that Constructing the low-resolution motion mesh includes: dividing the high-resolution motion field into a plurality of three-dimensional cubes; For each of the plurality of three-dimensional cubes, applying average pooling to force each cube to share the same motion vector; and The high-resolution motion field is down-sampled to a low-resolution motion mesh.

17. A non-transitory computer-readable storage medium, characterized in that The device is configured with instructions executable by one or more processors to cause the one or more processors to perform operations, the operations comprising: Training a multi-layer perceptron (MLP) feature decoder for dynamic radiation fields; capturing a first frame of the dynamic radiation field; estimating a high-resolution motion field from the first frame to a second frame, wherein the second frame immediately precedes the first frame in the dynamic radiation field and the high-resolution motion field includes motion vectors for voxels in the first frame pointing to corresponding voxels in the second frame; constructing a low-resolution motion mesh for the first frame by performing motion pooling on the high-resolution motion field; constructing a residual mesh for the first frame by warping features of the second frame and the low-resolution motion mesh using the trained MLP decoder; and The dynamic radiation field is encoded into a low-resolution representation based on a feature grid representation of a frame in the dynamic radiation field and a series of low-resolution motion grids and residual grids constructed at each time step of the dynamic radiation field.

18. The non-transitory computer-readable storage medium of claim 17, wherein: Constructing the residual grid for the first frame comprises: Obtaining a feature grid of the second frame; warping the feature mesh and the low-resolution motion mesh of the second frame into a base mesh of the first frame, wherein the base mesh roughly compensates for feature differences caused by inter-frame motion; and The residual grid for the first frame is learned by fixing the base grid and the trained MLP of the first frame and back-propagating gradients to update only parameters of the residual grid.

19. The non-transitory computer-readable storage medium of claim 17, wherein: The operations also include decoding the low-resolution representation, wherein the decoding includes: A feature grid for the first frame is generated based on the feature grid of the second frame, the low-resolution motion grid for the first frame, and the residual grid for the first frame.

20. The non-transitory computer-readable storage medium of claim 19, wherein: Generating the feature grid for the first frame comprises: applying the low-resolution motion mesh for the first frame to the feature mesh of the second frame to extract inter-frame redundancy and obtain a base feature mesh for the first frame; and The residual compensation in the residual grid is temporally added to the base feature grid for the frame to obtain the feature grid for the first frame.

Images