Variable code rate method and system for end-to-end joint optimization of volume video compression

Abstract

The invention provides a variable code rate method and system for end-to-end joint optimization of volume video compression. The method comprises the following steps: representing a volume video by using a three-plane feature grid and a 3D density grid; a volume video residual error grid is learned, and three-plane residual error modeling is completed; learning data distribution of the volume video by using an entropy model, and estimating the compressed size of the volume video; an analog quantization error is introduced, and the quantization error of overall potential representation is controlled, so that the compressed size is a variable bit rate; rendering the volume video expressed by the three-plane residual error; training, optimized representation and compression of a dynamic nerve radiation field are carried out by using an end-to-end progressive training method; and inputting a volume video to be compressed into the trained dynamic neural radiation field, carrying out three-plane residual modeling and entropy coding, obtaining a code stream with a variable bit rate, and realizing streaming transmission. According to the invention, a wide variable bit rate is realized through a single model, and excellent rate distortion performance is maintained so as to adapt to different requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of volumetric video representation and compression, and in particular to an end-to-end jointly optimized variable bit rate method and system for volumetric video compression. Background Art

[0002] In virtual reality and telepresence scenarios, realistic volumetric videos provide an immersive experience, showing great potential to become the next-generation video format. Traditional volumetric video reconstruction methods mainly rely on point cloud-based methods and depth-based techniques, which usually have difficulties in rendering quality and realism. Recently, neural radiance fields and 3D Gaussian sputtering have shown considerable potential in representing realistic volumetric videos. However, the storage and transmission of volumetric videos using these two methods remain challenging. Compared with the implicit modeling of neural radiance fields, 3D Gaussian sputtering uses an explicit point cloud representation, which is not conducive to effective compression. In summary, the compact representation and implicit modeling capabilities of neural radiance fields make them naturally suitable for volumetric video compression.

[0003] Neural radiance fields and their variants have achieved remarkable success in synthesizing new views, inspiring numerous derivative works focusing on dynamic scenes. Some recent techniques adopt deformation fields to capture the motion of voxels relative to the canonical space, introduce temporal voxel features or achieve better temporal reconstruction by joint training across multiple frames. However, most existing studies focus on improving the reconstruction quality of neural radiance field representations, often ignoring the critical need to reduce storage size and transmission bandwidth. This oversight poses significant challenges for practical applications, especially in streaming volumetric videos.

[0004] To address these issues, several methods have been proposed to compress explicit features of dynamic neural radiance fields. They use grid-based explicit representations to model the spatiotemporal feature space of dynamic scenes and employ traditional image coding techniques to compress the representation after training. Subsequent work uses a hybrid representation of three planes to model dynamic scenes and employs traditional video codecs to further reduce redundancy. However, these methods optimize representation and compression independently, ignoring the rate-distortion trade-off during the training phase, which ultimately limits their compression performance. To bridge this gap, another work introduced an end-to-end joint training method for the representation and compression of dynamic neural radiance fields, but it only supports a fixed rate and has slow rendering speed. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide an end-to-end jointly optimized variable bit rate method and system for volumetric video compression.

[0006] According to one aspect of the present invention, a variable bit rate method for end-to-end joint optimization of volumetric video compression is provided, comprising:

[0007] S1: Use tri-plane feature grids and 3D density grids to represent volumetric video;

[0008] S2: learning the residual grid of the volumetric video, compensating for errors between different frames and newly observed areas in the volumetric video, completing three-plane residual modeling, and obtaining a volumetric video represented by a three-plane residual;

[0009] S3: using an entropy model to learn the data distribution of the volumetric video represented by the three-plane residual, and estimating its compressed size; at the same time, introducing a simulated quantization error to control the quantization error of the overall potential representation, so that the compressed size is a variable bit rate;

[0010] S4: Rendering the volumetric video represented by the three-plane residual;

[0011] S5: Based on the entropy model and the rendering, using an end-to-end progressive training method to train the dynamic neural radiation field, optimize representation and compression;

[0012] S6: Input the volumetric video to be compressed into the trained dynamic neural radiation field, perform three-plane residual modeling and entropy coding, obtain a variable bit rate bit stream, and realize streaming transmission.

[0013] Preferably, the using of a three-plane feature grid and a 3D density grid to characterize the volumetric video comprises:

[0014] Represent a complete volumetric video as a three-plane feature grid and a 3D density grid;

[0015] Using light to interpolate the three-plane feature grid to obtain a first interpolation feature;

[0016] Interpolating the 3D density grid using light to obtain a second interpolation feature;

[0017] The first interpolation feature and the second interpolation feature are input into a multi-layer perceptron to reconstruct a scene.

[0018] Preferably, the learning of the residual grid of the volumetric video, compensating for errors between different frames in the volumetric video and newly observed areas, completing three-plane residual modeling, and obtaining a video volume expressed by the three-plane residual includes:

[0019] Segment the volumetric video sequence into multiple feature groups of equal length, each group containing multiple frames;

[0020] In each of the feature groups, the first frame is used as an independent key frame, and a window top is set in the key frame for learning a complete feature grid;

[0021] In each of the feature groups, the feature grid of each frame except the first frame is recursively reconstructed by the residual of the feature grid of the previous frame and the current frame.

[0022] Preferably, frames in the same feature group share one multilayer perceptron as a feature decoder.

[0023] Preferably, the entropy model is used to learn the data distribution of the volumetric video represented by the three-plane residual, and the compressed size thereof is estimated; and at the same time, a simulated quantization error is introduced to control the quantization error of the overall potential representation so that the compressed size is a variable bit rate, specifically:

[0024] First, the entropy model learns the data distribution of the volumetric video represented by the three-plane residual:

[0025]

[0026] in, is the Gaussian entropy, which indicates the estimated size of the compressed volume video. The Gaussian entropy is the quantized potential representation In quantifying context features Conditional and simulated quantization error a i Gaussian entropy under control, N(μ i ,σ i ′) represents the Gaussian distribution obtained by probability modeling, represents uniform sampling;

[0027] Then, a set of learnable quantization parameters A = {a 1 , a 2 , ..., a n}, and the predefined Lagrange multiplier Λ={λ 1 ,λ 2 , ..., λ n}, the quantization parameter and the Lagrange multiplier are used in combination to control the error and achieve a variable bit rate:

[0028]

[0029] in, represents the variable rate compression loss, Representation Encoding The estimated bit rate required, Measure the original residual R t Instead of rebuilding The distortion between the paired Lagrange multipliers λ i with a i Balance the trade-off between bitrate and distortion.

[0030] Preferably, rendering the video volume represented by the three-plane residual includes:

[0031] Accumulate features along a ray:

[0032]

[0033] Where n s represents the number of sampling points on the ray r, δ k Represents the interval between adjacent samples, density It is interpolated from the density grid V, k represents the position of the kth sampling point, and the combined feature f k is obtained by connecting the appearance features f from the three planes lk ,l∈L;

[0034] Compute the reconstructed color of the light using a multilayer perceptron sharing the same set of features: Where ω(d) represents the position encoding of the viewpoint direction, Φ represents a small multi-layer perceptron, Represents a cumulative feature.

[0035] Preferably, the training of the dynamic neural radiation field based on the entropy model and the rendering using an end-to-end progressive training method to optimize the representation and compression includes:

[0036] First stage:

[0037] The reconstructed features of the three-plane residual modeling that are downsampled to a low resolution are trained to output a low-resolution density grid and residual three-plane features;

[0038] Phase 2:

[0039] Upsampling the residual three-plane features generated in the first stage to high resolution as initialization for the second stage;

[0040] Based on the initialization data, a learnable variable rate entropy coding model is adopted and jointly trained with the three-plane residual modeling.

[0041] Preferably, the method adopts a learnable variable rate entropy coding model based on the initialization data and performs joint training together with residual dynamic modeling, specifically:

[0042]

[0043] in is the loss of the sth stage, γ 1 and γ 2 is the weight of the regular item; As a residual regularization term, it is used to ensure time continuity and minimize R t the magnitude of

[0044] represents the variable rate compression loss, in Representation Encoding The estimated bit rate required, and Measure the original residual R t Instead of rebuilding The distortion between the paired Lagrange multipliers λ i with a i Balance the trade-off between bitrate and distortion.

[0045] is the luminosity loss, in is the set of training pixel rays, c g (r) and are the true and reconstructed colors of ray r, respectively.

[0046] Preferably, the step of inputting the volumetric video to be compressed into a trained dynamic neural radiation field, performing three-plane residual modeling and entropy coding, obtaining a bit stream with a variable bit rate, and realizing streaming transmission comprises:

[0047] Perform three-plane residual modeling on the volume video to be compressed to obtain the video volume expressed by the three-plane residual;

[0048] The video volume represented by the three-plane residual is rounded and quantized according to the quantization step size;

[0049] The quantized result is entropy encoded to obtain the bit rate, and the bit stream contains feature grid information of variable bit rate.

[0050] According to a second aspect of the present invention, there is provided an end-to-end jointly optimized variable bit rate system for volumetric video compression, comprising:

[0051] Representation module: Use three-plane feature grids and 3D density grids to represent volumetric video;

[0052] Residual module: learning the residual grid of the volumetric video, compensating for errors between different frames and newly observed areas in the volumetric video, completing three-plane residual modeling, and obtaining the video volume expressed by the three-plane residual;

[0053] Variable bit rate module: using an entropy model to learn the data distribution of the video volume represented by the three-plane residual, estimating its compressed size; at the same time, introducing a simulated quantization error to control the quantization error of the overall potential representation, so that the compressed size is a variable bit rate;

[0054] Rendering module: rendering the video volume represented by the three-plane residual;

[0055] Training module: Based on the entropy model and the rendering, an end-to-end progressive training method is used to train the dynamic neural radiation field, optimize the representation and compression;

[0056] Coding module: The volumetric video to be compressed is input into the trained dynamic neural radiation field, and three-plane residual modeling and entropy coding are performed to obtain a variable bit rate bit stream to achieve streaming transmission.

[0057] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0058] The end-to-end jointly optimized variable bit rate method for volumetric video compression in the embodiment of the present invention is based on a three-plane residual model and an entropy coding model, introduces simulated quantization error, controls the quantization error of the overall potential representation, and uses an end-to-end progressive method for training, so that a single model can achieve a wide range of variable bit rates while maintaining excellent rate-distortion performance to adapt to different needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0060] Figure 1 is a flow chart of a variable bit rate method for end-to-end joint optimization of volumetric video compression in one embodiment of the present invention;

[0061] Figure 2 It is a schematic diagram of the framework structure of a variable bit rate method for end-to-end joint optimization of volumetric video compression in a preferred embodiment of the present invention;

[0062] Figure 3 It is a schematic diagram of a variable bit rate training process of end-to-end joint optimization of volumetric video compression in a preferred embodiment of the present invention;

[0063] Figure 4 is a comparison chart of detailed quantitative results of the method in the embodiment of the present invention and other methods on the ReRF and DNA-Rendering datasets;

[0064] Figure 5 It is a comparison chart of RD performance between the method of the embodiment of the present invention and ReRF and JointRF. DETAILED DESCRIPTION

[0065] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0066] Volumetric video will play a very important role in people's lives. However, there is currently a lack of a good volumetric video encoding and decoding framework, and there is no method for end-to-end joint optimization of volumetric video representation and compression, which will lead to the loss of dynamic details and reduced compression efficiency. At the same time, volumetric videos trained with a single model are difficult to be flexible, and it is rare to freely choose multiple rate-distortion performances at the decoding and rendering end according to device conditions and personal needs. To address this problem, in one embodiment of the present invention, a variable bit rate method for end-to-end joint optimization of volumetric video compression is provided to improve the reconstruction quality and compression rate of volumetric video, significantly improve rate-distortion performance, and enhance its flexibility.

[0067] Specifically, Figure 1 As shown, the main steps are as follows:

[0068] S1: Use tri-plane feature grids and 3D density grids to represent volumetric video;

[0069] S2: Learn the residual grid of the volumetric video, complete the three-plane residual modeling, and obtain the video volume expressed by the three-plane residual;

[0070] S3: Use the entropy model to learn the data distribution of the video volume represented by the three-plane residual and estimate its compressed size; at the same time, introduce simulated quantization error to control the quantization error of the overall potential representation, so that the compressed size is a variable bit rate;

[0071] S4: Rendering the video volume represented by the three-plane residual;

[0072] S5: Based on entropy model and rendering, an end-to-end progressive training method is used to train dynamic neural radiation fields (including two parts, namely the three-plane residual model and the variable bit rate coding model) to optimize representation and compression;

[0073] S6: Input the volumetric video to be compressed into the trained dynamic neural radiation field, perform three-plane residual modeling and entropy coding, obtain a variable bit rate bit stream, and realize streaming transmission.

[0074] The variable bit rate method for end-to-end joint optimization of volumetric video compression in the above embodiment can improve the reconstruction quality and compression rate of volumetric video, significantly improve the rate-distortion performance, and obtain multiple rate-distortion performances in one model to meet different needs.

[0075] In order to better represent the volumetric video, in a preferred embodiment of the present invention, a preferred solution for volumetric video representation is provided, that is, step S1, using a three-plane feature grid and a 3D density grid to represent the volumetric video, and the following steps can be adopted, such as Figure 2 As shown:

[0076] S1.1, use three-plane feature grid and 3D density grid to represent the neural radiation field. A complete neural radiation field is represented as a three-plane feature grid P = {P l |l∈L}, L={xy,yz,xz} and 3D density grid V, that is, F=(P,V), where F refers to the neural radiation field of the static scene. The formula indicates that this neural radiation field F is characterized by a three-plane feature grid P and a 3D density grid V.

[0077] S1.2, using light to interpolate the three-plane feature grid P to obtain a first interpolation feature;

[0078] S1.3, using light to interpolate the 3D density grid V to obtain a second interpolation feature;

[0079] S1.4, input the first interpolation feature and the second interpolation feature into a small universal multi-layer perceptron to quickly reconstruct the scene. The reconstruction process of the neural radiation field can be expressed as:

[0080]

[0081] in represents the interpolation function, π l is to project the 3D point x onto the feature plane l, f is to connect the features of the three planes, Φ is a small multi-layer perceptron (MLP), which decodes the color of point x based on the feature f and the encoded viewpoint direction ω(d). The density of point x is obtained by interpolation on the density grid. In this way, the color and depth information of the corresponding level can be obtained, and the reconstructed neural radiation field (volume video) is represented by the color c of the corresponding point and the density σ of the corresponding point.

[0082] The above embodiment decomposes the radiation field of each frame into three planes. In order to further obtain characteristic information of the radiation field, in another preferred embodiment of the present invention, step S2 uses a compact three-plane residual grid to compensate for errors and newly observed areas between different frames in the dynamic neural radiation field, and the following steps can be used:

[0083] S2.1, in order to reduce the inefficient operation of transmitting the entire radiation field in each frame and take advantage of the continuity of the sequence, the video sequence is divided into multiple feature groups (GoFs) of equal length, each group contains N frames. Generally, N can be set to 30.

[0084] S2.2, the feature grid of a frame in each group can be recursively reconstructed by combining the feature grid of the previous frame with the residual of the current frame.

[0085] This residual modeling builds frame-to-frame residuals based on forward reference, thereby exploiting inherent temporal coherence and significantly reducing the data required for accurate scene representation.

[0086] Because residual learning of long sequences will lead to error accumulation, each group needs to relearn a complete feature grid as a reference, and then the subsequent frames in the group learn the residuals with this feature grid. Therefore, in a preferred embodiment, the top of the window is set in each feature group, and the first frame is used as an independent modeling I-feature F 1 (key frame), and subsequent frames are designated as P-features They represent the compensated residual with the previous feature. After retrieving the reconstructed features of the previous frame from the decoded buffer, the entire feature of the current frame can be reconstructed by applying residual compensation. This reconstruction process can be expressed as:

[0087]

[0088] refers to the reconstructed features of the tth frame; Refers to the reconstructed features of the t-1th frame; Refers to the compensated residual of the features of the t-th frame and the t-1-th frame.

[0089] The hierarchical residual radiation field representation method in the above-mentioned embodiment effectively reduces the temporal redundancy in the extended sequence and provides scalable quality adjustment.

[0090] In a preferred embodiment, frames in the same group share a compact global multi-layer perceptron as a feature decoder for the spatiotemporal feature space, effectively reducing bitrate consumption while maintaining performance quality. This sharing approach customizes the streaming experience for fluctuations in network conditions or specific requirements for detail levels, improving the efficiency and flexibility of volumetric video transmission.

[0091] The above embodiment introduces a compact three-plane implicit residual representation, decomposes the radiation field of each frame into three planes, and simulates the residual information between adjacent timestamps in this feature space (the three-plane feature space formed after the radiation field of each frame is decomposed into three planes) to effectively capture high-dimensional appearance features in a compact plane and minimize the temporal redundancy of inter-frame modeling of long-term dynamic scenes.

[0092] S1-S2 in the above embodiment completes the three-plane residual modeling and obtains the video volume represented by the three-plane residual. The entropy model can be used to estimate its size after compression. Generally, it is a fixed bit rate value. In order to meet different needs, a wider range of bit rates need to be obtained. Therefore, in a preferred embodiment, the entropy model is used to learn the data distribution of the video volume represented by the three-plane residual to estimate its compressed size; at the same time, the simulated quantization error is introduced to control the quantization error of the overall potential representation, so that the compressed size is a variable bit rate. Specifically, as Figure 2 As shown:

[0093] The entropy model is introduced to estimate the actual distribution data of each residual grid by constructing a new distribution, and predict the bit rate after entropy coding. The entropy estimation process can be expressed as:

[0094]

[0095] in To quantify the potential In quantifying context features Gaussian entropy under condition and error control, N(μ i ,σ i ′) represents Gaussian distribution, U is uniform sampling, which is also called uniform sampling. For a grid Gi, its center is pi, and the grid contains k points. Finally, the k points are represented by the points closest to the grid center pi. The sampled points are uniform.

[0096] In order to reduce the quantization error, a set of learnable quantization parameters A = {a 1 ,a 2 ,…,a n}, and the predefined Lagrange multiplier Λ={λ 1 ,λ 2 ,…,λ n}. The quantization parameters are used in combination with the Lagrange multipliers. A set of quantization parameters and Lagrange multipliers corresponds to one error value, so these errors can be controlled and variable bit rates can be achieved by using different sets of quantization parameters and Lagrange multipliers, such as low bit rate, medium bit rate and high bit rate.

[0097] In some specific embodiments, the Lagrange multipliers are initially {0.0018, 0.0035, 0.0067, 0.0130, 0.025, 0.0483, 0.0932, 0.18}, and the quantization parameter A={a 1 ,a 2 ,…,a n}Initially it is {1.0000,1.3944,1.9293,2.6874,3.7268,5.1801,7.1957,10.0}.

[0098] The above-mentioned simulated quantization error refers to adding a uniform noise of a certain size to the feature grid to simulate the error caused by real quantization. Because real quantization is rounded with a certain quantization step size, the error caused is a random value uniformly distributed within a certain range, so this operation is reasonable and feasible.

[0099] To encourage better rate-distortion performance, a bit rate constraint is effectively imposed on the network update. In a preferred embodiment, the bit rate obtained in step S3 is integrated into the loss function. This process replaces the actual entropy coding process, so that the distribution model of the original data remains differentiable.

[0100] However, during model training, since the quantization operation is inherently non-differentiable, this embodiment applies a straight-through estimator (STE) to approximate the gradient during backpropagation to facilitate the flow of gradients through the quantization step. This approximation makes it possible to effectively optimize the learnable quantization parameter a during training. i , allowing the model to dynamically adjust the quantization step size and optimize the bit rate. The rate-distortion loss function of the variable rate model (the model composed of S1+S2+S3) is defined as:

[0101]

[0102] in Representation Encoding The estimated bit rate required, and Measure the original residual R t Instead of rebuilding The distortion between the paired Lagrange multipliers λ i with a i Balance the trade-off between bitrate and distortion.

[0103] The above embodiment provides a scheme for compression of a variable rate residual representation, which utilizes a learnable quantization step and an entropy model based on a miniature multi-layer perceptron, combined with a predefined set of Lagrange multipliers, to achieve a variable bit rate.

[0104] In order to achieve further joint optimization of representation and compression, in a preferred embodiment of the present invention, S5, an end-to-end progressive training method is used to train the dynamic neural radiation field, and the representation and compression are jointly optimized to further improve the rate-distortion performance. The following two training stages can be used, such as Figure 3 As shown:

[0105] In the first training stage (e.g., steps 0-12999), the network is trained only on the reconstructed features that are downsampled to a low resolution. The input to this stage includes the multi-view image of the current frame and the reconstructed features of the previous frame obtained from the decoded buffer. These reconstructed features are downsampled to low resolution as initialization for the coarse training phase. The output of this phase is low-resolution residual tri-planar features and a density grid. This training phase not only accelerates convergence, but also lays a solid foundation for the more detailed optimization in the second phase.

[0106] In the second training phase (e.g., 13,000-40,000 steps), the residual features generated in the first phase are upsampled to a higher resolution and used as initialization for this phase to reduce training time and increase convergence speed. In addition, a learnable variable rate entropy coding model is used in the second phase for joint training with residual dynamic modeling. During the training process, multiple lambdas are used in the entropy model to optimize the quantization parameter A to achieve variable rate bitstream. This joint training method effectively captures low-entropy high-dimensional appearance features, significantly improving compression efficiency while maintaining high rendering quality.

[0107] The multi-rate distortion loss function of the whole framework is defined as follows:

[0108]

[0109] in is the loss of the sth stage, γ 1 and γ 2 is the weight of the regular term. As a residual regularization term, it aims to ensure time continuity and minimize R t The amplitude. Indicates variable rate compression loss. is the luminosity loss,

[0110]

[0111] in is the set of training pixel rays, c g (r) and are the true and reconstructed colors of ray r, respectively.

[0112] In another embodiment of the present invention, during model training, a delayed rendering model is used to further accelerate the training and rendering process, specifically:

[0113] The features are accumulated along the ray, and the formula is expressed as:

[0114]

[0115] Where n srepresents the number of sampling points on the ray r, δ k Indicates the interval between adjacent samples. Density Interpolated from the density grid V. Combined features f k is obtained by connecting the appearance features f from the three planes lk ,l∈L. The reconstructed color of the light is then computed using a small global multilayer perceptron shared across frames in the same group of features (GoF): Where ω(d) represents the position encoding of the viewpoint direction, Represents volumetric accumulated features. This approach significantly reduces computational complexity because each ray only requires a single multi-layer perceptron decoding.

[0116] The trained neural radiation field in the above embodiment has the following advantages: it can accurately characterize the scene and reconstruct high-quality dynamic volumetric video. It has low entropy characteristics, which is convenient for subsequent compression (low entropy means that the data distribution is simple, and the bit stream obtained after compression is smaller). When the above training is completed, in one embodiment of the present invention, a three-plane residual modeling and entropy coding model obtained by adjusting the values ​​of lambda and a is used to perform three-plane residual modeling and entropy coding to obtain a variable bit rate bit stream and realize streaming transmission. Specifically, the following can be used:

[0117] Perform three-plane residual modeling on the volume video to be compressed to obtain the video volume expressed by the three-plane residual;

[0118] The video volume represented by the three-plane residual is rounded and quantized according to the quantization step size (real quantization, not the above-mentioned analog quantization);

[0119] The quantized result is entropy-coded to obtain a bit stream, which contains feature grid information of a variable bit rate.

[0120] Based on the same inventive concept, in another embodiment of the present invention, a variable bit rate system for end-to-end joint optimization of volumetric video compression is provided, comprising:

[0121] Representation module: Use three-plane feature grids and 3D density grids to represent volumetric video;

[0122] Residual module: learning the residual grid of the volumetric video, compensating for errors between different frames and newly observed areas in the volumetric video, completing three-plane residual modeling, and obtaining the video volume expressed by the three-plane residual;

[0123] Variable bit rate module: using an entropy model to learn the data distribution of the video volume represented by the three-plane residual, estimating its compressed size; at the same time, introducing a simulated quantization error to control the quantization error of the overall potential representation, so that the compressed size is a variable bit rate;

[0124] Rendering module: rendering the video volume represented by the three-plane residual;

[0125] Training module: Based on the entropy model and the rendering, an end-to-end progressive training method is used to train the dynamic neural radiation field, optimize the representation and compression;

[0126] Coding module: The volumetric video to be compressed is input into the trained dynamic neural radiation field, and three-plane residual modeling and entropy coding are performed to obtain a variable bit rate bit stream to achieve streaming transmission.

[0127] The modules / units in the above examples of the present invention may specifically refer to the implementation techniques of the steps corresponding to the end-to-end joint optimized variable bit rate method for volumetric video compression in the above embodiments, which will not be described in detail here.

[0128] Due to the low entropy and high reconstruction quality characteristics of the feature grid, the result obtained after the compression model is small, and a high reconstruction quality is retained after restoration. The final trained neural radiation field model can be evaluated for its reconstruction quality using the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM), and its bit rate is evaluated using KB per frame. In some specific embodiments, the Bjontegaard Delta Bit-Rate (BDBR) is used to comprehensively compare the compression efficiency, and the rendering time efficiency (RT) is evaluated by calculating the average rendering time (in seconds) per frame.

[0129] Figure 4 Detailed quantitative results are presented on the ReRF and DNA-Rendering datasets. Figure 4 In the figure, "Ours(Low)", "Ours(Middle)" and "Ours(High)" represent different performances of the method of the embodiment of the present invention, which uses a single model to provide a variable bitrate bitstream. Compared with other methods, "Ours(Middle)" achieves higher reconstruction quality at a lower bitrate; "Ours(High)" provides significantly better reconstruction quality at a much lower bitrate than K-planes and ReRF; "Ours(Low)" achieves comparable reconstruction quality at a significantly lower bitrate than TetriRF and JointRF. In addition, the rendering time of the method of the embodiment of the present invention and TetriRF is at least twice that of ReRF and JointRF.

[0130] Figure 5The RD performance comparison of the method VRVVC of the embodiment of the present invention with ReRF and JointRF is shown. It is worth noting that VRVVC of the embodiment of the present invention consistently outperforms these methods in terms of RD performance. For example, compared with TetriRF, the method of the embodiment of the present invention reduces the average BDBR by -46.25 on the training view and -48.27 on the test view of the ReRF dataset. Similarly, on the DNA-Rendering dataset, the method of the embodiment of the present invention observes an average BDBR savings of -81.86% on the training view and -83.51% on the test view.

[0131] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. An end-to-end jointly optimized variable bit rate method for volumetric video compression, characterized in that: include: Use tri-plane feature grids and 3D density grids to represent volumetric video; Learning the residual grid of the volumetric video, completing three-plane residual modeling, and obtaining a volumetric video represented by the three-plane residual; Using an entropy model to learn the data distribution of the volumetric video represented by the three-plane residual, and estimating its compressed size; At the same time, an analog quantization error is introduced to control the quantization error of the overall potential representation, so that the compressed size is a variable bit rate; Rendering the volumetric video represented by the three-plane residual; Based on the entropy model and the rendering, a dynamic neural radiation field is trained using an end-to-end progressive training method to optimize representation and compression; The volumetric video to be compressed is input into the trained dynamic neural radiation field, and three-plane residual modeling and entropy coding are performed to obtain a variable bit rate bit stream to achieve streaming transmission.

2. The variable bit rate method for end-to-end joint optimization of volumetric video compression according to claim 1, characterized in that: The method of using a three-plane feature grid and a 3D density grid to characterize a volumetric video includes: Represent a complete volumetric video as a three-plane feature grid and a 3D density grid; Using light to interpolate the three-plane feature grid to obtain a first interpolation feature; Interpolating the 3D density grid using light to obtain a second interpolation feature; The first interpolation feature and the second interpolation feature are input into a multi-layer perceptron to reconstruct a scene.

3. The variable bit rate method for end-to-end joint optimization of volumetric video compression according to claim 1, characterized in that: The learning of the residual grid of the volumetric video, completing the three-plane residual modeling, and obtaining the video volume expressed by the three-plane residual includes: Segment the volumetric video sequence into multiple feature groups of equal length, each group containing multiple frames; In each of the feature groups, the first frame is used as an independent key frame, and a window top is set in the key frame for learning a complete feature grid; In each of the feature groups, the feature grid of each frame except the first frame is recursively reconstructed by the residual of the feature grid of the previous frame and the current frame.

4. The variable bit rate method for end-to-end joint optimization of volumetric video compression according to claim 3, characterized in that: Frames in the same feature group share a multi-layer perceptron as a feature decoder.

5. The variable bit rate method for end-to-end joint optimization of volumetric video compression according to claim 1, characterized in that: The entropy model is used to learn the data distribution of the volumetric video represented by the three-plane residual, and its compressed size is estimated; at the same time, a simulated quantization error is introduced to control the quantization error of the overall potential representation, so that the compressed size is a variable bit rate, specifically: First, the entropy model learns the data distribution of the volumetric video represented by the three-plane residual: in, is the Gaussian entropy, which indicates the estimated size of the compressed volume video. The Gaussian entropy is the quantized potential representation In quantifying context features Conditional and simulated quantization error a i Gaussian entropy under control, N(μ i ,μ i ′ ) represents Gaussian distribution, represents uniform sampling; Then, a set of learnable quantization parameters A = {a1, a2, ..., a n }, and the predefined Lagrange multipliers Λ = {λ1, λ2, ..., λ n }, the quantization parameter and the Lagrange multiplier are used in combination to control the error and achieve a variable bit rate: in, represents the variable rate compression loss, Representation Encoding The estimated bit rate required, Measure the original residual R t Rather than rebuilding The distortion between the paired Lagrange multipliers λ i With α i Balance the trade-off between bitrate and distortion.

6. The variable bit rate method for end-to-end joint optimization of volumetric video compression according to claim 1, characterized in that: The rendering of the video volume represented by the three-plane residual comprises: Accumulate features along a ray: where n s represents the number of sampling points on the ray r, δ k Represents the interval between adjacent samples, density It is interpolated from the density grid V, k represents the position of the kth sampling point, and the combined feature f k is obtained by connecting the appearance features f from the three planes lk ,l∈L; Compute the reconstructed color of the light using a multilayer perceptron sharing the same set of features: Where ω(d) represents the position encoding of the viewpoint direction, Φ represents a small multi-layer perceptron, Represents volume accumulation characteristics.

7. The variable bit rate method for end-to-end joint optimization of volumetric video compression according to claim 1, characterized in that: The method of training a dynamic neural radiation field based on the entropy model and the rendering using an end-to-end progressive training method to optimize representation and compression includes: First stage: The reconstructed features of the three-plane residual modeling that are downsampled to a low resolution are trained to output a low-resolution density grid and residual three-plane features; Phase 2: Upsampling the residual three-plane features generated in the first stage to high resolution as initialization for the second stage; Based on the initialization data, a learnable variable rate entropy coding model is adopted and jointly trained with the three-plane residual modeling.

8. The variable bit rate method for end-to-end joint optimization of volumetric video compression according to claim 7, characterized in that: Based on the initialization data, a learnable variable rate entropy coding model is used to perform joint training together with residual dynamic modeling, specifically: in is the loss of the sth stage, γ1 and γ2 are the weights of the conventional terms; As a residual regularization term, it is used to ensure time continuity and minimize R t the magnitude of represents the variable rate compression loss, in Representation Encoding The estimated bit rate required, and Measure the original residual R t Rather than rebuilding The distortion between the paired Lagrange multipliers λ i with a i Balancing the trade-off between bitrate and distortion; is the luminosity loss, in is the set of training pixel rays, c g (r) and are the true and reconstructed colors of ray r, respectively.

9. The variable bit rate method for end-to-end joint optimization of volumetric video compression according to claim 1, characterized in that: The method of inputting the volumetric video to be compressed into the trained dynamic neural radiation field, performing three-plane residual modeling and entropy coding, obtaining a bit stream with a variable bit rate, and realizing streaming transmission includes: Perform three-plane residual modeling on the volume video to be compressed to obtain the video volume expressed by the three-plane residual; The video volume represented by the three-plane residual is rounded and quantized according to the quantization step size; The quantized result is entropy-coded to obtain a bit stream, which contains feature grid information of a variable bit rate.

10. An end-to-end jointly optimized variable bit rate system for volumetric video compression, characterized in that: include: Representation module: Use three-plane feature grids and 3D density grids to represent volumetric video; Residual module: learning the residual grid of the volumetric video, completing three-plane residual modeling, and obtaining the video volume expressed by the three-plane residual; Variable bit rate module: using an entropy model to learn the data distribution of the video volume represented by the three-plane residual and estimate its compressed size; At the same time, an analog quantization error is introduced to control the quantization error of the overall potential representation, so that the compressed size is a variable bit rate; Rendering module: rendering the video volume represented by the three-plane residual; Training module: Based on the entropy model and the rendering, an end-to-end progressive training method is used to train the dynamic neural radiation field, optimize the representation and compression; Coding module: The volumetric video to be compressed is input into the trained dynamic neural radiation field, and three-plane residual modeling and entropy coding are performed to obtain a variable bit rate bit stream to achieve streaming transmission.

Images