A variable rate 4D gaussian compression method

Abstract

The application provides a variable bit rate 4D Gaussian compression method, comprising the following steps: S1: using a complete 3D Gaussian to represent a first frame (key frame); S2: using a motion grid and two shared light-weight multi-layer perceptron (MLP) to perform motion estimation on an inter-frame Gaussian cell; S3: using a sparse compensation Gaussian to perform motion compensation on an inter-frame change region; S4: using a complete Gaussian expression of a previous frame in a buffer, a motion grid and a sparse compensation Gaussian of a current frame to reconstruct a complete Gaussian expression of the current frame, and storing the complete Gaussian expression in the buffer for reconstruction of a next frame; and S5: quantizing and entropy encoding the motion grid and the sparse compensation Gaussian which have been trained, compressing a model size, obtaining a variable bit rate code stream, and realizing streaming. The application realizes a wide variable bit rate through a single model, while keeping superior rate distortion performance, and can be used for streaming to adapt to different requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of volumetric video representation and compression, and in particular to a variable bitrate 4D Gaussian compression method. BACKGROUND

[0002] Free-viewpoint video (FVV) supports immersive real-time navigation of a scene from any viewpoint, enhancing user interactivity and realism, which makes FVV well-suited for applications such as entertainment, virtual reality, sports broadcasting, and telepresence. However, streaming and rendering high-quality FVV remains challenging, especially for sequences with large motion, complex background, and long duration. The main difficulty lies in developing an efficient representation and compression method that enables FVV to support limited bitrate streaming while maintaining high fidelity.

[0003] Traditional FVV reconstruction methods mainly rely on point cloud-based methods and depth-based techniques, which struggle to provide high-quality rendering and realism, especially in complex scenes. Neural Radiance Fields (NeRF) and its variants demonstrate impressive results in reconstructing FVV by learning continuous 3D scene representations, but they have limitations in supporting long sequences and streaming. Recent methods address these issues by compressing the explicit features of dynamic NeRF, however, these methods are often affected by slow training and rendering speeds.

[0004] Recently, 3D Gaussian Spraying (3DGS) has shown superior performance in rendering speed and quality for static scenes compared to NeRF-based methods. Several methods attempt to extend 3DGS to dynamic environments by incorporating temporal correspondence or temporal dependency, but these methods require loading all frames into memory for training and rendering, which limits their practicality in streaming applications. 3DGStream models the inter-frame transformation and rotation of 3D Gaussians as a neural transform cache, which reduces the per-frame memory requirement for FVV. However, the overall data volume is still large, hindering its ability to support efficient FVV transmission. Although a few studies have explored the compression of dynamic 3DGS, these methods face significant difficulties in handling real-world dynamic scenes with background, which limits their practical utility. Furthermore, they optimize the representation and compression independently, ignoring the rate-distortion (RD) trade-off during training, which ultimately limits the compression efficiency. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a variable bitrate 4D Gaussian compression method.

[0006] According to one aspect of the present application, a variable bitrate 4D Gaussian compression method is provided, comprising:

[0007] The first frame is compressed using a complete Gaussian representation, and a bit rate of the compressed Gaussian representation is estimated using an implicit entropy model, and a simulated quantization is introduced so that the size of the compressed Gaussian representation is a variable bit rate;

[0008] The first stage of training of the non-key frame uses a motion mesh and two globally shared light multi-layer perceptron (MLP) to estimate the motion of the Gaussian primitives between frames, and a motion-estimated Gaussian representation is obtained. An implicit entropy model is used to estimate the bit rate of the compressed motion mesh, and a simulated quantization is introduced so that the size of the compressed motion mesh is a variable bit rate. The motion mesh, the two globally shared light MLP, and the corresponding implicit entropy model are jointly optimized.

[0009] The second stage of training of the non-key frame uses a sparse compensated Gaussian based on the motion-estimated Gaussian representation to compensate for the changes between frames, and an implicit entropy model is used to estimate the bit rate of the compressed sparse compensated Gaussian, and a simulated quantization is introduced so that the size of the compressed sparse compensated Gaussian is a variable bit rate. The sparse compensated Gaussian and the corresponding implicit entropy model are jointly optimized.

[0010] The complete Gaussian representation of the current frame is reconstructed using the complete Gaussian representation of the first frame or the last frame in the buffer, the motion mesh of the current frame, and the sparse compensated Gaussian, and is stored in the buffer for reconstruction of the next frame.

[0011] The trained motion mesh and the sparse compensated Gaussian are quantized and entropy encoded, and the size of the motion mesh and the sparse compensated Gaussian is compressed to obtain a variable bit rate code stream, and streaming is realized.

[0012] Based on the representation of the first frame, and the first stage of training of the non-key frame, the second stage of training of the non-key frame, the reconstruction of the current frame, and the compression of the trained motion mesh and the sparse compensated Gaussian, the compression and streaming of the entire volumetric video are completed.

[0013] Preferably, the compression of the first frame using a complete Gaussian representation includes:

[0014] The entire three-dimensional space scene is represented using a set of Gaussian primitives G.

[0015] Each Gaussian primitive G is represented by a set of parameters {μ; R; f; s; α}, where μ is the center position, R is the rotation matrix, f is the SH coefficient of the color c dependent on the view angle, s is the scaling vector, and α is the transparency. For a point x located in the Gaussian primitive G, the spatial distribution of x is determined by

[0016] ​​​where ∑ = Rss T R T ;

[0017] When rendering, the rendering color c of a pixel is calculated by alpha blending the overlapping Gauss primitives in depth order, specifically:

[0018]

[0019] where α′ i is the projection of the transparency of the i-th Gauss primitive on the image plane, c i is the color of the i-th Gauss primitive in the viewing direction, and N represents the number of Gauss primitives in the entire three-dimensional space.

[0020] Preferably, the motion estimation of the inter-frame Gauss primitives is performed using a motion grid and two globally shared lightweight multi-layer perceptrons (MLPs), to obtain a motion-estimated Gauss representation, including:

[0021] The center positions of the Gauss primitives in the complete Gauss representation of the previous frame loaded from the buffer are position-encoded to be mapped to a multi-band;

[0022] The position encodings are trilinear-interpolated on a multi-resolution motion grid M t to generate motion features of different scales;

[0023] The motion features of different scales are concatenated and input into two shared lightweight multi-layer perceptrons Φ μ and Φ R respectively, to obtain the motion estimation of the translation and rotation of the Gauss primitives from the previous frame to the current frame;

[0024] Using the obtained motion estimation and the complete Gauss representation of the previous frame in the buffer, a motion-estimated transformed Gauss representation G′ t of the current frame is obtained.

[0025] Preferably, an implicit entropy model is used to estimate the bit rate of the motion grid or the sparse-compensated Gauss after compression, and a simulated quantization is introduced, so that the size after compression is a variable bit rate, including:

[0026] The simulated quantization effect with a step size of q is obtained by adding uniform noise to the motion grid M t or the sparse-compensated Gauss ΔG t , to obtain a quantized value The size of the added noise is consistent with the step size of the quantization;

[0027] An implicit entropy model is used to approximate the quantized value of the simulated quantized motion grid or the sparse-compensated Gauss ​ the probability mass function of to estimate the compressed bitrate, specifically:

[0028]

[0029] where is the quantized value the probability mass function of is the cumulative distribution function of the quantized value

[0030] Preferably, the joint optimization of the motion mesh, two globally shared light-weight MLPs and the corresponding implicit entropy model comprises:

[0031] The weighted sum of the distortion loss and the estimated model bitrate loss is taken as the total loss, and the gradient is backpropagated, specifically:

[0032]

[0033] wherein, denotes the total loss of the first stage, is the bitrate loss estimated from the quantized motion mesh N is the number of quantized values is the photometric loss, c g and are the real and reconstructed colors corresponding to the view angle, respectively; is the D-SSIM evaluation index between the real picture and the rendering result in training, λ2 is a weight parameter; the parameter λ1 balances the trade-off between bitrate and distortion, controlling the model size and reconstruction quality.

[0034] Preferably, the motion-compensated interframe change region using sparse compensation Gauss based on the Gaussian expression after motion estimation comprises:

[0035] On the basis of the Gaussian expression G′ t after the motion estimation transformation of the current frame, identify the suboptimal region that needs to be compensated, including the region whose gradient exceeds a predetermined threshold, and the region whose translation and rotation of the Gaussian cell exceeds a predetermined threshold;

[0036] For the region whose gradient exceeds a predetermined threshold, clone 1 Gaussian cell located in the region;

[0037] For the region whose translation and rotation of the Gaussian cell exceeds a predetermined threshold, clone 2 Gaussian cells located in the region, and compress their scales to one hundredth.

[0038] Preferably, the joint optimization of the sparse compensation Gauss and the corresponding implicit entropy model comprises:​​

[0039] The weighted sum of distortion loss and estimated model code rate loss is taken as the total loss, and the gradient is back-propagated, specifically:

[0040]

[0041] wherein, is the code rate loss estimated from the SH coefficients of the quantized sparse compensated Gaussians, M is the number of quantization values; is the luminance loss; the parameter λ1 balances the trade-off between bit rate and distortion, and controls the model size and reconstruction quality.

[0042] Preferably, the cloned Gaussians ΔG t are distributed around the original Gaussians in a normal distribution, and are optimized in the second training phase.

[0043] Preferably, the reconstruction of the complete Gaussian representation of the current frame using the complete Gaussian representation of the first frame or the previous frame in the buffer, the motion grid of the current frame and the sparse compensated Gaussians, and storing it in the buffer for the reconstruction of the next frame, comprises:

[0044] Once the training of the current frame is completed, i.e. the first phase and the second phase of training are completed, the complete Gaussian representation of the current frame is reconstructed Specifically,

[0045]

[0046] wherein and represent the reconstructed motion grid and the compensated Gaussians of the current frame, respectively, represent the operations of updating μ and R of each Gaussian element in stored in the reference buffer for the reconstruction of the next frame.

[0047] Preferably, the quantization and entropy coding of the trained motion grid and the sparse compensated Gaussians are performed to compress the size of the motion grid and the sparse compensated Gaussians, and obtain a variable bit rate code stream to realize streaming, comprising:

[0048] The quantization of the motion grid and the sparse compensated Gaussians is performed after the above training of each frame, and the formula is:

[0049]

[0050] ​​​​Wherein q is a quantization step, x is all data to be compressed, including the spherical correlation SH coefficient of the Gaussian sphere and the motion grid;

[0051] The quantized result is interval coded to obtain a bit stream, and the bit stream contains variable bit rate model information, and the formula is:

[0052] B t = E (Q (q x) - Q (q min (x)) ; ω t )

[0053] Wherein B t is the code stream corresponding to t time, E represents an entropy encoder, and ω t is the distribution of the corresponding data, which is obtained by statistics after the training is completed.

[0054] Compared with the prior art, the embodiment of the present application has at least one of the following beneficial effects:

[0055] The variable bit rate 4D Gaussian compression method in the embodiment of the present application realizes a wide range of variable bit rates through a single model, while maintaining superior rate distortion performance, and can be used for streaming to adapt to different needs. BRIEF DESCRIPTION OF DRAWINGS

[0056] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the following drawings:

[0057] Figure 1 is a flow chart of the variable bit rate 4D Gaussian compression method in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The present application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made. These all belong to the protection scope of the present application.

[0059] Volume video will play a very important role in people's life. However, at present, there is no good volume video coding framework, and there is no method of end-to-end joint optimization of volume video representation and compression, which will lead to loss of dynamic details and reduction of compression efficiency. At the same time, the volume video trained by a single model is difficult to have flexibility, and it is difficult to freely select multiple rate distortion performances according to the device situation and personal needs at the decoding rendering end. In order to solve this problem, an embodiment of the present application provides a variable bit rate 4D Gaussian compression method, and the specific steps are as follows: Figure 1 as shown in the figure, the main steps are as follows:

[0060] S1: Use the complete Gaussian compressed 3DGS representation of the key frame, i.e. the first frame, estimate the bit rate of the Gaussian compressed 3DGS using an implicit entropy model, and introduce simulated quantization so that the compressed size is variable bit rate;

[0061] By performing the above step S1, an accurate three-dimensional geometry (3DGS) representation of the key frame or the first frame can be obtained, which has extremely high reconstruction quality. In addition, the volume of the 3DGS model is small, and supports variable bit rate.

[0062] S2: Train the first stage of the non-key frame, use the motion mesh and two globally shared lightweight multi-layer perceptron (MLP) to estimate the motion of the inter-frame Gaussian cell, obtain the motion-estimated Gaussian representation, estimate the bit rate of the motion mesh compression using an implicit entropy model, and introduce simulated quantization so that the compressed size is variable bit rate; jointly optimize the motion mesh, two globally shared lightweight MLP, and the corresponding implicit entropy model;

[0063] In the foregoing step S2, based on the previous frame image, motion simulation of the Gaussian sphere is performed to obtain the initial representation of the non-key frame. The motion mesh has a small size, is encoded with variable bit rate, and at the same time ensures the accuracy of motion estimation. For the part that has appeared in the scene, the reconstruction quality is extremely high.

[0064] S3: Train the second stage of the non-key frame, based on the motion-estimated Gaussian representation, use the sparse compensation Gaussian to perform motion compensation on the inter-frame change region, estimate the bit rate of the sparse compensation Gaussian compression using an implicit entropy model, and introduce simulated quantization so that the compressed size is variable bit rate; jointly optimize the sparse compensation Gaussian and the corresponding implicit entropy model;

[0065] In the foregoing step S3, based on the results of step S2, motion compensation processing of the Gaussian sphere is performed, thereby obtaining the final representation of the non-key frame. The newly added Gaussian sphere has a small volume, is encoded with variable bit rate, and has high accuracy of motion compensation, which significantly improves the quality of the entire scene reconstruction.

[0066] S4: Use the complete Gaussian representation of the first frame or the previous frame in the buffer, the motion mesh and the sparse compensation Gaussian of the current frame to reconstruct the complete Gaussian representation of the current frame, and store it in the buffer for the reconstruction of the next frame;

[0067] In the foregoing step S4, the final representation of the non-key frame is reconstructed, which has a small volume, is encoded with variable bit rate, and at the same time ensures the high quality of the scene reconstruction, and prepares for the training work of the subsequent frame.

[0068] S5: quantize and entropy encode the trained motion mesh and sparse compensation Gaussians, compress the motion mesh and sparse compensation Gaussian size, obtain a variable bit rate code stream, and realize streaming.

[0069] Based on the representation of the first frame, and the non-key frame first stage training of cycle S2, the non-key frame second stage training of S3, the current frame reconstruction of S4, and the compression of the trained motion mesh and the sparse compensation Gaussian of S5, the compression and streaming of the entire volumetric video are completed.

[0070] The above embodiments can improve the reconstruction quality and compression rate of the volumetric video, improve the rate-distortion performance, enhance the flexibility, and enable streaming. The problems of large storage, poor reconstruction quality, and inability to stream in the prior art are solved. In the foregoing embodiments, implicit entropy models are used in steps S1, S2, and S3 to accurately calculate the bit rate of each model after compression, and analog quantization technology is integrated to realize variable bit rate of the compressed data size. By using this method, the motion mesh has low entropy characteristics, the amount of data after compression and encoding is further reduced, and the robustness of the model to quantization loss is also enhanced.

[0071] In a preferred embodiment, a preferred scheme of step S1 is provided, specifically:

[0072] A set of Gaussian primitives G is used as an explicit representation of the entire three-dimensional space scene similar to point cloud.

[0073] Each Gaussian primitive is composed of a set of optimizable parameters {μ; R; f; s; α}, where μ is the center position, R is the rotation matrix, f represents the SH coefficient of the view-dependent color c, s is the scaling vector, and α is the transparency. For a point x located in the Gaussian primitive, its spatial distribution is determined by , where ∑=Rss T R T .

[0074] During rendering, the rendering color c of the pixel is calculated by alpha blending the overlapping Gaussians in depth order, specifically:

[0075]

[0076] where α′ i is the transparency of the i-th Gaussian primitive on the image plane, c i is the color of the i-th Gaussian primitive in the viewing direction. N represents the number of Gaussian primitives in the entire three-dimensional space.

[0077] In order to better represent and compress the volumetric video, in a preferred embodiment of the present application, a preferred solution for the first stage training of the volumetric video, i.e. step S2, can be adopted, which comprises the following steps:

[0078] S2.1, motion estimation of the inter-frame Gaussian cells using the motion mesh and two globally shared light-weight multi-layer perceptrons, comprising:

[0079] the complete Gaussian representation of the previous frame loaded from the buffer position encoding of the center position of the Gaussian cell in the middle to map to the multi-band, specifically: position encoding.

[0080] position encoding on the multi-resolution motion mesh trilinear interpolation, generating motion features of different scales, wherein L is the resolution level;

[0081] concatenating the motion features of different scales and inputting them into two shared light-weight multi-layer perceptrons Φ μ and Φ R to obtain the motion estimation of the translation Δμ t and rotation ΔR t of the Gaussian cell from the previous frame to the current frame, realizing accurate motion prediction across multiple scales, capturing necessary transformations and effectively reducing inter-frame redundancy, specifically:

[0082]

[0083] wherein interp(·) represents the interpolation operation of the mesh.

[0084] using the obtained motion estimation and the complete Gaussian representation of the previous frame in the buffer, the Gaussian representation G′ t of the current frame after motion estimation transformation can be obtained, specifically:

[0085]

[0086] wherein C represents fixed parameters including f t-1 , s t-1 and α t-1 ; represents the reconstructed Gaussian representation of the previous frame;

[0087] represents all Gaussian spheres of the previous frame; represents the result obtained by applying all Gaussian spheres of the previous frame to the motion mesh of the current frame; μ t-1 represents the position of the Gaussian sphere of the previous frame, Δμ tdenotes the variable of the Gaussian sphere position of this frame relative to the last frame t denotes the variable of the Gaussian sphere rotation of this frame relative to the last frame t-1 denotes the rotation of the Gaussian sphere of the last frame.

[0088] S2.2, use a compact implicit entropy model to accurately estimate the bit rate of the compressed motion mesh, and introduce a simulated quantization, so that the compressed size is a variable bit rate, including:

[0089] Add uniform noise to the motion mesh and newly added Gaussian primitives to simulate the effect of quantization with a step size of q, so that the training process is robust while preserving the gradient flow;

[0090] Use a small and trainable implicit entropy model to approximate the motion mesh after simulated quantization or sparse compensation Gaussian quantization value probability mass function to estimate the compressed bit rate, specifically:

[0091]

[0092] P CDF is the cumulative distribution function. is the quantized data to be compressed. The entropy model can approximate the probability mass function (PMF) of the quantized data to be compressed by calculating the cumulative distribution function (CDF) of hat{y}.

[0093] S2.3, jointly optimize the motion mesh M t , two globally shared lightweight multi-layer perceptrons Φ μ and Φ R , and the corresponding implicit entropy model, and the weighted sum of the distortion loss and the estimated model code rate loss as the total loss during the process, and the gradient is backpropagated to train the model, specifically:

[0094]

[0095] wherein, is the code rate loss estimated from the quantized motion mesh N is the number of quantization values ; is the photometric loss, c g and are the real and reconstructed colors of the corresponding view, respectively. D-SSIM between the real picture and the rendered result in training, and λ2 is a weight parameter; parameter λ1 balances the trade-off between bit rate and distortion, thereby controlling the model size and reconstruction quality.

[0096] Similarly, in order to better represent and compress the volumetric video, in another preferred embodiment of the present application, a preferred scheme for the second stage training of the volumetric video, i.e. step S3, can adopt the following steps:

[0097] S3.1, using sparse compensation Gauss to compensate the inter-frame change significant area, including:

[0098] In the current frame motion estimation transformed Gauss expression G' t On this basis, identify the suboptimal area that needs to be compensated, mainly the area with significant gradient change and the larger Gauss cell area that undergoes large transformation in motion estimation;

[0099] For the area with significant gradient change, when the gradient exceeds the pre-defined gradient threshold τ g , clone 1 Gauss cell located at this position, denoted as To ensure accurate representation of the newly observed elements;

[0100] For the larger Gauss cell that undergoes large transformation in motion estimation, when the cell translation |Δμ t | and rotation |ΔR t | exceed the pre-defined threshold τ μ and τ R , clone 2 Gauss cells located at this position and compress their scale to one hundredth to To more accurately capture detailed motion dynamics;

[0101] The above newly compensated Gauss cell ΔG t is distributed around the original Gauss cell by And is optimized in the second training stage.

[0102] S3.2, using a compact implicit entropy model to accurately estimate the bit rate of the sparse compensation Gauss compression, and introducing simulated quantization, so that the compressed size is a variable bit rate, including:

[0103] By adding uniform noise to simulate the quantization effect with a step size of q, the training process is robust while preserving the gradient flow;

[0104] Using a small and trainable implicit entropy model to approximate the motion mesh M t The probability mass function of the quantized value after simulated quantization The estimated bit rate after compression is specifically:

[0105]

[0106] S3.3, jointly optimizing sparse compensation Gaussian ΔG t And the corresponding implicit entropy model, the weighted sum of the distortion loss and the estimated model code rate loss in the process is taken as the total loss, and the gradient is back propagated to train the model, specifically:

[0107]

[0108] Wherein, is the code rate loss estimated from the SH coefficients of the quantized sparse compensation Gaussian M is the number of quantization values; is the photometric loss, defined the same as in the first stage; The parameter λ1 balances the trade-off between bit rate and distortion, thereby controlling the model size and reconstruction quality. The similar strategy is also applied to the Gaussian representation of the first frame (key frame).

[0109] In a preferred embodiment, step S4: using the complete Gaussian representation of the previous frame in the buffer, the motion grid and the sparse compensation Gaussian of the current frame to reconstruct the complete Gaussian representation of the current frame, and storing it in the buffer for the reconstruction of the next frame, specifically:

[0110] Once the training of the current frame is completed, the complete Gaussian representation of the current frame is reconstructed Specifically:

[0111]

[0112] Wherein and represent the reconstructed motion grid and the compensation Gaussian of the current frame, represent the operation of updating μ and R of each Gaussian cell in

[0113] Of course, the needs to be stored in the reference buffer for the reconstruction of the next frame.

[0114] The above embodiment has completed the training of the motion grid and the sparse compensation Gaussian. In a preferred embodiment, step S5: quantizing and entropy encoding the trained motion grid and sparse compensation Gaussian to compress the model size, obtain a variable bit rate code stream, and realize streaming, specifically:

[0115] ​​​​After the above training of each frame, the motion mesh and the sparse compensation Gaussian are quantized, and the formula is:

[0116]

[0117] where q is the quantization step, x represents all the data to be compressed, including the spherical correlation coefficients of the Gaussian sphere and the motion mesh

[0118] The quantized results are interval encoded to obtain a bit stream, and the bit stream contains variable bit rate model information, and the formula is:

[0119] B t =E(Q(q·x)-Q(q·min(x));ω t )

[0120] where B t is the code stream corresponding to t time, E represents the entropy encoder, and ω t is the distribution of the corresponding data, which is obtained after the training is completed.

[0121] Due to the low entropy and high reconstruction quality characteristics of the feature mesh, the results obtained after the compression model are very small, and the reconstruction quality is high after recovery. The finally trained neural radiance field model (i.e. the 4D Gaussian compression framework with variable bit rate) can be evaluated by peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) to evaluate its reconstruction quality, and the MB of each frame is used to evaluate its bit rate. In order to comprehensively analyze the rate-distortion (RD) performance, Bjontegaard Delta Bit-Rate (BDBR) and Bjontegaard Delta PSNR (BD-PSNR) are used, and the rendering efficiency is evaluated by calculating the frames per second (FPS).

[0122] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various modifications or changes within the scope of the claims, which does not affect the essential content of the present application. The above preferred features can be used in combination in the case of not conflicting with each other.

Claims

1. A 4D Gaussian compression method with variable code rate, characterized in that, The method comprises the following steps: The first frame is represented by using complete Gaussian compression 3DGS, the bit rate of the Gaussian compression 3DGS is estimated by using an implicit entropy model, and analog quantization is introduced so that the size of the compression is a variable bit rate; In the first stage of training of the non-key frame, motion estimation is performed on the Gaussian primitives between frames by using a motion grid and two globally shared lightweight multi-layer perceptron MLPs, and a Gaussian expression after motion estimation is obtained; The bit rate of the motion grid after compression is estimated by using an implicit entropy model, and analog quantization is introduced so that the size of the compression is a variable bit rate; the motion grid, the two globally shared lightweight MLPs and the corresponding implicit entropy model are jointly optimized; In the second stage of training of the non-key frame, a region of inter-frame change is compensated by using a sparse compensation Gaussian based on the Gaussian expression after motion estimation, the bit rate of the sparse compensation Gaussian after compression is estimated by using an implicit entropy model, and analog quantization is introduced so that the size of the compression is a variable bit rate; the sparse compensation Gaussian and the corresponding implicit entropy model are jointly optimized; The complete Gaussian expression of the current frame is reconstructed by using the complete Gaussian expression of the first frame or the last frame in the buffer, the motion grid and the sparse compensation Gaussian of the current frame, and is stored in the buffer for the reconstruction of the next frame; The motion grid and the sparse compensation Gaussian that have been trained are quantized and entropy encoded, the size of the motion grid and the sparse compensation Gaussian is compressed, a variable bit rate code stream is obtained, and streaming is realized; Based on the representation of the first frame, and the first stage of training of the non-key frame, the second stage of training of the non-key frame, the reconstruction of the current frame, the compression of the trained motion grid and the sparse compensation Gaussian, the compression and streaming of the entire volumetric video are completed.

2. The 4D Gaussian compression method with variable code rate according to claim 1, wherein, The first frame is represented by using complete Gaussian compression 3DGS, which comprises the following steps: The entire three-dimensional space scene is represented by using a group of Gaussian primitives G; each Gaussian cell comprises a set of optimizable parameters {μ; R; f; s; a}, where μ is the center position, R is the rotation matrix, f is the SH coefficients of the view-angle dependent color c, s is the scaling vector, and a is the transparency; For a point x located in a Gaussian cell its spatial distribution is determined by , where ∑ = Rss T R T ; During rendering, the rendering color c of a pixel is calculated by alpha blending the overlapping Gaussian primitives in the order of depth, specifically: wherein a′ i is the projection of the transparency of the i-th Gaussian cell on the image plane, c i is the color of the i-th Gaussian cell in the viewing direction, and N represents the number of Gaussian cells present in the entire three-dimensional space.

3. The variable bit rate 4D Gaussian compression method of claim 1, wherein, The motion estimation is performed on the Gaussian primitives between frames by using a motion grid and two globally shared lightweight multi-layer perceptron MLPs, and a Gaussian expression after motion estimation is obtained, which comprises the following steps: The center position of the Gaussian primitive in the complete Gaussian expression of the last frame loaded from the buffer is position encoded to be mapped to a multi-band; encoding the position in a multi-resolution motion mesh M t performing trilinear interpolation on the motion features of the different scales; The motion features of different scales are connected and input into two shared lightweight multi-layer perceptrons Φ μ and Φ R , to obtain the motion estimation of the Gaussian cell translation and rotation from the previous frame to the current frame. Using the obtained motion estimation and the complete Gaussian representation of the previous frame on the buffer, the Gaussian representation G' of the current frame after motion estimation transformation is obtained t .

4. The method of claim 3, wherein, The bit rate of the motion grid or the sparse compensation Gaussian after compression is estimated by using an implicit entropy model, and analog quantization is introduced so that the size of the compression is a variable bit rate, which comprises the following steps: By adding uniform noise to the motion grid M t or sparse compensation Gauss ΔG t Adding uniform noise to simulate the quantization effect of the step size q, get the quantization value The size of the added noise is consistent with the step size of quantization; Approximating quantized motion mesh using implicit entropy model or sparse compensated gaussian of quantized values probability mass function to estimate the compressed bitrate, in particular: wherein is the probability mass function of the quantized value is the cumulative distribution function of the quantized value is the probability mass function of the quantized value is the cumulative distribution function of the quantized value 5. The method of claim 3, wherein the 4D Gaussian compression method is variable bit rate. The motion grid, the two globally shared lightweight MLPs and the corresponding implicit entropy model are jointly optimized, which comprises the following steps: The weighted sum of the distortion loss and the estimated model code rate loss is taken as the total loss, and the gradient is back propagated, specifically: in, This represents the total loss in the first phase. To extract from the quantized motion mesh The estimated bitrate loss, where N is the quantization value. Quantity; It is a loss of luminosity, c g and These are the real and reconstructed colors from the corresponding viewpoints; It is the D-SSIM evaluation metric between real images and the rendering results in training, λ2 is the weight parameter; parameter λ1 balances the trade-off between bit rate and distortion, and controls the model size and reconstruction quality.

6. The variable bit rate 4D Gaussian compression method of claim 1, wherein, The region of inter-frame change is compensated by using a sparse compensation Gaussian based on the Gaussian expression after motion estimation, which comprises the following steps: Gaussian representation G' of the current frame after motion estimation transform t On this basis, sub-optimal regions requiring compensation are identified, including regions where the gradient exceeds a predetermined threshold, and regions where the translation and rotation of the Gaussian cell exceeds a predetermined threshold; For the region whose gradient exceeds a predetermined threshold, one Gaussian primitive located in the region is cloned; For the region whose translation and rotation of the Gaussian cell exceeds a predetermined threshold, clone 2 Gaussian cells in the region and compress their scale to one hundredth.

7. The variable rate 4D Gaussian compression method of claim 6, wherein, The joint optimization of the sparse compensation Gaussian and the corresponding implicit entropy model comprises: The weighted sum of the distortion loss and the estimated model code rate loss is taken as the total loss, and the gradient is back propagated, specifically: where, is the number of quantized SH coefficients of the sparse compensated Gaussians is the estimated rate loss from the quantized SH coefficients of the sparse compensated Gaussians M is the quantization value is the luminance loss; the parameter λ1balances the trade-off between bit rate and distortion, controlling the model size and the reconstruction quality.

8. The variable bit rate 4D Gaussian compression method of claim 7, wherein, Cloned Gaussians t are distributed around the original Gaussians and are optimized in a second training phase.

9. The variable bit rate 4D Gaussian compression method of claim 1, wherein, The complete Gaussian expression of the current frame is reconstructed using the complete Gaussian expression of the first frame or the last frame in the buffer, the motion grid of the current frame and the sparse compensation Gaussian, and is stored in the buffer for the reconstruction of the next frame, and comprises: Once the training of the current frame is completed, i.e. the first and second stage training is completed, the complete Gaussian representation of the current frame will be reconstructed Specifically: in and These represent the motion mesh and compensated Gaussian mesh reconstructed for the current frame, respectively. Basis of representation right Each Gaussian element in The operation updates μ and R; It is stored in the reference buffer for reconstruction in the next frame.

10. The 4D Gaussian compression method with variable code rate according to claim 1, wherein, The trained motion grid and sparse compensation Gaussian are quantized and entropy encoded to compress the size of the motion grid and the sparse compensation Gaussian, obtain a variable bit rate code stream, and realize streaming, and comprises: After the above training of each frame, the motion grid and the sparse compensation Gaussian are quantized, and the formula is: Where q is the quantization step, and x is all the data to be compressed, including the spherical harmonics SH coefficients of the Gaussian sphere and the motion grid; The quantized results are interval encoded to obtain a bit stream containing variable bit rate model information, and the formula is: B t = E(Q(q · x) - Q(q · min(x)); ω t ) where B t is the code stream corresponding to t time, E represents an entropy encoder, ω t is the distribution of the corresponding data, which is obtained by statistics after the end of training.

Images