Layered representation compression and progressive transmission method of gaussian splash volume video

Abstract

The application provides a layered representation compression and progressive transmission method of Gaussian splat volume video. The layered representation compression method comprises the following steps: extracting scene surface geometry, constructing 4D Gaussian initial parameters as the basic representation of dynamic scene; calculating the importance measure of each Gaussian primitive, and dividing the Gaussian into L layers of progressive details; performing inter-frame modeling on the layered Gaussian through rigid transformation and residual compensation; calculating the average displacement of the Gaussian primitive, and dynamically adjusting the grouping length of the inter-frame modeling; injecting uniform noise simulation quantization error, and estimating the entropy of each attribute by using KDE; simultaneously constructing an end-to-end optimization weighted loss of distortion, code rate and time smoothness, and training the inter-frame modeling; and encoding the trained layered Gaussian parameters into 2D graphs and compressing by using H.264. The bit stream supporting multi-level quality selection is generated by the layered representation compression, the number of layers is selected for decoding according to the requirement, and progressive transmission and dynamic adaptation are realized.

Description

Technical Field

[0001] This invention relates to the field of volumetric video compression and streaming technology, specifically to a hierarchical characterization compression and progressive transmission method for Gaussian splash volumetric video, and also to a corresponding hierarchical characterization compression system and progressive transmission system. Background Technology

[0002] Volumetric video, with its free-viewpoint navigation capabilities, provides immersive 3D experiences for scenarios such as virtual reality and telemedicine. However, its high bandwidth requirements and real-time decoding challenges limit its widespread application. Existing dynamic neural radiation field methods have significant limitations when handling long sequences of dynamic scenes. While 3D Gaussian and its extensions achieve realistic rendering, they face the parameter explosion problem when modeling long sequences (directly storing features for each frame leads to a huge model size). For example, when the 3D Gaussian splatting method for static scenes is extended to dynamic scenes, Gaussian parameters need to be preloaded for the entire sequence, severely limiting the feasibility of streaming. Existing schemes for dynamic 3D Gaussian representations mainly fall into two categories: one models Gaussian properties as time functions, achieving high rate-distortion performance but ignoring streaming requirements; the other uses frame-by-frame rigid motion estimation, but the fixed reference frame leads to error accumulation in large motion scenes.

[0003] In addition, existing compression methods lack joint optimization of representation and compression: for example, the HPC method encodes dynamic scenes through residual mesh, but the decoding latency is high and cannot meet the needs of real-time applications; although 4DGC attempts joint optimization, it is not robust enough for large motion scenes.

[0004] The more fundamental problem is that traditional video coding frameworks (such as H.264) are difficult to apply directly to the high-dimensional feature space of neural radiation fields, while existing dynamic Gaussian compression methods generally lack hierarchical rate-distortion (RD) supervision mechanisms, resulting in the loss of dynamic details after compression. Therefore, how to achieve hierarchical representation of dynamic scenes, motion adaptive modeling, and end-to-end compression optimization in a single model remains a critical technical bottleneck that urgently needs to be overcome in the field of volumetric video streaming. Summary of the Invention

[0005] To address the aforementioned shortcomings in the prior art, this invention provides a hierarchical characterization compression and progressive transmission method for Gaussian splash volumetric video, and also relates to a corresponding hierarchical characterization compression system and progressive transmission system.

[0006] According to one aspect of the present invention, a hierarchical characterization compression method for Gaussian splash volume video is provided, comprising:

[0007] Extract the scene surface geometry and construct 4D Gaussian initial parameters as the basic representation of the dynamic scene;

[0008] Based on the basic representation of the dynamic scene, the importance metric of each Gaussian initial parameter is calculated, and the Gaussian is divided into L layers of progressive detail to obtain the layered Gaussian keyframe representation.

[0009] Based on the hierarchical Gaussian keyframe representation, inter-frame modeling is performed through rigid transformation and residual compensation;

[0010] Calculate the average displacement of the Gaussian primitives and dynamically adjust the grouping length for inter-frame modeling;

[0011] Uniform noise is injected to simulate quantization error, and KDE is used to estimate the entropy of each attribute to guide compression efficiency. At the same time, a weighted loss that optimizes distortion, bit rate and temporal smoothness is constructed to train inter-frame modeling.

[0012] The trained hierarchical Gaussian parameters are encoded into a 2D image and compressed using H.264 to complete the volumetric video compression.

[0013] According to a second aspect of the present invention, a hierarchical characterization compression system for Gaussian splash volume video is provided, comprising:

[0014] The parameter initialization module is used to extract the scene surface geometry and construct 4D Gaussian initial parameters as the basic representation of the dynamic scene.

[0015] The Gaussian model construction module calculates the importance metric of each Gaussian initial parameter based on the basic representation of the dynamic scene, divides the Gaussian into L layers of progressive detail, and obtains the layered Gaussian keyframe representation; for the layered Gaussian keyframe representation, inter-frame modeling is performed through rigid transformation and residual compensation; the average displacement of the Gaussian primitives is calculated, and the grouping length of the inter-frame modeling is dynamically adjusted.

[0016] The Gaussian model training module is used to inject uniform noise to simulate quantization error and use KDE to estimate the entropy of each attribute to guide compression efficiency; at the same time, it constructs an end-to-end weighted loss to optimize distortion, bit rate and temporal smoothness and trains the inter-frame model.

[0017] The dynamic scene compression module encodes the trained hierarchical Gaussian parameters into a 2D image and compresses it using H.264 to achieve volumetric video compression.

[0018] According to a third aspect of the present invention, a progressive transmission method for Gaussian splash volume video implemented using the layered characterization compression method described above is provided, comprising:

[0019] Based on the aforementioned volumetric video compression method, a bitstream supporting multi-level quality selection is generated;

[0020] The bitstream is decoded at the desired number of layers to achieve progressive transmission and dynamic adaptation.

[0021] According to a fourth aspect of the present invention, a progressive transmission system for Gaussian splash volume video implemented using the layered characterization compression method described above is provided, comprising:

[0022] A bitstream generation module that generates a bitstream that supports multi-level quality selection based on the volumetric video compression method;

[0023] The decoding module is used to select the number of layers to decode the bitstream as needed, so as to achieve progressive transmission and dynamic adaptation.

[0024] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0025] This invention provides a hierarchical representation compression and progressive transmission method for Gaussian splash volumetric video. It employs a progressive streaming framework for volumetric video, supporting progressive coding through an importance-aware Gaussian hierarchical representation. This allows users to adapt to various terminals by selecting different numbers of bitstreams at different levels. Our method can achieve multiple rendering performance levels after a single model and compression. Our compression uses H.264, ensuring compatibility with existing rendering pipelines and enabling real-time decoding and rendering on mobile devices.

[0026] The hierarchical representation, compression, and progressive transmission method for Gaussian splash volume video provided by this invention can achieve accurate non-keyframe representation through Gaussian rigid motion and residual deformation.

[0027] The hierarchical characterization compression and progressive transmission method for Gaussian splash volume video provided by this invention can dynamically select key frames based on the motion of the actual sequence through motion-aware adaptive Gaussian grouping, thereby improving reconstruction quality.

[0028] The hierarchical representation compression and progressive transmission method for Gaussian splash volumetric video provided by this invention employs an end-to-end entropy optimization learning strategy, including hierarchical RD supervision and attribute-specific entropy modeling. Hierarchical RD supervision optimizes the RD performance of each Gaussian layer, while attribute-specific entropy modeling captures the distribution of each Gaussian attribute. KDE is used for keyframes, and Gaussian distribution is used to estimate the residual values ​​of non-keyframes, allowing for more accurate estimation of attribute entropy and thus optimizing the model's RD performance. Attached Figure Description

[0029] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0030] Figure 1This is a flowchart illustrating the workflow of a layered characterization and compression method for Gaussian splash volume video in a preferred embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the components of a layered characterization and compression system for Gaussian splash volume video in a preferred embodiment of the present invention.

[0032] Figure 3 This is a flowchart illustrating the progressive transmission method for Gaussian splash volume video in a preferred embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of the constituent modules of a progressive transmission system for Gaussian splash volume video in a preferred embodiment of the present invention.

[0034] Figure 5 This is a schematic diagram illustrating the overall process of layered characterization compression and progressive transmission of Gaussian splash volume video in a specific application example of the present invention. Detailed Implementation

[0035] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0036] Existing methods for compressing and transmitting volumetric video often suffer from problems such as redundant modeling parameters for long-sequence dynamic scenes, low real-time decoding efficiency, and the inability of a single model to adapt to variable bitrate requirements. To address these issues, one embodiment of this invention provides a hierarchical representation compression method for Gaussian splash volumetric video. This method, based on an end-to-end compression optimization framework using hierarchical 4D Gaussian representation, achieves high-fidelity reconstruction and efficient compressed transmission of dynamic scenes.

[0037] Specifically, such as Figure 1 As shown, the hierarchical characterization compression method for Gaussian splash volume video provided in this embodiment may include:

[0038] S1, extract the scene surface geometry and construct 4D Gaussian initial parameters as the basic representation of the dynamic scene;

[0039] S2, based on the basic representation of the dynamic scene, calculates the importance metric of each Gaussian primitive (i.e. each Gaussian initial parameter), divides the Gaussian into L layers of progressive detail, and obtains the layered Gaussian keyframe representation;

[0040] S3, based on the hierarchical Gaussian keyframe representation, performs inter-frame modeling through rigid transformation and residual compensation;

[0041] S4 calculates the average displacement of the Gaussian primitive and dynamically adjusts the grouping length for inter-frame modeling;

[0042] S5 injects uniform noise to simulate quantization error and uses KDE to estimate the entropy of each attribute to guide compression efficiency; at the same time, it constructs an end-to-end weighted loss to optimize distortion, bit rate and temporal smoothness, and trains the inter-frame model.

[0043] S6 encodes the trained hierarchical Gaussian parameters into a 2D graph and uses H.264 compression to complete volumetric video compression for dynamic scenes.

[0044] In some preferred embodiments, the above-mentioned S1 may further include:

[0045] S11. The NeuS2 learning method is used to extract the geometric information of the scene surface and initialize the Gaussian distribution parameters, including: position μ, covariance ∑, opacity α and color c, to obtain the 4D Gaussian initial parameters.

[0046] S12 uses 4D Gaussian initial parameters to replace the traditional neural radiation field as the basic representation of dynamic scenes.

[0047] In some preferred embodiments, the above-mentioned S2 may further include:

[0048] S21, calculate the importance metric ψ for each Gaussian primitive as follows:

[0049] ψ=α+λ ψ S

[0050] In the formula, α represents opacity, and λ represents... ψ is a tradeoff coefficient used to balance the two influencing factors, and S is the spatial volume;

[0051] S22, the importance measure ψ is sorted in descending order, and the Gaussian is divided into L layers; among them, the first layer, as the base layer, retains the core structure of the scene, and the first to L layers, as the higher layers, progressively enhance the details.

[0052] Furthermore, in the above steps, position represents the coordinates of the Gaussian sphere in three-dimensional space, covariance represents the size and rotation angle, and opacity and color are used to calculate the position and color of the Gaussian sphere's projection on the plane.

[0053] In some preferred embodiments, the above-mentioned S3 may further include:

[0054] Using the hierarchical Gaussian keyframe representation as the reference basis for training subsequent inter-frame models, the complex motion of each layer of Gaussian keyframes is captured through hierarchical motion modeling, and frame differences are decomposed into rigid transformations and residual compensation to maintain temporal coherence, thus performing inter-frame modeling; where:

[0055] Frame differences are decomposed into rigid transformation and residual compensation, including:

[0056] S311, utilizing the Gaussian position of the previous frame As input, through a multi-resolution hash grid Capture motion features at different scales Then through a lightweight MLP network and Calculate the translation of the current frame and rotation Using the translation of the current frame and rotation Update Gaussian position and orientation, process the overall rigid displacement of the camera or object, and estimate the Gaussian rigid transformation between frames;

[0057] S312, based on rigid transformation, uses residual compensation to predict local detail deformation parameters, and obtains scaled residuals. Opacity residual and color residual The Gaussian properties (i.e. Gaussian distribution parameters) are updated using local detail deformation parameters to compensate for non-rigid dynamic details;

[0058] S313 decouples global motion from local details through hierarchical modeling using rigid transformation and residual compensation, thus completing inter-frame modeling.

[0059] In some preferred embodiments, the above-mentioned S4 may further include: including:

[0060] Set displacement threshold τ μ According to the Gaussian average translation Automatically divide group length; when Shorten the group leader period to improve reference quality; when The time extension group length is used to optimize compression efficiency, thereby dynamically adjusting the group length for inter-frame modeling.

[0061] In some preferred embodiments, the above-mentioned S5 may further include:

[0062] S51 introduces uniform noise to simulate quantization error and uses KDE to probabilistically model Gaussian properties, calculates the entropy rate of Gaussian properties, and guides compression efficiency.

[0063] S52, for the optimization of each set of Gaussian distribution parameters, construct a system incorporating the color error term L. color With entropy rate term L rate_key The loss function is used to form a weighted sum; by optimizing the Gaussian distribution parameters layer by layer, the optimal bit rate and reconstruction quality of each layer are obtained;

[0064] S53 optimizes subsequent Gaussian sequences within a group based on keyframes; specifically, during rigid motion, only uniform noise is used to simulate quantization error, relying on color error L. color Supervision is conducted; color loss L is introduced into the residual compensation process. color Entropy rate loss L rate_inter and smoothness loss L reg This constitutes a joint optimization objective, enhancing time consistency and reducing residual magnitude, thereby further optimizing reconstruction quality and storage efficiency.

[0065] In some preferred embodiments, the above-mentioned S6 may further include:

[0066] Differential quantization and channel tiling are performed on the Gaussian attributes of each layer to form a 2D single-channel image sequence that maintains spatial continuity. After alignment by group, layer and channel, the image is encoded and compressed using H.264.

[0067] Based on the same inventive concept, an embodiment of the present invention also provides a hierarchical characterization compression system for Gaussian splash volume video.

[0068] Specifically, such as Figure 2 As shown, the hierarchical characterization compression system for Gaussian splash volume video provided in this embodiment may include:

[0069] The parameter initialization module is used to extract the scene surface geometry and construct 4D Gaussian initial parameters as the basic representation of the dynamic scene.

[0070] The Gaussian model building module is based on the basic representation of the dynamic scene. It calculates the importance metric of each Gaussian primitive and divides the Gaussian into L layers of progressive detail. For the layered Gaussian, it performs inter-frame modeling through rigid transformation and residual compensation. It calculates the average displacement of the Gaussian primitive and dynamically adjusts the grouping length of the inter-frame modeling.

[0071] The Gaussian model training module is used to inject uniform noise to simulate quantization error and use KDE to estimate the entropy of each attribute to guide compression efficiency; at the same time, it constructs an end-to-end weighted loss to optimize distortion, bit rate and temporal smoothness and trains the inter-frame model.

[0072] The dynamic scene compression module encodes the trained hierarchical Gaussian parameters into a 2D image and compresses it using H.264 to achieve volumetric video compression for dynamic scenes.

[0073] The following describes in detail, with reference to preferred embodiments, the specific implementation methods of each functional module constituting the system provided in the above embodiments of the present invention.

[0074] The system provided by this preferred embodiment includes the following functional modules:

[0075] The parameter initialization module uses NeuS2 to extract surface geometry and construct 4D Gaussian initial parameters as the basic representation of the dynamic scene.

[0076] The Gaussian model construction module is based on the basic representation of the dynamic scene. It calculates the importance metric of each Gaussian initial parameter, divides the Gaussian into L layers of progressive detail, and obtains the layered Gaussian keyframe representation. For the layered Gaussian keyframe representation, it performs inter-frame modeling through rigid transformation and residual compensation. It calculates the average displacement of the Gaussian primitives and dynamically adjusts the grouping length of the inter-frame modeling.

[0077] The Gaussian model training module injects uniform noise during training to simulate quantization error and uses KDE to estimate the entropy of each attribute to guide the improvement of compression efficiency; at the same time, it constructs an end-to-end weighted loss to optimize distortion, bit rate and temporal smoothness.

[0078] The dynamic scene compression module encodes the trained hierarchical Gaussian parameters into a 2D image and compresses it with H.264. During decoding, the number of layers is selected as needed to achieve progressive transmission and dynamic adaptation.

[0079] In some preferred embodiments, the parameter initialization module described above is further implemented by including:

[0080] A 4D Gaussian primitive is used to replace the traditional neural radiation field as the basic representation of the dynamic scene. The geometric information of the scene surface is extracted by NeuS2, and the position μ, covariance ∑, opacity α, color c and other parameters of the Gaussian distribution are initialized to construct the initial volume representation of the dynamic scene.

[0081] In some preferred embodiments, the Gaussian model construction module is further implemented in the following ways:

[0082] The importance metric ψ (combining spatial volume S and opacity α) for each Gaussian primitive is calculated using the formula ψ = α + λ. ψ S. Sort Gaussians by ψ from largest to smallest, dividing them into L layers. The base layer (such as layer 1) preserves the core structure of the scene, while the higher layers (such as layers 2-6) progressively enhance details to support progressive real-time rendering with on-demand decoding.

[0083] Based on hierarchical Gaussian keyframe representations, these representations are used as a reference basis for training subsequent inter-frame movements. Hierarchical motion modeling effectively captures complex motions and decomposes frame differences into rigid transformations and residual deformations to maintain temporal coherence. Specifically, the Gaussian rigid transformation between frames is first estimated using the Gaussian position of the previous frame. As input, through a multi-resolution hash grid Capture motion features at different scales Then through a lightweight MLP network and Calculate the translation of the current frame and rotation Use it to update the Gaussian position and orientation to handle the overall rigid displacement of the camera or object.

[0084] Furthermore, based on the rigid transformation, residual compensation is used to predict local detail deformation parameters, including scaled residuals. Opacity residual and color residual Used to update Gaussian properties, compensate for non-rigid dynamic details such as clothing wrinkles and object deformation, reduce visual artifacts, and maintain temporal consistency. Through hierarchical modeling of rigid transformation and residual deformation, global motion is decoupled from local details. Rigid transformation handles large-scale displacements, while residual deformation captures subtle local changes. The combination of the two achieves a compact representation of dynamic scenes, further reducing the redundancy of motion parameters.

[0085] Furthermore, to address the reference failure caused by motion accumulation in dynamic scenes and the redundancy and error problems brought about by fixed grouping, an adaptive Gaussian grouping for motion perception is proposed: setting a displacement threshold τ μ According to the Gaussian average translation Automatically divides group lengths, especially during intense exercise. Shorten the set length to improve reference quality, during stable movement The extension group optimizes compression efficiency, thereby enabling flexible time modeling and efficient reconstruction.

[0086] In some preferred embodiments, the Gaussian model training module described above is further implemented in the following ways:

[0087] During training, an end-to-end entropy optimization training scheme is employed, achieving joint optimization of bit rate and reconstruction quality (RD) through differentiability and attribute-specific entropy modeling. Uniform noise is introduced during training to simulate quantization error, and KDE is used to estimate the entropy of each attribute to guide compression efficiency improvements. For optimization of each set of key reference frames, the loss function incorporates the color error L... color , and the entropy rate term L rate_key The Gaussian parameters are optimized layer by layer to ensure optimal RD performance at each level.

[0088] Furthermore, based on keyframes, subsequent Gaussian sequences within a group are optimized. To improve the accuracy and compactness of each layer, the rigid motion process uses only analog quantization and relies on L... color Supervision is performed. A color loss L is introduced during the residual optimization process. color Entropy rate loss L rate_inter and smoothness loss L reg This enhances time consistency and reduces residual amplitude, thereby improving reconstruction quality and storage efficiency.

[0089] In some preferred embodiments, the dynamic scene compression module described above is further implemented by including:

[0090] After training, the Gaussian attributes of each layer are differentially quantized and channel-tiled to form a 2D single-channel image sequence that maintains spatial continuity. This sequence is then aligned by group, layer, and channel and compressed using H.264 encoding. This encoding compression generates a bitstream that supports multi-level quality selection, further enabling on-demand adaptive decoding, smooth quality transitions, and real-time rendering.

[0091] It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding components in the system. Those skilled in the art can refer to the technical solution of the system to implement the steps of the method, and can also refer to the technical solution of the method to implement the composition of the system. That is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, which will not be elaborated here.

[0092] Based on the hierarchical characterization compression method and system provided in the above embodiments of the present invention, an embodiment of the present invention also provides a progressive transmission method for Gaussian splash volume video implemented using the above hierarchical characterization compression method or system.

[0093] Specifically, such as Figure 3 As shown, the progressive transmission method for Gaussian splash volume video provided in this embodiment may include:

[0094] M1, based on a hierarchical characterization compression method, generates a bitstream that supports multi-level quality selection;

[0095] M2 allows for the selection of the number of layers to decode the bitstream as needed, enabling progressive transmission and dynamic adaptation.

[0096] In the above embodiments of the present invention, decoding is the process of restoring the bitstream to Gaussian properties, including decoding the H.264 compressed bitstream into a 2D image using H.264, and then reassembling it into layered Gaussian parametric encoding. Progressive transmission refers to the fact that since the Gaussian parameters are explicitly layered, the 2D image and bitstream are also layered, allowing for free selection of layers during transmission; lower levels can be transmitted first, followed by higher levels. Dynamic adaptation means that different numbers of layers can be received on different devices, striking a balance between rendering quality and data processing volume.

[0097] Based on the volumetric video compression method and system provided in the above embodiments of the present invention, an embodiment of the present invention also provides a progressive transmission system for Gaussian splash volumetric video implemented using the above-mentioned hierarchical characterization compression method or system.

[0098] Specifically, such as Figure 4As shown, the progressive transmission system for Gaussian splash volume video provided in this embodiment may include:

[0099] The bitstream generation module generates bitstreams that support multi-level quality selection based on volumetric video compression methods.

[0100] The decoding module is used to select the number of layers to decode the bitstream as needed, so as to achieve progressive transmission and dynamic adaptation.

[0101] The technical solution provided by the above embodiments of the present invention will be further described in detail below with reference to a specific application example.

[0102] Building on previous work, volumetric video, with its free-viewpoint navigation capabilities, provides immersive 3D experiences for applications such as virtual reality and remote collaboration. However, its high bandwidth requirements and real-time decoding challenges severely restrict its widespread application. Traditional dynamic neural radiation field methods face issues of parameter redundancy and low compression efficiency when modeling long sequences of dynamic scenes. While existing 3D Gaussian and its extensions achieve high-quality rendering, they lack an end-to-end joint optimization mechanism for dynamic modeling and compressed transmission, resulting in loss of dynamic details, compression rate distortion, and poor performance, making it difficult to meet the requirements of streaming transmission and adaptive decoding.

[0103] This specific application example addresses the aforementioned problems by employing the hierarchical representation compression method and progressive transmission method for Gaussian splash volumetric video provided in the above embodiments of the present invention to achieve hierarchical 4D Gaussian representation compression and progressive volumetric video stream transmission.

[0104] Specifically, the hierarchical characterization compression method and progressive transmission method for Gaussian splash volumetric video involved in this specific application example have the following overall process: Figure 5 As shown, it includes the following steps:

[0105] Step S1: Use NeuS2 to extract surface geometry and construct 4D Gaussian initial parameters as the basic representation of the dynamic scene.

[0106] The traditional neural radiation field is replaced by a 4D Gaussian primitive as the basic representation of the dynamic scene. First, high-quality keyframes are reconstructed as references for subsequent frames. Then, the geometric information of the scene surface is extracted using NeuS2, and parameters such as the position μ, covariance ∑, opacity α, and color c of the Gaussian distribution are initialized to construct the initial volume representation of the dynamic scene.

[0107] Step S2: Calculate the importance metric for each Gaussian primitive and divide the Gaussian primitive into L layers of progressive detail.

[0108] To address the issue of high data volume in 3DGS, the importance metric ψ (combining spatial volume S and opacity α) for each Gaussian primitive is calculated using the formula ψ = α + λ. ψS. Sort the Gaussians into L layers according to ψ from largest to smallest. To maintain a balance between volume and opacity, λ... ψ Set to 1×10 5 The Gaussian representation is divided into L=6 layers. The base layer (such as layer 1) retains the core structure of the scene, while the higher layers (such as layers 2-6) progressively enhance details. The client can dynamically select the number of decoding layers based on network conditions and computing resources to ensure smooth playback and balance transmission overhead with reconstruction fidelity. This hierarchical representation also supports progressive rendering, allowing for efficient reconstruction of the scene at different levels of detail.

[0109] Step S3: For the layered Gaussian, perform inter-frame modeling in two steps, including rigid transformation and residual compensation.

[0110] Based on hierarchical Gaussian keyframe representations, these representations are used as a reference basis for training subsequent inter-frame movements. Hierarchical motion modeling effectively captures complex motions and decomposes frame differences into rigid transformations and residual deformations to maintain temporal coherence. Specifically, the Gaussian rigid transformation between frames is first estimated using the Gaussian position of the previous frame. As input, through a multi-resolution hash grid Capture motion features at different scales Then through a lightweight MLP network and Calculate the translation of the current frame and rotation Right now:

[0111]

[0112] Through equations and Update the Gaussian position and orientation to handle the overall rigid displacement of the camera or object.

[0113] Building upon the rigid transformation, residual compensation is further used to predict local detail deformation parameters, including scaled residuals. Opacity residual and color residual Using equations The Gaussian properties are updated to compensate for non-rigid dynamic details such as clothing wrinkles and object deformation, reducing visual artifacts and maintaining temporal consistency. By using hierarchical modeling of rigid transformation and residual deformation, global motion is decoupled from local details. Rigid transformation handles large-scale displacements, while residual deformation captures subtle local changes. The combination of the two achieves a compact representation of dynamic scenes, further reducing the redundancy of motion parameters.

[0114] Step S4: Calculate the average displacement of the Gaussian primitive and dynamically adjust the grouping length of the inter-frame modeling.

[0115] To address the issues of reference failure due to motion accumulation in dynamic scenes and redundancy and errors caused by fixed grouping, an adaptive Gaussian grouping method for motion sensing is proposed: setting a displacement threshold τ. μ According to the Gaussian average translation Automatic group length division:

[0116] when For scenes with large movements, such as characters moving quickly, use short grouping intervals (e.g., every 5 frames) to avoid rendering distortion caused by error accumulation.

[0117] when For stable scenarios: Increase the grouping interval (e.g., every 20 frames) to reduce redundancy in motion parameter storage.

[0118] This enables flexible time modeling and efficient reconstruction.

[0119] Step S5: During training, uniform noise is injected to simulate quantization error, and KDE is used to estimate the entropy of each attribute to guide the improvement of compression efficiency; at the same time, an end-to-end weighted loss is constructed to optimize distortion, bit rate and temporal smoothness.

[0120] During training, an end-to-end entropy optimization training scheme was used, achieving joint optimization of bit rate and reconstruction quality (RD) through differentiability and attribute-specific entropy modeling. Uniform noise was introduced during the keyframe optimization stage. To enhance quantization robustness, U represents a uniform distribution, q represents the quantization step size, and KDE (kernel density estimation) is used to probabilistically model Gaussian attributes (such as rotation, scaling, color, and opacity) and calculate their entropy rate. The loss function consists of the color (luminosity) error term L. color And entropy rate term L rate_key Composition, in the form of a weighted sum:

[0121]

[0122] Among them, L key λ represents the total loss for keyframe reconstruction. l This represents the hyperparameter that balances the error of each layer. λ represents the color reconstruction error of the l-th layer. rate_key The weighted hyperparameters representing the keyframe entropy loss are... This represents the entropy estimated from Gaussian properties at the same level using KDE. By optimizing layer by layer, optimal rate-distortion (RD) performance is ensured at each level, thereby improving overall compression efficiency and rendering quality.

[0123] Then, inter-frame optimization was performed on subsequent frames within the group based on keyframes. For Gaussian position and rotation parameters (i.e., rigid motion processes), analog quantization was used, relying solely on color loss L. colorSupervision is performed without introducing entropy constraints; however, during the residual deformation stage, an entropy rate loss L is further introduced to improve accuracy and compactness. rate_inter and time smoothing loss L reg This constitutes a joint optimization objective:

[0124]

[0125] Among them, L inter The total loss for non-keyframes, To estimate the bit rate of entropy of residual attributes (scaling, features, opacity) from Gaussian attributes at the same level using KDE, λ rate_inter The weighted hyperparameters for the keyframe entropy loss are... To apply L1 regularization to the residual values ​​to enhance time consistency and reduce residual amplitude, λ reg is a weighted hyperparameter for non-keyframe entropy loss. This joint training strategy significantly improves reconstruction quality and compression efficiency, obtaining a compact and high-fidelity hierarchical 4D Gaussian representation, supporting efficient volumetric video compression and storage transmission.

[0126] Step S6: Encode the trained hierarchical Gaussian parameters into a 2D graph and compress it with H.264. Select the number of layers as needed during decoding to achieve progressive transmission and dynamic adaptation.

[0127] After training, the Gaussian attributes of each level are explicitly separated, and a differential quantization strategy is adopted: higher-precision uint16 or uint32 is used for position parameters (such as μ) to reduce error sensitivity, while other attributes (such as scaling s, color f, and opacity α) are uniformly quantized using uint8. For ease of compression and transmission, each attribute channel is independently unfolded into a 2D single-channel image that maintains spatial continuity, and then organized into time-series frames after alignment by group, level, and channel. Subsequently, these sequences are efficiently compressed using an H.264 video encoder to generate a progressive bitstream that supports multi-level quality selection. The client can dynamically receive and decode the data of the corresponding level according to bandwidth and computing power, achieving smooth quality transitions, flexible adaptive viewing experience, and real-time rendering.

[0128] The hierarchical representation compression method and progressive transmission method for Gaussian splash volume video provided in the above embodiments of the present invention achieve high-fidelity reconstruction and efficient compression and transmission through the joint design of hierarchical representation, motion modeling, adaptive grouping, and end-to-end entropy optimization. The present invention first constructs a perceptually weighted hierarchical 4D Gaussian representation, dividing Gaussian primitives into L layers based on the importance of geometric volume and opacity. The base layer preserves the scene structure, while higher layers progressively enhance details, supporting on-demand selection of the number of layers during decoding to achieve progressive transmission and dynamic adaptation. For dynamic scenes, a hierarchical motion modeling strategy is proposed, decomposing inter-frame motion into rigid transformation and residual deformation. Combined with a motion-aware adaptive grouping mechanism, the reference frame is dynamically updated when the average displacement exceeds a threshold, balancing error accumulation and data redundancy. Noise perturbations in the compression process are simulated through differentiable quantization, and an attribute entropy model is constructed by combining kernel density estimation. Rendering distortion, bitrate estimation, and temporal smoothness constraints are integrated into a unified loss function, achieving joint optimization of representation and compression. Finally, the hierarchical Gaussian features are converted into a progressive bitstream through standard video coding, enabling real-time decoding and rendering on mobile devices. This invention breaks through the fixed bitrate limitation of traditional volumetric video compression, reducing storage and transmission costs while maintaining high-resolution details. It provides an efficient solution for scenarios such as free-viewpoint video streaming and AR / VR content distribution. Experiments have verified that its rate-distortion performance is superior to existing methods in various dynamic scenarios.

[0129] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0130] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A hierarchical characterization and compression method for Gaussian splash volumetric videos, characterized in that, include: Extract the scene surface geometry and construct 4D Gaussian initial parameters as the basic representation of the dynamic scene; Based on the fundamental representation of the dynamic scene, the importance metric of each Gaussian initial parameter is calculated, and the Gaussian is divided into L layers of progressive detail to obtain the layered Gaussian keyframe representation. Specifically, this includes: calculating the importance metric of each Gaussian primitive. for: = + S In the formula, For opacity, As a weighting factor, S For spatial volume; for the measure of importance The Gaussian keyframes are divided into L layers, arranged from largest to smallest. The first layer, which serves as the base layer, retains the core structure of the scene, while the layers from the first to the Lth layer progressively enhance the details, resulting in a layered Gaussian keyframe representation. Based on the hierarchical Gaussian keyframe representation, inter-frame modeling is performed through rigid transformation and residual compensation. Specifically, based on the rigid transformation, residual compensation is used to predict local detail deformation parameters, resulting in scaled residuals. Opacity residual and color residual The Gaussian distribution parameters are updated using the local detail deformation parameters to compensate for non-rigid dynamic details; Calculate the average displacement of the Gaussian primitive and dynamically adjust the grouping length for inter-frame modeling, specifically including: setting a displacement threshold. According to the Gaussian average translation Automatically divide group length; when > Shorten the group leader period to improve reference quality; when ≤ The time extension group length is used to optimize compression efficiency, thereby dynamically adjusting the group length for inter-frame modeling; Uniform noise is injected to simulate quantization error, and KDE is used to estimate the entropy of each attribute to guide compression efficiency. Simultaneously, an end-to-end weighted loss is constructed to optimize distortion, bitrate, and temporal smoothness, and used to train inter-frame modeling. Specifically, this includes optimizing each set of Gaussian distribution parameters and constructing a model incorporating color error. And entropy rate term The loss function is used to form a weighted sum; the optimal bit rate and reconstruction quality at each level are obtained by optimizing the Gaussian distribution parameters layer by layer; subsequent Gaussian distributions within the group are optimized based on keyframes; in the rigid motion process, only uniform noise is used to simulate quantization error, and color error is relied upon. Supervision is conducted; color loss is introduced into the residual compensation process. Entropy loss and smoothness loss This constitutes a joint optimization objective, enhancing time consistency and reducing residual magnitude, thereby further optimizing reconstruction quality and storage efficiency. The trained hierarchical Gaussian parameters are encoded into a 2D image and compressed using H.264 to complete the volumetric video compression.

2. The hierarchical characterization and compression method for Gaussian splash volume video according to claim 1, characterized in that, The extraction of scene surface geometry and the construction of 4D Gaussian initial parameters as the basic representation of the dynamic scene include: The NeuS2 learning method is used to extract the geometric information of the scene surface and initialize Gaussian distribution parameters, including: location. Covariance Opacity Given color c, obtain the initial parameters of the 4D Gaussian; The 4D Gaussian initial parameters are used to replace the traditional neural radiation field as the basic representation of the dynamic scene.

3. The hierarchical characterization and compression method for Gaussian splash volume video according to claim 1, characterized in that, The hierarchical Gaussian keyframe representation, through rigid transformation and residual compensation, performs inter-frame modeling and also includes: Using the hierarchical Gaussian keyframe representation as the reference basis for training subsequent inter-frame models, the complex motion of each layer of Gaussian keyframes is captured through hierarchical motion modeling, and frame differences are decomposed into rigid transformations and residual compensation to maintain temporal coherence, thus performing inter-frame modeling; where: The decomposition of frame differences into rigid transformation and residual compensation includes: Using the Gaussian position of the previous frame = As input, through a multi-resolution hash grid = Capture motion features at different scales = Then through a lightweight MLP network and lightweight MLP networks Calculate the translation of the current frame respectively and rotation Using the translation of the current frame and rotation Update Gaussian position and orientation, process the overall rigid displacement of the camera or object, and estimate the Gaussian rigid transformation between frames; By using hierarchical modeling with rigid transformation and residual compensation, global motion is decoupled from local details, thus completing inter-frame modeling.

4. The hierarchical characterization and compression method for Gaussian splash volume video according to claim 1, characterized in that, The step of encoding the trained hierarchical Gaussian parameters into a 2D graph and compressing it using H.264 includes: Differential quantization and channel tiling are performed on the Gaussian attributes of each layer to form a 2D single-channel image sequence that maintains spatial continuity. After alignment by group, layer and channel, the image is encoded and compressed using H.

264.

5. A hierarchical characterization compression system for implementing the hierarchical characterization compression method according to any one of claims 1-4, characterized in that, include: The parameter initialization module is used to extract the scene surface geometry and construct 4D Gaussian initial parameters as the basic representation of the dynamic scene. The Gaussian model construction module calculates the importance metric of each Gaussian initial parameter based on the basic representation of the dynamic scene, divides the Gaussian into L layers of progressive detail, and obtains the layered Gaussian keyframe representation; for the layered Gaussian keyframe representation, inter-frame modeling is performed through rigid transformation and residual compensation; the average displacement of the Gaussian primitives is calculated, and the grouping length of the inter-frame modeling is dynamically adjusted. The Gaussian model training module is used to inject uniform noise to simulate quantization error and uses KDE to estimate the entropy of each attribute to guide compression efficiency. Simultaneously, an end-to-end weighted loss for optimizing distortion, bitrate, and temporal smoothness is constructed to train inter-frame modeling; The dynamic scene compression module encodes the trained hierarchical Gaussian parameters into a 2D image and compresses it using H.264 to achieve volumetric video compression.

6. A progressive transmission method for Gaussian splash volume video implemented using the layered characterization compression method according to any one of claims 1-4, characterized in that, include: Based on the hierarchical characterization compression method, a bitstream supporting multi-level quality selection is generated; The bitstream is decoded at the required number of layers to achieve progressive transmission and dynamic adaptation.

7. A progressive transmission system for Gaussian splash volumetric video implemented using the layered characterization compression method according to any one of claims 1-4, characterized in that, include: A bitstream generation module that generates a bitstream that supports multi-level quality selection based on the hierarchical characterization compression method; The decoding module is used to select the number of layers to decode the bitstream as needed, so as to achieve progressive transmission and dynamic adaptation.

Images