Flow dynamic gaussian blurring characterization and compression method, device, equipment and medium
Patent Information
- Application Number
- CN202611400028.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-10
- Publication Date
- 2026-10-09
AI Technical Summary
[0022] As can be seen from the technical solution provided by the present invention, by constructing a point-based hierarchical implicit representation, the waste of parameters in unoccupied space is avoided; by using local neighborhood information through intra-scale aggregation, spatial redundancy is reduced; by fusing multi-resolution context through cross-scale aggregation, the non-uniform distribution characteristics of Gaussian points are adapted, and the neural network parameters can also be compressed, which can save transmission resources and reduce the bit rate.
Smart Images
Figure CN122888029A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of dynamic Gaussian splash compression technology, and in particular to a method, apparatus, equipment and medium for characterizing and compressing dynamic Gaussian splashes. Background Technology
[0002] Free-viewpoint video is an interactive video technology that allows users to freely change their viewing angle. It achieves real-time freezing of dynamic images and 360° free-viewpoint interaction through multi-camera array synchronous control, virtual viewpoint generation, and depth-based rendering techniques. It has wide applications in sports live streaming, variety show live streaming, and distance education, providing an immersive visual experience. However, this technology also faces challenges such as large data transmission volumes, high computational complexity, and slow rendering speeds. To address these issues, researchers have proposed multi-frame synchronization technology, efficient encoding and transmission schemes, and real-time rendering techniques to improve deployment efficiency and reduce hardware costs.
[0003] 3DGS (3-Dimensional Gaussian Splatting) is an advanced 3D scene representation and rendering technology designed to efficiently reconstruct and render complex scenes. It achieves a precise representation of the scene by decomposing objects in the scene into multiple 3D Gaussian distributed points, each containing attributes such as position, shape, color, and opacity. 3DGS's core advantages lie in its high-efficiency rendering performance and accurate capture of complex geometries, making it widely applicable in fields such as autonomous driving simulation, virtual reality, and augmented reality. Furthermore, 3DGS further optimizes rendering speed through adaptive density control and tile-based rasterization algorithms, supporting real-time rendering and enabling real-time interaction in dynamic scenes. Figure 1 The basic algorithm flow of 3DGS is demonstrated, in which SfM (Structure from Motion) is the motion inference structure.
[0004] In the compression research of static 3D Gaussian splashes (3DGS), in order to improve the compactness of representation and the fidelity of rendering, researchers proposed a neural Gaussian representation. This representation eliminates local spatial redundancy through structured implicit representation, and each neural Gaussian is defined as a set of attributes. The position of the local Gaussian is calculated based on the set of attributes, while the remaining attributes of the local Gaussian, such as color, rotation matrix, scaling matrix, and opacity, are predicted by an MLP (multilayer perceptron) with implicit features and relative viewpoint information as input.
[0005] Dynamic 3D Gaussian splashing can be seen as a temporal extension of static 3D Gaussian splashing. Building upon the use of Gaussian primitives to describe the spatial structure and appearance attributes of a scene, it further models the changes in the position, shape, color, or opacity of these primitives at different times. Therefore, compared to static scenes, dynamic scenes not only exhibit spatial redundancy within a single frame but also significant temporal redundancy between Gaussian primitives and their attributes at adjacent times. How to effectively compress these spatial and temporal redundancies while maintaining spatiotemporal consistency and rendering quality in dynamic scenes becomes a crucial problem that dynamic Gaussian splashing compression needs to address.
[0006] Currently, dynamic Gaussian splash compression schemes mainly include the following two types: Option 1: Dynamic Gaussian splash compression based on hash grid.
[0007] The hash grid-based dynamic Gaussian splash compression framework consists of two parts: initial frame encoding and prediction frame encoding. The initial frame uses a neural Gaussian as the scene representation primitive. To reduce the bandwidth required for transmission, the attributes of the neural Gaussian are quantized into integers, and a hash grid-assisted context model is introduced to estimate the entropy of the neural Gaussian attributes. Arithmetic coding compression is then used to generate the neural Gaussian attribute bitstream. The anchor coordinates of the initial frame are stored and transmitted using 32-bit floating-point numbers, forming the initial frame bitstream together with the neural Gaussian attribute bitstream. The decoding end performs arithmetic decoding and inverse quantization on the received bitstream, and generates a 2D image through viewpoint-adaptive neural Gaussian splash rendering. Furthermore, the decoding end stores the decoded neural Gaussian as a reference for subsequent prediction frames.
[0008] In the prediction frame, the framework uses the neural Gaussian obtained from the decoded previous frame as a reference. It learns a binary hash grid as a transform buffer to represent the residual information. Then, it adds the reference information and the residual information to obtain the neural Gaussian representation of the current frame (the current prediction frame). A two-dimensional image is then generated through viewpoint-adaptive neural Gaussian splash rendering. The neural Gaussian includes anchor coordinates and other attributes. The anchor coordinates are the same across all frames and originate from the initial frame, while the other attributes differ. For the current frame, the corresponding attributes from the previous frame are used as a reference. Specifically, the binary hash grid receives the anchor coordinates of the neural Gaussian in the initial frame as input. It uses the implicit embedding features learned and stored by the grid vertices as reference information, performs linear interpolation based on the position of the vertex coordinates and the anchor coordinates, predicts the features of the anchor position, and performs nonlinear operations on the predicted features through MLP to obtain the residual of the neural Gaussian. The residual is then added to the corresponding reference attribute to obtain the neural Gaussian attribute of the current frame.
[0009] However, to avoid excessive storage and computational overhead, the hash grid used in the above scheme does not assign implicit embedding features to every grid vertex in 3D space. Instead, it uses a hash function to map most grid vertices to a few implicit embedding features. One implicit embedding feature may correspond to multiple grid vertices, thus saving computation and storage, but at the cost of reduced reconstruction quality. Furthermore, the above scheme uses 32-bit floating-point numbers to store the MLP network parameters, lacking effective compression, resulting in storage and transmission overhead.
[0010] Option 2: Dynamic Gaussian splash compression based on implicit point representation.
[0011] In the research of streaming compression for dynamic Gaussian splashing, to avoid redundant modeling of unoccupied space in structured mesh methods, researchers have proposed methods based on point-like implicit representations, among which QUEEN (a streaming dynamic Gaussian splashing compression framework) is representative. QUEEN employs a point-by-point modeling strategy: assigning an independent learnable implicit embedding to each Gaussian point in the scene, and using a jointly optimized implicit decoder to predict the Gaussian attribute residuals of the current frame relative to the previous frame. For compression, QUEEN quantizes the implicit embeddings and introduces a learned gating module to sparsify the positional residuals, thereby reducing storage overhead.
[0012] However, the QUEEN method has the following inherent drawbacks: (1) Lack of local correlation modeling: QUEEN processes each Gaussian point independently and fails to utilize the spatial local correlation between adjacent points, resulting in high spatial redundancy in its implicit representation, which limits the compactness of the representation.
[0013] (2) Single-scale structure limitation: QUEEN adopts a single-scale implicit representation structure, which cannot adapt to the non-uniform distribution of Gaussian points in three-dimensional space. For regions with large density differences (such as foreground and background, moving and static regions), the single-scale design is difficult to capture motion or deformation features of different granularities at the same time, thus affecting prediction accuracy and reconstruction quality.
[0014] (3) Neural network parameters are not compressed: QUEEN only compresses the implicit embeddings, while ignoring the compression of the neural network parameters themselves. This leads to an unbalanced overall bit rate distribution, with the neural network parameters occupying a large amount of transmission resources, hindering further bit rate reduction and also being detrimental to end-to-end rate distortion optimization.
[0015] The aforementioned shortcomings limit the compression efficiency and reconstruction performance of QUEEN in streaming dynamic Gaussian splash compression. There is an urgent need for a new representation and compression framework that can take into account local correlation modeling, multi-scale feature representation and network parameter compression.
[0016] In view of this, the present invention is hereby proposed. Summary of the Invention
[0017] The purpose of this invention is to provide a method, apparatus, device and medium for characterizing and compressing dynamic Gaussian splashing, which is a novel representation and compression scheme that can take into account local correlation modeling, multi-scale feature expression and network parameter compression.
[0018] The objective of this invention is achieved through the following technical solution: A method for characterizing and compressing dynamic Gaussian splashing in streaming media, comprising: At the current time step t, the Gaussian neural network set obtained from the decoding of the previous frame is acquired. Multi-scale downsampling is performed on the anchor points in the Gaussian neural network set from the previous frame to obtain a multi-scale anchor point set. A learnable implicit embedding is initialized for each anchor point position, and compressed separately with the neural network parameters before being transmitted to the decoding end. Specifically, for the implicit embedding, bit rate estimation is performed based on a factorized entropy model after quantization, and arithmetic encoding is used to complete the compression based on the estimated bit rate. For the neural network parameters, bit rate estimation is performed based on a Gaussian entropy model after quantization, and arithmetic encoding is used to complete the compression based on the estimated bit rate. The decoder obtains the reconstructed implicit embeddings and neural network parameters, and combines the reconstructed implicit embeddings with the corresponding scale anchor positions to form a point-based hierarchical implicit representation. Within each scale, the implicit representations are aggregated by combining the reconstructed neural network parameters to obtain the aggregated representation for each scale. The aggregated representations for each scale are then aggregated layer by layer to obtain the final aggregated implicit representation. Based on the reconstructed neural network parameters, residual prediction is performed on the final aggregated implicit representation, and combined with the Gaussian neural set obtained from the previous frame decoding, the Gaussian neural set obtained from the current frame decoding is obtained.
[0019] A flow-based dynamic Gaussian splash characterization and compression device for implementing the aforementioned method includes: At the encoding end, at the current time step t, the Gaussian neural network set decoded from the previous frame is obtained. Multi-scale downsampling is performed on the anchor positions of the previous frame within the Gaussian neural network set to obtain a multi-scale anchor position set. A learnable implicit embedding is initialized for each anchor position at each scale, and compressed separately with the neural network parameters before being transmitted to the decoding end. Specifically, for the implicit embedding, bit rate estimation is performed based on a factorized entropy model after quantization, and arithmetic encoding is used to complete the compression based on the estimated bit rate. For the neural network parameters, bit rate estimation is performed based on a Gaussian entropy model after quantization, and arithmetic encoding is used to complete the compression based on the estimated bit rate. The decoding end is used to decode and obtain the reconstructed implicit embeddings and neural network parameters. The reconstructed implicit embeddings and corresponding scale anchor positions are combined into a point-based hierarchical implicit representation. Within each scale, the implicit representations are aggregated by combining the reconstructed neural network parameters to obtain the aggregated representation for each scale. The aggregated representations for each scale are then aggregated layer by layer to obtain the final aggregated implicit representation. Based on the reconstructed neural network parameters, residual prediction is performed on the final aggregated implicit representation, and combined with the Gaussian neural set obtained from the previous frame decoding, the Gaussian neural set obtained from the current frame decoding is obtained.
[0020] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.
[0021] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.
[0022] As can be seen from the technical solution provided by the present invention, by constructing a point-based hierarchical implicit representation, the waste of parameters in unoccupied space is avoided; by using local neighborhood information through intra-scale aggregation, spatial redundancy is reduced; by fusing multi-resolution context through cross-scale aggregation, the non-uniform distribution characteristics of Gaussian points are adapted, and the neural network parameters can also be compressed, which can save transmission resources and reduce the bit rate. Attached Figure Description
[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 A schematic diagram of the basic algorithm flow of 3DGS provided for the background technology of this invention.
[0025] Figure 2 This is a schematic diagram of a flow dynamic Gaussian splash characterization and compression method provided in an embodiment of the present invention.
[0026] Figure 3 This is a schematic diagram of implicit embedding compression provided in an embodiment of the present invention.
[0027] Figure 4 This is a schematic diagram of the frame group results of neural network compression provided in an embodiment of the present invention.
[0028] Figure 5This is a schematic diagram of neural network parameter compression for an I-frame provided in an embodiment of the present invention.
[0029] Figure 6 This is a schematic diagram of neural network parameter compression for P-frames provided in an embodiment of the present invention.
[0030] Figure 7 The following are performance comparison results of the present invention and the baseline model on the rate-distortion curve when using the N3DV dataset, as provided in the embodiments of the present invention.
[0031] Figure 8 The following are performance comparison results of the present invention and the baseline model on the rate-distortion curve when using the Meet Room dataset, as provided in the embodiments of the present invention.
[0032] Figure 9 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation
[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0034] First, the following explanations are provided for the terms that may be used in this article: The terms “comprising,” “including,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “comprising a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.
[0035] The following provides a detailed description of a flow dynamic Gaussian splash characterization and compression method, apparatus, equipment, and medium provided by the present invention. Contents not described in detail in the embodiments of the present invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, conventional conditions in the art or conditions recommended by the manufacturer shall apply. Instruments used in the embodiments of the present invention whose manufacturers are not specified are all conventional products that can be purchased commercially.
[0036] Example 1 This invention provides a method for characterizing and compressing dynamic Gaussian splashing in a flow cytometry manner, such as... Figure 2 As shown, it mainly includes: (1) At the current time step t, obtain the Gaussian neural set obtained from the decoding of the previous frame, perform multi-scale downsampling on the anchor positions of the previous frame in the Gaussian neural set to obtain a multi-scale anchor position set; initialize a learnable implicit embedding for each scale anchor position, and compress it together with the neural network parameters and transmit it to the decoding end; among them, for the implicit embedding, after quantization, perform bit rate estimation based on the factorized entropy model, and combine the estimated bit rate to perform arithmetic coding to complete the compression; for the neural network parameters, after quantization, perform bit rate estimation based on the Gaussian entropy model, and combine the estimated bit rate to perform arithmetic coding to complete the compression.
[0037] (2) The decoder decodes to obtain the reconstructed implicit embedding and neural network parameters, and combines the reconstructed implicit embedding with the corresponding scale anchor position to form a point-based hierarchical implicit representation; in each scale, the implicit representation is aggregated by combining the reconstructed neural network parameters to obtain the aggregated representation of each scale; then the aggregated representation of each scale is aggregated layer by layer to obtain the final aggregated implicit representation; the residual prediction is performed on the final aggregated implicit representation based on the reconstructed neural network parameters, and the Gaussian neural set obtained by decoding the previous frame is combined to obtain the Gaussian neural set obtained by decoding the current frame.
[0038] Preferably, the step of performing multi-scale downsampling on the anchor point positions of the previous frame in the Gaussian neural network set to obtain a multi-scale anchor point position set includes: The anchor point position of the previous frame in the Gaussian neural set is denoted as... Where t-1 is the previous time step, corresponding to the previous frame; anchor point position As the finest scale anchor point position Locations at various scales are obtained recursively through grid sampling: ; in, For the first Scale of grid size, For preset coefficients, , The corresponding representation is the first Scale, First The location of the anchor point for the scale; This is the floor symbol; By combining anchor point locations at all scales, a multi-scale anchor point location set is obtained. , where L is the total number of scales.
[0039] Preferably, the implicit representation aggregation by combining the reconstructed neural network parameters at each scale includes: For the Scale, its implicit representation is denoted as The implicit representation contains several target points, and each target point corresponds to a target implicit representation. The anchor points for each target point are a set containing the implicit representation anchor points. For each target implicit representation, the k nearest neighbor implicit representations are searched using the k-nearest neighbor algorithm based on the anchor point position. The relative positional offset between the target implicit representation anchor point and the k nearest neighbor implicit representation anchor points is calculated. This relative position is input into the first multilayer perceptron to obtain normalized aggregation weights. The implicit embeddings of the k nearest neighbor implicit representations are then weighted and summed. The weighted summation result is then input into the second multilayer perceptron to obtain the aggregated representation of the target implicit representation. All target implicit representations obtain their corresponding aggregated representations through the same process, thus obtaining the first... The aggregated representation of all implicit representations of the target at the scale, which serves as the first Aggregate characterization of scale; The parameters of the first and second multilayer perceptrons are derived from the parameters of the reconstructed neural network.
[0040] Preferably, the stepwise aggregation of the aggregation representation at each scale to obtain the final aggregation implicit representation includes: From the coarsest scale It begins by aggregating layer by layer until the finest scale is reached. Where L is the total number of scales; The representation at the current scale is upsampled to the spatial resolution of the next scale. The upsampled aggregated representation is then embedded and concatenated with the aggregated representation at the next scale along the channel dimension. The concatenated result is used as the new representation at the next scale. This process is repeated recursively until the finest scale is reached. This yields the final implicit representation of the aggregation; Where the current scale is the coarsest scale Then the representation at the current scale is the coarsest scale. Aggregate representation; if the current scale is not the coarsest scale Then the representation at the current scale is a new representation obtained by splicing.
[0041] Preferably, the residual prediction of the final aggregated implicit representation based on the reconstructed neural network parameters, and the obtaining of the Gaussian neural set obtained from the decoding of the previous frame, includes: Gaussian neural sets obtained from decoding the previous frame ,in, This is the anchor point position of the previous frame. This is the Gaussian offset of the previous frame. This is an implicit feature of the previous frame. This is the scaling factor for the previous frame; The final aggregated implicit representation is input into different prediction heads to obtain feature residuals. Anchor point translation amount Anchor point rotation amount and offset residual The parameters of the different prediction heads are all derived from the parameters of the reconstructed neural network. Update the neural Gaussian properties of the current frame using the following formula: Implicit features of the current frame for ; Anchor position of the current frame for: ; Gaussian offset of the current frame for: ; Current frame scaling factor Remain unchanged, that is ; Finally, the Gaussian neural set obtained from decoding the current frame is obtained. .
[0042] Preferably, for implicit embeddings, the bit rate estimation based on the factorized entropy model after quantization includes: Define the hierarchical implicit embedding corresponding to the anchor positions of all scales at the current time step t. ;in, For the first The implicit embedding corresponding to the anchor point position of the scale, where L is the total number of scales; Hierarchical implicit embedding Quantization is performed to obtain the quantized hierarchical implicit embedding. ; Bit rate estimation is performed using a factorized entropy model, which first progressively embeds the quantized data into a hierarchical implicit structure through several learnable layers. The distribution is transformed into a cumulative distribution, and the probability mass function is calculated. Then, the bit rate is estimated based on the probability mass function, as follows: ; ; in, For preset coefficients, The cumulative distribution function is... Based on The probability mass function, For about The mathematical expectation, where log is the logarithmic function. Based on Estimated bit rate.
[0043] Preferably, the step of performing bit rate estimation based on a Gaussian entropy model after quantization of neural network parameters, and then performing arithmetic encoding based on the estimated bit rate to complete compression includes: A compression strategy based on time-domain reference is adopted, and a frame group structure is introduced. The first frame of each frame group structure is an independently coded I-frame, and the subsequent frames are P-frames. Among them, I-frames are intra-coded frames, and P-frames are inter-predictive coded frames. If the current time step t corresponds to I-frame, then adjust the neural network parameters. Scaling quantization is performed, and the probability distribution of parameters is modeled using a Gaussian entropy model. This is used to estimate the bit rate, and arithmetic coding is performed based on the estimated bit rate to complete compression. If the current time step t corresponds to P frames, then a parameter-wise residual is introduced. Using the neural network parameters reconstructed from the previous time step t-1 The dynamic range of the residual Quantization is performed, and the probability distribution of the parameters is modeled using a Gaussian entropy model. This is used to estimate the bit rate, and arithmetic coding is performed based on the estimated bit rate to complete the compression.
[0044] The above-mentioned solution provided by the embodiments of the present invention avoids wasting parameters in unoccupied space by constructing a point-based hierarchical implicit representation; reduces spatial redundancy by utilizing local neighborhood information through intra-scale aggregation; adapts to the non-uniform distribution characteristics of Gaussian points by fusing multi-resolution context through cross-scale aggregation; and can also compress neural network parameters, thereby saving transmission resources and reducing bit rate.
[0045] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.
[0046] 1. Dynamic Gaussian representation based on point-based hierarchical implicit representation.
[0047] See Figure 2 This invention is based on the neural Gaussian set obtained by decoding the previous frame. By constructing multi-scale point-like implicit embedding and performing intra-scale implicit aggregation and cross-scale implicit aggregation, the neural Gaussian deformation parameters of the current frame are predicted, thereby realizing frame-by-frame reconstruction of dynamic scenes.
[0048] Let the current time step be Obtain the neural Gaussian set obtained from the decoding of the previous frame. , This is the anchor point position of the previous frame. This is the Gaussian offset of the previous frame. This is an implicit feature of the previous frame. This is the scaling factor for the previous frame. To construct a hierarchical, dotted implicit representation, the anchor point position of the previous frame is... Multi-scale downsampling is performed to obtain location sets at different scales. ,in This is the finest scale (corresponding to the original position). Increasing the corresponding scale coarsens the scale, where L is the total number of scales. The coarse-scale locations are obtained recursively through grid sampling. ; in, For the first Scale of grid size, For preset coefficients (e.g., can be set) ), , The corresponding representation is the first Scale, First The location of the anchor point for the scale.
[0049] Initialize a learnable implicit embedding for each location at each scale. After optimization, compression, and transmission, the decoding end obtains the reconstructed embedding. And combined with the corresponding positions to form a point-based hierarchical implicit representation. .
[0050] After obtaining the hierarchical implicit points, Inner-scale Latent Aggregation (ILA) is performed. This involves processing the implicit representations within each scale. For each target implicit representation, the k nearest neighbor implicit representations are searched using the k nearest neighbor algorithm, and the relative positional offset between the target implicit representation anchor point and each nearest neighbor implicit representation anchor point is calculated. The relative positions are input into a shared MLP (the first multilayer perceptron mentioned above), which outputs normalized aggregation weights for embedding each neighbor. A weighted sum is performed, and the weighted result is then fed into another MLP (the second multilayer perceptron mentioned earlier) to obtain an aggregated representation of the target's implicit representation. ILA modules at different scales share the same MLP parameters, and all MLP parameters are derived from the reconstructed neural network parameters obtained through decoding. The encoding and decoding processes involved will be described later.
[0051] Then, cross-scale implicit aggregation (CLA) is performed. Starting from the coarsest scale... Begin by aggregating layer by layer towards finer scales: this involves combining the current coarse-scale representation (if...) If the result is positive, it is the aggregated representation; otherwise, it is the new representation obtained by splicing. Upsample (Up) to the spatial resolution of the next finer scale (by copying to the corresponding position), and splice the upsampled label with the aggregated representation of the next finer scale along the channel dimension. Use the splicing result as the new representation of the next finer scale, and continue recursively to finer scales until the finest scale is reached. This yields the final implicit representation of the aggregation.
[0052] The final aggregated implicit representation is input into multiple prediction heads (which can be MLPs) to obtain feature residuals. Anchor point translation amount Anchor point rotation amount and offset residual The parameters of the prediction head also come from the reconstructed neural network parameters obtained through decoding; the encoding and decoding processes involved will be described later. Then, the neural Gaussian attributes of the current frame are updated as follows: implicit features are updated to... Anchor point position updated to The Gaussian offset is updated to... The scaling factor remains unchanged, i.e. Finally, the neural Gaussian set obtained from decoding the current frame is obtained. It is used for subsequent rendering or as input for encoding the next frame.
[0053] The above scheme avoids wasting parameters in unoccupied space by constructing a point-based hierarchical implicit representation; it reduces spatial redundancy by utilizing local neighborhood information through intra-scale aggregation; and it adapts to the non-uniform distribution characteristics of Gaussian points by fusing multi-resolution context through cross-scale aggregation. The proposed Gaussian attribute update mechanism can efficiently model inter-frame dynamic changes, providing compact and expressive implicit features for subsequent compression.
[0054] 2. End-to-end rate-distortion coding of combined implicit embedding and neural networks.
[0055] This section mainly introduces an end-to-end compression scheme for streaming dynamic Gaussian splashing, which combines implicit embedding and neural network parameters with rate distortion optimization to achieve low bit rate transmission.
[0056] (1) Implicit embedding compression.
[0057] like Figure 3 The diagram shown is a schematic of implicit embedding compression.
[0058] Define the hierarchical implicit embedding corresponding to the anchor positions of all scales at the current time step t. ;in, For the first The implicit embedding corresponding to the anchor point position of the scale, where L is the total number of scales.
[0059] Hierarchical implicit embedding Quantization is performed to obtain the quantized hierarchical implicit embedding. Since the quantization operation is not differentiable during training, this embodiment employs two different surrogate strategies: for distortion calculation, a gradient pass-through estimator (STE) is used to obtain the quantized embedding. For entropy estimation, quantization is simulated by adding uniform noise, i.e. , Indicates noise Follows uniform distribution , This indicates a range of uniform distribution; this range is merely an example and does not constitute a limitation.
[0060] After quantization, a factorized entropy model is used for bit rate estimation. The quantized hierarchical implicit embedding is first performed through several learnable layers. The distribution is transformed into a cumulative distribution, and the probability mass function is calculated. Then, the bit rate is estimated based on the probability mass function, as follows: ; ; in, For preset coefficients (e.g., set) ), The cumulative distribution function is... For based on The probability mass function, For about The expected value of the logarithm, where log is the logarithmic function. For based on Estimated bit rate.
[0061] Arithmetic coding is applied to the implicit embedding based on the estimated bit rate to achieve final compression, resulting in a compressed bitstream. After the compressed bitstream is transmitted to the decoding end, the decoding end uses the aforementioned probability quality function to perform arithmetic decoding to obtain the reconstructed hierarchical implicit embedding. .
[0062] (2) Neural network parameter compression.
[0063] This embodiment compresses the neural network parameters in dynamic Gaussian splashing. By analyzing the parameter distribution of corresponding network layers in adjacent frames, strong inter-frame correlations were found; therefore, a compression strategy based on time-domain reference was adopted. To prevent error propagation, a group of frames (GOP) structure was introduced, such as... Figure 4 As shown, each GOP begins with an independently coded I-frame, followed by several predictively coded P-frames.
[0064] like Figure 5As shown, if the current time step t corresponds to I-frame, then the neural network parameters... Scaling quantization is performed by modeling the probability distribution of parameters (i.e., the probability mass function) using a Gaussian entropy model, from which bit rate estimation is conducted. The estimated bit rate is then combined with arithmetic coding to complete compression. Specifically, scaling quantization is performed using the following formula: ,in, To quantize bit depth, Here are the quantized neural network parameters, and min(.) and max(.) are the minimum and maximum value functions, respectively. Minimum value and maximum value As side information, it is transmitted to the decoding end, which first decodes it to obtain... Press again Reconstructing Neural Network Parameters .
[0065] During training, distortion is calculated using STE, and entropy is estimated by adding uniform noise. A network-independent Gaussian entropy model is employed: the probability mass function of the parameters is... ,in, For network parameter index values, The mean is variance is Gaussian distribution, , The corresponding representation is used to obtain The mean and variance of For based on The probability mass function, bit rate , For about The mathematical expectation.
[0066] like Figure 6 As shown, if the current time step t corresponds to P frames, then an optimization is performed using a parameter-wise residual. Using the neural network parameters reconstructed from the previous time step t-1 The dynamic range of the residual Quantization is performed, and the probability distribution of the parameters is modeled using a Gaussian entropy model. This is used to estimate the bit rate, and arithmetic coding is performed based on the estimated bit rate to complete the compression.
[0067] Specifically: by introducing a learnable layer-by-layer scaling factor and parameter-wise residuals Collaborative modeling and reconstruction of network parameters. Because of the extremely small quantity, the original precision of transmission is maintained. Then further compression. The decoding end presses... Reconstructing Neural Network Parameters Residual Quantitative utilization Normalize the dynamic range: ; ; ; in, The residual after quantization for and The minimum value in, for and The maximum value in.
[0068] The entropy estimation of the residuals also uses the Gaussian entropy model: bit rate ,in, For based on The probability mass function, The mean is variance is Gaussian distribution, , The corresponding representation is used to obtain The mean and variance of For about The mathematical expectation.
[0069] Similarly, minimum value and maximum value As side information, it is transmitted to the decoding end, which first decodes the quantized residual. Then reconstruct the residuals according to the following formula. The neural network parameters reconstructed at the previous time step t-1 As a reference parameter, a layer-by-layer learnable scaling factor is introduced. A preliminary prediction is made of the reference parameters (i.e., the two are multiplied together), and then compared with the reconstructed residuals. Add them together to obtain the neural network parameters reconstructed at the current time step t. .
[0070] (3) End-to-end joint optimization.
[0071] For each time step t, the overall loss function is: ,in Lagrange multipliers are used for control rate distortion balance. Distortion term. The rendering loss is achieved using 3D Gaussian splashing, i.e. ,in and These represent the 1-norm distance loss and structural similarity loss between the rendered image and the ground image at time step t, respectively. Bit rate term. Determined by frame type: If it is an I-frame, then If it is a P-frame, then .
[0072] The above scheme utilizes the inter-frame correlation of network parameters in adjacent frames, performs time-domain prediction through learnable scaling factors and parameter-by-parameter residuals, and quantizes and entropy-encodes the residuals. Combined with GOP structure and end-to-end joint optimization, it achieves efficient compression of network parameters.
[0073] 3. Performance description of the above scheme.
[0074] Existing advanced streaming dynamic Gaussian splash compression methods were introduced as baseline methods for comparative experiments. The baseline methods in the comparative experiments included: 4DGV (4D Gaussian video), iFVC (intelligent free viewpoint video compression), and ComGS (compressed Gaussian splash). The test datasets were the N3DV dataset (a classic benchmark dataset for dynamic 3D scene reconstruction) and the Meet Room dataset (a benchmark dataset for evaluating the performance of dynamic 3D scene reconstruction methods in complex, high-motion scenes). The rate-distortion curve was used as the evaluation metric to measure the encoding performance of each algorithm. The bitrate was the file size required to encode each frame of the scene, and the distortion was the peak signal-to-noise ratio (PSNR).
[0075] Figures 7-8 As shown, the performance of the proposed method and the baseline model on the rate-distortion curves in the N3DV and Meet Room datasets is compared. The method closer to the top left corner represents the better performance, which shows that the proposed method can effectively improve the rate-distortion performance of the encoder.
[0076] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0077] Example 2 The present invention also provides a flow dynamic Gaussian splash characterization and compression device, which is mainly used to implement the method provided in the foregoing embodiments. The device mainly includes: At the encoding end, at the current time step t, the Gaussian neural network set decoded from the previous frame is obtained. Multi-scale downsampling is performed on the anchor positions of the previous frame within the Gaussian neural network set to obtain a multi-scale anchor position set. A learnable implicit embedding is initialized for each anchor position at each scale, and compressed separately with the neural network parameters before being transmitted to the decoding end. Specifically, for the implicit embedding, bit rate estimation is performed based on a factorized entropy model after quantization, and arithmetic encoding is used to complete the compression based on the estimated bit rate. For the neural network parameters, bit rate estimation is performed based on a Gaussian entropy model after quantization, and arithmetic encoding is used to complete the compression based on the estimated bit rate. The decoding end is used to decode and obtain the reconstructed implicit embeddings and neural network parameters. The reconstructed implicit embeddings and corresponding scale anchor positions are combined into a point-based hierarchical implicit representation. Within each scale, the implicit representations are aggregated by combining the reconstructed neural network parameters to obtain the aggregated representation for each scale. The aggregated representations for each scale are then aggregated layer by layer to obtain the final aggregated implicit representation. Based on the reconstructed neural network parameters, residual prediction is performed on the final aggregated implicit representation, and combined with the Gaussian neural set obtained from the previous frame decoding, the Gaussian neural set obtained from the current frame decoding is obtained.
[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0079] Example 3 The present invention also provides a processing device, such as Figure 9 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.
[0080] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.
[0081] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.
[0082] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.
[0083] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.
[0084] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.
Claims
1. A method for characterizing and compressing dynamic Gaussian splashing in a flow cytometry system, characterized in that, include: At the current time step t, the Gaussian neural network set obtained from the decoding of the previous frame is acquired. Multi-scale downsampling is performed on the anchor points in the Gaussian neural network set from the previous frame to obtain a multi-scale anchor point set. A learnable implicit embedding is initialized for each anchor point position, and compressed separately with the neural network parameters before being transmitted to the decoding end. Specifically, for the implicit embedding, bit rate estimation is performed based on a factorized entropy model after quantization, and arithmetic encoding is used to complete the compression based on the estimated bit rate. For the neural network parameters, bit rate estimation is performed based on a Gaussian entropy model after quantization, and arithmetic encoding is used to complete the compression based on the estimated bit rate. The decoder obtains the reconstructed implicit embeddings and neural network parameters, and combines the reconstructed implicit embeddings with the corresponding scale anchor positions to form a point-based hierarchical implicit representation. Within each scale, the implicit representations are aggregated by combining the reconstructed neural network parameters to obtain the aggregated representation for each scale. The aggregated representations for each scale are then aggregated layer by layer to obtain the final aggregated implicit representation. Based on the reconstructed neural network parameters, residual prediction is performed on the final aggregated implicit representation, and combined with the Gaussian neural set obtained from the previous frame decoding, the Gaussian neural set obtained from the current frame decoding is obtained.
2. The method for characterizing and compressing dynamic Gaussian splashing according to claim 1, characterized in that, The process of performing multi-scale downsampling on the anchor point positions of the previous frame in the Gaussian neural network to obtain a multi-scale anchor point position set includes: The anchor point position of the previous frame in the Gaussian neural set is denoted as... Where t-1 is the previous time step, corresponding to the previous frame; anchor point position As the finest scale anchor point position Locations at various scales are obtained recursively through grid sampling: ； in, For the first Scale of grid size, For preset coefficients, , The corresponding representation is the first Scale, First The location of the anchor point for the scale; This is the floor symbol; By combining anchor point locations at all scales, a multi-scale anchor point location set is obtained. , where L is the total number of scales.
3. The method for characterizing and compressing dynamic Gaussian splashing according to claim 1, characterized in that, The implicit representation aggregation by combining the reconstructed neural network parameters at each scale includes: For the Scale, its implicit representation is denoted as The implicit representation contains several target points, each target point corresponding to a target implicit representation, the th... The anchor points for each target point are a set containing the implicit representation anchor points. For each target implicit representation, the k nearest neighbor implicit representations are searched using the k-nearest neighbor algorithm based on the anchor point position. The relative positional offset between the target implicit representation anchor point and the k nearest neighbor implicit representation anchor points is calculated. This relative position is input into the first multilayer perceptron to obtain normalized aggregation weights. The implicit embeddings of the k nearest neighbor implicit representations are then weighted and summed. The weighted summation result is then input into the second multilayer perceptron to obtain the aggregated representation of the target implicit representation. All target implicit representations obtain their corresponding aggregated representations through the same process, thus obtaining the first... The aggregate representation of all implicit representations of the target at the scale, which serves as the first Aggregate characterization of scale; The parameters of the first and second multilayer perceptrons are derived from the parameters of the reconstructed neural network.
4. The method for characterizing and compressing dynamic Gaussian splashing in a flow cytometry system according to claim 1, characterized in that, The process of aggregating the aggregated representations at each scale layer by layer to obtain the final aggregated implicit representation includes: From the coarsest scale It begins by aggregating layer by layer until the finest scale is reached. Where L is the total number of scales; The representation at the current scale is upsampled to the spatial resolution of the next scale. The upsampled aggregated representation is then embedded and concatenated with the aggregated representation at the next scale along the channel dimension. The concatenated result is used as the new representation at the next scale. This process is repeated recursively until the finest scale is reached. This yields the final implicit representation of the aggregation; Where the current scale is the coarsest scale Then the representation at the current scale is the coarsest scale. Aggregate representation; if the current scale is not the coarsest scale Then the representation at the current scale is a new representation obtained by splicing.
5. The method for characterizing and compressing dynamic Gaussian splashing in a flow cytometry system according to claim 1, characterized in that, The reconstructed neural network parameters are used to perform residual prediction on the final aggregated implicit representation, and combined with the Gaussian neural set obtained from the previous frame decoding, the Gaussian neural set obtained from the current frame decoding includes: Gaussian neural set obtained from decoding the previous frame ,in, This is the anchor point position of the previous frame. This is the Gaussian offset of the previous frame. This is an implicit feature of the previous frame. This is the scaling factor for the previous frame; The final aggregated implicit representation is input into different prediction heads to obtain feature residuals. Anchor point translation amount Anchor point rotation amount and offset residual The parameters of the different prediction heads are all derived from the parameters of the reconstructed neural network. Update the neural Gaussian properties of the current frame using the following formula: Implicit features of the current frame for ; Anchor position of the current frame for: ; Gaussian offset of the current frame for: ; Current frame scaling factor Remain unchanged, that is ; Finally, the Gaussian neural set obtained from decoding the current frame is obtained. .
6. The method for characterizing and compressing dynamic Gaussian splashing in a flow cytometry system according to claim 1, characterized in that, For implicit embeddings, the bit rate estimation based on the factorized entropy model after quantization includes: Define the hierarchical implicit embedding corresponding to the anchor positions of all scales at the current time step t. ;in, For the first The implicit embedding corresponding to the anchor point position of the scale, where L is the total number of scales; Hierarchical implicit embedding Quantization is performed to obtain the quantized hierarchical implicit embedding. ; Bit rate estimation is performed using a factorized entropy model, which first progressively embeds the quantized data into a hierarchical implicit structure through several learnable layers. The distribution is transformed into a cumulative distribution, and the probability mass function is calculated. Then, the bit rate is estimated based on the probability mass function, as follows: ； ； in, For preset coefficients, The cumulative distribution function is... Based on The probability mass function, For about The mathematical expectation, where log is the logarithmic function. Based on Estimated bit rate.
7. The method for characterizing and compressing dynamic Gaussian splashing in a flow cytometry system according to claim 1, characterized in that, The bit rate estimation based on the Gaussian entropy model after quantization of neural network parameters includes: A compression strategy based on time-domain reference is adopted, and a frame group structure is introduced. The first frame of each frame group structure is an independently coded I-frame, and the subsequent frames are P-frames. Among them, I-frames are intra-coded frames, and P-frames are inter-predictive coded frames. If the current time step t corresponds to I-frame, then adjust the neural network parameters. Scaling quantization is performed, and the probability distribution of the parameters is modeled using a Gaussian entropy model, thereby estimating the bit rate; If the current time step t corresponds to P frames, then a parameter-wise residual is introduced. Using the neural network parameters reconstructed from the previous time step t-1 The dynamic range of the residual Quantization is performed, and the probability distribution of the parameters is modeled using a Gaussian entropy model, thereby estimating the bit rate.
8. A flow-type dynamic Gaussian splash characterization and compression device, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: At the encoding end, at the current time step t, the Gaussian neural network set decoded from the previous frame is obtained. Multi-scale downsampling is performed on the anchor positions of the previous frame within the Gaussian neural network set to obtain a multi-scale anchor position set. A learnable implicit embedding is initialized for each anchor position at each scale, and compressed separately with the neural network parameters before being transmitted to the decoding end. Specifically, for the implicit embedding, bit rate estimation is performed based on a factorized entropy model after quantization, and arithmetic encoding is used to complete the compression based on the estimated bit rate. For the neural network parameters, bit rate estimation is performed based on a Gaussian entropy model after quantization, and arithmetic encoding is used to complete the compression based on the estimated bit rate. The decoding end is used to decode and obtain the reconstructed implicit embeddings and neural network parameters. The reconstructed implicit embeddings and corresponding scale anchor positions are combined into a point-based hierarchical implicit representation. Within each scale, the implicit representations are aggregated by combining the reconstructed neural network parameters to obtain the aggregated representation for each scale. The aggregated representations for each scale are then aggregated layer by layer to obtain the final aggregated implicit representation. Based on the reconstructed neural network parameters, residual prediction is performed on the final aggregated implicit representation, and combined with the Gaussian neural set obtained from the previous frame decoding, the Gaussian neural set obtained from the current frame decoding is obtained.
9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.
10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.