3d gaussian splash compression method

Abstract

The application discloses a 3D Gaussian splash compression method, comprising the following steps: collecting a multi-view image sequence, extracting original 3D GS data and corresponding multi-modal semantic information; generating a self-organizing division structure based on the geometric features of the original 3D GS data and the multi-modal semantic information; processing the attributes of the data through a generative model according to the self-organizing division structure and by using the semantic information to obtain coded attribute data; and assembling the structural information and the coded attribute data into a compressed image data code stream. The application uses semantic guidance to divide the structure and restores details by using a generative model, solves the problems that key semantic features are prone to be lost and scene complexity adaptability is poor in the compression process, and realizes high-fidelity reconstruction at a high compression rate.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image data compression, and particularly relates to a 3D Gaussian Splatting compression method. BACKGROUND

[0002] As a new three-dimensional scene representation method, 3D Gaussian Splatting (3DGS) technology explicitly models a scene by using millions of attribute-bearing Gaussian spheres, and achieves real-time rendering speed while maintaining high rendering fidelity. This technology has broad application prospects in virtual reality (VR), digital twinning, dynamic scene reconstruction, and real-time transmission. However, the original data volume of 3DGS technology is extremely large, and a Gaussian sphere usually needs hundreds of bytes to store its position, covariance, color spherical harmonic coefficients, and opacity and other attributes, resulting in storage overhead of a single scene reaching hundreds of megabytes or even gigabytes, which seriously restricts its actual deployment and application in bandwidth and storage limited environments (such as mobile devices and network streaming). Therefore, it is of great research significance and practical value to develop an efficient 3DGS compression technology.

[0003] Currently, there are some research schemes for 3DGS data compression. One class of methods is based on clustering and anchor compression, for example, the Gaussian spheres in the space are clustered by using a hash grid or an octree structure, and the Gaussian attributes in the same cluster are represented by a small number of anchor parameters, so as to utilize the spatial redundancy. Another class of methods is based on quantization and entropy coding, which directly performs scalar or vector quantization (VQ) on the attributes of the Gaussian sphere (such as position coordinates, scaling factor, spherical harmonic coefficients, etc.), and combines entropy coding techniques such as arithmetic coding to losslessly compress the quantized index or residual. In addition, some researches use models such as autoencoders to learn a compact latent representation of the Gaussian parameter set, and reconstruct the Gaussian attributes through the decoder during decompression.

[0004] However, the existing compression schemes still have several deep technical problems in balancing the compression rate, fidelity, and scene adaptability. These problems mainly arise from the lack of scene content perception ability in the compression process and the irreversible loss of high-frequency information in the reconstruction process. SUMMARY

[0005] The application aims to provide a 3D Gaussian Splatting compression method to solve the above problems in the prior art.

[0006] The technical scheme provides a 3D Gaussian Splatting compression method, which comprises the following steps:

[0007] A multi-view image sequence was acquired, and raw 3D Gaussian splatter data, including the position, anisotropic covariance, color spherical harmonic coefficients, and opacity of the 3D Gaussian splatter, as well as the corresponding multimodal semantic information, was extracted. The multimodal semantic information includes multi-view images and text prompts. The multimodal semantic information was obtained through a cross-modal coding model.

[0008] Based on the geometric features and multimodal semantic information of the original 3D Gaussian splash data, a self-organizing partitioning structure is generated;

[0009] Based on the self-organizing partitioning structure and utilizing multimodal semantic information, the attributes of the original 3D Gaussian splash data are processed through a pre-configured generative model to obtain encoded attribute data.

[0010] The structural information used to restore the self-organizing partition structure and the encoded attribute data are assembled together into a compressed image data stream.

[0011] Beneficial effects: This invention utilizes semantic guidance to divide the structure and uses a generative model to recover details, solving the problems of easy loss of key semantic features and poor adaptability to scene complexity during compression, and achieving high-fidelity reconstruction under high compression rate. Attached Figure Description

[0012] Figure 1 A flowchart illustrating the steps of a 3D Gaussian splash compression method provided in this application embodiment.

[0013] Figure 2 A flowchart illustrating the steps for generating a self-organizing partition structure provided in this application embodiment.

[0014] Figure 3 A flowchart illustrating the steps for recursively determining the segmentation parameters provided in this application embodiment.

[0015] Figure 4 A flowchart illustrating the steps for obtaining encoded attribute data provided in this application embodiment. Detailed Implementation

[0016] It should be noted that the terms include and have, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0017] The research revealed that existing methods are semantically blind in compression decisions. They rely solely on the geometric proximity or statistical characteristics of data for compression, failing to distinguish between semantically critical regions (such as minute scratches on industrial parts or fine facial textures) and semantically redundant regions (such as blank walls or sky backgrounds). This leads to unreasonable bit allocation and severe distortion of key semantic features at high compression rates. Furthermore, existing spatial partitioning structures (such as static hash grids) are typically content-independent, with fixed granularity that cannot adapt to the complexity of scene content. This results in the loss of detail in geometrically complex and semantically important regions due to overly coarse partitioning, and storage redundancy in simple regions due to overly fine partitioning, leading to poor scene adaptability. Schemes relying on traditional quantization or standard autoencoders suffer from irreversible high-frequency details (such as specular reflections on materials and fine surface textures) during compression. Once these high-frequency components are discarded during quantization, the decoder cannot recover them, resulting in a generally smooth and blurry reconstructed scene that fails to meet the requirements of high-fidelity real-time rendering.

[0018] like Figure 1 As shown, a 3D Gaussian sputtering compression method is proposed, including the following steps:

[0019] A multi-view image sequence is acquired, and the original 3D Gaussian splatter data, including the position, anisotropic covariance, color spherical harmonic coefficients, and opacity of the 3D Gaussian splatter, as well as the corresponding multimodal semantic information, is extracted. The multimodal semantic information includes multi-view images and text prompts. The multimodal semantic information is obtained through a cross-modal coding model.

[0020] In other words, the process involves acquiring multi-view image sequences, extracting raw 3D Gaussian splatter data, and corresponding multimodal semantic information. The raw 3D Gaussian splatter data includes, but is not limited to: the position of the 3D Gaussian splatter, anisotropic covariance, color spherical harmonic coefficients, and opacity. The multimodal semantic information includes, but is not limited to: multi-view images and text prompts.

[0021] In this embodiment, the original 3D Gaussian splatter data is an explicit representation of the 3D scene, which can be constructed using existing multi-view geometric reconstruction and 3D Gaussian fitting algorithms. Exemplarily, the original 3D Gaussian splatter data specifically includes a set of parameters for a series of Gaussian spheres, such as: position (3D coordinates), anisotropic covariance (which can be represented by quaternions and scaling factors), color spherical harmonic coefficients (used to represent view-dependent colors), and opacity (α value). Optionally, multimodal semantic information is auxiliary information associated with the 3D scene content, used to guide compression strategies in subsequent compression steps, such as distinguishing semantically important regions from redundant regions. For example, a text prompt could be a red metallic car.

[0022] Preferably, to ensure that the semantic information from different modalities can be uniformly utilized in subsequent steps, this embodiment further includes: encoding multi-view images and text prompts using a cross-modal coding model (e.g., the CLIP model or a variant thereof), where the CLIP model can be a publicly available pre-trained model or a model fine-tuned on scene data; mapping and aligning them to a unified semantic space to generate semantic embeddings. Simultaneously, the semantic embeddings are aligned with the camera parameters corresponding to the original 3D Gaussian splash data to ensure a consistent correspondence between semantic information and geometric positions in three-dimensional space.

[0023] Optionally, it also includes: normalizing the attributes (such as location scale, covariance range, color spherical harmonic amplitude, etc.) of the original 3D Gaussian splash data to form a normalized configuration. This normalized configuration defines the baseline threshold, quantization step size, and stability constraint range on which subsequent steps (such as reversible stability constraints or rate control calibration) depend, so as to improve the robustness of the algorithm.

[0024] Based on the geometric features and multimodal semantic information of the original 3D Gaussian splash data, a self-organizing partitioning structure is generated.

[0025] Specifically, a self-organizing partitioning structure is a data structure used to hierarchically or regionally organize Gaussian sets in three-dimensional space. Unlike traditional static mesh partitioning, the generation process of a self-organizing partitioning structure is self-organizing; that is, the granularity, boundaries, and methods of partitioning are dynamically determined based on the geometric characteristics of the original 3D Gaussian splash data itself (such as the spatial density distribution and positional clustering of Gaussian spheres) and multimodal semantic information (such as the similarity or importance of semantic embeddings). It can be understood that generating a self-organizing partitioning structure aims to divide a large Gaussian set into several more manageable local regions (such as leaf nodes or hash units) so that differentiated compression strategies can be applied to the attributes of different regions in subsequent steps. For example, for regions with complex geometry and high semantic importance (such as faces), the structure can adaptively adopt a finer partitioning; while for regions with simple geometry and simple semantics (such as walls), a coarser-grained partitioning is used, thus achieving a better balance between compression ratio and fidelity.

[0026] In some implementations, the self-organizing partitioning structure can be a reversible partitioning tree. A reversible partitioning tree is generated through a recursive partitioning process that uses a joint cost function to guide each split. Optionally, the joint cost function takes into account geometric distortion, semantic divergence, bitrate cost, and reversibility stability constraints.

[0027] In other implementations, the self-organizing partitioning structure can be a multi-resolution hash grid. The construction process of the multi-resolution hash grid is optimized through an objective function that includes semantically weighted clustering error and grid redundancy penalty terms to achieve adaptive matching between grid resolution and scene content.

[0028] Based on the self-organizing partitioning structure and utilizing multimodal semantic information, the attributes of the original 3D Gaussian splash data are processed through a pre-configured generative model to obtain encoded attribute data.

[0029] In this embodiment, Gaussian attributes within each local region of the self-organizing partitioning structure are compressed and encoded. The generative model is a model capable of reconstructing high-fidelity data (especially high-frequency details) from a compact latent representation; preferably, a diffusion model or variational autoencoder (VAE) can be used. The process of processing the attributes of the original 3D Gaussian splatter data preferably includes attribute factorization: that is, decomposing the Gaussian attributes within each region (e.g., leaf nodes) into two parts: explicit low-frequency attributes (e.g., positional basis, low-order color harmonic coefficients, etc.) to safeguard the basic geometric and lighting primitives; and latent variables that map high-frequency details and perturbations. Accordingly, the encoded attribute data also contains two main parts: a bitstream obtained by quantizing and entropy encoding (e.g., arithmetic encoding) the explicit low-frequency attributes; and control parameters used to control the generative model's reconstruction of latent variables at the decoding end. During this process, multimodal semantic information (e.g., semantic embedding) is used as a conditional input to the generative model. This enables generative models to recover more accurate textures and lighting effects that conform to semantics when reconstructing high-frequency details, based on semantic cues (such as metallic luster), rather than blindly guessing. This solves the problem of traditional compression losing key semantic features at high compression rates.

[0030] The structural information used to restore the self-organizing partition structure and the encoded attribute data are assembled together into a compressed image data stream.

[0031] In this embodiment, a final, deterministic output file is generated. Structural information refers to all the data required by the decoder to accurately reconstruct the self-organizing partition structure bit-by-bit without retraining or optimization. For example, in a scheme using a reversible partitioning tree, this structural information includes the partitioning type of all nodes, quantization partitioning parameters, and the bound semantic codebook index. Preferably, the compressed image data bitstream has a predefined, hierarchical syntax structure. For example, the bitstream can be organized as follows: a header field for storing global information, such as a version number, a fixed random seed for deterministic decoding, an identifier of the global semantic codebook, and an identifier of the global rate control mapping table; a structure table for carrying the structural information; the decoder reads the structure table and reconstructs the self-organizing partition structure in memory; and a leaf node table or data table for carrying the encoded attribute data. The leaf node table or data table typically corresponds one-to-one with the leaf nodes in the structure table, storing the explicit low-frequency attribute bitstream for each leaf node, as well as control parameters (such as diffusion time step index, quantization step size set selection, entropy coding context selection, and semantic checksum) used to control the generative model (such as the diffusion model). Upon receiving the compressed image data bitstream, the decoder parses the header, reconstructs the structure based on the structure table, and finally traverses the leaf node table, combining it with the semantic codebook and performing the generative inverse process of the parameter domain (such as the diffusion inverse process) to reconstruct the latent variables. These latent variables are then synthesized with the explicit low-frequency attributes to ultimately restore the complete 3D Gaussian set for real-time rendering.

[0032] According to one aspect of this application, the process of acquiring raw 3D Gaussian splash data and multimodal semantic information includes:

[0033] By deploying multiple image acquisition devices in an objective physical scene, a multi-view image sequence of the objective physical scene is acquired, and the camera parameters of the image acquisition devices are obtained.

[0034] Acquire auxiliary multimodal data related to the objective physical scene, describing physical attributes or semantic concerns, wherein the auxiliary multimodal data includes text prompts;

[0035] Based on multi-view image sequences and camera parameters, multi-view geometric reconstruction and 3D Gaussian fitting are performed to construct an initial Gaussian set representing the three-dimensional shape of the objective physical scene. This initial Gaussian set constitutes the original 3D Gaussian splash data.

[0036] By using a cross-modal coding model, multi-view image sequences and auxiliary multimodal data are encoded, the encoding results are mapped to a unified semantic space and aligned with camera parameters to generate semantic embeddings that spatially correspond to the initial Gaussian set. These semantic embeddings constitute the corresponding multimodal semantic information.

[0037] For example, the objective physical scene can be an industrial assembly line or a museum exhibition hall. Image acquisition devices can be multiple (e.g., 8, 16, or more) fixed industrial cameras, high-definition cameras, or handheld / airborne devices (such as drones) moving around the physical scene to capture images. A multi-view image sequence is an optical snapshot of the physical scene from different physical viewpoints. Camera parameters are the physical data necessary for 3D reconstruction, including the intrinsic parameters (such as focal length, principal point, and distortion coefficients) and extrinsic parameters (i.e., its physical position and orientation in the world coordinate system) of each camera. Semantics are descriptive, used to indicate key features in the physical scene. For example, in an industrial scene, it could be the engineer's input text, "Point A has a tiny crack," or in a museum scene, the content creator's annotation, "Golden sculpture, high-reflection." This data provides the basis for subsequent compression algorithms to make semantic-aware decisions.

[0038] Optionally, the process of generating 3DGS data can employ Structure for Motion Reconstruction (SfM) technology, utilizing multi-view image sequences and camera parameters to reconstruct a sparse point cloud in 3D space. Based on this sparse point cloud, millions of Gaussian spheres are progressively grown and optimized using a 3D Gaussian splatting fitting algorithm (e.g., optimized via stochastic gradient descent). The optimization goal is to make the splatting rendering results of these Gaussian spheres at each viewpoint as consistent as possible with the pixel values ​​of the original physical image sequence. The final output initial Gaussian set is the original 3D Gaussian splatting data.

[0039] Preferably, the cross-modal coding model can be a pre-trained, powerful model, such as the CLIP (Contrastive Language–Image Pre-Training) model. This model can encode physical images and physical text / speech into the same high-dimensional unified semantic space, such that the vector of the word "metal" is spatially adjacent to the vector of its physical image. These semantic vectors are spatially aligned with an initial set of Gaussians in the three-dimensional physical space through projection and interpolation using camera parameters. The resulting semantic embeddings are semantic vectors associated with each Gaussian sphere or its spatial region, constituting the corresponding multimodal semantic information required for subsequent compression steps.

[0040] like Figure 2 As shown, in one possible implementation, the self-organizing partitioning structure is a reversible partitioning tree; the process of generating the self-organizing partitioning structure includes:

[0041] Based on geometric features and multimodal semantic information, the segmentation parameters used to construct the reversible partitioning tree are recursively determined. The recursion is terminated when the segmentation benefit corresponding to the segmentation parameter is lower than a preset threshold or the number of nodes reaches the upper limit. The segmentation parameters are quantized to form quantized segmentation parameters. The structural information used to restore the self-organizing partitioning structure includes the quantized segmentation parameters to support the decoding end in performing deterministic structural restoration of the reversible partitioning tree.

[0042] In this embodiment, the reversible partitioning tree is a tree-like data structure used for spatial partitioning, where each non-leaf node represents a spatial partition, and leaf nodes represent local regions containing several Gaussian spheres. The process typically starts from the root node containing all Gaussian spheres in the scene and initializes a candidate node set. For example, recursive determination refers to continuously selecting nodes from the candidate node set for partitioning in a loop until a stopping condition is met (e.g., reaching the maximum depth, the number of Gaussians within a node being below a threshold, or the partitioning benefit being too low). Quantization is crucial for enabling deterministic structure reconstruction at the decoding end. Since the floating-point operation environments of the encoder and decoder (which may run on different hardware) may have slight differences, directly transmitting floating-point partitioning parameters (such as the normal vector and threshold of the partitioning plane) may cause the decoder to reconstruct a tree structure inconsistent with the encoder (i.e., index ambiguity). Therefore, it is preferable to quantize the partitioning parameters into a fixed-decimal-bit format, for example, using 16-bit or 32-bit fixed-point numbers. During quantization, a minimum discrimination interval Δ is set to ensure reversibility and prevent decoding ambiguity caused by floating-point differences. These quantization and splitting parameters are written into the structure table. The decoder only needs to read these determined quantization values ​​to reconstruct the tree structure without any floating-point operations or re-optimization.

[0043] like Figure 3 As shown, in a further embodiment, the process of recursively determining the segmentation parameters includes:

[0044] Based on geometric features and multimodal semantic information, a joint cost function is constructed; using the joint cost function, the segmentation benefit of each node in the preset candidate node set is evaluated; based on the segmentation benefit, at least one node is selected from the candidate node set, and the segmentation parameters corresponding to the selected node are determined.

[0045] Specifically, the joint cost function J is a mathematical model used to evaluate the merits of a node (or a potential split). Preferably, this function is constructed to simultaneously consider four aspects: geometry, semantics, bitrate, and stability, and its general form can be expressed as: J = D geom +λ s •D sem +λ r •R bits +λ c •C stab ; where Dgeom For the cost of geometric distortion, D sem For semantic divergence cost, R bits For the sake of bitrate, C stab This is the cost of reversible stability constraints; λ s , λ r , λ c Weighting coefficients are used to balance the importance of semantic consistency, bitrate cost, and reversibility stability, respectively, to balance the importance of these cost terms. Evaluating the segmentation benefit of each node in the predefined candidate node set means that for each node in the candidate node set, the system tries one or more segmentation methods (e.g., axis-aligned plane segmentation or kd-segmentation) and calculates the joint cost of the child nodes after segmentation. The segmentation benefit is the difference between the cost before segmentation and the sum of the costs after segmentation.

[0046] In an optional embodiment, the joint cost function includes at least the following three parts: evaluating the fitting error within a node based on geometric features to obtain the geometric distortion cost; evaluating the semantic dispersion within a node based on multimodal semantic information to obtain the semantic dispersion cost; and evaluating the bit overhead required to write the segmentation parameters into the bitstream to obtain the bitrate cost.

[0047] Based on the splitting profit, select at least one node from the candidate node set, including:

[0048] The reduction in geometric distortion cost and semantic divergence cost is taken as the benefit, and the bitrate cost is taken as the cost. The unit bitrate distortion reduction benefit of each node in the candidate node set is calculated; the node with the highest unit bitrate distortion reduction benefit is selected.

[0049] In this embodiment, the geometric distortion cost D geom This is used to measure the fitting accuracy of the Gaussian set within a node to the local 3D space. Preferably, the geometric distortion cost can be specifically defined as a combination of weighted volume rendering error and voxel coverage difference on the node voxel or ray sample. The lower this cost, the higher the geometric consistency within the node. Semantic divergence cost D sem Semantic embedding is used to measure the semantic consistency of the Gaussian set within a node. Preferably, the semantic divergence cost can be calculated based on the normalized cosine distance between each semantic vector within the node and its semantic centroid. To enhance robustness, anomalous semantic samples can also be truncated. The lower the cost, the more semantically homogeneous the node. Bitrate cost R bitsThis is used to estimate the bit cost required to write the segmentation parameters (such as segmentation type and quantization segmentation parameters) into the bitstream for this segmentation. Preferably, the bitrate cost can be specifically modeled as the sum of the segmentation type flag, the number of quantization bits for the segmentation hyperplane direction and threshold, the node index overhead, and the bit estimate of the entropy coding context switching that may be caused. For example, the rate-distortion reduction benefit per unit bitrate is a rate-distortion optimization strategy. Specifically, the reduction in geometric distortion cost ΔD geom The reduction in semantic divergence cost ΔD sem (or its weighted sum) is considered as the benefit of this split, and the increment ΔR of the bitrate cost is considered as the benefit. bits This is considered a cost. The computer evaluates the benefit-cost ratio (e.g., benefit / cost) of each candidate node under the optimal splitting method and selects the node with the highest ratio (i.e., the node with the highest benefit per unit bitrate loss of distortion) as the node to be split in this recursion.

[0050] In some preferred embodiments, to further improve the robustness of the segmentation decision, a quantile constraint strategy can be adopted when solving for segmentation parameters (such as normal and threshold) to avoid the influence of noisy samples. In addition, a backoff strategy is also provided. For example, if the calculated benefit after segmentation is too low (i.e., the bitrate increment is high but the distortion benefit is insufficient), the segmentation can be canceled, the node can be marked as a leaf node, and recursion can be stopped.

[0051] In a preferred embodiment, the process of constructing the joint cost function further includes:

[0052] Based on the requirement of deterministic reconstruction of the reversible partition tree at the decoding end, reversible stability constraints are constructed; according to the reversible stability constraints, the decoding uncertainty caused by the splitting parameters during the quantization process is evaluated to obtain the cost of the reversible stability constraints; the joint cost function includes the cost of the reversible stability constraints.

[0053] In this embodiment, the reversible stability constraint cost C stabThis is a penalty term used to penalize segmentation choices that might lead to decoding uncertainty after quantization. Optionally, a reversible stability constraint is constructed, including setting a minimum discrimination interval for the segmentation parameters after quantization to avoid index ambiguity caused by differences in floating-point operations at the decoding end; wherein quantization includes converting the segmentation parameters to a fixed-decimal-bit format; the cost of the reversible stability constraint is that the minimum discrimination interval is not met after the segmentation parameters are converted to a fixed-decimal-bit format. Specifically, a minimum discrimination interval is set for the segmentation parameters after quantization (i.e., conversion to a fixed-decimal-bit format). This interval defines the minimum distance that the quantized segmentation threshold must maintain with the coordinates (projected onto the segmentation normal direction) of all Gaussian spheres within the node. Preferably, this constraint may further include setting a geometric safety distance with the nearest sample. This geometric safety distance is typically greater than the minimum discrimination interval to provide additional safety margin. When evaluating potential segments, the system first calculates the continuous-valued segmentation parameters, then quantizes them to a fixed-decimal-bit format, and then checks whether the quantized parameters meet the requirements of the minimum discrimination interval and / or geometric safety distance. If the condition is not met, a non-zero cost C for the reversible stability constraint is incurred. stab (For example, a large penalty value); if satisfied, the cost term is zero. This is achieved by applying the reversible stability constraint cost C. stab By incorporating the joint cost function J, the optimization process will automatically avoid unstable partitioning schemes that are prone to index ambiguity. In a preferred implementation, if the partitioning scheme does not meet the stability requirements after quantization, the system can attempt to fine-tune the threshold within the reversible range, or directly revert to the suboptimal partitioning scheme.

[0054] According to one aspect of this application, such as Figure 4 As shown, the encoded attribute data obtained includes:

[0055] Within the region defined by the self-organizing partitioning structure, the attribute factors of the original 3D Gaussian splatter data are decomposed into: explicit low-frequency attributes used to protect the geometric and lighting primitives; and latent variables that map high-frequency details and perturbations.

[0056] Encoded attribute data is generated based on the bits that encode explicit low-frequency attributes and the control parameters used to control the reconstruction of latent variables in the generative model.

[0057] In other words, the encoded attribute data includes bits that encode explicit low-frequency attributes, as well as control parameters used to control the reconstruction of latent variables in generative models.

[0058] In this embodiment, the regions determined by the self-organizing partitioning structure correspond to leaf nodes. Gaussian attributes (such as covariance and color spherical harmonics) simultaneously contain information that determines the basic structure (low frequency) and fine texture (high frequency) of the scene. If strong quantization is applied uniformly, the basic structure will collapse; if weak quantization is applied uniformly, the bitrate will be too high, thus requiring factorization. Preferably, the attributes are decoupled. Explicit low-frequency attributes form the skeleton of the scene, such as the Gaussian positional basis, the low-frequency components of the covariance principal axis, and the low-order terms of the color spherical harmonics (such as the DC term). This data has a significant impact on visual quality and will be extracted separately, using lighter quantization and efficient entropy coding (such as arithmetic coding), and then stored as encoded bits in the bitstream. Latent variables are the remaining high-frequency details, random perturbations, and fine texture information in the attributes (such as higher-order terms of the color spherical harmonics and asymmetric perturbations of the covariance). This data is the most easily lost in traditional compression. Optionally, the mapping process is achieved through a learnable transformation, such as using a variational autoencoder (VAE) or a reversible flow model. This transformation compresses high-dimensional, high-frequency detail information into a low-dimensional, compact latent space, forming latent variables. At the decoding end, a generative model (diffusion model) is responsible for reconstructing the high-frequency details from these latent variables (or their control parameters). The control parameters are the information needed to guide the generative model at the decoding end to perform deterministic reconstruction, including, for example, the diffusion time step index and quantization step size selection.

[0059] In one possible implementation, the generative model is a diffusion model; the 3D Gaussian splash compression method further includes:

[0060] An offline calibration rate control mapping table is used, which establishes a monotonic correspondence between the diffusion time step of the diffusion model and the preset target bit rate. The diffusion model is trained by minimizing the mean square error between the predicted noise and the actual noise. The determination of the control parameters is based on the rate control mapping table.

[0061] In other words, offline calibration establishes a rate control mapping table that establishes a monotonic correspondence between the diffusion time step of the diffusion model and the preset target bit rate; based on the rate control mapping table, control parameters are determined.

[0062] In this embodiment, the generative model is preferably a diffusion model. The diffusion model, by simulating the forward process of progressive noise addition and the reverse process of progressive denoising, demonstrates powerful high-frequency detail recovery capabilities in image and data generation tasks. The diffusion timestep (t) of the diffusion model is intrinsically related to the compression ratio (or bitrate R) and reconstruction quality: smaller timesteps t (closer to the original data) retain more details and have higher reconstruction quality, but require a higher bitrate for encoding; larger timesteps t (closer to pure noise) have lower information entropy, are easier to compress, and have a lower bitrate, but require the model to guess more details during reconstruction, which may lead to distortion. Therefore, this relationship can be quantified by offline calibration (e.g., experimenting on a large number of typical scene datasets) of the rate control mapping table. The monotonic correspondence means that as the target bitrate R increases, the chosen diffusion timestep t should monotonically decrease (i.e., retain more original information and reduce the degree of diffusion). Preferably, the process of determining the control parameters is as follows: during encoding, the system determines the target bitrate R (e.g., through a global bitrate allocation strategy or user specification), consults the rate control mapping table, finds the diffusion time step t that best matches R, and writes t (or its index) as part of the control parameters into the bitstream.

[0063] In a further embodiment, it also includes:

[0064] Construct a signal-to-noise ratio (SNR) function that describes the scheduling of the SNR of the diffusion model as a function of the diffusion time step;

[0065] Furthermore, the monotonic correspondence in the rate control mapping table is determined by R≈a×SNR(t). -b The SNR(t) is obtained by calibrating the fitted model of +c; where R is the target bit rate, t is the diffusion time step, and SNR(t) = α. t / (1-α t ), α t Let be the noise attenuation coefficient of the diffusion process, SNR(t) be the value of the signal-to-noise ratio function at time step t, and a, b, and c be the fitting constants obtained from calibration.

[0066] Specifically, the diffusion schedule of the diffusion model is defined; for example, linear attenuation or cosine attenuation scheduling is preferably used. The signal-to-noise ratio (SNR) function, SNR(t), is determined by this schedule and describes the ratio of signal energy to noise energy at time step t. Based on this, extensive observational experiments are conducted on a calibration dataset to record the SNR(t) values ​​corresponding to different t values ​​and the actual code rate R required to encode explicit low-frequency attributes and control parameters. Optionally, a fitting model R≈a×SNR(t)-b+c is used, and constants a, b, and c are obtained by regression analysis (such as least squares). This model provides an approximate analytical relationship from t to R. In some preferred embodiments, other forms of fitting models can also be used to better fit the nonlinear relationship under certain specific schedules (such as cosine attenuation). For example, a power-law attenuation model can be used as follows: R≈a•SNR(t). -b +c; where a, b, and c are constants obtained through observational experiments.

[0067] In a further embodiment, the rate control map establishes a joint entry for the diffusion time step and the corresponding quantization step size set selection and entropy coding context selection; and the process of determining the control parameters includes: based on the target bit rate, consulting the joint entry in the rate control map, selecting the diffusion time step index, and simultaneously obtaining the quantization step size set selection and entropy coding context selection corresponding to the diffusion time step index; the control parameters include the diffusion time step index, the quantization step size set selection, and the entropy coding context selection.

[0068] Specifically, in actual encoding, the final bit rate R is affected not only by the diffusion time step t, but also by the quantization step size (QP) used to quantize the explicit low-frequency attributes and the entropy coding context. Therefore, it is preferable to bind these three (t, QP, Context) as a joint entry. The rate control mapping table is no longer a simple function of t and R, but a discretized lookup table from {target bit rate R} to {joint entry (t_index, qp_set_index, context_set_index)}. Here, the quantization step size set selection (qp_set_index) is an index pointing to a set of predefined quantization steps (e.g., index 2 in {QP=10, QP=20, QP=30, ...}); the entropy coding context selection (context_set_index) is the context model index used for the entropy encoder (such as an arithmetic encoder); and t_index is the diffusion time step index. Exemplarily, the process of determining the control parameters correspondingly becomes: allocating bit budgets to each leaf node based on the global target bit rate. Preferably, this allocation is weighted. For example, based on factors such as semantic importance (e.g., semantic codebook), geometric error, or field of view coverage, a higher target bitrate R is assigned to semantically significant regions (e.g., faces, text), while a lower target bitrate R is assigned to the background (e.g., sky, walls). Based on the specific target bitrate R assigned to the leaf node, the rate control mapping table is consulted, and a joint entry with the closest bitrate that does not exceed the budget is selected. The three indices contained in this joint entry (diffusion time step index, quantization step size set selection, and entropy coding context selection) are written together as control parameters into the bitstream (e.g., the leaf node table).

[0069] In a further embodiment, the method further includes: configuring a decoding deterministic configuration for the inverse process of reconstructing latent variables at the decoding end of the diffusion model; the decoding deterministic configuration includes at least a fixed random seed and a fixed number of sampling steps to ensure the consistency of the decoding reconstruction results; the compressed image data bitstream contains the decoding deterministic configuration.

[0070] In this embodiment, the inherent randomness problem in the inverse diffusion model process (denoising process) is addressed. Standard diffusion model sampling (such as DDPM or DDIM) typically introduces random Gaussian noise in each denoising step, causing the high-frequency details (latent variables) reconstructed from two decodings to differ even with the same control parameters and bitstream. This inconsistency is unacceptable in compression applications. Therefore, this randomness can be eliminated by configuring a deterministic decoding configuration. A fixed random seed is used to initialize the decoder's random number generator, ensuring that the noise sequence introduced in each sampling step of the inverse diffusion process is consistent and reproducible. A fixed number of sampling steps ensures that the number of denoising iterations performed by the decoder is consistent with the preset number by the encoder (or during calibration). The compressed image data bitstream contains a deterministic decoding configuration, meaning that this configuration information (e.g., random seed value, sampling step value) is written into the bitstream. Preferably, it can be written as a global parameter in the header field for the decoder to read and set before starting the inverse process.

[0071] This embodiment implements an attribute processing and rate control method based on a diffusion model. In other words, a generative model (specifically a diffusion model) is used to perform factorization, rate control encoding, and deterministic configuration of attributes within a self-organizing partitioning structure (such as leaf nodes).

[0072] According to one aspect of this application, it also includes:

[0073] Using multimodal semantic information, a semantic codebook is trained through vector quantization; the local semantic summaries of each node in the self-organizing partition structure are mapped to the semantic codebook index in the semantic codebook; and the structural information contains the semantic codebook index, while the compressed image data bitstream contains information for identifying the semantic codebook.

[0074] Specifically, during the recursive partitioning process, each node (e.g., a leaf node) not only possesses geometric attributes but also aggregates the multimodal semantic information corresponding to its internal Gaussian sphere. The local semantic summary refers to the value obtained by aggregating all semantic embeddings (vectors) within a node, such as the mean or centroid of all semantic vectors within that node. The semantic codebook is a compact codebook trained using a vector quantization algorithm. This codebook contains K codewords, each representing a typical local semantic (e.g., sky, skin, metal, etc.). Preferably, to handle high-dimensional semantic vectors and improve efficiency, vector quantization can employ a product quantization mechanism and can be combined with an inverted file structure to efficiently train and index the codebook. After training, the local semantic summary (high-dimensional vector) of each node is compared with all codewords in the semantic codebook (e.g., cosine similarity is calculated), and mapped to the index of the nearest codeword, i.e., the semantic codebook index (an integer). The structural information includes the semantic codebook index, meaning that during the generation of the reversible partitioning tree, this semantic codebook index is bound to the corresponding node (e.g., a leaf node) and written into the structure table. The compressed image data bitstream contains information for identifying the semantic codebook, which means that the header field contains an identifier (such as a hash value or version number) to indicate which semantic codebook (which may be pre-trained and shared) the decoder should load.

[0075] Furthermore, it also includes: compiling a semantic codebook index to form a semantic index sequence; using an error check coding mechanism to process the semantic index sequence and generate a semantic check code; the encoded attribute data contains the semantic check code for the decoding end to perform consistency verification.

[0076] In this embodiment, the Gaussian attributes reconstructed by the decoder must semantically match the semantic codebook index of its corresponding leaf node. If the decoder, for some reason (such as bitstream error or model mismatch), causes the reconstructed semantics (e.g., reconstructing wood texture) to differ from the semantics marked in the structure table (e.g., the index points to metal), a mechanism is needed to detect this inconsistency. A semantic index sequence refers to a sequence formed by arranging the semantic codebook indices of one or more leaf nodes in a predetermined order (e.g., the traversal order of the leaf nodes in the structure table). Preferably, since the semantic index sequence is usually short, the error check coding can employ a lightweight short code mechanism, such as block check (e.g., parity check) or low-density check code (LDPC). The semantic check code is an additional, short set of check bits generated after processing the semantic index sequence through the error check coding mechanism. The encoded attribute data further includes the semantic check code, meaning that the check code is written into the bitstream; for example, preferably, it can be written into the leaf node table and stored together with other control parameters (e.g., diffusion time step index) corresponding to the leaf node.

[0077] In one possible embodiment, a decoding method corresponding to the 3D Gaussian splatter compression method includes: parsing a semantic check code from the compressed image data bitstream; using the semantic check code to perform a consistency verification on the decoded and reconstructed 3D Gaussian splatter data; and triggering a decoding backoff strategy when the consistency verification fails; the decoding backoff strategy includes: adjusting the diffusion time step index or quantization step size set selection item used to reconstruct the latent variables.

[0078] Specifically, the decoder reads the semantic checksum when parsing the leaf node table. Optionally, the decoder can perform consistency verification in one or a combination of two methods: based on error check coding, the decoder performs the inverse process to reconstruct the Gaussian properties, calculates the local semantic summary of the reconstructed properties, and then maps it to the semantic codebook to obtain the decoded semantic index sequence. The decoder uses the same error check coding mechanism (such as LDPC) as the encoder to check whether the decoded semantic index sequence matches the semantic checksum read in the bitstream. Based on a similarity threshold, the decoder can also calculate the cross-modal similarity (such as cosine similarity) between the reconstructed local semantic summary and the codewords corresponding to the semantic codebook index specified in the structure table, and check whether the similarity is higher than a preset verification threshold (e.g., 0.8). If the above verification fails (e.g., the error check coding mechanism fails, or the similarity is lower than the threshold), it indicates that the reconstruction quality is too low under the current parameters (e.g., diffusion time step t), resulting in semantic drift. At this time, the decoder triggers a decoding backoff strategy. The rollback strategy aims to retry reconstruction by using more conservative (i.e., higher bitrate, but more reliable quality) decoding parameters, rather than directly reporting an error. Preferably, the decoding rollback strategy has defined execution rules, such as: the decoder first reduces the diffusion time step index by one level (e.g., if the current index points to t=100, then rollback to t=80). The smaller the time step t, the more original information is preserved, and the less prone the reconstructed semantics are to errors. Alternatively, the decoder increases the quantization step size set selection by one level (e.g., if the current index points to QP=30, then rollback to QP=20). The smaller the quantization step size, the more accurate the explicit low-frequency attributes, which also helps improve reconstruction quality. After adjusting the parameters, the decoder re-executes the reverse process and performs consistency verification again. This process can be repeated until verification passes, or the preset maximum number of rollbacks is reached (at which point a safety net path can be used, such as rendering using only explicit low-frequency attributes).

[0079] According to one aspect of this application, the self-organizing partitioning structure is a multi-resolution hash grid; the process of generating the self-organizing partitioning structure includes: calculating semantic weights based on multimodal semantic information; and adjusting the grid cells of the multi-resolution hash grid according to the semantic weights to minimize the objective function that includes semantic weighted clustering error and grid redundancy penalty.

[0080] In this embodiment, a multi-resolution hash grid is an efficient scene representation structure. It typically consists of L layers (e.g., L=8 or L=16) of hash tables with different resolutions, ranging from a coarser resolution (e.g., N... min =16 3 From a finer resolution (e.g., N) max =2 19 The three-dimensional position coordinates of each Gaussian sphere are used to query this L-layer hash table. The query results (e.g., feature vectors) are interpolated and aggregated to represent the local geometric and appearance features of the Gaussian sphere. Unlike the static use of this mesh in traditional applications (such as NeRF), this embodiment focuses on its self-organizing properties, which are achieved through semantic weights and an objective function. Specifically, the process of calculating semantic weights is as follows: Multimodal semantic information of the scene (e.g., text describing a red metallic car) and its semantic embeddings are obtained. For each region or Gaussian sphere in space, the similarity (e.g., cosine similarity) between its own features (e.g., color, position) and the semantic embedding is calculated. This similarity is mapped to a scalar value, i.e., the semantic weight W. sem For example, in the area with red cars, W sem It will be assigned a higher value (such as 0.9), while in the area of ​​blank wall, W sem They will be assigned lower values ​​(e.g., 0.1). The process of adjusting the grid cells of the multi-resolution hash grid according to semantic weights is achieved by minimizing the objective function J. hash This is achieved through [the objective function]. The objective function can preferably be expressed as: J hash =J cluster (W sem )+λ reg *J redundancy ;where λ reg This is the regularization coefficient used to balance the two terms. Semantic weighted clustering error J cluster (W sem This is used to measure whether the fineness of the grid partitioning is sufficient. It can be modeled as a weighted clustering loss (e.g., k-means loss), penalizing those cells that are partitioned into the same hash grid cell but are geometrically far apart or have semantic weights W. sem Higher Gaussian spheres. When the mesh cells contain Gaussian spheres that are geometrically dispersed or semantically important (W... sem When the resolution is high, the semantically weighted clustering error term increases, driving the optimization process to split the grid cells, i.e., automatically enabling a finer resolution level for representation in that spatial region. Grid redundancy penalty J redundancy It is used to control the compression ratio, penalizing those that are enabled but do not significantly reduce J. cluster The hash table entries (i.e., redundant grid cells). When the geometry of the region is simple and the semantic weight W sem At low temperatures (e.g., on a blank wall), J clusterVery low, at this time J redundancy The item will play a leading role in driving the optimization process of merging cells, that is, disabling high-resolution levels in this area and using only coarse levels to represent it, thereby saving storage redundancy.

[0081] In summary, this embodiment, by introducing semantic weights and the optimization objective function, enables the structure of the multi-resolution hash grid (i.e., which levels and hash entries are activated) to adapt to the complexity and semantic importance of the scene content. This solves the problem of poor scene complexity adaptability caused by the static nature of the hash grid and the independence of granularity on space complexity.

[0082] In one embodiment of this application, a high compression ratio and high-fidelity reconstruction are achieved by combining self-organizing hashing and multimodal diffusion. Specifically, the original 3D Gaussian Splash (3DGS) parameters and optional text descriptions (e.g., text encoders from the CLIP model) are loaded. Semantic features extracted from the text descriptions are combined with properties such as the opacity of the Gaussian spheres themselves to calculate the semantic weight of each Gaussian sphere or its spatial region. Regions with high opacity and high semantic matching are assigned high weights. A multi-resolution hash grid (e.g., containing 3 initial levels) is initialized based on the scene's bounding box. The grid structure is dynamically adjusted by optimizing the objective function (which includes semantically weighted clustering error and grid redundancy penalty terms). Regions with high semantic weights or high geometric complexity will adaptively use a finer grid resolution, and vice versa. The final output is the optimized hash grid structure and the features stored in the hash table entries, which can be considered as clustering anchors for the Gaussian spheres. The clustering anchor features are concatenated with the multimodal (semantic) features. The concatenated features are used as input to the diffusion model. Preferably, the diffusion model can be implemented by a Transformer encoder containing multiple layers (e.g., 3 layers). This diffusion model is trained to predict the mean of Gaussian properties (e.g., low-frequency components or statistical means of Gaussian properties), while a corresponding noise estimation model is pre-trained to perform inverse denoising at the decoder. Adaptive quantization (e.g., based on a rate-controlled mapping table) and entropy coding (e.g., arithmetic coding) are performed on the anchor points, property means, and multimodal features. The encoded data is assembled into a final compressed bitstream. In the inverse compression process, the decoder performs entropy decoding and inverse quantization from the bitstream to recover the cluster anchor points, property means, and multimodal features. The decoder invokes the pre-trained diffusion noise estimation model and performs the inverse denoising process (i.e., the inverse diffusion process). In this denoising process, the model uses the anchor points, property means, and multimodal semantic features as conditions to progressively recover the high-frequency detail properties of the Gaussian from the compressed latent representation, such as fine scaling, rotation, and higher-order terms of the color spherical harmonics. The final output is a complete, high-fidelity 3D Gaussian splatter image.

[0083] This embodiment combines the structural adaptability of hash grids with the generative detail recovery capability of diffusion models through a collaborative mechanism of semantic guidance, self-organized compression, and diffusion reconstruction. This effectively solves the problem of severe high-frequency detail distortion and texture blurring under high compression rates, and achieves high-fidelity reconstruction.

[0084] In one embodiment of this application, the implementation process of applying the 3D Gaussian splash compression method to industrial assembly line monitoring is as follows:

[0085] Obtain raw data, including 3D Gaussian splash data and multimodal semantic information.

[0086] In this embodiment, the acquired data is closely tied to an objective physical scene (e.g., an automobile assembly line). Specifically, multiple (e.g., 16) industrial cameras deployed around the physical assembly line capture a sequence of multi-view images of the assembly line at a specific moment. This image sequence is an objective optical snapshot of the physical assembly line (including robotic arms, parts to be inspected, etc.) in three-dimensional space. Simultaneously, the camera parameters corresponding to the industrial cameras are acquired, such as their physical position in the factory coordinate system, focal length, distortion parameters, etc. A specific method for acquiring multimodal semantic information in this scenario is exemplified as follows: text annotations describing areas of physical interest input by remote monitoring experts or the maintenance system, such as requesting focused monitoring of the seam at point A of robotic arm #3 or detecting surface physical scratches on the car door in area B.

[0087] Based on the geometric features and multimodal semantic information of the original 3D Gaussian splash data, a self-organizing partitioning structure is generated.

[0088] In this embodiment, initial 3D Gaussian splatter data is generated by performing multi-view geometric reconstruction on a multi-view image sequence. It is important to emphasize that this 3D Gaussian splatter data is not an abstract number, but a direct digital representation of the three-dimensional geometry of the physical assembly line. The parameters (such as position and scaling) of each Gaussian sphere directly correspond to the geometric and appearance attributes of a real surface point on the physical assembly line. Based on this, the self-organizing partitioning structure is preferably implemented as a multi-resolution hash grid. This grid typically consists of L layers (e.g., L=8 or 16) of hash tables with different resolutions, used to represent three-dimensional spatial features. The construction process of this hash grid is not arbitrary, but driven and constrained by objective physical data. Specifically, based on the physical text annotation of the seam or surface physical scratch at point A, a high semantic weight is calculated. This weight directly drives the optimization process, which dynamically adjusts the physical storage structure of the hash grid (e.g., activating or deactivating hash table entries at different resolution levels) according to an objective function that includes semantically weighted clustering error and grid redundancy penalty. The semantically weighted clustering error term automatically applies a finer mesh resolution (i.e., finer granularity) to regions with high semantic weight or high geometric complexity (such as the spatial coordinate regions of physical seams or scratches); while the mesh redundancy penalty term automatically applies a coarser mesh (i.e., coarse partitioning) to physically unimportant regions (such as blank wall backgrounds). Therefore, the topological structure of the generated intermediate data (hash grid) is a direct mapping and optimized data structure of the geometric complexity and physical semantic importance of the physical world (assembly line).

[0089] Based on the self-organizing partitioning structure and utilizing multimodal semantic information, the attributes of the original 3D Gaussian splash data are processed through a generative model.

[0090] In this embodiment, a hash grid structure reflecting physical importance is used to differentially compress data representing physical assembly line attributes. Specifically, a diffusion model is preferably used as the generative model, which recovers high-frequency details through a progressive denoising inverse process. Simultaneously, the encoding process is based on an offline-calibrated rate control mapping table. This mapping table establishes a correspondence between the target bit rate and a set of encoding parameters. Preferably, this mapping table binds the target bit rate to a joint entry containing a diffusion time step index, a quantization step size set selection, and an entropy coding context selection. The selected rate control parameters are not globally uniform. Instead, the control parameters are differentially selected based on the high semantic weight of physical defect regions (such as the A-point seam). The system automatically allocates a higher bit budget to 3DGS data representing physical seams, i.e., selecting a lower diffusion time step (to retain more original details) and a finer quantization step size set through table lookup; while for data representing the physical background, a lower bit budget is allocated, i.e., selecting a higher diffusion time step (allowing more generative compression) and a coarser quantization step size.

[0091] The structural information used to restore the self-organizing partition structure, together with the encoded attribute data, are assembled into a compressed image data stream.

[0092] In this embodiment, the size of the compressed image data stream (e.g., reduced by a factor of 100 compared to the original data) is significantly reduced, enabling it to be transmitted in real-time from the physical assembly line to a physical remote monitoring center via a physical, bandwidth-constrained factory 5G network. Furthermore, when remote experts view the reconstructed 3D scene on a physical monitor in the monitoring center, thanks to intermediate processing tightly bound to physical data (defect location, semantics), the reconstructed image maintains high fidelity in physical detail at points such as the seam at point A or the scratch in area B. This allows remote experts to accurately identify millimeter-level physical damage, which would become blurred and missed if traditional uniform compression algorithms were used due to over-compression.

[0093] In summary, this embodiment optimizes the data transmission efficiency in the physical network and improves the display accuracy of the physical defect detection system on the physical display by processing sensor data from the physical world (industrial images, defect annotation text).

[0094] In another embodiment of this application, a compression method for streaming virtual reality (VR) content is implemented, comprising: acquiring raw data of 3D Gaussian splash data, corresponding to a large, objectively existing physical scene. Specifically, a physics museum exhibition hall is photographed using a professional multi-camera array (physical acquisition equipment) to acquire a sequence of multi-view images of the physical exhibition hall. The exhibition hall (e.g., containing a physical specular metal sculpture with complex lighting and shadows) is an objectively existing physical world. Simultaneously, the physical parameters of the camera array are acquired. Multimodal semantic information, corresponding to objective attribute descriptions of the physical scene, is acquired. For example, textual descriptions input by the content creator, such as the golden sculpture in the center of the exhibition hall having a surface with specular reflections or the fine physical texture of the oil paintings on the wall. The acquired image sequence of the physical exhibition hall is reconstructed to generate initial 3DGS data representing the three-dimensional geometry and appearance of the physical exhibition hall (including its physical sculpture). A self-organizing partitioning structure (e.g., a reversible partitioning tree optimized by a joint cost function including geometric distortion, semantic divergence, and bitrate cost, or a multi-resolution hash grid dynamically adjusted according to semantic weights) is generated to spatially partition the digital representation of the physical exhibition hall. An attribute factorization strategy is preferably adopted to process the attributes of the original 3D Gaussian splatter data. This strategy is not an abstract mathematical transformation, but rather decomposes the 3D Gaussian splatter data representing the physical sculpture into: explicit low-frequency attributes: representing the basic outline and base diffuse color of the physical sculpture (e.g., low-order terms of color spherical harmonics); latent variables: representing high-frequency details of the physical sculpture surface (e.g., high-order terms of color spherical harmonics). It should be noted that these latent variables directly correspond to physical semantics, i.e., the physical lighting effect corresponding to specular reflection and the surface details corresponding to fine physical textures. During compression, the system processes these latent variables through a diffusion model, a process guided by the physical semantics of specular reflection and selecting appropriate control parameters (e.g., diffusion time step index) based on an offline calibrated rate control mapping table. The structural information used to reconstruct the self-organizing partition structure, along with the encoded attribute data, are assembled into a compressed image data stream. The generated stream contains not only the compressed data but also, preferably, a decoding deterministic configuration. This decoding deterministic configuration (e.g., a fixed random seed and a fixed number of sampling steps) is crucial for ensuring the consistency of the reconstruction results from the diffusion inverse process at the decoding end, addressing the inherent randomness problem of generative models. This configuration is written into the header field of the stream and transmitted along with the compressed data. Specifically, the size of the stream is compressed (e.g., to 1 / 200th of the original data), overcoming the physical bandwidth limitations faced by user-worn physical VR glasses (with limited computing power and storage) when streaming over physical Wi-Fi networks. A lightweight diffusion decoder deployed on the physical processing chip (e.g., a mobile GPU or NPU) of the VR glasses receives this stream.The decoder reads the deterministic decoding configuration to ensure the stability of the reconstruction. It performs an inverse diffusion process and uses control parameters carried in the bitstream, associated with physical semantics (spectral reflection), to directionally and generatively recover the high-frequency details of specular reflection on the physical sculpture's surface. Users see a highly realistic 3D scene through the physical display of their VR glasses (e.g., two OLED screens). The gold sculpture is no longer a blurry patch due to overcompression, but instead exhibits a realistic metallic luster that conforms to physical laws, enhancing the user's physical immersion experience.

[0095] In summary, this embodiment processes physical scene data (exhibition hall images, lighting text) to adapt to the physical hardware limitations (bandwidth, computing power) of mobile devices, and controls the physical display device (VR screen) to reconstruct high-fidelity physical lighting effects.

[0096] In one possible embodiment, a 3D Gaussian splatter compression method includes: obtaining an initial 3D Gaussian set, the initial 3D Gaussian set containing attributes characterizing Gaussian geometry and appearance, and obtaining semantic embeddings associated with the initial 3D Gaussian set, the semantic embeddings being generated based on image, text, or speech data; generating a self-organizing partition structure based on the geometric features characterized by the attributes of the initial 3D Gaussian set and the semantic embeddings; processing the attributes of the initial 3D Gaussian set using a generative model based on the self-organizing partition structure and utilizing the semantic embeddings to obtain encoded attribute data; and assembling the structural information used to reconstruct the self-organizing partition structure and the encoded attribute data together into a compressed data stream.

[0097] According to one aspect of this application, a 3D Gaussian splash compression method includes: extracting semantic vectors from scene images and text descriptions to guide subsequent compression priorities. The multimodal input can be represented as: z s =f m (I, T); where I represents the scene image, T represents the text description, and the function f m It consists of a CLIP visual encoder fused with a Transformer. For each Gaussian point G... i Based on spatial distribution and semantic weight w i Self-organizing to multi-resolution hash tables. A self-organizing hash map is defined as L... hash To minimize clustering error and grid redundancy, the process can be expressed as: L hash =∑ j 1 / N j ∑ i∈j w i ·∣∣μ i -μ* j ||2 2 +λ·∑ j Dsem (z i , z j ); where μ i It is a hash distribution, μ* j D is the mean of the Gaussian positions within the cell. sem For semantic distance measurement, λ is the redundancy penalty coefficient, and w i It refers to semantic weights. Semantic weights are used to split grid cells into regions, reducing clustering errors. Cells with low semantic weights are merged to reduce redundancy; N j z is the number of Gaussian points in the j-th hash cell; i , z j Let G be the semantic vector of the Gaussian point. For each hash unit, the Gaussian attribute set {G} h} Mapped to the latent space via encoder E: z h =E({G h}); The compression process follows conditional diffusion forward noise addition: q(z t |z0)=μ(sqrt(α) t )z0,(1-α t )); where q(z) t |z0) represents the forward process probability distribution of the diffusion model, indicating the diffusion from the original latent representation z0 to the representation z at step t. t z0=z h ;α t Let be the noise attenuation coefficient, and let μ be the mean function of the Gaussian distribution, obtained by introducing noise perturbation to achieve a controllable compression ratio. The decoder D recovers high-frequency details based on a reverse diffusion process, which can be expressed as: p(z... t-1 ∣z t , z s )=μ(v(z t , z s ), ∑(z t , z s )); where p(z) t-1 ∣z t , z s Let z be the probability distribution of the inverse process of the diffusion model, representing the expression z from step t. t Reverting to step t-1, the condition is multimodal semantic guided feature z. s Multimodal semantic-guided features enable the model to focus on semantically key regions, such as highlights and texture edges, during the denoising process; v(z t , z s ) is the mean prediction function for the inverse diffusion process; ∑(z) t , z s Let L be the variance prediction function in the inverse diffusion process. The reconstruction loss can be expressed as: L rec=R(y*)+R(z*)+λ·D(x, x*); R(y*)=E[-log2(p y*∣z* (y*|z*))]; R(z*)=E[-log2(p z*∣Ψ (z*∣Ψ))];where y*, z*, x, x* represent the quantized latent representation, the super-prior latent representation, the original image, and the reconstructed image, respectively; R(y*) is the bitrate estimation loss for the quantized latent representation y*; R(z*) is the bitrate estimation loss for the super-prior latent representation z*; D(x, x*) is the reconstruction error measure between the original image x and the reconstructed image x*; p y*∣z* (y*|z*) represents the probability distribution of the quantized latent representation y* given the prior latent representation z*; p z*∣Ψ (z*|Ψ) represents the probability distribution of the latent representation z* given the prior model Ψ. The semantic consistency loss can be expressed as: L sem =∣∣Φ(I*, T*)-Φ(I, T)∣∣2 2 Where Φ() represents the pre-trained CLIP model, and I*, T*, I, and T represent the encoded image and text information, and the unencoded image and text information, respectively. Diffusion consistency loss: L diff =E t，ε ∣∣ε-ε θ (z t ,t,z s )∣∣;where E t，ε This represents the expectation of the distribution of time step t and the true noise ε, where ε is the expected value. θ z represents the noise predicted by the model. t ,t,z s Let L represent the diffused samples at time step t, the diffused samples at time step s, and the diffused samples at time step s, respectively. The overall loss is: L total =L rec +α1L sem +α2L diff +α3L hash , where α1, α2, and α3 are different weighting coefficients.

[0098] This embodiment utilizes a multimodal diffusion model to automatically identify semantically salient regions (such as object edges and reflective surfaces) during compression. These semantically salient regions are automatically assigned higher bit allocations and lower quantization errors, while redundant background regions are compressed more compactly, thus achieving semantically weighted compression. At the same bit rate, compared to the traditional uniformly quantized Gaussian point compression (GS-Compress) method, it improves the peak signal-to-noise ratio (PSNR) and reduces the perceptual quality index (LPIPS) on the Tanks and Temples and Mip-NeRF360 datasets, achieving a higher compression ratio without changing or improving visual quality. Traditional hash grids use fixed partitioning rules, with granularity independent of spatial complexity. This embodiment introduces a self-organizing hash map based on both semantic and geometric distances. The hash table can dynamically adjust its resolution according to scene complexity, allowing for finer-grained partitioning of complex regions and coarser partitioning of simple regions. Under the same compression ratio, it reduces the average number of Gaussians, lowers scene geometric errors compared to the Hierarchical Aggregate Clustering (HAC) method, and reduces storage redundancy while maintaining rendering consistency. Traditional generative compression uses autoencoders to recover Gaussian parameters using latent variables, resulting in smooth but texture-lacking reconstructions. This embodiment introduces a diffusion generative model, incorporating multimodal semantic conditions during the diffusion decoding stage to achieve high-frequency texture reconstruction and recover detailed features that traditional autoencoders cannot generate. At the same bitrate, the perceptual quality index (LPIPS) of diffusion decoding reconstruction is superior to that of autoencoders, improving texture recovery and significantly enhancing the preservation of visually visible details.

[0099] In one possible embodiment, generating the self-organizing partitioning structure can also involve: reading the initial Gaussian set, semantic embedding, and normalized configuration; establishing root nodes on a sparse spatial grid or octagonal space; and initializing a candidate node set as the starting point for recursive partitioning. The candidate node set and its initial Gaussian set are read, and volume rendering error and coverage error are evaluated on node voxels or ray samples to construct a geometric distortion metric. A weighted average reconstruction error of the voxel sampling points and the rendering error along the view direction is used: Geometric distortion cost = weighted volume rendering error + voxel coverage difference; a geometric statistical summary is output (containing the mean, variance, and voxel coverage rate of the reconstruction error for each candidate node). The candidate node set and semantic embedding are read, and the intra-class scatter and spherical divergence of the semantic centroid are calculated for the semantic vectors within each candidate node to form a semantic divergence metric. Normalized cosine distance is preferably used, and outlier samples are truncated to enhance robustness. A semantic statistical summary is output (containing the semantic centroid, intra-class divergence, and outlier percentage for each candidate node). Read the candidate node set and normalized configuration. Based on the bit overhead of writing segmented records into the bitstream plus the incremental bits brought about by entropy coding context changes, construct a rate cost model, including: segmentation type flag, number of quantized bits for segmentation hyperplane direction and threshold, node index and child node identification overhead, and bit estimation of potential context switching. Output the rate cost model parameters (bit budget and context penalty coefficient for different segmentation types). Read the normalized configuration and geometric distribution within the candidate nodes. To maintain decoding consistency after quantization of the segmentation hyperplane parameters, define reversible stability constraints. The constraints should include at least: lower limit of the discrimination interval after quantization, geometric safe distance from the nearest sample, and fixed decimal place format of the segmentation parameters. Output the stability threshold configuration (quantization scale, minimum discrimination interval, and safe distance). Read the geometric statistical summary, semantic statistical summary, rate cost model parameters, and stability threshold configuration to establish a joint objective: J = geometric distortion cost + λ s Semantic divergence cost + λ r • Bitrate cost + λ c • Cost of reversible stability constraints; where λ s , λ r , λ cThe algorithm calibrates on a fixed dataset using validation set grid search or Bayesian optimization to ensure stability across different scenarios. It reads the candidate node set, joint cost function definition, and weight configuration, and evaluates the potential benefits of planar segmentation versus axis-aligned segmentation for each candidate node, ranking them by distortion reduction per unit bitrate to obtain a priority segmentation node list. It then reads the priority segmentation node list, the initial Gaussian set within each node, and the semantic embedding, optimizing the segmentation normal and threshold under the joint cost, while employing robust strategies to avoid noise influence (such as quantile constraints in geometric and semantic statistics), outputting a candidate set of segmentation parameters (continuous solutions for the normal vector and threshold). Finally, it reads the candidate set of segmentation parameters and the stability threshold configuration, quantizing the segmentation normal and threshold to a fixed decimal place format, and then verifying whether the interval and safety distance meet stability requirements after quantization; if not, it adjusts the threshold within the reversible range or reverts to a suboptimal segmentation scheme, outputting the quantized segmentation parameters (which can be directly written into the bitstream's segmentation type and quantization parameters). Read the quantization segmentation parameters and bitrate cost model parameters, and back off when the bitrate increment is high but the distortion benefit is insufficient. Terminate the segmentation of the current branch when the weighted error of geometry and semantics meets the lower limit or reaches the upper limit of the layer number, outputting a segmentation stop flag and necessary backoff records to ensure the consistency of the subsequent structure. Read the quantization segmentation parameters and segmentation stop flag, and generate segmentation records according to the structure table requirements, including segmentation type, quantization parameters, node number, parent-child relationship, and semantic codebook index binding placeholders. Perform simple checks on the records (such as check bit consistency with node count), and output the quantization segmentation parameters and segmentation records. Perform vector quantization on the local semantics of nodes, train and quantize the semantic codebook (e.g., based on product quantization and inverted file structure), map the local semantic summary of each node to a semantic codebook index so that encoding and decoding consistently reference semantic conditions, and output the semantic codebook and the set of semantic codebook indices for each node. A reversible partitioning tree is generated based on the quantization segmentation parameters and the semantic codebook index set, and the leaf node set is derived. For each node, the segmentation type, quantization segmentation parameters, corresponding semantic codebook index and stop flag are recorded to ensure that the decoding end can restore the structure based on the records alone, without having to rerun the training.

[0100] In one possible embodiment, obtaining the encoded attribute data can also involve: reading the reversible partitioning tree and the set of leaf nodes, factorizing the Gaussian attributes within each leaf node: organizing the low-frequency components that ensure the geometric and lighting primitives into explicit low-frequency attributes (position primitives, low-frequency covariance principal axes, low-order color spherical harmonics, etc.), and mapping high-frequency details and perturbations into latent variables through learnable transformations (such as variational autoencoders or reversible flow), forming leaf-level attribute decomposition, and outputting explicit low-frequency attributes and latent variables as two paths for encoding and diffusion reconstruction. Reading the set of leaf nodes and the reversible partitioning tree, and combining it with typical scene datasets, setting several target bitrate sets and quality index sets for mapping calibration experiments, outputting a calibration input set (containing a statistical summary and target bitrate set for each leaf node). Reading the calibration input set and normalization configuration, selecting a diffusion schedule (such as linear or cosine decay) and defining a function of the signal-to-noise ratio changing with the diffusion time step, outputting the diffusion schedule configuration and the signal-to-noise ratio function. Read the diffusion scheduling configuration and signal-to-noise ratio (SNR) function. Conduct combined experiments with several quantization step sizes and entropy coding contexts at different diffusion time steps, recording the actual bitrate and reconstruction quality observations (for each leaf node). Output a set of observation curves (a ternary relationship between diffusion time step, bitrate, and quality). Read the set of observation curves and fit the relationship between the diffusion time step and the target bitrate using a family of monotonic functions. Recommended model: Bitrate ≈ a·SNR(diffusion time step) -b+c, where a, b, and c are scene-independent fitting constants, and SNR is the signal-to-noise ratio function. The continuous output is discretized and pruned to form a leaf-level lookup table from the diffusion time step index to the bit rate. This table is then combined with available quantization step size selections and entropy coding context selections to create joint entries, outputting a rate control mapping table (one for each leaf node or shared by category). The rate control mapping table and reserved verification data are read to verify the monotonicity and upper bound of the mapping under different scenarios and leaf nodes. If anomalies are found, the process returns to the combined experiment step to supplement edge samples or adjust the fitting model. The final rate control mapping table and its identifier are output. The leaf node set, semantic codebook, and final rate control mapping table are read. Based on semantic importance (e.g., semantic centroid representativeness), geometric error, and field coverage, the total available bit rate is allocated at the leaf level to obtain a bit budget allocation scheme. The bit budget allocation scheme and the final rate control mapping table are read. Under the premise of meeting the budget, a diffusion time step index and corresponding quantization step size selections and entropy coding context selections are selected for each leaf node, prioritizing the quality and stability of semantically significant regions. Read the diffusion time step index of each leaf node, fix the random seed and sampling steps for the reverse diffusion process, ensuring consistent reconstruction results at the decoding end under the same bitstream and configuration, and output the decoding deterministic configuration (random seed and sampling steps). Read the diffusion time step index, quantization step size set selection, entropy coding context selection, and decoding deterministic configuration of each leaf node, and check whether they are all within the allowable range of the final rate control mapping table; if out of bounds or conflicts are found, return to the step of selecting the diffusion time step index and the corresponding quantization step size set selection and entropy coding context selection, and reselect, outputting the leaf-level selection record set. Read the semantic codebook and leaf node set, summarize the local semantic summary for each leaf node and map it into a semantic index sequence (product quantization and inverted index structure codeword index sequences can be used). Read the semantic index sequence, select a short code error checking mechanism (such as block check or low-density check code), encode the semantic index sequence into a semantic check code, and ensure that the decoding end can verify the semantic consistency after local rendering. Read the normalized configuration and typical validation set, set similarity thresholds (e.g., lower limit for cross-modal similarity) and error check passing conditions to ensure no misjudgment under normal jitter and timely triggering during semantic shifts, and output the validation threshold configuration. Read the semantic checksum and validation threshold configuration, and agree on the backoff strategy at the decoding end: when validation fails or similarity is below the threshold, first reduce the diffusion time step index by one level or increase the quantization step size set selection by one level. If it still fails twice consecutively, a fallback path is activated (e.g., selecting a more conservative parameter set). Output the backoff strategy configuration as an explanation and field identifier, and execute it at the decoding end according to the protocol. Quantize and entropy encode the explicit low-frequency attributes according to the selected quantization step size set selection to obtain the explicit attribute bit string used for bitstream writing; at the same time, organize the diffusion time step index, entropy encoding context selection, and semantic checksum of each leaf node to form the field set required for bitstream assembly.

[0101] This invention theoretically optimizes the compression rate and fidelity in 3DGS compression through a collaborative mechanism of semantic guidance, self-organized compression, and diffusion reconstruction. In engineering, it improves storage efficiency, transmission bandwidth, and rendering consistency, demonstrating potential for industrial applications. Specifically, it acquires multimodal semantic information, such as text or images, corresponding to the 3D scene (e.g., physical defects, specular materials), in the initial stage. This semantic information is then used to guide subsequent compression processes. For example, during bit budget allocation, the system can weight regions based on semantic importance, assigning higher bit rates and finer quantization parameters to regions carrying key semantics (e.g., scratches on industrial parts), while employing more aggressive compression for redundant background regions (e.g., blank walls), thus achieving semantically aware differentiated compression. A self-organized partitioning structure is also proposed. Unlike static meshes, this structure's generation process is dynamic and adaptive. For example, the structure can be an invertible partitioning tree optimized by a joint cost function that simultaneously minimizes geometric distortion and semantic divergence; or a multi-resolution hash mesh whose grid cells are dynamically adjusted based on semantic weights and geometric complexity, automatically refining in complex regions and coarsely partitioning in simple regions. This allows the spatial partitioning structure to adaptively match the geometric and semantic complexity of the scene. It also employs a strategy of attribute factorization and generative reconstruction. Gaussian attributes are decomposed into explicit low-frequency attributes requiring precise encoding (such as basic contours) and easily lost high-frequency latent variables (such as texture details). At the decoding end, generative models (such as diffusion models) are used to reconstruct high-frequency details from the latent variables. This reconstruction process is not blind guessing, but is conditionally guided by multimodal semantic information and low-frequency attributes, thus enabling the recovery of physically accurate fine textures and lighting effects that would otherwise be lost in traditional quantization.

[0102] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A 3D Gaussian sputtering compression method, characterized in that, include: A multi-view image sequence was acquired, and raw 3D Gaussian splatter data, including the position, anisotropic covariance, color spherical harmonic coefficients, and opacity of the 3D Gaussian splatter, as well as the corresponding multimodal semantic information, was extracted. The multimodal semantic information includes multi-view images and text prompts. The multimodal semantic information was obtained through a cross-modal coding model. Based on the geometric features and multimodal semantic information of the original 3D Gaussian splash data, a self-organizing partitioning structure is generated; Based on the self-organizing partitioning structure and utilizing multimodal semantic information, the attributes of the original 3D Gaussian splash data are processed through a pre-configured generative model to obtain encoded attribute data; where the generative model is a model that can reconstruct high-fidelity data from a compact latent representation. The structural information used to restore the self-organizing partition structure and the encoded attribute data are assembled together into a compressed image data stream.

2. The method according to claim 1, characterized in that, The self-organizing partitioning structure is a reversible partitioning tree; Generate a self-organizing partition structure, including: Based on geometric features and multimodal semantic information, the splitting parameters used to construct the reversible partitioning tree are recursively determined. The recursion is terminated when the splitting benefit corresponding to the splitting parameter is lower than the preset threshold or the number of nodes reaches the upper limit. The segmentation parameters are quantized to form quantized segmentation parameters; The structural information used to reconstruct the self-organizing partition structure includes quantized segmentation parameters to support deterministic structural reconstruction of the reversible partition tree at the decoding end.

3. The method according to claim 2, characterized in that, The segmentation parameters are determined recursively, including: A joint cost function is constructed based on geometric features and multimodal semantic information; The joint cost function is used to evaluate the splitting benefit of each node in the pre-defined candidate node set. Based on the splitting reward, select at least one node from the candidate node set and determine its corresponding splitting parameters.

4. The method according to claim 3, characterized in that, The joint cost function includes: The geometric distortion cost is obtained by evaluating the fitting error within nodes based on geometric features. The semantic dispersion within a node is evaluated based on multimodal semantic information to obtain the semantic dispersion cost; Evaluate the bit overhead required to write the segmentation parameters into the bitstream to obtain the bitrate cost; Based on the splitting profit, select at least one node from the candidate node set, including: The reduction in geometric distortion cost and semantic divergence cost is taken as the benefit, and the bitrate cost is taken as the cost. The unit bitrate distortion reduction benefit of each node in the candidate node set is calculated. Choose the node with the highest per-unit bitrate distortion reduction benefit.

5. The method according to claim 4, characterized in that, Constructing a joint cost function further includes: Based on the requirement of deterministically restoring the reversible partitioning tree at the decoding end, reversible and stable constraints are constructed. Based on the reversible stability constraint, the decoding uncertainty caused by the segmentation parameter during the quantization process is evaluated, and the cost of the reversible stability constraint is obtained. The joint cost function includes the cost of reversible stability constraints.

6. The method according to claim 5, characterized in that, Constructing reversible stability constraints includes: A minimum discrimination interval is set for the segmentation parameters after quantization to avoid index ambiguity caused by differences in floating-point operations at the decoding end; the quantization process includes converting the segmentation parameters into a fixed decimal place format; The cost of reversible stability constraints arises when the minimum discrimination interval is not met after the segmentation parameters are converted to a fixed decimal place format.

7. The method according to claim 1, characterized in that, The encoded attribute data includes: Within the region defined by the self-organizing partitioning structure, the attribute factors of the original 3D Gaussian splatter data are decomposed into: explicit low-frequency attributes used to protect the geometric and lighting primitives; and latent variables that map high-frequency details and perturbations. Encoded attribute data is generated based on the bits that encode explicit low-frequency attributes and the control parameters used to control the reconstruction of latent variables in the generative model.

8. The method according to claim 7, characterized in that, The generative model is a diffusion model; the method further includes: An offline calibration rate control mapping table is provided, which establishes a monotonic correspondence between the diffusion time step of the diffusion model and the preset target bit rate; wherein, the diffusion model is trained by minimizing the mean square error between the predicted noise and the actual noise; The process of determining control parameters is based on the rate control mapping table; To reconstruct the inverse process of latent variables at the decoding end of the diffusion model, a decoding deterministic configuration is configured. The decoding deterministic configuration includes at least a fixed random seed and a fixed number of sampling steps, and the compressed image data bitstream contains the decoding deterministic configuration.

9. The method according to claim 8, characterized in that, Also includes: Construct a signal-to-noise ratio (SNR) function that describes the scheduling of the SNR of the diffusion model as a function of the diffusion time step; Furthermore, the monotonic correspondence in the rate control mapping table is determined by R≈a×SNR(t). -b +c was obtained from the calibration of the fitted model; Where R is the target bit rate, t is the diffusion time step, SNR(t) is the value of the signal-to-noise ratio function at diffusion time step t, and a, b, and c are the fitting constants obtained from calibration.

10. The method according to claim 8, characterized in that, The rate control mapping table establishes a joint entry for the diffusion time step, the corresponding quantization step size set selection, and the entropy coding context selection; Furthermore, the process of determining the control parameters includes: Based on the target bit rate, consult the joint entry in the rate control mapping table, select the diffusion time step index, and simultaneously obtain the quantization step size set selection and entropy coding context selection corresponding to the diffusion time step index; The control parameters include the diffusion time step index, quantization step size set selection, and entropy coding context selection.

Images