Model compression method, apparatus, device, and storage medium

Abstract

The application relates to the technical field of computer vision, and discloses a model compression method, device, equipment and storage medium, which comprises the following steps: acquiring an initial four-dimensional Gaussian model; calculating the fusion space-time importance score of each Gaussian cell in the initial four-dimensional Gaussian model, and pruning the Gaussian cell based on the fusion space-time importance score to obtain a four-dimensional Gaussian model; training the four-dimensional Gaussian model to obtain a trained four-dimensional Gaussian model; and compressing each Gaussian cell based on the feature parameters and the fusion space-time importance score of each Gaussian cell in the trained four-dimensional Gaussian model to obtain a compressed four-dimensional Gaussian model. By identifying and removing Gaussian cells with low contribution in the space-time dimension, and compressing the feature parameters of the remaining Gaussian cells after training, the model storage volume is compressed, the memory access overhead is reduced, and the deployment capability of the model in a resource-limited device is effectively improved under the premise of maintaining negligible visual quality loss.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a model compression method, apparatus, device and storage medium. Background Technology

[0002] Gaussian splashing technology uses an explicit Gaussian primitive representation, introducing time-varying attribute parameters (such as position, rotation, scaling, and spherical harmonic coefficients) to achieve high-quality real-time rendering of dynamic scenes. However, existing four-dimensional Gaussian splashing technology lacks an adaptive control mechanism for the number of primitives, resulting in a large number of low-contribution Gaussian primitives stored in full-precision floating-point format in both spatial and temporal dimensions. This consumes a large amount of memory and storage space, restricting the deployment of the model on terminal devices. Summary of the Invention

[0003] The main purpose of this application is to provide a model compression method, apparatus, device and storage medium, which aims to solve the technical problem that the existing four-dimensional Gaussian splashing technology consumes a large amount of memory and computing resources.

[0004] To achieve the above objectives, this application proposes a model compression method, which includes: Obtain an initial four-dimensional Gaussian model, which is a model obtained after initialization based on a multi-view image sequence; Calculate the spatiotemporal importance score of each Gaussian element in the initial four-dimensional Gaussian model, and prune the Gaussian elements based on the spatiotemporal importance score to obtain the four-dimensional Gaussian model. The four-dimensional Gaussian model is trained to obtain the trained four-dimensional Gaussian model; Based on the feature parameters of each Gaussian element in the trained four-dimensional Gaussian model and the fusion spatiotemporal importance score, each Gaussian element is compressed to obtain a compressed four-dimensional Gaussian model.

[0005] Optionally, the step of calculating the fusion spatiotemporal importance score of each Gaussian element in the initial four-dimensional Gaussian model includes: Based on the opacity of each Gaussian element in the initial four-dimensional Gaussian model, calculate the spatial importance score of each Gaussian element; The time importance score of each Gaussian element is calculated based on its normalized volume and cumulative dynamic change intensity. Based on the spatial importance score and the temporal importance score, the fusion spatiotemporal importance score of each Gaussian element is calculated.

[0006] Optionally, the step of calculating the spatial importance score of each Gaussian element based on the opacity of each Gaussian element in the initial four-dimensional Gaussian model includes: Obtain the opacity of each Gaussian element, and calculate the transmittance of each Gaussian element based on the opacity; For each Gaussian cell, the spatial importance score of the Gaussian cell is calculated using the opacity and the transmittance of all Gaussian cells in front of the Gaussian cell.

[0007] Optionally, the step of calculating the time importance score of each Gaussian element based on its normalized volume and cumulative dynamic change intensity includes: The activity distribution function of each Gaussian element in time is determined by a Gaussian function, wherein the parameters of the Gaussian function include the time center and the time scale. The dynamic change intensity of each Gaussian element is determined by the active distribution function; The cumulative dynamic change intensity of each Gaussian element is calculated based on the dynamic change intensity. The normalized volume of each Gaussian element is obtained, and the normalized volume is fused with the cumulative dynamic change intensity to generate the time importance score of each Gaussian element.

[0008] Optionally, the step of obtaining the initial four-dimensional Gaussian model includes: Three-dimensional reconstruction is performed on each frame of the received multi-view image sequence to obtain the initial three-dimensional point cloud corresponding to each frame of the image; The initial three-dimensional point cloud is stitched together in chronological order to obtain a four-dimensional spatiotemporal point cloud; The initial four-dimensional Gaussian model is obtained by initializing the four-dimensional spatiotemporal point cloud with the four-dimensional Gaussian primitive.

[0009] Optionally, the step of training the four-dimensional Gaussian model to obtain the trained four-dimensional Gaussian model includes: Randomly select time indices and corresponding observation perspectives from the multi-view image sequence to generate training data for each iteration; For each iteration, based on the training data, the image is rendered using the four-dimensional Gaussian model to obtain the rendered image; Calculate the composite loss between the rendered image and the corresponding real image, the composite loss including image reconstruction loss and entropy regularization loss; Based on the composite loss, the parameters of the four-dimensional Gaussian model are updated through backpropagation to obtain the trained four-dimensional Gaussian model.

[0010] Optionally, the step of compressing each Gaussian unit based on the feature parameters of each Gaussian unit in the trained four-dimensional Gaussian model and the fused spatiotemporal importance score to obtain a compressed four-dimensional Gaussian model includes: Based on the fusion spatiotemporal importance score, the color features in the feature parameters of each Gaussian element are classified to generate a first color feature and a second color feature. The fusion spatiotemporal importance score of the Gaussian element corresponding to the first color feature is less than that of the second color feature. Cluster the first color features and generate a shared codebook based on the clustering results; The parameters of the first color feature are replaced with the codebook index of the shared codebook to obtain the first compression result; The second color feature and other features in the feature parameters are compressed and stored using a preset precision format to obtain a second compression result; The compressed four-dimensional Gaussian model is determined based on the first compression result and the second compression result.

[0011] Furthermore, to achieve the above objectives, this application also proposes a model compression device, which includes: The model initialization module is used to obtain an initial four-dimensional Gaussian model, which is a model obtained after initialization based on a multi-view image sequence. The primitive pruning module is used to calculate the fusion spatiotemporal importance score of each Gaussian primitive in the initial four-dimensional Gaussian model, and prune the Gaussian primitives based on the fusion spatiotemporal importance score to obtain the four-dimensional Gaussian model. The model training module is used to train the four-dimensional Gaussian model to obtain the trained four-dimensional Gaussian model. The primitive compression module is used to compress each Gaussian primitive based on the feature parameters of each Gaussian primitive in the trained four-dimensional Gaussian model and the fusion spatiotemporal importance score, to obtain a compressed four-dimensional Gaussian model.

[0012] In addition, to achieve the above objectives, this application also proposes a model compression apparatus, the apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the model compression method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the model compression method described above.

[0014] This application discloses a method that introduces an adaptive pruning mechanism based on spatiotemporal importance to identify and remove Gaussian primitives with low contributions in both spatial and temporal dimensions, achieving a significant reduction in model parameters while maintaining rendering quality. Simultaneously, by designing a multi-scale-aware composite loss function, spatial details are preserved during training, while simultaneously driving the reasonable distribution of Gaussian primitives on object surfaces, effectively suppressing rendering artifacts and ensuring high-quality visual output. Furthermore, after training, codebook-based vector quantization converts high-precision floating-point parameters into compact discrete representations, compressing the feature parameters of the remaining Gaussian primitives. This reduces model storage size and memory access overhead while maintaining negligible visual quality loss, effectively improving the model's deployment capability on resource-constrained devices. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the first embodiment of the model compression method of this application; Figure 2 This is a schematic diagram of the overall architecture of the model compression method in this application; Figure 3 This is a flowchart illustrating the second embodiment of the model compression method of this application; Figure 4 This is a flowchart illustrating the third embodiment of the model compression method of this application; Figure 5 This is a flowchart illustrating the fourth embodiment of the model compression method of this application; Figure 6 This is a schematic diagram of the core compression process of a model compression method. Figure 7 This is a schematic diagram of the module structure of the model compression device in an embodiment of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the model compression method in this application embodiment.

[0018] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0020] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0021] In recent years, significant progress has been made in dynamic scene reconstruction techniques based on NeRF (Neural Radiance Fields). These methods utilize deep neural networks to implicitly learn the continuous changes in scene appearance over time, enabling the synthesis of visually high-fidelity new perspective video sequences from multi-view videos. However, these methods require intensive queries on complex neural networks to render each frame, resulting in lengthy training cycles and slow rendering speeds. This makes it difficult to meet real-time interactive requirements, severely limiting their practical deployment and application in latency-sensitive scenarios such as mobile terminals and virtual reality.

[0022] To overcome the aforementioned bottlenecks, 4D (Four-Dimensional) Gaussian splashing technology uses an explicit Gaussian primitive representation. By introducing time-varying attribute parameters (such as position, rotation, scaling, and spherical harmonic coefficients), it achieves high-quality real-time rendering of dynamic scenes. In this technology, traditional 4D Gaussian methods typically assume that Gaussian primitives undergo linear motion with a fixed temporal covariance, while the spatiotemporal Gaussian framework further introduces continuous spatiotemporal primitives to flexibly simulate dynamic motion, enabling more accurate capture of various motion modes, from rigid body transformations to complex non-rigid deformations.

[0023] While the Spatiotemporal Gaussian model excels in dynamic modeling and rendering, it still faces two challenges as practical applications increasingly demand lightweight models and efficient deployment: firstly, the lack of an adaptive mechanism for controlling the number of primitives leads to severe redundancy; secondly, the model parameters are always stored in full-precision floating-point format, both of which result in high storage and computational resource overhead, ultimately limiting the framework's deployment on terminal devices. Therefore, there is an urgent need to develop targeted compression and optimization methods to improve its practical applicability.

[0024] Therefore, this application provides a model compression method for dynamic 3D scene reconstruction. For a 4D Gaussian splash model based on spatiotemporal Gaussian, a collaborative model pruning and parameter quantization compression scheme is constructed. The aim is to systematically eliminate parameter redundancy in the spatial and temporal dimensions of the model, and to compress the attribute representation of Gaussian primitives through codebook quantization technology. This significantly reduces model storage volume and memory access overhead while maintaining negligible visual quality loss, and effectively improves its deployment capability on resource-constrained devices.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, model training, and program execution functions, such as a computer, or an electronic device capable of performing the above functions. The following description uses a model compression system as an example to illustrate this embodiment and the subsequent embodiments.

[0026] Based on this, embodiments of this application provide a model compression method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the model compression method of this application.

[0027] In this embodiment, the model compression method includes: Step S10: Obtain an initial four-dimensional Gaussian model, which is a model obtained after initialization based on a multi-view image sequence.

[0028] It should be noted that the initial four-dimensional Gaussian model is an explicit parameterized model used to represent dynamic 3D scenes. The basic building block of the model is a four-dimensional Gaussian primitive. Each Gaussian primitive contains a series of parameters describing the image's shape in 3D space (such as position, rotation, and scaling), appearance (such as color and opacity), and dynamic properties that evolve over time. The initial four-dimensional Gaussian model can synthesize corresponding 2D images from any viewpoint and time using Gaussian splash rendering technology. Furthermore, in this embodiment, the initial four-dimensional Gaussian model is also the initial model to be processed, requiring spatiotemporal pruning and quantization compression.

[0029] It is understandable that a multi-view image sequence is a collection of images acquired synchronously at consecutive time points from multiple different spatial observation angles for the same dynamic scene. It contains both the spatial structure information of the scene from different perspectives and the dynamic changes of the scene in the time dimension.

[0030] Specifically, as an optional implementation, obtaining the initial four-dimensional Gaussian model can be achieved by performing three-dimensional reconstruction on each frame of the multi-view image sequence using a structure for motion reconstruction and a multi-view stereo algorithm, resulting in a sparse or dense three-dimensional point cloud at each moment. Next, the three-dimensional point clouds from different moments are aligned and stitched together along the time dimension based on timestamps or through a temporal registration algorithm, forming a unified four-dimensional spatiotemporal point cloud sequence. Finally, based on this four-dimensional spatiotemporal point cloud, a four-dimensional Gaussian primitive is instantiated for each point, and initial values ​​are assigned to its position, color, scale, rotation, and other attributes, thereby completing the construction of the initial four-dimensional Gaussian model. For example, the open-source tool COLMAP can be used to perform three-dimensional reconstruction of each frame, and the reconstruction results can be combined in chronological order.

[0031] Alternatively, in another embodiment, an end-to-end neural network can be used to directly regress the initial four-dimensional Gaussian meta-parameter set from the multi-view image sequence, thereby skipping the explicit three-dimensional reconstruction and point cloud stitching steps.

[0032] It should be understood that the number of initial four-dimensional Gaussian primitives is directly related to the number of points in the four-dimensional spatiotemporal point cloud. Therefore, an overly dense point cloud results in a large number of Gaussian primitives in the initial four-dimensional Gaussian model, which in turn causes the initial four-dimensional Gaussian model to occupy a large amount of memory and storage space.

[0033] Furthermore, in order to provide a well-structured, spatiotemporally aligned initial set of primitives for subsequent compression processes, step S10 may include: Three-dimensional reconstruction is performed on each frame of the received multi-view image sequence to obtain the initial three-dimensional point cloud corresponding to each frame; the initial three-dimensional point cloud is stitched together in chronological order to obtain a four-dimensional spatiotemporal point cloud; four-dimensional Gaussian primitives are initialized based on the four-dimensional spatiotemporal point cloud to obtain an initial four-dimensional Gaussian model.

[0034] It should be noted that the initial 3D point cloud contains the spatial location (3D coordinates) information of the scene at a certain moment, and may also include appearance attributes such as color. The 4D spatiotemporal point cloud adds temporal information to the initial 3D point cloud.

[0035] Specifically, stitching together the initial 3D point clouds in chronological order can involve aligning and combining the reconstructed 3D point clouds from different times according to their timestamp order, forming a continuous data structure in four dimensions of spatiotemporal space (three-dimensional space and one-dimensional time), i.e., a four-dimensional spatiotemporal point cloud. The four-dimensional spatiotemporal point cloud reflects the trajectory of geometric elements changing over time in a dynamic scene. Optionally, temporal registration algorithms (such as feature point tracking or optical flow methods) can be used to more accurately align the point clouds from different times to compensate for inconsistencies caused by reconstruction errors or camera motion.

[0036] For example, the system first receives a sequence of multi-view images as input. Each frame is then independently reconstructed using a preprocessing layer to obtain initial 3D point clouds for each frame. These point clouds are then stitched together in chronological order to form a unified 4D spatiotemporal point cloud. Based on this point cloud, 4D Gaussian primitives are initialized. Each primitive contains parameters such as position, color, scaling, rotation, opacity, and dynamic and temporal attributes. During initialization, each spatiotemporal point in the point cloud is used as a seed to instantiate a four-dimensional Gaussian function. Each four-dimensional Gaussian primitive contains a series of parameters describing its spatial morphology (e.g., position, rotation, scaling), appearance (e.g., color, opacity), and dynamic attributes that evolve over time (e.g., velocity, acceleration, or deformation parameters). During initialization, the position parameter can be directly taken from the coordinates of the spatiotemporal point, while other parameters can be set to default values ​​or assigned using heuristic methods (e.g., setting an initial scale based on the local density of the point cloud).

[0037] It should be noted that if the multi-view image sequence is not uniformly sampled in time or has missing frames, virtual point clouds at intermediate moments can be generated through interpolation during the point cloud stitching stage to ensure the continuity of the four-dimensional spatiotemporal point cloud.

[0038] Step S20: Calculate the spatiotemporal importance score of each Gaussian element in the initial four-dimensional Gaussian model, and prune the Gaussian elements based on the spatiotemporal importance score to obtain the four-dimensional Gaussian model.

[0039] It should be noted that the spatiotemporal importance score is used to quantify the contribution of each four-dimensional Gaussian primitive to the final rendered image of the entire dynamic scene in both spatial and temporal dimensions. The higher the score, the more important the primitive. Pruning Gaussian primitives can be done based on the spatiotemporal importance score, removing primitives with contributions below a predetermined threshold or those ranked low, thereby directly reducing the number of Gaussian primitives in the model.

[0040] Understandably, as an optional implementation, calculating the fused spatiotemporal importance score includes two parallel sub-steps: First, spatial importance score is calculated by simulating the physical process of light accumulation. A primitive with low opacity or severely occluded by primitives in front contributes less to the current frame and thus has a lower spatial importance score. Second, temporal importance score is calculated by analyzing the changing patterns of primitive attributes over time. A primitive with smooth or almost static movement throughout the time series has a lower temporal importance score.

[0041] For example, when pruning based on the spatiotemporal importance score, all primitives can be sorted in descending order of score, and a certain percentage (e.g., 30%) of the primitives with the lowest scores can be removed directly. Alternatively, an absolute score threshold can be set to remove all primitives with scores below that threshold.

[0042] In another alternative implementation, pruning can be integrated with the iterative training process of the model, rather than being performed all at once. For example, after each round or every few rounds of training iterations, the importance scores of the primitives can be recalculated and pruning performed to achieve a dynamic, progressive model simplification that better preserves the model's expressive power during the compression process.

[0043] It should be understood that the spatiotemporal importance scores can also serve as a guide for subsequent optimization steps (such as quantization). For example, in subsequent steps, differentiated compression strategies can be adopted for primitives with different importance.

[0044] Step S30: Train the four-dimensional Gaussian model to obtain the trained four-dimensional Gaussian model.

[0045] It should be understood that while pruning reduces model size, it may also lose some useful information or introduce structural discontinuities. Targeted training can encourage the remaining Gaussian units to readjust their properties and spatial distribution, representing the entire dynamic scene more efficiently in a collaborative manner. This allows for model size compression while ensuring or even improving the final rendering fidelity, providing a high-quality, compact model foundation for subsequent quantization and compression steps.

[0046] Understandably, during the model training phase, an iterative optimization strategy based on random spatiotemporal sampling can be employed. In each iteration, a time index and multiple observation perspectives at that moment are randomly selected for batch training. The specific process includes mapping 4D Gaussian primitives to a two-dimensional image plane through Gaussian projection and rasterization, then rendering the image based on the projection results, calculating the composite loss function, and backpropagating the gradient to update the model parameters.

[0047] Alternatively, in another implementation, the training process can be multi-stage. For example, a high learning rate and large batch size can be used initially for rapid convergence, while a lower learning rate and finer sampling strategy can be used later for fine-tuning. Learning rate scheduling or early stopping strategies can also be introduced to prevent overfitting and ensure that the model achieves optimal performance on the validation set.

[0048] Step S40: Based on the feature parameters of each Gaussian element in the trained four-dimensional Gaussian model and the fusion spatiotemporal importance score, compress each Gaussian element to obtain a compressed four-dimensional Gaussian model.

[0049] It should be noted that feature parameters are the attribute data necessary to describe each Gaussian primitive, typically including spatial attributes (such as 3D position, rotation, and scaling) and appearance attributes (such as opacity and color features, where color features are often represented by spherical harmonic coefficients). Compression can reduce storage space by decreasing the precision of the parameter numerical representation (i.e., quantization).

[0050] Specifically, as an optional implementation, the compression of each Gaussian primitive can be achieved by classifying Gaussian primitives according to a fusion spatiotemporal importance score. For example, an importance score threshold can be set, and Gaussian primitives with scores above the threshold can be classified into a high-importance set, while those with scores below the threshold can be classified into a low-importance set. Next, a differentiated quantization strategy is implemented: for geometric parameters (such as position and rotation) common to primitives in the high-importance set and highly sensitive to numerical precision, they are uniformly converted into a low-bit-width floating-point format (such as FP16) for storage. For parameters of primitives in the low-importance set, a shared codebook is used for storage.

[0051] In another alternative implementation, the grading can be divided into more levels based on importance scores, with each level corresponding to a different quantization precision or codebook size, to achieve finer compression control.

[0052] In one example, reference Figure 2 , Figure 2This is a schematic diagram of the overall architecture of the model compression method in this application. Taking a multi-view image sequence as input, the initialization phase begins. First, 3D point cloud reconstruction is performed on each frame of the sequence using the COLMAP tool to obtain the three-dimensional spatial structure data corresponding to each frame. Then, all single-frame 3D point clouds are stitched together in chronological order to form a 4D spatiotemporal point cloud containing spatiotemporal dimensional information. Next, attribute parameters are configured for the 4D Gaussian primitives based on this 4D spatiotemporal point cloud to complete the 4D Gaussian initialization, thus obtaining the initial 4D Gaussian model. The model then enters the convergence judgment stage: if the model has met the convergence condition, the current model is directly used as the pruned 4D Gaussian model, and post-processing (vector quantization) is performed on it to complete the quantization and compression of feature parameters, and the final compressed model is output; if the model has not converged, it enters the iterative optimization process: first, random spatiotemporal sampling is performed, selecting random time indices and samples corresponding to the observation perspectives from multi-view image sequences; then, the adaptive control module is entered, sequentially completing spatiotemporal importance assessment (quantizing the spatiotemporal contribution of each 4D Gaussian unit) and adaptive pruning (removing redundant Gaussian units); the pruned model is processed by projection, mapping the 4D Gaussian units to a two-dimensional image plane, and then combined with the input ground truth image to perform rendering and loss calculation, obtaining the difference loss between the model output and the real image; finally, the parameters are updated based on the loss result, adjusting the attribute parameters of the 4D Gaussian units, and the updated model returns to the convergence judgment stage, repeating the above iterative process until the model converges, and then enters the post-processing and compression stage, finally generating and outputting the final compressed model.

[0053] In this embodiment, by introducing an adaptive pruning mechanism based on spatiotemporal importance, Gaussian primitives with low contribution in the spatial and temporal dimensions are identified and removed. The feature parameters of the remaining Gaussian primitives are compressed after training, thereby compressing the model storage volume and reducing memory access overhead while maintaining negligible visual quality loss, effectively improving the model's deployment capability on resource-constrained devices.

[0054] Reference Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the model compression method of this application. Based on the first embodiment described above, a second embodiment of the model compression method of this application is proposed. In the second embodiment, step S20 includes: Step S201: Calculate the spatial importance score of each Gaussian element based on the opacity of each Gaussian element in the initial four-dimensional Gaussian model.

[0055] It's important to note that opacity is a key attribute parameter of Gaussian primitives, representing the primitive's ability to block light during rendering. Its value typically ranges from 0 to 1. An opacity of 1 indicates complete opacity, while 0 indicates complete transparency. Transmittance is complementary to opacity and refers to the proportion of light remaining after passing through the primitive. For a single primitive, transmittance is typically calculated as 1 minus the opacity. Spatial importance score measures the direct impact of a primitive on the color formation of pixels in the image plane at a specific viewpoint.

[0056] Specifically, as an optional implementation, the spatial importance score can be calculated by first obtaining the current opacity value of each Hokkaido primitive. Next, a ray of light originating from the camera and passing through the pixels is simulated, and the remaining transmittance after being blocked by all other Hokkaido primitives along the path before reaching the current Hokkaido primitive is calculated; this is the cumulative transmittance. This process takes into account the occlusion relationships between primitives in three-dimensional space. Then, the opacity of the current primitive is multiplied by the cumulative transmittance reaching its position to obtain the primitive's contribution weight to the final pixel color. Finally, by summing the contribution weights of the primitive across all relevant pixels (e.g., the pixel area covered by its projection) and normalizing the result, the spatial importance score of the primitive is obtained. For example, a primitive with high opacity and little forward occlusion has a larger contribution weight and a higher spatial importance score; conversely, a primitive with low opacity or almost complete occlusion has a lower score.

[0057] In another alternative implementation, to simplify computation, physical occlusion can be omitted from strict simulation. Instead, factors such as the projected area of ​​the Gaussian unit on the image plane and the reciprocal of the distance from the camera can be used, combined with opacity, to approximate its spatial importance. Optionally, when calculating the spatial importance score, it can be based on one or more representative training views, or the average of all training views can be used to obtain more robust evaluation results.

[0058] Furthermore, in order to simulate the occlusion and accumulation effects of light during realistic rendering, and to more accurately reflect the visual contribution of each Gaussian unit in a single frame image, making the spatial importance assessment more consistent with the rendering physics process, step S201 may include: Obtain the opacity of each Gaussian element and calculate the transmittance of each Gaussian element based on the opacity; for each Gaussian element, calculate the spatial importance score of the Gaussian element using the opacity and the transmittance of all Gaussian elements in front of the Gaussian element.

[0059] In one example, in a batch of size Resolution is In the training images, for the first The contribution of each Gaussian to the rendered pixels (spatial importance score) Due to its opacity The transmittance, together with that of all Gaussians in front of it, determines the following:

[0060] in, Indicates the first The transmittance of the j-th Gaussian element preceding the j-th Gaussian element.

[0061] Step S202: Calculate the time importance score of each Gaussian element based on its normalized volume and cumulative dynamic change intensity.

[0062] It's important to note that, unlike spatial importance which focuses on contribution within a single frame, temporal importance focuses on the changing patterns and influence of primitive attributes over time. Normalized volume refers to the effective span or presence of a Gaussian primitive across its temporal active distribution, typically normalized to the [0, 1] interval. It reflects whether the primitive is a consistently present key element throughout the entire time series (e.g., from time t=0 to t=T) or only appears briefly. A primitive active for most of the time has a larger normalized volume. Cumulative dynamic change intensity refers to the integral or sum of the drastic changes in a Gaussian primitive's attributes (such as position, color, shape, etc.) over time. The more frequent and larger the changes, the higher the value, reflecting the amount of dynamic information carried by the primitive. A primitive that is stationary or in uniform linear motion has a very low dynamic change intensity; while a primitive undergoing acceleration, rotation, or deformation has a very high intensity. The time importance score is used to quantify the importance of primitives in the time dimension. A higher time importance score indicates that the primitives participate more extensively and significantly in the dynamic expression of the scene.

[0063] Furthermore, in order to effectively identify and distinguish between continuously active and rapidly changing primitives and static or slowly changing primitives, and thus prioritize the retention of time-sensitive primitives that are crucial to the expression of dynamic scenes during pruning, step S202 may include: The activity distribution function of each Gaussian element in time is determined by a Gaussian function, the parameters of which include the time center and the time scale. The dynamic change intensity of each Gaussian element is determined by the activity distribution function. The cumulative dynamic change intensity of each Gaussian element is calculated based on the dynamic change intensity. The normalized volume of each Gaussian element is obtained, and the normalized volume is fused with the cumulative dynamic change intensity to generate the time importance score of each Gaussian element.

[0064] It should be understood that a distribution function describing the "activity" or "influence" of each Gaussian primitive can be fitted over time. For example, a Gaussian function with a certain width centered at a specific time can be used to model this. The mean of this Gaussian function (i.e., the time center) represents the primitive's most active moment, and the standard deviation (i.e., the time scale) represents the range of its active duration.

[0065] In one example, for calculating the time importance score, the system first uses a time-centered approach. and time scale A Gaussian function with parameters is used to model the temporal activity distribution of the primitives, as follows:

[0066] This distribution at any time The intensity of dynamic change can be measured by its curvature, i.e., the function. The second derivative To represent, as follows:

[0067] Among them, the Gaussian time importance score Its normalized volume It is determined by the intensity of its cumulative dynamic change over the entire time series, as follows:

[0068] Understandably, a primitive that flashes briefly and changes little (small normalized volume, low cumulative intensity), or a background primitive that is always present but completely static (cumulative intensity of 0), will have a very low temporal importance score. These primitives contribute little to representing the main dynamics of the scene and can be prioritized for removal during subsequent pruning.

[0069] Step S203: Calculate the fusion spatiotemporal importance score of each Gaussian unit based on the spatial importance score and the temporal importance score.

[0070] Specifically, by integrating spatial importance scores With time importance score The spatiotemporal importance score of each Gaussian unit can be obtained as follows:

[0071] It should be understood that adaptive pruning is based on the fusion of spatiotemporal importance scores. First, all Gaussian primitives are sorted from highest to lowest score to form a pruning candidate queue. Based on the sorting results, low-importance primitives are directly removed, while the complete parameters of high-importance primitives are retained.

[0072] In this embodiment, importance scores are calculated separately for both spatial and temporal dimensions, and then fused to obtain a comprehensive spatiotemporal importance score. This helps to retain key primitives that significantly impact visual quality during pruning, while eliminating redundant primitives in space or time. This achieves a significant reduction in the number of model parameters while maintaining rendering quality.

[0073] Reference Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the model compression method of this application. Based on the above embodiments, a third embodiment of the model compression method of this application is proposed. In the second embodiment, step S30 includes: Step S301: Randomly select time indices and corresponding observation perspectives from the multi-view image sequence to generate training data for each iteration.

[0074] It's important to note that a time index refers to a unique identifier for a specific frame (moment) in a multi-view image sequence. The observation viewpoint is the camera pose corresponding to images acquired from different spatial locations at the same time index (same moment). Randomly selecting multiple observation views means that the model needs to understand and reconstruct the scene from multiple different spatial angles at the same moment. Training data consists of the input data units used in each iteration to perform forward rendering, calculate loss, and backpropagate. A training batch of training data can be formed by a randomly selected time index and a set of randomly selected observation views (and their corresponding real images) at that moment.

[0075] Understandably, by forcing the model to reconstruct the scene at random time points using random combinations of viewpoints, this method effectively enhances the model's ability to express continuity and consistency throughout the entire four-dimensional spatiotemporal domain. Compared to training with all data or data in a fixed order, this method makes fuller use of limited data, accelerates model convergence, and significantly improves the model's robustness when facing new viewpoints or temporal interpolation tasks.

[0076] Step S302: For each iteration, based on the training data, perform image rendering using the four-dimensional Gaussian model to obtain the rendered image.

[0077] It should be noted that by using the current model to be optimized (i.e., the pruned four-dimensional Gaussian model), the real imaging process can be simulated based on the training data randomly sampled in this round (specific time points and multiple observation perspectives), and the corresponding predicted image can be synthesized. The rendered image is the predicted output of the model parameters for the scene in the current state.

[0078] Understandably, by comparing the generated rendered image with the corresponding ground truth image in the training data, the resulting difference (i.e., loss) will accurately point out the shortcomings of the model's current prediction, thereby guiding subsequent gradient backpropagation and parameter updates, driving the model to continuously optimize in the direction of more accurately reconstructing dynamic scenes.

[0079] Step S303: Calculate the composite loss between the rendered image and the corresponding real image, wherein the composite loss includes image reconstruction loss and entropy regularization loss.

[0080] It should be noted that the image reconstruction loss is a loss function used to directly measure the difference between the rendered image and the real image. The entropy regularization loss acts on the regularization term of the model's internal parameters (specifically, the opacity of Gaussian units). Utilizing the concept of information entropy, it makes the distribution of the opacity parameter more deterministic, tending towards 0 (completely transparent) or 1 (completely opaque), thereby reducing fuzzy states and semi-transparent redundancy in the model. The composite loss is the weighted sum of the image reconstruction loss and the entropy regularization loss, and it is the final objective function that the gradient descent algorithm aims to minimize during training.

[0081] In one example, the composite loss function consists of two parts: image reconstruction loss and entropy regularization loss, as follows:

[0082] Among them, image reconstruction loss for (i.e., absolute error per pixel channel) loss and structural similarity loss A weighted combination of these factors is used to evaluate the difference between the rendered image and the ground truth image at different spatial scales, ensuring the structural integrity and detail realism of the scene. Entropy regularization loss is employed. The opacity parameter acting on the Gaussian element is defined as:

[0083] The aforementioned loss function, by driving the probability of Gaussian primitives to approach a binary distribution, can both control the effective number of Gaussians and suppress rendering artifacts caused by improper blending of semi-transparent primitives, thereby improving visual quality. Hyperparameters and The weights of different losses in the overall optimization objective are controlled separately.

[0084] Step S304: Based on the composite loss, update the parameters of the four-dimensional Gaussian model through backpropagation to obtain the trained four-dimensional Gaussian model.

[0085] Understandably, the backpropagation algorithm calculates the gradient of the loss with respect to each model parameter, and the optimization algorithm updates these parameters along the gradient descent direction, enabling the model to output a more realistic and structurally superior rendering result in the next iteration. Through repeated iterations of this process, the model parameters are continuously optimized from the pruned initial state, eventually converging to a high-performance state, namely the four-dimensional Gaussian model.

[0086] In this embodiment, by designing a multi-scale perceptual composite loss function, spatial details are preserved during the optimization process, while Gaussian elements are driven to be reasonably distributed on the object surface, thereby effectively suppressing rendering artifacts and ensuring high-quality visual output.

[0087] Reference Figure 5 , Figure 5 This is a flowchart illustrating the fourth embodiment of the model compression method of this application. Based on the above embodiments, a fourth embodiment of the model compression method of this application is proposed.

[0088] In the fourth embodiment, step S40 includes: Step S401: Based on the fusion spatiotemporal importance score, the color features in the feature parameters of each Gaussian element are classified to generate a first color feature and a second color feature. The fusion spatiotemporal importance score of the Gaussian element corresponding to the first color feature is less than that of the second color feature.

[0089] It should be noted that after the model training converges, the post-processing stage begins, employing a hierarchical quantization strategy to compress the parameters. Color features refer to the set of parameters used to describe the appearance color of Gaussian primitives, typically represented by a set of spherical harmonic function coefficients. Before compression, the color features among the feature parameters of each Gaussian primitive need to be classified. For example, a threshold τ is set, primitives with scores below τ are classified into the first class (corresponding to the first color feature), and primitives with scores above τ are classified into the second class (corresponding to the second color feature).

[0090] Specifically, as an optional implementation, when performing the classification, all Gaussian primitives are first sorted in descending order according to their spatiotemporal importance scores. Then, the dividing point is determined according to the target compression ratio or empirical proportion (e.g., classifying the bottom 30% of primitives as the first class), thereby generating the first color feature (color parameters of the low importance set) and the second color feature (color parameters of the high importance set).

[0091] In another alternative implementation, the threshold τ may not be fixed, but may be dynamically calculated based on the distribution of importance scores of all primitives, for example, using the median or mean of the scores as the boundary.

[0092] Understandably, the grading step determines whether the subsequent compression process can effectively protect the key color information that has the greatest impact on visual quality while reducing storage overhead, because the second color feature will be preserved with higher precision, while the first color feature will be subjected to stronger compression.

[0093] Step S402: Cluster the first color features and generate a shared codebook based on the clustering results.

[0094] Step S403: Replace the parameters of the first color feature with the codebook index of the shared codebook to obtain the first compression result.

[0095] It should be noted that the shared codebook is a lookup table consisting of a finite number (e.g., 8192) of typical color vectors. Each entry in the table is called a codeword, representing the central feature of a cluster. The codebook index is an integer number used to uniquely identify a specific codeword in the shared codebook. The first compression result refers to the transformation of the original color parameters, originally stored as floating-point arrays, into a sequence of integer indices that occupy a much smaller storage space after the substitution.

[0096] Specifically, as an optional implementation, firstly, the color parameters of all Gaussian primitives belonging to the first color feature (assuming each parameter is a D-dimensional floating-point vector) are collected to form a matrix of size M×D (M being the number of low-importance primitives). Then, a K-means clustering algorithm is run on this matrix, with the preset number of clusters K (e.g., K=8192) determining the size of the shared codebook. After the algorithm converges iteratively, K clusters are obtained, and the centroid vector of each cluster is a codeword. All codewords are arranged sequentially to generate the shared codebook. Next, for each color vector in the original matrix, its distance to all K codewords (e.g., Euclidean distance) is calculated. The closest codeword is found, and its index in the codebook (e.g., an integer between 0 and 255) is used as the new representation of the color vector, thus completing the parameter-to-codebook index replacement. Finally, the color information of all low-importance primitives is no longer stored as the original D floating-point numbers, but rather as a single, easily accessible index. The integer index of the bit is used to generate the first compression result.

[0097] In another alternative implementation, the clustering process can employ more efficient algorithms such as Mini-Batch K-means to handle large-scale data, or use methods such as hierarchical clustering and product quantization to construct a multi-level codebook, achieving higher compression rates or better reconstruction quality. The codebook index itself can also be further compressed using entropy encoding.

[0098] Understandably, through the above steps, a large amount of redundant and unimportant color information is highly compressed. Due to the existence of the shared codebook, decoding only requires a simple table lookup operation to approximately recover the color, greatly reducing the model's storage footprint while maintaining acceptable visual quality.

[0099] Step S404: Compress and store the second color feature and other features in the feature parameters using a preset precision format to obtain a second compression result.

[0100] This can be understood as the other features in the feature parameters referring to the attribute parameters necessary for Gaussian elements besides color features, mainly including geometric features (such as position, rotation, and scaling) and appearance features (such as opacity). The second compression result is a set of parameters with significantly reduced data size obtained after reducing the precision of these important parameters.

[0101] Specifically, as an optional implementation, first, a uniform preset precision format is determined, such as FP16 (16-bit floating-point). Then, for all Gaussian primitives classified as second color features, each floating-point value in their original color feature vector (such as spherical harmonic coefficients), which may be in FP32 format, is directly converted to FP16 format. Simultaneously, for the geometric features (position, rotation, scaling) and appearance features (such as opacity) of all Gaussian primitives in the model (including primitives belonging to the first and second color features), their original full-precision parameters are also converted to FP16 format. This conversion process typically involves simple type conversion or may require a small amount of saturation processing (to prevent overflow). After conversion, these parameters are stored in a new, lower-precision format, collectively forming the second compression result.

[0102] In another alternative implementation, different preset precision formats can be used for different types of features. For example, FP16 can be used for geometric parameters that are extremely sensitive to numerical precision, such as position and rotation, while more aggressive INT8 fixed-point quantization can be used for scaling factors or opacity. Similarly, for the second color feature, it can be further subdivided into multiple sub-levels based on the specific distribution of its importance scores, and different precision formats such as FP16 and BFLOAT16 can be applied to each sub-level to achieve finer compression control. Furthermore, the entire precision conversion process can be integrated with the model training process, i.e., quantization-aware training can be used to simulate the errors introduced by precision conversion during optimization, thereby making the final model parameters more robust to quantization.

[0103] Understandably, compared to codebook quantization, the advantages of the above methods are that they maintain the independence and differentiability of parameters, eliminating the need for table lookups during decoding and allowing direct use of low-precision values, resulting in high computational efficiency. Although some numerical errors are introduced, since these parameters are inherently important or sensitive to numerical range, using a floating-point format (such as FP16) that retains a certain dynamic range and precision can achieve a good balance between compression ratio and reconstruction quality.

[0104] Step S405: Determine the compressed four-dimensional Gaussian model based on the first compression result and the second compression result.

[0105] Understandably, the two parts of compressed data that have undergone differential processing are systematically reorganized and encapsulated, namely the first compressed result (codebook index) representing low-importance color features and the second compressed result (reduced precision values) representing high-importance color features and other key parameters, thereby generating a compressed four-dimensional Gaussian model that is structurally complete, data-compact, and can be loaded and interpreted by a standard renderer.

[0106] In this embodiment, the codebook-based vector quantization technique converts high-precision floating-point parameters into a compact discrete representation, reducing the model's storage overhead and memory bandwidth usage. This enables the compressed model to run efficiently in resource-constrained environments such as mobile terminals, breaking through the dependence of the original technology on high-performance computing devices.

[0107] For example, to help understand the implementation process of the model compression method obtained by combining the above embodiments, please refer to... Figure 6 , Figure 6This diagram illustrates the core compression process of a model compression method. Specifically, starting with an uncompressed 4D Gaussian model, the system first enters the spatiotemporal importance assessment module, where spatial and temporal contribution analyses are performed sequentially. The two contribution results are then fused to rank the importance, yielding the spatiotemporal importance level of each 4D Gaussian unit. This importance result is then passed to the importance judgment module, which distinguishes between low and high importance Gaussian units. Low-importance units are directly removed, while high-importance units are fully preserved, resulting in a pruned model. Next, based on the codebook, vector quantization is performed. Through parameter classification, codebook generation, cluster quantization, and parameter reconstruction, the corresponding parameters are quantized, resulting in the compressed 4D Gaussian model. Before vector quantization, loss function calculations, encompassing image reconstruction loss and entropy regularization loss, are performed simultaneously for model training. During vector quantization, the quantization strategy includes using FP16 format for basic storage of parameters such as model position, scaling, rotation, dynamic attributes, and time attributes. For color features, a differentiated quantization strategy is implemented based on the importance judgment results: color features with low importance and high sigma elements are processed by K-means vector clustering, while color features with high importance and high sigma elements are stored in FP16 format.

[0108] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the model compression method of this application. Any simple transformations based on this technical concept are all within the protection scope of this application.

[0109] This application also provides a model compression device; please refer to... Figure 7 The model compression device includes: The model initialization module 10 is used to obtain an initial four-dimensional Gaussian model, which is a model obtained after initialization based on a multi-view image sequence. The primitive pruning module 20 is used to calculate the fusion spatiotemporal importance score of each Gaussian primitive in the initial four-dimensional Gaussian model, and prune the Gaussian primitive based on the fusion spatiotemporal importance score to obtain the four-dimensional Gaussian model. The model training module 30 is used to train the four-dimensional Gaussian model to obtain the trained four-dimensional Gaussian model. The primitive compression module 40 is used to compress each Gaussian primitive based on the feature parameters of each Gaussian primitive in the trained four-dimensional Gaussian model and the fusion spatiotemporal importance score, to obtain a compressed four-dimensional Gaussian model.

[0110] The model compression apparatus provided in this application, employing the model compression method described in the above embodiments, can solve the technical problem of existing four-dimensional Gaussian splashing technology consuming large amounts of memory and computing resources from Gaussian elements. Compared with the prior art, the beneficial effects of the model compression apparatus provided in this application are the same as those of the model compression method provided in the above embodiments, and other technical features in the model compression apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0111] This application provides a model compression device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the model compression method in Embodiment 1 above.

[0112] The following is for reference. Figure 8 The diagram illustrates a structural schematic of a model compression device suitable for implementing embodiments of this application. The model compression device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The model compression device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0113] like Figure 8As shown, the model compression device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the model compression device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the model compression device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows model compression devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0114] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0115] The model compression device provided in this application, employing the model compression method described in the above embodiments, can solve the technical problem of existing four-dimensional Gaussian splashing technology consuming a large amount of memory and computing resources from Gaussian elements. Compared with the prior art, the beneficial effects of the model compression device provided in this application are the same as those of the model compression method provided in the above embodiments, and other technical features of this model compression device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0116] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0117] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0118] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the model compression method in the above embodiments.

[0119] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0120] The aforementioned computer-readable storage medium may be included in the model compression device; or it may exist independently and not be assembled into the model compression device.

[0121] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the model compression device, cause the model compression device to perform the model compression method described above.

[0122] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0124] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0125] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described model compression method. This solves the technical problem that existing four-dimensional Gaussian splashing techniques consume large amounts of memory and computing resources from Gaussian elements. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the model compression method provided in the above embodiments, and will not be repeated here.

[0126] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A model compression method, characterized in that, The model compression method includes: Obtain an initial four-dimensional Gaussian model, which is a model obtained after initialization based on a multi-view image sequence; Calculate the spatiotemporal importance score of each Gaussian element in the initial four-dimensional Gaussian model, and prune the Gaussian elements based on the spatiotemporal importance score to obtain the four-dimensional Gaussian model. The four-dimensional Gaussian model is trained to obtain the trained four-dimensional Gaussian model. Based on the feature parameters of each Gaussian element in the trained four-dimensional Gaussian model and the fusion spatiotemporal importance score, each Gaussian element is compressed to obtain a compressed four-dimensional Gaussian model.

2. The model compression method as described in claim 1, characterized in that, The step of calculating the fusion spatiotemporal importance score of each Gaussian element in the initial four-dimensional Gaussian model includes: Based on the opacity of each Gaussian element in the initial four-dimensional Gaussian model, calculate the spatial importance score of each Gaussian element; The time importance score of each Gaussian element is calculated based on its normalized volume and cumulative dynamic change intensity. Based on the spatial importance score and the temporal importance score, the fusion spatiotemporal importance score of each Gaussian element is calculated.

3. The model compression method as described in claim 2, characterized in that, The step of calculating the spatial importance score of each Gaussian element based on the opacity of each Gaussian element in the initial four-dimensional Gaussian model includes: Obtain the opacity of each Gaussian element, and calculate the transmittance of each Gaussian element based on the opacity; For each Gaussian cell, the spatial importance score of the Gaussian cell is calculated using the opacity and the transmittance of all Gaussian cells in front of the Gaussian cell.

4. The model compression method as described in claim 2, characterized in that, The step of calculating the time importance score of each Gaussian element based on its normalized volume and cumulative dynamic change intensity includes: The activity distribution function of each Gaussian element in time is determined by a Gaussian function, wherein the parameters of the Gaussian function include the time center and the time scale. The dynamic change intensity of each Gaussian element is determined by the active distribution function; The cumulative dynamic change intensity of each Gaussian element is calculated based on the dynamic change intensity. The normalized volume of each Gaussian element is obtained, and the normalized volume is fused with the cumulative dynamic change intensity to generate the time importance score of each Gaussian element.

5. The model compression method according to any one of claims 1 to 4, characterized in that, The steps for obtaining the initial four-dimensional Gaussian model include: Three-dimensional reconstruction is performed on each frame of the received multi-view image sequence to obtain the initial three-dimensional point cloud corresponding to each frame of the image; The initial three-dimensional point cloud is stitched together in chronological order to obtain a four-dimensional spatiotemporal point cloud; The initial four-dimensional Gaussian model is obtained by initializing the four-dimensional spatiotemporal point cloud with the four-dimensional Gaussian primitive.

6. The model compression method according to any one of claims 1 to 4, characterized in that, The step of training the four-dimensional Gaussian model to obtain the trained four-dimensional Gaussian model includes: Randomly select time indices and corresponding observation perspectives from the multi-view image sequence to generate training data for each iteration; For each iteration, based on the training data, the image is rendered using the four-dimensional Gaussian model to obtain the rendered image; Calculate the composite loss between the rendered image and the corresponding real image, the composite loss including image reconstruction loss and entropy regularization loss; Based on the composite loss, the parameters of the four-dimensional Gaussian model are updated through backpropagation to obtain the trained four-dimensional Gaussian model.

7. The model compression method according to any one of claims 1 to 4, characterized in that, The step of compressing each Gaussian element based on the feature parameters of each Gaussian element in the trained four-dimensional Gaussian model and the fused spatiotemporal importance score to obtain a compressed four-dimensional Gaussian model includes: Based on the fusion spatiotemporal importance score, the color features in the feature parameters of each Gaussian element are classified to generate a first color feature and a second color feature. The fusion spatiotemporal importance score of the Gaussian element corresponding to the first color feature is less than that of the second color feature. Cluster the first color features and generate a shared codebook based on the clustering results; The parameters of the first color feature are replaced with the codebook index of the shared codebook to obtain the first compression result; The second color feature and other features in the feature parameters are compressed and stored using a preset precision format to obtain a second compression result; The compressed four-dimensional Gaussian model is determined based on the first compression result and the second compression result.

8. A model compression device, characterized in that, The device includes: The model initialization module is used to obtain an initial four-dimensional Gaussian model, which is a model obtained after initialization based on a multi-view image sequence. The primitive pruning module is used to calculate the fusion spatiotemporal importance score of each Gaussian primitive in the initial four-dimensional Gaussian model, and prune the Gaussian primitives based on the fusion spatiotemporal importance score to obtain the four-dimensional Gaussian model. The model training module is used to train the four-dimensional Gaussian model to obtain the trained four-dimensional Gaussian model. The primitive compression module is used to compress each Gaussian primitive based on the feature parameters of each Gaussian primitive in the trained four-dimensional Gaussian model and the fusion spatiotemporal importance score, to obtain a compressed four-dimensional Gaussian model.

9. A model compression device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the model compression method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the model compression method as described in any one of claims 1 to 7.

Images