End-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression

Abstract

The invention provides an end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression, and the method comprises the steps: constructing initial dynamic Gaussian scene representation and executing preheating training, and obtaining a dynamic Gaussian field; performing dynamic and static decomposition on the dynamic Gaussian field by using an adaptive mechanism; implementing densification and pruning capable of differential budget control and importance guidance for the decomposed dynamic Gaussian field; executing three-stage joint optimization training with controllable budget based on the densified and pruned dynamic Gaussian field; and performing dual-mode mixed compression on the trained dynamic Gaussian field to complete end-to-end joint optimization of controllable dynamic Gaussian scene representation and dual-mode compression. According to the method, the constraint condition is directly fused into a training cycle, an efficient model naturally adaptive to target equipment is directly constructed, a framework for formalizing dynamic scene reconstruction into a budget constraint optimization problem is realized, and the problem that Gaussian points in a dynamic Gaussian reconstruction scene are uncontrolled to proliferate or depend on heuristic pruning in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of dynamic Gaussian scene reconstruction representation and compression technology. Specifically, it relates to an end-to-end joint optimization method for long-sequence controllable dynamic Gaussian scene representation and dual-mode compression, and also to a corresponding joint optimization system, computer terminal and computer-readable storage medium. Background Technology

[0002] Dynamic Gaussian splashing is gradually emerging as a powerful paradigm for real-time 4D scene reconstruction. However, existing dynamic variant methods suffer from uncontrolled Gaussian point proliferation or reliance on heuristic pruning. Model training focuses solely on improving the quality metrics of the rendered image, leading to Gaussian primitive stacking, uncontrollable utilization, representation redundancy, unpredictable model complexity, and excessive memory overhead. Therefore, current models are inefficient for customization on specific devices. However, pruning a large, single model after training often only yields suboptimal results while reducing the original reconstruction quality. Summary of the Invention

[0003] To address the aforementioned shortcomings in the prior art, this invention provides an end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression, along with a corresponding joint optimization system, computer terminal, and computer-readable storage medium.

[0004] According to a first aspect of the present invention, an end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression is provided, comprising: Construct an initial dynamic Gaussian scene representation and perform preheating training to obtain a dynamic Gaussian field; The dynamic Gaussian field is decomposed into static and dynamic components using an adaptive mechanism. Differentiable budget control and importance-guided densification and pruning are implemented for the decomposed dynamic Gaussian field; A three-stage joint optimization training with controllable execution budget is based on the dynamic Gaussian field after densification and pruning. The trained dynamic Gaussian field is subjected to dual-mode hybrid compression to achieve end-to-end joint optimization of controllable dynamic Gaussian scene representation and dual-mode compression.

[0005] Preferably, the step of constructing an initial dynamic Gaussian scene representation and performing preheating training includes: Using a multi-view dynamic sequence as input, a set of Gaussian points is initialized to represent the initial dynamic Gaussian scene, where each Gaussian point... All are preset to static, including 3D center. Rotation ,scale Opacity spherical harmonic coefficient Through rendering loss Perform containing Short-run optimization with quantitative constraints is used to obtain an initial Gaussian prior covering the scene geometry and coarse motion, providing a reliable foundation for subsequent dynamic and static decomposition and budget control; among which... The sparsity of the loss function is a tradeoff. To render the initial dynamic video sequence, A multi-view dynamic video sequence calibrated by the camera. The SSIM loss is for the two images.

[0006] Preferably, the step of using an adaptive mechanism to perform dynamic-static decomposition of the dynamic Gaussian field includes: Assign a translation attribute to each Gaussian point of the dynamic Gaussian field. To characterize its motion, a smooth histogram is constructed. And identify the two most significant peaks corresponding to the static Gaussian group and the dynamic Gaussian group; By locating the local minimum between the two peaks and calculating the percentiles of data dependence, an adaptive motion threshold is automatically determined. Based on this, the initial set of Gaussian points is unsupervisedly divided into mutually exclusive static subsets. With dynamic subsets ; Static subsets retain time-invariant properties but allow lightweight inter-frame translation, while dynamic subsets are configured with explicit temporal parameters, including position interpolation vectors. Rotation interpolation vector ,scale spherical harmonic coefficient and activation time window The opacity of the dynamic Gaussian decays smoothly outside the window and is activated only during the period when the object is visible, thus effectively modeling spatiotemporal sparsity.

[0007] Preferably, the densification and pruning of the decomposed dynamic Gaussian field by implementing differentiable budget control and importance guidance includes: Define a uniform importance score : In the formula, for, for, Let be a Gaussian point. To fuse parameters such as position gradient, opacity, depth, anisotropic scale, and motion amplitude, To integrate parameters such as photometric residual, pixel coverage, and multi-view consistency, For global normalization; By using hard sigmoid gating through temperature annealing, continuous activation variables are introduced for each Gaussian. ; Define differentiable effective quantity and construct budget loss ;in, The number of constraints; In importance score Under the guidance of Gaussians exceeding a set threshold are cloned or split. Gaussian elements with a value less than or equal to a set threshold are randomly removed. A closed-loop Gaussian enrichment and pruning process is then performed to ensure the total number of Gaussian elements converges to a predetermined target during training. .

[0008] Preferably, the three-stage joint optimization training based on the optimized and pruned dynamic Gaussian field with controllable execution budget includes: The budget-controlled three-stage joint optimization training includes: a warm-up stage (stage one), a budget control stage (stage two), and a stabilization and fine-tuning stage (stage three); wherein: During the warm-up phase, only rendering loss is optimized. Initialize Gaussian coverage; During the budget control phase, the total loss is jointly optimized. ,in, To apply smoothing regularization to the time-series interpolation vector of the dynamic Gaussian, To construct budget loss, To measure the weighting coefficients of the construction budget loss, The weighting coefficients for smoothing regularization; During the stable fine-tuning phase, when the differentiable effective quantity Convergence approaches the constraint quantity Then, a fixed gating is used to freeze Gaussian existence, with only geometric and appearance parameters fine-tuned; the rendering loss in the final loss function... Controlling distortion D Budget loss implicitly controls the bit rate through Gaussian quantity constraints. R , and regular terms As a sparse constraint on dynamic redundancy, they together constitute the rate-distortion optimization objective.

[0009] Preferably, the step of performing dual-mode hybrid compression on the trained dynamic Gaussian field to complete the end-to-end joint optimization of controllable dynamic Gaussian scene representation and dual-mode compression includes: For static Gaussians, use The criteria separate foreground and background; foreground attributes are quantified and spatially reordered using a KD tree to ensure local continuity, then residuals are predicted based on neighborhood and entropy encoding is performed; background is stored directly. For dynamic Gaussians, outliers are separated to optimize data centrality, the remaining quantized attributes are reshaped into 2D images by channel, and the image sequences with the same attribute are grouped and compressed using a standard video encoder. The final model is stored as a bitstream and then restored after decoding.

[0010] Preferably, the above method further includes: evaluating the optimization results; wherein: Evaluate the quality of dynamic Gaussian scene reconstruction using PSNR and SSIM; Compression efficiency is measured by the storage volume of the reconstructed scene-related storage files.

[0011] According to a second aspect of the present invention, an end-to-end joint optimization system for controllable dynamic Gaussian scene representation and dual-mode compression is provided, comprising: The Dynamic Gaussian Field Construction Module is used to construct an initial dynamic Gaussian scene representation and perform preheating training to obtain a dynamic Gaussian field. A dynamic-static decomposition module, which uses an adaptive mechanism to perform dynamic-static decomposition on the dynamic Gaussian field; The densification and pruning module implements differentiable budget control and importance-guided densification and pruning for the decomposed dynamic Gaussian field. The three-stage joint optimization module performs budget-controlled three-stage joint optimization training based on the dynamic Gaussian field after densification and pruning. The hybrid compression module is used to perform dual-mode hybrid compression on the trained dynamic Gaussian field, completing the end-to-end joint optimization of controllable dynamic Gaussian scene representation and dual-mode compression.

[0012] According to a third aspect of the present invention, a computer terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to perform the method described in any one of the above inventions, or to run the system described in the above inventions.

[0013] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described in any one of the above-described inventions, or to run the system described in the above-described inventions.

[0014] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art: The end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression provided by this invention significantly improves the controllability of dynamic Gaussian scene reconstruction by technicians and reduces the requirements of storage and hardware devices for dynamic Gaussian scene reconstruction by reconstructing dynamic Gaussian scenes with a controllable model scale and compressing them efficiently.

[0015] The end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression provided by this invention adaptively decomposes the dynamic scene into static and dynamic Gaussian fields for reconstruction under a fixed Gaussian point budget. The static Gaussian field is modeled using time-invariant parameters and a lightweight inter-frame transform; the dynamic Gaussian field explicitly models the temporal changes in position and rotation, and achieves visibility-aware activation through a time-activated window.

[0016] The end-to-end joint optimization method of controllable dynamic Gaussian scene representation and dual-mode compression provided by this invention also introduces a differentiable budget controller. By using a unified importance score that integrates geometric, motion and perception cues, it guides the adaptive densification and pruning of Gaussian points, so that the total number accurately converges to the target value and precisely controls the number of Gaussian points.

[0017] This invention provides an end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression. The training employs a three-stage strategy: initial warm-up, joint optimization rendering and budget loss, and fixed Gaussian existence fine-tuning. It also proposes dual-mode hybrid compression: after separating the foreground and background using static Gaussian, the foreground Gaussian is compressed using KD-tree spatial reordering and residual predictive coding; dynamic Gaussian first separates outliers, then reshapes the attributes into an image sequence, which is then efficiently compressed by a video encoder. Through this approach, high compression ratios and high-fidelity reconstruction are achieved while strictly controlling model complexity, significantly improving rate-distortion performance, making it suitable for high-quality free-viewpoint video applications on resource-constrained devices. Attached Figure Description

[0018] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the end-to-end joint optimization method of controllable dynamic Gaussian scene representation and dual-mode compression in a preferred embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the components of an end-to-end joint optimization system for controllable dynamic Gaussian scene representation and dual-mode compression in a preferred embodiment of the present invention. Detailed Implementation

[0020] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0021] Existing dynamic variant methods suffer from uncontrolled Gaussian point proliferation or reliance on heuristic pruning, leading to low customization efficiency for various current models on specific devices. Pruning a large, single model after training often only yields suboptimal results while reducing the original reconstruction quality.

[0022] To address the aforementioned issues, one embodiment of the present invention provides an end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression. This method directly integrates constraints into the training loop to achieve optimality under constraints, thereby constructing an efficient model that is naturally adapted to the target device from the outset. It realizes a framework that formalizes dynamic scene reconstruction as a budget-constrained optimization problem, thereby solving the problem of uncontrolled proliferation of Gaussian points or reliance on heuristic pruning in the dynamic Gaussian reconstruction scene in the prior art.

[0023] Specifically, such as Figure 1 As shown, the end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression provided in this embodiment may include: S1: Construct the initial dynamic Gaussian scene and perform warm-up training; S2: Use an adaptive mechanism to perform dynamic-static decomposition of the Gaussian field; S3: Implement differentiable budgetary control and importance-guided densification and pruning; S4: Perform three-stage joint optimization training with controllable execution budget; S5: Perform dual-mode hybrid compression on the trained dynamic Gaussian field.

[0024] In some preferred embodiments, the above-mentioned S1 may further include: Taking the multi-view dynamic sequence as input, a set of Gaussian points is initialized to represent the initial dynamic Gaussian scene. Each Gaussian point... All are preset to static, including 3D center. Rotation ,scale Opacity spherical harmonic coefficient Through rendering loss Perform containing Short-run optimization with quantitative constraints is used to obtain an initial Gaussian prior covering the scene geometry and coarse motion, providing a reliable foundation for subsequent dynamic and static decomposition and budget control. The sparsity of the loss function is a tradeoff. To render the initial dynamic video sequence, A multi-view dynamic video sequence calibrated by the camera. The SSIM loss is for the two images.

[0025] In some preferred embodiments, the above-mentioned S2 may further include: Assign translation property to each Gaussian point To characterize its motion, a smooth histogram is constructed. It identifies the two most significant peaks corresponding to the static and dynamic Gaussian groups. By locating the local minimum (valley) between the two peaks, it calculates the data-dependent percentile and automatically determines the adaptive motion threshold. Based on this, the Gaussian set (i.e., the set of Gaussian points obtained from initialization) is unsupervisedly divided into mutually exclusive static subsets. With dynamic subsets Static subsets retain time-invariant properties but allow lightweight inter-frame translation, while dynamic subsets are configured with explicit temporal parameters, including position interpolation vectors. Rotation interpolation vector ,scale spherical harmonic coefficient and activation time window Its opacity decays smoothly outside the window and is activated only when the object is visible, thus effectively modeling spatiotemporal sparsity.

[0026] In some preferred embodiments, the above-mentioned S3 may further include: Define a uniform importance score : in It integrates position gradient, opacity, depth, anisotropic scale, and motion amplitude. Integrating luminance residual, pixel coverage, and multi-view consistency For global normalization, introduce continuous activation variables for each Gaussian. This is achieved through hard sigmoid gating via temperature annealing. Define the number of differentiable effective quantities. and construct budget loss Guided by importance scores, the closed-loop density (high) is increased. Right now Gaussian cloning or splitting exceeding a set threshold (where the set threshold can be 0.5) and pruning (low Right now (Randomly remove Gaussians less than or equal to a set threshold) to ensure the total number of Gaussians converges precisely to the preset target during training. .

[0027] In some preferred embodiments, the above-mentioned S4 may further include: Phase 1 Warm-up: Optimizing Rendering Loss Only Initialize Gaussian coverage; Phase Two Budget Control: Jointly Optimize Total Loss ,in Apply a smoothing regularization to the time-series interpolation vector of the dynamic Gaussian. To construct budget loss, To measure the weighting coefficients of the construction budget loss, The weighting coefficients for smoothing regularization; Phase 3 Stable Fine-Tuning: When Convergence Approach Then, Gaussian existence is frozen (fixed gating), and only geometric and appearance parameters are fine-tuned; the final loss function contains... Controlling distortion D Budget loss implicitly controls bitrate R (Through Gaussian scalar constraints), and regularization terms It can be regarded as a sparse constraint on dynamic redundancy, which together constitutes the rate-distortion optimization objective, and the constraint conditions are directly integrated into the training loop.

[0028] In some preferred embodiments, the above-mentioned S5 may further include: For static Gaussian, The criteria separate foreground and background. After quantizing the foreground attributes, a KD-tree is used for spatial reordering to ensure local continuity. Then, residual prediction based on neighborhood is performed and entropy encoding is applied, while the background is directly stored. For dynamic Gaussian models, outliers are separated to optimize data centrality. The remaining quantized attributes are reshaped into 2D images by channel. After grouping image sequences with the same attribute, they are compressed using a standard video encoder. The final model is stored as a bitstream and can be recovered after decoding.

[0029] In some preferred embodiments, the above method may further include: Reconstruction quality was evaluated using PSNR and SSIM, while compression efficiency was measured by the storage volume of the relevant storage files in the reconstruction scenario.

[0030] On multiple dynamic scene datasets, the method provided by the above embodiments of the present invention achieves efficient model compression with a small Gaussian number error, and its rate-distortion performance is significantly better than that of existing methods.

[0031] Based on the same inventive concept, an embodiment of the present invention also provides an end-to-end joint optimization system for controllable dynamic Gaussian scene representation and dual-mode compression.

[0032] Specifically, such as Figure 2 As shown, the end-to-end joint optimization system for controllable dynamic Gaussian scene representation and dual-mode compression provided in this embodiment may include: Dynamic Gaussian Field Construction Module: Constructs the initial dynamic Gaussian scene and performs preheating training. Further, the specific implementation of this module is as follows: Given a multi-view dynamic video sequence that has been calibrated by the camera (Where v is the view index and t is the timestamp), the system first initializes an initial set of Gaussian points to represent the entire dynamic scene. In this initial construction phase, all Gaussian points are assumed to be static, meaning their geometric and appearance parameters remain unchanged over time. Each Gaussian point... Parametrically rendered as a standard 3D Gaussian splash: in, It is a 3D center. Rotation matrix As a scale, For opacity, Let be the spherical harmonic coefficients. At this stage, only short-term optimization is performed using the standard rendering loss function, i.e. Furthermore, unlike previous dynamic Gaussian methods that allowed for the free addition or deletion of Gaussian volumes after training, this method explicitly constrains the representational capacity during the optimization process. This allows us to obtain an initial Gaussian prior covering the scene geometry and coarse motion, providing a reliable foundation for subsequent dynamic and static decomposition and budget control.

[0033] Dynamic-Static Decomposition Module: This module utilizes an adaptive mechanism to perform dynamic-static decomposition on the Gaussian field. Furthermore, the specific implementation of this module is as follows: Assign translation property to each Gaussian point To characterize its motion, a smooth histogram is constructed. It identifies the two most significant peaks for the corresponding static and dynamic populations. By locating the local minimum (valley) between the two peaks, it calculates the data-dependent percentiles and automatically determines the adaptive motion threshold. Based on this, Gaussian sets are unsupervisedly divided into mutually exclusive static subsets. With dynamic subsets ,but: , , The static Gaussian preserves time-invariant properties but allows for lightweight inter-frame translation, while the dynamic Gaussian configures explicit temporal parameters, including position interpolation vectors. Rotation interpolation vector ,scale spherical harmonic coefficient and activation time window Its opacity decays smoothly outside the window and is activated only when the object is visible, thus effectively modeling spatiotemporal sparsity.

[0034] Densification and Pruning Module: Implements differentiable budget control and importance-guided densification and pruning. Furthermore, the specific implementation of this module is as follows: Define a uniform importance score : in, It integrates position gradient, opacity, depth, anisotropic scale, and motion amplitude. Integrating luminance residual, pixel coverage, and multi-view consistency For global normalization; introduce continuous activation variables for each Gaussian. This is achieved through hard sigmoid gating via temperature annealing; a differentiable effective quantity is defined. and construct budget loss Guided by importance scores, the closed-loop density is increased (higher density). Gaussian cloning or splitting) and pruning (low) (randomly remove Gaussians), so that the total number of Gaussians is... During training, it converges precisely to the preset target. The error is less than 2%.

[0035] The three-stage joint optimization module performs budget-controlled three-stage joint optimization training. Furthermore, the specific implementation of this module is as follows: Phase 1 Warm-up: Optimizing Rendering Loss Only Initialize the Gaussian coverage. The pre-trained Gaussian volume serves as a basic prior, providing reliable guidance for subsequent dynamic scene modeling, differentiable budget control, and dynamic-static decomposition. Phase Two Budget Control: Jointly Optimize Total Loss in Apply a smoothing regularization to the time-series interpolation vector of the dynamic Gaussian. and The strength of the budget penalty term and the regularization term are controlled separately; Phase 3 Stable Fine-Tuning: When Convergence Approach Then, freeze the existence of Gaussians (fixed) Gating), only in rendering loss With regularization loss Under the joint constraints, the appearance and geometric parameters of the Gaussian body are fine-tuned. The final loss function contains... Controlling distortion D Budget loss implicitly controls bitrate R (Through Gaussian scalar constraints), and regularization terms It can be regarded as a sparse constraint on dynamic redundancy, which together constitutes the rate-distortion optimization objective.

[0036] Hybrid Compression Module: Performs dual-mode hybrid compression on the trained dynamic Gaussian field. Further, the specific implementation of this module is as follows: For a static Gaussian, calculate the mean distance of all Gaussian volumes to the scene center. and standard deviation and with The criterion separates the foreground and background Gaussian volumes. After quantizing the foreground attributes, spatial reordering is performed using a KD-tree to ensure local continuity. Then, residual prediction is based on neighborhood and entropy encoding is applied, while the background is stored directly. For dynamic Gaussians, outliers are separated to optimize data centrality. The remaining quantized attributes are reshaped into 2D images by channel. After grouping image sequences with the same attribute, they are compressed using a standard video encoder. The final model is stored as a bitstream.

[0037] Evaluation module: This module evaluates the optimization results. Further, its specific implementation is as follows: Reconstruction quality was evaluated using PSNR and SSIM, while compression efficiency was measured by the storage volume of the relevant storage files in the reconstruction scenario.

[0038] It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding components in the system. Those skilled in the art can refer to the technical solution of the system to implement the steps of the method, and can also refer to the technical solution of the method to implement the composition of the system. That is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, which will not be elaborated here.

[0039] An embodiment of the present invention also provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to execute the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0040] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0041] A processor is used to execute computer programs stored in memory to implement the various steps of the methods or various modules of the systems involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method and system embodiments.

[0042] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0043] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when executed by a processor, the computer program can be used to perform the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0044] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0045] The following detailed explanation, using a specific application example, further illustrates the application effect of the technical solution provided by the above embodiments of the present invention in the task of dynamic 3D scene reconstruction and lightweight deployment in the field of computer vision.

[0046] This specific application example uses the reconstruction of dynamic 3D scenes from sports events as an example. In this task, the end-to-end joint optimization method of controllable dynamic Gaussian scene representation and dual-mode compression is deeply adapted to the reconstruction characteristics of 3D Gaussian sputtering technology in high-speed motion scenes. Addressing the pain points of large amplitude of athlete limb movements, the tendency for Gaussian kernels to proliferate disorderly, the difficulty in constraining temporal motion consistency, and the high costs of model storage and computation in sports events, a complete technical chain from modeling to compression is constructed. The core objective is to reconstruct dynamic 3D scenes with high accuracy based on multi-view sports video sequences, while simultaneously achieving lightweight modeling to meet the requirements of real-time rendering and lightweight deployment.

[0047] This example uses a football match scene as the object, and inputs a 600-frame dynamic video sequence (including running, passing, and other actions) captured simultaneously from 20 different perspectives. The camera pose is obtained using COLMAP, and the single-frame resolution is 1920×1080. The specific application process is as follows: 1. Controllable initialization of dynamic Gaussian scenes To address the dynamic characteristics of sports scenarios, a controllable dynamic Gaussian representation is constructed using a data-driven initialization strategy: All Gaussian kernels are initialized to static type, and the geometric and motion parameters are optimized only through rendering loss to quickly cover the core scene structure such as the field and athletes, providing a reliable prior for subsequent dynamic modeling. By introducing a unified importance score as a pre-guidance method, and integrating geometric position, range of motion and perceptual contribution cues, dynamic key areas such as athletes' limbs and soccer balls are marked in advance, providing an initial basis for the addition and deletion of Gaussian kernels; By learning motion features during the initialization phase, each Gaussian kernel is given initial motion attributes, avoiding non-physical drift in subsequent dynamic modeling and ensuring the temporal continuity of the scene.

[0048] Compared to traditional unconstrained initialization, this step can quickly establish the basic structure of the scenario, which improves the convergence speed of subsequent dynamic optimization by 30%, while reserving precise adjustment space for budget control.

[0049] 2. Adaptive Dynamic-Static Allocation and Budget Constraint Optimization By leveraging the dynamic-static partitioning characteristics of sports scenarios, optimal resource allocation is achieved through adaptive allocation and differential budget control. Automatic region segmentation based on Gaussian kernel motion amplitude: Constructing a motion attribute histogram, determining dynamic-static boundaries through bimodal detection and valley thresholding, and assigning sites, billboards, etc., to static areas. Athletes, footballs, and other sports were assigned to dynamic fields. No manual labeling is required; The dynamic field Gaussian kernel is equipped with dedicated time parameters (position / rotation interpolation vector, activation window), which only takes effect during the motion period, reducing redundant calculations; the static field keeps the properties unchanged and adapts to the inter-frame viewpoint changes only through lightweight transformations. The differential budget controller constrains the number of Gaussian kernels through a quadratic loss function and guides additions and deletions by combining importance scores: high-scoring Gaussian kernels (such as athlete limb regions) are preferentially split to supplement details, while low-scoring kernels are randomly pruned to ensure that the total number accurately matches the target budget (error <2%).

[0050] Experiments show that when the target Gaussian number is 300k, the dynamic region is allocated 251.4k Gaussians and the static region is allocated 48.6k Gaussians. Compared with uniform allocation, the dynamic region reconstruction accuracy is improved by 15% and the model volume is reduced by 40%.

[0051] 3. Three-stage training and dual-mode compression optimization The model's performance and compactness are further improved through three-stage training and dual-mode compression: The three-stage training is progressive: the warm-up stage (500 iterations) completes basic modeling, the budget execution stage (29,500 iterations) activates budget control and dynamic-static allocation, and the fine-tuning stage (10,000 iterations) fixes the number of kernels, optimizes appearance and geometric parameters, and suppresses motion flickering. Dual-mode compression specifically eliminates redundancy: the static Gaussian kernel is compressed through spatial reordering and predictive coding using a KD tree, while the dynamic Gaussian kernel separates 5% outliers and reshapes the image into a 2D image sequence, which is then efficiently compressed using a video encoder.

[0052] Application effect verification: This method achieves a reconstruction accuracy of 32.12dBPSNR in football match scenarios, with a model size of only 32.8MB, an encoding time of 16 seconds, a decoding time of 0.5 seconds, and a mobile rendering frame rate of 145FPS, which fully meets the requirements of real-time reconstruction and lightweight deployment of sports events.

[0053] The end-to-end joint optimization method and system for controllable dynamic Gaussian scene representation and dual-mode compression provided in the above embodiments of the present invention formalizes dynamic scene reconstruction into a framework of budget-constrained optimization problem for the first time. By reconstructing and representing dynamic Gaussian scenes at a controllable model scale and compressing them efficiently, it can significantly improve the controllability of dynamic Gaussian scene reconstruction by technicians and reduce the requirements of dynamic Gaussian scene reconstruction on storage and hardware devices.

[0054] The end-to-end joint optimization method and system for controllable dynamic Gaussian scene representation and dual-mode compression provided in the above embodiments of the present invention adaptively decomposes dynamic scenes into static and dynamic Gaussian fields for reconstruction under a fixed Gaussian point budget. The static Gaussian field uses time-invariant parameters and lightweight inter-frame transform for modeling; the dynamic Gaussian field explicitly models the temporal changes in position and rotation, and achieves visibility perception activation through a time activation window. To accurately control the number of Gaussian points, the present invention also introduces a differentiable budget controller, using a unified importance score that integrates geometric, motion, and perceptual cues to guide the adaptive densification and pruning of the Gaussian field, ensuring the total number accurately converges to the target value. Training employs a three-stage strategy: initial warm-up, joint optimization rendering and budget loss, and solidified Gaussian existence fine-tuning. Furthermore, a dual-mode hybrid compression is proposed: the static Gaussian field separates the foreground and background, then compresses the foreground Gaussian field using KD-tree spatial reordering and residual predictive coding; the dynamic Gaussian field first separates outliers, then reshapes the attributes into an image sequence, which is then efficiently compressed by a video encoder. This invention achieves high compression ratio and high-fidelity reconstruction while strictly controlling model complexity, significantly improving rate distortion performance, and is suitable for high-quality free-viewpoint video applications on resource-constrained devices.

[0055] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0056] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. An end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression, characterized in that, include: Construct an initial dynamic Gaussian scene representation and perform preheating training to obtain a dynamic Gaussian field; The dynamic Gaussian field is decomposed into static and dynamic components using an adaptive mechanism. Differentiable budget control and importance-guided densification and pruning are implemented for the decomposed dynamic Gaussian field; A three-stage joint optimization training with controllable execution budget is based on the dynamic Gaussian field after densification and pruning. The trained dynamic Gaussian field is subjected to dual-mode hybrid compression to achieve end-to-end joint optimization of controllable dynamic Gaussian scene representation and dual-mode compression.

2. The end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression according to claim 1, characterized in that, The construction of the initial dynamic Gaussian scene representation and the execution of preheating training include: Using a multi-view dynamic sequence as input, a set of Gaussian points is initialized to represent the initial dynamic Gaussian scene, where each Gaussian point... All are preset to static, including 3D center. Rotation ,scale Opacity spherical harmonic coefficient Through rendering loss Perform containing Short-run optimization with quantitative constraints is used to obtain an initial Gaussian prior covering the scene geometry and coarse motion, providing a reliable foundation for subsequent dynamic and static decomposition and budget control; among which... The sparsity of the loss function is a tradeoff. To render the initial dynamic video sequence, A multi-view dynamic video sequence calibrated by the camera. The SSIM loss is for the two images.

3. The end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression according to claim 1, characterized in that, The step of using an adaptive mechanism to perform dynamic-static decomposition of the dynamic Gaussian field includes: Assign a translation attribute to each Gaussian point of the dynamic Gaussian field. To characterize its motion, a smooth histogram is constructed. And identify the two most significant peaks corresponding to the static Gaussian group and the dynamic Gaussian group; By locating the local minimum between the two peaks and calculating the percentiles of data dependence, an adaptive motion threshold is automatically determined. Based on this, the initial set of Gaussian points is unsupervisedly divided into mutually exclusive static subsets. With dynamic subsets ; Static subsets retain time-invariant properties but allow lightweight inter-frame translation, while dynamic subsets are configured with explicit temporal parameters, including position interpolation vectors. Rotation interpolation vector ,scale spherical harmonic coefficient and activation time window The opacity of the dynamic Gaussian decays smoothly outside the window and is activated only during the period when the object is visible, thus effectively modeling spatiotemporal sparsity.

4. The end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression according to claim 1, characterized in that, The aforementioned density enhancement and pruning, which involves differentiable budget control and importance guidance for the decomposed dynamic Gaussian field, includes: Define a uniform importance score : In the formula, for, for, Let be a Gaussian point. To fuse parameters such as position gradient, opacity, depth, anisotropic scale, and motion amplitude, To integrate parameters such as photometric residual, pixel coverage, and multi-view consistency, For global normalization; By using hard sigmoid gating through temperature annealing, continuous activation variables are introduced for each Gaussian. ; Define differentiable effective quantity and construct budget loss ;in, The number of constraints; In importance score Under the guidance of Gaussians exceeding a set threshold are cloned or split. Gaussian elements with a value less than or equal to a set threshold are randomly removed. A closed-loop Gaussian enrichment and pruning process is then performed to ensure the total number of Gaussian elements converges to a predetermined target during training. .

5. The end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression according to claim 1, characterized in that, The three-stage joint optimization training based on the optimized and pruned dynamic Gaussian field with controllable execution budget includes: The budget-controlled three-stage joint optimization training includes: a warm-up stage (stage one), a budget control stage (stage two), and a stabilization and fine-tuning stage (stage three); wherein: During the warm-up phase, only rendering loss is optimized. Initialize Gaussian coverage; During the budget control phase, the total loss is jointly optimized. ,in, To apply smoothing regularization to the time-series interpolation vector of the dynamic Gaussian, To construct budget loss, To measure the weighting coefficients of the construction budget loss, The weighting coefficients for smoothing regularization; During the stable fine-tuning phase, when the differentiable effective quantity Convergence approaches the constraint quantity Then, a fixed gating is used to freeze Gaussian existence, with only geometric and appearance parameters fine-tuned; the rendering loss in the final loss function... Controlling distortion D Budget loss implicitly controls the bit rate through Gaussian quantity constraints. R , and regular terms As a sparse constraint on dynamic redundancy, they together constitute the rate-distortion optimization objective.

6. The end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression according to claim 1, characterized in that, The step of performing dual-mode hybrid compression on the trained dynamic Gaussian field to achieve end-to-end joint optimization of controllable dynamic Gaussian scene representation and dual-mode compression includes: For static Gaussians, use The criteria separate foreground and background; foreground attributes are quantified and spatially reordered using a KD tree to ensure local continuity, then residuals are predicted based on neighborhood and entropy encoding is performed; background is stored directly. For dynamic Gaussians, outliers are separated to optimize data centrality, the remaining quantized attributes are reshaped into 2D images by channel, and the image sequences with the same attribute are grouped and compressed using a standard video encoder. The final model is stored as a bitstream and then restored after decoding.

7. The end-to-end joint optimization method for controllable dynamic Gaussian scene representation and dual-mode compression according to any one of claims 1-6, characterized in that, It also includes: evaluating the optimization results; where: Evaluate the quality of dynamic Gaussian scene reconstruction using PSNR and SSIM; Compression efficiency is measured by the storage volume of the reconstructed scene-related storage files.

8. An end-to-end joint optimization system for controllable dynamic Gaussian scene representation and dual-mode compression, characterized in that, include: The Dynamic Gaussian Field Construction Module is used to construct an initial dynamic Gaussian scene representation and perform preheating training to obtain a dynamic Gaussian field. A dynamic-static decomposition module, which uses an adaptive mechanism to perform dynamic-static decomposition on the dynamic Gaussian field; The densification and pruning module implements differentiable budget control and importance-guided densification and pruning for the decomposed dynamic Gaussian field. The three-stage joint optimization module performs budget-controlled three-stage joint optimization training based on the dynamic Gaussian field after densification and pruning. The hybrid compression module is used to perform dual-mode hybrid compression on the trained dynamic Gaussian field, completing the end-to-end joint optimization of controllable dynamic Gaussian scene representation and dual-mode compression.

9. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can be used to perform the method of any one of claims 1-7, or to run the system of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program can be used to perform the method of any one of claims 1-7, or to run the system of claim 8.

Images