A mobile terminal-oriented dynamic distillation and lightweight 3D Gaussian scene reconstruction method

Abstract

The application discloses a mobile terminal-oriented dynamic distillation and lightweight 3D Gaussian scene reconstruction method, and relates to the technical field of computer vision and three-dimensional reconstruction. The method aims to solve the problem that the existing 3D Gaussian sputtering (3DGS) model occupies high video memory and has large rendering calculation amount, and is difficult to run smoothly on a mobile terminal device. The application introduces a sliding window mechanism to monitor the point cloud growth trend, and adaptively triggers a "teacher-student" distillation strategy in the geometric structure stable period. After triggering, the high-order SH model is solidified as a teacher model, and the spherical harmonic coefficient of the current model is truncated and down-ordered as a student model in the physical layer. The rendering image and depth image of the teacher model are used to supervise the student model training, so that the high-frequency details are maintained while the parameter amount is reduced. In addition, combined with a global pruning strategy based on cumulative rendering contribution degree and a gradient pruning strategy, invalid redundant points are further removed. Experiments show that the method can maintain high-quality rendering effect while greatly reducing the model storage volume and memory occupation, and significantly improves the rendering frame rate on a mobile device.

Description

Technical Field

[0001] This invention belongs to the fields of computer graphics, computer vision and 3D reconstruction technology, and specifically relates to a method for 3D Gaussian scene reconstruction, model compression and lightweight training under mobile hardware resource-constrained environments. Background Technology

[0002] In recent years, with the rapid development of mobile augmented reality applications, achieving photorealistic real-time 3D rendering on mobile devices has become a research hotspot in academia and industry. Neural Radiation Field (NeRF) technology has made breakthroughs in implicit 3D representation and novel perspective synthesis, but its neural network-based volumetric rendering mechanism results in massive inference computation, making real-time interaction on mobile devices difficult. As an alternative, 3D Gaussian Sputtering (3DGS) technology uses an anisotropic set of 3D Gaussian spheres as an explicit representation of the scene, combined with an efficient differentiable rasterization pipeline, achieving rendering speeds of over 100 frames per second on PCs. However, directly migrating this technology to mobile devices faces severe challenges: standard 3DGS stores higher-order spherical harmonic function coefficients for each Gaussian point, resulting in model files reaching hundreds of megabytes or even gigabytes, which is extremely unfriendly to mobile devices with limited storage space; simultaneously, the unified memory architecture of mobile devices creates bandwidth bottlenecks when rendering high-order models due to the GPU frequently reading large amounts of coefficient data, causing frame drops, device overheating, and reduced battery life. Existing lightweight strategies have significant limitations. The method of training first and then compressing not only increases training overhead but also struggles to correct geometric distortions caused by compression. While training at low levels throughout reduces the workload, it easily loses high-frequency details, making it difficult to meet the demands of high-quality vision. Therefore, a new training method is needed that can intelligently adjust during training to balance high-precision construction with lightweight distillation, significantly reducing computational and storage overhead while maintaining visual quality. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies, such as large 3D Gaussian model size, high rendering load on mobile devices, and low efficiency of existing compression methods, and to provide a dynamic distillation and lightweight 3D Gaussian scene reconstruction method for mobile deployment. This method embeds the model lightweighting process into the training loop, achieving a balance between high fidelity and low computational consumption.

[0004] The specific steps are as follows: S1. Initial training phase: Initialize the 3D Gaussian model, perform regular training and conservative pruning within the preset initial iteration interval, and open a sliding window to record historical data on the number of point clouds. S2, Dynamic Trigger Detection Phase: When training enters a specific detection interval, the geometric stability index is calculated based on the rate of change of the number of points in the sliding window; when the geometric stability index meets the convergence threshold or reaches the upper limit of the detection interval, the distillation lightweight mechanism is triggered. S3, Teacher-Student Distillation Architecture Construction Phase: After the triggering mechanism, the current complete spherical harmonic order model state is immediately saved as the teacher model; at the same time, the current training model is physically reduced to the student model, and the optimizer state is reset. S4. Hybrid Loss Distillation Training Phase: In subsequent training, while the student model learns from real images, it uses the rendered images and depth maps generated by the teacher model for supervised learning, and optimizes the low-order student model through the hybrid loss function. S5. Contribution-based global pruning stage: During training, the cumulative contribution of each Gaussian point is calculated; when the preset densification cutoff iteration number is reached, the Gaussian point cloud is globally pruned based on the cumulative contribution, redundant points are removed, and the final lightweight model is obtained. This invention significantly reduces model size (50%-70%) while substantially reducing GPU memory bandwidth usage and improving frame rates for mobile rendering through physical reduction, dynamic triggering distillation mechanisms, and contribution-based pruning strategies. The seamless integration of the training and lightweighting processes allows the model to be deployed directly after training without the need for a separate compression step. The multimodal joint distillation strategy avoids the "dull" effect of low-order models, ensuring the final visual quality. Attached Figure Description

[0005] Appendix Figure 1 This is a flowchart of the dynamic distillation and lightweight 3D Gaussian scene reconstruction method for mobile deployment of the present invention; Appendix Figure 2 This is a flowchart of the dynamic triggering mechanism before distillation of the present invention; Appendix Figure 3 This is a flowchart of the multimodal collaborative distillation training process of the present invention; Appendix Figure 4 This is a comparison chart of the lightweight design of this invention and the parameters of the original model; Detailed Implementation

[0006] This invention provides a dynamic distillation and lightweight 3D Gaussian scene reconstruction method for mobile devices. To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description, in conjunction with a specific embodiment, further illustrates the invention.

[0007] I. Overall System Initialization and Environment Setup In this implementation, the first step is to build a computing environment optimized for mobile devices. The system is developed based on the PyTorch framework and integrates CUDA operators for accelerating 3D Gaussian rendering. During the data preparation phase, the input consists of a multi-view image sequence of the scene to be reconstructed and sparse point clouds obtained through structures such as COLMAP. The core of the initialization phase lies in the construction of the GaussianModel. This method sets the initial Spherical Harmonics (SH) order to 3, meaning that each Gaussian point needs to store one basic color coefficient and 15 higher-order color coefficients, totaling 48 floating-point numbers to represent color. In addition, the Gaussian center position, opacity, scaling, and quaternion rotation also need to be initialized. To adapt to the storage limitations of mobile devices, the system pre-configures optimizer type monitoring during initialization, utilizing SparseGaussianAdam to optimize memory usage.

[0008] II. Dynamic Perception and Intelligent Triggering Logic of Training Status Unlike traditional fixed-round training methods, this invention implements an intelligent triggering mechanism based on the evolution trend of point clouds. After training begins, the system starts a monitoring thread to record the number of Gaussian points after each iteration in real time. Figure 2 During implementation, the program sets a sliding window W, whose length is typically set to 500 iterations. The system maintains a queue to store the point cloud size within this window. Whenever the number of iterations reaches a multiple of the window size, the system calculates the point cloud growth rate between the start and end points of the window. That is: The judgment logic is as follows: if it is within the specified sensitive interval (e.g., between 6000 and 8000 iterations). If the absolute value of the density is less than a preset threshold of 0.01 (i.e., the growth rate is less than 1%), the system determines that the current geometric topology has stabilized and no further large-scale geometric densification is needed. If the stabilization condition is not met after 8000 iterations, the system will execute a hard-triggered strategy. The significance of this dynamic awareness is that scenes of different complexities converge at different speeds. Dynamic triggering ensures that each scene enters the lightweighting stage when its geometry is most complete, avoiding the loss of details due to premature compression or the computational waste caused by late compression.

[0009] III. Truncation and Reduction of Higher-Order Features at the Physical Level Once the trigger condition is met, the system immediately enters the physics reduction phase. This is one of the key steps in achieving lightweighting in this invention. At the moment of triggering, the system first performs a complete backup of the current full-order (SH-3) Gaussian model, storing it in memory as the "teacher model." Subsequently, the system performs physics-level parameter pruning on the active model (student model). Specifically, the tensor storing the spherical harmonic coefficients is sliced, retaining only the first 3 channels (corresponding to SH-1 order), and the parameters of the remaining 12 channels are directly erased from memory. At this point, it is forcibly locked to 1. This step directly reduces the storage overhead of color in the model by approximately 75%. Due to the abrupt change in parameter shape, the original optimizer state (first-order momentum and second-order momentum) is no longer applicable. Therefore, this implementation executes a new optimizer. This approach ensures that the model can restart convergence with extremely low parameter counts and find the optimal solution in low-dimensional space.

[0010] IV. Collaborative Distillation Training Based on Teacher-Student Architecture After physical order reduction, the student model retains only the basic coefficients of the spherical harmonic function. and This leads to significant loss of detail and rendering blur when the model represents visual features with strong anisotropy—such as highlights on metallic surfaces, specular reflections, and view-dependent colors at object edges. To salvage visual accuracy under the constraint of extremely low parameter count, this invention implements a cross-dimensional online collaborative distillation algorithm, such as... Figure 3 .

[0011] 1. Mathematical representation and data flow of the distillation mechanism In each training iteration, the system randomly selects a camera viewpoint using a uniform distribution. At this point, the system has two parallel rendering branches: Teacher branch: Generates high-fidelity reference images using a full-order model (SH-3) that locks instantly upon triggering. and associated depth map .

[0012] Student branch: Generate predicted images using the current physical reduction model to be optimized (SH-1). and predicted depth map .

[0013] 2. Construction of a Multi-Constraint Collaborative Loss Function To achieve the synchronous transfer of color consistency and geometric topological stability, this invention constructs a composite loss function. The loss function consists of three main parts: ground truth supervision, color distillation, and depth distillation. Its complete mathematical definition is as follows:

[0014] (1) Among them, As a truth-checking term, it is defined as: This combination method involves setting hyperparameters. (Usually a value of 0.2) effectively balances pixel-level intensity differences with topological similarity in the perceptual dimension.

[0015] (2) Distillation of color space knowledge To compensate for the loss of higher-order spherical harmonic coefficients in the student model, this invention introduces the rendering result of the teacher model as a soft label. The color distillation term is defined as the average absolute error between the student and teacher rendered images in RGB space.

[0016] In this embodiment, color weights are set. This loss term forces the lower-order student model to fit the complex anisotropic light and shadow distribution produced by the teacher model by adjusting its basic coefficients and spatial location distribution, thereby achieving... The next step is to simulate close to The visual perception of the steps.

[0017] (3) Depth-consistent geometric distillation Physical reduction often involves a redistribution of parameter gradients, which can easily lead to "drifting" of the point cloud in the depth dimension or the generation of thickness artifacts. Therefore, this invention utilizes precise geometric priors provided by the teacher model for depth constraints: Set weight coefficients The depth distillation loss function effectively locks in the scene's geometric topology by limiting the consistency between the student model and the teacher model in the depth map projection, preventing spatial structure collapse during aggressive pruning and parameter reduction.

[0018] 3. Dynamic Evolution of Distillation Training In actual implementation, this co-distillation process exhibits significant stage-specific characteristics. In the initial stage of distillation, due to... The strong guiding role of the model leads to a rapid parameter remapping process in the student model, transforming the nonlinear visual features stored in a high-dimensional tensor by the teacher model into a linear combination in a low-dimensional space. As training progresses, this process, through traditional... By combining supervision with the joint optimization proposed in this invention, the model can still maintain sharp edge contours and accurate lighting transitions even in the low computing power environment commonly found on mobile devices, ultimately achieving high-quality scene reconstruction under bidirectional compression of point cloud scale and feature dimension.

[0019] This collaborative distillation scheme not only solves the performance degradation caused by lightweighting, but also significantly improves the generalization ability of student models in unobserved perspectives through deep guidance of teacher models, laying the algorithmic foundation for real-time interaction of 3D scenes in mobile browsers or embedded applications.

[0020] V. Cumulative Contribution Statistics and Overall Importance Pruning To further compress the number of point clouds, this implementation introduces an evaluation system based on global contribution. The system defines a tensor of equal length to the number of point clouds to store the cumulative contribution score of each Gaussian point. In each rendering loop, the visibility filter of the CUDA renderer is used to statistically analyze in real time which Gaussian points participated in the pixel composition of the current frame. As long as a point is projected into screen space and its contribution value is not zero, its corresponding value is incremented by 1. This accumulation mechanism is maintained throughout the mid-to-late stages of training. When the iteration reaches the densification endpoint, the system performs a "global pruning." The program calculates the average contribution score of all points and sets a retention threshold (e.g., 10% of the average). Gaussian points that exist in space but are rarely observed, or whose contribution to the final image is negligible, are permanently removed. In this way, the present invention can accurately identify and remove redundant transparent points and outliers outside the scene.

[0021] VI. Parameter Fixing and Storage after Mobile Optimization After completing 30,000 rounds of training (or other preset endpoints), the system performs final parameter solidification. During the parameter solidification stage of this invention, the system performs a final activation filter on the opacity values ​​stored in the memory. Perform a nonlinear mapping to calculate its actual physical opacity. Subsequently, a point cloud density evaluation operator is defined. The following formula is used to filter Gaussian points across the entire domain:

[0022] This step further compresses the model storage size by eliminating spurious Gaussian points with extremely high transmittance (i.e., almost transparent) in space, while ensuring that the PSNR index does not decrease significantly and the reconstruction quality is not significantly affected.

[0023] Compared to the original 3DGS, the final generated model file not only reduces the number of point clouds by 40%-60%, but also significantly reduces the attribute bit width of individual points due to physical downsizing. When deployed on mobile devices, because the SH order is 1, the number of instructions the GPU needs to calculate spherical harmonics is greatly reduced, thus improving the throughput of the rendering pipeline.

[0024] Analysis of the implementation effects of the method of this invention reveals that the strategy of combining dynamic distillation with physical reduction can significantly reduce the size of the Gaussian point cloud and the dimension of single-point features while maintaining high reconstruction quality and visual consistency. Figure 4 Compared to the unweighted 3D Gaussian model, the lightweight model generated by this invention significantly reduces storage volume, parameter size, and rendering computational complexity, while showing no significant degradation in key visual features such as main geometric contours, occlusion relationships, and lighting transitions. This indicates that the online collaborative distillation mechanism proposed in this invention can effectively compensate for the performance loss caused by model downsizing. Regarding training costs, due to the introduction of a dual-path rendering distillation architecture, the training time for the lightweight version slightly increases from 27 minutes and 16 seconds to 32 minutes and 45 seconds (an increase of approximately 5 minutes and 29 seconds), which is an acceptable offline computational cost. Further analysis shows that by introducing color and depth joint distillation constraints from the teacher model during training, this invention enables the low-order student model to effectively inherit the visual expressive capabilities of the high-order model within a limited parameter space, thus maintaining stable reconstruction results even with significant model compression. The additional training overhead introduced by this method occurs offline and does not affect the actual running efficiency of the model on the terminal side. Conversely, due to the significant reduction in model size and rendering computation, this invention can effectively alleviate memory bandwidth pressure and shading computation burden when deployed on mobile devices, improving overall rendering smoothness and system stability.

Claims

1. A dynamic distillation and lightweight 3D Gaussian scene reconstruction method for mobile devices, characterized in that, Includes the following steps: S1. Initial Training Phase: Initialize the 3D Gaussian model, perform regular training and conservative pruning within the preset initial iteration interval, and simultaneously open a sliding window to record historical data on the number of point clouds. S2. Dynamic Trigger Detection Phase: When training enters a specific detection interval, the geometric stability index is calculated based on the rate of change of the number of points in the sliding window; when the geometric stability index meets the convergence threshold or reaches the upper limit of the detection interval, the distillation lightweight mechanism is triggered. S3. Construct a teacher-student distillation architecture: After the mechanism is triggered, immediately save the current complete spherical harmonic order model state as the teacher model; at the same time, perform a physical order reduction operation on the current training model, use it as the student model, and reset the optimizer state; S4. Hybrid Loss Distillation Training Phase: In subsequent training, while the student model learns from real images, it uses the rendered images and depth maps generated by the teacher model for supervised learning, and optimizes the low-order student model through the hybrid loss function. S5. Contribution-based global pruning stage: During training, the cumulative contribution of each Gaussian point is calculated; when the preset densification cutoff iteration number is reached, the Gaussian point cloud is globally pruned based on the cumulative contribution, redundant points are removed, and the final lightweight model is obtained.

2. The method according to claim 1, characterized in that, The specific implementation of step S2 includes: establishing a first-in-first-out (FIFO) queue of length W as a sliding window to store the total number P of Gaussian point clouds in the most recent W iterations; calculating the rate of change of ΔP; if |ΔP| < ε, where ε is a preset convergence threshold, then it is determined that the scene geometry has converged, and step S3 is triggered immediately; if the convergence condition is not met at the end of the detection interval, step S3 is forcibly triggered to ensure the execution of the lightweight process.

3. The method according to claim 1, characterized in that, The physical order reduction operation in step S3 specifically refers to: first, deeply copying and freezing the current full-parameter model to generate the teacher model; then, truncating the spherical harmonic coefficient (SH) parameter tensor of the 3D Gaussian model, retaining only the coefficient data of order 0 or low order (such as order 1); and forcibly locking the active spherical harmonic order of the model to the reduced order. Physical compression of model parameters is achieved by removing the memory usage of higher-order SH coefficients.

4. The method according to claim 1, characterized in that, The hybrid distillation loss function Ltotal in step S4 is composed of the following three linearly combined parts: the first part is the ground truth supervision loss LGT (including L1 loss and structural similarity loss Lssim) between the student model rendering result and the real view image; the second part is the rendering color consistency loss LRGB between the student model and the teacher model under the same view; and the third part is the rendering depth consistency loss LDepth between the student model and the teacher model under the same view.

5. The method according to claim 1, characterized in that, The specific implementation of step S5 includes: initializing a counting vector C with a length equal to the total number of Gaussian points, with an initial value of all zeros; during each forward rendering process, using the rasterizer's visibility filter to identify the indices of Gaussian points participating in imaging from the current viewpoint; updating the counting vector C accordingly, such that Ci ← Ci+1; calculating the mean μc of the counting vector C when a preset pruning iteration round is reached; and constructing a pruning mask M that satisfies Ci < α. The Gaussian points of μc are marked as removed (where α is the retention coefficient) and permanently deleted from the model.

6. The method according to claim 1, characterized in that, Also includes: After performing the model reduction operation, the gradient threshold during the Gaussian point densification process is adjusted; the point cloud splitting and cloning thresholds after triggering distillation are increased to maintain the stability of the geometric structure at low parameter scales.

7. A dynamic distillation and lightweight 3D Gaussian scene reconstruction method system for mobile devices, characterized in that, include: Initialization module: Used to build the initial 3D Gaussian representation and perform warm-up training; Dynamic monitoring module: used to monitor the geometric stability of point clouds in real time through a sliding window and output trigger signals; Model conversion module: used to perform teacher model backup and physical reduction of student model when a trigger signal is received; Joint distillation module: used to perform the calculation of the blending loss function, which includes color consistency and depth consistency, and backpropagation; Pruning and optimization module: Used to perform global redundant point removal based on rendering contribution statistics.

Images