Three-dimensional model generation method and device based on path tracking distillation, and storage medium

Abstract

The application discloses a three-dimensional model generation method and device based on path tracking distillation and a storage medium, the method uses an initial rendering image dataset as a training sample, the gradient deviation between the image data predicted by a two-dimensional teacher model optimized by distribution alignment and obtained through training and the initial rendering image data is small, and the training difficulty of three-dimensional asset data is reduced, in an iteration process, the prediction ability of the two-dimensional teacher model is gradually and continuously recovered by using a dynamic weight, and in the recovery process, a three-dimensional model, namely three-dimensional asset data, is trained, compared with the prior art which directly uses a two-dimensional teacher model to create a huge adaptation gap, produces a sharp deviation and a very high statistical variance, and is extremely easy to fall into a local optimum, the method provided by the application uses a relatively moderate gradient calculation process, can achieve convergence stability, realizes three-dimensional model geometry stability, and can quickly and efficiently obtain good visual detail quality.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method, apparatus, and storage medium for generating three-dimensional models based on path tracing distillation. Background Technology

[0002] With the booming development of cutting-edge fields such as virtual reality, augmented reality, and digital twins, the industrial demand for high-quality, high-precision 3D assets has exploded. However, traditional 3D modeling processes heavily rely on the knowledge and manual labor of professional artists, resulting in high production costs and lengthy development cycles, making it difficult to meet the needs of large-scale content production. In recent years, text-to-3D technology, with its extremely low creative threshold, powerful automated generation capabilities, and high degree of creative freedom, has become a core research hotspot and industrial application direction.

[0003] Currently, mainstream 3D text generation techniques typically employ Score Distillation Sampling (SDS) and its variants. The core logic of this approach lies in utilizing diffusion models pre-trained on large-scale 2D image datasets (such as LAION-5B) as a powerful general prior knowledge base (i.e., the teacher model). Through differentiable rendering techniques, a 2D view generated from a 3D representation (such as 3D Gaussian Splash 3DGS, NeRF, etc.) is projected onto a high-dimensional noisy space. The teacher model is then used to calculate the score gradient of this view relative to a given text prompt, thereby guiding the iterative optimization of the 3D parameters.

[0004] However, existing technologies face a severe out-of-distribution (OOD) challenge in practical applications, especially in the early stages of the generation process. Since 3D assets typically begin with randomly initialized point clouds, isotropic Gaussian distributions, or untrained neural weights, the resulting rendered images are extremely coarse in terms of geometric topology, texture continuity, and lighting consistency. The statistical properties of these initially degraded images in both pixel and feature spaces deviate significantly from the high-quality real-world image distribution fitted by the pre-trained 2D diffusion model. Because the teacher model has never encountered such low-quality samples with severe visual noise and structural defects during training, a huge mismatch gap exists between the rendered image distribution and the teacher model's predicted distribution.

[0005] Due to the inherent limitations of such out-of-domain distribution problems, pre-trained fractional distillation networks often exhibit drastic deviations and extremely high statistical variances in their output gradient directions when processing views located at or outside the edges of their distribution support sets. During the initial stages of the generation task, this inaccurate and noisy gradient guidance can easily cause the optimization algorithm to get stuck in local optima or lead to violent oscillations in the optimization trajectory. The most direct technical consequence is the collapse of the geometric structure of the 3D model. Furthermore, conventional fractional distillation schemes generally lack a complete, real-time-aware quantitative evaluation system for distribution deviation and a dynamic correction mechanism. In the process of guiding 3D optimization using 2D priors with fixed weights or simple linear scheduling, the failure to effectively address distribution misalignment often results in visual artifacts such as abnormally high saturation, excessively smoothed details and textures, and a lack of material realism. Ultimately, this leads to 3D assets that fail to meet production-grade application standards in terms of both visual fidelity and physical accuracy.

[0006] Therefore, how to construct a mechanism that can effectively quantify, track, and smoothly bridge the difference between the initial 3D rendering distribution and the actual distribution of the teacher model, thereby improving the convergence stability, geometric stability, and visual detail quality of 3D model generation, is a core issue that urgently needs to be addressed in this field and presents significant technical challenges. Summary of the Invention

[0007] This invention provides a method for generating 3D models based on path tracing distillation, which can improve the convergence stability, geometric stability, and visual detail quality of 3D model generation.

[0008] This invention provides a method for generating three-dimensional models based on path-tracing distillation, comprising:

[0009] Step (1): Convert the text prompts entered by the user into initial 3D asset data, and use a differentiable renderer to obtain an initial rendered image dataset based on the initial 3D asset data;

[0010] Step (2): Use the initial rendered image dataset as the training sample set to train the pre-trained two-dimensional teacher model with the low-rank adaptation module, update the parameters of the low-rank adaptation module, obtain the distribution-aligned optimized two-dimensional teacher model, and construct the loss function based on the image data output by the distribution-aligned optimized two-dimensional teacher model and the initial rendered image data to train the initial three-dimensional asset data.

[0011] The low-rank adaptation module with updated parameters is frozen, and dynamic weights are configured for the frozen low-rank adaptation module. The dynamic weights change continuously from 1 to 0 based on the increase of the number of iteration steps.

[0012] Step (3): Iterate through steps (1)-(2) until the iteration count exceeds the threshold, and then stop iterating to obtain the final three-dimensional asset data.

[0013] Preferably, when the number of iteration steps is less than or equal to the preset number of preheating steps, the dynamic weight changes continuously from 1 to 0 as the number of iteration steps increases;

[0014] When the number of iteration steps is greater than the preset number of warm-up steps, the dynamic weight is 0.

[0015] Preferably, converting user-inputted text prompts into initial 3D asset data includes:

[0016] Based on the text prompts input by the user, the initialization model is used to extract features and construct the corresponding initial 3D asset data;

[0017] The initialization model includes a pre-trained feedforward reconstruction network or a geometric extraction algorithm.

[0018] Preferably, the initial three-dimensional asset data consists of multiple Gaussian elements, and the parameter set of each Gaussian element includes the center position μ, rotation quaternion q, scaling factor s, opacity α, and spherical harmonic function coefficient c;

[0019] The initial three-dimensional asset data is represented in the form of three-dimensional Gaussian splash.

[0020] Preferably, the initialization model includes an efficient 3D model synthesis method based on point cloud diffusion or a stable and fast 3D mesh reconstruction method.

[0021] Preferably, an initial rendered image dataset is obtained based on the initial 3D asset data using a differentiable renderer, including:

[0022] The initial 3D asset data is projected from multiple random viewpoints using a differentiable renderer to obtain a set of initial rendered image datasets corresponding to the camera viewpoints.

[0023] Preferably, the pre-trained two-dimensional teacher model is a stable diffusion model.

[0024] The present invention also provides a three-dimensional model generation apparatus based on path-tracing distillation, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the three-dimensional model generation method based on path-tracing distillation.

[0025] The present invention also provides a storage medium that, when executed by a processor, implements the described method for generating three-dimensional models based on path-tracing distillation.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] This invention utilizes the initial rendered image dataset as training samples. The image data predicted by the distribution-aligned optimized 2D teacher model obtained through training has a small gradient deviation from the initial rendered image data, reducing the training difficulty of 3D asset data. During the iteration process, the predictive ability of the 2D teacher model is gradually and continuously restored using dynamic weights. During the restoration process, the corresponding 3D model, i.e., 3D asset data, is trained. Compared with existing technologies that directly use the 2D teacher model to create a huge adaptation gap, resulting in severe bias and extremely high statistical variance, and are extremely prone to getting trapped in local optima, the method provided by this invention uses a more moderate gradient calculation process, which can achieve convergence stability, realize the geometric stability of the 3D model, and obtain better visual detail quality quickly and efficiently. Attached Figure Description

[0028] Figure 1 A flowchart of a three-dimensional model generation method based on path tracing distillation is provided for a specific embodiment of the present invention;

[0029] Figure 2 The diagram illustrates the iterative process in the path-tracing distillation-based 3D model generation method provided in a specific embodiment of the present invention. Detailed Implementation

[0030] This invention addresses the technical bottlenecks in the initial stage of 3D asset generation using 2D pre-trained priors, which are caused by significant deviations between the rendered distribution and the real image distribution (i.e., out-of-domain distribution problems). These bottlenecks result in unstable optimization processes, excessive gradient prediction variance, and low visual fidelity of the 3D model. The invention provides a 3D model generation method based on path tracing distillation. This method effectively bridges the distribution gap in the early stages of generation by constructing and tracing intermediate distribution paths, aiming to significantly reduce initial matching losses while ensuring high fidelity and semantic consistency of the generated assets.

[0031] The specific embodiments of the present invention provide a method for generating three-dimensional models based on path-tracing distillation, such as... Figure 1 As shown, it includes:

[0032] Step (1): Convert the user-inputted text prompts into initial three-dimensional (3D) asset data, and obtain an initial rendered image dataset based on the initial three-dimensional asset data using a differentiable renderer.

[0033] In one specific embodiment, this embodiment converts user-inputted text prompts into initial 3D asset data, including: constructing corresponding initial 3D asset data by performing feature extraction using an initialization model based on user-inputted text prompts; the initialization model includes a pre-trained feedforward reconstruction network or a geometric extraction algorithm.

[0034] The initialization model provided in this specific embodiment of the invention extracts the vertex spatial coordinates, color coefficients, opacity, and scaling parameters of the geometric skeleton based on the received text prompt word 'y', thereby constructing an initial 3D Gaussian point cloud and obtaining initial 3D asset data. Explicit 3D Gaussian Splatting is used as the underlying representation, leveraging its efficient differentiable rendering characteristics to provide a parameter space with clear geometric topological constraints for the subsequent gradient distillation process. This provides a definite parameter starting point for subsequent differentiable rendering, avoiding the topological structure chaos problem that easily occurs in the initial stage of implicit representation. In practice, this initialization process can be combined with feedforward networks such as the efficient 3D model synthesis method Point-E based on point cloud diffusion or the stable fast 3D mesh reconstruction method SF3D (Stable Fast3D). These methods quickly generate a preliminary Gaussian distribution structure and determine the corresponding Gaussian parameters, thereby achieving rapid initialization of 3D Gaussian attributes and ensuring that the initial model has a basically correct proportional relationship on a macroscopic scale.

[0035] The initial three-dimensional asset data provided in the specific embodiments of the present invention consists of multiple Gaussian elements. The parameter set of each Gaussian element includes the center position μ, rotation quaternion q, scaling factor s, opacity α, and spherical harmonic function coefficient c.

[0036] In one specific embodiment, the method of obtaining an initial rendered image dataset based on the initial 3D asset data using a differentiable renderer includes: projecting the initial 3D asset data from multiple random viewpoints using a differentiable renderer to obtain a set of initial rendered image datasets corresponding to camera viewpoints. Since the geometry and texture of the initial assets, i.e., the initial rendered image dataset, are in a non-converged state, the statistical characteristics of its initial rendered image set differ significantly from the data distribution of the pre-trained 2D teacher model. To bridge this distribution bias, this specific embodiment of the invention actively reduces the quality of the predicted distribution of the pre-trained 2D teacher model to bridge the gap with the initial rendered images and avoid excessive gradients.

[0037] Step (2): Path construction method: Use the initial rendered image dataset as the training sample set, train the pre-trained two-dimensional teacher model with low-rank adaptation module, update the parameters of the low-rank adaptation module, obtain the distribution-aligned optimized two-dimensional teacher model, and construct the loss function based on the image data output by the distribution-aligned optimized two-dimensional teacher model and the initial rendered image data based on the text prompt words.

[0038] The updated low-rank (LoRA) adapter module is frozen, and dynamic weights are configured on the frozen low-rank adapter module. The dynamic weights change continuously from 1 to 0 based on the increase of the number of iteration steps.

[0039] Step (3): Iterate through steps (1)-(2) until the iteration count exceeds the threshold, and then stop iterating to obtain the final three-dimensional asset data.

[0040] Specifically, such as Figure 2 As shown, during the iteration process, the original pre-training distribution parameters of the pre-trained two-dimensional teacher model... Without changing the parameters, after obtaining the distribution-aligned optimized two-dimensional teacher model, the parameters of the low-rank adaptation module are frozen, i.e. The process remains unchanged, but the scheduling path control logic is based on the dynamic weights adjusted according to the number of iteration steps. Continuous intermediate distribution path for regulation ,in iter The number of iterations is used to perform path tracking and gradient distillation, and optimization is achieved through backpropagation. The three-dimensional asset parameters are updated. In this way, the specific embodiment of the present invention successfully decomposes the originally huge distribution differences into a series of continuous intermediate distribution states with smaller distribution spans.

[0041] In each iteration step, such as Figure 2 As shown, through the dynamic weight function The contribution ratio of the low-rank adaptation module to the distillation gradient is adjusted, that is, the contribution ratio gradually decreases as the number of iterations increases. This continuously improves the quality of the 3D asset data while gradually restoring the predictive ability of the pre-trained 2D teacher model, making the quality of the 3D asset data more compatible with this predictive ability. Figure 2 As shown in the logical architecture, by continuously interpolating the original teacher model distribution with the LoRA distribution controlled by dynamic weights, the system can generate more accurate guiding gradients.

[0042] The dynamic weight function provided in the specific embodiments of the present invention The core function is to establish a distribution scheduling mechanism that automatically adjusts as the optimization process progresses. In the initial stage of distillation optimization, due to the significant distribution bias between the initial observation distribution of the 3D assets and the pre-trained distribution of the teacher model, the path tracing distillation module assigns higher weight coefficients to the path node models, ensuring that the guided gradient is preferentially generated from a finely tuned intermediate distribution state with better local consistency. As the optimization iterations advance, the dynamic weight function... The system guides a smooth reduction in dependence on intermediate distributions and gradually backtracks towards the original, unadjusted 2D teacher model distribution. This dynamic tracking process not only achieves substantial compression of gradient variance but also ensures a seamless transition of 3D assets from macroscopic structural correction to microscopic texture refinement. The dynamic weighting function... During the tracking process, the following scheduling relationship is preferred: , This represents the current iteration step. The preset number of preheating steps, This is the decay rate control coefficient. In the early stages of optimization, this module corrects the macroscopic geometry by assigning higher weights to the path node model; as the iteration progresses, the weight components are gradually reduced to achieve a smooth transition to the original distribution, ultimately obtaining optimized high-fidelity 3D asset data.

[0043] To further demonstrate the implementation details and significant effects of this invention in a specific generation pipeline, the method described in this invention can be integrated as a core plugin into advanced 3D generation frameworks based on fractional distillation, such as LucidDreamer. In this exemplary application, the total number of iterations for the entire 3D asset generation is set to 5000 steps. Before optimization begins, the path construction module first enters the offline fine-tuning stage. Guided by the input text prompts, the system uses initialization methods such as Pointe to obtain initial Gaussian assets and collects 32 representative rendering views. Subsequently, the LoRA plugin is attached to the pre-trained Stable Diffusion 2.1 model. In terms of fine-tuning parameter configuration, the rank of LoRA is preferably set to 4, and the learning rate is usually maintained at [value missing]. These parameter settings aim to ensure that the fine-tuning process accurately captures the degradation distribution characteristics of the initial assets without compromising the pre-trained model's original general semantic understanding capabilities, thereby improving guidance accuracy while maintaining generative diversity. After a total of 200 fine-tuning iterations, the system saves the converged final low-rank adaptive weight model. This model defines a complete evolution vector in the weight space for the migration from the current asset's "degraded observation distribution" to the "ideal target distribution". In subsequent generation processes, the system does not need to store multiple intermediate node models; instead, it uses a dynamic weight function. By continuously interpolating the single-weight model, a smooth intermediate distribution path is implicitly reconstructed. This path representation method based on single-model weight interpolation not only significantly reduces storage overhead and computational complexity, but also ensures strong continuity of the generated trajectory in the parameter space, avoiding gradient instability that may be caused by discrete node switching.

[0044] In the subsequent online optimization process, the path tracing distillation module is responsible for guiding the asset's evolution. In this example, the first 1000 steps are configured as a path tracing warm-up period, and the following 4000 steps are a regular texture refinement period. Within the first 1000 steps, the path tracing distillation module applies a dynamic weighting function to adjust the gradient contribution ratio. The following scheduling formula is satisfied:

[0045]

[0046] In the very early stages of optimization, because the asset structure is not yet stable, the system assigns extremely high weights to the fine-tuned path node models. This ensures that gradient guidance primarily originates from the "intermediate neighborhood distribution" that is highly consistent with the current asset distribution. This significantly suppresses the erroneous gradient directions and high variance noise generated by the original teacher model when processing coarse assets. As the number of iterations increases and asset quality gradually improves, the weight function guides the system to smoothly backtrack towards the original, un-fine-tuned cooperative teacher model distribution. After more than 1000 steps, the weights are reset to zero, and the system is entirely guided by the original pre-trained model. This continuous weight interpolation mechanism ensures that the 3D assets are always guided by an accurate and low-variance distribution throughout the evolution process, effectively avoiding the convergence imbalance phenomenon common in the generation domain.

[0047] Experimental data demonstrate that this invention exhibits significant technical advantages in the aforementioned LucidDreamer ensemble example of the interval score distillation-based 3D generation framework. During the initial generation phase (i.e., the first 1000 steps), by introducing a Stable Target Field (STF) quantization evaluation and path repair mechanism, the initial matching loss is significantly reduced. Compared to the comparative experiment without the path-tracking mechanism of this invention, the performance decreased by approximately 40%. This significant reduction signifies a fundamental reversal of gradient prediction bias, enabling the macroscopic geometry of the 3D Gaussian point cloud to converge stably in a very short time. Further multi-view quantitative evaluation was performed on the final generated 3D model, using the CLIP (Contrastive Language-Image Pre-training) model to calculate its feature cosine similarity to the input text prompts. The results show that the average CLIP score of the baseline method was 40.27, while the CLIP score of the final asset jumped to 42.36 after adopting the path-tracking distillation module of this invention. This fully demonstrates that this invention can guide assets to move on a more accurate distribution path, ensuring that the generated results achieve high standards for industrial applications in terms of visual fidelity and semantic consistency.

[0048] On the other hand, the present invention also provides a three-dimensional model generation apparatus based on path-tracing distillation, including a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the three-dimensional model generation method based on path-tracing distillation.

[0049] On the other hand, the present invention also provides a storage medium that, when executed by a processor, implements the described method for generating three-dimensional models based on path-tracing distillation.

Claims

1. A method for generating a three-dimensional model based on path-tracing distillation, characterized in that, include: Step (1): Convert the text prompts entered by the user into initial 3D asset data, and use a differentiable renderer to obtain an initial rendered image dataset based on the initial 3D asset data; Step (2): Use the initial rendered image dataset as the training sample set to train the pre-trained two-dimensional teacher model with the low-rank adaptation module, update the parameters of the low-rank adaptation module, obtain the distribution-aligned optimized two-dimensional teacher model, and construct the loss function based on the image data output by the distribution-aligned optimized two-dimensional teacher model and the initial rendered image data to train the initial three-dimensional asset data. The low-rank adaptation module with updated parameters is frozen, and dynamic weights are configured for the frozen low-rank adaptation module. The dynamic weights change continuously from 1 to 0 based on the increase of the number of iteration steps. Step (3): Iterate through steps (1)-(2) until the iteration count exceeds the threshold, then stop iterating to obtain the final three-dimensional asset data; During the iteration process, the original pre-trained distribution parameters of the pre-trained two-dimensional teacher model... Without changing the parameters, after obtaining the distribution-aligned optimized two-dimensional teacher model, the parameters of the low-rank adaptation module are frozen, i.e. The process remains unchanged, but the scheduling path control logic is based on the dynamic weights adjusted according to the number of iteration steps. Continuous intermediate distribution path for regulation , where iter is the iteration step, performs path tracking to distill gradients, and updates 3D asset parameters through backpropagation optimization; Dynamic weight function The following scheduling relationship is satisfied during the tracing process: iter is the iteration number. The preset number of preheating steps is denoted by C, which is the decay rate control coefficient. In the early stages of optimization, the path node model is assigned a higher weight through the path tracing distillation module to correct the macroscopic geometry. As the iteration progresses, the weight components are gradually reduced to achieve a smooth transition to the original distribution, ultimately obtaining optimized high-fidelity 3D asset data.

2. The method for generating a three-dimensional model based on path-tracing distillation according to claim 1, characterized in that, When the number of iteration steps is less than or equal to the preset number of warm-up steps, the dynamic weight changes continuously from 1 to 0 as the number of iteration steps increases; When the number of iteration steps is greater than the preset number of warm-up steps, the dynamic weight is 0.

3. The method for generating a three-dimensional model based on path-tracing distillation according to claim 1, characterized in that, Convert user-inputted text prompts into initial 3D asset data, including: Based on the text prompts input by the user, the initialization model is used to extract features and construct the corresponding initial 3D asset data; The initialization model includes a pre-trained feedforward reconstruction network or a geometric extraction algorithm.

4. The method for generating a three-dimensional model based on path-tracing distillation according to claim 1, characterized in that, The initial three-dimensional asset data consists of multiple Gaussian elements, and the parameter set of each Gaussian element includes the center position μ, rotation quaternion q, scaling factor s, opacity α, and spherical harmonic function coefficient c; The initial three-dimensional asset data is represented in the form of three-dimensional Gaussian splash.

5. The method for generating a three-dimensional model based on path-tracing distillation according to claim 3, characterized in that, The initialization model includes efficient 3D model synthesis methods based on point cloud diffusion or stable and fast 3D mesh reconstruction methods.

6. The method for generating a three-dimensional model based on path-tracing distillation according to claim 1, characterized in that, Based on the initial 3D asset data, an initial rendered image dataset is obtained using a differentiable renderer, including: The initial 3D asset data is projected from multiple random viewpoints using a differentiable renderer to obtain a set of initial rendered image datasets corresponding to the camera viewpoints.

7. The method for generating a three-dimensional model based on path-tracing distillation according to claim 1, characterized in that, The pre-trained two-dimensional teacher model is a stable diffusion model.

8. A three-dimensional model generation device based on path-tracing distillation, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the three-dimensional model generation method based on path tracing distillation as described in any one of claims 1-7.

9. A storage medium, characterized in that, The program is stored on a storage medium, and when the program is executed by a processor, it implements the three-dimensional model generation method based on path tracing distillation as described in any one of claims 1-7.

Images