Three-dimensional model generation method based on two-stage mixed representation 3DGS

Abstract

The invention discloses a three-dimensional model generation method based on two-stage mixed representation 3DGS, and the method comprises the steps: firstly, rapidly generating a basic 3D structure through a pre-trained 3D diffusion model, and converting the basic 3D structure into initial 3DGS representation through a Gaussian parameterization algorithm based on local curvature; then, an optimization strategy of geometry and texture decoupling is adopted, improved fractional distillation sampling (SDS) loss and multi-view consistency constraints are combined, and the geometry and texture of the 3DGS are finely adjusted; and finally, efficient optimization is realized through a dynamic Gaussian management mechanism, and a 3D model with a complex topological structure and rich details is generated. According to the method, the generation speed and quality are remarkably improved, and the method is suitable for real-time 3D content generation in the fields of game development, virtual reality and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional model generation, and in particular to a three-dimensional model generation method based on two-stage hybrid representation 3DGS. BACKGROUND

[0002] In recent years, the field of artificial intelligence generated content (AIGC) has made significant progress, with input modalities including text, images, videos, audio, and three-dimensional data. Among them, three-dimensional data, as the visual modality closest to the real world, carries a wealth of information. Three-dimensional content generation not only has important academic value and practical significance, but also faces enormous technical challenges. The generation of 3D models is the core of computer graphics and has been a research hotspot for decades. With the advent of advanced neural representations and generation models, the field of 3D content generation is rapidly developing, enabling the creation of increasingly high-quality and diverse 3D models.

[0003] Compared with 2D image generation, 3D content generation faces unique challenges such as data scarcity, complex representation, and large computational load, while also providing new opportunities for the study of generation models. Early 3D generation methods mainly relied on parameterized models and traditional computer graphics techniques, which were limited by their expressive ability and generalization. Rule-based procedural generation methods were widely used in game scene generation. These methods generate 3D content by combining predefined syntax rules and parameterized components, with high controllability but lack of creativity and diversity. As the demand for content diversity and realism increases in application scenarios, data-driven generation methods have gradually replaced traditional rule-based methods.

[0004] 3DGS splatting (3D Gaussian Splatting, 3DGS) as a new 3D representation method, with its explicit, efficient and high-quality rendering characteristics, has quickly become a research hotspot in the field of three-dimensional generation. Compared with traditional NeRF (Neural Radiance Fields) methods, 3DGS parameterizes the scene through explicit Gaussian distribution, avoiding the computational complexity of implicit decoding, while supporting real-time rendering and efficient editing. The core advantage of 3DGS lies in its adaptive model complexity - during training, the number of Gaussian points will be dynamically increased or decreased according to the complexity of the scene, ensuring accurate representation in detail areas and avoiding resource waste in blank areas. In the past few years, 3DGS has made significant breakthroughs in generation modeling, multi-modal conditional control, and structured representation optimization, promoting the rapid generation and interactive editing of high-quality 3D content.

[0005] Although existing text or image 3D generation models have made significant progress in generation speed and quality, they still face a series of key challenges. From geometric consistency and texture details to computational efficiency and physical plausibility, these issues constrain the performance of generation models in practical applications. Multi-view inconsistency is a core bottleneck of current generation methods, although some methods introduce multi-view diffusion priors or geometric consistency constraints, the generated 3D models may still have problems such as texture flickering, structural drift, or semantic ambiguity. For example, SDS-based methods rely on 2D diffusion models for optimization, but since the diffusion model itself lacks strict 3D understanding, the rendering results of different views may contradict each other. Even with multi-view diffusion models such as MVDream, the generation of complex topological structures (such as interleaved objects, slender structures) still cannot guarantee global consistency. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of the prior art and provide a two-stage hybrid representation 3DGS-based three-dimensional model generation method. This method quickly generates a basic 3D structure using a pre-trained 3D diffusion model and converts it to an initial 3DGS representation. Then, the geometry and texture of the 3DGS are refined, and a dynamic Gaussian management mechanism is used for efficient optimization to generate 3D models with complex topological structures and rich details, suitable for 3D content generation in fields such as game development, virtual reality, and film production.

[0007] The purpose of the present application is achieved by the following technical solution: a two-stage hybrid representation 3DGS-based three-dimensional model generation method, comprising the following steps: (1) Design a coarse-grained 3DGS generation model using a pre-trained 3D diffusion model Stable Diffusion-2-1-Base to obtain a coarse-grained 3DGS generation model. This model can generate a 128-resolution 3D model 3DGS representation from a single image or a single text, and export its mesh representation; (2) Design a multi-view consistent fractional distillation sampling loss MVS-SDS. By introducing a neighborhood consistency weighted loss on top of the fractional distillation sampling loss SDS, the MVS-SDS loss is obtained, which makes the rendering results of different views consistent in semantics and structure; (3) Design a geometry and texture decoupling module, which includes a geometry generation module based on a conditional diffusion model and a texture synthesis module. The output geometry and texture content of the 3DGS generation model in step (1) are deconstructed as input to the subsequent optimization module; (4) Based on the MVS loss in step (2), a 3DGS optimization module is constructed, which can optimize the 3DGS representation input in step (3) to a 1024-resolution 3DGS representation, and finally output the Mesh representation of the 3D model.

[0008] Specifically, in step (1), the 3DGS generation model used employs a series of 3DGS to represent the scene, with each 3DGS containing its center location. Color, opacity Covariance To facilitate optimization, the covariance matrix is ​​decomposed into vectors for scaling. and quaternions used to represent rotations Therefore, the overall 3DGS scene is represented as follows: ; When calculating the color C(r) of the corresponding pixel rendered by ray r, 3DGS maps it to the 2D pixel space using a sputtering algorithm, as follows: ; Where N is the number of 3DGS along ray r. This represents the color of the i-th 3DGS element. It is the offset between the center of 3DGS and the pixel; G represents the spatial distribution of 3DGS.

[0009] Specifically, in step (1), a coarse-grained 3DGS generation model is obtained using the pre-trained 3D diffusion model Stable Diffusion-2-1-Base, which can generate a 3DGS representation of a 128-resolution 3D model from a single image or a single piece of text, and derive its mesh representation. First, the visual feature vector is extracted from the input image or text using the image-text contrast pre-trained model CLIP, and then input into the pre-trained 3D implicit function generation model Shap-E to obtain the initial mesh. The Shap-E model first trains an encoder to map 3D data to a latent space of implicit neural representations, using an MLP network to parameterize and model shape and texture. Then, a Transformer-based 3DGS diffusion model is trained, capable of learning to generate new 3D representations in the latent space. This model achieves conditional generation from text to 3D through a cross-attention mechanism while preserving unconditional generation. The Shap-E model is then fine-tuned on the Objaverse dataset to obtain higher-resolution 3D representations. The Transformer-based 3DGS diffusion model uses a mesh. Initialize a specific 3DGS point , grid Vertex as the center coordinate of the 3DGS function According to 3DGS central position Perform Poisson reconstruction and apply it to 3DGS. Export as Grid ; and Representing grids respectively The colors of the vertices and mesh vertices; colors From the vertex The surrounding 3DGS colors The obtained mesh is represented as follows: Where V represents the grid. The number of triangles in the diagram; Indicates the formation of a grid The coordinates of the three vertices of the mesh are obtained; the resulting mesh representation is used for subsequent decoupling of 3D model geometry and texture, as well as export to other applications for further editing.

[0010] Specifically, in step (2), the MVS loss is expressed as follows: It is the total loss function. It is the basic SDS loss introduced in DreamFusion. It is a neighborhood consistency loss. It is a hyperparameter that controls the intensity of neighborhood consistency loss; It is a latent variable of noise. It's real noise. y is the noise predicted by the diffusion model, w(t) is the latent variable of the input text prompt, and w(t) is the weight corresponding to time step t. The neighborhood consistency loss establishes pixel-level constraints among the rendered images of N views, establishes pixel correspondence in the corresponding pixels through 3D geometric projection, and makes the pixels similar in color and semantics by matching the local neighborhood of the pixels. It is pixel-level matching based on projection geometry, where p is the pixel coordinate of the reference viewpoint k. It is the set of visible viewpoints of p. It is the projection function from viewpoint k to j; semantic neighborhood term In China, The neighborhood-weighted similarity constraint, where It is a neighborhood of p. is a feature extractor of a pre-trained visual encoder CLIP-ViT, is a cosine distance of feature similarity, is an adaptive weight function, represents the color standard deviation of the local window, is the gradient of the depth map, is a normalization factor; the adaptive weight can reduce the weight at the texture edge and the geometric discontinuous area to avoid false matching.

[0011] Further, the geometry and texture decoupling module in step (3) is specifically: generating an initial Mesh representation from the 3D GS model through Poisson reconstruction, and retaining the vertex position and face patch topology as its geometric representation; that is, obtaining a texture representation, the texture representation reversely projects the rendered RGB image to the mesh surface and bakes it as a texture map; by uniformly selecting 8 azimuth angles and 3 elevation angles, plus top view and bottom view to render the corresponding RGB image, each pixel in the RGB image is reversely projected to the texture image based on the UV coordinates, and the pixels with a camera space z-axis and a surface normal cosine value less than 0.1 are excluded to avoid texture image blur at the mesh boundary; the reversely projected texture image is used as the model texture initialization of the next optimization stage.

[0012] Further, when rendering the image, the radius of the camera pose used is 3.5 to 5.5, the azimuth angle is -180 degrees to 180 degrees, and the elevation angle is 30 degrees to 150 degrees.

[0013] Further, step (4) is specifically: using a pre-trained diffusion model Stable Diffusion-2-1-Base as the 3D model prior information obtained from the 2D image in the optimization stage, using the MVS loss in step (2) for training, so that the generated 3D asset has 3D consistency provided by the 3D generation model and fine details from the 2D diffusion model; first, based on the input feature vector y and the given camera pose, a random noise level i is given, and random noise is added to to generate a noise image , then according to the fractional distillation sampling loss SDS, the difference between the randomly generated noise and the real noise predicted by the model is calculated, and the gradient direction of the neighborhood consistency loss in the MVS loss is used to optimize the 3DGS.

[0014] The beneficial effects of the present application are: 1. A new two-stage 3DGS generation framework is proposed. Compared with single-stage 3D generation methods, this method combines the advantages of generation effect based on 2D prior optimization model and the advantages of generation speed based on native 3D model, and can better balance the generation efficiency and generation effect of 3D model, and quickly generate high-quality 3D model.

[0015] 2. A new MVS-SDS loss is designed based on SDS loss, which introduces multi-view consistent information in the optimization process, constrains the consistency of the generated model in different views, and has better visual effect.

[0016] 3. The 2D prior-based optimization model and the native 3D-based generation model are innovatively combined and applied to the field of 3D generation, and competitive results are achieved, which is a 3DGS model suitable for the field of 3D generation, efficient and capable of generating high-quality 3D model. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is the overall flowchart of the present application; Figure 2 is a two-stage 3DGS model structure diagram of the present application; Figure 3 is a model generation effect diagram of the present application. DETAILED DESCRIPTION

[0018] The specific embodiments of the present application will be further described in detail below in combination with the drawings and specific embodiments. The following drawings are used to illustrate the present application, but not to limit the scope of the present application.

[0019] Example 1 Please refer to Figure 1 , Figure 2 , Figure 1 is a three-dimensional model generation method algorithm flowchart based on two-stage hybrid representation 3DGS provided by the present application. It includes the following steps: Step 1: design a relatively coarse-grained 3D generation method, use a pre-trained 3D diffusion model to generate a basic 3D structure, and derive its Mesh representation; The 3D generation model used in step 1 uses a series of 3DGS to represent the scene, each 3DGS contains the center position , color, opacity and covariance . In order to facilitate optimization, the covariance matrix is decomposed into a vector for scaling and a quaternion for representing rotation. Therefore, the 3DGS is represented as: When calculating the color C(r) of the corresponding pixel rendered by ray r, 3DGS maps it to the 2D pixel space using a sputtering algorithm, as follows: Where N is the number of 3DGS along ray r. This represents the color of the i-th 3DGS element. It is the offset between the center of the 3DGS and the pixel. G represents the spatial distribution of the 3DGS.

[0020] Coarse-grained 3D generative models first extract visual feature vectors from the input image or text using a pre-trained CLIP model, and then input these vectors into a pre-trained 3D diffusion model, Shap-E, to obtain an initial mesh. Shap-E first trains an encoder to map 3D data to a latent space of implicit neural representations, parameterized by an MLP network, enabling flexible modeling of shape and texture. Subsequently, a Transformer-based diffusion model is trained to learn and generate new 3D representations in the latent space. Conditional generation from text to 3D is achieved through a cross-attention mechanism, while retaining the flexibility of unconditional generation. The Shap-E model used was fine-tuned on the Objaverse dataset to better learn higher-resolution 3D representations.

[0021] Use grid Initialize 3DGS , grid Vertex as the center coordinate of the 3DGS function According to 3DGS central position Perform Poisson reconstruction and apply it to 3DGS. Export as Grid . and Representing grids respectively The colors of the vertices and mesh vertices. (Color) From the vertex The surrounding 3DGS colors The obtained mesh is represented as follows: Where V represents the grid. The number of triangles in the diagram. Indicates the formation of a grid The coordinates of the three vertices of the mesh. The resulting mesh representation facilitates subsequent decoupling of 3D model geometry and texture, as well as exporting it to other applications for further editing; Step 2: Design a MVS loss that introduces a neighborhood consistency weighted loss on top of the SDS loss to ensure that the rendered results from different views remain consistent in semantics and structure.

[0022] The MVS loss is shown as follows: is the total loss function, is the base SDS loss introduced in DreamFusion, is the neighborhood consistency loss, is a hyperparameter that controls the strength of the neighborhood consistency loss. is the latent variable of the noise, is the real noise, is the predicted noise by the diffusion model, y is the input text prompt latent variable, and w(t) is the weight corresponding to time step t. The neighborhood consistency loss establishes pixel-level constraints between the rendered images from N views, establishes pixel correspondence through 3D geometric projection in the corresponding pixels, and matches the local neighborhood of the pixels in color and semantics. is the pixel-level matching based on projection geometry, p is the pixel coordinate of the reference view k, is the set of visible views of p, is the projection function from view k to j. Semantic neighborhood item where the neighborhood weighted similarity constraint of is introduced, where is the neighborhood of p, is the feature extractor of the pre-trained visual encoder CLIP-ViT, is the cosine distance of feature similarity, is an adaptive weight function, represents the color standard deviation of the local window, is the gradient of the depth map, is the normalization factor. The adaptive weight can reduce the weight in the texture edge and geometric discontinuous area to avoid false matching.

[0023] Step 3: Design a geometry and texture decoupling module to deconstruct the output geometry and texture content of the 3D GS model in step 1 as input for the subsequent optimization module. The initial Mesh generated from the 3D GS through Poisson reconstruction retains its vertex positions and face topology as its geometric representation; that is, a texture representation is obtained, which reversely projects the rendered RGB image onto the mesh surface and bakes it into a texture map. By uniformly selecting 8 azimuth angles and 3 elevation angles, plus top and bottom views, the corresponding RGB images are rendered, each pixel in the RGB image is reversely projected onto the texture image based on the UV coordinates, and the pixels with a normal cosine value less than 0.1 in the z direction of the camera space are excluded to avoid unstable projection at the mesh boundary, that is, the texture image is blurred; the reversely projected texture image is used as the model texture initialization of the next optimization stage.

[0024] Step 4: A 3D GS optimization module is constructed based on the MVS loss, which can optimize the input 3D GS representation into a higher resolution 3D GS representation, and finally output the Mesh representation of the 3D model.

[0025] A pre-trained diffusion model Stable Diffusion-2-1-Base is used as the 3D model prior information obtained from the 2D image in the optimization stage, which is trained using the MVS loss in step 2 to ensure that the generated 3D asset has good 3D consistency provided by the 3D generation model and fine details from the 2D diffusion model. First, based on the input feature vector y and the given camera pose, a random noise level i is given, and a random noise is added to the image to generate a noisy image Then, the difference between the randomly generated noise and the model-predicted real noise is calculated according to the SDS loss, and the gradient direction of the neighborhood consistency loss in the MVS loss is used to optimize the 3D GS.

[0026] The resolution of the rendered image in the optimization process is 1024x1024, and the number of iterations is 5000. When rendering the image, the radius of the camera pose used is 3.5 to 5.5, the azimuth angle is -180 degrees to 180 degrees, and the elevation angle is 30 degrees to 150 degrees. In the optimization stage using the geometric constraint Gaussian, the learning rate of the position of the 3D GS is . The final 3D model can be generated in only 1 minute per optimization process.

[0027] The final effect of the present application is to input an image or a text, and the text prompt is: a high-resolution blue and white porcelain image; and then generate a 3D model described by the image or the text, as shown in Figure 3 , that is, a generated high-quality blue and white porcelain image is obtained.

[0028] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.

Claims

1. A three-dimensional model generation method based on two-stage hybrid representation of 3D GS, characterized in that, The method comprises the following steps: (1) a coarse-grained 3DGS generation model is designed, a pre-trained 3D diffusion model Stable Diffusion-2-1-Base is used to obtain a coarse-grained 3DGS generation model, which can generate a 128-resolution 3D model from a single image or a single text, and export the mesh representation of the 3DGS representation; (2) a multi-view consistent fractional distillation sampling loss MVS-SDS is designed, which introduces a neighborhood consistency weighted loss on the fractional distillation sampling loss SDS to obtain the MVS-SDS loss, so that the rendering results under different views are consistent in semantics and structure; (3) a geometry and texture decoupling module is designed, which includes a geometry generation module and a texture synthesis module based on a conditional diffusion model, and the output geometry and texture content of the 3DGS generation model in step (1) are deconstructed as the input of the subsequent optimization module; (4) a 3DGS optimization module is constructed based on the MVS loss in step (2), which can optimize the 3DGS representation input in step (3) to a 1024-resolution 3DGS representation, and finally output the mesh representation of the 3D model. 2.The method of claim 1, wherein, In step (1), the 3DGS generation model used represents the scene using a series of 3DGSs, each containing a center position , color, opacity , and covariance ; for ease of optimization, the covariance matrix is decomposed into a vector for scaling and a quaternion for representing rotation ; thus, the overall 3DGS scene representation is: ； When calculating the color C(r) of the corresponding pixel rendered by the light ray r, the 3DGS is mapped into the 2D pixel space by the ray marching algorithm, and the calculation is as follows: ； where N is the number of 3DGS along the ray r, represents the color of the ith 3DGS, is the offset between the center of the 3DGS and the pixel; G represents the 3DGS spatial distribution. 3.The method of claim 2, wherein, In step (1), a coarse-grained 3DGS generation model is obtained using the pre-trained 3D diffusion model Stable Diffusion-2-1-Base. This model can generate a 3DGS representation of a 128-resolution 3D model from a single image or a single piece of text, and derive its mesh representation. First, the visual feature vector is extracted from the input image or text using the image-text contrast pre-trained model CLIP. This vector is then input into the pre-trained 3D implicit function generation model Shap-E to obtain the initial mesh. The Shap-E model first trains an encoder to map 3D data to a latent space of implicit neural representations, using an MLP network to parameterize and model shape and texture. Then, a Transformer-based 3DGS diffusion model is trained, capable of learning to generate new 3D representations in the latent space. This model achieves conditional generation from text to 3D through a cross-attention mechanism while preserving unconditional generation. The Shap-E model is then fine-tuned on the Objaverse dataset to obtain higher-resolution 3D representations. The Transformer-based 3DGS diffusion model uses a mesh. Initialize a specific 3DGS point , grid Vertex as the center coordinate of the 3DGS function According to 3DGS central position Perform Poisson reconstruction and apply it to 3DGS. Export as Grid ; and Representing grids respectively The colors of the vertices and mesh vertices; colors From the vertex The surrounding 3DGS colors The obtained mesh is represented as follows: ； where V represents the number of triangles in the mesh represents the coordinates of the three vertices of the Mesh forming the mesh The resulting mesh representation is such that subsequent 3D model geometry and texture decoupling and export to other applications for further editing can be performed.​ 4. The method of claim 1, wherein the method is characterized by, In the step (2), the MVS loss is shown as follows: ； ； ； ； ； ； ； is the total loss function, is the base SDS loss introduced in DreamFusion, is the neighborhood consistency loss, is a hyper-parameter to control the strength of the neighborhood consistency loss; is the latent variable of the noise, is the real noise, is the predicted noise by the diffusion model, y is the input text prompt latent variable, w(t) is the weight corresponding to time step t; the neighborhood consistency loss establishes pixel-level constraints between the rendered images of N views, establishes pixel correspondence through 3D geometric projection in the corresponding pixels, and makes them similar in color and semantics by matching the local neighborhood of the pixels; is the pixel-level matching based on projection geometry, p is the pixel coordinate of the reference view k, is the set of visible views of p, is the projection function from view k to j; semantic neighborhood item , the neighborhood weighted similarity constraint of is introduced, where is the neighborhood of p, is the feature extractor of the pre-trained visual encoder CLIP-ViT, is the cosine distance of feature similarity, is an adaptive weight function, represents the color standard deviation of the local window, is the gradient of the depth map, is a normalization factor; The adaptive weight can reduce the weight in the texture edge and the geometric discontinuous area to avoid false matching.

5. The method of claim 1, wherein the method further comprises: The geometry and texture decoupling module in the step (3) is as follows: the initial mesh representation is generated from the 3DGS model by Poisson reconstruction, and the vertex position and face topology are retained as its geometry representation; that is, the texture representation is obtained, which inversely projects the rendered RGB image onto the mesh surface and bakes it into a texture map; by uniformly selecting 8 azimuth angles and 3 elevation angles, plus the top view and the bottom view, the corresponding RGB image is rendered, each pixel in the RGB image is inversely projected onto the texture image based on the UV coordinates, and the pixels with a camera space z-axis and a surface normal cosine value less than 0.1 are excluded to avoid texture image blur at the mesh boundary; the inversely projected texture image is used as the model texture initialization for the next optimization stage.

6. The method of claim 5, wherein the two-stage hybrid representation based 3D model generation method is characterized by, When rendering the image, the radius of the camera pose used is 3.5 to 5.5, the azimuth angle is -180 degrees to 180 degrees, and the elevation angle is 30 degrees to 150 degrees.

7. The method of claim 1, wherein the method further comprises: The step (4) is specifically: using a pre-trained diffusion model Stable Diffusion-2-1-Base as prior information of a 3D model obtained from a 2D image in an optimization stage, training using the MVS loss in the step (2) so that the generated 3D asset has 3D consistency provided by the 3D generation model and fine details from the 2D diffusion model; first, based on the input feature vector y and the given camera pose, a random noise level i is given, random noise is added to generate a noisy image , then according to the score distillation sampling loss SDS, the difference between the randomly generated noise and the model predicted real noise is calculated, and the gradient direction of the neighborhood consistency loss in the MVS loss is used to optimize the 3DGS.

Images