Three-dimensional object generation method and system based on grid-guided Gaussian splashing

Abstract

The invention discloses a three-dimensional object generation method and system based on grid-guided Gaussian splashing. The method comprises the following steps of: 1, recombining an input text into a structured text description by using instructive text reconstruction operation; 2, generating a three-dimensional grid through structured text description; 3, initializing a three-dimensional Gaussian splashing model by using the three-dimensional grid; 4, optimizing parameters of the three-dimensional Gaussian splashing model through a distillation fraction sampling technology; and 5, performing fine trimming on the three-dimensional Gaussian splash model after basic optimization through grid driving, and outputting a high-precision and high-fidelity three-dimensional object. According to the method, the generation quality and geometric rationality of the three-dimensional object are remarkably improved, the consumption of computing resources is greatly reduced, meanwhile, the diversity of generation results is enriched, and the scene adaptation and function expansion capabilities are enhanced. The method has comprehensive breakthrough in the aspects of generation quality, resource efficiency, result diversity, expansion suitability and the like, is very wide in application range, and has very wide market prospects and social values.

Description

Technical Field

[0001] This invention belongs to the field of textural 3D object technology under artificial intelligence computer vision, and provides a method and system for generating 3D objects based on mesh-guided Gaussian splashing. Background Technology

[0002] Text-based 3D object generation technology is gradually becoming a significant breakthrough. It allows non-professional users to quickly generate detailed 3D objects solely based on natural language descriptions. This technology significantly lowers the technical barrier to traditional manual 3D modeling and shows broad application prospects in various fields such as game animation, augmented reality, virtual reality, and medicine. Unlike traditional methods that require extensive domain knowledge and human intervention, this technology truly democratizes the creation of complex 3D objects. The realistic simulation data it generates greatly deepens human understanding of complex 3D structures. Despite these advantages, significant challenges remain. Low-quality and limited-quantity 3D datasets lead to poor output quality from directly trained 3D diffusion models; optimization-based methods are prone to geometric distortion, generating geometrically flawed 3D objects; computational costs are high during generation; and there are issues such as insufficient diversity and poor scalability.

[0003] In summary, the challenge of text-to-3D object generation lies in how to generate detailed 3D objects while maintaining good computational efficiency. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a three-dimensional generation method and system based on mesh-guided Gaussian splashing.

[0005] A 3D generation method based on mesh-guided Gaussian splashing is proposed, and the technical solution adopted to solve its technical problems is as follows:

[0006] Step (1): Obtain a structured text description from the input text: Use the guided text reconstruction operation to reorganize the input text into a structured text description, which helps in subsequent image generation based on the diffusion model.

[0007] Step (2): Generate a 3D mesh through structured text description: Use the structured text description as input to the text-based image diffusion model to obtain a single view that is highly consistent with the semantics. Then, generate multi-view images from different perspectives through the multi-view diffusion model. Input the multi-view images into the mesh generator to obtain a 3D mesh, providing priors for the initialization of the 3D Gaussian splash model.

[0008] Step (3): Initialize the 3D Gaussian splash model with a 3D mesh: Extract the vertex coordinates and color information of the 3D mesh to generate a point cloud. To enrich the details in the point cloud, add auxiliary point clouds. Then use all the point clouds to initialize the corresponding parameters in the 3D Gaussian splash model for subsequent optimization.

[0009] Step (4) Basic optimization of the three-dimensional Gaussian splash model: The parameters of the three-dimensional Gaussian splash model are optimized by distillation fraction sampling technology, which effectively improves the precision of the three-dimensional object.

[0010] Step (5): Refine the basic optimized 3D Gaussian splash model using mesh-driven method: Extract the 3D mesh from the optimized 3D Gaussian splash model, use the extracted 3D mesh as a guide to refine and optimize the 3D Gaussian splash model again, thereby significantly improving the model quality and outputting a high-precision, high-fidelity 3D object.

[0011] Furthermore, step (1) is as follows:

[0012] First, the input text is restructured using guided text reconstruction techniques. Convert to structured text description Effectively guides subsequent text-based image diffusion models The image generation process. This step ensures the generation of a single view. To minimize background interference, thereby improving the quality and fidelity of subsequent 3D mesh generation, single view The formula for generating it is:

[0013]

[0014] Furthermore, step (2) is as follows:

[0015] Background removal method for single view using rembg Predicting object masks, separating target objects, and obtaining a single view of the focused object. A single view with the background removed. Input to multi-view diffusion model By analyzing the model Adjust absolute elevation angle (From +20° to −10°) and relative azimuth (Starting at 30°, with increments of 60°), generate Zhang's multi-view images:

[0016]

[0017] in, This represents the generated set of multi-view images.

[0018] Multi-view images When the input mesh generator generates voxel representations, each multi-view image Its corresponding camera parameters via image encoder Encoding is performed. The encoded multi-view features are then processed by a three-plane decoder. Decoded into a three-dimensional voxel shape representation:

[0019]

[0020] in, This represents the reshaping and deconvolution operations. To characterize the generated three-dimensional voxel shape, It is a learnable three-plane token.

[0021] The FlexiCubes module was ultimately adopted. Characterization of three-dimensional voxel shapes Convert to high-quality 3D mesh :

[0022]

[0023] in, Includes vertex coordinates With corresponding color attributes .

[0024] Furthermore, the training process of the mesh generator is divided into two stages:

[0025] First, optimize the generation of three-dimensional voxel shape representations. The generation process utilizes a multi-dimensional constraint loss mechanism involving view rendering, segmentation masking, and perceptual similarity. accomplish:

[0026]

[0027] in, , They represent the first Rendered images from different perspectives compared to real images. , To predict the mask and the actual mask, In order to perceive similarity, , This represents the weighting parameter.

[0028] Next, optimize the 3D mesh. FlexiCubes modules are calculated using depth maps, normal map constraints, and regularization terms. loss :

[0029]

[0030] in, and These are the rendered depth map and the actual depth map, respectively. , Corresponding to the predicted normal map and the true normal map , and Represents the weight parameters, symbol Represents element-wise multiplication, regularization term Used to prevent overfitting.

[0031] Furthermore, step (3) is as follows:

[0032] The initialization process is divided into several stages. First, the 3D mesh is... The vertex coordinates and color attributes are converted into the original point cloud, denoted as . ,in Let the vertex coordinates be... This corresponds to the color. This conversion allows the data to be seamlessly integrated into subsequent 3D Gaussian splash models.

[0033] To further enrich the details of the point cloud, the original point cloud was surrounded by... Randomly sample 1 million points within the axis-aligned bounding box to generate a random point cloud. This step introduces finer details and denser representations for subsequent optimization.

[0034] Subsequently, from random point clouds Selection from original point cloud Distance less than threshold The points form an auxiliary point cloud. . use To achieve efficient nearest neighbor search:

[0035] in, This indicates the nearest neighbor search algorithm. This represents a binary search tree.

[0036] This selection ensures the addition of auxiliary point clouds. Fits closely to the original mesh geometry. Select point color. Color of nearest neighbor Random noise is added to enhance realism:

[0037]

[0038] Among them, noise intensity , Sampling from a uniform distribution.

[0039] Next, the present invention will use the original point cloud With auxiliary point cloud The points are then merged to form a complete point cloud. :

[0040]

[0041] Finally, the merged complete point cloud Used to initialize the parameters of the 3D Gaussian splash model

[0042] .Location With color Each by and Settings; Opacity Initially 0.1; covariance according to The nearest neighbor distance for each point in the point cloud is obtained, and this nearest neighbor distance is calculated using a nearest neighbor search algorithm. This initialization lays the foundation for efficient and accurate optimization.

[0043] Furthermore, step (4) is as follows:

[0044] The initialized 3D Gaussian splash model was optimized using fractional distillation sampling (SDS) technology. The rendering equation for Gaussian splash is: ,in Representing camera parameters, optimization is guided by SDS gradients:

[0045]

[0046] in, For the weight function, Representing the diffusion model Predictive noise, The actual injected noise, It is a parameter gradient operator, For time step t and noise Mathematical expectation It is an image At time step Images with added noise Indicates time step, This indicates that the SDS loss is related to the parameter. The gradient.

[0047] Furthermore, step (5) is as follows:

[0048] To further improve detail and fidelity on top of basic optimization, this invention extracts a 3D mesh from the optimized 3D Gaussian splash model and refines the 3D Gaussian splash model accordingly.

[0049] First, the three-dimensional space is divided into 16 parts. 3 Furthermore, for overlapping voxel blocks, Gaussians whose centers are located outside the voxel blocks are removed; 8 operations are performed within each voxel block. 3 The dense grid queries ultimately form a global 128 3 A dense mesh was then used. The mesh point locations were subsequently calculated based on the optimized 3D Gaussian splash model. density :

[0050]

[0051] in, The Gaussian set corresponding to the optimized dense mesh. , , They represent the first The opacity, covariance, and position of a Gaussian. This indicates the location of mesh points. The Marching Cubes algorithm is used to extract the surface mesh with a threshold of 1. Next, textures are generated for the surface mesh: the texture map is initialized, UV coordinates are unfolded, and vertices are mapped to texture space; textures are rendered from 14 perspectives (8 azimuth angles, 3 elevation angles, and top and bottom), and the results are projected into UV space. Finally, SDS optimization is performed again, guided by the textured mesh, significantly improving model quality and detail, outputting a high-precision, high-fidelity 3D object.

[0052] Furthermore, the present invention also provides a 3D generation system based on mesh-guided Gaussian splashing, the system comprising the following modules:

[0053] Structured text description module: It uses guided text reconstruction operations to reorganize the input text into a structured text description, which helps in subsequent diffusion-based image generation.

[0054] 3D Mesh Module: The structured text description is used as input to the text-generated image diffusion model to obtain a single view that is highly consistent with the semantics. Then, a multi-view diffusion model is used to generate multi-view images from different perspectives. The multi-view images are then input into the mesh generator to obtain a 3D mesh.

[0055] The 3D Gaussian splash model initialization module extracts vertex coordinates and color information from the 3D mesh to generate a point cloud. To enrich the details in the point cloud, auxiliary point clouds are added. Then, all the point clouds are used to initialize the corresponding parameters in the 3D Gaussian splash model.

[0056] Optimization module: The parameters of the 3D Gaussian splash model are optimized by distillation fraction sampling technology, which effectively improves the precision of the 3D object.

[0057] Refinement Module: Extracts a 3D mesh from the optimized 3D Gaussian splash model, uses the extracted 3D mesh as a guide to refine and optimize the 3D Gaussian splash model again, and outputs a high-precision, high-fidelity 3D object.

[0058] Beneficial effects of this invention:

[0059] This invention proposes a high-quality text-based method for generating 3D objects, addressing issues such as low quality, geometric inconsistencies, high computational costs, insufficient diversity, and poor scalability. Through algorithm optimization and an innovative model architecture, this invention significantly improves the quality and geometric plausibility of generated 3D objects, substantially reduces computational resource consumption, enriches the diversity of generated results, and enhances the method's scene adaptability and functional scalability. This is particularly important for scenarios requiring high-precision modeling and efficient iteration, such as digital twin construction, virtual simulation experiments, and high-precision film and television modeling. Due to the comprehensive breakthroughs in generation quality, resource efficiency, result diversity, and scalability, this invention has a very wide range of applications. Besides traditional film and television games and industrial design, it can be extended to metaverse content creation, intelligent building pre-visualization, digital restoration of cultural relics, and many other fields, possessing a very broad market prospect and social value. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the overall framework of the method of the present invention.

[0061] Figure 2 It is the single-view induced mesh generation module constructed in the method of this invention.

[0062] Figure 3 It is the mesh-guided three-dimensional Gaussian splash model initialization module constructed in the method of this invention.

[0063] Figure 4 This is the two-stage three-dimensional Gaussian splash optimization module constructed in the method of this invention. Detailed Implementation

[0064] The detailed parameters of the present invention will be further described below.

[0065] This invention mainly includes: 1. Reorganizing input text through guided text reconstruction: This invention first reorganizes the input text into a structured text description through guided text reconstruction, effectively guiding the subsequent image generation process based on the diffusion model. 2. 3D mesh generation: This invention inputs the structured text description into the text-generated image diffusion model to obtain a semantically aligned single view. Then, a 3D mesh is obtained through a multi-view diffusion model and a mesh generator, providing a reliable prior for the 3D Gaussian splash model. 3. Initialization of the 3D Gaussian splash model: This invention extracts the vertex coordinates and colors of the generated 3D mesh and converts it into a point cloud. To further enrich the point cloud details, this invention adds auxiliary point clouds to the original point cloud for subsequent optimization. Then, the fused point cloud is used to initialize the corresponding parameters in the 3D Gaussian splash model for subsequent optimization. 4. Basic optimization of the 3D Gaussian splash model: This invention uses fractional distillation sampling technology to optimize the parameters of the 3D Gaussian splash model, effectively improving the refinement of the 3D object. 5. Mesh-driven 3D Gaussian splash model refinement: To further improve detail and fidelity on the basis of basic optimization, this invention extracts meshes from the optimized 3D Gaussian splash model and refines the 3D Gaussian splash model with textured meshes as a guide, thereby significantly improving model quality and detail and outputting high-precision, high-fidelity 3D objects.

[0066] like Figure 1 As shown, the present invention provides a method and system for generating three-dimensional objects based on mesh-guided Gaussian splashing, specifically a high-quality text-described three-dimensional object generation framework.

[0067] Step (1) uses guided text reconstruction to reconstruct the input text into a structured text description, which helps with subsequent diffusion-based image generation. For example... Figure 2 As shown, the details are as follows:

[0068] Enter text Structured text is obtained by reconstructing imperative text. The format for the instruction text reconstruction is "a ((full-body:2)) shot of a ((single:2)) [Input Text], isolated on graybackground, 4k, highly detailed". "a ((full-body:2))" forces the generation of a full-body shot with a weight of 2 to avoid generating only a partial shot. "((single:2))" forces the generation of a semantically correct, complete object with a weight of 2. "[Input Text]" represents the input text, and "4k" indicates a 4k resolution image.

[0069] Structured text helps to focus on the object when generating a single view later, and avoids generating complex backgrounds that may affect the generation of 3D objects.

[0070] Step (2) In this invention, the structured text description is input into the text-generated graph diffusion model to obtain a semantically aligned single view. Then, a three-dimensional mesh is obtained through a multi-view diffusion model and a mesh generator, providing a reliable prior for the three-dimensional Gaussian splash model, such as... Figure 2 As shown, the details are as follows:

[0071] The structured text T is input into the Wensheng graph diffusion model. Get a single view The diffusion model used for the textural image was stabilityai / stable-diffusion-xl-base-1.0, with a guidance scale of 7.5, 50 denoising steps, and an output resolution of 1024×1024. This resulted in a single view. Then, it is input into the multi-view diffusion model. Get a collection of multi-view images The multi-view synthetic diffusion model uses Zero123++ to generate six 320×320 view images at once. The absolute elevation angle φ of each image ranges from +20° to −10°, and the relative azimuth angle Θ starts at 30° and increments in 60° increments. This results in a multi-view image set. The input data is fed into a mesh generator to generate a 3D mesh. The training data for the mesh generator comes from the Objaverse dataset, sampled from a subset of approximately 270,000 complete and high-quality 3D objects. For each scene, 32 viewpoint images are rendered, with 6 randomly selected images as input and 4 images used for supervision to enhance robustness to different camera configurations. The mesh generator training consists of two phases: the first phase has a learning rate of 4.0 × 10⁻⁶. −4 Cosine annealing to 4.0×10 −5 Loss weight , The second-stage learning rate is 4.0 × 10⁻⁶. −5 Cosine annealing to 0 results in a loss weight. , , .

[0072] Step (3) In this invention, vertex coordinates and color information of the 3D mesh are extracted to generate a point cloud. To enrich the details in the point cloud, an auxiliary point cloud is added. The fused point cloud is then used to initialize the corresponding parameters in the 3D Gaussian splash model, facilitating subsequent optimization, such as... Figure 3 As shown, the details are as follows:

[0073] This invention will generate a mesh The vertex and color attributes are converted into the original point cloud, denoted as . ,in Let the vertex coordinates be... For corresponding colors. To further enrich the point cloud details, this invention surrounds... Randomly sample 1 million points within the axis-aligned bounding box to generate a random point cloud. Subsequently, this invention utilizes random point clouds... Selecting from the original point cloud Distance less than threshold The points constitute auxiliary . use Achieve efficient nearest neighbor search. ,in This indicates the nearest neighbor search algorithm. This represents a binary search tree. This selection ensures that newly added points fit closely to the original mesh geometry. Selected point color. Color by nearest neighbor Random noise is added to enhance realism. Noise intensity , The sample is taken from a uniform distribution. Next, the original point cloud is fused with the newly added point cloud to form a complete point cloud. Ultimately, the merged point cloud Used to initialize the parameters of the 3D Gaussian splash model .Location With color Each by and Settings; Opacity Initially 0.1; covariance according to The nearest neighbor distance for each point in the point cloud is obtained, and this nearest neighbor distance is calculated using a nearest neighbor search algorithm. This initialization lays the foundation for efficient and accurate optimization.

[0074] Step (4) The 3D Gaussian splash model renders 2D images from different perspectives using Gaussian splashing, and optimizes the parameters of the 3D Gaussian splash model using fractional distillation sampling technology, effectively improving the detail of the 3D object, such as... Figure 4 As shown, the details are as follows:

[0075] This invention uses fractional distillation sampling (SDS) technology to optimize the initialized 3D Gaussian splash model. The rendering equation for Gaussian splash is: ,in This represents the camera parameters. Optimization is guided by SDS gradients, and the optimization formula for SDS is: in, For the weight function, Representing the diffusion model Predictive noise, The actual injected noise, It is a parameter gradient operator, For time step t and noise Mathematical expectation It is an image At time step Images with added noise Indicates time step, This indicates that the SDS loss is related to the parameter. The gradient of the 3D Gaussian splash model is used for optimization. The stabilityai / stable-diffusion-2-1-base model is employed. The rendered image is first generated at a size of 1024×1024, then downsampled to a resolution of 512×512 to match the diffusion model input. The learning rate for each parameter is: position. 5×10 −5 Opacity 10 −2 ,color Encoding is performed using spherical harmonic functions with a learning rate of 1.25 × 10⁻⁶. −2 covariance Decomposed into scale and rotation, with learning rates of 10. −3 With 10 −2 .

[0076] Step (5) Extract the mesh from the optimized 3D Gaussian splash model. Use the extracted mesh as a guide to refine and optimize the 3D Gaussian splash model, thereby significantly improving the model quality and outputting high-precision, high-fidelity 3D objects, such as... Figure 4 As shown, the details are as follows:

[0077] This invention divides the three-dimensional space into 16 parts. 3 Furthermore, for overlapping voxel blocks, Gaussians whose centers are located outside the voxel blocks are removed; 8 operations are performed within each block. 3 The dense grid queries ultimately form a global 128 3 A dense mesh was then used. The location of each mesh point was subsequently calculated based on the optimized 3D Gaussian splash. density in For the optimized Gaussian set, , , These represent its opacity, covariance, and position, respectively. The MarchingCubes algorithm is used to extract the surface mesh with a threshold of 1. Next, textures are generated for the mesh: the texture map is initialized, UV coordinates are unfolded, and vertices are mapped to texture space. Textures are rendered from 14 perspectives: 8 azimuth angles, 3 elevation angles, and top and bottom views, and the results are projected into UV space. Finally, SDS optimization is performed again, guided by the textured mesh, significantly improving model quality and detail, outputting a high-precision, high-fidelity 3D object. The SDS optimization process consists of 800 iterations. The first 500 iterations have timestamps uniformly sampled between 0.02 and 0.98; the last 300 iterations are scaled down to 0.02–0.55 to focus on finer structural details. The entire 3D object generation process is completed in approximately 20 minutes on a single NVIDIA RTX 3090 GPU.

[0078] Table 1: Quantitative evaluation results on T3Bench compared to state-of-the-art methods

[0079] Methods Time (s) Single. w / Surr. Multi. Aver. Score Shap-E 27 25.5 23.1 13.6 20.7 DreamFusion 5400 24.4 24.6 16.1 21.7 ProlificDreamer 36000 49.4 44.8 35.8 43.3 RichDreamer 5400 48.6 43.1 28.4 40.0 DreamGaussian 300 19.8 14.1 10.9 14.9 GaussianDreamer 900 54.0 48.6 34.5 45.7 Ours 1200 55.4 46.5 46.8 49.6

[0080] Table 1 above shows the results at T 3 Quantitative comparison results were performed on Bench. The method of this invention achieved the highest average score across three metrics: Single., Single. with surrounding environment, and Multi., while also possessing a competitive 3D object generation speed, fully demonstrating its superior ability to generate high-quality, semantically consistent 3D objects.

[0081] Table 2: Guided Text Reconstruction (ITR) Ablation Experiments on T3Bench

[0082] Methods Single. w / Surr. Multi. Aver. Score w / o ITR 52.3 40.5 34.7 42.5 w / ITR 55.4 46.5 46.8 49.6

[0083] Table 2 above shows the results at T 3 Quantitative comparisons were performed on Bench. The guided text reconstruction operation used in this invention significantly improved the average scores of the three indicators—Single, Single with surrounding environment, and Multi—compared to those without guided text reconstruction, fully demonstrating the positive role of guided text reconstruction in improving the quality of the generated data.

[0084] Table 3: Two-stage 3D Gaussian splashing optimized ablation experiment on T3Bench

[0085] Methods Single. w / Surr. Multi. Aver. Score w / o Optimization 49.6 40.1 40.6 43.4 w / Optimization 55.4 46.5 46.8 49.6

[0086] Table 3 above shows the results at T 3Quantitative comparison results were performed on Bench. The two-stage three-dimensional Gaussian splash optimization operation used in this invention significantly improved the average scores of the three indicators—Single., Single. with surrounding environment (w / Surr.), and Multi.—compared to the operation without the two-stage three-dimensional Gaussian splash optimization operation, demonstrating the practical value and technical advantages of the two-stage three-dimensional Gaussian splash optimization operation.

[0087] Table 4: Mesh-driven ablation experiments on T3Bench

[0088] Methods Single. w / Surr. Multi. Aver. Score w / o Refinement 54.5 45.8 46.2 48.8 w / Refinement 55.4 46.5 46.8 49.6

[0089] Table 4 above shows the results in T 3 Quantitative comparisons were performed on the Bench. The mesh-driven optimization operation used in this invention significantly improved the average scores of the three metrics—Single, Single with surrounding environment, and Multi—compared to that without mesh-driven optimization, demonstrating the effectiveness of the mesh-driven optimization operation.

Claims

1. A three-dimensional generation method based on mesh-guided Gaussian splashing, characterized in that, Includes the following steps: Step (1): Obtain a structured text description from the input text: Use guided text reconstruction operations to reorganize the input text into a structured text description; Step (2): Generate a 3D mesh through structured text description: Use the structured text description as input to the text-based image diffusion model to obtain a single view that is highly consistent with the semantics. Then, generate multi-view images from different perspectives through the multi-view diffusion model. Input the multi-view images into the mesh generator to obtain a 3D mesh. Step (3): Initialize the 3D Gaussian splash model with a 3D mesh: Extract the vertex coordinates and color information of the 3D mesh to generate a point cloud. To enrich the details in the point cloud, add auxiliary point clouds. Then use all the point clouds to initialize the corresponding parameters in the 3D Gaussian splash model. Step (4) Perform basic optimization on the three-dimensional Gaussian splash model: optimize the parameters of the three-dimensional Gaussian splash model through distillation fraction sampling technology; Step (5): Refine the basic optimized 3D Gaussian splash model using mesh-driven method: Extract the 3D mesh from the optimized 3D Gaussian splash model, use the extracted 3D mesh as a guide to refine and optimize the 3D Gaussian splash model again, and output a high-precision, high-fidelity 3D object.

2. The three-dimensional generation method based on mesh-guided Gaussian splashing according to claim 1, characterized in that, Step (1) is as follows: First, the input text is restructured using guided text reconstruction techniques. Convert to structured text description Effectively guides subsequent text-based image diffusion models The process of generating a single view, single view The formula for generating it is: 。 3. The three-dimensional generation method based on mesh-guided Gaussian splashing according to claim 1, characterized in that, Step (2) is as follows: Background removal method for single view using rembg Predicting object masks, separating target objects, and obtaining a single view of the focused object. ; Remove the background from the single view Input to multi-view diffusion model Through the multi-view diffusion model Adjust absolute elevation angle and relative azimuth ,generate Zhang's multi-view images: in, This represents the generated set of multi-view images; Multi-view images When the input mesh generator generates voxel representations, each multi-view image Its corresponding camera parameters via image encoder Encoding is performed; the encoded multi-view features are then processed by a three-plane decoder. Decoded into a three-dimensional voxel shape representation: in, This represents the reshaping and deconvolution operations. To characterize the generated three-dimensional voxel shape, A learnable three-plane token; The FlexiCubes module was ultimately adopted. Characterization of three-dimensional voxel shapes Convert to high-quality 3D mesh : in, Includes vertex coordinates With corresponding color attributes .

4. The three-dimensional generation method based on mesh-guided Gaussian splashing according to claim 3, characterized in that, The training process of the mesh generator is divided into two stages: First, optimize the generation of three-dimensional voxel shape representations. The generation process utilizes a multi-dimensional constraint loss mechanism involving view rendering, segmentation masking, and perceptual similarity. accomplish: in, , They represent the first Rendered images from different perspectives compared to real images. , To predict the mask and the actual mask, In order to perceive similarity, , Indicates the weighting parameter; Next, optimize the 3D mesh. FlexiCubes modules are calculated using depth maps, normal map constraints, and regularization terms. loss : in, and These are the rendered depth map and the actual depth map, respectively. , Corresponding to the predicted normal map and the true normal map , and Represents the weight parameters, symbol Represents element-wise multiplication, regularization term Used to prevent overfitting.

5. The three-dimensional generation method based on mesh-guided Gaussian splashing according to claim 4, characterized in that, Step (3) is as follows: First, the three-dimensional mesh The vertex coordinates and color attributes are converted into the original point cloud, denoted as . ,in Let the vertex coordinates be... For the corresponding color; Surrounding the original point cloud Randomly sample 1 million points within the axis-aligned bounding box to generate a random point cloud. ; From random point cloud Selection from original point cloud Distance less than threshold The points form an auxiliary point cloud. ;use To achieve efficient nearest neighbor search: in, This indicates the nearest neighbor search algorithm. This represents a binary search tree; Select point color Color of nearest neighbor The additional random noise is generated using the following formula: Among them, noise intensity , Sampling from a uniform distribution; Next, the original point cloud With auxiliary point cloud The points are then merged to form a complete point cloud. : Finally, the merged complete point cloud Used to initialize the parameters of the 3D Gaussian splash model ;Location With color respectively by and Settings; Opacity Initially 0.1; covariance according to The nearest neighbor distance for each point in the cloud is obtained, and this nearest neighbor distance is calculated using the nearest neighbor search algorithm.

6. The three-dimensional generation method based on mesh-guided Gaussian splashing according to claim 5, characterized in that, Step (4) is as follows: The initialized 3D Gaussian splash model was optimized using fractional distillation sampling; the rendering equation for Gaussian splash is: ,in Representing camera parameters, optimization is guided by SDS gradients: in, For the weight function, Representing the diffusion model Predictive noise, The actual injected noise, It is a parameter gradient operator, For time step t and noise Mathematical expectation It is an image At time step Images with added noise Indicates time step, This indicates that the SDS loss is related to the parameter. The gradient.

7. A three-dimensional generation method based on mesh-guided Gaussian splashing according to claim 6, characterized in that, Step (5) is as follows: First, the three-dimensional space is divided into 16 parts. 3 Furthermore, for overlapping voxel blocks, Gaussians whose centers are located outside the voxel blocks are removed; 8 operations are performed within each voxel block. 3 The dense grid queries ultimately form a global 128 3 Dense grid; Subsequently, the grid point positions are calculated based on the optimized 3D Gaussian splash model. density : in, The Gaussian set corresponding to the optimized dense mesh. , , They represent the first The opacity, covariance, and position of a Gaussian. The grid point positions are indicated; the Marching Cubes algorithm is used to extract the surface grid with a threshold of 1; then, a texture is generated for the surface grid: the texture map is initialized, the UV coordinates are unfolded and the vertices are mapped to the texture space; the texture is rendered from 14 perspectives, including 8 azimuth angles, 3 elevation angles and top and bottom, and the results are projected into the UV space; Finally, SDS optimization is performed again, guided by the textured mesh, to output a high-precision, high-fidelity 3D object.

8. A three-dimensional generation method based on mesh-guided Gaussian splashing according to claim 6, characterized in that, The format of the instruction text reconstruction is "a ((full-body:2)) shot of a ((single:2)) [Input Text],isolated on gray background, 4k, highly detailed"; where a ((full-body:2)) means to force the generation of a full-body shot with a weight of 2, ((single:2)) means to force the generation of a semantically correct complete object with a weight of 2, [Input Text] means the input text, and 4k means the image quality is 4k resolution.

9. A three-dimensional generation system based on mesh-guided Gaussian splashing, characterized in that, The system includes the following modules: Structured text description module: It uses guided text reconstruction operations to reorganize the input text into a structured text description, which helps with subsequent image generation based on the diffusion model; 3D Mesh Module: The structured text description is used as input to the text-generated image diffusion model to obtain a single view that is highly consistent with the semantics. Then, a multi-view diffusion model is used to generate multi-view images from different perspectives. The multi-view images are then input into the mesh generator to obtain a 3D mesh. 3D Gaussian splash model initialization module: Extracts vertex coordinates and color information from the 3D mesh to generate point cloud. To enrich the details in the point cloud, auxiliary point cloud is added. Then, all the point cloud is used to initialize the corresponding parameters in the 3D Gaussian splash model. Optimization module: Optimizes the parameters of the 3D Gaussian splash model using distillation fraction sampling technology; Refinement Module: Extracts a 3D mesh from the optimized 3D Gaussian splash model, uses the extracted 3D mesh as a guide to refine and optimize the 3D Gaussian splash model again, and outputs a high-precision, high-fidelity 3D object.

Images