A method for generating 3D models based on 3D Gaussian.

Abstract

This invention discloses a method for generating 3D models based on 3D Gaussian models. Its key features include using a 3D Gaussian model as the initial 3D model carrier, employing a diffusion model generated from a 2D image to optimize the rendering results of the 3D Gaussian model from various perspectives, and using a mesh model as the final 3D model carrier. Specifically, the method includes: randomly sampling camera positions and rendering the 3D Gaussian model to obtain a rendered image; using the rendered image and text prompts, updating the 3D Gaussian parameters through backpropagation via the diffusion model; converting the trained 3D Gaussian model into a textured mesh and optimizing it using differentiable rendering to obtain the final 3D model. Compared with existing technologies, this invention utilizes a 3D Gaussian model from the field of 3D reconstruction, combined with 2D image generation technology, to achieve rapid generation of 3D models. It effectively solves the problem of extremely long rendering times associated with radiation field-based 3D generation methods, and has promising applications in the field of 3D model generation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-generated content technology, and in particular to a method for generating 3D models based on 3D Gaussian. Background Technology

[0002] Thanks to training sets of billions of image-text pairs and the introduction of two-dimensional diffusion models, natural language-driven two-dimensional image generation technology has achieved a major breakthrough. Applying this method to three-dimensional model generation requires a large-scale dataset of three-dimensional model-text pairs and an efficient architecture for three-dimensional model generation, neither of which currently exists.

[0003] Currently, 3D model generation generally employs neural radiation field-based methods. These methods use the neural radiation field as the 3D model carrier and a 2D generated model as the guiding model. After calculating the loss using the 2D generated model, backpropagation is performed to update the neural radiation field parameters. This iterative update process yields the final 3D model. However, because the neural radiation field uses an implicit function format, it suffers from problems such as difficulty in decoupling texture and geometry, and high computational cost, requiring tens of hours to generate a single 3D model.

[0004] In summary, existing 3D model generation technologies use 2D image generation techniques to supervise the learning of 3D models. This approach suffers from problems such as difficulty in decoupling texture and geometry, and high computational cost. Generating a single 3D model often takes tens of hours, resulting in low efficiency and poor quality. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a 3D model generation method based on 3D Gaussian. This method uses 3D Gaussian as the initial 3D model carrier, employs a diffusion model generated from a 2D image to optimize the rendering results of the 3D Gaussian from various perspectives, and uses a mesh model as the final 3D model carrier to improve the quality of the 3D model. The method uses a diffusion model generated from a 2D image to optimize the rendering results of the 3D Gaussian from various perspectives, uses rendered images and text prompts, updates the 3D Gaussian parameters through backpropagation using the diffusion model, uses a mesh model to convert the trained 3D Gaussian into a textured mesh, and uses differentiable rendering for optimization, resulting in a final 3D model with high efficiency and high quality. This invention accelerates the optimization process by rapidly generating a 3D Gaussian model and extracting geometric and texture information from the 3D Gaussian model for refinement, solving the problem of excessive time consumption in radiation field-based 3D model generation methods, and has promising application prospects.

[0006] The specific technical solution to achieve the purpose of this invention is: a method for generating a 3D model based on 3D Gaussian, characterized by using 3D Gaussian as the initial 3D model carrier, using the diffusion model generated from the 2D image to optimize the rendering results of the 3D Gaussian from various perspectives, and using a mesh model as the final 3D model carrier to improve the quality of the 3D model. The method specifically includes the following steps:

[0007] Step 1: Use a 3D Gaussian model as the initial 3D model carrier and perform image rendering. The specific steps are as follows:

[0008] A1) Randomly sample the camera's placement position in a spherical coordinate system and convert the coordinates into vectors in three-dimensional space to represent the camera's specific position.

[0009] A2) Based on the preset image resolution and the camera's placement and orientation, construct the corresponding rendering transformation matrix.

[0010] A3) Use a 3D Gaussian model to store the geometric and material information of the 3D model. By applying rendering equations, calculate the pixel color under each viewpoint, and then generate an image corresponding to the camera viewpoint.

[0011] Step 2: Use the diffusion model generated from the 2D image to optimize the rendering results of the 3D Gaussian image from various perspectives. The specific steps are as follows:

[0012] B1) Combining the angle information randomly sampled in the spherical coordinate system, the object description text input by the user is expanded to include orientation information, and the CLIP model is used to convert the text into a high-dimensional embedding matrix.

[0013] B2) Introduce Gaussian noise into the image and randomly select the time step of the diffusion process within a certain range. Adjust the degree of mixing between the image and the noise according to this time step to generate a noisy rendered image.

[0014] B3) The noise image, text embedding matrix, and diffusion time points are used as inputs to the denoising model in the two-dimensional diffusion model to predict the noise distribution.

[0015] B4) By comparing the actual noise with the predicted noise, the error is calculated and the parameters of the three-dimensional Gaussian model are updated using the backpropagation algorithm. After 500 iterations, a three-dimensional Gaussian model that matches the generated object description text is obtained.

[0016] Step 3: Use a mesh model as the final 3D model carrier and further optimize it. The specific steps are as follows:

[0017] C1) A mesh model is used to accurately represent the geometry and surface material of the 3D model. The MarchingCubes algorithm is used to extract the geometry based on the opacity information of the 3D Gaussian model.

[0018] C2) Randomly select multiple perspectives to render the 3D Gaussian model, and obtain material information through a multi-view back-projection algorithm to obtain texture maps.

[0019] C3) The camera position and orientation are sampled using the same method as in step 1, and a differentiable renderer is used to render the mesh model at a higher resolution than the image rendered in step 1.

[0020] C4) When the mesh model is rendered to the image, the differentiable renderer tracks the UV coordinates of the projection of each corresponding pixel. These coordinates will be used to query the color in the corresponding texture map. Using the same method as in step 2, the parameters of the mesh model and texture map are updated. After 50 iterations, a 3D mesh model that matches the generated object description text is obtained.

[0021] Compared with existing technologies, this invention has the advantages of high efficiency, high quality, and rapid generation of 3D models. By utilizing the 3D Gaussian model in the field of 3D reconstruction and combining it with 2D image generation technology, an explicit 3D model in mesh format can be generated in just three minutes, realizing the rapid generation of 3D models. This effectively solves the problem of the extremely long time consumption of 3D generation methods based on radiation fields and has certain application prospects in the field of 3D model generation. Attached Figure Description

[0022] Figure 1 This is a flowchart of the present invention;

[0023] Figure 2 Flowchart for rendering images from a 3D Gaussian model;

[0024] Figure 3 Flowchart for training a 3D Gaussian model based on natural language supervision;

[0025] Figure 4 This is a flowchart of a grid model trained under natural language supervision. Detailed Implementation

[0026] To facilitate understanding of the present invention, the invention will be described in detail below with reference to the accompanying drawings.

[0027] See Figure 1 This invention uses a 3D Gaussian model as the initial 3D model carrier, uses a diffusion model generated from a 2D image to optimize the rendering results of the 3D Gaussian model from various perspectives, and uses a mesh model as the final 3D model carrier. Specifically, it includes the following steps:

[0028] Step 1: 3D Gaussian rendering of the image;

[0029] Step 2: Supervised training of a 2D diffusion model with a 3D Gaussian model;

[0030] Step 3: Supervised training of the grid model using a two-dimensional diffusion model.

[0031] This invention takes generated object description text as input, uses a two-dimensional diffusion model to supervise the training of a three-dimensional Gaussian model based on the rendering results of the three-dimensional Gaussian model, initializes the mesh model based on the opacity of the three-dimensional Gaussian points, obtains material maps through a multi-view back projection algorithm, and uses a two-dimensional diffusion model to supervise the training of the mesh model and material maps to obtain a three-dimensional mesh model that conforms to the generated object description text.

[0032] See Figure 2 Step 1 involves rendering the 3D Gaussian model. The camera position and orientation are obtained by sampling in a spherical coordinate system. The rotation and translation matrices are derived based on the camera position and rendering resolution. Information such as opacity, positive definite matrix, and color is stored using 3D Gaussian points. Finally, the rendered image is obtained using the Gaussian rendering equation. The specific steps are as follows:

[0033] S100: Randomly sample the camera position in spherical coordinates. For the polar angle φ, the sampling range is set to [-10°, 90°], for the azimuth angle θ, the sampling range is set to [0°, 360°], and for the distance r to the origin, the sampling range is set to [1, 1.5]. Finally, the sampling results in the spherical coordinate system are converted into a three-dimensional vector x, representing the camera position.

[0034] S110: Set the resolution of the rendered image to 64*64, take the image center coordinate y as the endpoint and the camera coordinate x as the starting point, and calculate the camera direction d. Using the camera position x as the ray origin and the camera direction d as the ray direction, construct the rotation matrix R and the translation vector t, and combine them with the intrinsic parameter matrix K to obtain the rotation and translation matrix W.

[0035] S120: Using the rotation and translation matrix W as input, construct and train a 3D Gaussian to learn the shape and material of the 3D model. Each 3D Gaussian point contains the following attributes: center position x represented by a 3D vector, opacity α represented by a scalar, a 3*3 symmetric positive definite matrix Σ, and color c represented by a 3D vector.

[0036] The Jacobian matrix J is used to approximate the non-affine transformation from camera space to ray space. For each Gaussian center point (u0, u1, u2), the Jacobian matrix J can be approximated by the following equation (a):

[0037]

[0038] Where (x0,x1,x2) are the coordinates of the points in the ray space; and (u0,u1,u2) are the coordinates of the points in the camera space.

[0039] The covariance matrix Σ of the 3D Gaussian can be transformed into the covariance matrix Σ′ expressed by the following equation (b): Using the rotation and translation matrix W and the Jacobian matrix J,

[0040] Σ′=JWΣW T J T (b)

[0041] Among them, W T J T These are the rotation and translation matrices and the Jacobian matrix after transpose, respectively.

[0042] The pixel color C is obtained using the following rendering equation, and finally the rendered image I corresponding to the camera position and orientation is obtained. The rendering equation is expressed by the following equation (c):

[0043]

[0044] Where C is the pixel color; c i Let α be the color of the i-th Gaussian point; i x represents the opacity of the i-th Gaussian point; i Let p be the coordinates of the i-th Gaussian point; Σ′ is the covariance matrix in ray space; p i Let be the brightness of the i-th Gaussian point.

[0045] See Figure 3 In step 2, a two-dimensional diffusion model is used to train a three-dimensional Gaussian model. In this invention, the user-input object description text is added with directional descriptors based on randomly sampled camera positions and encoded to obtain a text embedding matrix. Noise is added to the rendered image based on uniformly sampled diffusion time steps. The text embedding matrix, diffusion time steps, and the rendered image with added noise are used as inputs to the noise prediction module in the two-dimensional diffusion model to predict diffusion noise. Finally, the three-dimensional Gaussian model is updated based on the error between the sampled noise and the predicted noise. After multiple iterations, a three-dimensional Gaussian model that conforms to the object description text is obtained.

[0046] S200: First, the user-input descriptive text is converted into matrix form. Based on the randomly sampled spherical coordinate angles in step 1, directional descriptive words are added to the user-input generated object description text. Next, the generated object description text is encoded using the CLIP model (Contrastive Language-Image Pre-Training) to obtain a text embedding matrix τ with dimensions (77,768).

[0047] S210: Add noise to the rendered image by randomly sampling Gaussian noise of the same dimension as the rendered image, in a uniformly distributed manner. At random sampling diffusion time step t, the rendered image I is added to Gaussian noise ∈ at a ratio of (1-t):t, thus obtaining the rendered image z with added noise. t .

[0048] S220: Add noise to the rendered image z t Using the text embedding matrix τ and the diffusion time step t as inputs to the noise prediction module in the two-dimensional diffusion model, the predicted diffusion noise is obtained.

[0049] S230: Calculate the error between the sampled noise and the predicted noise. The gradient of the error is calculated through backpropagation, and the parameters of the 3D Gaussian model are updated. After a certain number of iterations, a 3D Gaussian model that matches the generated object description text is obtained.

[0050] See Figure 4 Step 3 initializes and trains the mesh model. The mesh model and texture map are initialized based on the three-dimensional Gaussian model obtained in step 2. The camera position and orientation are sampled on the spherical coordinate system and differentiable rendering is performed to obtain the rendered image. The parameters of the mesh model and texture map are updated using a two-dimensional diffusion model. After a certain number of iterations, a three-dimensional mesh model that conforms to the generated object description text is obtained.

[0051] S300: To initialize the mesh model from the 3D Gaussian, the Marching Cubes algorithm is used to obtain the geometric information of the mesh model. According to the following equation (d), the 128^3 squares in space are queried to determine whether the d(x) corresponding to the square is greater than 0.8, thereby determining whether it is the boundary of the mesh model:

[0052]

[0053] Where, α i x represents the opacity of the i-th Gaussian point; i Let be the coordinates of the i-th Gaussian point; Σ′ is the covariance matrix in ray space.

[0054] S310: In order to obtain the material information of the mesh model, 20 random viewpoints are selected to render the 3D Gaussian, and the rendered RGB images are back-projected onto the mesh model obtained in 3.1 initialization to obtain the corresponding texture maps.

[0055] S320: The camera position and orientation are sampled using the same method as in step 1, and the mesh model is rendered with a resolution of 256x256 using a differentiable renderer.

[0056] S330: When the mesh is rendered to the image, the differentiable renderer tracks the 3D coordinates of the projection of each corresponding pixel. These coordinates will be used to query the color in the corresponding texture field. Using the same method as in step 2, the parameters of the mesh model and texture map are updated. After a certain number of iterations, a 3D mesh model that matches the generated object description text is obtained.

[0057] The above is merely a further description of the present invention and is not intended to limit the scope of this patent. Any equivalent implementation of the present invention should be included within the scope of the claims of this patent.

Claims

1. A method for generating 3D models based on 3D Gaussian, characterized in that, Using generated object description text as input, and based on the rendering results of a 3D Gaussian model, a 3D Gaussian model is trained under supervision using a 2D diffusion model. The mesh model is initialized based on the opacity of the 3D Gaussian points. Material maps are obtained through a multi-view back-projection algorithm. The mesh model and material maps are then trained under supervision using a 2D diffusion model to obtain a 3D mesh model that conforms to the generated object description text. This method specifically includes the following steps: Step 1: 3D Gaussian rendering of the image 1.1) Randomly sample the camera position in the spherical coordinate system and convert the sampling result into a three-dimensional vector x representing the camera position. For the polar angle φ, the sampling range is set to [-10°, 90°], for the azimuth angle θ, the sampling range is set to [0°, 360°], and for the distance r to the origin, the sampling range is set to [1, 1.5]. 1.2) Using the image center coordinate y as the endpoint and the camera coordinate x as the starting point, calculate the camera direction d, and construct the rotation matrix R and translation vector t with the 3D vector x as the ray starting point and d as the ray direction. Combine the intrinsic parameter matrix K to obtain the rotation and translation matrix W, and set the resolution of the rendered image to 64*64. 1.3) Using the rotation and translation matrix W as input, a 3D Gaussian is constructed and trained to learn the shape and material of the 3D model. Each 3D Gaussian point contains: the center position x represented by a 3D vector, the opacity α represented by a scalar, a 3*3 symmetric positive definite matrix Σ, and the color c represented by a 3D vector. The acquisition of the 3D Gaussian rendered image specifically includes: 1.3.1) The Jacobian matrix J is used to approximate the non-affine transformation from camera space to ray space. For each Gaussian center point (u0, u1, u2), the Jacobian matrix J is expressed by the following equation (a): Where x0, x1, x2 are coordinate points in ray space; u0, u1, u2 are coordinate points in camera space; 1.3.2) By using the rotation and translation matrix W and the Jacobian matrix J, the covariance matrix Σ of the 3D Gaussian is transformed into the covariance matrix Σ in ray space by the following equation (b). ′ : Σ′=JWΣW T J T (b); Among them, W T J T These are the rotation and translation matrices and the Jacobian matrix after transposition, respectively; 1.3.3) Using the rendering equation shown in equation (c) below, the pixel color C is obtained: Where C is the pixel color; c i Let α be the color of the i-th Gaussian point; i x represents the opacity of the i-th Gaussian point; i Let Σ be the coordinates of the i-th Gaussian point; ′ p is the covariance matrix in ray space; i Let be the brightness of the i-th Gaussian point; 1.3.4) Obtain the rendered image I corresponding to the camera's position and orientation; Step 2: Supervised training of a two-dimensional diffusion model with a three-dimensional Gaussian model 2.1) Convert the user-input description text into matrix form. Based on the randomly sampled spherical coordinate angles in step 1, add the orientation descriptive words to the user-input generated object description text. Use the CLIP model to encode the generated object description text to obtain a text embedding matrix τ with dimensions of 77,768. 2.2) Randomly sampled Gaussian noise of the same dimension as the rendered image, in a uniformly distributed manner. At random sampling diffusion time step t, the rendered image I is added to Gaussian noise ∈ at a ratio of 1-t:t to obtain the rendered image z9 with noise added; 2.3) Add noise to the rendered image z : Using the text embedding matrix τ and the diffusion time step t as inputs to the noise prediction module in the two-dimensional diffusion model, the predicted diffusion noise is obtained. 2.4) Calculate the error between the sampled noise and the predicted noise. The gradient is calculated through backpropagation, the parameters of the 3D Gaussian are updated and iterated 500 times to obtain a 3D Gaussian that matches the generated object description text. Step 3: Supervised training of the mesh model using a two-dimensional diffusion model 3.1) The Marching Cubes algorithm is used to obtain the geometric information of the mesh model. The mesh model is initialized from the 3D Gaussian matrix, and the following equation (d) is used to query the 128^3 squares in space to determine whether the d(x) corresponding to the square is greater than 0.8, thereby determining whether it is the boundary of the mesh model: in, α i x represents the opacity of the i-th Gaussian point; i Let Σ be the coordinates of the i-th Gaussian point; Σ′ is the covariance matrix in ray space; 3.2) Randomly select 20 viewpoints to render the 3D Gaussian, and back-project the rendered RGB images onto the mesh model initialized in 3.1) to obtain the corresponding texture maps, i.e. the material information of the mesh model; 3.3) The camera position and orientation are sampled using the same method as in step 1, and the resolution of the mesh model is set to 256x256 using a differentiable renderer; 3.4) When the mesh model is rendered to the image, a differentiable renderer is used to track the 3D coordinates of the projection of each corresponding pixel and query the color in the corresponding texture field. Using the same method as in step 2, the parameters of the mesh model and texture map are updated and iterated 50 times to obtain a 3D mesh model that matches the generated object description text.

Images