Three-dimensional model generation method based on three-dimensional Gaussian

Abstract

The invention discloses a three-dimensional model generation method based on three-dimensional Gaussian, which is characterized in that the three-dimensional Gaussian is used as an initial three-dimensional model carrier, a diffusion model for generating a two-dimensional image is used for optimizing the rendering result of each view angle of the three-dimensional Gaussian, and a grid model is used as a final three-dimensional model carrier. The method specifically comprises the following steps: randomly sampling the position of a camera and rendering a three-dimensional Gaussian model to obtain a rendered image; the three-dimensional Gaussian parameters are updated through diffusion model back propagation by using rendered image and text prompt; and converting the trained three-dimensional Gaussian into a grid with textures, and performing optimization by using differentiable rendering to obtain a final three-dimensional model, and the like. Compared with the prior art, the method has the advantages that the three-dimensional Gaussian model in the field of three-dimensional reconstruction is utilized, the two-dimensional image generation technology is combined, rapid generation of the three-dimensional model is achieved, the problem that a three-dimensional generation method based on a radiation field consumes long time is well solved, and the method has certain application prospects in the field of three-dimensional model generation.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence content generation, and in particular to a three-dimensional model generation method based on three-dimensional Gaussian. Background Art

[0002] Thanks to billions of training sets of image-text pairs and the introduction of 2D diffusion models, natural language-driven 2D image generation has made great breakthroughs. Applying this approach to 3D model generation requires large-scale 3D model-text pair datasets and efficient architectures for 3D model generation, neither of which currently exists.

[0003] At present, the generation of 3D models generally adopts the generation method based on neural radiation field, using neural radiation field as the carrier of 3D model and 2D generation model as the guide model. After calculating the loss through 2D generation model, back propagation is performed to update the neural radiation field parameters, and the final 3D model is obtained through iterative update. Since neural radiation field is in implicit function format, there are problems such as difficulty in decoupling texture and geometry and large amount of calculation. It takes dozens of hours to generate a 3D model.

[0004] In summary, the existing 3D model generation technology uses 2D image generation technology to perform supervised learning on the 3D model. There are problems such as difficulty in decoupling texture and geometry and large amount of calculation. It often takes dozens of hours to generate a 3D model, which is inefficient and of poor quality. Summary of the invention

[0005] The purpose of the present invention is to provide a three-dimensional model generation method based on three-dimensional Gaussian in view of the deficiencies of the prior art. The three-dimensional Gaussian is used as the initial three-dimensional model carrier, and the diffusion model for generating two-dimensional images is used to optimize the rendering results of the three-dimensional Gaussian at various viewing angles, and the mesh model is used as the final three-dimensional model carrier to improve the quality of the three-dimensional model. The method uses the diffusion model for generating two-dimensional images to optimize the rendering results of the three-dimensional Gaussian at various viewing angles, uses the rendered image and text prompts, updates the three-dimensional Gaussian parameters through the diffusion model back propagation, uses the mesh model to convert the trained three-dimensional Gaussian into a textured mesh, and uses differentiable rendering for optimization, and obtains the final three-dimensional model with high efficiency and high quality. The present invention solves the problem that the three-dimensional model generation method based on the radiation field is too time-consuming by quickly generating a three-dimensional Gaussian model and extracting geometric and texture information from the three-dimensional Gaussian model to refine and accelerate the optimization process, and has a good application prospect.

[0006] The specific technical solution for achieving the purpose of the present invention is: a 3D model generation method based on 3D Gaussian, characterized in that the 3D Gaussian is used as the initial 3D model carrier, the diffusion model for generating the 2D image is used to optimize the rendering results of the 3D Gaussian at each viewing angle, and the mesh model is used as the final 3D model carrier to improve the quality of the 3D model. The method specifically includes the following steps:

[0007] Step 1: Use 3D Gaussian as the initial 3D model carrier and perform image rendering. The specific steps are:

[0008] A1) Randomly sample the camera's placement position in the spherical coordinate system and convert the coordinates into vectors in three-dimensional space to represent the specific position of the camera.

[0009] A2) Construct a corresponding rendering transformation matrix for the preset image resolution according to the camera's placement and orientation.

[0010] A3) Use a 3D Gaussian model to store the geometric information and material information of the 3D model, and calculate the pixel color at each viewing angle by applying a rendering equation, thereby generating an image corresponding to the camera viewing angle.

[0011] Step 2: Use the diffusion model generated by the two-dimensional image to optimize the rendering results of the three-dimensional Gaussian at each viewing angle. The specific steps are:

[0012] B1) Combined with the randomly sampled angle information in the spherical coordinate system, the object description text entered by the user is expanded to include orientation information, and the CLIP model is used to convert the text into a high-dimensional embedding matrix.

[0013] B2) Introducing Gaussian noise into the image, and randomly selecting the time step of the diffusion process within a certain range, and adjusting the mixing degree of the image and the noise according to this time step, thereby generating a rendered image with noise.

[0014] B3) The noise image, text embedding matrix and diffusion time points are used as inputs of the denoising model in the two-dimensional diffusion model to predict the noise distribution.

[0015] B4) By comparing the actual noise with the predicted noise, calculating the error and using the back propagation algorithm to update the parameters of the three-dimensional Gaussian model, after 500 iterations, a three-dimensional Gaussian that conforms to the generated object description text is obtained.

[0016] Step 3: Use the mesh model as the final 3D model carrier and further optimize it. The specific steps are:

[0017] C1) Use the mesh model to accurately represent the geometric structure and surface material of the 3D model, and use the MarchingCubes algorithm to extract the geometric structure based on the opacity information of the 3D Gaussian model.

[0018] C2) Randomly select multiple perspectives to render the three-dimensional Gaussian model, and obtain material information through a multi-perspective back-projection algorithm to obtain a texture map.

[0019] C3) The camera position and orientation are sampled using the same method as step 1, and a differentiable renderer is used to render the mesh model at a higher resolution than the image rendered in step 1.

[0020] C4) When the mesh model is rendered to the image, the differentiable renderer tracks the UV coordinates of each corresponding pixel projection. These coordinates will be used to query the color in the corresponding texture map. The parameters of the mesh model and texture map are updated using the same method as step 2. After 50 iterations, a 3D mesh model that conforms to the generated object description text is obtained.

[0021] Compared with the prior art, the present invention has the advantages of high efficiency, good quality and rapid generation of 3D models. It utilizes the 3D Gaussian model in the field of 3D reconstruction and combines it with 2D image generation technology to generate an explicit 3D model in a grid format in only three minutes, thereby achieving rapid generation of 3D models and effectively solving the problem of extremely long time consumption of 3D generation methods based on radiation fields. It has certain application prospects in the field of 3D model generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the present invention;

[0023] Figure 2 Rendering image flow chart for 3D Gaussian model;

[0024] Figure 3 Flowchart for training a 3D Gaussian model based on natural language supervision;

[0025] Figure 4 Flowchart for training a grid model based on natural language supervision. DETAILED DESCRIPTION

[0026] In order to facilitate the understanding of the present invention, the present invention is described in detail below with reference to the accompanying drawings.

[0027] See also Figure 1 The present invention uses a three-dimensional Gaussian as an initial three-dimensional model carrier, uses a diffusion model for generating a two-dimensional image to optimize the rendering results of the three-dimensional Gaussian at each viewing angle, and uses a grid model as a final three-dimensional model carrier, specifically including the following steps:

[0028] Step 1: 3D Gaussian rendering image;

[0029] Step 2: supervised training of 3D Gaussian using 2D diffusion model;

[0030] Step 3: 2D diffusion model supervises the training of grid model.

[0031] The present invention takes the generated object description text as input, uses a two-dimensional diffusion model to supervise the training of the three-dimensional Gaussian model according to the rendering result of the three-dimensional Gaussian model, then initializes the grid model according to the opacity of the three-dimensional Gaussian points, obtains the material map through a multi-view back projection algorithm, and uses the two-dimensional diffusion model to supervise the training of the grid model and the material map to obtain a three-dimensional grid model that conforms to the generated object description text.

[0032] See also Figure 2 , the step 1 renders the three-dimensional Gaussian model, samples in the spherical coordinate system to obtain the camera position and direction, obtains the rotation and translation matrix according to the camera position and the rendering resolution, uses the three-dimensional Gaussian points to store information such as opacity, positive definite matrix, color, etc., and uses the Gaussian rendering equation to obtain the rendered image, and the specific steps are as follows:

[0033] S100: Randomly sample the camera position. Randomly sample in spherical coordinates. For the polar angle φ, the sampling range is set to [-10°, 90°], for the azimuth angle θ, the sampling range is set to [0°, 360°], for the distance r to the origin, the sampling range is set to [1, 1.5]. Finally, the sampling result of the spherical coordinate system is converted into a three-dimensional vector x, which represents the position of the camera.

[0034] S110: Set the resolution of the rendered image to 64*64, take the image center coordinate y as the end point, the camera coordinate x as the starting point, and calculate the camera direction d. Use the camera position x as the ray starting point and the camera direction d as the ray direction to construct the rotation matrix R and the translation vector t, and combine the intrinsic parameter matrix K to obtain the rotation and translation matrix W.

[0035] S120: Take the rotation and translation matrix W as input, construct and train a three-dimensional Gaussian to learn the shape and material of the three-dimensional model. Each three-dimensional Gaussian point contains the following attributes: the center position x represented by a three-dimensional vector, the opacity α represented by a scalar, the 3*3 symmetric positive definite matrix Σ, and the color c represented by a three-dimensional vector.

[0036] The Jacobian matrix J is used to approximate the non-affine transformation from camera space to ray space. For each Gaussian center point (u0, u1, u2), the Jacobian matrix J can be approximated as follows (a):

[0037]

[0038] Among them, (x0,x1,x2) is the coordinate point in ray space; (u0,u1,u2) is the coordinate point in camera space.

[0039] By rotating and translating the matrix W and the Jacobian matrix J, the covariance matrix Σ of the 3D Gaussian can be converted into the covariance matrix Σ′ represented by the following formula (b):

[0040] Σ′=JWΣW T J T (b).

[0041] Among them, W T , J T are the transposed rotation and translation matrices and Jacobian matrices respectively.

[0042] The pixel color C is obtained using the following rendering equation, and finally the rendered image I corresponding to the camera position and direction is obtained. The rendering equation is expressed by the following formula (c):

[0043]

[0044] Where C is the pixel color; c i is the color of the i-th Gaussian point; α i is the opacity of the i-th Gaussian point; x i is the coordinate of the i-th Gaussian point; Σ′ is the covariance matrix in the ray space; p i is the brightness of the i-th Gaussian point.

[0045] See also Figure 3 In step 2, the two-dimensional diffusion model is used to train the three-dimensional Gaussian model. The invention adds the generated object description text input by the user to the orientation description word according to the randomly sampled camera position, and encodes it to obtain a text embedding matrix, adds noise to the rendered image according to the uniformly sampled diffusion time step, and uses the text embedding matrix, the diffusion time step, and the rendered image with the added noise as the input of the noise prediction module in the two-dimensional diffusion model to predict the diffusion noise, and finally updates the three-dimensional Gaussian model according to the error between the sampled noise and the predicted noise, and obtains the three-dimensional Gaussian model that conforms to the generated object description text after multiple iterations.

[0046] S200: First, the description text input by the user is converted into a matrix form. Based on the spherical coordinate angles randomly sampled in step 1, the orientation description words are added to the generated object description text input by the user. Then, the CLIP model (Contrastive Language-Image Pre-Training) is used to encode the generated object description text to obtain a text embedding matrix τ with a dimension of (77,768).

[0047] S210: Add noise to the rendered image, randomly sample Gaussian noise ∈ with the same dimension as the rendered image, and distribute it uniformly In the random sampling diffusion time step t, the rendered image I is added with Gaussian noise ∈ in the ratio of (1-t):t to obtain the rendered image z with added noise t .

[0048] S220: Add the noise to the rendered image z t Together with the text embedding matrix τ and the diffusion time step t, it is used as the input of the noise prediction module in the two-dimensional diffusion model to obtain the predicted diffusion noise

[0049] S230: Calculate the error between the sampled noise and the predicted noise The error is back-propagated to calculate the gradient and update the parameters of the three-dimensional Gaussian model. After a certain number of iterations, a three-dimensional Gaussian model that conforms to the generated object description text is obtained.

[0050] See also Figure 4 In step 3, the mesh model is initialized and trained, the mesh model and texture map are initialized according to the three-dimensional Gaussian model obtained in step 2, the camera position and direction are sampled on the spherical coordinate system and differentiable rendering is performed to obtain a rendered image, and the parameters of the mesh model and texture map are updated using a two-dimensional diffusion model. After a certain number of iterations, a three-dimensional mesh model that meets the description text of the generated object is obtained.

[0051] S300: In order to initialize the mesh model from the three-dimensional Gaussian, the Marching Cubes algorithm is used to obtain the geometric information of the mesh model. According to the following formula (d), 128^3 squares in the space are queried to determine whether the d(x) corresponding to the square is greater than 0.8 to determine whether it is the boundary of the mesh model:

[0052]

[0053] Among them, α i is the opacity of the i-th Gaussian point; x i is the coordinate of the i-th Gaussian point; Σ′ is the covariance matrix in the ray space.

[0054] S310: In order to obtain the material information of the mesh model, 20 viewing angles are randomly selected to render the three-dimensional Gaussian, and the rendered RGB image is back-projected onto the mesh model initialized in 3.1 to obtain the corresponding texture map.

[0055] S320: The camera position and direction are sampled using the same method as step 1, and a differentiable renderer is used to set the mesh model rendering resolution to a 256x256 image.

[0056] S330: When the mesh is rendered to the image, the differentiable renderer tracks the 3D coordinates of each corresponding pixel projection. These coordinates will be used to query the color in the corresponding texture field. The same method as step 2 is used to update the parameters of the mesh model and texture map. After a certain number of iterations, a three-dimensional mesh model that conforms to the generated object description text is obtained.

[0057] The above is only a further explanation of the present invention and is not intended to limit this patent. Any equivalent implementation of the present invention should be included in the scope of the claims of this patent.

Claims

1. A three-dimensional model generation method based on three-dimensional Gaussian, characterized in that: The generated object description text is used as input, and according to the rendering result of the three-dimensional Gaussian model, the three-dimensional Gaussian model is supervised and trained using a two-dimensional diffusion model, and the mesh model is initialized according to the opacity of the three-dimensional Gaussian point. The material map is obtained through a multi-view back-projection algorithm, and the mesh model and the material map are supervised and trained using a two-dimensional diffusion model to obtain a three-dimensional mesh model that conforms to the generated object description text. The method specifically includes the following steps: Step 1: 3D Gaussian Rendering 1.1) Randomly sample the camera position in the spherical coordinate system and convert the sampling result into a three-dimensional vector x representing the camera position. For the polar angle φ, the sampling range is set to [-10°, 90°], for the azimuth angle θ, the sampling range is set to [0°, 360°], and for the distance r to the origin, the sampling range is set to [1, 1.5]; 1.2) Take the image center coordinate y as the end point and the camera coordinate x as the starting point, calculate the camera direction d, and use the three-dimensional vector x as the ray starting point and d as the ray direction to construct the rotation matrix R and translation vector t. Combine the intrinsic parameter matrix K to obtain the rotation and translation matrix W, and set the resolution of the rendered image to 64*64; 1.3) Take the rotation and translation matrix W as input, construct and train a 3D Gaussian to learn the shape and material of the 3D model, where each 3D Gaussian point contains: the center position x represented by the 3D vector, the opacity α represented by the scalar, the 3*3 symmetric positive definite matrix Σ and the color c represented by the 3D vector. The 3D Gaussian rendering image is obtained specifically including: 1.3.1) The Jacobian matrix J is used to approximate the non-affine transformation from camera space to ray space. For each Gaussian center point (u0, u1, u2), the Jacobian matrix J is expressed by the following formula (a): Among them, x0, x1, x2 are coordinate points in ray space; u0, u1, u2 are coordinate points in camera space; 1.3.2) By rotating and translating the matrix W and the Jacobian matrix J, the covariance matrix Σ of the 3D Gaussian is converted from the following formula (b) to the covariance matrix Σ′ in the ray space: Σ′=JWΣW T J T (b); Among them, W T , J T are the transposed rotation and translation matrices and Jacobian matrices respectively; 1.3.3) Use the rendering equation shown in the following formula (c) to obtain the pixel color C: Where C is the pixel color; c i is the color of the i-th Gaussian point; a i is the opacity of the i-th Gaussian point; x i is the coordinate of the i-th Gaussian point; Σ′ is the covariance matrix in the ray space; p i is the brightness of the i-th Gaussian point; 1.3.4) Obtain a rendered image I corresponding to the camera position and direction; Step 2: 2D Diffusion Model Supervised Training of 3D Gaussian 2.1) Convert the description text input by the user into a matrix form. Based on the spherical coordinate angles randomly sampled in step 1, add the orientation description words to the generated object description text input by the user. Use the CLIP model to encode the generated object description text to obtain a text embedding matrix τ with a dimension of 77,768. 2.2) Randomly sample Gaussian noise ∈ with the same dimension as the rendered image, in a uniform distribution In the random sampling diffusion time step t, the rendered image I is added with Gaussian noise ∈ in the ratio of 1-t:t to obtain the rendered image z9 with added noise; 2.3) Add the noise to the rendered image z : Together with the text embedding matrix τ and the diffusion time step t, it is used as the input of the noise prediction module in the two-dimensional diffusion model to obtain the predicted diffusion noise 2.4) Calculate the error between the sampling noise and the predicted noise The gradient is calculated by back propagation, the parameters of the three-dimensional Gaussian are updated and iterated 500 times to obtain a three-dimensional Gaussian that meets the description text of the generated object; Step 3: 2D Diffusion Model Supervised Training Grid Model 3.1) Use the Marching Cubes algorithm to obtain the geometric information of the mesh model, initialize the mesh model from the 3D Gaussian, and query the 128^3 squares in the space using the following formula (d) to determine whether the d(x) corresponding to the square is greater than 0.8, thereby determining whether it is the boundary of the mesh model: in, α i is the opacity of the i-th Gaussian point; x i is the coordinate of the i-th Gaussian point; Σ′ is the covariance matrix in the ray space; 3.2) Randomly select 20 viewing angles to render the 3D Gaussian, and back-project the rendered RGB image onto the mesh model initialized in 3.1 to obtain the corresponding texture map, i.e., the material information of the mesh model; 3.3) The camera position and direction are sampled using the same method as step 1, and the resolution of the mesh model rendering is set to 256x256 images using a differentiable renderer; 3.4) When the mesh model is rendered to the image, use a differentiable renderer to track the 3D coordinates of each corresponding pixel projection and query the color in the corresponding texture field; use the same method as step 2 to update the parameters of the mesh model and texture map and iterate 50 times to obtain a 3D mesh model that conforms to the generated object description text.

2. The three-dimensional model generation method based on three-dimensional Gaussian according to claim 1, characterized in that: The step 1 uses a three-dimensional Gaussian as an initial three-dimensional model carrier and performs image rendering, and the specific steps are as follows: A1) randomly sampling the camera's placement position in the spherical coordinate system, and converting the coordinate into a vector x in three-dimensional space to represent the specific position of the camera; A2) constructing a corresponding rendering transformation matrix for a preset image resolution according to the camera's placement and orientation; A3) Use the 3D Gaussian model to store the geometric and material information of the 3D model, and calculate the pixel color c at each viewing angle by applying the rendering equation i , generating an image corresponding to the camera’s viewpoint.

3. The three-dimensional model generation method based on three-dimensional Gaussian according to claim 1, characterized in that: The step 2 generates a diffusion model of a two-dimensional image to optimize the rendering results of the three-dimensional Gaussian at each viewing angle, and the specific steps are as follows: B1) expanding the object description text input by the user to include orientation information based on the randomly sampled angle information in the spherical coordinate system, and converting the text into a high-dimensional embedding matrix using the CLIP model; B2) introducing Gaussian noise into the image, randomly selecting a time step t of the diffusion process within the range of 0.02 to 0.98, and adjusting the mixing degree of the image and the noise according to the time step t to generate a rendered image with noise; B3) using the noise image, text embedding matrix and diffusion time points as inputs to the denoising model in the two-dimensional diffusion model to predict the noise distribution; B4) By comparing the actual noise with the predicted noise, the error is calculated and the parameters of the three-dimensional Gaussian model are updated using the back propagation algorithm. After 500 iterations, a three-dimensional Gaussian that conforms to the generated object description text is obtained.

4. The three-dimensional model generation method based on three-dimensional Gaussian according to claim 1, characterized in that: The step 3 uses the mesh model as the final 3D model carrier and further optimizes it. The specific steps are as follows: C1) Using a mesh model to represent the geometric structure and surface material of a 3D model, and using the Marching Cubes algorithm to extract the geometric structure from the opacity information of the 3D Gaussian model; C2) randomly selecting multiple perspectives to render the three-dimensional Gaussian model, and obtaining material information through a multi-perspective back-projection algorithm to obtain a texture map; C3) The camera position and direction are sampled using the same method as step 1, and the mesh model is rendered into an image with a resolution of 256x256 using a differentiable renderer; C4) Use a differentiable renderer to track the UV coordinates of each corresponding pixel projection, and use it to query the color in the corresponding texture map. Use the same method as step 2 to update the parameters of the mesh model and texture map and iterate 50 times to obtain a three-dimensional mesh model that conforms to the generated object description text.

Images