Three-dimensional object generation method based on linearization look-ahead VSD

Abstract

A three-dimensional object generation method based on linearized look-ahead VSD comprises the steps that updated parameters of a LoRA network are expanded at the positions of the parameters before updating through Taylor series, and the optimization process of the LoRA network for simulating approximate three-dimensional parameter distribution is matched with the optimization process of three-dimensional object parameters through a first-order item; and the generation effect is greatly improved, and meanwhile, the calculation time of each step is slightly increased. According to the method, very ideal performance improvement is achieved on the task of generating the three-dimensional object based on the text, and meanwhile, the method can be integrated with other parallel methods for three-dimensional distillation.

Description

Technical Field

[0001] The present invention relates to a technology in the field of three-dimensional vision processing, specifically a three-dimensional object generation method based on a linearized lookahead variational score distillation model (Linearized Lookahead Variational Score Distillation). Background Art

[0002] The optimization-based 3D object generation method uses the object representation as an optimizable network parameter, and generates a large model by distilling the pre-trained image, so that the images rendered by the 3D object to be optimized at each viewing angle are as close as possible to the image distribution in the image model. Although compared with the sampling-based method, the optimization method can produce more realistic, clearer and higher-quality 3D objects, its renderings at each viewing angle still have a certain gap with the image quality generated by the model; at the same time, it requires a long distillation optimization process, which hinders further large-scale application. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a three-dimensional object generation method based on linearized forward-looking VSD. Based on an in-depth analysis of the principle and implementation method of the original variational score distillation model, the problems existing in the update and optimization process of the three-dimensional object parameters and the LoRA network are discovered, and the linear terms and high-order terms are separated through Taylor series decomposition, and the influence of these two terms on the effect is decoupled and compared, and finally the original gradient is corrected in a way that the linear terms are retained. While the present invention has achieved a very ideal performance improvement in the task of generating three-dimensional objects based on text, it can be integrated with other parallel methods for three-dimensional distillation.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a three-dimensional object generation method based on linearized forward-looking VSD. The updated parameters of the LoRA network are expanded at the parameters before the update by using Taylor series, and the first-order terms are used to simulate the optimization process of the LoRA network with approximate three-dimensional parameter distribution to match the optimization process of the three-dimensional object parameters, thereby greatly improving the generation effect at the cost of only slightly increasing the calculation time of each step. Technical Effects

[0006] The present invention uses the variational score distillation model (VSD) as the main framework, and on its basis, makes corresponding corrections to the gradient used for updating by only increasing the cost of the forward process of the LoRA network once for each iterative optimization. An effective linear forward correction term is separated, which greatly improves the generation quality at the cost of increasing extremely low time cost, and can generate a generation effect close to that of more optimized existing technologies with only a small number of optimization steps. Compared with the prior art, the present invention can generate very realistic three-dimensional objects in the rough generation stage of the first stage, which significantly improves the generation quality and generation efficiency of three-dimensional objects, and can be combined with other distillation methods used in parallel for three-dimensional generation, thereby improving the generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a flow chart of the present invention;

[0008] Figure 2 A schematic diagram of the source of gradient updates to distinguish the innovation of the present invention from the baseline method;

[0009] In the figure: (a) represents the update method of VSD, (b) represents the update method of L-VSD, (c) represents the update method of the present invention, blue and red arrows represent the update gradient sources of the three-dimensional object parameters, green, yellow and pink arrows represent the update gradient sources of the LoRA network parameters under the three methods respectively;

[0010] Figure 3 Generate a comparison chart of the effects in the first stage with a resolution of 64*64;

[0011] Figure 4 Generate a comparison chart of the effects at all stages with a resolution of 64*64;

[0012] Figure 5 Generate a comparison chart of the effects in the first stage with a resolution of 256*256;

[0013] Figure 6 It is a comparison diagram of the effects of the embodiments. DETAILED DESCRIPTION

[0014] like Figure 1 As shown, this embodiment relates to a three-dimensional object generation method based on linearized forward-looking VSD, under the specific environment setting of the open source threestudio framework, including:

[0015] Step 1: Randomly initialize the three-dimensional object parameters θ0, set T to the total number of optimization steps set; load the pre-trained Wensheng graph model ε pretrainAfter adding the LoRA network layer between all its linear layers, the LoRA network layer is initialized in the same way as the default initialization of the linear layer to obtain the LoRA fine-tuned model

[0016] Step 2: At the beginning of the loop, randomly select a camera perspective and represent the internal and external parameters as the camera matrix c, and treat the three-dimensional object parameters θ i Render the image x of the corresponding perspective, randomly select the time step t, sample the standard Gaussian noise ε to add noise to the image x, and get the noisy image x t .

[0017] Step 3: Add the noisy image x t , the text embedding vector y and the time step t are respectively input into the pre-trained text graph model ε in step 1 pretrain And the LoRA fine-tuned model The prediction noise ε is generated later pretrain (x t ,t,y) and

[0018] Step 4: Randomly select time step t' again to obtain the corresponding noisy image x t' , by calculating the LoRA fine-tuned model The prediction loss By back-propagating the gradient, we get in:

[0019] Step 5: Analyze the existence of and The gap between exist The Taylor series expansion at At the same time, according to the update of LoRA network parameters This term is the gradient obtained in step 3, thus obtaining Among them: The first-order term is The higher-order terms are

[0020] Step 6: Use PyTorch forward automatic differentiation to calculate the linear correction term Combined with the predicted noise calculated in step 3, the optimized gradient of the three-dimensional object parameters is calculated

[0021] Step 7: Update the 3D object parameters using the gradients obtained in step 6 At the same time, the gradient of the LoRA model is used Update LoRA fine-tuned model Network parameters Return to step 2.

[0022] Step 8: Until i ≥ T, the loop ends and the optimized three-dimensional object parameters θ are obtained T .

[0023] This embodiment is run under the open source threestudio framework, and uses the parameter settings of the original variational score distillation model. The pre-trained Wensheng graph model uses Stable-Diffusion 1.5. All total optimization steps are T=15000, η=1e-4; at the same time, other baseline methods are also run under this open source framework with corresponding default parameters.

[0024] As shown in Table 1, this is the generation effect of this method and other baseline methods on 6 different instances. This method uses two different quantitative indicators, FID and angle of CLIP similarity, to reflect the effects of different methods. Due to the strong independence between instances, the following table will report the performance comparison between instances independently.

[0025] Table 1

[0026] like Figure 2 As shown in (a), the source of the gradient update of the three-dimensional object parameters in the original variational score distillation model is shown, θ i+1 Depends on θ i and The following figure shows the source of the gradient update of the three-dimensional object parameters in the simple forward variational score distillation model, θ i+1 Depends on θ i and The display in the figure is the source of the three-dimensional object parameter gradient update of the present invention, θ i+1 At the same time, it depends on θ i , and

[0027] like Figure 3 As shown, the quality of the three-dimensional assets (last column) generated in the first stage of the present invention is better than that of the three-dimensional assets generated in the first stage of various baseline methods. The image resolution during the distillation process of this embodiment is 64*64.

[0028] like Figure 4 As shown, the quality of the three-dimensional assets (last column) generated by the present invention in the third stage is better than that of the three-dimensional assets generated by various baseline methods in the third stage. The image resolution during the distillation process of this embodiment is 64*64.

[0029] like Figure 5As shown, the quality of the three-dimensional assets (last column) generated in the first stage of the present invention is better than that of the three-dimensional assets generated in the first stage of various baseline methods. The image resolution during the distillation process of this embodiment is 256*256.

[0030] like Figure 6 As shown, the asset quality comparison of the present invention combined with other baseline methods in the first stage is demonstrated, thereby proving that the present invention can be combined with other parallel baseline methods to further optimize the generation effect.

[0031] Compared with the prior art, the present invention performs Taylor series expansion at the network stage, and matches the update of three-dimensional parameters with the update of the network after LoRA fine-tuning by calculating the linearized forward correction term obtained by gradient backpropagation and forward automatic differentiation again. In order to obtain the gradient correction term, the present invention only adds the forward process of the LoRA network once. If the calculation of the correction term requires the parameters of the entire LoRA network, each optimization step will consume 0.3s more time; the present invention can achieve a balance between time efficiency and performance improvement by using only the last layer of the LoRA network. This measure can consume only 0.1s more time in each optimization step. While achieving a very ideal performance improvement in the task of generating three-dimensional objects based on text, it can be integrated with other parallel methods for three-dimensional distillation.

[0032] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A three-dimensional object generation method based on linearized forward-looking VSD, characterized in that: By using Taylor series to expand the updated parameters of the LoRA network at the parameters before the update, the first-order terms are used to simulate the optimization process of the LoRA network with approximate three-dimensional parameter distribution and match it with the optimization process of the three-dimensional object parameters, which greatly improves the generation effect at the cost of only a minimal increase in the calculation time of each step.

2. The three-dimensional object generation method based on linearized forward-looking VSD according to claim 1 is characterized in that: include: Step 1, randomly initialize the three-dimensional object parameters θ0, and set T to the total number of optimization steps; Load the pre-trained Wensheng graph model ε pretrain After adding the LoRA network layer between all its linear layers, the LoRA network layer is initialized in the same way as the default initialization of the linear layer to obtain the LoRA fine-tuned model Step 2: At the beginning of the loop, randomly select a camera perspective and represent the internal and external parameters as the camera matrix c, and treat the three-dimensional object parameters θ i Render the image x of the corresponding perspective, randomly select the time step t, sample the standard Gaussian noise ε to add noise to the image x, and get the noisy image x t ; Step 3: Add the noisy image x t , the text embedding vector y and the time step t are respectively input into the pre-trained text graph model ε in step 1 pretrain And the LoRA fine-tuned model The prediction noise ε is generated later pretrain (x t , t, y) and Step 4: Randomly select time step t′ again to obtain the corresponding noisy image x t′ , by calculating the LoRA fine-tuned model The prediction loss By back-propagating the gradient, we get in: Step 5: Analyze the existence of and The gap between exist The Taylor series expansion is At the same time, according to the update of LoRA network parameters This term is the gradient obtained in step 3, so we get Among them: The first-order term is The higher-order terms are Step 6: Use PyTorch forward automatic differentiation to calculate the linear correction term Combined with the predicted noise calculated in step 3, the optimized gradient of the three-dimensional object parameters is calculated Step 7: Update the 3D object parameters using the gradients obtained in step 6 At the same time, the gradient of the LoRA model is used Update LoRA fine-tuned model Network parameters Return to step 2; Step 8: Until i ≥ T, the loop ends and the optimized three-dimensional object parameters θ are obtained T .

Images