Initial noise optimization method of new view angle synthesis model based on diffusion model

Abstract

The invention discloses an initial noise optimization method of a new view angle synthesis model based on a diffusion model, belongs to the diffusion model, and can be used in the field of new view angle synthesis. According to the optimization method, firstly, a machine learning network is provided, and the network is composed of an encoder and a decoder; then training an encoder-decoder network according to initial noise-optimized noise pair acquired by a new view angle synthesis model based on a diffusion model; and finally, inserting the trained encoder-decoder network into the new visual angle synthesis model. The invention aims to change initial noise into optimized noise through network action, so that a model generation result is improved on the premise of not finely tuning a model.

Description

Technical Field

[0001] This invention relates to diffusion models, and more specifically to a novel method for optimizing the initial noise of a synthetic model based on diffusion models. Background Technology

[0002] Novel perspective synthesis from a single image is a long-standing and highly valuable research problem in computer vision. It refers to predicting how a 3D object will appear from other perspectives using only a single reference image. This technique can be applied to various downstream tasks, such as game design and virtual / augmented reality. Due to its powerful generative capabilities, diffusion models have gradually become the mainstream method for solving the novel perspective synthesis problem. Most methods address novel perspective synthesis by fine-tuning the diffusion model. However, this fine-tuning method requires significant computational resources.

[0003] Currently, some methods suggest that selecting specific initial noise can improve the generation results of novel perspective synthesis models based on diffusion models. However, how to provide targeted initial noise based on a single reference image remains a problem that urgently needs to be solved. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a novel method for optimizing the initial noise of a synthetic model based on a diffusion model. This method transforms the initial noise into optimized noise through network processing, thereby improving the model generation results without fine-tuning the model.

[0005] To address the problems in the existing technology, the technical solution adopted by this invention is as follows:

[0006] A novel method for optimizing initial noise in a synthetic model based on a diffusion model includes the following steps:

[0007] Step 1: Based on the dataset collected by the new perspective synthetic model based on the diffusion model;

[0008] Step 2: Train the encoder-decoder network according to the loss function, and run the new perspective synthesis model inserted into the encoder-decoder network, specifically the initial noise. Divide by coefficient get ,coefficient Preset parameter values ​​for a new perspective synthetic model based on a diffusion model. Variational autoencoder encoding with reference image The output result is fed into the encoder-decoder network. Then output the results With noise Add element by element and then multiply by the coefficient. The predicted optimized noise is obtained. ; predicted optimized noise instead of the initial noise The denoising network is input into a new view synthesis model to generate multiple images of the 3D object from different perspectives.

[0009] Preferably, the data set in step 1 includes: noise , the variational autoencoder encoding of the reference image, the real optimized noise rich in image semantic information obtained by the noise . .

[0010] Preferably, the encoder-decoder network in step 2 is composed of an encoder and a decoder.

[0011] Preferably, the running steps of the encoder in the encoder-decoder network in step 2 are:

[0012] Step (1), splice the noise and the variational autoencoder encoding of the reference image ;

[0013] Step (2), the spliced result is subjected to convolution layer, batch normalization layer and RELU nonlinear layer to obtain feature tensor ;

[0014] Step (3), maximum pooling operation is performed on the feature tensor to obtain tensor ; tensor is sequentially put into convolution layer, batch normalization layer, RELU nonlinear layer, convolution layer, batch normalization layer to obtain tensor ; tensor is added to and then put into RELU nonlinear layer to obtain tensor ; tensor is sequentially put into convolution layer, batch normalization layer, RELU nonlinear layer, convolution layer, batch normalization layer to obtain tensor ; tensor is added to and then put into RELU nonlinear layer to obtain feature tensor ;

[0015] Step (4), the feature tensor is sequentially put into convolution layer, batch normalization layer, RELU nonlinear layer, convolution layer, batch normalization layer to obtain tensor ; the feature tensor ​Put into the convolution layer, batch normalization layer in turn, get the tensor , Add with , put into the RELU nonlinear layer, get the tensor ; Put the tensor Into the convolution layer, batch normalization layer, RELU nonlinear layer, convolution layer, batch normalization layer in turn, get the tensor ; Put the tensor Add with , put into the RELU nonlinear layer, get the feature tensor .

[0016] Preferably, the running step of the decoder in the encoder-decoder network in step 1 is:

[0017] Step i, put the feature tensor Into the convolution layer, ELU nonlinear layer, convolution layer and Pixel Shuffle layer in turn, get the tensor ; Put the tensor With the feature tensor Splicing, get the tensor ;

[0018] Step ii, put the tensor Into the convolution layer, ELU nonlinear layer, convolution layer, ELU nonlinear layer, convolution layer and Pixel Shuffle layer in turn, get the tensor . Put the tensor With the feature tensor Splicing, get the tensor ;

[0019] Step iii, put the tensor Into the convolution layer, ELU nonlinear layer, convolution layer, ELU nonlinear layer, convolution layer Pixel Shuffle layer and convolution layer in turn, get the final result, that is, the output of the encoder-decoder network .

[0020] Preferably, the specific process of the encoder-decoder network training part in step 2 is:

[0021] Step a, set the optimizer to Adam, set the loss function to smooth L1 function, train the network for m rounds in total, each round contains s / n batches, and each batch contains n groups of data; The initial learning rate is set to L, and the learning rate is multiplied by a number z after every x rounds. It should be noted that when s / n is not divisible, the excess is put into a new batch;

[0022] Step b, the specific process of each batch in each round is as follows:

[0023] b-1, encode the noise and the reference image into a variational autoencoder input into the encoder-decoder network to obtain an output result

[0024] b-2, put the output result and the difference between the noise and the noise into the loss function together

[0025] b-3, according to the loss value, update the encoder-decoder network weight using the Adam optimizer

[0026] Step c, after each round of training, test y groups of data using the current encoder-decoder network weight, calculate the loss value of each group of data, and obtain the average value of the y groups of data loss value. The network weight with the lowest average value is retained as the optimal weight; the calculation process of the loss value of each group of data is as follows:

[0027] c-1, encode the noise and the reference image into a variational autoencoder input into the encoder-decoder network to obtain an output result

[0028] c-2, put the output result and the difference between the noise and the noise into the loss function together

[0029] Advantages

[0030] The method of the application changes the initial noise of the new view synthesis model based on the diffusion model into optimized noise through the encoder-decoder network, improves the generation result of the model, and has the following advantages: the optimized noise obtained through the encoder-decoder network can add more details to the generation result of the new view synthesis model based on the diffusion model, and improve the quality of the generation result; the method does not need to fine-tune the new view synthesis model based on the diffusion model, greatly improving the efficiency. At the same time, the addition of the encoder-decoder network only slightly increases the inference time; strong generalization ability, the method can be applied to various new view synthesis models based on the diffusion model. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is the overall structure of the encoder-decoder network.

[0032] Figure 2 is the specific structure of the encoder in the encoder-decoder network.

[0033] Figure 3 ​​This describes the specific structure of the decoder in the encoder-decoder network.

[0034] Figure 4 This is a schematic diagram of inserting an encoder-decoder network into a new perspective synthesis model.

[0035] Figure 5 The following is a comparison of the noise effect after processing by the method of the present invention: (a) is the initial noise, and (b) is the noise after processing. Detailed Implementation

[0036] A novel method for optimizing initial noise in a synthetic model based on a diffusion model is proposed, the specific process of which is as follows:

[0037] Step 1: Based on the dataset collected by the Sv3d synthetic model, a novel perspective model based on a diffusion model,

[0038] The specific content of the dataset is: noise. The shape is (21, 4, 72, 72), corresponding to the reference image, and is composed of noise. The obtained real optimized noise rich in image semantic information The shape is (21, 4, 72, 72). The reference image is a single PNG image with a 3D main object and a transparent background, in RGBA format, and a size of 576×576. The reference image is encoded by a variational autoencoder obtained from a novel perspective synthesis model based on a diffusion model. The shape is (21, 4, 72, 72). The real optimized noise in this embodiment is obtained by conventional semantic information addition methods in the art. The training set in this embodiment contains 359 sets of data.

[0039] Step 2: Train the encoder-decoder network using the dataset collected from the novel perspective synthesis model Sv3d based on a diffusion model. The encoder-decoder network consists of two parts: an encoder and a decoder.

[0040] Specifically,

[0041] Step (1): The noise in group h... Variational autoencoder encoding with reference image The data is stitched together in the second dimension. This variational autoencoder is derived from the Sv3d model. The shape of the stitched result is (21×h, 8, 72, 72), where h is the number of data sets input each time.

[0042] Step (2): The concatenated result is processed through a convolutional layer (i.e., a Conv2d layer, with 8 input channels, 64 output channels, a kernel size of 7×7, a stride of 2×2, padding of 3×3, and no bias term), a batch normalization layer (i.e., a BatchNorm2d layer, with 64 feature channels, a numerical stability term set to 1e−5, a momentum of 0.1, using learnable affine transformation parameters, and tracking the mean and variance during runtime), and a ReLU nonlinear layer to obtain the feature tensor. The shape is (21×h, 64, 36, 36).

[0043] Step (3): For the feature tensor Perform max pooling (i.e., a MaxPool2d layer with a 3×3 kernel size, a stride of 2, a padding of 1, an inflation factor of 1, and without using ceil mode) to obtain a tensor. The shape is (21×h, 64, 18, 18). The tensor... The following layers are sequentially added: a convolutional layer (Conv2d layer, 64 input channels, 64 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), a batch normalization layer (BatchNorm2d layer, 64 feature channels, numerical stability term set to 1e−5, momentum 0.1, using learnable affine transformation parameters, and tracking the mean and variance during runtime), a ReLU nonlinear layer, another convolutional layer (Conv2d layer, 64 input channels, 64 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), and another batch normalization layer (BatchNorm2d layer, 64 feature channels, numerical stability term set to 1e−5, momentum 0.1, using learnable affine transformation parameters, and tracking the mean and variance during runtime), resulting in a tensor. The shape is (21×h, 64, 18, 18); the tensor With tensor After element-wise addition, the result is placed in a ReLU nonlinear layer to obtain a tensor. The shape is (21×h, 64, 18, 18); the tensor The following layers are sequentially added: a convolutional layer (Conv2d layer, 64 input channels, 64 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), a batch normalization layer (BatchNorm2d layer, 64 feature channels, numerical stability term set to 1e−5, momentum 0.1, using learnable affine transformation parameters, and tracking the mean and variance during runtime), a ReLU nonlinear layer, another convolutional layer (Conv2d layer, 64 input channels, 64 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), and another batch normalization layer (BatchNorm2d layer, 64 feature channels, numerical stability term set to 1e−5, momentum 0.1, using learnable affine transformation parameters, and tracking the mean and variance during runtime), resulting in a tensor. The shape is (21×h, 64, 18, 18); the tensor With tensor After element-wise addition, the result is fed into a ReLU nonlinear layer to obtain the feature tensor. The shape is (21×h, 64, 18, 18).

[0044] Step (4): Convert the feature tensor The following layers are sequentially added: a convolutional layer (Conv2d layer, 64 input channels, 128 output channels, 3×3 kernel, 2×2 stride, 1×1 padding, no bias term), a batch normalization layer (BatchNorm2d layer, 128 feature channels, numerical stability term set to 1e−5, momentum 0.1, using learnable affine transformation parameters, and tracking the mean and variance during runtime), a ReLU nonlinear layer, another convolutional layer (Conv2d layer, 128 input channels, 128 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), and another batch normalization layer (BatchNorm2d layer, 128 feature channels, numerical stability term set to 1e−5, momentum 0.1, using learnable affine transformation parameters, and tracking the mean and variance during runtime), resulting in a tensor. The shape is (21×h, 128, 9, 9); the feature tensor The following layers are sequentially inserted: a convolutional layer (Conv2d layer, 64 input channels, 128 output channels, 1×1 kernel, 2×2 stride, no bias term) and a batch normalization layer (BatchNorm2d layer, 128 feature channels, numerical stability term set to 1e−5, momentum 0.1, using learnable affine transformation parameters, and tracking the mean and variance during runtime), to obtain a tensor. The shape is (21×h, 128, 9, 9). and After element-wise addition, the result is placed in a ReLU nonlinear layer to obtain a tensor. The shape is (21×h, 128, 9, 9); the tensor The following layers are sequentially added: a convolutional layer (Conv2d layer, 128 input channels, 128 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), a batch normalization layer (BatchNorm2d layer, 128 feature channels, numerical stability term set to 1e−5, momentum 0.1, using learnable affine transformation parameters, and tracking the mean and variance during runtime), a ReLU nonlinear layer, another convolutional layer (Conv2d layer, 128 input channels, 128 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), and another batch normalization layer (BatchNorm2d layer, 128 feature channels, numerical stability term set to 1e−5, momentum 0.1, using learnable affine transformation parameters, and tracking the mean and variance during runtime), resulting in a tensor. The shape is (21×h, 128, 9, 9); the tensor and After element-wise addition, the result is fed into a ReLU nonlinear layer to obtain the feature tensor. The shape is (21×h, 128, 9, 9).

[0045] Step (5): Convert the feature tensor The following layers are sequentially added: a convolutional layer (Conv2d layer, 128 input channels, 64 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), an ELU nonlinear layer (ELU activation layer, α parameter 1.0, in-place operation), a convolutional layer (Conv2d layer, 64 input channels, 256 output channels, 1×1 kernel, 1×1 stride, no bias term, initialized with ICNR), and a pixel shuffle layer (PixelShuffle layer, magnification factor 2), to obtain a tensor. The shape is (21×h, 64, 18, 18); the tensor With feature tensor By splicing, a tensor is obtained. The shape is (21×h, 128, 18, 18); the tensor The following layers are sequentially added: a convolutional layer (Conv2d layer, 128 input channels, 64 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), an ELU nonlinear layer (ELU activation layer, α parameter 1.0, in-place operation), a convolutional layer (Conv2d layer, 64 input channels, 32 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), an ELU nonlinear layer (ELU activation layer, α parameter 1.0, in-place operation), a convolutional layer (Conv2d layer, 32 input channels, 128 output channels, 1×1 kernel, 1×1 stride, no bias term, initialized with ICNR), and a pixel shuffle layer (PixelShuffle layer, magnification factor 2), to obtain a tensor. The shape is (21×h, 32, 36, 36); the tensor With feature tensor By splicing, a tensor is obtained. The shape is (21×h, 96, 36, 36).

[0046] Step (6): Convert the tensor The following layers are sequentially placed: a convolutional layer (Conv2d layer, 96 input channels, 32 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), an ELU nonlinear layer (ELU activation layer, α parameter 1.0, in-place operation), a convolutional layer (Conv2d layer, 32 input channels, 16 output channels, 3×3 kernel, 1×1 stride, 1×1 padding, no bias term), an ELU nonlinear layer (ELU activation layer, α parameter 1.0, in-place operation), a convolutional layer (Conv2d layer, 16 input channels, 64 output channels, 1×1 kernel, 1×1 stride, no bias term, initialized with ICNR), and a PixelShuffle layer (PixelShuffle...). The encoder-decoder network consists of two layers (with a magnification of 2) and convolutional layers (Conv2d layers, 16 input channels, 4 output channels, 3×3 kernel size, 1×1 stride, 1×1 padding, and no bias term) to obtain the final result, which has a shape of (21×h, 4, 72, 72). ;

[0047] The training process is as follows:

[0048] Step (7) Set the optimizer to Adam, the loss function to the smooth L1 function, and train the network for a total of 600 rounds. Each round contains 45 batches (if the number is not even, the extra batches are put into a new batch), and each batch contains 8 sets of data. The initial learning rate is set to 0.0003, and after every 200 rounds, the learning rate is multiplied by a number 0.8.

[0049] The specific process for each round and each batch in step (8) is as follows:

[0050] a. Noise Variational autoencoder encoding of the reference image The input is fed into an encoder-decoder network to obtain the output result. b. Output the results With noise and noise The difference between them is put into the loss function; c. Based on the loss value, the Adam optimizer is used to update the encoder-decoder network weights;

[0051] Step (9) After each round of training, use the current encoder-decoder network weights to test the y (y=30) groups of data, calculate the loss value of each group of data, and obtain the average value of the loss value of the y groups of data. Keep the network weight with the lowest average value as the optimal weight; the calculation process of the loss value of each group of data is as follows: a. Remove noise Variational autoencoder encoding of the reference image The input is fed into an encoder-decoder network to obtain the output result. b. Output the results With noise and noise The differences between them are included in the loss function;

[0052] A novel perspective synthesis model running an insertion encoder-decoder network with initial noise Divide by coefficient get , Variational autoencoder encoding with reference image It will be input into the encoder-decoder network, and the network output will be... With noise Add element by element and then multiply by the coefficient. This result is the predicted optimization noise. Predicted optimization noise It will replace the initial noise The input is fed into a new perspective synthesis model denoising network to generate 21 3D object images from different perspectives. (Coefficients) The parameter values ​​in this embodiment are preset for the synthetic network based on the new perspective of the diffusion model. It is 700.0007.

[0053] The optimized noise and the initial noise were fed into the Sv3d model, and the final partial results are as follows: Figure 5 As shown in the figure, the image obtained by optimizing the noise is clearer and has richer details compared to the image obtained with the initial noise.

Claims

1. A novel method for optimizing initial noise in a synthetic model based on a diffusion model, characterized in that, Includes the following steps: Step 1: Based on the dataset collected by the new perspective synthetic model based on the diffusion model; Step 2: Train the encoder-decoder network according to the loss function, and run the new perspective synthesis model inserted into the encoder-decoder network, specifically the initial noise. Divide by coefficient get ,coefficient Preset parameter values ​​for a new perspective synthetic model based on a diffusion model. Variational autoencoder encoding with reference image The output result is fed into the encoder-decoder network. Then output the results With noise Add element by element and then multiply by the coefficient. The predicted optimized noise is obtained. Predicted optimization noise Replace initial noise The input is fed into the denoising network of the new perspective synthesis model to generate multiple 3D object images from different perspectives.

2. The initial noise optimization method for a novel perspective synthesis model based on a diffusion model according to claim 1, characterized in that, The dataset in step 1 includes: noise The reference image corresponding to the noise The obtained real optimized noise rich in image semantic information The reference image is encoded by a variational autoencoder obtained through a novel perspective synthesis model based on a diffusion model. .

3. The initial noise optimization method for a novel perspective synthesis model based on a diffusion model according to claim 1, characterized in that, The encoder-decoder network in step 2 consists of two parts: an encoder and a decoder.

4. The initial noise optimization method for a novel perspective synthesis model based on a diffusion model according to claim 3, characterized in that, The operation steps of the encoder in the encoder-decoder network in step 2 are as follows: Step (1) Noise Variational autoencoder encoding with reference image To splice; Step (2): The concatenated result is processed by a convolutional layer, a batch normalization layer, and a ReLU nonlinear layer to obtain the feature tensor. ; Step (3): For the feature tensor Perform max pooling to obtain tensors ; Tensor By sequentially inserting convolutional layers, batch normalization layers, ReLU nonlinear layers, convolutional layers, and batch normalization layers, a tensor is obtained. ; Tensor and After addition, a ReLU nonlinear layer is inserted to obtain a tensor. ; Tensor By sequentially inserting convolutional layers, batch normalization layers, ReLU nonlinear layers, convolutional layers, and batch normalization layers, a tensor is obtained. ; Tensor and After addition, the result is fed into a ReLU nonlinear layer to obtain the feature tensor. ; Step (4): Convert the feature tensor By sequentially inserting convolutional layers, batch normalization layers, ReLU nonlinear layers, convolutional layers, and batch normalization layers, a tensor is obtained. ; the feature tensor By sequentially inserting convolutional layers and batch normalization layers, tensors are obtained. , and After addition, a ReLU nonlinear layer is inserted to obtain a tensor. ; Tensor By sequentially inserting convolutional layers, batch normalization layers, ReLU nonlinear layers, convolutional layers, and batch normalization layers, a tensor is obtained. ; Tensor and After addition, the result is fed into a ReLU nonlinear layer to obtain the feature tensor. .

5. The initial noise optimization method for a novel perspective synthesis model based on a diffusion model according to claim 1, characterized in that, The operation steps of the decoder in the encoder-decoder network in step 2 are as follows: Step i: Convert the feature tensor By sequentially adding convolutional layers, ELU nonlinear layers, convolutional layers, and pixel shuffle layers, a tensor is obtained. ; Tensor With feature tensor By splicing, a tensor is obtained. ; Step ii: Convert the tensor By sequentially inserting convolutional layers, ELU nonlinear layers, convolutional layers, ELU nonlinear layers, convolutional layers, and pixel shuffle layers, a tensor is obtained. ; Tensor With feature tensor By splicing, a tensor is obtained. ; Step iii: Convert the tensor By sequentially inserting convolutional layers, ELU nonlinear layers, convolutional layers, ELU nonlinear layers, convolutional pixel shuffle layers, and another convolutional layer, the final result is obtained, which is the output of the encoder-decoder network. .

6. The initial noise optimization method for a novel perspective synthesis model based on a diffusion model according to claim 1, characterized in that, The specific process of the encoder-decoder network training part in step 2 is as follows: Step a, set the optimizer to Adam, the loss function to the smooth L1 function, the network is trained for a total of m rounds, each round contains s / n batches, and each batch contains n sets of data; the initial learning rate is set to L, and after every x rounds, the learning rate is multiplied by a number z. It should be noted that when s / n is not divisible, the batch is directly incremented by 1. Step b, the specific process for each round and each batch is as follows: b-1, Noise Variational autoencoder encoding of the reference image The input is fed into an encoder-decoder network to obtain the output result. ; b-2, Output the results With noise and noise The differences between them are included in the loss function; b-3. Based on the loss value, use the Adam optimizer to update the encoder-decoder network weights; Step c: After each round of training, test the y groups of data using the current encoder-decoder network weights, calculate the loss value for each group of data, and obtain the average loss value of the y groups of data. Retain the network weights with the lowest average loss value as the optimal weights; the calculation process for the loss value of each group of data is as follows: c-1, Noise Variational autoencoder encoding of the reference image The input is fed into an encoder-decoder network to obtain the output result. ; c-2, Output the results With noise and noise The differences between them are included in the loss function.

Images