Three-dimensional reconstruction method of frequency domain optimization sparse view based on epipolar features and conditional diffusion model

Abstract

The invention relates to a three-dimensional reconstruction method of a frequency domain optimization sparse view based on epipolar features and a conditional diffusion model. The method comprises the following steps: S1, selecting a plurality of images with camera pose information from an image library comprising all information of an object as input images P1; s2, extracting a feature map from the input image P1 by using a CNN network, and carrying out auxiliary feature extraction by adopting multi-layer linear attention; s3, extracting features from the extracted feature map by using an antipode feature method to obtain features; and S4, taking the features acquired by each input image P1 as training conditions of a conditional diffusion model, gradually adding Gaussian noise into the features, and training the conditional diffusion model to recover noise images. According to the method, the neural radiation field is optimized by using the epipolar feature method and the conditional diffusion model, so that the training speed of the diffusion model can be accelerated, the problems of artifacts and fuzziness are effectively improved, and the reconstruction effect can be more natural.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a three-dimensional reconstruction method of frequency domain optimized sparse views based on epipolar features and a conditional diffusion model. Background Art

[0002] As an important branch of computer vision, 3D reconstruction has always received extensive attention from researchers and has important application potential in the field of industrial design. The NeRF neural radiation field and a series of models derived from it have demonstrated powerful capabilities in the field of 3D reconstruction. They can achieve high-quality 3D reconstruction of specified objects in actual scenes by learning the color density information of specified objects in actual scenes. However, this process requires a large amount of view information for learning and training, which greatly increases resource consumption. In actual applications, the view of the specified object in the actual scene is incomplete and the amount of information is not complete, which leads to unsatisfactory 3D reconstruction effects.

[0003] Therefore, sparse view 3D reconstruction has received widespread attention. However, sparse views lack a lot of information compared to multiple views. The traditional neural radiation field single method 3D reconstruction effect is not ideal, and it is prone to noise artifacts. More importantly, the generated views and 3D results are different from the real scene. A single training neural radiation field consumes a lot of resources, so other auxiliary methods are often used to optimize the neural radiation field.

[0004] For example, although the conditional diffusion model cannot generate 3D scenes, after training, it can better obtain new views of specified objects in actual scenes, which provides the missing information for the neural radiation field. However, the diffusion model also has certain limitations. For example, when using the original view for training, due to the large amount of information, the training speed is slow, and there are certain differences between the generated image and the original image. Summary of the invention

[0005] The purpose of the present invention is to provide a frequency domain optimized sparse view 3D reconstruction method based on epipolar features and conditional diffusion model, which can be used to generate realistic and detail-rich scene images and perform 3D reconstruction under the condition of sparse known views.

[0006] The object of the present invention is achieved by the following technical solution: A three-dimensional reconstruction method of frequency domain optimized sparse views based on epipolar features and conditional diffusion model, comprising the following steps:

[0007] S1. Select several images with camera pose information from a gallery containing all information of an object as input image P1;

[0008] S2. Use CNN network to extract feature maps of input image P1, and use LiteMLA multi-layer linear attention for auxiliary feature extraction;

[0009] S3. Extract features from the extracted feature map using the epipolar feature method to obtain features;

[0010] S4. The features obtained from each input image P1 are used as training conditions for the conditional diffusion model, and Gaussian noise is gradually added to the above features. The conditional diffusion model is trained to restore the noisy image, and the above steps are repeated to finally obtain a pre-trained conditional diffusion model based on the antipodal features;

[0011] S5. Establish a NeRF neural radiation field based on hash coding and volume rendering, realize three-dimensional reconstruction according to the input image P1, and generate a new perspective image P2 through neural network rendering;

[0012] S6. Input the new perspective image P2 generated by the NeRF neural radiation field into the pre-trained conditional diffusion model to generate a new perspective image P3;

[0013] S7. Use frequency domain optimization to compare the new perspective image P2 generated by the NeRF neural radiation field and the new perspective image P3 generated by the pre-trained conditional diffusion model, and obtain the difference loss of the contrast images;

[0014] S8. Repeat steps S5-S7, use the difference loss in step S7 to update the parameters of the NeRF neural radiation field, and use PSNR, SSIM, and LPIPS to compare the new perspective image generated by the NeRF neural radiation field with the original image to evaluate the final 3D reconstruction quality.

[0015] Compared with the prior art, the advantages of the present invention are:

[0016] The present invention uses a multi-layer linear attention mechanism to strengthen feature extraction, and uses the polar feature method to extract features, so that the conditional diffusion model can understand the reconstructed object more comprehensively, is more conducive to the reconstruction of the object, and has better robustness than other methods.

[0017] The present invention uses the extreme features to train the conditional diffusion model, which can not only provide a more comprehensive understanding of the object and achieve better view generation, but also accelerate the training speed because the conditional diffusion model processes features faster than images.

[0018] The present invention uses a frequency domain optimization method to compare the relative distribution distance of images in the frequency domain to optimize the model, which can better focus on local details. Compared with traditional perceptual loss, overfitting is greatly reduced.

[0019] The present invention is not limited to the three-dimensional reconstruction of the neural radiation field. The use of the epipolar characteristic method and the conditional diffusion model to optimize the neural radiation field can not only accelerate the training speed of the diffusion model, effectively improve the artifact and blur problems, but also make the reconstruction effect more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The present invention is a flowchart of an embodiment of a method for three-dimensional reconstruction of frequency-domain optimized sparse views based on epipolar features and a conditional diffusion model.

[0021] Figure 2 Schematic diagram of the overall structure of LiteMLA multi-layer linear attention.

[0022] Figure 3 Schematic diagram of the principle of the pole characteristic method.

[0023] Figure 4 Schematic diagram of training and sampling of the conditional diffusion model.

[0024] Figure 5 Schematic diagram of the frequency domain optimization method. DETAILED DESCRIPTION

[0025] The present invention is described in detail below with reference to the accompanying drawings and embodiments:

[0026] like Figure 1-5 It is a schematic diagram of an embodiment of a method for three-dimensional reconstruction of frequency domain optimized sparse views based on epipolar features and conditional diffusion model provided by the present invention.

[0027] A 3D reconstruction method of frequency domain optimized sparse views based on epipolar features and conditional diffusion model, comprising the following steps:

[0028] Step S1

[0029] Select several images with camera pose information from a library containing all information of an object as input images P1;

[0030] The gallery contains at least 200 images, each with corresponding camera pose information.

[0031] Step S2

[0032] The CNN network (convolutional neural network) is used to extract feature maps of the input image P1, and LiteMLA multi-layer linear attention is used for auxiliary feature extraction;

[0033] The process of LiteMLA multi-layer linear attention assisted feature extraction is as follows:

[0034] S2-1. Project the feature map extracted by the CNN network through a linear layer to generate three sets of features: Q (Query), K (Key), and V (Value);

[0035] S2-2. Use three small kernel convolutions to aggregate local features of Q, K, and V to generate multi-scale Q, K, and V;

[0036] S2-3. Use ReLU linear attention to calculate the attention scores of three small kernel convolution local feature aggregations;

[0037] S2-4. Concatenate or fuse the attention scores of the three small kernel convolution local features in the feature dimension;

[0038] S2-5. Input the fusion result into the linear layer for projection to generate the final output feature map.

[0039] The principle of LiteMLA multi-layer linear attention is as follows Figure 2 As shown, the sizes of the three small cores are 1*1, 3*3, and 5*5 respectively.

[0040] Step S3

[0041] The extracted feature map is used to extract features using the epipolar feature method to obtain the features:

[0042] y=h ψ (π,C)

[0043] Where π represents the pose of the given target camera, C represents the input image, and h ψ is the extremum feature extraction method, and y represents the acquired features.

[0044] Step S4

[0045] The features obtained from each input image P1 are used as training conditions for the conditional diffusion model, and Gaussian noise is gradually added to the above features. The conditional diffusion model is trained to restore the noisy image. The above steps are repeated to finally obtain a pre-trained conditional diffusion model based on the antipodal features.

[0046] The training process of the conditional diffusion model is:

[0047] L OSS =E z,∈～N(0,1),t,y [||∈-∈ φ (z t ,t,y)||^ 2 ]

[0048] Where E is the expected value of the calculated random variable, z is a random Gaussian noise with a mean of 0 and a variance of 1, and z tis z after adding noise after t time steps, t is the sampling time step, y is the feature extracted by the antipodal feature method, where ∈ and ∈ φ are the variables to which noise is added and the noise variables predicted given the feature y and time step t, respectively.

[0049] Step S5

[0050] A NeRF neural radiation field based on hash coding and volume rendering is established. After three-dimensional reconstruction is achieved based on the input image P1, a new perspective image P2 is generated through neural network rendering.

[0051] Step S6

[0052] The new perspective image P2 generated by the NeRF neural radiation field is input into the pre-trained conditional diffusion model to generate a new perspective image P3;

[0053] Step S7

[0054] Frequency domain optimization is used to compare the new perspective image P2 generated by the NeRF neural radiation field and the new perspective image P3 generated by the pre-trained conditional diffusion model, and the difference loss of the contrast images is obtained;

[0055] The process of frequency domain optimization is as follows:

[0056] The images to be compared are subjected to discrete Fourier transform and transferred to the frequency domain space to obtain the amplitude component and the phase component. The amplitude components X2 and X3 are used to predict the distribution distance difference of the frequency domain features of the two images through the SWD function. The difference loss of the compared images is finally obtained as:

[0057] L Floss (P2,P3)=SWD(X2,X3)

[0058] Among them, P2 is the new perspective image generated by the NeRF neural radiation field, P3 is the new perspective image generated in the pre-trained conditional diffusion model, X2 is the amplitude component extracted by discrete Fourier transform based on P2, and X3 is the amplitude component extracted by discrete Fourier transform based on P3. SWD is used to calculate the distribution distance of two amplitude components to measure image similarity.

[0059] Step S8

[0060] Repeat steps S5-S7, use the difference loss in step S7 to update the parameters of the NeRF neural radiation field, and use PSNR, SSIM, and LPIPS to compare the new perspective image generated by the NeRF neural radiation field with the original image to evaluate the final 3D reconstruction quality.

[0061] In step S5, a NeRF neural radiation field based on hash coding and volume rendering is first established, and only the parameters of the NeRF neural radiation field are updated in subsequent repetitions.

[0062] The above description is only a preferred specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and shall be included in the protection scope of the present invention.

Claims

1. A 3D reconstruction method of frequency domain optimized sparse views based on epipolar features and conditional diffusion model, characterized in that: The following steps are involved: S1. Select several images with camera pose information from a gallery containing all information of an object as input image P1; S2. Use CNN network to extract feature maps of input image P1, and use LiteMLA multi-layer linear attention for auxiliary feature extraction; S3. Extract features from the extracted feature map using the epipolar feature method to obtain features; S4. The features obtained from each input image P1 are used as training conditions for the conditional diffusion model, and Gaussian noise is gradually added to the above features. The conditional diffusion model is trained to restore the noisy image, and the above steps are repeated to finally obtain a pre-trained conditional diffusion model based on the antipodal features; S5. Establish a NeRF neural radiation field based on hash coding and volume rendering, realize three-dimensional reconstruction according to the input image P1, and generate a new perspective image P2 through neural network rendering; S6. Input the new perspective image P2 generated by the NeRF neural radiation field into the pre-trained conditional diffusion model to generate a new perspective image P3; S7. Use frequency domain optimization to compare the new perspective image P2 generated by the NeRF neural radiation field and the new perspective image P3 generated by the pre-trained conditional diffusion model, and obtain the difference loss of the contrast images; S8. Repeat steps S5-S7, use the difference loss in step S7 to update the parameters of the NeRF neural radiation field, and use PSNR, SSIM, and LPIPS to compare the new perspective image generated by the NeRF neural radiation field with the original image to evaluate the final 3D reconstruction quality.

2. The method for 3D reconstruction of frequency domain optimized sparse views based on epipolar features and conditional diffusion model according to claim 1, characterized in that: The process of LiteMLA multi-layer linear attention-assisted feature extraction in step S2 is as follows: S2-1. Project the feature map extracted by the CNN network through a linear layer to generate three sets of features: Q, K, and V; S2-2. Use three small kernel convolutions to aggregate local features of Q, K, and V to generate multi-scale Q, K, and V; S2-3. Use ReLU linear attention to calculate the attention scores of three small kernel convolution local feature aggregations; S2-4. Concatenate or fuse the attention scores of the three small kernel convolution local features in the feature dimension; S2-5. Input the fusion result into the linear layer for projection to generate the final output feature map.

3. The method for 3D reconstruction of frequency domain optimized sparse views based on epipolar features and conditional diffusion model according to claim 1, characterized in that: The training process of the conditional diffusion model in step S4 is: L OSS =E z,∈～N(0,1),t,y [||∈-∈ φ (z t ,t,y)||^ 2 ] Where E is the expected value of the calculated random variable, z is a random Gaussian noise with a mean of 0 and a variance of 1, and z t is z after adding noise after t time steps, t is the sampling time step, y is the feature extracted by the antipodal feature method, where ∈ and ∈ φ are the variables to which noise is added and the noise variables predicted given the feature y and time step t, respectively.

4. The method for 3D reconstruction of frequency domain optimized sparse views based on epipolar features and conditional diffusion model according to claim 1, characterized in that: The process of frequency domain optimization in step S7 is as follows: The images to be compared are subjected to discrete Fourier transform and transferred to the frequency domain space to obtain the amplitude component and the phase component. The amplitude components X2 and X3 are used to predict the distribution distance difference of the frequency domain features of the two images through the SWD function. The difference loss of the compared images is finally obtained as: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> Floss <h2 style=";text-align:left;direction:ltr"> (P2,P3) = SWD(X2,X3) Among them, P2 is the new perspective image generated by the NeRF neural radiation field, P3 is the new perspective image generated in the pre-trained conditional diffusion model, X2 is the amplitude component extracted by discrete Fourier transform based on P2, and X3 is the amplitude component extracted by discrete Fourier transform based on P3. SWD is used to calculate the distribution distance of two amplitude components to measure image similarity.

5. The method for 3D reconstruction of frequency domain optimized sparse views based on epipolar features and conditional diffusion model according to claim 1, characterized in that: The image gallery in step S1 includes at least 200 images, each of which has corresponding camera pose information.

Images