Monocular image three-dimensional model reconstruction method based on diffusion model and neural radiation field

Abstract

The invention discloses a monocular image three-dimensional model reconstruction method based on a diffusion model and a neural radiation field. The monocular image three-dimensional model reconstruction method comprises the following steps: S1, constructing a Zero-1-to-3 diffusion model network; s2, constructing a neural radiation field network; s3, constructing a two-dimensional neural rendering module network; s4, using a pre-trained Zero-1-to-3 diffusion model, taking the monocular image as input, and obtaining a plurality of images at different angles through the diffusion model; s5, training the neural radiation field network, minimizing loss through back propagation and optimization, and obtaining a reconstruction result through volume rendering; s6, rendering the reconstructed image from a low-resolution image to a high-resolution image by using a two-dimensional neural rendering module network; and S7, taking the rendered multi-view image as the input of the neural radiation field network again, and obtaining a final reconstruction result. According to the method, the technical problem of performing three-dimensional reconstruction on the monocular image can be effectively solved, and the reconstruction quality is further improved while the reconstruction quality is ensured.

Description

Technical Field

[0001] The present invention relates to a method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field. Background Art

[0002] 3D reconstruction is a key research area in computer vision and graphics. It aims to restore the geometric structure and appearance of real-world 3D scenes from 2D images, videos, or other sensor data. With the recent surge in popularity of concepts like the metaverse and digital humans, 3D model reconstruction has become a hot topic in the technology.

[0003] Neural radiance field is an emerging 3D reconstruction method based on neural networks. It is used to synthesize high-quality new perspectives from input images at different viewpoints, thereby achieving 3D reconstruction. Neural radiance field uses the volume rendering technology in computer graphics. It regards the 3D scene as a continuous function represented by the radiation field (implicitly represented, using a function to describe the scene geometry). The radiation field defines the radiation intensity and color when observing the scene from any direction and position. To model the radiation field, neural radiance field uses a deep neural network structure that maps the 3D position and viewing direction of the input scene to radiation field values. Finally, the radiation field is sampled using volume rendering technology to reconstruct the scene.

[0004] The Neural Radiance Field method has been demonstrated to be effective through extensive qualitative and quantitative experiments. The introduction of the Neural Radiance Field method is a significant contribution to computer vision and computer graphics, a successful example of integrating deep learning with computer graphics, and provides a new approach to 3D reconstruction, inspiring subsequent research by numerous researchers.

[0005] Although the neural radiance field method has achieved impressive results, it also has some shortcomings and limitations:

[0006] 1. Requires a large number of viewpoints: During the training phase, Neural Radiance Field requires input images and corresponding camera parameters from many different viewpoints. This means that we need to collect a large amount of data from different viewpoints to train the network model. However, the vast majority of existing open-source image datasets are monocular images (images from a single viewpoint). This makes 3D reconstruction from monocular images a challenge for Neural Radiance Field.

[0007] 2. Issues with 3D reconstruction quality: Neural radiance field methods require high coverage of different viewpoints. Reconstruction results can exhibit significant errors, especially for less frequently observed viewpoints. If the training data lacks coverage of certain viewpoints, neural radiance field methods may generate inaccurate or unrealistic results for these viewpoints, resulting in poor quality reconstructed 3D models.

[0008] Therefore, how to reconstruct a high-quality three-dimensional model from multiple perspectives from a monocular image and improve the quality of multi-perspective generation of the neural radiation field has become a problem that needs to be solved in this technology. Summary of the Invention

[0009] The purpose of the present invention is to overcome the existing defects and provide a method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field, which can effectively solve the technical problem of three-dimensional reconstruction of a monocular image, and ensure the reconstruction quality while further improving the reconstruction quality.

[0010] The technical solution to achieve the above purpose is:

[0011] A method for reconstructing a three-dimensional model from a monocular image based on a diffusion model and a neural radiation field, comprising:

[0012] Step S1, constructing a Zero-1-to-3 (a framework that achieves the goal of changing the camera viewpoint of an object by providing only a single RGB image) diffusion model network;

[0013] Step S2, constructing a neural radiation field network;

[0014] Step S3, constructing a two-dimensional neural rendering module network;

[0015] Step S4: Using a pre-trained Zero-1-to-3 diffusion model, the monocular image is used as input and multiple images at different angles are obtained through the diffusion model.

[0016] Step S5, training the neural radiation field network, minimizing the loss using back propagation and optimization, and obtaining the reconstruction result through volume rendering;

[0017] Step S6, using a two-dimensional neural rendering module network to render the reconstructed image from a low-resolution image to a high-resolution image;

[0018] In step S7, the rendered multi-view image is used as input to the neural radiance field network again to obtain the final reconstruction result.

[0019] Preferably, in step S1, constructing a Zero-1-to-3 diffusion model network includes:

[0020] Given a single RGB image of an object make and are the relative camera rotation and translation of the desired viewpoint, respectively, and the goal is to learn a diffusion model f that synthesizes a new image under this camera transformation as shown below:

[0021]

[0022] Where, For the composite image.

[0023] Preferably, in step S2, a neural radiation field network is constructed, and an implicit rendering process is constructed by training a neural network model using the original neural radiation field network, including:

[0024] Neural Radiation Field Model F θ It receives the position, direction and three-dimensional coordinates of the light as input, and outputs the volume density and color radiation field properties. Finally, through the volume rendering algorithm, it synthesizes them into the final neural radiation field network according to the volume density and color information of the sampling points.

[0025] Preferably, in step S3, the two-dimensional neural rendering module network is mainly composed of a deconvolution layer and a bilinear upsampling network, and constructing the two-dimensional neural rendering module network includes:

[0026] Introducing the idea of ​​the deconvolution layer in DCGAN (Deep Convolutional Generative Adversarial Network) into the 2D neural rendering module network;

[0027] The upsampling layer in DCGAN enlarges the low-resolution feature map to a higher resolution, slides the filter on the input, and performs the opposite operation of convolution to expand the dimension of the input feature map.

[0028] Preferably, in step S4, a pre-trained Zero-1-to-3 diffusion model is used, a monocular image is taken as input, and multiple images at different angles are obtained through the diffusion model, including:

[0029] Given a dataset of paired images and their camera extrinsics {(x,x (R，T) ,R,T)}, fine-tune the diffusion model to learn control over camera parameters without destroying the rest of the representation;

[0030] Use a U-Net with encoder ∈ and denoiser ∈ θ and the potential diffusion architecture of decoder D;

[0031] At diffusion time step t ~ [1,1000], let c(x,R,T) be the embedding of the input view and the relative camera exterior, and solve the fine-tuned model using the following formula:

[0032]

[0033] After training the model θ, the diffusion model f generates images by performing iterative denoising from Gaussian noise images conditioned on c(x, R, T);

[0034] After passing this fine-tuned Zero-1-to-3 diffusion model, a monocular image is then generated into a multi-view image.

[0035] Preferably, in step S5, training a neural radiation field network, minimizing the loss by back propagation and optimization, and obtaining a reconstruction result by volume rendering comprises:

[0036] Represent a static scene as a continuous 5D function that outputs the value of each point (x, y, z) in each direction in space. The emitted radiance, and thus the density at each point, controls the accumulated radiance for rays passing through (x,y,x), i.e.:

[0037] Represent a continuous scene as a 5D vector-valued function whose input is a 3D position x = (x, y, z) and a 2D viewing direction Its output is the emitted color c = (r, g, b) and volume density σ;

[0038] Represent the direction as a 3D Cartesian unit vector d, use a multi-layer perceptron network to approximate the continuous 5D scene representation: (x, d) → (c, σ), and optimize its weights Emit color from each input 5D coordinate mapped to its corresponding volume density and direction;

[0039] Neural Radiance Field optimizes a deep fully connected neural network without any convolutional layers by Regressing to a single volume density and view-dependent RGB color representation of the function, that is:

[0040] The dataset of captured RGB images of the scene, the corresponding camera poses and intrinsic parameters, and the scene boundaries. For synthetic data, the real camera poses, intrinsic parameters and boundaries are used. For real data, the above parameters are estimated from the motion bag using the COLMAP structure.

[0041] In each optimization iteration, a batch of camera rays is randomly sampled from all pixels in the dataset, and for each ray a color is rendered from the coarse and fine sample sets;

[0042] The loss function is simply the total squared error between the rendered colors of the coarse and fine renderings and the true pixel colors. By optimizing and training the network parameters, the network model can learn the radiation field representation in the scene. In volume rendering, there are coarse rendering and fine rendering. The total squared error between the rendered and the true pixel colors is defined as:

[0043]

[0044] Where, is the set of each batch of rays, C(r), and are the ground truth value, rough volume prediction value and fine volume prediction value RGB color of ray r respectively;

[0045] Among them, the volume rendering equation can be divided into the ideal case and the actual situation of the algorithm, as shown in the following equations:

[0046]

[0047] δ i =t i+1 -t i ;

[0048] In the formula, C(r) represents the ideal case, which is to calculate the pixel color value by integration. represents the actual calculation method of the neural radiation field algorithm, which is approximated by summing the integral method. t is the distance between the sampling point and the origin, σ is the density value, T(t) is the opacity, c is the final rendered pixel color, r(s) is the parameterized light path, t n is the near boundary of the ray integration, t f is the far boundary of the ray integration, σ i is the density of the i-th sampling point, δ i is the distance between adjacent sampling points, c i is the color of the i-th sampling point, T i is the cumulative transmittance from the i-th sampling point to the camera, t i+1 is the distance value of the i+1th sampling point along the light, t i is the distance value of the i-th sampling point along the light;

[0049] Finally, the image in the corresponding posture is obtained through the volume rendering strategy.

[0050] Preferably, it is characterized in that, in said step S6, rendering the reconstructed image from a low-resolution image to a high-resolution image using a two-dimensional neural rendering module network comprises:

[0051] Combining the idea of ​​deconvolution layer and bilinear upsampling network, the combination of deconvolution layer and bilinear upsampling network is equivalent to an upsampling process, thereby restoring the image resolution.

[0052] Finally, the image after two-dimensional neural rendering is used as the input of the neural radiation field again, and a high-resolution reconstructed image is finally obtained.

[0053] Preferably, in step S7, the rendered multi-view image is used again as input to the neural radiance field network to obtain the final reconstruction result, including:

[0054] The high-resolution multi-view images processed by the 2D neural rendering module are again used as input to the neural radiance field network;

[0055] The neural radiance field network uses the improved multi-view images for further optimization and adjustment to obtain the final reconstruction results.

[0056] The beneficial effects of the present invention are:

[0057] The present invention generates multi-view images from a monocular image using a Zero-1-to-3 diffusion model. This is then reconstructed using a neural radiance field network for 3D reconstruction. This image quality is then improved using a 2D neural rendering module. Finally, the image is re-input into the neural radiance field network to generate a high-quality 3D model. This process fully leverages deep learning and rendering technology to achieve an end-to-end reconstruction process from monocular images to high-quality 3D models.

[0058] The proposed method addresses the neural radiance field algorithm's dependence on multi-view images by introducing a diffusion model. Traditional neural radiance field methods require images from multiple viewpoints as input, while the proposed method can use monocular images for 3D reconstruction by introducing a diffusion model. The diffusion model can infer the depth information of the scene from a single viewpoint image, thus overcoming the limitation of traditional methods requiring multi-view images. In this way, high-quality 3D reconstruction can be achieved using neural radiance fields without collecting a large number of images from different viewpoints.

[0059] By introducing a 2D neural rendering module network, the resolution quality of the reconstructed multi-view images is enhanced. The multi-view images generated by the traditional neural radiation field method may have a low resolution problem, resulting in unclear details in the reconstructed results. Therefore, the introduction of a 2D neural rendering module, drawing on the deconvolution layer in DCGAN, is used to improve the resolution and quality of the image and enhance the expression of details and clarity. This can enhance the ability to express details in the reconstructed results, making the final 3D model more realistic and accurate.

[0060] The present invention overcomes the dependence of traditional neural radiation field methods on multi-view images and improves the quality of reconstruction results by introducing a two-dimensional neural rendering module, which enables the present invention to be applied in a wider range of scenarios and obtain higher quality three-dimensional reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of a method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field according to the present invention;

[0062] Figure 2 This is a diagram of the neural radiation field network structure in the present invention;

[0063] Figure 3 This is a network structure diagram of the two-dimensional rendering module in the present invention. DETAILED DESCRIPTION

[0064] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0065] The present invention will be further described below with reference to the accompanying drawings.

[0066] like Figure 1 As shown, a method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field includes:

[0067] Step S1: construct a Zero-1-to-3 diffusion model network.

[0068] In this embodiment, the Zero-1-to-3 diffusion model is used to generate multi-view images from multiple perspectives. It uses a monocular image as input, utilizes deep learning techniques to infer the scene's depth information, and generates images from multiple perspectives. This simulates the effect of observing a scene from multiple perspectives, providing multi-view image data for the subsequent neural radiance field network.

[0069] In the embodiment, constructing a Zero-1-to-3 diffusion model network includes:

[0070] Given a single RGB image of an object The goal is to synthesize images of the object from different camera viewpoints, so and are the relative camera rotation and translation of the desired viewpoint, respectively, and the goal is to learn a diffusion model f that synthesizes a new image under this camera transformation as shown below:

[0071]

[0072] Where, To synthesize the image, the present invention estimates A new view x that is perceptually similar to a real, but unobserved, view R，T resemblance.

[0073] Step S2, constructing a neural radiation field network, such as Figure 2 shown.

[0074] In an embodiment, a neural radiation field network receives a multi-view image generated by a diffusion model as input. It maps the multi-view image into a three-dimensional space through volume rendering technology and estimates the color and density information of each spatial position, thereby obtaining a three-dimensional model representation that includes the geometry and texture information of the scene.

[0075] In the embodiment, a neural radiation field network is constructed, and an implicit rendering process is constructed by training a neural network model using the original neural radiation field network, thereby generating correct photos at any perspective, specifically including:

[0076] Neural Radiation Field Model F θ It receives the position, direction and three-dimensional coordinates of the light as input, and outputs the volume density and color radiation field properties. Finally, through the volume rendering algorithm, it synthesizes them into the final neural radiation field network according to the volume density and color information of the sampling points.

[0077] Step S3, construct a two-dimensional neural rendering module network, such as Figure 3 shown.

[0078] In an embodiment, in order to further improve the quality of the reconstructed three-dimensional model, a deconvolution layer in DCGAN is introduced to design and use a two-dimensional neural rendering module, which receives a multi-view image generated by a neural radiation field as input and further processes it to improve the resolution and quality of the image. By using deep learning technology, the module can enhance the detail expression and clarity of the image, thereby obtaining a high-resolution multi-view image.

[0079] In the embodiment, the two-dimensional neural rendering module network is mainly composed of a deconvolution layer and a bilinear upsampling network. The two-dimensional neural rendering module network is constructed, including:

[0080] This paper refers to the design concept of StyleNerf upsampler and is inspired by HeadNerf's 2D neural rendering module network. It modifies some of the networks in the 2D neural rendering module network, namely:

[0081] Introducing the idea of ​​deconvolution layer in DCGAN into the 2D neural rendering module network;

[0082] The upsampling layer in DCGAN, also known as the transposed convolution layer, enlarges the low-resolution feature map to a higher resolution, slides the filter on the input, and performs the opposite operation of convolution to expand the dimension of the input feature map.

[0083] In step S4, a pre-trained Zero-1-to-3 diffusion model is used, the monocular image is taken as input, and multiple images at different angles are obtained through the diffusion model.

[0084] In this embodiment, a pre-trained Zero-1-to-3 diffusion model is used, a monocular image is used as input, and multiple images at different angles are obtained through the diffusion model, including:

[0085] Although the diffusion model is trained on large-scale Internet data and can cover most viewpoints of most objects, these viewpoints cannot be controlled and adjusted in the pre-trained model. To this end:

[0086] Given a dataset of paired images and their camera extrinsics {(x,x (R，T) ,R,T)}, fine-tune the diffusion model to learn control over camera parameters (including but not limited to pose information) without destroying the rest of the representation;

[0087] Use a U-Net with encoder ∈ and denoiser ∈ θ and the potential diffusion architecture of decoder D;

[0088] At diffusion time step t ~ [1,1000], let c(x,R,T) be the embedding of the input view and the relative camera exterior, and solve the fine-tuned model using the following formula:

[0089]

[0090] After training the model θ, the diffusion model f generates images by performing iterative denoising from Gaussian noise images conditioned on c(x, R, T);

[0091] After passing through this fine-tuned Zero-1-to-3 diffusion model, a monocular image can be generated into a multi-view image, that is, a multi-view image.

[0092] In step S5, the neural radiation field network is trained, and the loss is minimized using back propagation and optimization, and the reconstruction result is obtained through volume rendering.

[0093] In an embodiment, a neural radiation field network is trained, loss is minimized using backpropagation and optimization, and a reconstruction result is obtained by volume rendering, including:

[0094] Represent a static scene as a continuous 5D function that outputs the value of each point (x, y, z) in each direction in space. The emitted radiance, and the density at each point, acts like differential opacity, controlling the accumulated radiance for a ray passing through (x,y,x), i.e.:

[0095] Represent a continuous scene as a 5D vector-valued function whose input is a 3D position x = (x, y, z) and a 2D viewing direction Its output is the emitted color c = (r, g, b) and volume density σ;

[0096] Represent the direction as a 3D Cartesian unit vector d, use a multi-layer perceptron network to approximate the continuous 5D scene representation: (x, d) → (c, σ), and optimize its weights Emit color from each input 5D coordinate mapped to its corresponding volume density and direction;

[0097] Neural Radiance Field optimizes a deep fully connected neural network without any convolutional layers by Regressing to a single volume density and view-dependent RGB color representation of the function, that is:

[0098] The dataset of captured RGB images of the scene, the corresponding camera poses and intrinsic parameters, and the scene boundaries. For synthetic data, the real camera poses, intrinsic parameters and boundaries are used. For real data, the above parameters are estimated from the motion bag using the COLMAP structure.

[0099] In each optimization iteration, a batch of camera rays is randomly sampled from all pixels in the dataset, and for each ray a color is rendered from the coarse and fine sample sets;

[0100] The loss function is simply the total squared error between the rendered colors of the coarse and fine renderings and the true pixel colors. By optimizing and training the network parameters, the network model can learn the radiation field representation in the scene. In volume rendering, there are coarse rendering and fine rendering. The total squared error between the rendered and the true pixel colors is defined as:

[0101]

[0102] Where, is the set of each batch of rays, C(r), and are the ground truth, coarse volume prediction, and fine volume prediction RGB colors of ray r, respectively, although the final rendering comes from But the present invention will also minimize The loss value of , so that the weight distribution of the coarse network can be used to distribute samples in the fine network;

[0103] Among them, the volume rendering equation can be divided into the ideal case and the actual situation of the algorithm, as shown in the following equations:

[0104]

[0105] δ i =t i+1 -t i ;

[0106] In the formula, C(r) represents the ideal case, which is to calculate the pixel color value by integration. represents the actual calculation method of the neural radiation field algorithm, which is approximated by the summation method. t is the distance between the sampling point and the origin, σ is the density value, T(t) is the opacity, c is the final rendered pixel color, which is the color value obtained by integrating along the light, r(s) is the parameterized light path, defined as: r(s) = o + sd, o is the origin of the light, s is the distance parameter along the light, d is the unit direction vector, t n is the near boundary of the ray integration, t f is the far boundary of the ray integral, defining the sampling range of volume rendering, σ i is the density of the i-th sampling point, indicating the probability density of light being blocked at this position, which determines the transmittance of light, δ i is the distance between adjacent sampling points, c i is the color of the i-th sampling point, which is determined by the output of the neural network, T i is the cumulative transmittance from the i-th sampling point to the camera, indicating the probability that the light is not absorbed before reaching the point, t i+1 is the distance value of the i+1th sampling point along the light, t i is the distance value of the i-th sampling point along the light, which is used to divide the integration interval;

[0107] Since the color values ​​and opacity in the image are not continuous values, the formula also needs to be discretized, and the number of sampling points calculated is also limited. Finally, the image in the corresponding posture is obtained through the volume rendering strategy.

[0108] Step S6: Using a two-dimensional neural rendering module network, the reconstructed image is rendered from a low-resolution image to a high-resolution image.

[0109] In an embodiment, a two-dimensional neural rendering module network is used to render the reconstructed image from a low-resolution image to a high-resolution image, including:

[0110] To build a 2D neural rendering module network, we draw inspiration from StyleNerf and HeadNerf, combine the ideas of deconvolution layers and bilinear upsampling networks, and introduce the deconvolution layer in DCGAN to amplify low-resolution feature maps to higher resolutions. Bilinear upsampling is a commonly used upsampling method that increases the resolution of an image through interpolation.

[0111] Combining the idea of ​​deconvolution layer and bilinear upsampling network, the combination of deconvolution layer and bilinear upsampling network is equivalent to an upsampling process, thereby restoring the image resolution.

[0112] Finally, the image after two-dimensional neural rendering is used as the input of the neural radiation field again, and a high-resolution reconstructed image is finally obtained.

[0113] In step S7, the rendered multi-view image is used as input to the neural radiance field network again to obtain the final reconstruction result.

[0114] In an embodiment, the rendered multi-view image is used again as input to the neural radiance field network to obtain the final reconstruction result, including:

[0115] The high-resolution multi-view images processed by the 2D neural rendering module are again used as input to the neural radiance field network;

[0116] The neural radiance field network uses the improved multi-view images for further optimization and adjustment to obtain the final reconstruction results.

[0117] Working principle:

[0118] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reconstructing a three-dimensional model from a monocular image based on a diffusion model and a neural radiation field, characterized in that: include: Step S1, constructing a Zero-1-to-3 diffusion model network; Step S2, constructing a neural radiation field network; Step S3, constructing a two-dimensional neural rendering module network; Step S4: Using a pre-trained Zero-1-to-3 diffusion model, the monocular image is used as input and multiple images at different angles are obtained through the diffusion model. Step S5, training the neural radiation field network, minimizing the loss using back propagation and optimization, and obtaining the reconstruction result through volume rendering; Step S6, using a two-dimensional neural rendering module network to render the reconstructed image from a low-resolution image to a high-resolution image; In step S7, the rendered multi-view image is used as input to the neural radiance field network again to obtain the final reconstruction result.

2. The method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field according to claim 1, characterized in that: In step S1, constructing a Zero-1-to-3 diffusion model network includes: Given a single RGB image x∈ of an object Let R∈ and T∈ are the relative camera rotation and translation of the desired viewpoint, respectively, and the goal is to learn a diffusion model f that synthesizes a new image under this camera transformation as shown below: Where, For the composite image.

3. The method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field according to claim 2, characterized in that: In step S2, a neural radiation field network is constructed. The original neural radiation field network is used to train a neural network model to construct an implicit rendering process, including: Neural Radiation Field Model F θ It receives the position, direction and three-dimensional coordinates of the light as input, and outputs the volume density and color radiation field properties. Finally, through the volume rendering algorithm, it synthesizes them into the final neural radiation field network according to the volume density and color information of the sampling points.

4. The method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field according to claim 3, characterized in that: In step S3, the two-dimensional neural rendering module network is mainly composed of a deconvolution layer and a bilinear upsampling network, and the two-dimensional neural rendering module network is constructed, including: Introducing the idea of ​​deconvolution layer in DCGAN into the 2D neural rendering module network; The upsampling layer in DCGAN enlarges the low-resolution feature map to a higher resolution, slides the filter on the input, and performs the opposite operation of convolution to expand the dimension of the input feature map.

5. The method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field according to claim 4, characterized in that: In step S4, a pre-trained Zero-1-to-3 diffusion model is used to take a monocular image as input, and multiple images at different angles are obtained through the diffusion model, including: Given a dataset of paired images and their camera extrinsics {(x,x (R，T) ,R,T)}, fine-tune the diffusion model to learn control over camera parameters without destroying the rest of the representation; Use a U-Net with encoder ∈ and denoiser ∈ θ and the potential diffusion architecture of decoder D; At diffusion time step t ~ [1,1000], let c(x,R,T) be the embedding of the input view and the relative camera exterior, and solve the fine-tuned model using the following formula: After training the model θ, the diffusion model f generates images by performing iterative denoising from Gaussian noise images conditioned on c(x, R, T); After passing this fine-tuned Zero-1-to-3 diffusion model, a monocular image is then generated into a multi-view image.

6. The method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field according to claim 5, characterized in that: In step S5, the neural radiation field network is trained, and the loss is minimized by back propagation and optimization, and a reconstruction result is obtained by volume rendering, including: Represent a static scene as a continuous 5D function that outputs the value of each point (x, y, z) in each direction in space. The emitted radiance, and thus the density at each point, controls the accumulated radiance for rays passing through (x,y,x), i.e.: Represent a continuous scene as a 5D vector-valued function whose input is a 3D position x = (x, y, z) and a 2D viewing direction Its output is the emitted color c = (r, g, b) and volume density σ; Represent the direction as a 3D Cartesian unit vector d, use a multi-layer perceptron network to approximate the continuous 5D scene representation: (x, d) → (c, σ), and optimize its weights Emit color from each input 5D coordinate mapped to its corresponding volume density and direction; Neural Radiance Field optimizes a deep fully connected neural network without any convolutional layers by Regressing to a single volume density and view-dependent RGB color representation of the function, that is: The dataset of captured RGB images of the scene, the corresponding camera poses and intrinsic parameters, and the scene boundaries. For synthetic data, the real camera poses, intrinsic parameters and boundaries are used. For real data, the above parameters are estimated from the motion bag using the COLMAP structure. In each optimization iteration, a batch of camera rays is randomly sampled from all pixels in the dataset, and for each ray a color is rendered from the coarse and fine sample sets; The loss function is simply the total squared error between the rendered colors of the coarse and fine renderings and the true pixel colors. By optimizing and training the network parameters, the network model can learn the radiation field representation in the scene. In volume rendering, there are coarse rendering and fine rendering. The total squared error between the rendered and the true pixel colors is defined as: Where, is the set of each batch of rays, C(r), and are the ground truth value, rough volume prediction value and fine volume prediction value RGB color of ray r respectively; Among them, the volume rendering equation can be divided into the ideal case and the actual situation of the algorithm, as shown in the following equations: δ i =t i+1 -t i ; In the formula, C(r) represents the ideal case, which is to calculate the pixel color value by integration. represents the actual calculation method of the neural radiation field algorithm, which is approximated by summing the integral method. t is the distance between the sampling point and the origin, σ is the density value, T(t) is the opacity, c is the final rendered pixel color, r(s) is the parameterized light path, t n is the near boundary of the ray integration, t f is the far boundary of the ray integration, σ i is the density of the i-th sampling point, δ i is the distance between adjacent sampling points, c i is the color of the i-th sampling point, T i is the cumulative transmittance from the i-th sampling point to the camera, t i+1 is the distance value of the i+1th sampling point along the light, t i is the distance value of the i-th sampling point along the light; Finally, the image in the corresponding posture is obtained through the volume rendering strategy.

7. The method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field according to claim 6, characterized in that: In step S6, the reconstructed image is rendered from a low-resolution image to a high-resolution image using a two-dimensional neural rendering module network, including: Combining the idea of ​​deconvolution layer and bilinear upsampling network, the combination of deconvolution layer and bilinear upsampling network is equivalent to an upsampling process, thereby restoring the image resolution. Finally, the image after two-dimensional neural rendering is used as the input of the neural radiation field again, and a high-resolution reconstructed image is finally obtained.

8. The method for reconstructing a three-dimensional model of a monocular image based on a diffusion model and a neural radiation field according to claim 7, characterized in that: In step S7, the rendered multi-view image is used as input to the neural radiance field network again to obtain the final reconstruction result, including: The high-resolution multi-view images processed by the 2D neural rendering module are again used as input to the neural radiance field network; The neural radiance field network uses the improved multi-view images for further optimization and adjustment to obtain the final reconstruction results.

Images