Small-sample image reconstruction method based on neural radiance field combined with diffuse texture constraints

Abstract

The present invention discloses a small sample image reconstruction method based on a neural radiation field combined with a diffusion texture constraint. The method is specifically: first, a 3D simulation image of the target is used to reconstruct the surface geometry; then a small sample two-dimensional RGB image of the same target as the two-dimensional 3D simulation image is used, and the reconstruction result is optimized by a diffusion model. During the optimization process, in order to improve the quality of samples "generated" based on the diffusion model, the present invention adds texture similarity constraints and structural similarity constraints. The present invention uses simulation images under various perspectives, rather than a small number of reference images with limited perspectives, as auxiliary knowledge to perform high-quality surface geometry reconstruction, which solves the problem of easy overfitting in the early stage of small sample reconstruction. On the basis of the semantic constraints of the diffusion model, the present invention introduces texture similarity constraints and structural similarity constraints based on the measured image design, thereby improving the reconstruction quality.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology and relates to a knowledge-assisted small sample neural radiation field reconstruction method, which aims to solve the problem of overfitting that easily occurs in image reconstruction under small sample conditions, resulting in blurred structure or false texture of the reconstruction result. Background Art

[0002] Traditional 3D reconstruction methods typically require dense observation of the scene, acquiring data from uniformly sampled perspectives. NeRF (Neural Radiance Fields) acquires image data from different viewpoints, using the 5D coordinates consisting of the viewpoint pose and the direction of the observation ray as the network input. The output includes the pixel volume density and corresponding color information at the sampling point. NeRF significantly improves the synthesis quality of 3D reconstruction by encoding parameters using a deep fully connected neural network and utilizing continuous voxels to address the limitations of increased image resolution.

[0003] Classic NeRF typically requires images from a large number of viewpoints as input. When the number of reference images is small, the quality of the reconstruction deteriorates significantly. Consequently, in recent years, increasing research has focused on improving the quality of NeRF reconstruction in small-sample NeRF scenarios. There are two main approaches to small-sample NeRF reconstruction: 1) Reconstruction without additional knowledge, simply by adding additional regularization terms to the network; 2) Reconstruction with additional knowledge.

[0004] When only a few reference views are available, NeRF training is prone to overfitting, resulting in poor reconstruction quality. To reconstruct high-quality models from limited reference images, many studies have introduced additional regularization terms in the network to prevent overfitting during NeRF training. However, due to the limited reference view angles, regularized NeRF reconstruction may not fully restore the target model.

[0005] The main goal of the present invention is to achieve reconstruction of a complete target model without being restricted by the reference view angle. Therefore, the present invention adopts diffuse texture as a constraint and introduces prior assistance such as semantic information to achieve this goal.

[0006] Furthermore, to more realistically reconstruct the target 3D model even with small sample sizes, many studies have proposed two-stage reconstruction methods, including a coarse reconstruction in the first stage and a fine reconstruction in the second stage. Unlike these mainstream algorithms, the present invention employs a two-stage reconstruction method that first reconstructs surface features and then reconstructs texture features. Summary of the Invention

[0007] In order to overcome the shortcomings of existing research, the present invention provides a small sample image reconstruction method based on the neural radiation field combined with diffusion texture constraints, aiming to solve the problem of overfitting in the neural radiation field reconstruction under small sample conditions, which leads to blurred structure or false texture in the reconstruction results.

[0008] In a first aspect, the present invention provides a small sample image reconstruction method based on a neural radiation field combined with a diffusion texture constraint, comprising the following steps:

[0009] Step 1: Build a neural radiation field model and perform initial training on it to achieve surface geometry reconstruction of the image;

[0010] Step 2: Retrain the neural radiation field model and diffusion model after initial training to achieve diffusion texture reconstruction of the image;

[0011] Step 3: Use the neural radiation field and diffusion model after secondary training to reconstruct the three-dimensional image.

[0012] Preferably, step one is:

[0013] The two-dimensional 3D simulated image and the corresponding camera internal and external parameters of the image are used as input to initially train the neural radiation field model, and the 3D representation of the current image scene with geometric characteristics is output, and the 3D representation includes the color and density of the pixel points.

[0014] The two-dimensional 3D simulation image uses several images of a scene;

[0015] Given a randomly sampled camera viewpoint, the output of the initially trained neural radiance field model is rendered into an image using a signed distance field-based volume rendering technique.

[0016] Preferably, step 2 is:

[0017] 2.1 Input a small sample of 2D RGB images of the same target as the 2D 3D simulation image as a reference image into the initially trained neural radiance field model, and output a 3D representation of the current image scene with texture characteristics, including the color and density of the pixels;

[0018] 2.2 Given a randomly sampled camera viewpoint, use volume rendering technology to render the output of the neural radiation field model after the second training in step 2.1 into an image with texture characteristics;

[0019] 2.3 Construct a diffusion model and its output constraints, and then train the diffusion model;

[0020] Preferably, step 2.3 is specifically: the diffusion model adopts a U-Net denoising model with a self-attention mechanism; the U-Net denoising model with a self-attention mechanism includes a U-Net basic network and a Transformer network;

[0021] The image with texture characteristics obtained in step 2.2 is subjected to noise processing to obtain a noisy image;

[0022] The sample category of the small sample 2D RGB image is used as auxiliary knowledge input to the Transformer network, and the Transformer network outputs the corresponding attention weight;

[0023] The U-Net basic network receives the noisy image and the attention weights output by the Transformer network. During the denoising process, the attention weights are continuously injected into the semantic vectors in the knowledge auxiliary to achieve the semantic information injection function and obtain the predicted noise;

[0024] The error between the predicted noise and the noise added to the image with texture characteristics is calculated, and the network parameters of the U-Net denoising model with self-attention mechanism are updated through back propagation based on the error.

[0025] The output constraints of the diffusion model are specifically:

[0026] (1) Semantic constraints based on diffusion model

[0027] Semantic constraints use the distillation loss function L SDS , which is calculated as follows:

[0028]

[0029] Where φ represents the network parameters of the U-Net denoising model with self-attention mechanism, G θ Represents the neural radiation field model after secondary training, w(t) represents the weight, ∈ φ represents the diffusion model, x t represents the noisy image, t represents the signal-to-noise ratio, ∈ represents the noise, y represents the sample category of the small sample 2D RGB image, x represents the small sample 2D RGB image; θ represents the neural radiation field parameter after secondary training;

[0030] (2) Texture similarity constraint

[0031] Texture similarity constraint uses texture loss function L CLIP-C , which is calculated as follows:

[0032]

[0033] in represents the language-image training loss, X represents the reference image, g θ (β) represents the denoised image after the diffusion model processing, n represents The number of values ​​greater than the threshold;

[0034] (3) Structural similarity constraints

[0035] The texture similarity constraint uses the structural loss function L depth , and its calculation method is as follows:

[0036]

[0037] Where d(β i ) represents the depth map in the reference direction, d represents the depth map of the denoised image after being processed by the diffusion model, Cov represents convolution, and Var represents variance;

[0038] (4) The semantic constraint, texture similarity constraint, and structural similarity constraint are aggregated into a diffusion texture constraint, which uses the diffusion texture loss function L total (φ,G θ ), which is calculated as follows:

[0039] L total (φ,G θ )=ae -bx w SDS L SDS (φ,G θ )+w CLIP L CLIP-C (X,g θ (β))+w depth L depth

[0040] Among them ae -bx represents the decaying exponential function, a represents the amplitude parameter of the decaying exponential function, b is the decay rate parameter, w SDS 、w CLIP 、w depth are the weight coefficients of the semantic constraint, texture similarity constraint, and structural similarity constraint of the diffusion model respectively;

[0041] The diffusion texture constrained back propagation is used to update the neural radiation field parameters after secondary training, thereby completing the complete three-dimensional model reconstruction of the target.

[0042] In a second aspect, the present invention provides a small sample image reconstruction system for implementing the method, including a neural radiation field and diffusion model after secondary training.

[0043] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described above.

[0044] In a fourth aspect, the present invention provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described is implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention proposes a small sample image reconstruction method based on a neural radiation field combined with a diffusion texture constraint. The method is divided into two stages: in the first stage, the surface geometric model of the target is reconstructed through the classical neural radiation field and volume rendering based on the signed distance field; on the basis of the first stage, the second stage uses a small number of reference images and optimizes the reconstruction results through the diffusion texture constraint. The two-stage reconstruction strategy proposed in the present invention can avoid the overfitting problem in the case of small samples to a certain extent. Compared with other existing mainstream open source algorithms, the method proposed in the present invention is based on the assistance of simulated images and model categories, which can get rid of the problem of limited viewing angle of the reference image, thereby reconstructing a complete target model. In addition, compared with other mainstream open source algorithms, the method of the present invention can generate high-fidelity textures and has considerable generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 This is a general schematic diagram of the knowledge-assisted small sample neural radiation field reconstruction method of the present invention;

[0049] Figure 2 This is a schematic diagram of the neural radiation field principle in the present invention;

[0050] Figure 3 Schematic diagram of semantic constraints based on the diffusion model in the present invention;

[0051] Figure 4 Schematic diagram of semantic embedding of the diffusion model in the present invention;

[0052] Figure 5 This is a graph comparing the experimental results of the present invention with other methods. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] A small sample image reconstruction method based on neural radiation field combined with diffusion texture constraint Figure 1 The specific steps are as follows:

[0055] Step 1: Construct the Neural Radiation Field model NeRF and perform initial training on it to achieve surface geometry reconstruction of the image;

[0056] like Figure 2 As shown in the figure, the overall process of NeRF is mainly based on a number of images of a given scene and the camera's internal and external parameters corresponding to the images. After encoding and MLP, a 3D representation of the scene is reconstructed. The 3D representation includes the color and density of the pixels. There are many forms of 3D representation. NeRF uses the radiation field, which can be regarded as a function: if a ray is emitted from an angle into a static space, the density σ of each pixel in the space can be queried, as well as the color c (c = (R, G, B)) presented at that position under the ray angle, that is:

[0057] F:(x,y,z,θ,φ)→(R,G,B,σ)

[0058] Where (x, y, z) represents the position of the point in space, (θ, φ) represents the ray angle, and σ represents the density.

[0059] To visualize the Neural Radiance Field, we will next perform volume rendering based on a signed distance field. Intuitively, volume rendering involves knowing the camera's focal point. The line connecting the focal point and the pixel can produce a ray. The color value of the pixel can be obtained by summing the colors of all points along this ray. For a given image pixel, NeRF represents the line connecting the camera's focal point and the pixel, i.e., a ray, as:

[0060] {p(t)=o+tv|t≥0}

[0061] Where o represents the camera center and v represents the light direction, so the volume rendering formula is expressed as follows:

[0062]

[0063] Where C(o,v) represents the color of the target pixel block after rendering calculation, w(t) represents the weight value of point p(t), and c(p(t),v) is the color value of spatial point p in the viewing direction v.

[0064] The volume rendering method based on surface signed distance field reconstruction combines the signed distance field (SDF) with volume rendering calculation to solve this problem. The signed distance field is based on the surface of the object and uses signed numbers to represent the distance between the grid points in the radiation field and the surface: the distance to the grid points outside the object is positive; the distance to the grid points inside is negative, and the surface of the object is represented by a zero level set:

[0065] S={x∈R 3 |f(x)=0}

[0066] where f(x) is the signed distance function.

[0067] The weight function w(t) is set to:

[0068]

[0069] Where: ρ(t) represents the opacity density function, f represents the signed distance function, Represents the cumulative transparency of light. The improved weight function w(t) reaches its maximum value on the object surface S, reducing the deviation of shape reconstruction.

[0070] Step 2: retrain the neural radiation field model and diffusion model after the initial training to achieve diffusion texture reconstruction of the image;

[0071] Step 2.1: Input a small sample 2D RGB image of the same target as the 2D 3D simulation image as a reference image into the initially trained neural radiance field model, and output a 3D representation of the current image scene with texture characteristics, including the color and density of the pixels;

[0072] Step 2.2: Given a camera perspective, use volume rendering technology to render the output of the neural radiation field (NeRF) in step 2.1 into an image with texture characteristics.

[0073] Step 2.3: Build a diffusion model

[0074] The diffusion model mainly refers to semantic constraints, where the semantic constraints focus on whether the image with texture characteristics rendered by NeRF is consistent with the corresponding target type. Figure 3As shown in the figure, the similarity between the rendered image and its target type is evaluated based on the diffusion model: noise is added to the rendered image (noise addition), the target type semantics is embedded into the diffusion model, and the diffusion model is iteratively "filtered" (noise prediction) to generate a new image that conforms to the target type semantics. The similarity between "noise addition" and "noise prediction" is compared to evaluate the similarity between the rendered image and its target type semantics.

[0075] Specifically, the diffusion model training process is like "noising" and then "denoising". The training process mainly includes: 1) adding noise to images in the training set; 2) inputting it into the U-Net and letting the U-Net predict the noise based on semantic information; 3) calculating the error between the predicted noise and the added noise; 4) updating the U-Net parameters through backpropagation.

[0076] like Figure 4 As shown in the figure, the self-attention mechanism is used to add semantic information about the target type: the CLIP model's TextEncoder is first used to compress the target type semantics into a semantic embedding vector. During the U-Net denoising process, the attention mechanism is continuously used to inject this embedding vector, thus achieving semantic information injection.

[0077] The simplest way to supervise NeRF using the diffusion model denoising process is to imitate the training process of the diffusion model: give the NeRF rendered image x render Add similar noise α t x render +σ t ε. If the image x rendered by NeRF render Close to the real image, then the noise ε predicted by the diffusion model pred It should be close to the added noise ε. The loss function L can be used diff To measure the proximity of the noise:

[0078] L diff =E[w(t)||ε pred -ε| 2 ]=E[w(t)||UNet(α t x i +σ t ε|t)-ε|| 2 ]

[0079] Where w(t) is the weight of the scheduler in the diffusion model.

[0080] Observe L diff The calculation process of the loss function finds x render =NeRF(camera,θ), and at the same time, x renderIt is also necessary to superimpose the U-Net operation. Therefore, if we directly use L diff To optimize the loss function, the gradient corresponding to U-Net must be calculated at each step, which is difficult and time-consuming. Therefore, the distillation loss function L is used. SDS For substitution, the corresponding gradient calculation method is as follows:

[0081]

[0082] Where φ represents the network parameters of the U-Net denoising model with self-attention mechanism, G θ Represents the neural radiation field output after secondary training, w(t) represents the weight, ∈ φ represents the diffusion model, x t represents the noisy image, t represents the signal-to-noise ratio, ∈ represents the noise, y represents the sample class of the small sample 2D RGB image, x represents the small sample 2D RGB image; θ represents the neural radiation field parameter;

[0083] Step 2.4: Construct output constraints of the diffusion model;

[0084] Optimizing NeRF solely based on semantic constraints can generate new synthetic images that roughly conform to the semantics of the target type. However, because the given semantics cannot describe all the details of the target scene, the new synthetic images do not fully match the input reference image in terms of texture details. Therefore, a texture similarity constraint is imposed on the denoised image in the diffusion model to force the NeRF-generated image to "align" with the reference image.

[0085] Using CLIP image encoder, design texture similarity constraint loss function L CLIP-D :

[0086]

[0087] where ε CLIP It is the CLIP image encoder, X i represents the reference image, Represents the image after U-Net denoising.

[0088] The CLIP image encoder is based on a unique convolutional neural network architecture, the VisionTransformer (ViT), which is based entirely on a self-attention mechanism. Unlike traditional convolutional neural networks, ViT does not require pooling or convolutional layers to downsample images or reduce feature dimensionality, making it better suited for images of varying scales and resolutions. Furthermore, ViT preserves the spatial relationships between pixels in an image by incorporating positional encoding.

[0089] In addition, for targets with complex shapes such as tanks, the front reference image is useless or even counterproductive for the reconstruction of the back. CLIP-D A threshold ε is set based on the threshold, and the reference images below the threshold are discarded (the reference images with too large angle differences are removed). For the similarities above the threshold, the average value is taken as the standard for similarity evaluation, and the final texture similarity constraint loss function L is obtained. CLIP-C :

[0090]

[0091] in represents the language-image training loss, X represents the reference image, g θ (β) represents the image generated by the diffusion model, n represents The number of values ​​greater than the threshold,

[0092] New images generated by NeRF optimized with semantic and texture similarity constraints still suffer from issues such as shape ambiguity, concave surfaces, overly flat geometric shapes, and depth ambiguity. To address these issues, we leverage the surface geometry model learned in stage 1 (surface geometry reconstruction) to construct structural similarity constraints and further optimize the small-sample NeRF reconstruction.

[0093] Taking into account the computational time, feature robustness, and detail representation capabilities of each structural feature, depth information is selected as the structural feature: by estimating and comparing the depth information of the target in the new synthesized image and the reference image (the relative distance of each pixel in the target area from the camera), the structural similarity between the two is evaluated.

[0094] An off-the-shelf single-view depth estimator is used to estimate the depth d of the rendered view (new synthesized image) and the depth d(ref) of the reference image. To address the errors and scale mismatch in depth estimation, a negative Pearson correlation is used for regularization.

[0095] The specific calculation method is: Figure 1 As shown in the figure, the depth maps of the surface model (reconstructed in stage 1) at multiple angles (front, back, left, and right) are selected as the benchmark, the depth map of the image after diffusion model denoising is compared with the benchmark, and the corresponding similarity value is calculated:

[0096]

[0097] Where d(β i ) represents the depth map in the reference direction, d represents the depth map of the image denoised by the diffusion model, Cov represents convolution, and Var represents variance.

[0098] The above three constraints (semantic, texture, and structural similarity constraints) are summarized into the diffusion texture constraint:

[0099] L total (φ,G θ )=ae -bx w SDS L SDS (φ,G θ )+w CLIP L CLIP-C (X,g θ (β))+w depth L depth

[0100] Among them: ae -bx is the decaying exponential function, a represents the amplitude parameter of the decaying exponential function, b is the decay rate parameter, w SDS 、w CLIP 、w depth They are the weight coefficients of the semantic constraint, texture similarity constraint, and structural similarity constraint of the diffusion model respectively.

[0101] To L SDS The reason for adding the decay exponential function is that, relatively speaking, in the iterative optimization process of NeRF, semantic constraints correspond to "coarse-grained" optimization, while texture and structural constraints correspond to "fine-grained" optimization; in the early stage of iterative optimization, "coarse-grained" optimization adjustments are performed first; and in the later stage of iterative optimization, "fine-grained" optimization adjustments are more needed; therefore, by setting the decay exponential function, the weight of the semantic constraints is gradually reduced.

[0102] The diffusion texture constrained back propagation is used to update the neural radiation field parameters after secondary training, thereby completing the complete three-dimensional model reconstruction of the target.

[0103] Step 3: Reconstruct the 3D image using the neural radiation field and diffusion model after secondary training; specifically:

[0104] The two-dimensional RGB image to be reconstructed is input into the neural radiation field after secondary training, and a 3D representation of the current image scene with texture characteristics is output; given a camera perspective, the 3D representation of the current image scene with texture characteristics is rendered into an image with texture characteristics using volume rendering technology.

[0105] The diffusion model is used to evaluate the similarity between the image output by the neural radiation field after secondary training and its target type, and the diffusion texture constrained back propagation is used to update the neural radiation field parameters, thereby completing the complete three-dimensional model reconstruction of the target.

[0106] A. Experimental Setup

[0107] The effectiveness of the proposed method was tested on targets such as drones and tanks using a single measured image and specific knowledge support. The knowledge support primarily consisted of simulated images of the targets and their types. The simulated images were primarily derived from simulation software, and the target types were used as language information in the diffusion model.

[0108] Model training:

[0109] In the first stage, the present invention uses NeRF and volume rendering based on signed distance fields to reconstruct the surface geometry of the target, wherein the interior of the surface is selected as the region of interest of the signed distance field. In NeRF, the present invention samples 512 rays per batch and performs 10,000 iterations at a rendering resolution of 400*400 on a single NVIDIA RTX1080Ti GPU. The training lasts for 8 hours.

[0110] In the second stage, sampling is performed according to the currently mainstream small-sample camera sampling method: novel views are randomly sampled with a probability of 75%, and predefined reference views are sampled with a probability of 25%. When rendering using NeRF, the present invention also randomly enlarges the focal length to improve rendering quality.

[0111] For the distillation loss function L SDS : The present invention randomly samples t from 200 to 600 and sets w(t) to a uniform weight according to the time step. The purpose of the present invention is to align the "generated" image with the measured image, and the present invention unifies w(t) to 10.

[0112] Training speed: The second phase uses Adam similar to the first phase, with a learning rate of 0.001. The second phase then repeats for 5,000 iterations at a rendering resolution of 800 x 800. The entire training process takes approximately 13 hours on a single NVIDIA RTX 1080 Ti GPU.

[0113] B. Comparative Experiment

[0114] This paper compares the proposed knowledge-assisted small sample neural radiation field reconstruction method with five other mainstream open source algorithms, including 1) DietNeRF; 2) SinNeRF; 3) DreamFusion; 4) Point-E; and 5) FreeNeRF.

[0115] C. Result Analysis

[0116] Qualitative comparison: First, the proposed method is compared with SinNeRF and DietNeRF: Due to the lack of multi-view supervision, SinNeRF and DietNeRF have problems of structure loss and texture artifacts when reconstructing more complex objects in 3D. Secondly, the proposed method is compared with DreamFusion, PointE and FreeNeRF: The proposed method and DreamFusion use 2D diffusion as a priori, PointE uses a 3D diffusion model as a priori, and FreeNeRF is based on regularized neural radiation field reconstruction. Figure 5 As shown in

[15] , the models they generated cannot be faithfully aligned with the reference image and the textures are smooth and artificial. In contrast, the proposed method can generate high-fidelity 3D models with accurate geometry and realistic textures.

[0117] Quantitative comparison: If you want to generate a high-fidelity 3D model, it should be very similar to the input image in the reference view, and the view under the novel view should also have a certain similarity with the reference view, and should also be semantically similar. The following indicators are used to evaluate these two aspects in this invention: 1) LPIPS, an indicator for evaluating image reconstruction quality. The calculation result of LPIPS reflects the difference between the reference image and the reconstructed image, 2) Contextual distance, an indicator for measuring the pixel-level similarity between the rendering of the novel view and the reference. It is mainly used to evaluate the degree of difference between the image generation or reconstruction algorithm and the reference image when generating a novel view, 3) CLIP score, an indicator for evaluating the semantic similarity between text and images.

[0118] For targets such as drones and tanks, multiple reconstructions were performed from a single reference image at multiple different angles, and three metrics were statistically analyzed. The results are shown in Table 1. The proposed method outperforms the comparison algorithm in reconstruction quality for targets such as drones and tanks at both the reference and novel perspectives.

[0119] Table 1 Quantitative comparison

[0120]

[0121]

Claims

1. A small sample image reconstruction method based on a neural radiation field combined with diffuse texture constraints, characterized by: The following steps are involved: Step 1: Build a neural radiation field model and perform initial training on it to achieve surface geometry reconstruction of the image; Step 2: Perform secondary training on the neural radiation field model and diffusion model after initial training to achieve diffusion texture reconstruction of the image; specifically: 2.1 Input a small sample of 2D RGB images of the same target as the 2D 3D simulation image as a reference image into the initially trained neural radiance field model, and output a 3D representation of the current image scene with texture characteristics, including the color and density of the pixels; 2.2 Given a camera viewpoint, use volume rendering technology to render the output of the neural radiation field model after the second training in step 2.1 into an image with texture characteristics; 2.3 Construct a diffusion model and its output constraints, and then train the diffusion model; The output constraints of the diffusion model are specifically: (1) Semantic constraints based on diffusion model Semantic constraints use distillation loss function , which is calculated as follows: Formula (1) in Represents the network parameters of the U-Net denoising model with self-attention mechanism, Represents the neural radiation field model after secondary training, represents the weight, represents the diffusion model, represents the noisy image, t represents the signal-to-noise ratio, represents noise, y represents the sample category of the small sample two-dimensional RGB image, and x represents the small sample two-dimensional RGB image; represents the neural radiation field parameters after secondary training; (2) Texture similarity constraint Texture similarity constraint uses texture loss function , which is calculated as follows: Formula (2) in represents the language-image training loss, X represents the reference image, Represents the denoised image after diffusion model processing, n represents The number of values ​​greater than the threshold; (3) Structural similarity constraints Texture similarity constraint uses structural loss function , and its calculation method is as follows: Formula (3) in represents the depth map in the reference direction, d represents the depth map of the denoised image after being processed by the diffusion model, Cov represents convolution, and Var represents variance; (4) The semantic constraint, texture similarity constraint, and structural similarity constraint are aggregated into a diffusion texture constraint, which uses the diffusion texture loss function , and its calculation method is as follows: Formula (4) in represents the decaying exponential function, represents the amplitude parameter of the decaying exponential function, is the decay rate parameter, 、 、 are the weight coefficients of the semantic constraint, texture similarity constraint, and structural similarity constraint of the diffusion model respectively; Using diffusion texture constrained back propagation, the neural radiation field parameters after secondary training are updated to complete the reconstruction of the complete 3D model of the target; Step 3: Use the neural radiation field and diffusion model after secondary training to reconstruct the three-dimensional image.

2. The method according to claim 1, characterized in that Step 1 is specifically: The two-dimensional 3D simulated image and the corresponding camera internal and external parameters of the image are used as input to initially train the neural radiation field model, and the 3D representation of the current image scene with geometric characteristics is output, and the 3D representation includes the color and density of the pixel points. The two-dimensional 3D simulation image uses several images of a scene; Given a camera viewpoint, the output of the initially trained neural radiance field model is rendered into an image using a signed distance field-based volume rendering technique.

3. The method according to claim 1, characterized in that The diffusion model adopts a U-Net denoising model with a self-attention mechanism.

4. The method according to claim 3, characterized in that The U-Net denoising model with self-attention mechanism includes a U-Net basic network and a Transformer network; The image with texture characteristics obtained in step 2.2 is subjected to noise processing to obtain a noisy image; The sample category of the small sample 2D RGB image is used as auxiliary knowledge input to the Transformer network, and the Transformer network outputs the corresponding attention weight; The U-Net basic network receives the noisy image and the attention weights output by the Transformer network. During the denoising process, the attention weights are continuously injected into the semantic vectors in the knowledge auxiliary to achieve the semantic information injection function and obtain the predicted noise; The error between the predicted noise and the noise added to the image with texture characteristics is calculated, and the network parameters of the U-Net denoising model with self-attention mechanism are updated through back propagation based on the error.

5. A small sample image reconstruction system implementing the method according to any one of claims 1 to 4, characterized in that: Includes neural radiation fields and diffusion models after secondary training.

6. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 4.

7. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 4 is implemented.

Images