Nerve radiation field three-dimensional reconstruction method based on potential spatial diffusion
Through the joint training of latent space diffusion technology and hidden space encoding codec, the problems of error and boundary blur in the neural radiation field method are solved, and a higher quality three-dimensional reconstruction effect is achieved.
Patent Information
- Application Number
- CN202411005852.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-08
AI Technical Summary
The existing three-dimensional reconstruction methods of neural radiation field are prone to errors when generating scene blocks, resulting in artifacts and blurred boundaries, affecting the accuracy and sense of realism of reconstruction.
The three-dimensional reconstruction method of neural radiation field based on latent spatial diffusion is adopted. Through the joint training of hidden space encoding and decoder, combined with latent spatial diffusion technology, noise is gradually added and removed, the three-dimensional scene is reconstructed, and the reconstruction quality and performance indicators are improved.
It significantly improves the accuracy and reality of three-dimensional reconstruction, reduces artifacts and boundary blurring, and improves the quality and performance of reconstruction results.
Smart Images

Figure CN118918260A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a method for three-dimensional reconstruction of neural radiation fields based on potential space diffusion. Background Art
[0002] 3D scene reconstruction is an important problem in the field of computer vision and graphics, which aims to restore the geometry and appearance information of a 3D scene from a series of 2D images. This problem has wide applications in fields such as virtual reality, augmented reality, and 3D printing. However, since 2D images only contain limited perspective and lighting information, while 3D scenes have infinite complexity and diversity, 3D scene reconstruction has always been a very challenging task.
[0003] In recent years, deep learning methods have made remarkable progress in the field of three-dimensional scene reconstruction. Among them, neural radiation field, as a novel and effective method, can recover three-dimensional geometry and appearance information from two-dimensional images, but it still faces some limitations. One of them is that the model often makes errors when generating scene blocks, resulting in the appearance of bodies that do not exist in the actual scene. This error manifests itself as artifacts similar to cloud effects in the scene, which brings a certain sense of unreality to the reconstruction results. In addition, the neural radiation field model has challenges in dealing with object boundaries, resulting in obvious blurring effects on the boundaries, which in turn affects the accuracy and realism of the overall reconstruction. In order to solve the above problems, the present invention proposes a method based on the potential diffusion model, which provides a new direction for the neural radiation field reconstruction method by reconstructing the basic neural network of the neural radiation field, and is no longer limited to the multi-layer perceptron. And it can significantly improve the reconstruction quality and quantitative indicators of the scene. Summary of the invention
[0004] The present invention proposes a three-dimensional reconstruction method of neural radiation field based on latent space diffusion, which can improve the quality of neural radiation field reconstruction and corresponding performance indicators.
[0005] The present invention adopts the following technical solutions.
[0006] A method for three-dimensional reconstruction of neural radiation fields based on potential space diffusion is characterized by comprising the following steps:
[0007] Step S1: Use a camera to capture views of various perspectives of a real scene, and use an algorithm to calculate the spatial position and direction of the camera for this view set, and then organize it into a data set for preprocessing;
[0008] Step S2: Input each view and camera pose, sample the scene space, and use the pre-trained neural radiance field model to obtain the color and density information of each sampling point in the scene, and combine them into the features of the sampling point;
[0009] Step S3: Input the features of all sampling points into the encoder to encode the latent space, obtain the potential representation of the entire space, and restore it into a new feature with the same shape as the original feature through the decoder, and make the original feature and the new feature loss, and jointly train the encoder and decoder;
[0010] Step S4: Use latent space diffusion to the latent representation, gradually add noise, obtain the residual of each step, and obtain the random noise representation. Then use the inverse process to input random noise and gradually restore it to the latent representation. The trained decoder is decoded into new features and the volume rendering method is used to reconstruct the complete three-dimensional scene.
[0011] The step S1 is specifically as follows:
[0012] Step S11: using a handheld camera to capture view images of various perspectives in a real scene, ensuring that different angles and fields of view are covered;
[0013] Step S12: Calculate the position of the camera in space and the shooting direction according to the motion trajectory of the camera and the posture information during shooting;
[0014] Step S13: sorting and preprocessing the calculated camera position and direction data, removing abnormal values and erroneous data, and converting the camera coordinate system into the world coordinate system.
[0015] The step S2 is specifically as follows:
[0016] Step S21: input each view and the corresponding camera pose information into the neural radiation field, including the position coordinates Position and the viewing direction Direction of the camera in the world coordinate system;
[0017] Step S22: Perform random sampling in the scene, take a ray Ray in a random direction, take a sampling point Sample on the ray Ray, and output the position information of all sampling points and input them into the pre-trained neural radiation field model;
[0018] Step S23: Output the intermediate result of the neural radiation field model, that is, the color RGB and density Density corresponding to the spatial position of each sampling point in the space, and combine the color RGB and density Density of each sampling point into a feature vector Feature representing the entire scene Sample .
[0019] The step S3 is specifically as follows:
[0020] Step S31: The obtained neural radiation field scene feature representation vector Feature Sample Input the latent space encoder Encoder and encode it into the potential feature representation Feature of the scenelantent_Sample ;
[0021] Step S32: Feature lantent_Sample Input decoder Decoder, restore it to the same Sample Feature vectors of the same dimension autoencoder_Sample ;
[0022] Step S33: Feature autoencoder_Sample With Feature Sample As loss, back propagation and gradient descent are performed to jointly train the encoder Encoder and decoder Decoder.
[0023] The process of the encoder is specifically as follows:
[0024] Let R represent the number of rays taken in space, S represent the number of sampling points on each ray, and represent the scene feature vector Feature with dimension (R, S, 4) Sample Split into a Feature with dimension (R, S, 3) RGB With a feature of dimension (R, S, 1) Density , and input the encoder Encoder, the two features are projected into Features with dimensions (R, S, N) through the multi-layer perceptron MLP mlp_RGB With Feature mlp_Density (Here N represents 128), use dot product to fuse the two features to get Feature lantent_Sample ; Specifically expressed as:
[0025] Feature RGB , Feature Density =Split(Feature Sample )
[0026] Feature mlp_RGB =MLPS1(Feature RGB )
[0027] Feature mlp_Density =MLPS2(Featu.re Density )
[0028] Feature lantent_Sample =Feature mlp_Density *Feature mlp_RGB
[0029] Where Split represents the segmentation operation, and MLPS1 and MLPS2 represent two multi-layer perceptrons respectively.
[0030] The training process of the decoder is specifically as follows:
[0031] The obtained feature lantent_Sample The input decoder is projected into a feature with dimensions of (R, S, 3) through multi-layer perceptrons MLPS3 and MLPS4 respectively. Decoder_RGB With the feature of dimension (R, S, 1) Decoder_Density , and concatenate them together according to the last dimension to form a Feature autoencoder_Sample ; Combine it with Feature Sample As MSE loss, the decoder is trained by gradient descent and back propagation; specifically:
[0032] Feature Decoder_RGB =MLPS3(Feature lantent_Sample )
[0033] Feature Decoder_Density =MLPS4(Feature lantent_Sample )
[0034] Feature autoenoder_Sample =Cat(Feature Decoder_RGB , Feature Decoder_Density )
[0035] MLPS3 and MLPS4 represent two multi-layer perceptrons, and Cat represents the concatenation operation.
[0036] Specifically:
[0037] Loss mse =(Feature autoencoder_Sample -Feature Sample ) 2
[0038] Among them, Loss mse Represents the calculated MSE loss value.
[0039] The step S4 is specifically as follows:
[0040] Step S41: Potential representation Feature lantet_Sample Add noise step by step to get a random noise representation;
[0041] Step S42: Input a random noise random , step-by-step denoising, that is, inversely processing the random noise representation to reconstruct it into a new potential representation feature diffusion _ lantent_Sample ;
[0042] Step S43: Reconstruct the potential representation feature Feature diffusion_lantent_Sample Input the trained decoder to restore it to the same state as Feature Sample New feature representation of the same dimension Decoder_diffusion_Sample ;
[0043] Step S44: Feature Decoder_diffusion_Sample Intermediate result feature of alternative neural radiation field Sample The subsequent volume rendering process is carried out to reconstruct a complete three-dimensional scene.
[0044] The specific method of adding noise is:
[0045] The potential representation of the feature lantent_Sample Gradually add noise to get a set of noise representation features noise :
[0046]
[0047] Feature lantent_t represents the potential representation of the t-th step of noise addition, Feature lantent_0 represents the initial potential representation, q(Feature lantent_t |Feature lantent_0 ) represents the random noise that needs to be added to the initial features in step t, N represents the set of natural numbers, I represents the set of integers, and α k represents the value of α at the kth step, α t represents the value of α at step t, It means the value of α multiplied from the first step to the tth step at step t. The α of different steps is predefined and gradually decreases.
[0048] When t is the last step, q(Feature noise |Feature lantent_Sample ) represents the process of adding noise from potential feature noise to random noise, which can be expressed as:
[0049] Feature noise =Feature lantent_Sample +q(Feature noise |Feature lantent_Sample ).
[0050] The specific method for removing noise is:
[0051] Noise removal is the reverse process of the above step-by-step noise addition. A U-Net is used to fit the reverse process:
[0052]
[0053] p(Feature lantent_t-1 |Feature lantent_t ) represents the random noise that needs to be removed for the t-1 step feature. Let t and Feature lantent_t It is known that the process of gradual noise addition can be performed by simply using a neural network to fit ε, and the entire training process is equivalent to the process of training the parameters of the U-Net network.
[0054] The specific method of volume rendering is:
[0055] Given the camera position coordinates Position and viewing direction Direction, through the input Feature Deroder_diffusion_Sample , split into the volume density Density and color RGB of each sampling point, and obtain the color COLOR of the object and its surface at the viewing angle of the position; the specific method is as follows:
[0056]
[0057] Where RGB i is the color of the i-th sampling point along the direction, Weight i is the weight of the color of the i-th sampling point along the direction, Density i Represents the density of the i-th sampling point along the direction, Density j Represents the density of the sampling point along the direction, Distance i Represents the distance between the i+1th and ith sampling points, Distance j Represents the distance between the h+1th and the h+1th sampling points, S represents the upper limit of the sampling points, which is 65536 here; finally, on the surface of the object in the scene to be synthesized, a complete three-dimensional model is rendered by tracking the camera light.
[0058] Compared with the prior art, the present invention has the following beneficial effects:
[0059] 1. Compared with the existing neural radiation field method, the present invention reconstructs the basic neural network of the neural radiation field while maintaining the inference speed as much as possible, improves the reconstruction accuracy of the method, and makes it more suitable for real scenes;
[0060] 2. The present invention innovatively uses the potential diffusion model method. Different from the neural radiation field method, the present invention only uses the intermediate results of the pre-trained neural radiation field model to train an independent feature generation model and replaces the original intermediate results to enter the volume rendering stage;
[0061] 3. The present invention reconstructs and trains a new direction for neural radiation field reconstruction, and combines it with the latent space method to improve the quality of three-dimensional reconstruction and corresponding performance indicators. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:
[0063] Attached Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0064] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0065] Please refer to Figure 1 The present invention provides a method for three-dimensional reconstruction of neural radiation field based on potential space diffusion, comprising the following steps:
[0066] Step S1: Use a camera to capture views of various perspectives of a real scene, and use an algorithm to calculate the spatial position and direction of the camera for this view set, and then organize it into a data set for preprocessing;
[0067] Step S2: Input each view and camera pose, sample the scene space, and use the pre-trained neural radiance field model to obtain the color and density information of each sampling point in the scene, and combine them into the features of the sampling point;
[0068] Step S3: Input the features of all sampling points into an encoder, encode them in latent space, obtain the potential representation of the entire space, and restore them into new features with the same shape as the original features through a decoder, and make losses on the original features and the new features, and jointly train the encoder and decoder;
[0069] Step S4: Use latent space diffusion to the latent representation, gradually add noise, obtain the residual of each step, and obtain a random noise representation. Then use the inverse process to input a random noise and gradually restore it to the latent representation. The trained decoder is decoded into new features and the volume rendering method is used to reconstruct the complete three-dimensional scene.
[0070] The step S1 is specifically as follows:
[0071] Step S11: using a handheld camera to capture view images of various perspectives in a real scene, ensuring that different angles and fields of view are covered;
[0072] Step S12: Calculate the position of the camera in space and the shooting direction according to the motion trajectory of the camera and the posture information during shooting;
[0073] Step S13: sorting and preprocessing the calculated camera position and direction data, removing abnormal values and erroneous data, and converting the camera coordinate system into the world coordinate system.
[0074] The step S2 is specifically as follows:
[0075] Step S21: input each view and the corresponding camera pose information into the neural radiation field, including the position coordinates Position and the viewing direction Direction of the camera in the world coordinate system;
[0076] Step S22: Perform random sampling in the scene, take a ray Ray in a random direction, take a sampling point Sample on the ray Ray, and output the position information of all sampling points and input them into the pre-trained neural radiation field model;
[0077] Step S23: Output the intermediate result of the neural radiation field model, that is, the color RGB and density Density corresponding to the spatial position of each sampling point in the space, and combine the color RGB and density Density of each sampling point into a feature vector Feature representing the entire scene Sample .
[0078] The step S3 is specifically as follows:
[0079] Step S31: The obtained neural radiation field scene feature representation vector Feature Sample Input the latent space encoder Encoder and encode it into the potential feature representation Feature of the scene lantent_Sample ;
[0080] Step S32: Feature lantent_Sample Input decoder Decoder, restore it to the same Sample Feature vectors of the same dimension autoencoder_Sample ;
[0081] Step S33: Feature autoencoder_Sample With Feature sample As loss, back propagation and gradient descent are performed to jointly train the encoder Encoder and decoder Decoder.
[0082] The process of the encoder is specifically as follows:
[0083] Let R represent the number of rays taken in space, S represent the number of sampling points on each ray, and represent the scene feature vector Feature with dimension (R, S, 4) Sample Split into a Feature with dimension (R, S, 3) RGBWith a feature of dimension (R, S, 1) Density , and input the encoder Encoder, the two features are projected into Features with dimensions (R, S, N) through the multi-layer perceptron MLP mlp_RGB With Feature mlp_Density (Here N represents 128), use dot product to fuse the two features to get Feature lantent_Sample ; Specifically expressed as:
[0084] Feature RGB , Feature Density =Split(Feature Sample )
[0085] Feature mlp_RGB =MLPS1(Feature RGB )
[0086] Feature mlp_Density =MLPS2(Feature Density )
[0087] Feature lantent_Sample =Feature mlp_Density *Feature mlp_RGB
[0088] Where Split represents the segmentation operation, and MLPS1 and MLPS2 represent two multi-layer perceptrons respectively.
[0089] The training process of the decoder is specifically as follows:
[0090] The obtained feature lantent_Sample The input decoder is projected into a feature with dimensions of (R, S, 3) through multi-layer perceptrons MLPS3 and MLPS4 respectively. Decoder_RGB With the feature of dimension (R, S, 1) Decoder_Density , and concatenate them together according to the last dimension to form a Feature autoencoder_Sample ; Combine it with Feature Sample As MSE loss, the decoder is trained by gradient descent and back propagation; specifically:
[0091] Feature Decoder_RGB =MLPS3(Feature lantent_Sample )
[0092] Feature Decoder_Density =MLPS4(Feature lantent_Sample )
[0093] Feature autoencoder_Sample =Cat(Feature Decoder_RGB , Feature Decoder_Density )
[0094] MLPS3 and MLPS4 represent two multi-layer perceptrons, and Cat represents the concatenation operation.
[0095] Specifically:
[0096] Loss mse =(Feature autoencoder_Sample -Feature Sample ) 2
[0097] Among them, Loss mse Represents the calculated MSE loss value.
[0098] The step S4 is specifically as follows:
[0099] Step S41: Potential representation Feature lantent_Sample Add noise step by step to get a random noise representation;
[0100] Step S42: Input a random noise random , step-by-step denoising, that is, inversely processing the random noise representation to reconstruct it into a new potential representation feature diffusion_lantent_Sample ;
[0101] Step S43: Reconstruct the potential representation feature Feature diffusion_lantent_Sample Input the trained decoder to restore it to the same state as Feature Sample New feature representation of the same dimension Decoder_diffusion_Sample ;
[0102] Step S44: Feature Decoder_diffusion_Sample Intermediate result feature of alternative neural radiation field Sample The subsequent volume rendering process is carried out to reconstruct a complete three-dimensional scene.
[0103] The specific method of adding noise is:
[0104] The potential representation of the feature lantent_Sample Gradually add noise to get a set of noise representation features noise :
[0105]
[0106] Featurelantent_t represents the potential representation of the t-th step of noise addition, Feature lantent_0 represents the initial potential representation, q(Feature lantent_t |Feature lantent_0 ) represents the random noise that needs to be added to the initial features in step t, N represents the set of natural numbers, I represents the set of integers, and α k represents the value of α at the kth step, α t represents the value of α at step t, It means the value of α multiplied from the first step to the tth step at step t. The α of different steps is predefined and gradually decreases.
[0107] When t is the last step, q(Feature noise |Feature lantent_Sample ) represents the process of adding noise from potential feature noise to random noise, which can be expressed as:
[0108] Feature noise =Feature lantent_Sample +q(Feature noise |Feature lantent_Sample ).
[0109] The specific method for removing noise is:
[0110] Noise removal is the reverse process of the above step-by-step noise addition. A U-Net is used to fit the reverse process:
[0111]
[0112] p(Feature lantent_t-1 |Feature lantent_t ) represents the random noise that needs to be removed for the t-1 step feature. Let t and Feature lantent_t It is known that the process of gradual noise addition can be performed by simply using a neural network to fit ε, and the entire training process is equivalent to the process of training the parameters of the U-Net network.
[0113] The specific method of volume rendering is:
[0114] Given the camera position coordinates Position and viewing direction Direction, through the input Feature Decoder_diffusion_Sample , split into the volume density Density and color RGB of each sampling point, and obtain the color COLOR of the object and its surface at the viewing angle of the position; the specific method is as follows:
[0115]
[0116] Where RGB i is the color of the i-th sampling point along the direction, Weight i is the weight of the color of the i-th sampling point along the direction, Density i Represents the density of the i-th sampling point along the direction, Density j Represents the density of the jth sampling point along the direction, Distance i Represents the distance between the i+1th and ith sampling points, Distance j represents the distance between the j+1th and the j+2nd sampling points, S represents the upper limit of the sampling points, which is 65536 here; finally, on the surface of the object in the scene to be synthesized, a complete three-dimensional model is rendered by tracking the camera light. The above description is only a preferred embodiment of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.
Claims
1. A 3D reconstruction method of neural radiation field based on latent space diffusion, characterized by: The following steps are involved: Step S1: Use a camera to capture views of various perspectives of a real scene, and use an algorithm to calculate the spatial position and direction of the camera for this view set, and then organize it into a data set for preprocessing; Step S2: Input each view and camera pose, sample the scene space, and use the pre-trained neural radiance field model to obtain the color and density information of each sampling point in the scene, and combine them into the features of the sampling point; Step S3: Input the features of all sampling points into the encoder to encode the latent space, obtain the potential representation of the entire space, and restore it into a new feature with the same shape as the original feature through the decoder, and make the original feature and the new feature loss, and jointly train the encoder and decoder; Step S4: Use latent space diffusion to the latent representation, gradually add noise, obtain the residual of each step, and obtain the random noise representation. Then use the inverse process to input random noise and gradually restore it to the latent representation. The trained decoder is decoded into new features and the volume rendering method is used to reconstruct the complete three-dimensional scene.
2. The method for 3D reconstruction of neural radiation field based on latent space diffusion according to claim 1, characterized in that: The step S1 is specifically as follows: Step S11: using a handheld camera to capture view images of various perspectives in a real scene, ensuring that different angles and fields of view are covered; Step S12: Calculate the position of the camera in space and the shooting direction according to the motion trajectory of the camera and the posture information during shooting; Step S13: sorting and preprocessing the calculated camera position and direction data, removing abnormal values and erroneous data, and converting the camera coordinate system into the world coordinate system.
3. The method for 3D reconstruction of neural radiation field based on latent space diffusion according to claim 1, characterized in that: The step S2 is specifically as follows: Step S21: input each view and the corresponding camera pose information into the neural radiation field, including the position coordinates Position and the viewing direction Direction of the camera in the world coordinate system; Step S22: Perform random sampling in the scene, take a ray Ray in a random direction, take a sampling point Sample on the ray Ray, and output the position information of all sampling points and input them into the pre-trained neural radiation field model; Step S23: Output the intermediate result of the neural radiation field model, that is, the color RGB and density Density corresponding to the spatial position of each sampling point in the space, and combine the color RGB and density Density of each sampling point into a feature vector Feature representing the entire scene Sample .
4. The method for 3D reconstruction of neural radiation field based on latent space diffusion according to claim 1, characterized in that: The step S3 is specifically as follows: Step S31: The obtained neural radiation field scene feature representation vector Feature Sample Input the latent space encoder Encoder and encode it into the potential feature representation Feature of the scene lantent_Sample ; Step S32: Feature lantent_Sample Input decoder Decoder, restore it to the same Sample Feature vectors of the same dimension autoencoder_Sample ; Step S33: Feature autoencoder_Sample With Feature Sample As loss, back propagation and gradient descent are performed to jointly train the encoder Encoder and decoder Decoder.
5. The method for 3D reconstruction of neural radiation field based on latent space diffusion according to claim 4, characterized in that: The process of the encoder is specifically as follows: Let R represent the number of rays taken in space, S represent the number of sampling points on each ray, and represent the scene feature vector Feature with dimension (R, S, 4) Sample Split into a Feature with dimension (R, S, 3) RGB With a feature of dimension (R, S, 1) Density , and input the encoder Encoder, the two features are projected into Features with dimensions (R, S, N) through the multi-layer perceptron MLP mlp_RGB With Feature mlp_Density (Here N represents 128), use dot product to fuse the two features to get Feature lantent_Sample ; Specifically expressed as: Feature RGB ,Feature Density =Split(Feature Sample ) Feature mlp_RGB =MLPS1(Feature RGB ) Feature mlp_Density =MLPS2(Feature Density ) Feature lantent_Sample =Feature mlp_Density *Feature mlp_RGB Where Split represents the segmentation operation, and MLPS1 and MLPS2 represent two multi-layer perceptrons respectively.
6. The method for 3D reconstruction of neural radiation field based on latent space diffusion according to claim 4, characterized in that: The training process of the decoder is specifically as follows: The obtained feature lantent_Sample The input decoder is projected into a feature with dimensions of (R, S, 3) through multi-layer perceptrons MLPS3 and MLPS4 respectively. Decoder_RGB With the feature of dimension (R, S, 1) Decoder_Density , and concatenate them together according to the last dimension to form a Feature autoancoder_Sample ; Combine it with Feature Sample As MSE loss, the decoder is trained by gradient descent and back propagation; specifically: Feature Decoder_RGB =MLPS3(Feature lantent_Sample ) Feature Deroder_Density =MLPS4(Feature lantent_Sample ) Feature autoe.ncoder_Sample =Cat(Feature Decoder_RGB ,Feature Decoder_Density ) MLPS3 and MLPS4 represent two multi-layer perceptrons, and Cat represents the concatenation operation. The MSE loss is: Loss mse =(Featre autoencoder_Sample -Feature Sample ) 2 Among them, Loss mse Represents the calculated MSE loss value.
7. The method for 3D reconstruction of neural radiation field based on latent space diffusion according to claim 1, characterized in that: The step S4 is specifically as follows: Step S41: Potential representation Feature lantent_Sample Add noise step by step to get a random noise representation; Step S42: Input a random noise random , step-by-step denoising, that is, inversely processing the random noise representation to reconstruct it into a new potential representation feature Feature diffusion_lantent_sample ; Step S43: Reconstruct the potential representation feature Feature diffusion_lantent_Sample Input the trained decoder to restore it to the same state as Feature sample New feature representation of the same dimension Decoder_diffusion_Sample ; Step S44: Feature Deroder_diffusion_Sample Intermediate result feature of alternative neural radiation field Sample The subsequent volume rendering process is carried out to reconstruct a complete three-dimensional scene.
8. The method for 3D reconstruction of neural radiation field based on latent space diffusion according to claim 7, characterized in that: The specific method of adding noise is: The potential representation of the feature lantent_Sample Gradually add noise to get a set of noise representation features noise : Feature lantent_t represents the potential representation of the t-th step of noise addition, Feature lantent_0 represents the initial potential representation, q(Featture lantent_t |Feature lanntent_0 ) represents the random noise that needs to be added to the initial features in step t, N represents the set of natural numbers, I represents the set of integers, and α k represents the value of α at the kth step, α t represents the value of α at step t, It means the value of α multiplied from the first step to the tth step at step t. The α of different steps is predefined and gradually decreases. When t is the last step, q(Feature noise |Feature lantent_Sample ) represents the process of adding noise from potential feature noise to random noise, which can be expressed as: Feature noise =Feature lantent_Sample +q(Feature noise |Feature lantent_Sample )。 9. The method for 3D reconstruction of neural radiation field based on latent space diffusion according to claim 7, characterized in that: The specific method for removing noise is: Noise removal is the reverse process of the above step-by-step noise addition. A U-Net is used to fit the reverse process: p(Feature lanent_t-1 |Feature lantent_t ) represents the random noise that needs to be removed for the t-1 step feature. Let t and Feature lantent_t It is known that the process of gradual noise addition can be performed by simply using a neural network to fit ε, and the entire training process is equivalent to the process of training the parameters of the U-Net network.
10. The method for 3D reconstruction of neural radiation field based on latent space diffusion according to claim 7, characterized in that: The specific method of volume rendering is: Given the camera position coordinates Position and viewing direction Direction, through the input Feature Decoder_diffusion_Sample , split into the volume density Density and color RGB of each sampling point, and obtain the color COLOR of the object and its surface at the viewing angle of the position; the specific method is as follows: Where RGB i is the color of the i-th sampling point along the direction, Weight i is the weight of the color of the i-th sampling point along the direction, Density i Represents the density of the i-th sampling point along the direction, Density j Represents the density of the jth sampling point along the direction, Distance i Represents the distance between the i+1th and ith sampling points, Distance j Represents the distance between the j+1th and jth sampling points, S represents the upper limit of the sampling points, which is 65536 here; finally, on the surface of the object in the scene to be synthesized, a complete three-dimensional model is rendered by tracking the camera light.
Citation Information
Cited By
Remote sensing three-dimensional reconstruction method and system based on combination of NeRF and diffusion model
CN122530418A
