Patch extraction-based generalized neural radiation field reconstruction method

Abstract

The invention belongs to the technical field of three-dimensional reconstruction and machine learning, and discloses a generalized neural radiation field reconstruction method based on patch extraction, which is used for implicitly learning a static three-dimensional scene based on a volume rendering algorithm of multi-view stereoscopic vision and a neural radiation field so as to realize new view angle synthesis of any angle of a complex scene. And performing three-dimensional reconstruction of the scene. The method comprises the following steps: firstly, extracting two-dimensional image features of a source view and a target view, and constructing a three-dimensional cost volume by using a plane scanning algorithm to distort a coordinate system of reference view features through homography transformation; secondly, for the cost volume constructed by the current scene, inputting the cost volume into a three-dimensional convolutional neural network to obtain a neural coding volume; inputting the neural coding volume into a multi-layer perceptron, and performing regression to obtain volume density and color so as to construct a neural radiation field; then, a target view is obtained through rendering of the current neural radiation field, patches of the target view and a reference view are extracted respectively, and reference view patch features are extracted through a pre-trained VGG-16 low-layer network and compared with the target view patch features; and finally, calculating average absolute error quantization to obtain a content feature difference between the target view patch and the reference view patch, and taking the content feature difference as a regular term of an overall loss function to improve the model rendering quality. The method can be used for generalization multi-view three-dimensional reconstruction, local feature enhancement can be carried out through an efficient training mechanism, and the rendering capability of image details and object boundaries is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular focuses on 3D reconstruction technology of multi-view geometry. Specifically, the present invention proposes a new method for 3D rendering based on neural radiance fields. The technology aims to improve the rendering effect of 3D reconstruction of generalizable sparse input views. The technology has potential applications in the fields of robotics, virtual reality and augmented reality.

[0002] Vision-based 3D reconstruction technology is widely used in fields such as autonomous driving and digital twins. Image data is low-cost and easy to obtain, and restoring the 3D structure of a scene from an image is becoming an increasingly important research direction. Although traditional 3D reconstruction algorithms are widely used in industry, the reconstruction effect of deep learning methods is closer to the real scene, so deep learning algorithms are also being used in more and more scenarios. Background Art

[0003] With the rapid development of 3D technology, 3D reconstruction technology has been widely used and developed rapidly in many fields. 3D reconstruction technology can reconstruct accurate 3D models from 2D images and can be widely used in cultural heritage protection, urban planning, virtual reality and other fields. In recent years, the rapid development of 3D reconstruction technology based on neural radiance field (NeRF) has made great breakthroughs in reconstruction and rendering. The purpose of the research on neural radiance field is to synthesize images of the same scene from different perspectives, and finally reconstruct the 3D representation of the scene. In the rendering process, different perspectives can be used as input to reconstruct the view of the scene.

[0004] NeRF has made great breakthroughs in neural reconstruction and rendering, but it is limited by long-term scene-by-scene rendering and lacks generalization capabilities on new scenes. We propose a generalizable neural radiance field reconstruction method based on patch extraction, PMSNeRF. Our method can effectively improve the generalization ability and synthesis quality of new view synthesis. By introducing the VGG-16 low-level network to extract patch features from the target view and performing local feature enhancement during training, our model can better capture image differences and significantly improve the reconstruction effect. Inspired by TTUR, we perform patch-based local training after every 64 global training steps, which improves training efficiency and effectively optimizes the overall performance of the model while maintaining efficient utilization of computing resources. We show that PMSNeRF outperforms a variety of generalizable NeRF models on multiple synthetic and real datasets, and has significant advantages in dealing with rendering artifacts and complex geometric structures.

[0005] Reconstructing and rendering real scenes through images is crucial for the application of computer vision. In recent years, a variety of NeRF-based methods have been proposed and attempted to solve the problem of cross-scene training of models. The unification of these methods is to train the NeRF network based on a set of target views rendered from source views with similar perspectives. Although these methods also show good generalization ability for new scenes, the generation effect of rendered images is still insufficient. To solve these problems, we propose a generalizable neural radiance field reconstruction method based on patch extraction. Our method aims to improve the synthesis quality and generalization ability of new perspective synthesis. It introduces the patch feature extraction of the VGG-16 low-level network for reference view, enhances local features through an efficient training mechanism, and improves the rendering ability of image details and object boundaries. Summary of the invention

[0006] The present invention uses the source views of three adjacent viewpoints in the scene as input to train the model, and finally inputs the coordinates and orientation of the reference viewpoint into the model to output the rendered image under that viewpoint, so as to finally obtain the rendering result of the entire scene. The present invention effectively improves the generalization ability and synthesis quality of sparse reconstruction based on neural radiation fields, uses a smaller number of input views to render the target scene, can better capture image differences, and has significant advantages in processing rendering artifacts and complex structures.

[0007] The present invention combines multi-view stereo geometry with neural radiation field. First, a two-dimensional convolutional neural network (2D Convolutional Neural Network, 2D CNN) is used to extract image features F, and then the feature map is warped to 128 planes of different depths under the reference view orientation through homography transformation to obtain a cost volume (Cost Volume). Through a three-dimensional convolutional neural network (3D Convolutional Neural Network, 3D CNN), the cost volume is used as input and then a neural encoding volume (Neural Encoding Volume) is output. Subsequently, the volume density σ and color c required for volume rendering are finally obtained by regression from the neural encoding volume through a multilayer perceptron (Multilayer Perceptron, MLP). Inspired by the Two Time-Scale Update Rule (TTUR), we perform local training of patch extraction after every 64 times of the above global training to enhance local features, and add regularization terms for feature constraints.

[0008] Based on the above invention ideas, the present invention proposes a generalizable neural radiation field reconstruction method based on patch extraction, which specifically includes the following steps:

[0009] S1, extracts the 2D image features of the source view and the target view, and constructs a 3D cost volume by distorting the coordinate system of the reference view features through homography transformation.

[0010] S2, uses a 3D convolutional neural network to transform the cost volume into a neural encoding volume and regresses the radiation field through a multi-layer perceptron.

[0011] S3, after every 64 global training steps, a patch of 64×64 size is randomly selected for the current target view for local training. The target view is rendered according to the parameters of the current multi-layer perceptron, and the feature extraction of the reference view and the target view is performed to obtain the corresponding feature value F of the image. r With F t .

[0012] S4, for global training, we define the loss function based on the pixel difference between the target view and the reference view. In local training, we add a regularized loss term constrained by feature difference and perform gradient descent via back-propagation to update the network parameters.

[0013] A generalizable neural radiation field reconstruction method based on patch extraction, the purpose of step S1 is to obtain a feature map for the currently input reference view and source view and obtain the current global training cost volume through a plane scanning algorithm. The cost volume is inferred through geometric perception, and a general function can be learned through the cost volume, which is an important factor for the generalizability of the reconstructed neural radiation field. This step specifically includes the following sub-steps:

[0014] S11, extracting two-dimensional image features. The two-dimensional image features are extracted through a two-dimensional convolutional neural network T to obtain neural features of the local image appearance. The neural network consists of seven layers of downsampling convolutional layers. The input is converted into a feature map

[0015] F i =T(I i ) (1)

[0016] Where H and W represent the length and width of the input image, and C is the number of feature channels obtained.

[0017] S12, warping feature map. After obtaining the feature map, the plane scanning algorithm is used to obtain the warping feature of the source view i to the reference view through homography transformation. Given the camera internal and external parameters Φ = [K, R, t] in the data set, the warping matrix of the homography transformation that maps the source image to the reference view is calculated:

[0018]

[0019] Among them, H i(z) is the distortion matrix from source view i to the reference view at depth z, K is the intrinsic matrix, R and t are the camera rotation and translation matrices, and the features of each source view can be transformed to the reference view through the corresponding distortion matrix:

[0020] F i,z (u,v)=F i (H i (z)[u,v,1] T ) (3)

[0021] F i,z is the warped feature map of view i at the reference view depth z. (u,v) represents the pixel position in the reference view.

[0022] S13, construct the cost volume. The cost volume is constructed by the warped feature map, and we use a variance-based metric to calculate the cost volume. For each voxel in the cost volume centered at coordinates (u, v, z), its cost feature vector is calculated as follows:

[0023] P(u,v,z)=Var(F i,z (u,v)) (4)

[0024] Var represents the variance of the M views. The variance-based cost coding volume encodes the image appearance changes of different input views, which is also an important reason why our method can be cross-scene.

[0025] A generalizable neural radiation field reconstruction method based on patch extraction, the purpose of step S2 is to input the cost volume constructed by the current scene into a three-dimensional convolutional neural network to obtain a neural coding volume. The neural coding volume is input into a multi-layer perceptron, and the volume density and color are regressed to construct a neural radiation field. This step specifically includes the following sub-steps:

[0026] S21, using a 3D convolutional neural network to convert the cost volume into a neural encoding volume. We train a deep 3D convolutional neural network B to convert the successfully constructed image feature cost volume into a neural feature volume S with C channels.

[0027] S=B(P) (5)

[0028] The 3D neural network is a 3D UNet, which consists of downsampling convolution, upsampling convolution and skip connection. The 3D convolutional neural network can effectively infer and propagate scene appearance information, so that a meaningful scene encoding volume S can be obtained through the neural network, and our network can learn the meaningful scene geometry and appearance in the neural features of each voxel in the encoding.

[0029] S22, input the neural coding volume into the multi-layer perceptron to regress the volume density and color. Use trilinear interpolation to obtain the neural feature value at the three-dimensional coordinate from the neural coding volume.

[0030] f=S(x) (6)

[0031] The three-dimensional coordinates x and direction d of a given arbitrary position are used as the input of the multi-layer perceptron. Directly inputting the coordinates and orientation into the network will result in poor rendering performance in representing high-frequency changes in color and geometric shapes. Therefore, we need to use a high-frequency function to map the input to a high-dimensional space, which can better fit data containing high-frequency changes. Mapping to high dimensional In the space, the encoding function is as follows:

[0032] γ(p)=(sin(2 0 πp),cos(2 0 πp),...,sin(2 L-1 πp),cos(2 L-1 πp)) (7)

[0033] γ(·) is applied to the 3D coordinates of x and the orientation d, and for γ(x) we set L = 10, and for γ(d) we set L = 4. From three input views of a scene, we can regress the volume density and color of the scene, thereby modeling the entire neural radiance field.

[0034] A generalizable neural radiation field reconstruction method based on patch extraction, the purpose of step S3 is to reconstruct the current neural radiation field. Inspired by TTUR, we perform local training on the patch every 64 global trainings to improve the reconstruction effect of the radiation field. The feature values ​​of the reference view patch and the feature values ​​of the target view patch obtained by the current radiation field rendering are extracted through the pre-trained Visual Geometry Group (VGG) low-level network for local training. This step specifically includes the following sub-steps:

[0035] S31, use the pre-trained VGG-16 low-level network to extract the reference view patch features and compare them with the target view patch features. After every 64 global trainings, we randomly obtain a 64×64 pixel reference view patch in the current reference view, and use the network obtained by the current training to render the target view. According to the coordinates of the reference view patch, the corresponding 64×64 pixel target view patch in the target view is obtained. We refer to the relevant research on style transfer and choose the VGG low-level network conv3_2 and the previous convolutional layers.

[0036] F i=VGG low-layer (Patch i ) (8)

[0037] Thus, the reference view patch features F r and the target view patch feature F t The low-level network of the Visual GeometryGroup achieved excellent results in the content reconstruction experiment. The conv3_2 and previous convolutional layers we used will retain the details, such as color and texture information, when extracting features. Our 3D reconstruction task experiment requires that the texture and color of the target view are infinitely close to the reference view. Feature extraction through VGG-16 is an important step in the task.

[0038] A generalizable neural radiation field reconstruction method based on patch extraction, the purpose of step S4 is that the reference view features and target view features extracted by the VGG-16 low-level network reflect the structural information of the image. We obtain the content feature error between the reference view patch and the target view patch, and add the regularization term to the loss function through the content error to improve the rendering quality. This step specifically includes the following sub-steps:

[0039] S41, quantize the content feature difference and use it as the regularization term of the overall loss function. In order to quantize the reference view patch feature F r and the target view patch feature F t , we calculate their mean absolute error:

[0040]

[0041] Where N is the total number of elements in the feature map. This error is used as a regularization term in the overall loss function to make the target view more consistent with the reference view content.

[0042] We improve the rendering quality by comparing the content features of the reference view and the target view and adding a regularization term. Specifically, we refer to the difference in content features between the target view patch and the reference view patch, and quantify this difference by the image content information captured by the low-level network of VGG-16. The difference is expressed by the mean absolute error L vgg Representation and used as a regularization term in the overall loss function to train the network to generate images with more consistent content information.

[0043] When training globally, our model depends entirely on the rendering supervision of the reference image. We first reconstruct the radiance field using the following loss:

[0044]

[0045] In the local training of patch extraction, we add the image features of the patch for supervision and balance the image pixel loss and feature loss through parameters. The total loss is:

[0046] L=L c +λL vgg (11)

[0047] Where L c is the squared loss between the pixel colors of the reference view and the target view, where L vgg represents the difference between the features extracted from the VGG-16 network, and λ is the regularization coefficient used to balance the original loss and feature loss.

[0048] Thus, we have achieved a generalizable neural radiance field reconstruction method based on patch extraction.

[0049] Compared with the prior art, the main improvements of the present invention are as follows:

[0050] 1. This invention introduces a generalizable 3D reconstruction method based on neural radiation field, and extracts view patch features through the VGG-16 low-level network. This enhances the model's ability to capture local details, thereby improving the quality of the synthesized view.

[0051] 2. The present invention performs local feature enhancement. During the training process, we compare the feature differences between the reference view patch and the target view patch obtained by rendering, so that the model can effectively adjust the network and improve the rendering effect.

[0052] 3. Based on the step-skipping regularization strategy, we learned the training strategy of TTUR and performed patch extraction after every 64 steps of global training. This method not only improves the training efficiency, but also optimizes the model performance while ensuring efficient use of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the process of constructing a neural coding volume in the generalizable neural radiation field reconstruction method based on patch extraction of the present invention.

[0054] Figure 2 This is the overall process of the generalizable neural radiation field reconstruction method based on patch extraction of the present invention.

[0055] Explanation of terms

[0056] NeRF is the abbreviation of Neural Radiance Fields, which means "neural radiation field". It is a method to reconstruct the 3D representation of a scene through images of multiple perspectives of a scene. Its implementation method refers to [B.Mildenhall, PP Srinivasan, M. Tancik, JT Barron, R. Ramamoorthi and R. Ng, "NeRF: Representing scenes as neural radiance fields for view synthesis", Proc. Eur. Conf. Comput. Vis., pp. 405-421, 2020.]

[0057] PMSNeRF is the abbreviation of Patch-based Neural Radiance Fields for Generalizable Multi-View Reconstruction, which means a generalizable neural radiance field reconstruction method based on patch extraction proposed in the present invention.

[0058] VGG is the abbreviation of the Visual Geometry Group, which stands for "deep convolutional neural network architecture". It is a network that can extract features to achieve classification and positioning tasks. VGG-16 refers to a network architecture with 16 convolutional layers. Its implementation method is referenced in [K.Simonyan and A.Zisserman, "Very deep convolutional networks for large-scale image recognition", 2014.]

[0059] TTUR is the abbreviation of Two Time-Scale Update Rule, which means "two time-scale update rule", which is a method to improve the convergence, quality and diversity problems in generative network training. It inspired the present invention to perform interval local training. Its implementation method is referenced in the literature [M.Heusel, H.Ramsauer, T.Unterthiner, B.Nessler and S.Hochreiter, "GANs trained by a two time-scale update rule converge to alocal nash equilibrium", Proc.31st Int.Conf.Neural Inf.Process.Syst., pp.6626-6637, 2017.] DETAILED DESCRIPTION

[0060] The present invention is further described in conjunction with the accompanying drawings.

[0061] Example

[0062] The process of constructing the neural coding volume based on the generalizable neural radiation field reconstruction method based on patch extraction is shown in the attached figure. Figure 1 shown.

[0063] First, compare the reference view with the source view I i Input step S1, and extract the two-dimensional features F of the image through the 2D CNN i , the image features of the source view are transformed homographically using formula (2), and warped to the reference view perspective through a plane scanning algorithm. Then the cost feature vector of each voxel is calculated based on the variance metric, and finally the cost volume of the scene is calculated. The cost volume is input into the 3D CNN, and finally the neural encoding volume that encodes the meaningful scene geometry and appearance in the scene is output.

[0064] The implementation process of the generalizable neural radiation field reconstruction method based on patch extraction is shown in the attached figure. Figure 2 As shown, it includes the following steps:

[0065] After obtaining the neural coding volume according to the steps of S1 and S2, the neural coding value f of the three-dimensional coordinates of the target view pixel is obtained in the neural coding volume through trilinear interpolation. The corresponding neural coding value of the pixel, its three-dimensional coordinates and the view orientation are input into the multi-layer perceptron MLP, and the volume density σ and color rgb of the scene are regressed. The rendered target view image is obtained according to the volume rendering formula, and then the loss L is calculated according to the pixel difference. rgb , and use back propagation to update the network parameters. So far, the global training step of the network based on the model before patch extraction has been carried out. After every 64 global trainings, local patch extraction training is performed to optimize the network. Randomly extract a patch image of 64×64 pixels in the reference view of the current scene, directly input the three-dimensional coordinates and orientation d of the patch pixel into the multi-layer perceptron MLP, and use volume rendering regression to obtain the patch image Target Patch corresponding to the target view. The reference view patch and the rendered target view patch are respectively input into the pre-trained VGG-16 network for feature extraction. Finally, compare the reference view patch feature F r With the target view F t Patch features optimize network parameters.

[0066] After the model training of the generalizable neural radiation field reconstruction method based on patch extraction is completed, the source views of three different perspectives of the scene to be reconstructed and a reference view are input into the model. According to the steps S1-S4 given above, the radiation field of the scene can be reconstructed to obtain the rendering result of the current scene.

[0067] Application Examples

[0068] The generalizable neural radiation field reconstruction method based on patch extraction provided in the embodiment is used to reconstruct three different data sets (DTU data set, Realistic Synthetic data set, Forward-Facing data set, the source of DTU data set can be found in references [Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola and Henrik "Large scale multi-view stereopsis evaluation", Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 406-413, 2014.】,Realistic Synthetic dataset source see reference [B. Mildenhall, PPSrinivasan, M. Tancik, JTBarron, R. Ramamoorthi and R. Ng,"NeRF: Representing scenes as neural radiance fields for view synthesis", Proc. Eur. Conf. Comput. Vis., pp. 405-421, 2020.】,Forward-Facing dataset source see reference [Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, et al.,"Local light field fusion: Practical view synthesis with prescriptive sampling guidelines", ACM Transactions on Graphics (SIGGRAPH), 2019.】) We only train on 88 scenes in the DTU dataset and test on the other 16 scenes of DTU as well as the Realistic Synthetic dataset and the Forward-Facing dataset.

[0069] At the same time, the generalizable neural radiation field reconstruction method (PMSNeRF) based on patch extraction provided by the present invention is compared with three different baseline models (PixNeRF, IBRNet, MVSNeRF), using PSNR, SSIM, and LPIPS as our evaluation indicators. The larger the value of PSNR and SSIM, the better the reconstruction effect; the smaller the LPIPS value, the better the reconstruction effect. The quantitative comparison results of the reconstruction are shown in Table 1:

[0070] Table 1 Quantitative comparison results of new view synthesis

[0071]

[0072] The remaining methods in the table are described as follows:

[0073] PixelNeRF: Uses a two-dimensional neural network to extract image features from sample points and uses an image encoder to synthesize new perspectives from sparse views, improving the generalization ability of the model based on NeRF. [A.Yu, V.Ye, M.Tancik and A.Kanazawa, "PixelNeRf: Neural radiance fields from one or few images", Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit., pp. 4578-4587, 2021.]

[0074] IBRNet: Improves generalization ability by leveraging neighboring view image features as input [Wang Qianqian, Wang Zhicheng, Genova Kyle, P. Srinivasan Pratul, Zhou Howard, T. Barron Jonathan, et al., "Ibrnet: Learning multi-view image-based rendering", CoRR, vol. abs / 2102.13090, 2021.].

[0075] MVSNeRF: Inspired by MVSNet [Y.Yao,Z.Luo,S.Li,T.Fang and L.Quan, "MVSNet: Depthinference for unstructured multi-view stereo",European Conf.on Computer Vision(ECCV),2018.], it uses multi-view stereo vision to improve the generalization ability of the model and achieve high-quality reconstruction. [A.Chen,Z.Xu,F.Zhao,X.Zhang,F.Xiang,J.Yu,et al., "MVSNeRF:Fast generalizableradiance field reconstruction from multi-view stereo",Intl.Conf.on Computer Vision(ICCV),pp.14124-14133,2021.]

[0076] It can be seen from the experimental results in Table 1 that the generalizable neural radiation field reconstruction method PMSNeRF based on patch extraction proposed in the present invention improves the rendering image quality of the target view and obtains higher quality neural radiation field reconstruction results compared with other baseline models.

[0077] It can be seen that the patch-extraction-based generalizable neural radiance field reconstruction method proposed in the present invention reconstructs new complex scenes using a pre-trained network without training each scene one by one. We use the multi-view stereo architecture to construct a cost volume to learn the geometric features of the scene. Then, inspired by style transfer, the network is optimized by enhancing the local features of the extracted patches through the VGG-16 low-level network, and the training efficiency is improved through interval training. Our work improves the rendering quality based on the generalizable model of neural radiance fields, and the indicator results on multiple datasets verify the effectiveness of our method.

[0078] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make other various specific variations and combinations that do not deviate from the essence of the present invention based on these technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A generalizable neural radiation field reconstruction method based on patch extraction, characterized in that: The following steps are involved: S1, extract the 2D image features of the source view and the target view, and construct a 3D cost volume by distorting the coordinate system of the reference view features through homography transformation. S2, use a 3D convolutional neural network to convert the cost volume into a neural coding volume, and regress the radiation field through a multi-layer perceptron. S3, after every 64 global training steps, randomly select a 64×64 patch of the current target view for local training. Render the target view according to the parameters of the current multi-layer perceptron, and extract features from the reference view and the target view to obtain the corresponding feature value F of the image. r With F t S4, for global training, we define the loss function based on the pixel difference between the target view and the reference view. In local training, we add a regularized loss term constrained by feature difference and perform gradient descent via back-propagation to update the network parameters.

2. The generalizable neural radiation field reconstruction method based on patch extraction according to claim 1, characterized in that: In step S1, for the currently input reference view and source view, a feature map is obtained and the cost volume of the current global training is obtained through a plane scanning algorithm. The cost volume is inferred through geometric perception, and a general function can be learned through the cost volume. This step S1 specifically includes the following sub-steps: S11, extracting two-dimensional image features. The two-dimensional image features are extracted through a two-dimensional convolutional neural network T to obtain neural features of the local image appearance. The neural network consists of seven layers of downsampling convolutional layers. The image The input is converted into a feature map F i =T(I i ) (1) Where H and W represent the length and width of the input image, and C is the number of feature channels obtained. S12, warping feature map. After obtaining the feature map, the plane scanning algorithm is used to obtain the warping feature of the source view i to the reference view through homography transformation. Given the camera internal and external parameters Φ = [K, R, t] in the data set, the warping matrix of the homography transformation that maps the source image to the reference view is calculated: Among them, H i (z) is the distortion matrix from source view i to the reference view at depth z, K is the intrinsic matrix, R and t are the camera rotation and translation matrices, and the features of each source view can be transformed to the reference view through the corresponding distortion matrix: F i,z (u,v)=F i (H i (z)[u,v,1] T ) (3) F i,z is the warped feature map of view i at the reference view depth z. (u,v) represents the pixel position in the reference view. S13, construct the cost volume. The cost volume is constructed by the warped feature map, and we use a variance-based metric to calculate the cost volume. For each voxel in the cost volume centered at coordinates (u, v, z), its cost feature vector is calculated as follows: P(u,v,z)=Var(F i,z (u,v)) (4) Var represents the variance calculated for M views. The variance-based cost coding volume encodes the image appearance variations across different input views.

3. The generalizable neural radiation field reconstruction method based on patch extraction according to claim 1, characterized in that: In step S2, the cost volume constructed for the current scene is input into a three-dimensional convolutional neural network to obtain a neural coding volume. The neural coding volume is input into a multi-layer perceptron to regress the volume density and color, thereby constructing a neural radiation field. This step S2 specifically includes the following sub-steps: S21, using a 3D convolutional neural network to convert the cost volume into a neural encoding volume. We train a deep 3D convolutional neural network B to convert the successfully constructed image feature cost volume into a neural feature volume S with C channels. S=B(P) (5) The 3D neural network is a 3D UNet, which consists of downsampling convolution, upsampling convolution and skip connection. The 3D convolutional neural network can effectively infer and propagate scene appearance information, so that a meaningful scene encoding volume S can be obtained through the neural network, and our network can learn the meaningful scene geometry and appearance in the neural features of each voxel in the encoding. S22, input the neural coding volume into the multi-layer perceptron to regress the volume density and color. Use trilinear interpolation to obtain the neural feature value at the three-dimensional coordinate from the neural coding volume. f=S(x) (6) The three-dimensional coordinates x and direction d of a given arbitrary position are used as the input of the multi-layer perceptron. Directly inputting the coordinates and orientation into the network will result in poor rendering performance in representing high-frequency changes in color and geometric shapes. Therefore, we need to use a high-frequency function to map the input to a high-dimensional space, which can better fit data containing high-frequency changes. Mapping to high dimensional In the space, the encoding function is as follows: γ(p)=(sin(2 0 πp),cos(2 0 πp),...,sin(2 L-1 πp),cos(2 L-1 (7) γ(·) is applied to the 3D coordinates of x and the orientation d, and for γ(x) we set L = 10, and for γ(d) we set L = 4. From three input views of a scene, we can regress the volume density and color of the scene, thereby modeling the entire neural radiance field.

4. The generalizable neural radiation field reconstruction method based on patch extraction according to claim 1, characterized in that: In step S3, after reconstructing the current neural radiation field, inspired by TTUR, we perform local training on the patch every 64 global trainings to improve the reconstruction effect of the radiation field. The feature values ​​of the reference view patch and the feature values ​​of the target view patch rendered by the current radiation field are extracted through the pre-trained Visual Geometry Group (VGG) low-level network for local training. The pre-trained VGG-16 low-level network is used to extract the reference view patch features and compare them with the target view patch features. After every 64 global trainings, we randomly obtain a 64×64 pixel reference view patch in the current reference view and use the network obtained by the current training to render the target view. According to the coordinates of the reference view patch, the corresponding 64×64 pixel target view patch in the target view is obtained. We refer to the relevant research on style transfer and select the VGG low-level network conv3_2 and the previous convolutional layers. F i =VGG low-layer (Patch i ) (8) Thus, the reference view patch features F r and the target view patch feature F t The low-level network of the Visual GeometryGroup achieved excellent results in the content reconstruction experiment. The conv3_2 and previous convolutional layers we used will retain the details, such as color and texture information, when extracting features. Our 3D reconstruction task experiment requires that the texture and color of the target view are infinitely close to the reference view. Feature extraction through VGG-16 is an important step in the task.

5. The generalizable neural radiation field reconstruction method based on patch extraction according to claim 1, characterized in that: In step S4, the reference view features and target view features extracted by the VGG-16 low-layer network reflect the structural information of the image, and the content feature error between the reference view patch and the target view patch is obtained. The regularization term is added to the loss function through the content error to improve the rendering quality. Step S4 specifically includes the following sub-steps: S41, quantize the content feature difference and use it as the regularization term of the overall loss function. In order to quantize the reference view patch feature F r and the target view patch feature F t , we calculate their mean absolute error: Where N is the total number of elements in the feature map. This error is used as a regularization term in the overall loss function to make the target view more consistent with the reference view content. We improve the rendering quality by comparing the content features of the reference view and the target view and adding a regularization term. Specifically, we refer to the difference in content features between the target view patch and the reference view patch, and quantify this difference by the image content information captured by the low-level network of VGG-16. The difference is expressed by the mean absolute error L vgg Representation and used as a regularization term in the overall loss function to train the network to generate images with more consistent content information. When training globally, our model depends entirely on the rendering supervision of the reference image. We first reconstruct the radiance field using the following loss: In the local training of patch extraction, we add the image features of the patch for supervision and balance the image pixel loss and feature loss through parameters. The total loss is: L=L c +λL vgg (11) Where L c is the squared loss between the pixel colors of the reference view and the target view, where L vgg represents the difference between the features extracted from the VGG-16 network, and λ is the regularization coefficient used to balance the original loss and feature loss.

Images