A patch-extraction-based generalizable neural radiance field reconstruction method

Abstract

The application belongs to the technical field of three-dimensional reconstruction and machine learning, and discloses a generalizable neural radiance field reconstruction method based on patch extraction, which is based on a multi-view stereo vision and a volume rendering algorithm of a neural radiance field to implicitly learn a static three-dimensional scene, realize new view synthesis of a complex scene at any angle, and perform three-dimensional reconstruction of the scene. Firstly, two-dimensional image features of a source view and a target view are extracted, and a three-dimensional cost volume is constructed by using a plane scanning algorithm to distort the coordinate system of reference view features through homography transformation. Secondly, the cost volume constructed for the current scene is input into a three-dimensional convolutional neural network to obtain a neural encoding volume. The neural encoding volume is input into a multi-layer perception to regress volume density and color, so as to construct a neural radiance field. Then, the current neural radiance field is used for rendering to obtain a target view, patches of the target view and the reference view are extracted respectively, and reference view patch features and target view patch features are extracted by using a pre-trained VGG-16 low-level network to compare the features. Finally, the average absolute error is calculated to quantify the content feature difference between the target view patch and the reference view patch, and the difference is used as a regular term of an overall loss function to improve the rendering quality of the model. The application can be used for generalizable multi-view three-dimensional reconstruction, can enhance local features through an efficient training mechanism, and can improve the rendering capability for image details and object boundaries.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically focusing on 3D reconstruction techniques based on multi-view geometry. Specifically, this invention proposes a novel method for 3D rendering based on neural radiance fields. This technique aims to improve the rendering effect of generalizable sparse input view 3D reconstruction. This technology has potential applications in fields such as robotics, virtual reality, and augmented reality.

[0002] Vision-based 3D reconstruction technology is widely used in fields such as autonomous driving and digital twins. Image data is inexpensive and readily available, making the recovery of the 3D structure of scenes from images an increasingly important research direction. Although traditional 3D reconstruction algorithms are more widely used in industrial applications, deep learning methods produce reconstruction results closer to real-world scenes, leading to their increasing application in various scenarios. Background Technology

[0003] With the rapid development of 3D technology, 3D reconstruction technology has been widely applied and rapidly developed in many fields. 3D reconstruction technology can reconstruct accurate 3D models from 2D images and can be widely used in cultural heritage protection, urban planning, virtual reality, and many other fields. In recent years, the rapidly developing neural radiation field (NeRF)-based 3D reconstruction technology has achieved significant breakthroughs in reconstruction and rendering. The research goal of NeRF is to synthesize images of the same scene from different perspectives, ultimately reconstructing a 3D representation of the scene. During the rendering process, using different perspectives as input allows for the reconstruction of the scene's view.

[0004] NeRF has achieved significant breakthroughs in neural reconstruction and rendering, but it is limited by long-duration scene-by-scene rendering and lacks generalization ability on new scenes. We propose a generalizable neural radiation field reconstruction method based on patch extraction—PMSNeRF. Our method effectively improves the generalization ability and synthesis quality of new viewpoints. By introducing a VGG-16 low-level network to extract patch features from the target view and performing local feature enhancement during training, our model can better capture image differences and significantly improve reconstruction results. Inspired by TTUR, we perform patch-based local training every 64 global training steps, improving training efficiency and effectively optimizing the overall model performance while maintaining efficient use of computational resources. We demonstrate that PMSNeRF outperforms various generalizable NeRF models on multiple synthetic and real datasets, and has significant advantages in handling rendering artifacts and complex geometric structures.

[0005] Reconstructing and rendering realistic scenes from images is crucial for computer vision applications. In recent years, various NeRF-based methods have been proposed to address the problem of cross-scene training. These methods share the commonality of training the NeRF network based on a set of source views rendered from similar perspectives. While these methods demonstrate good generalization ability for new scenes, the generated image quality still falls short. To address these issues, we propose a generalizable neural radiation field reconstruction method based on patch extraction. Our approach aims to improve the synthesis quality and generalization ability of new perspectives by introducing a VGG-16 low-level network for patch feature extraction from the reference view. An efficient training mechanism is used for local feature enhancement, improving the rendering ability of image details and object boundaries. Summary of the Invention

[0006] This invention trains a model using source views from three neighboring perspectives within a scene as input. Finally, it inputs the coordinates and orientation of a reference view to the model and outputs a rendered image from that viewpoint, thus obtaining the final rendering result of the entire scene. This invention effectively improves the generalization ability and synthesis quality of sparse reconstruction based on neural radiation fields. It renders the target scene using a smaller number of input views, better captures image differences, and has significant advantages in handling rendering artifacts and complex structures.

[0007] This invention combines multi-view stereo geometry and neural radiation fields. First, a 2D Convolutional Neural Network (2D CNN) is used to extract image features F. Then, homography transformation is used to warp the feature map to 128 planes at different depths in the reference view orientation, obtaining the cost volume. A 3D Convolutional Neural Network (3D CNN) is then used as input to output a neural encoding volume. Subsequently, a Multilayer Perceptron (MLP) is used to regress the neural encoding volume to obtain the volume density σ and color c required for final volume rendering. Inspired by the Two Time-Scale Update Rule (TTUR), after every 64 iterations of the above global training, a patch-extracted local training is performed to enhance local features, and a regularization term is added for feature constraint.

[0008] Based on the above-mentioned inventive concept, this invention proposes a generalizable neural radiation field reconstruction method based on patch extraction, which specifically includes the following steps:

[0009] S1 extracts the two-dimensional image features of the source view and the target view, and constructs a three-dimensional cost volume by distorting the coordinate system of the reference view features through homography transformation.

[0010] S2 uses a three-dimensional convolutional neural network to transform the cost volume into a neural coding volume, and then uses a multilayer perceptron to regress the radiation field.

[0011] S3: After every 64 global training steps, a 64×64 patch of the current target view is randomly selected for local training. The target view is rendered according to the parameters of the current multilayer perceptron, and feature extraction is performed on both the reference view and the target view to obtain the corresponding image feature values ​​F. r With F t .

[0012] S4. For global training, we define a loss function based on the pixel difference between the target view and the reference view. In local training, we add a regularized loss term constrained by feature differences and use gradient descent via backpropagation to update the network parameters.

[0013] A generalizable neural radiation field reconstruction method based on patch extraction is disclosed. Step S1 aims to obtain feature maps for the current input reference view and source view, and then obtain the current global training cost volume using a planar scanning algorithm. The cost volume is inferred through geometric perception, and a general function can be learned from the cost volume, which is a crucial factor in the generalizability of the reconstructed neural radiation field. This step specifically includes the following sub-steps:

[0014] S11, Extract 2D Image Features. A 2D convolutional neural network T is used to extract 2D image features, obtaining the neural features of the local image appearance. This neural network consists of seven downsampling convolutional layers. The image... The input is transformed into a feature map.

[0015] F i =T(I i (1)

[0016] Where H and W represent the length and width of the input image, and C is the number of feature channels obtained.

[0017] S12, Distortion Feature Map. After obtaining the feature map, a planar scanning algorithm is used to obtain the distortion features from the source view i to the reference view through homography transformation. Given the camera intrinsic and extrinsic parameters Φ = [K, R, t], the distortion matrix that maps the source image to the viewpoint of the reference view through homography transformation is calculated:

[0018]

[0019] Where Hi (z) is the twist matrix from the source view i to the reference view at depth z, where K is the intrinsic matrix, and R and t are the camera rotation and translation matrices. The features of each source view can be transformed to the reference view through the corresponding twist matrix:

[0020] F i,z (u,v)=F i (H i (z)[u,v,1] T (3)

[0021] F i,z It is the distortion feature map of view i at the reference view depth z. (u,v) represents the pixel position in the reference view.

[0022] S13, Construct the cost volume. The cost volume is constructed from the distorted feature map, and we compute the cost volume using a variance-based metric. For each voxel in the cost volume centered at coordinates (u,v,z), its cost eigenvector is computed as follows:

[0023] P(u,v,z)=Var(F i,z (u,v)) (4)

[0024] Var represents the variance calculated for the M views. The variance-based cost encoding volume encodes the image appearance variations across different input views, which is a key reason why our method can be used across different scenarios.

[0025] A generalizable neural radiation field reconstruction method based on patch extraction is disclosed. Step S2 aims to input the cost volume constructed for the current scene into a three-dimensional convolutional neural network to obtain the neurally encoded volume. This neurally encoded volume is then input into a multilayer perceptron to regress the volume density and color, thereby constructing the neural radiation field. This step specifically includes the following sub-steps:

[0026] S21, using a 3D convolutional neural network to transform the cost volume into a neural coding volume. We train a deep 3D convolutional neural network B to successfully construct the image feature cost volume into a neural feature volume S with C channels.

[0027] S=B(P) (5)

[0028] This 3D neural network is a 3D UNet composed of downsampling convolutions, upsampling convolutions, and skip connections. This 3D convolutional neural network can effectively infer and propagate scene appearance information, thereby obtaining a meaningful scene encoding volume S. Our network can learn meaningful scene geometry and appearance from the encoded voxel neural features.

[0029] S22, the neural coding volume is input into a multilayer perceptron for regression analysis to obtain volume density and color. Trilinear interpolation is then used to obtain the neural feature values ​​at these three-dimensional coordinates from the neural coding volume.

[0030] f = S(x) (6)

[0031] We take the 3D coordinates x and orientation d of any given location as input to a multilayer perceptron. Directly inputting these coordinates and orientation into the network leads to poor rendering performance in representing high-frequency changes in color and geometry. Therefore, we need to use a high-frequency function to map the input to a high-dimensional space, which can better fit data containing high-frequency changes. This is achieved by using the function γ... Mapping to higher dimensions In space, the encoding function is as follows:

[0032] γ(p)=(sin(2 0 πp),cos(2 0 πp),...,sin(2 L-1 πp),cos(2 L-1 πp)) (7)

[0033] γ(·) is applied to the three-dimensional coordinates of x and the orientation d. For γ(x), we set L = 10, and for γ(d), we set L = 4. From three input views of a scene, the volume density and color of the scene can be regressed, thus modeling the entire neural radiation field.

[0034] A generalizable neural radiation field reconstruction method based on patch extraction is proposed. Step S3 aims to improve the reconstruction effect of the radiation field by performing local patch training after reconstructing the current neural radiation field, inspired by TTUR (Theoretical and Technical Research). This is achieved by extracting feature values ​​from the reference view patch and the target view patch rendered from the current radiation field using a pre-trained Visual Geometry Group (VGG) low-level network for local training. This step specifically includes the following sub-steps:

[0035] In step S31, we extract reference view patch features using a pre-trained VGG-16 low-level network and compare them with target view patch features. After every 64 global training iterations, we randomly select a 64×64 pixel reference view patch from the current reference view and render the target view using the currently trained network. Based on the coordinates of the reference view patch, we obtain the corresponding 64×64 pixel target view patch in the target view. Referring to relevant research on style transfer, we selected the VGG low-level network conv3_2 and its preceding convolutional layers.

[0036] F i=VGG low-layer (Patch i (8)

[0037] Thus, the reference view patch feature F is obtained respectively. r With target view patch feature F t The low-level networks of the Visual Geometry Group achieved excellent results in content reconstruction experiments. Our chosen conv3_2 and preceding convolutional layers preserved details such as color and texture information during feature extraction. Our 3D reconstruction experiment required the texture and color of the target view to be infinitely close to the reference view; feature extraction using VGG-16 was a crucial step in this task.

[0038] A generalizable neural radiation field reconstruction method based on patch extraction is proposed. Step S4 aims to ensure that the reference view features and target view features extracted by the VGG-16 low-level network reflect the structural information of the image. We obtain the content feature error between the reference view patch and the target view patch, and use this content error to add a regularization term to the loss function to improve rendering quality. This step specifically includes the following sub-steps:

[0039] S41, quantize the content feature differences and use them as the regularization term of the overall loss function. To quantize the reference view patch features F... r and target view patch feature F t We calculate their mean absolute error:

[0040]

[0041] Where N is the total number of elements in the feature map. This error is used as a regularization term in the overall loss function, making the content of the target view more consistent with that of the reference view.

[0042] We improve rendering quality by adding a regularization term after comparing the content features of the reference view and the target view. Specifically, we compare the content feature differences between the target view patch and the reference view patch, and quantify this difference using image content information captured by the low-level VGG-16 network. This difference is expressed as the mean absolute error L... vgg This is represented and used as a regularization term in the overall loss function to train the network to generate more consistent content information images.

[0043] During global training, our model depends entirely on the rendering supervision of the reference image. We first reconstruct the radiation field using the following loss:

[0044]

[0045] In the local training of the patch extraction, we incorporate the image features of the patch for supervision, and use parameters to balance the image pixel loss and feature loss. The total loss is:

[0046] L = L c +λL vgg (11)

[0047] Where L c The squared loss between the pixel colors of the reference view and the target view is given by L. vgg This represents the difference between features extracted from the VGG-16 network, and λ is a regularization coefficient used to balance the original loss and the feature loss.

[0048] Thus, we have realized a generalizable neural radiation field reconstruction method based on patch extraction.

[0049] Compared with the prior art, the main improvements of the present invention are as follows:

[0050] 1. This invention introduces a generalizable three-dimensional reconstruction method based on neural radiation fields, through...

[0051] The VGG-16 low-level network is used for view patch feature extraction. This enhances the model's ability to capture local details, thereby improving the quality of the synthesized view.

[0052] 2. This invention performs local feature enhancement. During training, we compare the feature differences between the rendered reference view patch and the target view patch to enable the model to effectively adjust the network and improve the rendering effect.

[0053] 3. This invention is based on a skip-step regularization strategy. We learned the TTUR training strategy and performed a patch extraction after every 64 steps of global training. This approach not only improves training efficiency but also optimizes model performance while ensuring efficient use of computational resources. Attached Figure Description

[0054] Figure 1 This invention describes the process of constructing the neural coding volume using a generalizable neural radiation field reconstruction method based on patch extraction.

[0055] Figure 2 This is the overall process of the patch-based generalizable neural radiation field reconstruction method of the present invention.

[0056] Terminology Explanation

[0057] NeRF is an abbreviation for Neural Radiance Fields, which is a method for reconstructing a 3D representation of a scene from images taken from multiple perspectives.

[0058] PMSNeRF is an abbreviation for Patch-based Neural Radiance Fields for Generalizable Multi-View Reconstruction, representing a patch-based generalizable neural radiation field reconstruction method proposed in this invention.

[0059] VGG is an abbreviation for the Visual Geometry Group, which stands for "Deep Convolutional Neural Network Architecture." It is a network that can extract features to achieve classification and localization tasks. VGG-16 indicates a network architecture with 16 convolutional layers.

[0060] TTUR is an abbreviation for Two Time-Scale Update Rule, which is a method to improve convergence, quality, and diversity issues in generative network training. It inspired this invention to perform interval local training. Detailed Implementation

[0061] The present invention will be further described with reference to the accompanying drawings.

[0062] Example

[0063] The process of constructing the neural coding volume for a generalizable neural radiation field reconstruction method based on patch extraction is attached. Figure 1 As shown.

[0064] First, compare the reference view with the source view I. i Input step S1, and extract the two-dimensional features F of the image through the 2D CNN. i The image features of the source view are homography transformed using formula (2) and warped to the reference view perspective using a planar scanning algorithm. Then, the cost feature vector of each voxel is calculated based on the variance metric, and the cost volume of the scene is finally calculated. The cost volume is input into a 3D CNN, and the final output is a neurally encoded volume that encodes the meaningful scene geometry and appearance.

[0065] The implementation process of the patch-based generalizable neural radiation field reconstruction method is attached. Figure 2 As shown, it includes the following steps:

[0066] After obtaining the neural coding volume according to steps S1 and S2, trilinear interpolation is used to obtain the neural coding value f of the three-dimensional coordinates of the target view pixel within the neural coding volume. The corresponding neural coding value of the pixel, its three-dimensional coordinates, and the view orientation are input into a multilayer perceptron (MLP) to regress and obtain the scene's volume density σ and color RGB. The rendered target view image is obtained according to the volume rendering formula, and the loss L is calculated based on the pixel differences. rgbBackpropagation is used to update network parameters. This completes the global training step of the network based on the model before patch extraction. After every 64 global training iterations, local patch extraction training is performed to optimize the network. A 64×64 pixel patch image is randomly extracted from the reference view of the current scene. The 3D coordinates and orientation d of this patch pixel are directly input into a multilayer perceptron (MLP), and volume rendering regression is used to obtain the target patch image corresponding to the target view. The reference view patch and the rendered target view patch are then input into a pre-trained VGG-16 network for feature extraction. Finally, the features F of the reference view patch are compared. r With target view F t Patch features optimize network parameters.

[0067] After the training of the generalizable neural radiation field reconstruction method based on patch extraction is completed, the source views of the scene to be reconstructed from three different perspectives and a reference view are input into the model. By following the steps S1-S4 given above, the radiation field of the scene can be reconstructed and the rendering result of the current scene can be obtained.

[0068] Application examples

[0069] The patch-based generalizable neural radiation field reconstruction method provided in the examples demonstrates the results on three different datasets (DTU dataset, Realistic Synthetic dataset, and Forward-Facing dataset). We trained on only 88 scenes in the DTU dataset and tested it on the other 16 scenes in DTU, as well as the Realistic Synthetic and Forward-Facing datasets.

[0070] Meanwhile, the patch-based generalizable neural radiation field reconstruction method (PMSNeRF) provided in this invention was compared with three different baseline models (PixNeRF, IBRNet, and MVSNeRF), using PSNR, SSIM, and LPIPS as our evaluation metrics. Higher values ​​for both PSNR and SSIM indicate better reconstruction results; lower values ​​for LPIPS also indicate better reconstruction results. The quantitative comparison results of the reconstructions are shown in Table 1.

[0071] Table 1. Quantitative comparison results of novel perspective synthesis

[0072]

[0073] The remaining methods in the table are described below:

[0074] PixelNeRF: It uses a two-dimensional neural network to extract image features from sample points and uses an image encoder to synthesize new perspectives from sparse views, improving the model's generalization ability based on NeRF.

[0075] • IBRNet: Improves generalization by using features from neighboring view images as input.

[0076] • MVSNet: Inspired by MVSNet, it uses multi-view stereo vision to improve the generalization ability of the model and achieve high-quality reconstruction.

[0077] As can be seen from the experimental results in Table 1, the patch-based generalizable neural radiation field reconstruction method PMSNeRF proposed in this invention improves the rendering image quality of the target view and obtains higher quality neural radiation field reconstruction results compared with other baseline models.

[0078] Therefore, the patch-based generalizable neural radiation field reconstruction method proposed in this invention reconstructs new complex scenes using a pre-trained network without training each scene individually. We utilize a multi-view stereo architecture to construct cost volume to learn the geometric features of the scene. Then, inspired by style transfer, we optimize the network by performing local feature enhancement on the extracted patches using a VGG-16 low-level network, and improve training efficiency through interval training. Our work improves the rendering quality of the generalizable neural radiation field model, and metrics on multiple datasets validate the effectiveness of our method.

[0079] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A generalizable neural radiation field reconstruction method based on patch extraction, characterized in that, Includes the following steps: S1, extract the two-dimensional image features of the source view and the target view, and construct the three-dimensional cost volume by distorting the coordinate system of the reference view features through homography transformation; S2 uses a three-dimensional convolutional neural network to transform the cost volume into a neural coding volume, and then regresses the radiation field through a multilayer perceptron. S3, after every 64 global training steps, a 64×64 patch of the current target view is randomly selected for local training. The target view is then rendered according to the parameters of the current multilayer perceptron. Feature extraction is performed on both the reference view and the target view to obtain the corresponding image feature values ​​F. r With F t ; S4. For global training, a loss function is defined based on the pixel difference between the target view and the reference view. In local training, a regularized loss term constrained by feature differences is added, and gradient descent is performed through backpropagation to update the network parameters. In step S1, for the current input reference view and source view, feature maps are obtained, and the current global training cost volume is obtained through a planar scanning algorithm. The cost volume is inferred through geometric perception, and a general function is learned through the cost volume. Step S1 specifically includes the following sub-steps: S11, Extracting 2D Image Features: A 2D convolutional neural network T is used to extract 2D image features, obtaining neural features of the local image appearance. This neural network consists of seven downsampling convolutional layers. The input is transformed into a feature map. F i =T(I i ) (1) Where H and W represent the length and width of the input image, and C is the number of feature channels obtained; S12, Distortion Feature Map: After obtaining the feature map, a planar scanning algorithm is used to obtain the distortion features from the source view i to the reference view through homography transformation. Given the camera intrinsic and extrinsic parameters Φ = [K, R, t], the distortion matrix that maps the source image to the viewpoint of the reference view through homography transformation is calculated. Where H i (z) is the twist matrix from source view i to reference view at depth z, K is the intrinsic matrix, and R and t are the camera rotation and translation matrices. The features of each source view can be transformed to the reference view through the corresponding twist matrix: F i,z (u,v)=F i (H i (z)[u,v,1] T ) (3) F i,z It is the distortion feature map of view i at the reference view depth z, where (u,v) represents the pixel position in the reference view; S13, Construct the cost volume. The cost volume is constructed from the distorted feature map and calculated using a variance-based metric. For each voxel in the cost volume centered at coordinates (u,v,z), its cost feature vector is calculated as follows: P(u,v,z)=Var(F i,z (u,v)) (4) Var represents the variance of the M views, and the cost-encoded volume based on the variance encodes the image appearance variations of different input views; In step S3, after reconstructing the current neural radiation field, inspired by TTUR, a local training is performed for the patch every 64 global training iterations to improve the reconstruction effect of the radiation field. The feature values ​​of the reference view patch and the feature values ​​of the target view patch obtained by rendering the current radiation field are extracted by the pre-trained VGG low-level network for local training. The features extracted from the reference view patch using a pre-trained VGG-16 low-level network are compared with those of the target view patch. After every 64 global training iterations, a 64×64 pixel reference view patch is randomly selected from the current reference view. The target view is then rendered using the currently trained network. Based on the coordinates of the reference view patch, the corresponding 64×64 pixel target view patch is obtained in the target view. The VGG low-level network conv3_2 and the preceding convolutional layers are selected. F i =VGG low-layer (Patch i ) (5) Thus, the reference view patch feature F is obtained respectively. r With target view patch feature F t .

2. The generalizable neural radiation field reconstruction method based on patch extraction according to claim 1, characterized in that, In step S2, the cost volume constructed for the current scene is input into a three-dimensional convolutional neural network to obtain the neural encoded volume. The neural encoded volume is then input into a multilayer perceptron to regress the volume density and color, thereby constructing the neural radiation field. Step S2 specifically includes the following sub-steps: S21, using a 3D convolutional neural network to transform the cost volume into a neural coding volume, train a deep 3D convolutional neural network B to successfully transform the constructed image feature cost volume into a neural feature volume S with C channels: S=B(P) (6) The 3D neural network is a 3D UNet composed of downsampling convolution, upsampling convolution and skip connections. This 3D convolutional neural network can effectively infer and propagate scene appearance information. Thus, a meaningful scene encoding volume S can be obtained through this neural network. The network can learn the meaningful scene geometry and appearance in each voxel neural feature in the encoding. S22, the neural coding volume is input into a multilayer perceptron for regression to obtain volume density and color, and trilinear interpolation is used to obtain neural feature values ​​at three-dimensional coordinates from the neural coding volume: f = S(x) (7) Through the function γ Mapping to higher dimensions In space, the encoding function is as follows: γ(p)=(sin(2 0 πp),cos(2 0 πp),...,sin(2 L-1 πp),cos(2 L-1 (8) γ(·) is applied to the three-dimensional coordinates of x and the orientation d. For γ(x), L = 10, and for γ(d), L = 4. The volume density and color of the scene can be regressed from three input views of a scene, thereby modeling the entire neural radiation field.

3. The generalizable neural radiation field reconstruction method based on patch extraction according to claim 1, characterized in that, In step S4, the reference view features and target view features extracted by the VGG-16 low-level network reflect the structural information of the image, and the content feature error between the reference view patch and the target view patch is obtained. The rendering quality is improved by adding a regularization term to the loss function through the content error. Step S4 specifically includes the following sub-steps: S41, quantize the content feature differences and use them as the regularization term of the overall loss function, in order to quantize the reference view patch features F. r and target view patch feature F t Calculate their mean absolute error: Where N is the total number of elements in the feature map, this error is used as a regularization term in the overall loss function to make the content of the target view more consistent with that of the reference view. Specifically, the differences in content features between the target view patch and the reference view patch are analyzed, and this difference is quantified using image content information captured by the low-level network of VGG-16. This difference is expressed as the mean absolute error L. vgg This is represented as a regularization term in the overall loss function to train the network to generate more consistent content-information images. During global training, the model depends entirely on the rendering supervision of the reference image. The radiation field is first reconstructed using the following loss: In the local training of patch extraction, image features of the patch are incorporated for supervision, and parameters are used to balance image pixel loss and feature loss. The total loss is: L=L c +λL vgg (11) Where L c The squared loss between the pixel colors of the reference view and the target view is given by L. vgg This represents the difference between features extracted from the VGG-16 network, and λ is a regularization coefficient used to balance the original loss and the feature loss.

Images