Neural radiation field-based few-sample view synthesis method

Abstract

The invention provides a few-sample view synthesis method based on a neural radiation field. The method comprises the following steps: carrying out data sampling on a group of static images of a scene or an object to generate input data containing a spatial position and a visual angle direction; encoding the data through Fourier feature mapping, and introducing a frequency regularization strategy to control introduction of high-frequency information; implicit reconstruction of a scene is carried out by using a multi-input segmentation network, and early termination is carried out according to uncertainty in each stage, so that the calculation complexity is reduced and the rendering efficiency is improved; a final rendered image is generated by stage rendering and aggregating the output of each stage. According to the method, through the improved Fourier feature mapping and segmented network structure, the rendering precision of the neural radiation field in few-sample view synthesis is effectively improved, noise artifacts are reduced, and the training efficiency and the image quality are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a few-sample view synthesis method based on neural radiation field. Background Art

[0002] With the rapid development of virtual reality and computer graphics, methods based on neural radiance fields (NeRF) have achieved remarkable results in 3D scene reconstruction and view synthesis. NeRF approximates the radiance field of a 3D scene through a neural network and can generate realistic images. However, in the task of few-shot view synthesis, NeRF is prone to noise artifacts, especially under low-quality or insufficient input images, resulting in blurry and unstable synthesized images.

[0003] In addition, traditional neural radiation field methods mainly rely on a large amount of training data and the learning of high-frequency signals, which easily leads to the model overfitting low-frequency information and neglecting the restoration of details when facing complex scenes. Although some methods have improved the performance of neural networks through position encoding mapping and deep learning technology, there are still problems such as insufficient capture of high-frequency information and poor stability in high-frequency areas, especially in resource-constrained scenarios. These problems are particularly prominent.

[0004] At present, although the multi-input network and segmented network in the existing technology can reduce the computational complexity and improve the training efficiency to a certain extent, it is still difficult to effectively suppress noise artifacts and reconstruct details when dealing with few-sample scenes. The lack of effective frequency regularization strategy and reasonable control of different frequency components makes the performance of existing methods unstable in complex scenes and difficult to meet the requirements of high-precision rendering and efficient computing. Therefore, a new method is urgently needed to solve the problem of noise artifacts in few-sample view synthesis and improve rendering efficiency and the quality of synthesized images. Summary of the invention

[0005] In view of this, the present invention provides a few-sample view synthesis method based on neural radiation field to solve the noise artifact problem in few-sample view synthesis and improve rendering efficiency and the quality of synthesized images.

[0006] The present invention provides a few-sample view synthesis method based on neural radiation field, the method comprising:

[0007] Obtain static image data of a scene or object and extract information about its spatial position and viewing direction;

[0008] Perform Fourier feature mapping on the input data, encode the spatial position and viewing direction into a high-dimensional feature vector, and dynamically introduce high-frequency information in combination with the frequency regularization strategy to gradually improve the network's ability to capture high-frequency details;

[0009] A multi-input segmentation network is used to implicitly reconstruct the scene, gradually predicting the color and volume density of spatial points from coarse to fine, reducing computational complexity and improving training efficiency;

[0010] Combined with the early termination mechanism, when the uncertainty of the network prediction is lower than a certain threshold, the calculation is stopped in advance, thereby reducing unnecessary calculations and improving rendering speed;

[0011] Using stage rendering technology, the rendering results of each stage are aggregated, and the final pixel color is calculated through the volume rendering formula to generate a high-quality synthetic image.

[0012] The present invention also introduces a random Fourier feature filtering method to enhance the local control ability of the network, further reduce noise artifacts and restore details. This method effectively improves the realism and stability of the rendered image, especially when processing a small number of samples, and can greatly improve the quality of the synthesized view. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flow chart of the view synthesis method;

[0012] Figure 2 It is a diagram of the Fourier feature map segmentation network structure;

[0013] Figure 3 Output module structure. DETAILED DESCRIPTION

[0014] This paper proposes a few-sample view synthesis method based on neural radiation field, which is mainly used in the optimization and evaluation of multi-scenario interactive medical science practice system. By combining Fourier feature mapping and frequency regularization method, the data sampling process can be accurately controlled to ensure the diversity of training data and the efficient learning ability of the model.

[0015] The core technical solution of the present invention is based on a deep learning framework. It optimizes the data sampling process by processing the Fourier feature mapping of spatial position and viewing direction, and combines the frequency mask control strategy to enhance the diversity and robustness of data sampling.

[0016] This solution uses dynamic frequency masking technology to gradually introduce high-frequency features at different training stages, avoiding over-reliance on high-frequency noise in the early stages and ensuring the learning of low-frequency features in the early stages of training. At the same time, the network can dynamically adjust the sampling strategy to improve its adaptability to different scenarios and conditions.

[0017] The present invention introduces data sampling and frequency regularization techniques:

[0018] The present invention enhances the feature learning ability of the model in different frequency ranges by dynamically sampling the Fourier eigenvectors of spatial position and viewing direction, and has good adaptability to high-frequency details in complex environments.

[0019] At the same time, by dynamically adjusting the frequency mask strategy, the speed of introducing high-frequency information can be effectively controlled, so that the network pays more attention to low-frequency information in the early stage of training. As the training deepens, high-frequency information is gradually introduced, ultimately improving the model's ability to capture detailed features.

[0020] In order to make the technical solution and operation flow of the present invention clearer, the application of data sampling and Fourier feature mapping is described in detail below in conjunction with a specific flowchart.

[0021] Figure 1 The overall workflow of an efficient view synthesis method combining Fourier features and early termination is shown in.

[0022] The present invention proposes a data sampling strategy based on dynamic frequency mask to improve the efficiency of model training and ensure that information of different frequencies can be effectively learned.

[0023] Starting from the left, a set of static images of a scene or object are processed first.

[0024] The input spatial position p and viewing direction d are encoded by Fourier features, thereby converting low-frequency and high-frequency information into high-dimensional feature vectors.

[0025] For the spatial position p = (x, y, z) and the viewing direction d = (θ, φ), we first perform position encoding and convert it into a Fourier feature vector, which is specifically expressed as: p′=[p,γ S (p)] and d′=[d, γ S (d)] Among them, γ S (p) and γ S (d) represents the eigenvector obtained by Fourier mapping function.

[0026] Then Figure 1 The second step in the Fourier feature mapping is to map the low-frequency spatial position and perspective information to a higher-dimensional space, thereby helping the network to better capture the spatial structure.

[0027] In order to gradually introduce more high-frequency information, the present invention adopts a frequency masking strategy, which linearly increases the visibility of frequency components in the Fourier eigenvector according to the training time step t, thereby avoiding the influence of high-frequency noise in the early stage of training.

[0028] For a feature vector p′ of length L+3, we introduce a linearly increasing frequency mask α to adjust the visibility of different frequency components in the Fourier feature vector according to the training time step.

[0029] The frequency mask function α(t, T, L) controls the visibility of different frequency components and is specifically expressed as: Where t is the current training iteration, T is the total number of training iterations, and L is the length of the Fourier eigenvector.

[0030] Through this frequency masking strategy, the network mainly learns low-frequency information in the early stage of training, and gradually introduces more high-frequency information as the training progresses, thereby improving the expressiveness of the model.

[0031] In addition, in order to reduce noise artifacts and enhance the local control ability of the network, the present invention combines a filtering method based on random Fourier features. The method includes a smoothing operator and a restoration operator, which can effectively restore detail information.

[0032] First, the smoothing operator is implemented through spherical normalization, and then the normalization result is multiplied by the scaling vector to implement the restoration operator. The scaling vector is the result of linear transformation of another Fourier feature of the input data, which is used to control the restoration sharpness at each position.

[0033] The smoothing operator is implemented by spherical normalization, while the restoration operator enhances the details by scaling the vector of Fourier features. The specific process is: y1=θ1(ω L (t, T; p′) y i =o(θ i (y i-1 ))⊙F x , i = 2, ..., k - 1 y k =θ k (y k-1 ) Among them, θ i is the network module of the i-th layer, o(v) = v / ||v|| is the spherical normalization operation, F x is the scaling vector for the restore operation.

[0034] Scaling vector F x Defined as: F x =[cos(2πpB T ), sin(2πpB T )]M T Among them, B is a matrix randomly drawn from the standard normal distribution, and M is the linear transformation matrix.

[0035] Through this filtering method, the network can not only effectively remove high-frequency noise, but also restore the details in the image and improve the quality of the final rendering effect.

[0036] In order to reduce the computational complexity of the model and shorten the training time, the present invention adopts a multi-input segmented network structure and combines it with an early termination mechanism to optimize the computational efficiency and training time of the model.

[0037] Figure 2 The specific process of the multi-input segmented network structure is demonstrated. There are four stages of segmented networks in total, and each stage network consists of three modules: network module, branch module, and output module.

[0038] In each stage of the network, the input features are first processed using the network module, and then the uncertainty of each query sample is predicted through the branch module. If the uncertainty is lower than the predetermined threshold, the sample can exit early and obtain the final prediction result; otherwise, the sample will enter the next more powerful stage network.

[0039] The multi-input segmentation network structure consists of four stages, each of which includes three modules: network module, branch module and output module.

[0040] The network module is responsible for feature processing and reduces the amount of calculation;

[0041] The branch module predicts uncertainty and decides whether to forward the query sample to the next stage;

[0042] The output module decodes the latent features into color and volume density to complete the rendering.

[0043] To effectively control uncertainty, the network adjusts the training process by calculating the uncertainty value for each query point.

[0044] Specifically, after the output of each stage, the uncertainty is calculated through a linear layer and the mean square error L is used MSE to oversee this process.

[0045] When the error of a certain pixel is large, the uncertainty of the related sample points is also high, so the penalty needs to be increased.

[0046] In order to avoid excessive uncertainty, the present invention introduces a regularization loss L0 to encourage the uncertainty of each query point to be as close to zero as possible.

[0047] The total uncertainty loss consists of two parts, namely, the uncertainty loss L SE and regularization loss L0, whose weighted summation gives the overall uncertainty loss Lunct , the specific formula is as follows: L unct =α1L SE +α2L0

[0048] Among them, L SE It is used to represent uncertainty loss, L0 is regularization loss, and α1 and α2 are weight coefficients. The specific formula is:

[0049] In the above formula, R represents a set of rays in each training batch, N is the number of sampling points on each ray, and δ i is the uncertainty value.

[0050] The task of the output module is to decode the latent features of the network output into color and volume density, which are then used to render the generated image.

[0051] When the uncertainty is low, the query sample can exit the network early and directly enter the output module. Otherwise, the query sample will continue to enter the next stage of the network module for processing.

[0052] The multi-input segmented network of the present invention is represented by the following formula: f1:(ω L (t, T; p′))→(y i , δ i ) f i :(y i-1 ,ω L (t, T; p′))→(y i , δ i ), i = 2, 3, 4 o i :(y i ,d′)→(c i , σ i ), i = 1, 2, 3, 4

[0053] Among them, f i Represents the network module and branch module of the i-th stage. When the uncertainty δ of the branch module prediction i When it is lower than the threshold, the query sample will be forwarded to the corresponding output module o i , directly output color and volume density. If the uncertainty is high, the sample will continue to enter the next stage of the network for processing.

[0054] By combining a multi-input segmentation network and an early termination mechanism, the model can effectively reduce computational complexity, reduce training time, and improve the accuracy and quality of rendering results.

[0055] Figure 3 The output module structure diagram of the present invention is given.

[0056] The stage rendering method of the present invention is used to improve rendering efficiency by screening sample points whose uncertainty is lower than a threshold in a segmented network so that the sample points exit the network, thereby reducing unnecessary calculations.

[0057] For points with higher uncertainty, continue to the next stage of the network for refinement.

[0058] In each stage, sample points with low uncertainty are terminated early, while points with higher uncertainty are sent to deeper stages for refinement.

[0059] This processing method reduces computational overhead and improves rendering efficiency by gradually refining the prediction results of sample points.

[0060] However, when the uncertainty of a sample point exceeds a threshold and enters a deep network, training instability may occur.

[0061] Specifically, the deep network may not have been trained enough at this coordinate point, so it outputs a nearly random value, resulting in an unstable increase in the loss value and affecting the stability of the training process.

[0062] In order to address the problem of training instability, the present invention proposes a solution: during training, each stage of the segmentation network outputs the color and volume density of all query sample points.

[0063] These outputs will be used to calculate the mean squared error loss at each stage and uncertainty loss

[0064] The loss functions of each stage will be summed with the same weight to obtain the overall loss function L, which is used to supervise the network. The overall loss function is:

[0065] Where D represents the number of stage networks, is the mean square error loss of the i-th stage, is the uncertainty loss of the i-th stage. The weight coefficients β1=1 and β2=0.1 are used to balance the two losses.

[0066] By summing the weighted losses, the present invention effectively avoids the instability caused by insufficient training of the deep-stage network, thereby ensuring the stability of the network during training and the quality of rendered images.

[0067] It should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A few-sample view synthesis method based on neural radiation field, characterized in that: The method comprises: By sampling a set of static images of a scene or object, relevant data including spatial position and viewing direction are obtained; Perform Fourier feature mapping on the data to map the spatial position and viewing direction into a high-dimensional feature vector; Use a multi-input segmentation network for implicit reconstruction to obtain the color and volume density of spatial points; The image is synthesized by stage rendering and early termination is performed according to the uncertainty threshold to reduce unnecessary calculations.

2. The method according to claim 1, characterized in that The Fourier feature map comprises: Perform position encoding on the spatial position and the viewing direction respectively to generate Fourier feature vectors; Concatenate the original input data with the Fourier eigenvector to generate a high-dimensional feature vector; The frequency mask α is introduced to dynamically adjust the visibility of different frequency components in the Fourier eigenvector, thereby controlling the introduction of high-frequency information.

3. The method according to claim 1, characterized in that The method dynamically controls the visibility of high-frequency components in Fourier eigenvectors through a frequency regularization strategy to avoid interference from unstable high-frequency signals in the early stage of network training.

4. The method according to claim 1, characterized in that: The multi-input segmented network includes multiple stage networks, each of which is composed of a network module, a branch module and an output module: The network module is used for feature processing, which reduces the computational complexity by reducing the number of model layers; The branch module is used to predict the uncertainty of the query sample. Samples with low uncertainty will be passed to the output module, and samples with high uncertainty will be forwarded to the next stage; The output module is responsible for decoding the latent features into color and volume density.

5. The method according to claim 4, characterized in that The branch module calculates uncertainty and determines whether the query sample exits the current stage according to the uncertainty value. When the uncertainty is lower than a preset threshold, the query sample exits early and enters the output module.

6. The method according to claim 1, characterized in that The stage rendering method exits early by screening sample points whose uncertainties are lower than a threshold, thereby reducing the amount of calculation, and the remaining points with higher uncertainties enter the subsequent stage for further refinement.

7. The method according to claim 1, characterized in that During the training process, the method uses mean square error loss and uncertainty loss to optimize the stage network, and obtains the overall loss function through weighted summation to ensure the stability and accuracy between networks at different stages.

8. The method according to claim 7, wherein the overall loss function comprises: Mean square error loss L MES , used to measure the error uncertainty loss L between the predicted image and the real image unct , which is used to penalize query points with high prediction uncertainty.

9. The method according to claim 1, characterized in that: The method combines Fourier features and random Fourier filtering to enhance the network's ability to restore local details, reduce noise artifacts, and improve the quality of the synthesized image.

10. The method according to claim 1, characterized in that The stage rendering method aggregates the rendering results of each stage, calculates the expected color of the pixel using the volume rendering formula, and finally generates a high-quality view synthesis image.

Images