Neural radiance field rendering method and device based on neural basis and tensor decomposition

Abstract

This invention discloses a neural radiation field rendering method and apparatus based on neural basis functions and tensor decomposition, belonging to the field of computer vision technology. The method includes: acquiring multi-view images and their camera parameters to construct a dataset; sampling the radiation field from tensor decomposition according to the camera parameters under a selected training viewpoint to obtain density feature tensors and texture feature tensors, which are then converted into density values ​​and neural basis functions, respectively; calculating the affine matrix and bias through a coefficient network based on the neural basis functions and performing affine transformations in conjunction with the observation direction to obtain coefficients; inputting the neural basis functions and coefficients into a neural hybrid network to obtain the color of a single sampling point; calculating pixel colors using volumetric rendering techniques and training the entire network; setting a new viewpoint and acquiring corresponding camera parameters; and rendering the image using the trained network parameters. This invention can efficiently reconstruct high-quality 3D scenes from multi-view image data, effectively reconstruct complex surface reflection effects, and achieve realistic image rendering from new viewpoints.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a neural radiation field rendering method and device based on neural basis and tensor decomposition. Background Art

[0002] 3D reconstruction has important application value in many fields such as digital twins, virtual reality, autonomous driving and robotics. Among many 3D reconstruction technologies, 3D reconstruction technology based on neural radiance fields (reference title: Nerf: Representing scenes as neural radiance fields for view synthesis (NeRF: Representing scenes as neural radiance fields for view synthesis)) has shown great potential in generating high-quality new perspective images and depth estimation with its unique advantages. Compared with traditional methods, NeRF uses a multilayer perceptron network (MLP) to represent 3D scenes as radiation fields in generating high-quality new perspective images and depth estimation. This representation method makes the color and density of each point in the scene closely related to the observation angle, so as to capture the propagation and accumulation process of light in the scene, thereby generating realistic new perspective images.

[0003] The core of NeRF lies in its differentiable volume rendering technology, which enables NeRF to continuously render the radiation field into images, thereby achieving accurate synthesis of new perspective images. During the training process, NeRF calculates the reconstruction loss of the training images and uses gradient backpropagation to optimize the MLP network representing the radiation field. This optimization process enables NeRF to continuously adjust its internal parameters to better fit the real scene and further improve the quality and realism of new perspective images. This technological breakthrough has opened up new paths for research in the field of computer vision and provided strong support for the development of technologies such as virtual reality and augmented reality.

[0004] However, although NeRF has made significant progress in generating new perspective image quality, it still faces certain challenges when dealing with scenes with complex surface reflection effects. The root of this problem is that NeRF mainly relies on volume rendering technology to synthesize images. Volume rendering technology often has difficulty in accurately capturing and simulating the interaction between light and the surface of objects. When dealing with scenes with complex surface reflection effects, NeRF may experience problems such as image quality degradation and loss of details. Therefore, NeRF still needs to be continuously improved and optimized to better meet the needs of practical applications.

[0005] In summary, how to further improve NeRF's ability to handle complex surface reflection effects and image rendering quality while maintaining its advantages has become an important issue that needs to be urgently addressed in the current field of computer vision. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a neural radiation field rendering method and device based on neural basis and tensor decomposition, which can efficiently reconstruct high-quality three-dimensional scenes from multi-view image data, effectively reconstruct complex surface reflection effects, and realize realistic image rendering from a new perspective.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0008] An embodiment of the present invention provides a neural radiation field rendering method based on neural basis and tensor decomposition, comprising the following steps:

[0009] Preprocessing stage: preprocess the collected multi-view images and obtain the camera parameters of each image, and construct the preprocessed images and their camera parameters into a data set;

[0010] Training phase: based on the camera parameters under the selected training perspective, the radiation field of tensor decomposition is sampled to obtain density feature tensor and texture feature tensor, the density feature tensor is converted into density value, and the texture feature tensor is converted into neural basis; based on the neural basis, the coefficient network is used to calculate the affine matrix and bias, and the affine transformation is performed using the affine matrix, bias and observation direction to obtain the coefficient of the neural basis; the neural basis and its coefficients are input into the neural hybrid network to obtain the color of a single sampling point, and the color of the sampling point is weighted along the light according to the density value using volume rendering technology to calculate the final pixel color; the network training in the training phase is performed based on the data set and the image reconstruction loss function;

[0011] Rendering stage: set a new perspective and obtain the corresponding camera parameters, load the trained network parameters, render according to the process of the training stage and obtain the final rendered image.

[0012] Preferably, the preprocessing of the collected multi-view images and obtaining the camera parameters of each image includes:

[0013] The collected multi-view images are preprocessed with a unified resolution, and the camera parameters of each image are obtained using the Structure from Motion (SFM) algorithm. The camera parameters include the camera pose and focal length.

[0014] Preferably, sampling the radiation field of tensor decomposition according to the camera parameters under the selected training viewing angle to obtain a density feature tensor and a texture feature tensor comprises:

[0015] The scene is modeled by the radiation field of tensor decomposition. The tensor decomposition formula is as follows:

[0016] ,

[0017] in, Indicates sampling point The feature tensor at , represents the Hadamard product, represents the total number of ranks, represents the index of the rank, , and Respectively represent sampling points exist , and The tensor of the axis, , and Respectively represent sampling points exist , and tensor in direction;

[0018] Select the training view and project it according to the camera parameters of the training view. The coordinates of the grid are extracted to obtain the feature tensor , the feature tensor Divide the feature channel dimension to obtain the density feature tensor and texture feature tensor .

[0019] Preferably, converting the density feature tensor into a density value comprises:

[0020] The ReLU activation function is used to convert the density feature tensor into actual density values.

[0021] Preferably, converting the texture feature tensor into a neural basis comprises:

[0022] The texture feature tensor is passed through a linear layer without bias and activation to generate a neural basis. The formula is as follows:

[0023] ,

[0024] in, Represents neural base, represents the weight matrix of the linear layer, Represents neural basis The number of channels, , and The tensor decomposition is , and The rank of the axis associated with the texture feature tensor.

[0025] Preferably, the affine matrix and the bias are calculated based on the neural basis through the coefficient network, and the coefficients of the neural basis are obtained by performing affine transformation using the affine matrix, the bias and the observation direction, including:

[0026] The neural base , sampling points after position coding The coordinate tensor of , Ambient Lighting Tensor And the camera observation direction tensor after position encoding Input coefficient network In the calculation, the affine matrix and bias , the formula is as follows:

[0027] ,

[0028] Neural base The coefficient of By observing the direction Perform affine transformation and the formula is as follows:

[0029] ,

[0030] in, represents the Hadamard product.

[0031] Preferably, the coefficient network is constructed using a multi-layer perceptron network.

[0032] Preferably, the neural basis and its coefficient are input into the neural hybrid network together to obtain the color of a single sampling point, and the color of the sampling point is weighted along the light according to the density value using the volume rendering technology to calculate the final pixel color, including:

[0033] The neural base and its coefficient Input Neural Hybrid Network , get the color of a single sampling point , the formula is as follows:

[0034] ,

[0035] Multiple sampling is performed along the light, and the pixel color of each sampling point is obtained using volume rendering technology. , the formula is as follows:

[0036] ,

[0037] ,

[0038] in, represents the total number of sampling points along the ray, represents the index of the sampling point along the ray, represents the cumulative transmittance of the first n-1 sampling points, Indicates The density value corresponding to the sampling point is Indicates The light sampling step corresponding to the sampling points is Indicates the value used to calculate the cumulative transmittance The index of the first -1 sample point.

[0039] Preferably, the neural hybrid network is constructed using a multi-layer perceptron network.

[0040] Preferably, the image reconstruction loss function includes a rendering loss function and a regularization term, and the formula is as follows:

[0041] ,

[0042] in, represents the image reconstruction loss function, represents the rendering loss function, Represents the actual pixel color, represents the rendered pixel color, represents the square of the L2 norm, Represents the regular term Hyperparameters of

[0043] The formula for the regularization term is as follows:

[0044] ,

[0045] in, represents the total number of parameters of the tensor decomposition, represents the squared difference between adjacent parameters, and denote the vector and matrix of tensor decomposition, respectively. and represents a hyperparameter for adjusting the smoothness.

[0046] To achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a neural radiation field rendering device based on neural basis and tensor decomposition, comprising: a preprocessing module, a training module and a rendering module;

[0047] The preprocessing module is used to preprocess the collected multi-view images and obtain the camera parameters of each image, and construct the preprocessed images and their camera parameters into a data set;

[0048] The training module is used to sample the radiation field of tensor decomposition according to the camera parameters under the selected training perspective to obtain a density feature tensor and a texture feature tensor, convert the density feature tensor into a density value, and convert the texture feature tensor into a neural basis; based on the neural basis, an affine matrix and a bias are calculated through a coefficient network, and the coefficient of the neural basis is obtained by performing an affine transformation using the affine matrix, the bias and the observation direction; the neural basis and its coefficient are input into a neural hybrid network together to obtain the color of a single sampling point, and the color of the sampling point is weighted along the light according to the density value using a volume rendering technique to calculate the final pixel color; network training of the training module is performed based on a data set and an image reconstruction loss function;

[0049] The rendering module is used to set a new viewing angle and obtain corresponding camera parameters, load the trained network parameters, render according to the process of the training module and obtain the final rendered image.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The present invention compactly stores high-dimensional density and texture feature tensors through tensor decomposition, simulates realistic surface reflection effects related to viewing angles through texture feature tensors, and renders surface colors related to viewing angles through volume rendering technology based on the obtained combination of neural bases and their coefficients. It can efficiently reconstruct high-quality three-dimensional scenes from multi-view image data, and effectively reconstruct complex surface reflection effects, thereby realizing realistic image rendering under new viewing angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 It is a flowchart of a neural radiation field rendering method based on neural basis and tensor decomposition provided by an embodiment of the present invention;

[0054] Figure 2 It is an overall framework diagram of a neural radiation field rendering method based on neural basis and tensor decomposition provided by an embodiment of the present invention;

[0055] Figure 3 is a comparison chart of experimental results of different methods provided by embodiments of the invention on a forward-facing data set;

[0056] Figure 4is a line graph of quantitative experimental results of different methods provided in the embodiments of the invention on the NeRF-synthetic data set;

[0057] Figure 5 It is a structural schematic diagram of a neural radiation field rendering device based on neural basis and tensor decomposition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0059] The inventive concept of the present invention is: variable surface materials, complex environmental lighting and geometric structures make reverse learning from multi-view images challenging. Therefore, in response to the problem of poor image rendering quality in the prior art when processing scenes with complex surface reflections, an embodiment of the present invention provides a neural radiation field rendering method and device based on neural basis and tensor decomposition, which solves this challenging reverse learning problem by using volume representation. The representation compactly stores high-dimensional density and texture feature tensors through tensor decomposition, simulates realistic surface reflection effects related to perspective through texture feature tensors, and further renders surface colors related to perspective through a combination of neural basis and coefficients, thereby effectively reconstructing complex surface reflection effects and further improving image rendering quality.

[0060] Figure 1 is a flowchart of a neural radiation field rendering method based on neural basis and tensor decomposition provided by an embodiment of the present invention, Figure 2 is an overall framework diagram of a neural radiation field rendering method based on neural basis and tensor decomposition provided by an embodiment of the present invention. Figure 1 and Figure 2 As shown, the embodiment provides a neural radiation field rendering method based on neural basis and tensor decomposition, comprising the following steps:

[0061] Step 1: Preprocessing stage: Preprocess the collected multi-view images and obtain the camera parameters of each image, and construct the preprocessed images and their camera parameters into a data set.

[0062] Step 1-1: Preprocess the multi-view images of a single scene captured by a digital camera to a uniform resolution.

[0063] Step 1-2: Use the motion structure recovery algorithm to obtain the camera internal and external parameters (including camera pose and focal length) corresponding to each image.

[0064] Step 1-3: Construct the preprocessed images and their camera parameters into a dataset, and divide the dataset into a training set and a test set in proportion.

[0065] Step 2: Training phase: Sample the radiation field of tensor decomposition according to the camera parameters under the selected training perspective to obtain density feature tensor and texture feature tensor, convert the density feature tensor into density value, and convert the texture feature tensor into neural basis; calculate the affine matrix and bias through the coefficient network based on the neural basis, and use the affine matrix, bias and observation direction to perform affine transformation to obtain the coefficient of the neural basis; input the neural basis and its coefficients into the neural hybrid network to obtain the color of a single sampling point, and use volume rendering technology to weight the color of the sampling point along the light according to the density value to calculate the final pixel color; perform network training in the training phase based on the data set and image reconstruction loss function.

[0066] Step 2-1: Model the scene through the radiation field of tensor decomposition. The embodiment implements a lightweight and efficient volume (radiation field) representation method through tensor decomposition, which can not only store volume data in a highly compact way, but also directly propagate gradients to spatial points in the volume. The feature tensor at , each spatial point Ability to efficiently query feature tensors in the volume by grid sampling The tensor decomposition formula is as follows:

[0067] ,

[0068] in, Indicates sampling point The feature tensor at , represents the Hadamard product, represents the total number of ranks, represents the index of the rank, , and Respectively represent sampling points exist , and The tensor of the axis, , and Respectively represent sampling points exist , and A tensor in the direction.

[0069] Step 2-2: Initialize the ambient lighting into a learnable ambient lighting tensor , in order to optimize the environment lighting during training.

[0070] Step 2-3: Select the training view and project it according to the camera parameters of the training view. Coordinates , through grid sampling in the volume The feature tensor is extracted from , the feature tensor Divide the feature channel dimension to obtain the density feature tensor and texture feature tensor .

[0071] Step 2-4: Use the ReLU activation function to transform the density feature tensor Convert to actual density value , to ensure that the output density values ​​are non-negative and have the appropriate activation range.

[0072] Step 2-5: Consider the geometry, material or lighting representation in the scene as some implicit features, and the neural base It uses neural networks to learn the sparse dictionary representations corresponding to these features. The texture feature tensor It is generated through a linear layer without bias and activation, and the formula is as follows:

[0073] ,

[0074] in, represents the weight matrix of the linear layer, Represents neural basis The number of channels, , and The tensor decomposition is , and The rank of the axis associated with the texture feature tensor.

[0075] Step 2-6: Neural base , sampling points after position coding The coordinate tensor of , Ambient Lighting Tensor And the camera observation direction tensor after position encoding Input coefficient network In the calculation, the affine matrix and bias , the formula is as follows:

[0076] .

[0077] Step 2-7: Neural Basis The coefficient of By observing the direction Perform affine transformation and the formula is as follows:

[0078] ,

[0079] in, represents the Hadamard product.

[0080] Step 2-8: Neural Hybrid Network It is implemented as a multi-layer perceptron (MLP) to approximate the mathematical expression of the coefficient-weighted basis to reconstruct the data. MLP has several advantages over static weighted sums, including greater flexibility, nonlinearity, and the ability to capture coefficients. With neural base Specifically, through the neural hybrid network The neural base and its coefficient As input to implement blending, generating perspective and lighting-dependent colors As output, the formula is as follows:

[0081] .

[0082] Step 2-9: Multiple sampling along the ray, Sampling points ,in, represents the origin of the light, Represents the light sampling parameters, represents the light sampling step length, Represents the total number of sampling points along the ray. Volume rendering technology is used to obtain the pixel color of each sampling point , the formula is as follows:

[0083] ,

[0084] ,

[0085] in, represents the total number of sampling points along the ray, represents the index of the sampling point along the ray, represents the cumulative transmittance, Indicates The density value corresponding to the sampling point is Indicates The light sampling step corresponding to the sampling points is Indicates the value used to calculate the cumulative transmittance The indices of the first n-1 sample points.

[0086] Step 2-10: Density Feature Tensor , texture feature tensor and the ambient lighting tensor The learnable parameters of other networks are optimized by minimizing the image reconstruction loss function, back-propagating the loss calculated by the loss function according to the gradient, and optimizing the network parameters in the training phase. The image reconstruction loss function includes the rendering loss function and the regularization term, and the formula is as follows:

[0087] ,

[0088] in, represents the image reconstruction loss function, represents the rendering loss function, Represents the actual pixel color, represents the rendered pixel color, represents the square of the L2 norm, Represents the regular term Hyperparameters of .

[0089] In order to realize the texture feature tensor and the density feature tensor The spatial smooth distribution of , using the total variation (TV) loss to regularize the learnable parameters in the tensor decomposition, the formula of the regularization term is as follows:

[0090] ,

[0091] in, represents the total number of parameters of the tensor decomposition, represents the squared difference between adjacent parameters, and denote the vector and matrix of tensor decomposition, respectively. and represents a hyperparameter for adjusting the smoothness.

[0092] Step 3: Rendering stage: Set a new perspective and obtain the corresponding camera parameters, load the trained network parameters, render according to the process of the training stage and obtain the final rendered image.

[0093] Step 3-1: Select a new viewing angle and obtain the corresponding camera parameters.

[0094] Step 3-2: Load the network parameters obtained during the training phase and use these parameters for rendering.

[0095] Step 3-3: Render the image for the new perspective according to the process of steps 2-3 to 2-9 in the training phase, which does not involve calculating the loss function and gradient back propagation. The final rendered image is generated by calculating the color of each pixel.

[0096] In one embodiment, the method of the present invention is implemented using PyTorch 1.12, and the rank of the density feature tensor is The rank associated with the texture feature tensor They are [16, 16, 16] and [48, 48, 48] respectively. The levels are all set to 2. Each ambient lighting tensor The number of channels is set to 32, and the neural base Number of channels Set to 27. Coefficient Network Implemented as an 8-layer MLP network with 256 hidden nodes and Leaky-ReLU activation function. Neural Hybrid Network It is implemented as a 3-layer MLP with 128 hidden nodes and ReLU activation function. In the example, the number of training iterations is set to 100,000, the batch size of training is 5,000 rays, and the initial learning rate is 0.02. On an NVIDIA A100 GPU, it takes about 2 hours to train a scene in the NeRF-synthetic dataset.

[0097] Based on the experimental conditions provided in the above embodiment, the present method is compared with the following methods: tensor radiation field (TensorF, reference title is Tensorf: Tensorial radiance fields (Tensorf: Tensor radiation field)), direct voxel grid optimization (DVGO, reference title is Improved direct voxel grid optimization for radiance fields reconstruction (Improved direct voxel grid optimization for radiance field reconstruction)), three-dimensional Gaussian splashing algorithm (3DGS, reference title is 3D Gaussian Splatting for Real-Time Radiance Field Rendering (Improved three-dimensional Gaussian splashing algorithm for real-time radiation field rendering)), K-planes (K-Planes, reference title is K-planes: Explicit radiance fields inspace, time, and appearance (K-planes: Explicit radiance fields in space, time and appearance)), instant neural graphics primitives (Instant-NGP, reference title is Instant neural graphics primitives with amultiresolution hash encoding (Real-time neural graphics primitives with multi-resolution hash encoding), PlenOxels (Plenoxels: Radiance fields without neuralnetworks), Neural Radiance Fields (NeRF, Nerf: Representing scenes as neural radiance fields for view synthesis), Local Light Field Fusion (LLFF, Local lightfield fusion: Practical view synthesis with prescriptive sampling guidelines), Neural Basis Extension (Nex, Nex: Real-time view synthesis with neural basis expansion).

[0098] The following evaluation indicators are used to evaluate the image rendering quality: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learning-perceptual image patch similarity (LPIPS). Table 1 shows the quantitative new perspective image generation results of the method provided by the embodiment of the present invention and some comparative methods on the NeRF-Synthetic dataset. The experimental results show that the method shows significant performance improvement in the benchmark dataset of new perspective image generation. For example, using PSNR as the evaluation indicator, the performance of the method on the NeRF-Synthetic dataset exceeds the original NeRF, achieving a significant improvement of 3 dB.

[0099] Table 1 Comparison of image rendering quality of different methods on the NeRF-Synthetic dataset

[0100]

[0101] In addition, Table 2 also shows the quantitative new perspective image generation results of this method and some comparison methods on the Forward-facing dataset. Figure 3 The visualization images and true values ​​rendered by different methods are displayed. It can be seen that this method has better detail reconstruction and rendering effects than other methods.

[0102] Table 2 Comparison of image rendering quality of different methods on the Forward-facing dataset

[0103]

[0104] In another embodiment, the method adjusts the number of training views from the default 100 to 200 and 300. The experimental results are detailed in Table 3, and FIG4 shows a line graph based on the data in Table 3. These graphs intuitively show the performance trends of various methods including the present method under different numbers of training views. It can be observed from the graphs that the present method always outperforms other methods in performance. In particular, as the number of training views increases, the present method shows a continuous lead in performance indicators, and the improvement is more significant, and this trend is particularly prominent in the PSNR indicator.

[0105] Table 3 Comparison of image rendering quality of different methods in the material scene of NeRF-synthetic dataset with different numbers of training viewpoints

[0106]

[0107] In summary, the neural radiation field rendering method based on neural basis and tensor decomposition provided by the embodiment of the present invention can generate high-quality images under new perspectives by efficiently processing and optimizing multi-perspective images. Experimental results show that compared with existing methods, the present invention shows significant performance improvements in multiple evaluation indicators, especially in peak signal-to-noise ratio, structural similarity, and learning-perceived image block similarity. In addition, the present invention also explores the impact of the number of training perspectives on the rendering quality. Experimental results show that as the number of training perspectives increases, the method of the present invention shows a continuous lead in performance indicators, and the improvement is more significant. This research result further verifies the effectiveness and robustness of the present method.

[0108] Based on the same inventive concept, Figure 5 As shown, an embodiment of the present invention further provides a neural radiation field rendering device 500 based on neural basis and tensor decomposition, including: a preprocessing module 510, a training module 520 and a rendering module 530.

[0109] The preprocessing module 510 is used to preprocess the collected multi-view images and obtain the camera parameters of each image, and construct the preprocessed images and their camera parameters into a data set.

[0110] The training module 520 is used to sample the radiation field of the tensor decomposition according to the camera parameters under the selected training perspective to obtain a density feature tensor and a texture feature tensor, convert the density feature tensor into a density value, and convert the texture feature tensor into a neural basis; based on the neural basis, an affine matrix and a bias are calculated through a coefficient network, and the coefficients of the neural basis are obtained by affine transformation using the affine matrix, the bias and the observation direction; the neural basis and its coefficients are input into a neural hybrid network together to obtain the color of a single sampling point, and the color of the sampling point is weighted along the light ray according to the density value using volume rendering technology to calculate the final pixel color; network training of the training module is performed based on the data set and the image reconstruction loss function.

[0111] The rendering module 530 is used to set a new viewing angle and obtain corresponding camera parameters, load the trained network parameters, render according to the process of the training module and obtain the final rendered image.

[0112] It should be noted that the neural radiation field rendering device based on neural basis and tensor decomposition provided in the above embodiment belongs to the same inventive concept as the neural radiation field rendering method based on neural basis and tensor decomposition. The specific implementation process is detailed in the embodiment of the neural radiation field rendering method based on neural basis and tensor decomposition, which will not be repeated here.

[0113] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A neural radiation field rendering method based on neural basis and tensor decomposition, characterized in that: The following steps are involved: Preprocessing stage: preprocess the collected multi-view images and obtain the camera parameters of each image, and construct the preprocessed images and their camera parameters into a data set; Training phase: (1) According to the camera parameters under the selected training perspective, the radiation field of the tensor decomposition is sampled to obtain the density feature tensor and the texture feature tensor, including: The scene is modeled by the radiation field of tensor decomposition. The tensor decomposition formula is as follows: ， in, Indicates sampling point The feature tensor at , represents the Hadamard product, represents the total number of ranks, represents the index of the rank, , and Respectively represent sampling points exist , and The tensor of the axis, , and Respectively represent sampling points exist , and tensor in direction; (2) Initialize the ambient lighting into a learnable ambient lighting tensor , in order to optimize the ambient lighting during training; (3) Select the training view and project it according to the camera parameters of the training view. The coordinates of the grid are extracted to obtain the feature tensor , the feature tensor Divide the feature channel dimension to obtain the density feature tensor and texture feature tensor ; (4) Use the ReLU activation function to convert the density feature tensor into actual density values; (5) The texture feature tensor is passed through a linear layer without bias and activation to generate a neural basis. The formula is as follows: ， in, Represents neural base, represents the weight matrix of the linear layer, Represents neural basis The number of channels, , and The tensor decomposition is , and The rank of the axis associated with the texture feature tensor; (6) Based on the neural basis, the affine matrix and bias are calculated through the coefficient network, and the coefficients of the neural basis are obtained by affine transformation using the affine matrix, bias and observation direction, including: The neural base , sampling points after position coding The coordinate tensor of , Ambient Lighting Tensor And the camera observation direction tensor after position encoding Input coefficient network In the above example, the coefficient network is constructed using a multilayer perceptron network, and the affine matrix is ​​calculated. and bias , the formula is as follows: ， Neural base The coefficient of By observing the direction Perform affine transformation and the formula is as follows: ， in, represents the Hadamard product; (7) Input the neural basis and its coefficients into the neural hybrid network to obtain the color of a single sampling point. Use volume rendering technology to weight the color of the sampling point along the light according to the density value to calculate the final pixel color. (8) Network training in the training phase based on the data set and image reconstruction loss function; Rendering stage: set a new perspective and obtain the corresponding camera parameters, load the trained network parameters, render according to the process of the training stage and obtain the final rendered image.

2. The neural radiation field rendering method based on neural basis and tensor decomposition according to claim 1, characterized in that: The neural basis and its coefficients are input into the neural hybrid network to obtain the color of a single sampling point, and the color of the sampling point is weighted along the light according to the density value using the volume rendering technology to calculate the final pixel color, including: The neural base and its coefficient Input Neural Hybrid Network , get the color of a single sampling point , the formula is as follows: ， Multiple sampling is performed along the light, and the pixel color of each sampling point is obtained using volume rendering technology. , the formula is as follows: ， ， in, represents the total number of sampling points along the ray, represents the index of the sampling point along the ray, represents the cumulative transmittance of the first n-1 sampling points, Indicates The density value corresponding to the sampling point is Indicates The light sampling step corresponding to the sampling points is Indicates the value used to calculate the cumulative transmittance The index of the first -1 sample point.

3. The neural radiation field rendering method based on neural basis and tensor decomposition according to claim 1 or 2, characterized in that: The neural hybrid network is constructed using a multilayer perceptron network.

4. The neural radiation field rendering method based on neural basis and tensor decomposition according to claim 1, characterized in that: The image reconstruction loss function includes the rendering loss function and the regularization term, and the formula is as follows: ， in, represents the image reconstruction loss function, represents the rendering loss function, Represents the actual pixel color, represents the rendered pixel color, represents the square of the L2 norm, Represents the regular term Hyperparameters of The formula for the regularization term is as follows: ， in, represents the total number of parameters of the tensor decomposition, represents the squared difference between adjacent parameters, and denote the vector and matrix of tensor decomposition, respectively. and represents a hyperparameter for adjusting the smoothness.

5. A neural radiation field rendering device based on neural basis and tensor decomposition, characterized in that: include: Preprocessing module, training module and rendering module; The preprocessing module is used to preprocess the collected multi-view images and obtain the camera parameters of each image, and construct the preprocessed images and their camera parameters into a data set; The training module is used to: (1) According to the camera parameters under the selected training perspective, the radiation field of the tensor decomposition is sampled to obtain the density feature tensor and the texture feature tensor, including: The scene is modeled by the radiation field of tensor decomposition. The tensor decomposition formula is as follows: ， in, Indicates sampling point The feature tensor at , represents the Hadamard product, represents the total number of ranks, represents the index of the rank, , and Respectively represent sampling points exist , and The tensor of the axis, , and Respectively represent sampling points exist , and tensor in direction; (2) Initialize the ambient lighting into a learnable ambient lighting tensor , in order to optimize the ambient lighting during training; (3) Select the training view and project it according to the camera parameters of the training view. The coordinates of the grid are extracted to obtain the feature tensor , the feature tensor Divide the feature channel dimension to obtain the density feature tensor and texture feature tensor ; (4) Use the ReLU activation function to convert the density feature tensor into actual density values; (5) The texture feature tensor is passed through a linear layer without bias and activation to generate a neural basis. The formula is as follows: ， in, Represents neural base, represents the weight matrix of the linear layer, Represents neural basis The number of channels, , and The tensor decomposition is , and The rank of the axis associated with the texture feature tensor; (6) Based on the neural basis, the affine matrix and bias are calculated through the coefficient network, and the coefficients of the neural basis are obtained by affine transformation using the affine matrix, bias and observation direction, including: The neural base , sampling points after position coding The coordinate tensor of , Ambient Lighting Tensor And the camera observation direction tensor after position encoding Input coefficient network In the above example, the coefficient network is constructed using a multilayer perceptron network, and the affine matrix is ​​calculated. and bias , the formula is as follows: ， Neural base The coefficient of By observing the direction Perform affine transformation and the formula is as follows: ， in, represents the Hadamard product; (7) Input the neural basis and its coefficients into the neural hybrid network to obtain the color of a single sampling point. Use volume rendering technology to weight the color of the sampling point along the light according to the density value to calculate the final pixel color. (8) Network training in the training phase based on the data set and image reconstruction loss function; The rendering module is used to set a new viewing angle and obtain corresponding camera parameters, load the trained network parameters, render according to the process of the training module and obtain the final rendered image.

Images