A neural radiance field relighting method based on progressive mesh refinement

Abstract

The application provides a neural radiance field relighting method based on progressive grid refinement, relates to the technical field of computer graphics and novel view synthesis, and adopts a grid structure with gradually progressive resolution, and combines a frequency coding technology to perform feature extraction on each layer grid through a multilayer perceptron, and gradually refines the three-dimensional structure and illumination information of a scene layer by layer. Specifically, low-resolution grids are combined with low-frequency coding, and high-resolution grids are combined with high-frequency coding, the feature expression capability under different scales is effectively enhanced through residual connection, the relighting effect of the scene is optimized, and the details and definition of a rendered image are improved. The training process of the method is efficient, fast training and reasoning can be realized on a consumer-level graphics card, the method has good scalability, is widely applicable to various environment perception tasks such as virtual reality, augmented reality and automatic driving, and has wide application prospects.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and novel perspective synthesis, and in particular to a neural radiation field relighting method based on progressive mesh refinement. Background Technology

[0002] In recent years, with the rapid development of computer vision and graphics technologies, the fields of 3D reconstruction and image rendering have witnessed significant innovation and optimization. Driven by emerging application demands, especially in virtual reality, augmented reality, and film and television production, the need for high-quality 3D scene reconstruction and detailed rendering is increasing. The emergence of neural radiation field technology marks a breakthrough in deep learning-based 3D rendering methods. Neural radiation field technology utilizes neural networks to learn the 3D structure and lighting information of a scene from 2D image data. Taking 2D images from multiple perspectives as input, it models each point in the scene using a multilayer perceptron (MLP). Unlike traditional 3D reconstruction methods, neural radiation field technology parameterizes the radiation and density fields in 3D space and, combined with differentiable volume rendering techniques, can generate high-fidelity images with photorealistic quality. This technology not only improves the accuracy of 3D scene reconstruction but also demonstrates significant advantages in tasks such as image generation and new perspective synthesis, showing broad application prospects in virtual reality, autonomous driving, and interior and exterior design.

[0003] Re-enhanced neural radiation fields, as an innovative derivative technology, aim to simulate the visual effects of scenes under different lighting conditions. Based on the combination of neural radiation field technology and lighting models for new perspective image rendering, re-enhanced neural radiation fields can quickly generate high-quality images of scenes under new lighting conditions by learning the interaction between the distribution of light sources and the 3D scene. Although re-enhanced neural radiation fields have made significant progress in lighting reconstruction, due to the complexity of lighting and the diversity of scene geometry, they often exhibit instability when handling high-frequency details and complex lighting interactions, potentially resulting in uneven lighting distribution or loss of detail. Summary of the Invention

[0004] To address the problems existing in current neural radiation field techniques for relighting tasks, this invention provides a neural radiation field relighting method based on progressive mesh refinement.

[0005] A neural radiation field relighting method based on progressive mesh refinement includes the following steps:

[0006] Step 1: Construct the dataset and divide it into training and test sets;

[0007] Step 1-1: Collect images taken by the camera from different perspectives to construct a dataset, and obtain the images and corresponding intrinsic parameters in the dataset. The intrinsic parameters include the camera's viewpoint direction v, the camera's position o when the image was taken, and the light source position P. l ;

[0008] Steps 1-2: Divide the dataset into training and test sets according to a set ratio;

[0009] Step 2: Emit a sampling ray from the camera position o in the dataset and perform discrete point sampling;

[0010] Step 2-1: For each image in the dataset, generate a sampling ray p(t) based on the camera's viewpoint direction: p(t) = o + tv, where o is the origin of the sampling ray. That is, three-dimensional data, where v is the direction of the sampling ray. That is, two-dimensional data, the sampling ray starts from the camera position, t represents the sampling distance, passes through the pixels on the two-dimensional image, and extends to the three-dimensional scene under the corresponding view of the image;

[0011] Step 2-2: Perform random uniform sampling on the sampling ray, using the following sampling formula:

[0012] Among them, t i t represents the sampling distance of the i-th sampling point. n t represents the near-end sampling point on the ray. f This represents the far-end sampling point on the ray, and N represents the number of sampling points;

[0013] Step 3: Construct a progressive multi-resolution feature grid and perform feature extraction;

[0014] Step 3-1: Multi-resolution representation of the mesh: Construct meshes of multiple resolutions in a progressive manner, with each mesh layer corresponding to spatial detail information at different scales; specifically represented as G. r ={G1,G2…G n},in G r N represents the grid of the r-th layer. r This is the resolution of the mesh layer, and n is the number of mesh layers;

[0015] Step 3-2: Feature Extraction Process: In the progressive mesh, features are extracted layer by layer to refine the mesh. A Multilayer Perceptron (MLP) is used to extract 3D spatial features from meshes of different resolutions. For each mesh layer G... r The MLP network takes the sampling point position x and sampling ray direction v obtained in step 2 as input to obtain the feature representation of the input sampling point learned by the MLP, namely the SDF value.

[0016] Step 3-3: Progressive training: Adopt a progressive training strategy. First, train on a low-resolution grid, and gradually introduce higher-resolution grid learning using a residual network for training and optimization.

[0017] Step 4: Construct position encodings of corresponding frequencies according to the progressive feature grid.

[0018] Step 4-1: Utilize the idea of progressive multi-resolution grid refinement and combine it with position encoding technology to generate encodings of different frequencies for each sampling point of the neural radiance field. Specifically, grids of different resolutions correspond to Fourier position encodings of different frequencies. For each grid point, use its spatial coordinates (X, Y, Z) to perform position frequency encoding using Fourier transform. Specifically, the position frequency encoding function is expressed as:

[0019] β(x): {sin(2 0 πx), cos(2 0 πx), sin(2 1 πx), cos(2 1 πx)... sin(2 L πx), cos(2 L πx)},

[0020] where x is the spatial coordinate of the grid point, X is the horizontal coordinate, representing the distance of the point along the horizontal direction relative to a certain reference plane (usually the horizontal plane). Y is the depth coordinate, representing the distance of the point along the front-back direction relative to the reference plane. Z is the vertical coordinate, representing the distance of the point along the up-down direction relative to the reference plane. L is the maximum frequency level of the encoding, corresponding to the resolution of the grid, and 2 L represents the calibration index for the high-frequency part;

[0021] Step 4-2: Progressive refinement encoding: Use Fourier encoding with a set minimum frequency to capture large-scale scene features; while on high-resolution grids, the frequency gradually increases. Assume the grid has n levels. For the l-th level grid, the maximum frequency of its frequency encoding is 2 l-1 , l < n, that is, as the resolution increases, the frequency also increases exponentially, expressed as follows:

[0022] β<​​​​​​​​​​​​​

[0023] Step 5: Connect multiple multilayer perceptron (MLP) networks using residual networks to achieve implicit representation of the scene;

[0024] The implicit expression is defined as a function mapping:

[0025] F: x,v→σ,c, where x is the spatial coordinate of the sampling point and v is the direction of the sampling ray. The network structure consists of multiple multilayer perceptrons (MLPs), specifically including F1, F2, ... F1. n The input to each MLP is the feature vector output by the previous MLP and the position encoding set of the corresponding frequency in this layer;

[0026] Step 5-1: The input to the first-layer MLP network F1 is the concatenation of the unencoded spatial coordinates x of the sampling points and the frequency-encoded data β1(x). The output vector is the feature vector O1 learned in this layer, expressed as:

[0027] F1:x,β1(x)→O1,l=1

[0028] Step 5-2: The MLP network of the second layer and each subsequent layer takes the output O from the previous layer and the position-frequency code β(x) of the current layer as input; specifically, the MLP network of the l-th layer takes the output O of the l-1 layer MLP as input. l-1 and the position frequency coding set β of this level l (x) is taken as input, and the intermediate feature vector O learned at this level is output. l The formula is expressed as follows:

[0029] F l :O l-1 ,β l (x)→O l ,l>1

[0030] Step 5-3: Connect the various network layers using a residual network to form the overall network architecture. The frequency encoding at each location is processed by the corresponding network layer to generate feature outputs. The residual network structure is used to connect the frequency outputs of each layer, as shown below:

[0031] O l =O l-1 +w l β l (x)+k, where w l These are the weighting coefficients, where k is a constant;

[0032] Step 5-4: Perform spectral encoding on the sampling ray direction v:

[0033]

[0034] Step 5-5: The feature vector output by the last layer of the network is O n The output is passed through a linear layer to obtain the final learned SDF value sdf. Then, the sampled ray direction information γ(v) obtained after frequency encoding is combined with the feature vector O. n The concatenation is performed and fed into an additional fully connected ReLU layer. Finally, the color value c is predicted by the sigmoid activation function.

[0035] Step 6: Sampling point visibility estimation, reconstructing lighting and shadows in the relighting task; perform visibility estimation for each sampling point in the scene to determine whether the point will be directly illuminated by a light source under given lighting conditions;

[0036] Step 6-1: Calculate sampling point p i With respect to the position P of the light source l The straight-line distance Z in three-dimensional space frag The formula is as follows:

[0037] Z frag =‖xP l ||

[0038] Where x is the spatial coordinate of the sampling point, P l These are the spatial coordinates of the light source point.

[0039] Step 6-2: Switch the camera position o to the light source point position P l Repeat steps 2 to 3 to learn the SDF values ​​of the sampling points on the ray, and denote the sampling points with an SDF value of 0 as q; connect the sampling points p. i With respect to the position P of the light source l The point on the line connecting two points is denoted as q if it has an SDF value of 0. i Calculate q i Point relative to the position of the light source P l The straight-line distance Z in three-dimensional space shadow The formula is as follows:

[0040] Z shadow =‖x q -P l ||

[0041] Where, x q For sampling point q i Spatial coordinates, P l These are the spatial coordinates of the light source point;

[0042] Step 6-3: Estimate sampling point p i Visibility, through Z frag and Z shadow To determine visibility; when Zfrag Greater than Z shadow If there is an obstruction on the line connecting the sampling point and the light source, the sampling point is located in the shadow area and its visibility is 0; otherwise, the visibility is 1. The formula is as follows:

[0043]

[0044] Among them, V p The visibility of the sampling ray at that sampling point;

[0045] Step 7: Render the image using the volume rendering formula;

[0046] Step 7-1: Calculate the volume density σ corresponding to each sampling point on the sampling ray. i :

[0047] Ω(x)=se -sx / (1+e -sx ) 2 ,σ i =Ω(F(p) i ))

[0048] Where Ω(x) represents the density function, s is the hyperparameter, F is the network function for learning the SDF value, and p i Indicates the sampling point;

[0049] Step 7-2: Calculate the cumulative value of the volume density corresponding to all sampling points on the sampling ray, i.e., the opacity T(t):

[0050]

[0051] Where t is the sampling distance and σ is the volume density of the sampling points.

[0052] Step 7-3: Calculate the density weight ω(t) corresponding to each sampling point on the sampling ray:

[0053] ω(t)=T(t)σ(t)

[0054] Step 7-4: Use the visibility V of the sampled ray p The density weights ω(t) of all sampling points on the sampling ray and the color values ​​c corresponding to the sampling points are calculated by summing to obtain the color value C(p) of the pixel:

[0055] Where N is the number of sampling points, ω i c represents the density weight value of the sampling points. i This represents the color learned by the sampling point.

[0056] Step 8: Calculate the overall loss;

[0057] Step 8-1: Calculate rendering loss;

[0058] The rendering loss is expressed as follows:

[0059]

[0060] Where C(p) represents the rendered RGB value, C gt (p) represents the actual RGB value of the pixel, R(p) is the set of all rays, and p is the sampled ray.

[0061] Step 8-2: Calculate the SDF regularization loss;

[0062] The SDF regularization loss is expressed as follows:

[0063]

[0064] Where x is the sampling point on the sampling ray p.

[0065] Step 8-3: Calculate the total loss;

[0066] The overall loss is a weighted sum of the rendering loss and the SDF regularization loss, expressed as follows:

[0067] l = l color +λl eikonal , where λ is a hyperparameter.

[0068] Step 9: Iterative training. Iteratively train the MLP network, repeating steps 2 through 8, using different training samples each time for optimization, until the neural network's performance reaches the predetermined convergence criterion. Finally, save the trained model.

[0069] Step 10: Using volume rendering formulas Generate a new perspective image.

[0070] The beneficial effects of adopting the above technical solution are as follows:

[0071] This invention provides a neural radiation field relighting method based on progressive mesh refinement. This method solves the problems of blurred details and high-frequency perceptual ambiguity in existing neural radiation field-based relighting methods. By employing progressive mesh refinement and frequency coding optimization techniques, it can generate smoother and clearer rendered images while preserving details, thus significantly improving the quality of scene reconstruction. The proposed network architecture uses an end-to-end training model, independent of various levels of supervision, exhibiting good performance and simple implementation. Its modular design facilitates later upgrades and optimizations and possesses strong deployability. This method has low hardware requirements and can be directly applied to consumer-grade graphics cards, making it widely applicable to tasks in augmented reality, autonomous driving, game design, interior and exterior design, and other fields. Specifically, it has the following beneficial effects:

[0072] 1. This invention effectively enhances the reconstruction capability of neural radiation fields for high-frequency details by combining progressive mesh refinement and frequency coding. Through dynamic adjustment of different resolution meshes and corresponding frequencies, fine incremental details are provided to the neural radiation field, significantly improving image quality in relighting tasks.

[0073] 2. This invention introduces a residual network structure, combined with gradually increasing frequency coding, making the network more flexible and efficient in processing high-frequency information. Simultaneously, this design optimizes the network training process, reduces training time, and effectively overcomes the problems caused by spectral bias.

[0074] 3. This invention effectively solves the problems of blurred detail reconstruction and high-frequency perception ambiguity in existing neural radiation field methods under varying illumination conditions, and significantly improves image clarity and detail reconstruction accuracy.

[0075] 4. This invention provides an end-to-end training framework that is simple, efficient, and requires no complex hierarchical supervision. This method has good scalability and optimization potential, can be directly deployed on consumer-grade graphics cards, and is widely applicable in fields such as augmented reality, autonomous driving, and interior and exterior design, demonstrating significant practical application value. Attached Figure Description

[0076] Figure 1 This is an overall flowchart of the neural radiation field relighting method according to an embodiment of the present invention;

[0077] Figure 2 This is a network structure diagram of the neural radiation field in an embodiment of the present invention. Detailed Implementation

[0078] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0079] A neural radiation field relighting method based on progressive mesh refinement, such as... Figure 1 As shown, it includes the following steps:

[0080] Step 1: Input the dataset;

[0081] Step 1-1: Collect images taken by the camera from different perspectives to construct a dataset, and obtain the images and corresponding intrinsic parameters in the dataset. The intrinsic parameters include the camera's viewpoint direction v, the camera's position o when the image was taken, and the light source position P. l ;

[0082] Steps 1-2: Divide the dataset into training and testing sets. The dataset contains data from 6 scenes, each with 600 images at a resolution of 512×512 pixels. Specifically, 500 images are used for training and 100 images are used for testing. The training and testing sets maintain the same distribution characteristics to ensure the model's generalization ability.

[0083] Step 2: Emit a sampling ray from camera position o in each image of the dataset, and generate a ray based on the viewing direction of the sampling ray. The ray is emitted from the camera position, passes through the pixels of the 2D image, and extends along the 3D scene where each pixel is located. Each sampling ray is uniformly sampled, and the feature value of each sampling point is calculated.

[0084] Step 2-1: For each image in the dataset, generate a sampling ray p(t) based on the camera's viewpoint direction: p(t) = o + tv, where o is the origin of the sampling ray. That is, three-dimensional data, where v is the direction of the sampling ray. That is, two-dimensional data, the sampling ray starts from the camera position, t represents the sampling distance, passes through the pixels on the two-dimensional image, and extends to the three-dimensional scene under the corresponding view of the image;

[0085] Step 2-2: Perform random uniform sampling on the sampling ray, using the following sampling formula:

[0086] Among them, t i t represents the sampling distance of the i-th sampling point. n t represents the near-end sampling point on the ray. f This represents the far-end sampling point on the ray, and N represents the number of sampling points;

[0087] Step 3: Construct a progressive multi-resolution feature mesh for feature extraction. Initial training is performed on a low-resolution mesh, then higher-resolution meshes are gradually introduced to refine the geometric features, thereby improving the model's performance. For example... Figure 2 As shown.

[0088] Step 3-1: Multi-resolution representation of the mesh: Multiple resolution meshes are constructed in a progressive manner, with each layer corresponding to spatial detail information at different scales. Low-resolution meshes primarily capture macroscopic structures, while high-resolution meshes refine the details to more microscopic geometric features. Specifically, this is represented as G. r ={G1,G2…G n},in G r N represents the grid of the r-th layer. r This represents the resolution of the mesh layer, and n is the number of mesh layers. Each mesh layer refines the representation of the object's surface and details; higher-level meshes can capture more detailed information.

[0089] Step 3-2: Feature Extraction Process: In the progressive grid, features are extracted layer by layer to refine the structure. Each layer of the neural network extracts features based on the current resolution of the grid. A multilayer perceptron (MLP) is used to extract 3D spatial features from grids of different resolutions; for each grid layer G... r The MLP network takes the sampling point position x and sampling ray direction v obtained in step 2 as input to obtain the feature representation of the input sampling point learned by the MLP, namely the SDF value.

[0090] Step 3-3: Progressive Training: A progressive training strategy is adopted, starting with training on a low-resolution mesh and gradually introducing higher-resolution meshes using the residual network for more refined training and optimization. This approach helps the network learn the coarse structure of the scene in the early stages, and then gradually optimizes the details in subsequent training, thereby improving training efficiency and convergence speed.

[0091] Step 4: Construct position codes for the corresponding frequencies based on the progressive feature grid;

[0092] Step 4-1: Utilizing the idea of ​​progressive multi-resolution grid refinement combined with positional encoding techniques, generate codes of different frequencies for each sampling point of the neural radiation field. Specifically, different resolution grids correspond to different frequency Fourier positional codes, which will be used to enhance the feature representation capability of the neural network at different scales. For each grid point, positional frequency encoding is performed using Fourier transform with its spatial coordinates (X, Y, Z). Specifically, the positional frequency encoding function is expressed as:

[0093] β(x): {sin(2 0 πx),cos(2 0 πx),sin(2 1 πx),cos(2 1 πx)...sin(2 L πx),cos(2L πx)}

[0094] Where x is the spatial coordinate of the grid point, X is the horizontal coordinate, representing the distance of the point along the horizontal direction relative to a certain reference plane (usually the horizontal plane). Y is the depth coordinate, representing the distance of the point along the front-back direction relative to the reference plane. Z is the vertical coordinate, representing the distance of the point along the up-down direction relative to the reference plane. L is the maximum frequency level of the encoding, corresponding to the resolution of the grid, 2 L is the calibration index for the high-frequency part, which can help the model better capture high-frequency details;

[0095] Step 4-2: Progressive refinement encoding: In the process of progressive grid refinement, the resolution of the grid gradually increases with the increase of the level. To match the grid resolution, the frequency encoding also increases accordingly. On the low-resolution grid, we use the Fourier encoding with the set minimum frequency to capture the large-scale scene features; while on the high-resolution grid, the frequency gradually increases to capture more refined high-frequency details. Assuming the grid has n levels, for the l-th level grid, the maximum frequency of its frequency encoding is 2 l-1 , l < n, that is, as the resolution increases, the frequency also grows exponentially, expressed as follows:

[0096] β l (x) = {sin(2 0 πx), cos(2 0 πx), sin(2 1 πx), cos(2 1 πx)... sin(2 l-1 πx), cos(2 l-1 πx)}

[0097] Step 5: Connect multiple multi-layer perceptrons MLP using a residual network to achieve an implicit representation of the scene;

[0098] The implicit representation is defined as a function mapping:

[0099] F: x, v → σ, c, where x is the spatial coordinate of the sampling point, v is the direction of the sampling ray, The network structure is composed of multiple multi-layer perceptrons MLP, specifically including F1, F2,... F n , and the input of each MLP is the feature vector output by the previous MLP and the set of position encodings corresponding to the frequency of this layer;

[0100] Step 5-1: The input of the first-level MLP network F1 is the concatenation of the unencoded spatial coordinates x of the sampling points and the data β1(x) after frequency encoding, and the output vector is the feature vector O1 learned by this layer, expressed as:

[0101] F1:x,β1(x)→O1,l=1

[0102] Step 5-2: The MLP network of the second layer and each subsequent layer takes the output O from the previous layer and the position-frequency code β(x) of the current layer as input; specifically, the MLP network of the l-th layer takes the output O of the l-1 layer MLP as input. l-1 and the position frequency coding set β of this level l (x) is taken as input, and the intermediate feature vector O learned at this level is output. l The formula is expressed as follows:

[0103] F l :O l-1 ,β l (x)→O l ,l>1

[0104] Step 5-3: A residual network is used to connect the various network layers to form the overall network architecture. This progressive residual structure can enhance the feature representation of each layer layer by layer. Each location frequency encoding generates feature outputs through the corresponding network layer. To better fuse information from different frequencies, a residual network structure is used to connect the outputs of each frequency layer, as shown below:

[0105] O l =O l-1 +w l β l (x)+k, where w l These are weighting coefficients used to control the contribution of each level's output, and k is a constant. This progressive residual structure enables the model to learn incremental information relative to the previous level, thereby progressively improving the network's representation ability.

[0106] Step 5-4: Perform spectral encoding on the sampling ray direction v using existing frequency encoding formulas:

[0107]

[0108] Step 5-5: The feature vector output by the last layer of the network is O n The output is passed through a linear layer to obtain the final learned SDF value sdf. Then, the sampled ray direction information γ(v) obtained after frequency encoding is combined with the feature vector O. n The concatenation is performed and fed into an additional fully connected ReLU layer. Finally, the color value c is predicted by the sigmoid activation function.

[0109] Step 6: Sampling Point Visibility Estimation and Reconstruction of Lighting and Shadows in the Relighting Task. Accurate lighting and shadow calculation is crucial in the relighting task. Visibility is estimated for each sampling point in the scene to determine whether that point will be directly illuminated by a light source under given lighting conditions.

[0110] Step 6-1: Calculate sampling point p i With respect to the position P of the light source l The straight-line distance Z in three-dimensional space frag The formula is as follows:

[0111] Z frag =‖xP l ||

[0112] Where x is the spatial coordinate of the sampling point, P l These are the spatial coordinates of the light source point.

[0113] Step 6-2: Switch the camera position o to the light source point position P l Repeat steps 2 to 3 to learn the SDF values ​​of the sampling points on the ray, and denote the sampling points with an SDF value of 0 as q; connect the sampling points p. i With respect to the position P of the light source l The point on the line connecting two points is denoted as q if it has an SDF value of 0. i Calculate q i Point relative to the position of the light source P l The straight-line distance Z in three-dimensional space shadow The formula is as follows:

[0114] Z shadow =‖x q -P l ||

[0115] Where, x q For sampling point q i Spatial coordinates, P l These are the spatial coordinates of the light source point;

[0116] Step 6-3: Estimate sampling point p i Visibility, through Z frag and Z shadow To determine visibility; when Z frag Greater than Z shadow If there is an obstruction on the line connecting the sampling point and the light source, the sampling point is located in the shadow area and its visibility is 0; otherwise, the visibility is 1. The formula is as follows:

[0117]

[0118] Among them, V pThe visibility of the sampling ray at that sampling point;

[0119] Step 7: Render the image using the volume rendering formula;

[0120] Step 7-1: Calculate the volume density σ corresponding to each sampling point on the sampling ray. i :

[0121] Ω(x)=se -sx / (1+e -sx ) 2 ,σ i =Ω(F(p) i ))

[0122] Where Ω(x) represents the density function, s is the hyperparameter, F is the network function for learning the SDF value, and p i Indicates the sampling point.

[0123] Step 7-2: Calculate the cumulative value of the volume density corresponding to all sampling points on the sampling ray, i.e., the opacity T(t):

[0124]

[0125] Where t is the sampling distance and σ is the volume density of the sampling points.

[0126] Step 7-3: Calculate the density weight ω(t) corresponding to each sampling point on the sampling ray:

[0127] ω(t)=T(t)σ(t)

[0128] Step 7-4: Use the visibility V of the sampled ray p The density weights ω(t) of all sampling points on the sampling ray and the color values ​​c corresponding to the sampling points are calculated by summing to obtain the color value C(p) of the pixel:

[0129] Where N is the number of sampling points, ω i c represents the density weight value of the sampling points. i This represents the color learned by the sampling point.

[0130] Step 8: Calculate the overall loss; train the network jointly using the rendering loss and SDF regularization loss.

[0131] Step 8-1: Calculate rendering loss;

[0132] The mean square error between the actual RGB value of a pixel and the rendered RGB value is calculated as follows:

[0133]

[0134] Where C(p) represents the rendered RGB value, C gt (p) represents the actual RGB value of the pixel, R(p) is the set of all rays, and p is the sampled ray.

[0135] Step 8-2: Calculate the SDF regularization loss;

[0136] The SDF regularization loss is used to ensure that the learned SDF function has a gradient close to 1 near the object's surface, thus ensuring that the SDF accurately represents the object's boundaries. It is expressed as follows:

[0137]

[0138] Where x is the sampling point on the sampling ray p.

[0139] Step 8-3: Calculate the total loss;

[0140] The overall loss is a weighted sum of the rendering loss and the SDF regularization loss, expressed as follows:

[0141] l = l color +λl eikonal , where λ is a hyperparameter.

[0142] Step 9: During the experiment, we trained the neural network on an Nvidia 2080Ti GPU using the PyTorch framework. Each model underwent 1000k iterations of training and was optimized using the Adam optimizer. We used PSNR (Peak Signal-to-Noise Ratio) as the evaluation metric. PSNR is primarily used to evaluate the detail reproduction and structural consistency between the generated image and the real image.

[0143] Step 10: In the testing phase, we use the trained model to synthesize images from the new perspective. This is done using the volume rendering formula. We tested it in 6 different scenarios (Verdant Guardian, Blush Elephant, Blushing Girl, Tiny Guardian, Wise Owl, Colorful Bloom).

[0144] Comparison of PSNR↑ metrics on the Area light-Blender dataset

[0145]

[0146] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A neural radiation field re-illumination method based on progressive mesh refinement, characterized in that, Includes the following steps: Step 1: Construct the dataset and divide it into training and test sets; Step 1-1: Collect images taken by the camera from different perspectives to construct a dataset, and obtain the images and corresponding intrinsic parameters in the dataset. The intrinsic parameters include the camera's viewpoint direction v, the camera's position o when the image was taken, and the light source position P. l ; Steps 1-2: Divide the dataset into training and test sets according to a set ratio; Step 2: Emit a sampling ray from the camera position o in the dataset and perform discrete point sampling; Step 3: Construct a progressive multi-resolution feature grid and perform feature extraction; Step 3-1: Multi-resolution representation of the mesh: A multi-resolution mesh is constructed in a progressive manner, with each mesh layer corresponding to spatial detail information at different scales. Specifically, this is represented as follows: ,in r = 1, 2, ..., n Represents the grid of the r-th layer. This is the resolution of the mesh layer, and n is the number of mesh layers; Step 3-2: Feature extraction process: In the progressive mesh, features are extracted layer by layer to refine the mesh, and multilayer perceptron (MLP) is used to extract 3D spatial features from meshes of different resolutions. For each mesh layer The MLP network takes the sampling point position x and sampling ray direction v obtained in step 2 as input to obtain the feature representation of the input sampling point learned by the MLP, namely the SDF value. Step 3-3: Progressive Training: A progressive training strategy is adopted, which first trains on a low-resolution grid and then gradually uses the residual network to learn on a higher-resolution grid for training and optimization. Step 4: Construct position codes for the corresponding frequencies based on the progressive feature grid; Step 4-1: Utilizing the concept of progressive multi-resolution grid refinement combined with position coding technology, generate codes of different frequencies for each sampling point of the neural radiation field. Specifically, different resolution grids correspond to different frequency Fourier position codes. For each grid point, using its spatial coordinates (X,Y,Z), perform position frequency coding using Fourier transform. Specifically, the position frequency coding function is expressed as: ； Where x represents the spatial coordinates of the grid point, X is the horizontal coordinate, representing the distance of the point relative to the horizontal plane along the horizontal direction; Y is the depth coordinate, representing the distance of the point relative to the horizontal plane along the front-back direction; Z is the vertical coordinate, representing the distance of the point relative to the horizontal plane along the vertical direction; L is the maximum frequency level of the encoding, corresponding to the grid resolution, 2 L The calibration index indicates the high-frequency range; Step 4-2: Progressive refinement encoding: Use Fourier encoding with a set minimum frequency to capture large-scale scene features; while on a high-resolution grid, the frequency gradually increases; the grid has n levels, and for the l-th level grid, the maximum frequency of its frequency encoding is 2 l-1 , l < n, that is, as the resolution increases, the frequency also grows exponentially. The representation of the position frequency encoding function corresponding to the l-th level grid is as follows: ； Step 5: Connect multiple multilayer perceptron (MLP) networks using residual networks to achieve implicit representation of the scene; The implicit expression is defined as a function mapping: F Where x is the spatial coordinate of the sampling point, and v is the direction of the sampling ray. , An MLP network consists of multiple multilayer perceptron MLPs, where the multilayer perceptron MLPs are F1, F2, ..., F... n The input to each MLP is the feature vector output by the previous MLP and the position encoding set of the corresponding frequency in this layer; Step 5-1: The input to the first-level MLP network F1 is the uncoded spatial coordinates x of the sampling points and the frequency-coded data. The vectors are concatenated, and the output vector is the feature vector learned in this layer. , represented as: ； Step 5-2: The MLP network of the second layer and each subsequent layer takes as input the output O from the previous layer and the position-frequency code of the current layer. Specifically, the L-th level MLP network will process the output of the (l-1)-th level MLP. and the location frequency coding set of this level As input, and output the intermediate feature vector learned at this level. The formula is expressed as follows: ; Step 5-3: Connect the various network layers using a residual network to form the overall network architecture. The frequency encoding at each location is processed by the corresponding network layer to generate feature outputs. The residual network structure is used to connect the frequency outputs of each layer, as shown below: ,in These are the weighting coefficients, where k is a constant; Step 5-4: Perform spectral encoding on the sampling ray direction v: ； Step 5-5: The feature vector output by the last layer of the network is The output is passed through a linear layer to obtain the final learned SDF value, and then the sampled ray direction information obtained after frequency encoding is used. With feature vectors The concatenation is performed and fed into an additional fully connected ReLU layer. Finally, the color value c is predicted by the sigmoid activation function. Step 6: Sampling point visibility estimation, reconstructing the lighting and shadows in the relighting task, and determining whether the point will be directly illuminated by the light source under given lighting conditions; Step 6-1: Calculate sampling point p i With respect to the position P of the light source l The straight-line distance Z in three-dimensional space frag The formula is as follows: ； Where x is the spatial coordinate of the sampling point, P l These are the spatial coordinates of the light source point; Step 6-2: Switch the camera position o to the light source point position P l Repeat steps 2 to 3 to learn the SDF values ​​of the sampling points on the ray, and denote the sampling points with an SDF value of 0 as q; connect the sampling points p. i With respect to the position P of the light source l The sampling point with an SDF value of 0 on the straight line connecting the two points is denoted as q. i Calculate q i Point relative to the position of the light source P l The straight-line distance Z in three-dimensional space shadow The formula is as follows: ; Where, x q For sampling point q i Spatial coordinates, P l These are the spatial coordinates of the light source point; Step 6-3: Estimate sampling point p i The visibility of the sampled ray is determined by Z. frag and Z shadow To determine visibility; when Z frag Greater than Z shadow If there is an obstruction on the line connecting the sampling point and the light source, the sampling point is located in the shadow area and its visibility is 0; otherwise, the visibility is 1 and the sampling point p... i The visibility of the sampled ray is expressed as follows: ； Among them, V p The visibility of the sampling ray at that sampling point; Step 7: Render the image using the volume rendering formula; Step 7-1: Calculate the volume density corresponding to each sampling point on the sampling ray. : , ; in, Let p represent the density function, s be the hyperparameter, F be the network function for learning the SDF value, and p be the density function. i Indicates the sampling point; Step 7-2: Calculate the cumulative value of the volume density corresponding to all sampling points on the sampling ray, i.e., the opacity. : ； Where t is the sampling distance. The volume density of the sampling points; Step 7-3: Calculate the density weight corresponding to each sampling point on the sampling ray. : ; Step 7-4: Use the visibility V of the sampled ray p Density weights of all sampling points on the sampling ray The color value c corresponding to the sampling point is calculated by summing the results to obtain the color value of the pixel. : Where N is the number of sampling points. This represents the density weight value of the sampling points. This represents the color learned by the sampling point; Step 8: Calculate the overall loss; Step 8-1: Calculate rendering loss; The rendering loss is expressed as follows: ; in, This represents the rendered RGB values. R(p) represents the actual RGB value of a pixel, where R(p) is the set of all rays and p is the sampled ray. Step 8-2: Calculate the SDF regularization loss; The SDF regularization loss is expressed as follows: ； in, Let be the sampling point on the sampling ray p, i.e., the spatial coordinates of the sampling point; Step 8-3: Calculate the total loss; The overall loss is a weighted sum of the rendering loss and the SDF regularization loss, expressed as follows: ,in For hyperparameters; Step 9: Iterative training. Iteratively train the MLP network, repeating steps 2 to 8 each time, using different training samples for optimization, until the performance of the MLP network reaches the predetermined convergence criterion; finally, save the trained MLP network. Step 10: Using volume rendering formulas Generate a new perspective image.

2. The neural radiation field re-illumination method based on progressive mesh refinement according to claim 1, characterized in that, Step 2 includes the following steps: Step 2-1: For each image in the dataset, generate a sampling ray based on the camera's viewpoint direction. : Where o is the camera position, which is also the origin of the sampling ray. That is, three-dimensional data, where v is the direction of the sampling ray. That is, two-dimensional data, with the sampling ray starting from the camera position. This represents the sampling distance, extending from the pixels on a two-dimensional image to the three-dimensional scene corresponding to that image's view. Step 2-2: Perform random uniform sampling on the sampling ray, using the following sampling formula: ,in, This represents the sampling distance of the i-th sampling point. Indicates the near-end sampling point on the ray. This represents the far-end sampling point on the ray, and N represents the number of sampling points.

Images