NeRF-based three-dimensional reconstruction method

Abstract

The invention discloses a three-dimensional reconstruction method based on NeRF. The three-dimensional reconstruction method comprises the following steps: step 1, synchronously acquiring multi-view images; 2, preprocessing the original image; step 3, extracting self-adaptive visual angle fusion features of the multi-visual angle image preprocessed in the step 2; 4, three-dimensional rays are generated, and coordinate transformation is completed; 5, setting and implementing a lightweight dynamic sampling strategy; 6, calculating a density field and a color field, and obtaining a loss function value after volume rendering is completed; and step 7, generating a three-dimensional mesh model, and completing optimization of the three-dimensional mesh model. The invention belongs to the technical field of computer vision and three-dimensional reconstruction, and solves the problems of low efficiency of multi-view feature fusion, high computing resource consumption and particularly small object detail missing in the prior art.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and three-dimensional reconstruction, and relates to a three-dimensional reconstruction method based on NeRF. BACKGROUND

[0002] Three-dimensional reconstruction, as a hot research direction in computer vision tasks, has shown wide application potential in virtual reality, industrial design, e-commerce and other fields. Traditional three-dimensional reconstruction methods can be roughly divided into two categories: one is geometry-based three-dimensional reconstruction, such as structure from motion (SFM) and multi-view stereo matching (MVS); the other is single-view reconstruction based on deep learning, such as Pixel2Mesh and 3D-R2N2. However, these methods each have obvious limitations. The geometry-based method highly depends on the accurate matching of feature points, and is therefore very sensitive to noise, occlusion and weak texture areas. The single-view reconstruction method based on deep learning often leads to poor integrity and missing details of the reconstructed model due to the limitation of input information.

[0003] In recent years, the neural radiance field (NeRF) technology models the radiation and density distribution of a scene through an implicit neural field, achieving highly realistic view synthesis. Compared with traditional three-dimensional reconstruction methods, NeRF does not need to rely on explicit geometric priors, and can complete reconstruction only with multi-view images and camera poses, and generate sub-pixel level details. However, direct use of existing NeRF technology still faces some key problems: 1) low efficiency of view fusion, multi-view feature fusion usually relies on simple splicing or averaging, without fully considering the weight difference between views, resulting in poor geometric consistency; 2) huge consumption of computing resources, dense sampling and large-scale multi-layer perceptron (MLP) inference make the training process extremely time-consuming, often taking hours or even days; 3) insufficient reconstruction accuracy for small objects, especially for objects with complex surface texture or reflective properties (such as earphones and metal shell objects), the reconstructed results are prone to have holes and blur.

[0004] Although there are some improved solutions, such as Mip-NeRF which improves the anti-aliasing ability through cone sampling, and Instant-NGP which accelerates the training process using hash coding, these improved solutions have not fundamentally solved the above problems. Therefore, it is urgent to develop a new adaptive view fusion mechanism and lightweight dynamic sampling strategy to significantly improve the accuracy and efficiency of three-dimensional reconstruction and fill the gap in the current technical field. SUMMARY

[0005] The purpose of the present application is to provide a three-dimensional reconstruction method based on NeRF, which solves the problems of inefficient multi-view feature fusion, large consumption of computing resources, and especially missing details of small objects in the prior art.

[0006] The technical scheme adopted by the present application is a NeRF-based three-dimensional reconstruction method, which is implemented according to the following steps: Step 1, synchronously collecting multi-view images; Step 2, preprocessing the original images; Step 3, extracting adaptive view angle fusion features of the multi-view images preprocessed in step 2; Step 4, generating three-dimensional rays and completing coordinate conversion; Step 5, setting and implementing a lightweight dynamic sampling strategy; Step 6, calculating a density field and a color field, and obtaining a loss function value after completing volume rendering; Step 7, generating a three-dimensional mesh model and completing optimization of the three-dimensional mesh model.

[0007] The present application has the beneficial effects that an adaptive view angle fusion mechanism and a lightweight dynamic sampling strategy are proposed, the adaptive view angle fusion mechanism dynamically weights multi-view features by using a cross-view attention module, can significantly improve the geometric consistency and precision of a three-dimensional model, and enhances the visual quality and generalization ability of the three-dimensional model; the lightweight dynamic sampling strategy combines feature map guided gradient sampling and hash coding, greatly reduces the computational overhead, and improves the reconstruction efficiency; the finally generated three-dimensional mesh model is optimized and can be directly applied to various practical scenarios, has the characteristics of high efficiency and practicality, realizes efficient and high-quality three-dimensional reconstruction, and is especially suitable for high-precision and real-time reconstruction of small objects. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a flowchart of the method of the present application; Figure 2 is a block diagram of the adaptive view angle fusion mechanism in the method of the present application; Figure 3 is a block diagram of the lightweight dynamic sampling strategy in the method of the present application; Figure 4 is an architecture block diagram of the NeRF radiation field in the method of the present application. DETAILED DESCRIPTION

[0009] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0010] The NeRF-based three-dimensional reconstruction method of the present application is implemented according to the following steps: Step 1, synchronously collecting multi-view images, The target object is synchronously photographed by a pre-calibrated multi-camera array (four view angles of front, back, left and right), ensuring that there is no occlusion between view angles and the light is uniform, and the specific process is as follows: 1.1) Calibrate the camera parameters: Use the checkerboard calibration method to calibrate the intrinsic parameters (focal length, principal point) and extrinsic parameters (relative pose) of the four cameras to ensure spatial alignment of images from different viewpoints. 1.2) Control the shooting environment: In a constant temperature and dust-free laboratory, use a ring-shaped LED light source to provide uniform lighting to avoid reflections or shadows; 1.3) Capture and store images: Trigger four cameras simultaneously to capture PNG format images. These images are saved by naming them according to the viewing angle to obtain the original images. Step 2: Preprocess the original image. Perform color correction, denoising and sharpening on the original image obtained in step 1. The specific operations are as follows: 2.1) Perform color correction on the original image, Place a standard whiteboard in the scene and optimize the color correction matrix K and bias vector B based on the white balance algorithm. The correction formula is: , in, Represents the rectified image matrix, with dimensions of H×W×3, where H is the image height, W is the width, and 3 represents the RGB channels; K is a 3×3 color correction matrix, which minimizes the color error of the whiteboard area ( ) is optimized and used for linear transformation of color space; is the original input image matrix, with the same dimension as same; B is a 3D bias vector used to eliminate the color shift caused by ambient light, thereby obtaining a corrected image; 2.2) Perform non-local mean denoising (e.g., NL-Means) to remove noise from the rectified image, significantly improving the quality of the rectified image. The specific process is as follows: For each pixel in the rectified image ,First, define a local image block centered on it With a search window ; The local image block Also known as a neighborhood block, for example, the size is 7×7 pixels; the search window Usually it needs to be much larger than the neighborhood block, for example, the size is 21×21 pixels; then, calculate each pixel in the rectified image Neighborhood blocks and their search windows Other pixels in The similarity between the neighborhood blocks is used to assign weights to each pixel and use these weights to search the window. The pixel values ​​within are weighted averaged to estimate the final denoised pixel value, thereby removing noise and retaining image details. The denoising formula is: , = , in, Represents the pixel after denoising x The value of is the normalization factor, ensuring that the sum of the weights is 1, that is, = , Indicates that in the current search window The rest of the pixels y Relative to the current pixel x The allocation weight of Indicates x The neighborhood block centered on express y The neighborhood block centered on h is a smoothing parameter that controls the weight decay rate and is usually set to 10. Thus, the preprocessed multi-view image is obtained.

[0011] Step 3: extract the adaptive perspective fusion features of the multi-view image preprocessed in step 2. Reference Figure 2 In this step, multi-view features are dynamically integrated through the Cross-View Attention (CVA) module. The specific process is as follows: 3.1) Set up the backbone network and output shallow features. To more efficiently extract multi-scale features from multi-view images, this step uses the lightweight and high-performance EfficientNet-B3 network as the backbone network. The EfficientNet-B3 network uses a compound scaling strategy to balance depth, width, and resolution while maintaining high precision.

[0012] Specific configuration includes: During the input processing phase, images from all perspectives are uniformly scaled to 512×512 pixels to meet the network input requirements. In terms of hierarchical structure, shallow feature extraction is performed by the first three modules (Conv1, MBConv1, and MBConv2). These modules use 3×3 convolution and depthwise separable convolution to gradually extract low-level features such as edges and textures. The MBConv1 module contains 16 channels, an expansion factor of 1, and the activation function is Swish. Deep semantic extraction is performed by the second two modules (MBConv3 and Conv5) through 5×5 convolution and a channel attention mechanism (SE Block) to capture the global semantic information of the target object. The MBConv3 module outputs a 40-channel feature map, and the Conv5 layer uses 1×1 convolution to increase the feature dimension to 256. In the output adjustment stage, the global pooling layer and classification head in the original EfficientNet-B3 network are removed, and the multi-scale feature maps (MBConv2: 256×256×32, Conv5: 64×64×256) are retained, and shallow features are output for further processing by the subsequent cross-view attention module.

[0013] 3.2) Construct a cross-view attention module to obtain preliminary fusion features. The cross-view attention module fuses multi-view features through a dynamic weighting mechanism, solving the problem of geometric inconsistency caused by fixed weights between views in traditional methods.

[0014] The core structure of the cross-view attention module is divided into three parts: First, in the query-key generation layer, the 256-dimensional feature map of each view (from the Conv5 layer) is passed through three independent linear layers (256→32) to generate the query vector , key vector , value vector For example, for four-view input (front, back, left, right), a total of four groups are generated Q, K, V , the dimensions are all 32; Secondly, the attention weight is calculated, the correlation between perspectives is calculated by dot product, and a scaling factor is introduced to prevent gradient disappearance. The calculation formula is: , in, , indicating the viewing angle i Perspective j The attention weight of is in the range of [0,1]; Perspective i The query vector of dimension 32 is obtained by mapping the feature vector through a linear layer; Perspectivej The key vector has a dimension of 32 and is generated in the same way as ; is the scaling factor to prevent the dot product result from being too large. Softmax The gradient disappears; Finally, feature fusion is performed and weighted aggregation value vector is obtained. V , generate a preliminary fusion feature map, the calculation formula is: ; 3.3) Output the optimized fusion feature map, The initial fusion feature is reduced to 128 dimensions by a 1×1 convolution layer and concatenated with the shallow features (MBConv2 layer output) obtained by the backbone network to enhance detail retention. Finally, the optimized fusion feature map is obtained. .

[0015] Step 4: Generate three-dimensional rays and complete coordinate transformation. In this step, the two-dimensional image pixels obtained in step 3 need to be mapped to a three-dimensional space coordinate system to generate three-dimensional rays for subsequent sampling. The specific implementation process is as follows: 4.1) Normalization of pixel coordinates, For each two-dimensional image of a viewing angle, the pixel coordinates ( u , v ) is converted to normalized device coordinates NDC. The conversion formula is as follows: , in, W and H are the width and height of the image, respectively. The normalized device coordinates map pixel coordinates to the range [−1, 1] to facilitate subsequent three-dimensional space calculations; 4.2) Calculate the ray direction, In the camera coordinate system, the vector of the ray direction By normalized coordinates and OK, the calculation formula is as follows: , in, n is the near plane distance, and is the focal length of the camera; This formula calculates the ray direction by normalizing the device coordinates and camera parameters to ensure that the ray starts from the camera origin and points to the corresponding pixel in the image plane; 4.3) Convert to world coordinate system, Using the camera's external parameters, the ray direction is converted from the camera coordinate system to the world coordinate system. The conversion formula is as follows: , where, is the direction of the ray in the world coordinate system, is the origin of the ray in the world coordinate system; rotation matrix R and translation vector represent the position and pose of the camera in the world coordinate system, respectively; 4.4) Ray parameterization, The calculation formula of each ray is as follows: , where, t is the parameter, which represents the offset of the ray in the direction, and the parameter range is usually determined by the bounding box estimation or preset value, which is used to limit the sampling range of the ray in the scene.

[0016] Step 5, set and implement the lightweight dynamic sampling strategy, Referring to Figure 3 , this step is based on the fusion feature map obtained in step 3 and the three-dimensional rays generated in step 4 , through hierarchical dynamic sampling and hash coding technology, the calculation cost is greatly reduced while ensuring the reconstruction accuracy, the implementation process is as follows: 5.1) Hierarchical dynamic sampling, Traditional NeRF uses uniform dense sampling method, which needs to sample hundreds of thousands of points along the direction of the ray in the object bounding box, the calculation cost is extremely high, while this step significantly reduces the number of sampling points through a two-stage sampling strategy, the specific description is as follows: 5.1.1) Implement coarse sampling: uniformly sample 64 points along each ray in the object bounding box, the number of sampling points can be adjusted appropriately according to different objects to ensure the coverage of the overall density distribution of the scene; 5.1.2) Determine the fine sampling area (guided by the fusion feature map output in step 3): First, extract the spatial gradient from the fusion feature map For each channel of C , use the Sobel operator to calculate the horizontal gradient C y and the vertical gradient x , the calculation formula is as follows: , = , = ; Secondly, for each channel, a corresponding gradient heat map is generated, which refers to The gradient magnitude map in the spatial dimension (height, width) reflects the severity of the feature change. The gradient magnitude C The calculation formula is: , Next, multi-channel aggregation is performed to take the average or maximum value of the gradient amplitudes of multiple channels to generate the final multi-channel heat map G ; Finally, the high gradient area is identified, preferably the threshold value r = 0.1 is set, and G The area of ​​r is identified as the high gradient area, that is, the area determined to be finely sampled; 5.1.3) Complete fine sampling: In the selected fine sampling area, insert additional sampling points along the ray direction (for example, insert 2 points per segment) to ensure that the sampling density of the detail area is increased by 3 times.

[0017] 5.2) Use hash coding to speed up, To address the computational complexity of traditional coordinate encoding (such as position encoding), this step uses multi-resolution hash encoding to map 3D coordinates to a compact hash table. The specific process is as follows: 5.2.1) Constructing a multi-resolution hash table: First, set up 8 layers of hash tables with different resolutions, according to Out of each level l Resolution, such as layer 1 = 32, 8th layer = 512; Then, each sampled 3D coordinate point is hashed and encoded at the level l quantize it to the nearest neighbor point and pass it through the hash function Mapped to the hash table location, the hash table size is ; Finally, each hash table entry stores an 8-dimensional feature vector ∈ , optimize the feature value through training; 5.2.2) Perform feature interpolation and fusion: Since the actual sampling points are usually non-integer points, relying solely on the discrete grid features obtained from the hash table will cause the feature space to show "jumps". Therefore, it is necessary to use trilinear interpolation to further obtain the continuous feature representation of the sampling point. In each layer of the hash table, the feature values ​​of the 8 points around it are taken and the feature of the point is calculated by trilinear interpolation. , and then the features of the 8-layer hash table , ,..., Concatenate to get 64-dimensional vector as the input of the density prediction network branch in step 6; for the view direction , a 16-dimensional vector is obtained by Spherical Harmonics encoding, which is concatenated with the coordinate feature and used as the input of the color prediction network branch in step 6.

[0018] This step 5 finally obtains a 64-dimensional vector as the input of the density prediction network in step 6, and a 16-dimensional vector as the input of the color prediction network.

[0019] Step 6, calculate the density field and the color field, and obtain the loss function value after completing the volume rendering, Referring to Figure 4 , this step divides the NeRF radiation field into a density field and a color field through a NeRF network, and predicts the volume density and color information in the three-dimensional space respectively, which significantly reduces the computational complexity and improves the detail reconstruction capability. The specific process is as follows: 6.1) Construct a density prediction network branch to output the density corresponding to the coordinate point

[0020] In this density prediction network branch, the input is the hash encoding of the 3D coordinate , which is generated by the hash table in step 5; The density prediction network branch includes four fully connected layers. The first layer maps the input dimension from 64 to 256, uses the ReLU activation function, and uses the He normal distribution for weight initialization; the second layer further reduces the dimension from 256 to 128, uses the ReLU activation function; the third layer reduces the dimension from 128 to 64, uses the ReLU activation function, and outputs a 64-dimensional intermediate feature feature ; the fourth layer maps the intermediate feature feature from the dimension 64 to 1, and the density prediction network branch finally outputs the density value corresponding to the coordinate point ; the set of density values is the density field. In this way, the density is predicted through the density prediction network branch, avoiding the interference of color information on geometric modeling.

[0021] 6.2) Construct a color prediction network branch, In this color prediction network branch, the intermediate feature feature output in the density network branch is concatenated with the 16-dimensional vector obtained by Spherical Harmonics encoding of the view direction to obtain an input vector with a dimension of 80; The color prediction network branch contains four fully connected layers. The first layer maps the input dimension from 80 to 256, uses the ReLU activation function, and uses the He normal distribution for weight initialization; the second layer reduces the dimension from 256 to 128, also uses the ReLU activation function, and the bias initialization is set to 0; the third layer further reduces the dimension from 128 to 64 and introduces residual connections to alleviate the gradient disappearance problem; the fourth layer reduces the dimension from 64 to 3, uses Sigmoid as the activation function, and finally outputs the normalized RGB color value. c , after integration it is called the color field; 6.3) Implement volume rendering, Volume rendering is a 3D scene visualization technology based on density field and color field. The input of volume rendering includes the position of sampling points, density field, color field and camera parameters; the output of volume rendering is a rendered image. ; The final result of the volume rendering process is a rendered image, which is used to calculate the loss function value of the NeRF network together with the real image. The loss function value is used to guide the iterative optimization of the network parameters. For the prediction network, the core is the density value. With color value c Perform multiple iterations of optimization, and the density value This is directly related to the generation of the 3D mesh model in step 7. Steps 4 and 5 above map the 2D image to a 3D sampling space, while volume rendering maps the 3D sampling space back to 2D. These two processes can be considered as inverse processes, so it is necessary to ensure that the sampling points used during volume rendering remain the same as in the previous steps. This requires that the ray generation and layered sampling strategies used are consistent with those used before, which is the "maintain consistency" description in the following sub-steps.

[0022] The specific steps are as follows: 1) Ray generation: This process is consistent with the 3D ray generation in step 4, and the ray is obtained ; 2) Stratified sampling: This process is consistent with the stratified dynamic sampling in step 5; 3) Numerical integration: First, calculate the transmittance at each sampling point , the calculation formula is: , in, is the distance between adjacent points, t i Represents rays A sampling point on is the density value corresponding to the point obtained by the density prediction network branch, Then, the transmittance of the ray is calculated , and the color contribution of all sampling points on the ray is calculated , wherein, t i represents a certain sampling point on the ray , and represents the color value corresponding to the point obtained by the color prediction network branch; 4) Pixel color assignment: assign the integral result to the corresponding RGB value of the pixel ; 6.4) Neural network loss function, use the loss function used in the original NeRF network, that is, calculate the L1 error between the rendered image and the real image , and the calculation formula is as follows: , wherein, represents the final loss function value, N is the total number of pixels, is the RGB value of the rendered image at pixel p , is the RGB value of the real image at pixel p , is the L1 norm, which calculates the absolute error. (Here, is the L1 norm, which calculates the absolute error; and the final is referred to as L1 error, that is, the mean absolute error).

[0023] Step 7, generate a three-dimensional grid model and complete the optimization of the three-dimensional grid model, In this step, the three-dimensional grid model needs to be generated by extracting the isosurface from the density field obtained in step 6, and the specific process is as follows: 7.1) Isosurface extraction: use the Marching Cubes algorithm to extract the isosurface. The Marching Cubes algorithm traverses all cubes in the three-dimensional voxel grid, and according to the relationship between the density values of the vertices of the cube and the set density threshold =0.5, determines the object surface by determining the position of the isosurface in the voxel cube, thereby generating an initial three-dimensional grid. The marching cubes algorithm can effectively extract continuous isosurface from discrete density field, and provides a basis for subsequent mesh optimization.

[0024] 7.2) Optimization of the initial three-dimensional mesh: the initial three-dimensional mesh is smoothed by using a Laplace smoothing algorithm, which adjusts the position of each vertex by iteratively calculating the Laplace value of each vertex, i.e. the difference between the vertex and its neighborhood vertices, so as to reduce the noise and jagged effect of the mesh. Specifically, the smoothing coefficient is iterated multiple times, and the smoothing coefficient may be set to 0.3 to ensure that certain detail information is retained while smoothing, and the displacement correction amount of Laplace smoothing is calculated as follows: , wherein, is the displacement correction amount of the vertex i , the neighborhood vertex set of the vertex , the three-dimensional coordinates of the vertex i , the three-dimensional coordinates of the i-th vertex in the neighborhood vertex set of the vertex , i V j i N(i) j Then, the position of the vertex is adjusted according to the displacement correction amount , and the update formula is as follows: , After updating all vertices, the optimized three-dimensional mesh model is obtained.

[0025] The overall method of the application comprises seven steps, and the core innovation point lies in the concept of density field. In step 5, a set of three-dimensional sampling points is obtained, and in step 6, the density value corresponding to each sampling point is obtained through the density network prediction branch, so that a set of density values is obtained, which is the density field. The density field is also a three-dimensional space like the space of the sampling points. In the final step 7, the isosurface of the density field is extracted by the marching cubes algorithm in step 7.1), and a preliminary three-dimensional mesh model can be obtained. Step 7.2) is the optimization of the preliminary three-dimensional mesh model, and a three-dimensional mesh model that can restore the real object more accurately is obtained.

[0026] Embodiment 1 The implementation object of this embodiment 1 is a headset.

[0027] Reference Figure 1 ​​​​​, the NeRF three-dimensional reconstruction method based on adaptive view angle fusion and light dynamic sampling is implemented according to the application, including the following steps: Step 1, a pair of earphones is synchronously photographed by a calibrated multi-camera array (four views of front, back, left and right), ensuring that there is no occlusion between the views and the light is uniform, and the specific implementation process is as follows: 1.1) When photographing four views of images, select a time period with uniform light to photograph, while keeping the distance between the cameras and the object in different views equal to ensure the consistency of the views; 1.2) Ensure that each view covers the ear cover, head beam and connecting parts when photographing; 1.3) Collect 400 groups of original images, name and store them according to the view.

[0028] Step 2, pre-process the obtained original images; 2.1) Color correction: optimize the correction matrix K and bias B through the whiteboard area to ensure that the error between the ear cover metal color and the true color is E<1.0; 2.2) Denoising: effectively retain the ear cover mesh texture by non-local mean denoising (neighborhood window 7x7, h=10).

[0029] Step 3, the adaptive view angle fusion feature extraction of embodiment 1 is as shown in Figure 2 , and the specific implementation process is as follows: 3.1) Extract multi-scale features by the backbone network (EfficientNet-B3), and give more weight to the front view by the cross-view attention module (CVA) to highlight the details of the ear cover; 3.2) Output the fused feature map , and the response intensity of the ear cover edge channel is effectively improved.

[0030] Step 4, strictly follow the light dynamic sampling strategy for sampling, and the implementation process is as follows: 4.1) Coarse sampling: uniformly sample 64 points along each ray from the camera origin point in the set object bounding box; 4.2) Fine sampling: based on the gradient heat map G generated by the fusion feature map (threshold r=0.1), sample 3 times at the edge of the ear cover; 4.3) Hash coding: set the hash table level to 8 to effectively capture the details of the ear cover mesh.

[0031] Step 5, input the generated in step 4 and to Figure 4The illustrated NeRF network, which is composed of a density prediction network branch and a color prediction network branch, can output a density field and a color field respectively through multiple iterations of training, wherein the density field is used as input in step 7 to generate a three-dimensional mesh model.

[0032] Step 6, calculate the density field and the color field, after completing volume rendering, use the loss function used in the original NeRF network, that is, calculate the L1 error between the rendered image and the real image , the formula is as follows: , wherein, represents the final loss value, N is the total number of pixels, is the RGB value of the rendered image at pixel p , is the RGB value of the real image at pixel p , is the L1 norm, which calculates the absolute error.

[0033] Step 7, generate an initial mesh model by isosurface extraction, and then obtain a final optimized three-dimensional mesh model after multiple Laplace smoothing processes, the specific implementation process is as follows: 7.1) Isosurface extraction: Marching Cubes algorithm generates an initial mesh; 7.2) Laplace smoothing: iterate multiple times, and the roughness of the head beam surface is greatly reduced.

[0034] In this embodiment 1, three indicators of evaluation index peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and training time are used to compare the method of the present application with the existing three-dimensional reconstruction network NeRF and the improved network Instant-NGP based on NeRF. PSNR measures the overall fidelity of the rendered image and the real image, and the higher the value, the smaller the noise and the lower the color and geometric error. SSIM evaluates the local similarity of two images in structure, brightness and contrast, and the closer the value to 1, the higher the visual consistency.

[0035] The calculation formula of PSNR is as follows: , wherein, MAX is the maximum range of pixel value, and MSE is the mean square error.

[0036] The calculation formula of SSIM is as follows: , wherein, , is the image mean, , is the standard deviation, is the covariance.

[0037] Table 1 shows a comparison of experimental results using the proposed method, the existing NeRF method, and the existing Instant-NGP method. As can be seen from Table 1, the proposed method achieves higher PSNR values ​​than the other two network methods, and its SSIM values ​​are closer to 1, indicating that the proposed method achieves better reconstruction of the subject's headphones.

[0038] Table 1. Comparison of experimental results of Example 1, NeRF and Instant-NGP

[0039] Example 2 The implementation object of this embodiment 2 is a metal shell printer.

[0040] Reference Figure 1 According to the NeRF 3D reconstruction method based on adaptive view fusion and lightweight dynamic sampling of the present invention, the following steps are included: Step 1: Use a calibrated multi-camera array (front, back, left, and right viewpoints) to simultaneously capture a metal-cased printer, ensuring that there are no obstructions and uniform lighting between viewpoints. The specific implementation process is as follows: 1.1) When capturing images from all four perspectives, choose a time period with uniform lighting and maintain the same distance between the camera and the object to ensure consistent viewing angles. 1.2) When shooting, ensure that every angle of view covers important parts, and focus on capturing corners and seams; 1.3) Use a circular LED array and polarizing filter to eliminate metal reflection interference.

[0041] 1.4) A total of 400 sets of original images were collected and stored by name according to viewing angle.

[0042] Step 2: preprocessing the acquired original image; 2.1) Color Correction: The correction matrix K optimizes the contrast between metallic gray and black components; 2.2) Denoising: Non-local mean denoising (neighborhood window 7×7, h=12) is used to preserve the texture of the printer seams.

[0043] Step 3, the adaptive perspective fusion feature extraction of this embodiment 2 is as follows Figure 2 As shown, the specific implementation process is as follows: 3.1) The backbone network (EfficientNet-B3) extracts multi-scale features, and the Cross-View Attention Module (CVA) assigns more weight to side views to accurately extract the seam features of the printer. 3.2) Output fusion feature map , the channel response intensity in the seam area is effectively improved.

[0044] Step 4: Sampling is performed strictly according to the lightweight dynamic sampling strategy. The implementation process is as follows: 4.1) Coarse sampling: along each ray starting from the camera origin within the set object bounding box 64 points are uniformly sampled; 4.2) Fine Sampling: Gradient Heatmap Generated Based on Fusion Feature Map G (threshold r = 0.15), the sampling is intensified 4 times at the seams; 4.3) Hash coding: Set the hash table level to 7.

[0045] Step 5: and Input to Figure 4 The NeRF network shown in the figure consists of a density prediction network branch and a color prediction network branch. Through multiple iterative training, it can output density field and color field respectively. The density field is used as input in step 7 to generate a three-dimensional mesh model.

[0046] Step 6: Calculate the density field and color field. After volume rendering is completed, use the loss function used in the original NeRF network to calculate the rendered image. With real images The L1 error is the same as in Example 1.

[0047] Step 7: Generate an initial mesh model through isosurface extraction, and then obtain the final optimized 3D mesh model through multiple Laplace smoothing processes. The specific implementation process is as follows: 7.1) Isosurface extraction: Use the Marching Cubes algorithm and set the degree threshold = 0.5, extract the shell mesh; 7.2) Laplace smoothing: Iterate more than 3 times to effectively eliminate jagged edges in the printout.

[0048] The evaluation indicators used in this embodiment 2 are the same as those used in embodiment 1.

[0049] Table 2 shows a comparison of experimental results using the proposed method, the existing NeRF, and the existing Instant-NGP. As can be seen from Table 2, the proposed method achieves significantly higher PSNR values ​​than the other two networks, and its SSIM values ​​are closer to 1. This indicates that the proposed method achieves superior reconstruction results for the metal housing printer used in the experiment.

[0050] Table 2, Example 2 and the experimental results of NeRF and Instant-NGP are compared

[0051] Example 3 The implementation object of this embodiment 3 is a ceramic tea cup.

[0052] Referring to Figure 1 , according to the NeRF three-dimensional reconstruction method based on adaptive view angle fusion and light dynamic sampling of the application, the following steps are included: Step 1, synchronously shoot a ceramic tea cup through the calibrated multi-camera array (four views of front, back, left and right), ensure that there is no occlusion between views and the light is uniform, and the specific implementation process is as follows: 1.1) When shooting four views of images, choose a time period with uniform light to shoot, and at the same time keep the distance between the cameras and the object in different views equal to ensure the consistency of the views; 1.2) Ensure that each view covers the rim, body and handle when shooting; 1.3) Collect a total of 400 original images, store them according to the view name.

[0053] Step 2, pre-process the obtained original images; 2.1) Color correction: the correction matrix K optimizes the contrast of white porcelain and blue glaze; 2.2) Denoising: through non-local mean denoising (neighborhood window 7x7, h=11), the glaze flake texture is preserved.

[0054] Step 3, the adaptive view angle fusion feature extraction of this embodiment 3 is as shown in Figure 2 , and the specific implementation process is as follows: 3.1) The backbone network (EfficientNet-B3) extracts multi-scale features, and the cross-view attention module (CVA) gives more weight to the front view to highlight the details of the rim curve; 3.2) Output the fused feature map , and the channel response strength of the handle and body junction is effectively improved.

[0055] Step 4, strictly follow the light dynamic sampling strategy for sampling, and the implementation process is as follows: 4.1) Coarse sampling: uniformly sample 64 points along each ray from the camera origin point in the set object bounding box; 4.2) Based on the gradient heat map G generated by the fusion feature map (threshold r=0.12), sample 3 times at the rim and handle; 4.3) Hash coding: Set the hash table level to 8.

[0056] Step 5, same as example 1.

[0057] Step 6, same as example 1.

[0058] Step 7, generate an initial mesh model by isosurface extraction, and then obtain a final optimized three-dimensional mesh model after multiple Laplace smoothing treatments. The specific implementation process is as follows: 7.1) Isosurface extraction: use the Marching Cubes algorithm, set the degree threshold = 0.5, extract the tea cup surface mesh; 7.2) Laplace smoothing: iterate 3 times, and the smoothness of the connection between the cup mouth and the handle is significantly improved.

[0059] The evaluation index used in this example 3 is the same as example 1.

[0060] Table 3 shows the experimental results of the method of the present application and the NeRF of the prior art and the Instant-NGP of the prior art. As can be seen from Table 3, the PSNR value of the method of the present application is greater than that of the other two network results, and the SSIM value is closer to 1 than that of the other two network results, that is, the reconstruction effect of the method of the present application on the ceramic tea cup is better.

[0061] Table 3, comparison of experimental results of example 3 with NeRF and Instant-NGP

[0062] Example 4 The implementation object of this example 4 is a sports shoe.

[0063] Referring to Figure 1 , the NeRF three-dimensional reconstruction method based on adaptive view angle fusion and lightweight dynamic sampling is implemented according to the present application, which includes the following steps: Step 1, synchronously shoot a sports shoe through a calibrated multi-camera array (four views of front, back, left and right), ensure that there is no occlusion between views and the light is uniform, and the specific implementation process is as follows: 1.1) When shooting images of four views, select a time period with uniform light to shoot, and at the same time keep the distance between the cameras and the object in different views equal to ensure the consistency of the views; 1.2) Ensure that each view covers the toe, body, sole and shoelace area when shooting; 1.3) Collect a total of 400 groups of original images, and store them according to the view name.

[0064] Step 2, pre-process the obtained original images; 2.1) Color correction: the correction matrix K optimizes the color consistency of the leather and rubber outsole; 2.2) Denoising: through non-local mean denoising (neighborhood window 7x7, h=12), the texture of the upper mesh is preserved.

[0065] Step 3, the adaptive view angle fusion feature extraction of Example 4 is as shown in Figure 2 The specific implementation process is as follows: 3.1) The backbone network (EfficientNet-B3) extracts multi-scale features, and the cross-view attention module (CVA) gives more weight to the front view, and highlights the texture of the upper mesh more; 3.2) Output fusion feature map The channel response intensity of the shoelace crossing area is effectively improved.

[0066] Step 4, strictly according to the lightweight dynamic sampling strategy for sampling, the implementation process is as follows: 4.1) Coarse sampling: within the set object bounding box, along each ray starting from the camera origin, 64 points are uniformly sampled; 4.2) Fine sampling: based on the gradient heat map G generated by the fusion feature map (threshold r=0.14), the sampling is encrypted by 4 times at the shoelace hole and mesh texture; 4.3) Hash coding: set the hash table level to 8.

[0067] Step 5, same as Example 1.

[0068] Step 6, same as Example 1.

[0069] Step 7, generate an initial mesh model by isosurface extraction, and then obtain a final optimized three-dimensional mesh model after multiple Laplace smoothing processing, the specific implementation process is as follows: 7.1) Isosurface extraction: use the Marching Cubes algorithm, set the degree threshold =0.5, extract the upper and shoelace mesh; 7.2) Laplace smoothing: iterate more than 3 times, and the overall sawtooth is significantly reduced.

[0070] The evaluation index adopted in Example 4 is the same as Example 1.

[0071] Table 4 shows the experimental results of the method of the present application and the existing technology NeRF and the existing technology Instant-NGP. As can be seen from Table 4, the PSNR value of the method of the present application is greater than that of the other two network results, and the SSIM value is closer to 1 than that of the other two network results, that is, the reconstruction effect of the method of the present application on the sports shoes is better.

[0072] Table 4, experimental results of Example 4 compared with NeRF and Instant-NGP

[0073] Example 5 The implementation object of this embodiment 5 is a metal key.

[0074] Reference Figure 1 , according to the adaptive view angle fusion and light dynamic sampling based on NeRF three-dimensional reconstruction method of the application, comprising the following steps: Step 1, a metal key is synchronously photographed by a calibrated multi-camera array (four views of front, back, left and right), ensuring that there is no occlusion between views and the light is uniform, and the specific implementation process is as follows: 1.1) When taking four view images, choose a time period with uniform light to take pictures, while keeping the distance between the cameras and the object in different views equal to ensure the consistency of the view; 1.2) Ensure that each view covers the key head, tooth groove and tail hole when shooting; 1.3) Use a ring-shaped LED and a diffuse reflection plate to eliminate metal highlights.

[0075] 1.4) Collect 400 groups of original images and store them according to the view angle.

[0076] Step 2, pre-process the obtained original images; 2.1) Color correction: the correction matrix K optimizes the color temperature of silver-white metal; 2.2) Denoising: through non-local mean denoising (neighborhood window 7x7, h=10), the fine texture of the tooth groove is preserved.

[0077] Step 3, the adaptive view angle fusion feature extraction of this embodiment 5 is as shown in Figure 2 , the specific implementation process is as follows: 3.1) The backbone network (EfficientNet-B3) extracts multi-scale features, and the cross-view attention module (CVA) gives more weight to the front view, highlighting the details of the tooth groove; 3.2) Output fusion feature map , the channel response intensity of the keyhole edge is effectively improved.

[0078] Step 4, strictly follow the light dynamic sampling strategy for sampling, the implementation process is as follows: 4.1) Coarse sampling: uniformly sample 64 points along each ray from the camera origin within the set object bounding box; 4.2) Fine sampling: generate a gradient heat map based on the fusion feature mapG (Threshold r = 0.11), 3 times of encryption sampling at the tooth groove and the circular hole; 4.3) Hash coding: set the hash table level to 7.

[0079] Step 5, same as Example 1.

[0080] Step 6, same as Example 1.

[0081] Step 7, generate an initial mesh model by isosurface extraction, and then obtain a final optimized three-dimensional mesh model after multiple Laplace smoothing treatments. The specific implementation process is as follows: 7.1) Isosurface extraction: use the Marching Cubes algorithm, set the degree threshold = 0.5, and extract the key whole shape mesh; 7.2) Laplace smoothing: iterate more than 3 times to effectively eliminate the sawtooth at the tooth groove.

[0082] The evaluation index used in this embodiment 5 is the same as that in Example 1.

[0083] Table 5 shows the experimental results of the method of the present application and the NeRF of the prior art and the Instant-NGP of the prior art. As can be seen from Table 5, the PSNR value of the method of the present application is greater than that of the other two network results, and the SSIM value is closer to 1 than that of the other two network results, i.e. the reconstruction effect of the method of the present application on the metal key object is better.

[0084] Table 5, comparison of experimental results of Example 5 with NeRF and Instant-NGP

[0085] Example 6 The implementation object of this embodiment 6 is a potted plant.

[0086] Referring to Figure 1 , the NeRF three-dimensional reconstruction method based on adaptive view angle fusion and lightweight dynamic sampling is implemented according to the present application, which includes the following steps: Step 1, synchronously shoot a potted plant through a calibrated multi-camera array (four views of front, back, left and right), ensure that there is no occlusion between views and the light is uniform, and the specific implementation process is as follows: 1.1) When shooting images of four views, select a time period with uniform light to shoot, and at the same time keep the distance between the cameras and the object in different views equal to ensure the consistency of the views; 1.2) Ensure that each view covers the leaves, stems and edges of the flowerpot when shooting; 1.3) Collect a total of 400 original images, and store them according to the view name.

[0087] Step 2, pre-process the acquired original image; 2.1) Color correction: the correction matrix K optimizes the color saturation of green leaves and brown soil; 2.2) Denoising: through non-local mean denoising (neighborhood window 7x7, h=13), the serrated edges of the leaves are preserved.

[0088] Step 3, the adaptive view angle fusion feature extraction of this embodiment 6 is as shown in Figure 2 , and the specific implementation process is as follows: 3.1) The backbone network (EfficientNet-B3) extracts multi-scale features, and the cross-view attention module (CVA) gives the same weight to each view angle, fully extracting the leaf layer structure in each direction; 3.2) Output fusion feature map , the channel response intensity of the pot edge is effectively improved.

[0089] Step 4, strictly follow the lightweight dynamic sampling strategy for sampling, the implementation process is as follows: 4.1) Coarse sampling: in the set object bounding box, along each ray starting from the camera origin, uniformly sample 64 points; 4.2) Fine sampling: based on the gradient heat map generated by the fusion feature map G (threshold r=0.13), 3 times of sampling are added at the leaf overlap; 4.3) Hash coding: set the hash table level to 8.

[0090] Step 5, same as embodiment 1.

[0091] Step 6, same as embodiment 1.

[0092] Step 7, generate an initial mesh model by isosurface extraction, and then after multiple Laplace smoothing processing, get the final optimized three-dimensional mesh model, the specific implementation process is as follows: 7.1) Isosurface extraction: use the Marching Cubes algorithm, set the degree threshold =0.5, extract the leaf, stem and pot mesh; 7.2) Laplace smoothing: iterate more than 3 times to effectively eliminate the serrations at the leaf overlap.

[0093] The evaluation index used in this embodiment 6 is the same as that in embodiment 1.

[0094] Table 6 shows the experimental results of the method of the application and the prior art NeRF and the prior art Instant-NGP. As can be seen from Table 6, the PSNR values of the method of the application are all greater than those of the other two networks, and the SSIM values are all closer to 1 than those of the other two networks, that is, the reconstruction effect of the method of the application on the metal shell printer is better.

[0095] Table 6, Example 6 and the experimental results of NeRF and Instant-NGP

[0096] In summary, the NeRF three-dimensional reconstruction method based on adaptive view angle fusion and lightweight dynamic sampling of the application can significantly improve the geometric consistency and accuracy of the three-dimensional model by shooting images from multiple view angles and using adaptive view angle fusion technology to extract image features from each view angle. At the same time, the lightweight dynamic sampling strategy greatly reduces the computational overhead and improves the reconstruction efficiency. The final generated three-dimensional mesh model performs well in overall integrity after optimization.

Claims

1. A 3D reconstruction method based on NeRF, characterized in that: Follow these steps to implement: Step 1: synchronously collect multi-view images; Step 2, preprocessing the original image; Step 3, extracting the adaptive perspective fusion features of the multi-view image preprocessed in step 2; Step 4: Generate three-dimensional rays and complete coordinate transformation; Step 5: Set up and implement a lightweight dynamic sampling strategy; Step 6: Calculate the density field and color field, and obtain the loss function value after completing volume rendering; Step 7: Generate a three-dimensional mesh model and complete the optimization of the three-dimensional mesh model.

2. The NeRF-based 3D reconstruction method according to claim 1, wherein: In step 1, the target object is captured synchronously using a pre-calibrated multi-camera array. The specific process is as follows: 1.1) Calibrate the camera parameters: Use the checkerboard calibration method to calibrate the intrinsic and extrinsic parameters of the four cameras to ensure spatial alignment of the images from each viewpoint. 1.2) Control the shooting environment: In a constant temperature and dust-free laboratory, use a ring-shaped LED light source to provide uniform lighting to avoid reflections or shadows; 1.3) Capture and store images: Trigger four cameras simultaneously to capture PNG format images. These images are saved by naming them according to the viewing angle to obtain the original images.

3. The NeRF-based 3D reconstruction method according to claim 1, wherein: In step 2, the specific process is: 2.1) Perform color correction on the original image, Place a standard whiteboard in the scene and optimize the color correction matrix K and bias vector B based on the white balance algorithm. The correction formula is: , in, Represents the rectified image matrix, with dimensions of H×W×3, where H is the image height, W is the width, and 3 represents the RGB channels; K is a 3×3 color correction matrix; is the original input image matrix, with the same dimension as same; B is the 3D bias vector, from which the corrected image is obtained; 2.2) Perform non-local mean denoising operation to remove noise from the rectified image. For each pixel in the rectified image ,First, define a local image block centered on it With a search window ; The local image block Also called neighborhood block; the search window Usually it needs to be much larger than the neighborhood block; then, calculate each pixel in the rectified image Neighborhood blocks and their search windows Other pixels in The similarity between the neighborhood blocks is used to assign weights to each pixel and use these weights to search the window. The pixel values ​​within are weighted averaged to estimate the final denoised pixel value, thereby removing noise and retaining image details. The denoising formula is: ， = ， in, Represents the pixel after denoising x The value of is the normalization factor, ensuring that the sum of the weights is 1, that is, = , Indicates that in the current search window The rest of the pixels y Relative to the current pixel x The allocation weight of Indicates x The neighborhood block centered on express y The neighborhood block centered on h is the smoothing parameter.

4. The NeRF-based 3D reconstruction method according to claim 1, wherein: In step 3, the specific process is: 3.1) Set up the backbone network and output shallow features. The lightweight and high-performance EfficientNet-B3 network is used as the backbone network. The EfficientNet-B3 network balances depth, width, and resolution through a composite scaling strategy. Specific configuration includes: In the input processing stage, all perspective images are uniformly scaled to 512×512 pixels to meet the network input requirements. In terms of hierarchical structure, shallow feature extraction is completed by the first three modules, namely Conv1, MBConv1, and MBConv2. These three modules use 3×3 convolution and depth-wise separable convolution to gradually extract low-level features of edges and textures. The MBConv1 module contains 16 channels, an expansion factor of 1, and an activation function of Swish. Deep semantic extraction is completed by the second two modules, namely MBConv3 and Conv5, through 5×5 convolution and channel attention mechanism to capture the global semantic information of the target object. The MBConv3 module outputs a 40-channel feature map, and the Conv5 layer increases the feature dimension to 256 through 1×1 convolution. In the output adjustment stage, the global pooling layer and classification head in the original EfficientNet-B3 network are removed, and the multi-scale feature maps are retained. MBConv2: 256×256×32, Conv5: 64×64×256, and shallow features are output. 3.2) Construct a cross-view attention module to obtain preliminary fusion features. The core structure of the cross-view attention module is divided into three parts: First, in the query-key generation layer, the 256-dimensional feature map of each view is passed through three independent linear layers to generate the query vector , key vector , value vector ; Secondly, the attention weight is calculated, the correlation between perspectives is calculated through dot product, and a scaling factor is introduced to prevent gradient disappearance. The calculation formula is: ， in, , indicating the viewing angle i Perspective j The attention weight of is in the range of [0,1]; Perspective i The query vector of dimension 32 is obtained by mapping the feature vector through a linear layer; Perspective j The key vector has a dimension of 32 and is generated in the same way as ; is the scaling factor; Finally, feature fusion is performed and weighted aggregation value vector is obtained. V , generate a preliminary fusion feature map, the calculation formula is: ； 3.3) Output the optimized fusion feature map, The initial fusion feature is reduced to 128 by a 1×1 convolution layer and concatenated with the shallow features obtained by the backbone network to enhance the detail retention capability, and finally the optimized fusion feature map is obtained. .

5. The NeRF-based 3D reconstruction method according to claim 1, wherein: In step 4, the specific process is: 4.1) Normalization of pixel coordinates. For each two-dimensional image at each viewing angle, the pixel coordinates ( u , v ) is converted to normalized device coordinates NDC. The conversion formula is as follows: ， in, W and H are the width and height of the image, respectively. The normalized device coordinates map pixel coordinates to the range [−1, 1] to facilitate subsequent three-dimensional space calculations; 4.2) Calculate the ray direction, In the camera coordinate system, the vector of the ray direction By normalized coordinates and OK, the calculation formula is as follows: ， in, n is the near plane distance, and is the focal length of the camera; 4.3) Convert to the world coordinate system. Use the camera's external parameters to convert the ray direction from the camera coordinate system to the world coordinate system. The conversion formula is as follows: ， in, is the ray direction in the world coordinate system, is the ray origin in the world coordinate system; rotation matrix R and translation vectors Respectively represent the position and posture of the camera in the world coordinate system; 4.4) Ray parameterization, The calculation formula for each ray is as follows: ， in, t is a parameter, indicating that the ray Offset in direction, parameter range Usually determined by bounding box estimation or preset values.

6. The NeRF-based 3D reconstruction method according to claim 1, wherein: In step 5, the specific process is: 5.1) Perform layered dynamic sampling, The two-stage sampling strategy significantly reduces the number of sampling points, as follows: 5.1.1) Implement coarse sampling: along each ray within the object bounding box 64 points are sampled evenly, and the number of sampling points can be adjusted appropriately according to different objects; 5.1.2) Determine the fine sampling area: First, from the fusion feature map Extract the spatial gradient from For each channel, the Sobel operator is used to calculate the horizontal gradient C x and vertical gradients C y , the calculation formula is as follows: ， in, = , = ; Secondly, for each channel, a corresponding gradient heat map is generated, which refers to The gradient amplitude map in the spatial dimension reflects the severity of the feature change. The gradient amplitude C The calculation formula is: , Next, multi-channel aggregation is performed to take the average or maximum value of the gradient amplitudes of multiple channels to generate the final multi-channel heat map G ; Finally, identify the high gradient area, set the threshold r, and The area of ​​r is identified as the high gradient area, that is, the area determined to be finely sampled; 5.1.3) Complete fine sampling: insert additional sampling points along the ray direction within the selected fine sampling area; 5.2) Use hash coding to speed up, Multi-resolution hash coding is used to map 3D coordinates to a compact hash table. The process is as follows: 5.2.1) Constructing a multi-resolution hash table: First, set up 8 layers of hash tables with different resolutions, according to Out of each level l Resolution; Then, hash code each sampled 3D coordinate point, at the level l quantize it to the nearest neighbor point and pass it through the hash function Mapped to a hash table location; finally, each hash table entry stores an 8-dimensional feature vector ∈ , optimize the feature value through training; 5.2.2) Perform feature interpolation and fusion: In each layer of the hash table, the feature values ​​of the 8 points around it are taken and the feature of the point is calculated by trilinear interpolation. , and then the features of the 8-layer hash table , ,..., Splicing to get 64-dimensional vector , as the input of the density prediction network branch in step 6; for the viewing direction ( ), then the 16-dimensional vector is obtained by spherical harmonic encoding , and is concatenated with the coordinate features as the input of the color prediction network branch in step 6.

7. The NeRF-based 3D reconstruction method according to claim 1, wherein: In step 6, the specific process is: 6.1) Construct a density prediction network branch to output the density corresponding to the coordinate point In this density prediction network branch, the input is the hash code of the 3D coordinates , the code is generated by the hash table in step 5; the density prediction network branch contains four fully connected layers; 6.2) Constructing the color prediction network branch, In the color prediction network branch, the intermediate features output from the density network branch are feature and viewing direction ( ) is a 16-dimensional vector obtained by spherical harmonic encoding Splicing is performed to obtain an input vector with a dimension of 80; the color prediction network branch contains four fully connected layers; 6.3) Implement volume rendering, The input of volume rendering includes the position of sampling points, density field, color field and camera parameters; the output of volume rendering is the rendered image ; 6.4) Neural Network Loss Function, Use the loss function used in the original NeRF network, that is, calculate the rendered image With real images The L1 error is calculated as follows: ， in, Represents the final loss function value, N is the total number of pixels, To render an image in pixels p The RGB value at For real images in pixels p The RGB value at For the L1 norm, calculate the absolute error.

8. The NeRF-based 3D reconstruction method according to claim 7, wherein: In step 6.3), the specific process is: 1) Ray generation: This process is consistent with the 3D ray generation in step 4, and the ray is obtained ; 2) Stratified sampling: This process is consistent with the stratified dynamic sampling in step 5; 3) Numerical integration: First, calculate the transmittance at each sampling point , the calculation formula is: ， in, is the distance between adjacent points, t i Represents rays A sampling point on That is, the density value obtained by the density prediction network branch corresponding to this point. Then, through the transmittance Calculate the color contribution of all sampling points along the entire ray and , the calculation formula is as follows: ， in, t i Represents rays A sampling point on Indicates the color value obtained by the color prediction network branch at this point; 4) Pixel color assignment: the integration result Assign values ​​to pixels The corresponding RGB value.

9. The NeRF-based 3D reconstruction method according to claim 1, wherein: In step 7, the specific process is: 7.1) Isosurface extraction: The Marching Cubes algorithm is used to extract isosurfaces. The Marching Cubes algorithm traverses all cubes in the 3D voxel grid and extracts the isosurfaces according to the density values ​​of the cube vertices. With the set density threshold The relationship between , the area above this value is determined as the surface of the object, and the position of the isosurface in the voxel cube is determined to generate the initial three-dimensional grid; 7.2) Optimization of the initial 3D mesh: The initial 3D mesh is smoothed using the Laplace smoothing algorithm. The Laplace smoothing algorithm iteratively calculates the Laplace value of each vertex, that is, the difference between the vertex and its neighboring vertices, to adjust the vertex position, thereby reducing the noise and jagged effect of the mesh. The displacement correction amount of the Laplace smoothing is The calculation formula is: , in, is the vertex i The displacement correction amount, is the vertex i The set of neighboring vertices of is the vertex i The three-dimensional coordinates of V j Vertex i The set of neighboring vertices N(i) The j The three-dimensional coordinates of the vertices; Then, according to the displacement correction amount To adjust the position of the vertex, the update formula is: ， After updating all vertices, an optimized three-dimensional mesh model is obtained.

Images