Method and system for 3d surface reconstruction and camera pose self-correction based on neural radiance fields

Abstract

This invention provides a method and system for 3D surface reconstruction and camera pose self-correction based on neural radiation fields. The invention employs a coarse-to-fine training strategy, applicable to NeRF-based methods, to predict camera pose and reconstruct 3D surfaces. Experiments on open-source datasets demonstrate that this invention exhibits stronger robustness to input data, achieving high-quality reconstruction even in complex scenes such as sparse view input, complex lighting environments, and 360-degree surround views. Compared to previous techniques, this invention typically retains more detail and achieves higher camera correction accuracy within the same timeframe.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional surface reconstruction technology, and particularly relates to a method and system for three-dimensional surface reconstruction and camera pose self-correction based on neural radiation fields. Background Technology

[0002] Existing 3D scene reconstruction techniques can be divided into two categories: traditional methods and neural network-based methods. Traditional methods mainly rely on techniques such as ray tracing and visual geometry. Ray tracing technology determines information such as objects and light sources in a scene by tracing rays in the scene and generating corresponding images. Its main advantage is that it can generate high-quality, high-fidelity images and can handle very complex scenes. However, ray tracing technology has a very high computational cost and high requirements for computing resources, making it unsuitable for applications with high real-time requirements. Visual geometry techniques include Structure from Motion (SfM) and Simultaneous Localization and Mapping (SLAM). These methods typically use multiple 2D images and reconstruct objects in the 3D scene by estimating the camera pose and the 3D geometric information of objects in the scene. COLMAP is a general-purpose SfM-based scene reconstruction and camera pose self-correction method. It reconstructs 3D scenes from disordered or ordered images and predicts camera pose based on feature matching. These methods typically require a large number of two-dimensional images and are highly sensitive to noise, occlusions, and changes in lighting in the images. They also struggle to capture complex surface details and transparent objects, and their accuracy and efficiency are easily affected by shooting conditions and scene structure.

[0003] In recent years, the most representative work in neural network-based 3D scene reconstruction technology has been the Neural Radiance Fields (NeRF) approach. NeRF is a deep learning-based 3D scene reconstruction method that implicitly models the reconstruction problem from a 2D image to a 3D scene as a 5D radiance field, and models the radiance function of each point in the scene as a neural network. Given the camera's position, orientation, and the corresponding 2D image, a ray is emitted from any pixel along the camera's pose direction into the scene. Sampling points are selected along the ray in 3D space, and the geometric density and color at that point are obtained through the radiance function represented by the neural network. The colors on the ray are then superimposed to achieve rendering of a single pixel. Repeating this process for all pixels in the image yields renderings from new perspectives. This method can learn information such as scene depth and color from the input image, thereby generating high-quality, high-precision 3D reconstruction results.

[0004] Existing NeRF-based 3D surface reconstruction methods add surface constraints, such as the signed distance function (SDF), to NeRF to help improve the quality of surface information from the implicit representation of NeRF and achieve more accurate surface reconstruction.

[0005] Currently, NeRF-based 3D reconstruction techniques typically require accurate camera pose information as input to render accurate pixel colors and 3D surfaces in the scene. However, obtaining accurate camera pose is often difficult in real-world applications. On one hand, camera calibration is usually accomplished by placing calibration objects (e.g., a checkerboard pattern) in the scene and using the known geometry of the calibration objects to estimate camera parameters. However, calibration objects are often difficult to obtain, especially with the increasing prevalence of smartphones and other camera devices, making image capture easier than obtaining camera pose. On the other hand, even if a suitable calibration object is selected, in real-world scenarios such as field deployment, it may interfere with the camera's perception task, resulting in noisy predicted camera poses, which will significantly affect the accuracy of the reconstruction.

[0006] Existing techniques have proposed improvements to NeRF, such as distinguishing foreground and background and using bundled adjustments, which can reconstruct 3D scenes from inaccurate camera poses while simultaneously correcting them. These techniques typically have the following drawbacks: 1) They usually require dense training inputs; 2) They usually require stable ambient lighting conditions, and the reconstruction quality is not high under complex lighting environments and complex scene geometry; 3) Most methods can only complete the reconstruction and self-correction tasks under conditions where the deviation between cameras is small, i.e., all images need to face the same direction of the scene, and they cannot complete the task under input conditions where the viewpoint is sparse and surrounds the scene 360 ​​degrees. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of NeRF-based 3D reconstruction methods by proposing a 3D surface reconstruction and camera pose self-correction method and system based on neural radiation fields.

[0008] The technical solution adopted in this invention is as follows:

[0009] A method for 3D surface reconstruction and camera pose self-correction based on neural radiation fields includes:

[0010] Obtain a set of multi-view views to be reconstructed;

[0011] Camera pose prediction is performed on multi-viewpoints to obtain the estimated camera pose for each view.

[0012] The multi-view view is processed to a higher level of detail. The processed multi-view view and camera pose are then input into the NeRF network for iterative training. The NeRF network parameters and camera pose are optimized by minimizing the loss function until the loss function is minimized or the preset number of iterations is reached. Finally, the NeRF network outputs the 3D surface reconstruction result of the multi-view view.

[0013] In each iteration of training, the multi-viewpoint is processed with varying degrees of fineness based on the magnitude of the loss function, as shown below:

[0014] P n =α1·P n-1 +α2·Loss n-1

[0015] Where P represents the adjustment parameter for the fineness of multi-view processing, n represents the number of training iterations, Loss represents the loss function of the NeRF network, α is a hyperparameter fixed during training, and the initial value of the camera pose is the estimated camera pose.

[0016] The loss function includes the loss between the volume-rendered synthetic image and the real image of the 3D surface reconstruction result based on the multi-view output of the NeRF network.

[0017] This invention employs a coarse-to-fine training strategy, which is applicable to NeRF-based methods to predict camera pose and reconstruct 3D surfaces.

[0018] Furthermore, since images from adjacent viewpoints have strong correlation and matching properties, this invention proposes to obtain the matching relationship of feature points between adjacent viewpoint images based on image feature extraction and feature matching methods, and to optimize the training behavior of the MLP network by combining prior information from feature point matching. Specifically, the loss function also includes a Gaussian mixture distribution intersection-union ratio loss for two rays with triangular correspondence, expressed as follows:

[0019]

[0020]

[0021] Where V is the set of three-dimensional points obtained by discretization sampling in three-dimensional space; v represents the coordinates of a point in three-dimensional space; S i Let S represent the three-dimensional coordinates of the i-th sampling point on the camera ray, i.e., the mean of the Gaussian function. i ∈S, where S is the set of sample points input to the NeRF network in volume rendering; σ MoG w represents the standard deviation of a Gaussian distribution. irepresents the weight corresponding to the i-th sampling point, and the value is the volume density value estimated by the NeRF network corresponding to the i-th sampling point; n represents the number of sampling points; MoG1(v) and MoG2(v) are the Gaussian mixture distributions of two rays with triangular correspondence; the origin of the two rays with triangular correspondence is the feature point pair obtained based on feature point extraction and matching of multi-view views.

[0022] Furthermore, the loss between the volume rendering synthesized image and the real image of the 3D surface reconstruction result based on the multi-view output of the NeRF network is... img It is expressed as follows:

[0023]

[0024] Where R represents the camera ray set, This represents the actual color value of each pixel in a real multi-view. The volume rendering synthesized image represents the predicted color value of each pixel in the 3D surface reconstruction result of the multi-view view output by a NeRF-based coarse network on sparse sampling. The volume rendering synthesized image represents the predicted color value of each pixel in the multi-view 3D surface reconstruction result of the NeRF-based fine network output on fine sampling.

[0025] Furthermore, in the camera ray set, each camera ray is represented as: r(t) = o + td, where o is the ray origin, o ∈ O', O' is the set of camera ray origin points O', d represents the ray direction calculated from the camera parameters, t represents the distance between the sampling point on the ray and the ray origin, and the set of camera ray origin points O' includes the set of pixels sampled on the image and the subset K' of sampling points obtained from feature point extraction and matching based on multi-view views.

[0026] Furthermore, the method for extracting and matching the feature point pairs is as follows:

[0027] The trained image matching model is used to extract the visual features of each image from the multi-view view and map them into feature vectors. The similarity between images is measured by calculating the Euclidean distance or cosine similarity between the feature vectors of different images. Images with high similarity are recorded as image pair sequences. Feature point extraction and matching algorithms are applied to the image pairs in the image pair sequence to obtain matching feature point pairs.

[0028] Furthermore, the feature point extraction algorithm is the SIFT algorithm or the SuperPoint method based on neural networks; the matching algorithm is the KNN algorithm or the SuperGlue method based on neural networks.

[0029] Furthermore, the method for refining the multi-view image is as follows: after performing Gaussian filtering on the original view, downsampling is performed, and then interpolation is used to obtain the original size image. The refinement of the generated image is adjusted by adjusting the parameters of the Gaussian kernel and adjusting the downsampling ratio accordingly based on the Gaussian kernel parameters.

[0030] Alternatively, an image generation function can be implicitly modeled using a convolutional neural network with the same input and output dimensions. The convolutional neural network and the NeRF network are trained synchronously and iteratively, and the image generation effect is supervised by the original view. As the training iterations increase, a gradually refined image is obtained.

[0031] Furthermore, the NeRF network has an MLP structure.

[0032] Furthermore, the MLP network includes a position coding layer, the coding function of which is expressed as follows:

[0033] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp))

[0034] Where L is the dimension parameter, and p represents the input variable to be encoded, including the three-dimensional coordinate X and the two-dimensional ray direction d. In practice, different L values ​​are specified for X and d to obtain different encoding functions γ(X) and γ(d), and position encoding is performed accordingly.

[0035] A three-dimensional surface reconstruction and camera pose self-correction system based on neural radiation fields includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned three-dimensional surface reconstruction and camera pose self-correction method based on neural radiation fields.

[0036] The beneficial effects of this invention are:

[0037] 1. Existing technologies for achieving high-quality 3D scene reconstruction typically require accurate camera pose as training input. This invention achieves camera correction while modeling the scene, thus requiring lower accuracy in camera pose input.

[0038] 2. Existing technologies that simultaneously achieve camera pose correction and 3D scene reconstruction typically require dense view input, stable ambient lighting, and small deviations between adjacent cameras. This invention has stronger robustness to input data and can achieve high-quality reconstruction even in complex scenes such as sparse view input, complex lighting environments, and 360-degree surround views.

[0039] 3. Compared with existing technologies, the reconstruction results of this invention can usually retain more details and have higher camera correction accuracy in the same amount of time. Attached Figure Description

[0040] Figure 1 This is a flowchart of Example 1;

[0041] Figure 2 This is a flowchart of Example 2;

[0042] Figure 3 It is a multi-view view under poor lighting conditions (significant changes in lighting);

[0043] Figure 4 for Figure 3 The results of the reconstruction of the multi-view view are shown in the following columns: the left column is the real model, the middle column is the reconstruction result of the existing NeRF method, and the right column is the reconstruction result using the method of Example 3. Detailed Implementation

[0044] The present invention will now be further described with reference to specific embodiments.

[0045] Example 1:

[0046] This invention discloses a method for 3D surface reconstruction and camera pose self-correction based on neural radiation fields. Employing a coarse-to-fine training strategy, it is applicable to NeRF-based methods, achieving the goal of predicting camera pose and reconstructing 3D surfaces. The method's flow is as follows: Figure 1 As shown, the specific steps include:

[0047] Step 1: Obtain a set of multi-view views to be reconstructed; these multi-view views can be sparse viewpoints, complex lighting environments, 360-degree surround views, etc.

[0048] Step 2: Perform camera pose prediction on the multi-viewpoints to obtain the estimated camera pose for each view. In this step, traditional camera pose prediction methods such as COLMAP (Schonberger, Johannes L., and Jan-Michael Frahm. "Structure-from-motion revisited." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016), SfM, etc., can be used to make a preliminary prediction of the camera pose and obtain a coarse camera pose. This step allows for errors in the coarse camera pose.

[0049] Step 3: The multi-view images are then refined to a finer level. The refined multi-view images and camera poses are then input into the NeRF network for iterative training. The NeRF network parameters and camera poses are optimized by minimizing the loss function until the loss function is minimized or a preset number of iterations is reached. Finally, the NeRF network outputs the 3D surface reconstruction results of the multi-view images. This invention proposes a coarse-to-fine image processing strategy for camera pose optimization. Unlike existing methods that directly input the original view into the NeRF network for training, this invention processes the original view before input. As the number of iterations in the NeRF network training increases, coarse to fine processed images are obtained sequentially from the original view. These processed images are then used as input to the NeRF network. The coarse image only needs to include the general features of the current view (outline, color, etc.), and more detailed features are gradually added with each iteration. Compared to direct input, the coarse-to-fine image processing strategy helps improve camera correction accuracy, thereby improving the accuracy of scene reconstruction. Specifically, in each iteration, the multi-view images are refined according to the magnitude of the loss function, as shown below:

[0050] P n =α1·P n-1 +α2·Loss n-1

[0051] Where P represents the adjustment parameter for the fineness of multi-view processing, n represents the number of training iterations, Loss represents the loss function of the NeRF network, α is a hyperparameter fixed during training, and the initial value of the camera pose is the estimated camera pose.

[0052] The image detail processing methods available here include: 1) performing Gaussian filtering on the original view, then downsampling, and then interpolating (nearest neighbor interpolation is used in the experiment, but other interpolation schemes are also available) to obtain the original size image. The detail of the generated image can be adjusted by adjusting the parameters of the Gaussian kernel and the downsampling ratio corresponding to the Gaussian kernel parameters; 2) using a lightweight convolutional neural network (CNN) with the same input and output sizes to implicitly model the image generation function. The CNN and the NeRF network are trained synchronously, and the image generation effect is supervised by the original view. As the training iterations increase, gradually refined images will be obtained, and gradually refined 3D scenes will be obtained from the NeRF network.

[0053] Taking the image processing method based on Gaussian kernel filtering and downsampling as an example, the image refinement process is as follows:

[0054] Gaussian kernel filtering is a common image processing technique designed to smooth images and reduce noise. The Gaussian kernel function is typically represented as a two-dimensional Gaussian distribution function, which includes two parameters: the center point and the variance.

[0055]

[0056] Where G(x,y) represents the value of the Gaussian function at the point (x,y), x0 and y0 represent the center point of the Gaussian kernel, and σ represents the standard deviation of the Gaussian kernel function.

[0057] When performing Gaussian kernel filtering on an image, a Gaussian kernel function is typically used to convolve each pixel in the image. The formula is as follows:

[0058]

[0059] Among them I in (x,y) represents the pixel value at point (x,y) in the input image. out (x,y) represents the pixel value at (x,y) in the output image, G(i,j) represents the value of the Gaussian kernel function at (i,j), which is used as a weight in the formula, k represents the size of the Gaussian kernel, and K represents the sum of the weights, used for normalization to ensure that the filtered pixel value is within the range of the original pixel value. For the same input image, the larger the standard deviation of the Gaussian kernel function used for filtering (i.e., the parameter σ in the above formula), the blurrier (coarser) the output image; the smaller the standard deviation of the Gaussian kernel function, the closer the output image is to the real image.

[0060] The standard deviation of the Gaussian kernel function is the adjustment parameter P for the level of detail processing of the multi-view view. By recursively updating the standard deviation of the Gaussian kernel function based on the magnitude of the loss function, the level of detail processing of the multi-view view can be controlled, as shown below:

[0061] σ n =α1·σ n-1 +α2·Loss n-1

[0062] Where σ represents the standard deviation of the Gaussian kernel function, n represents the number of iterations, L represents the training loss of NeRF, and α is a fixed hyperparameter during training (α1<1), used to adjust the influence of the training loss on the level of refinement and the adjustment magnitude of σ with iteration. During the iteration process, σ has a fixed decreasing trend, and the magnitude of the decrease is adjusted according to the performance of NeRF (i.e., the training loss). When NeRF performs well (the training loss is small), σ will decrease in an exponential function form close to α1; when NeRF performs poorly (the training loss is large), the decrease magnitude of σ will be slowed down, allowing the training to converge better and achieving the goal of adaptive refinement.

[0063] The aforementioned 3D surface reconstruction method based on neural radiation fields models a continuous scene as an implicit function, denoted as F, using a NeRF network. ΘThe function :(X,d)→(c,δ) takes a five-dimensional vector as input, including three-dimensional spatial coordinates X and two-dimensional view direction d, where the two-dimensional view direction d is obtained based on the camera pose; the output is three-dimensional RGB color information c and one-dimensional volume density δ. Generally, this function is approximated by continuous optimization using a multilayer perceptron (MLP). The parameters of the MLP and the camera pose are optimized by minimizing the loss function.

[0064] The loss function includes the loss between the volume-rendered synthetic image and the real image of the 3D surface reconstruction result based on the multi-view output of the NeRF network.

[0065] Existing methods synthesize and render images using volume rendering. Volume rendering is a 2D projection technique used to display discrete 3D sampled datasets. It renders a 2D projection of the 3D dataset based on the camera's spatial position relative to the geometry, the color of voxels in space, and the volume density. The camera ray is denoted as r(t) = o + td, where o is the origin of the ray, d represents the ray direction calculated from the camera parameters (i.e., the 2D viewpoint direction), t represents the distance between the sampling point on the ray and the origin, and the boundaries between the near and far points of the ray are denoted as t0. n and t f Then, the predicted value C(r) of the color of each pixel in the volume rendering synthetic image of the 3D surface reconstruction result based on the multi-view output of the NeRF network can be expressed as:

[0066] in

[0067] Where T(t) represents the ray from t n The cumulative volume density up to position t, δ(r(t)) represents the volume density of the ray at position t, and c(r(t),d) represents the RGB color of the ray at position t. In the implementation of the existing method, t... n to t f Divide the system into N uniform intervals, randomly select a sample point from each interval, and discretize the integral into a summation form:

[0068] in

[0069] Where, Δ i =t i+1 -t i , where represents the distance between adjacent sampling points, δ represents the volume density at the sampling point, and c represents the RGB color at the sampling point. This function uses the three-dimensional RGB color information c and the one-dimensional volume density δ predicted by the NeRF network to differentially calculate the predicted value of the image pixel color.

[0070] Furthermore, a hierarchical sampling strategy can be adopted. To achieve hierarchical sampling, existing methods simultaneously optimize two MLPs (referred to as coarse and fine networks, respectively). First, a first set of sparse, uniform random samples is performed along the ray as sparse sampling. The coarse network is used to estimate the sparse sample points, and the probability density function of the volume density along the ray direction is obtained based on the estimation result (i.e., the output of the coarse network), roughly revealing the volume density distribution. Then, a second set of fine sampling points is obtained by inverse transformation of the probability density function, where samples are more densely distributed in areas with higher volume density. The fine network is used to estimate all sample points from both the coarse and fine sampling, and the output of the fine network is calculated as the volume rendering result. Using mean squared error, the loss between the volume rendering synthesized image and the real image based on the multi-view 3D surface reconstruction results from the NeRF network output is expressed as follows:

[0071]

[0072] Where R represents the camera ray set, This represents the actual color value of each pixel in a real multi-view view. This represents the predicted color value of each pixel in the volume-rendered composite image of the 3D surface reconstruction results from a multi-view view output by a NeRF-based coarse network on coarse sampling. The volume rendering synthesized image represents the predicted color value of each pixel in the multi-view 3D surface reconstruction result of the NeRF-based fine network output on fine sampling. This represents the mean square error.

[0073] Example 2:

[0074] As an optional implementation, this invention designs another loss term based on the triangulation relationship between point pairs in the image and voxels in 3D space during 3D reconstruction. That is, for image pairs with feature point matching relationships, any matching feature point pair should correspond to the same coordinate in 3D space. In other words, two camera rays with these two points as the origin of the rays and along the corresponding image camera direction should intersect at the actual location of the surface point in 3D space.

[0075] Specifically, the method flow is as follows: Figure 2 As shown, the specific steps include:

[0076] Step 1: Obtain a set of multi-view views to be reconstructed; these multi-view views can be sparse viewpoints, complex lighting environments, 360-degree surround views, etc.

[0077] Step 2: Perform camera pose prediction and feature point extraction and matching on the multi-view views to obtain the estimated camera pose and feature point pair set for each view.

[0078] The specific method for feature point extraction and matching is as follows: the trained image matching model is used to extract the visual features of each image in the multi-view view and map them into feature vectors. The similarity between images is measured by calculating the Euclidean distance or cosine similarity between the feature vectors of different images. Images with high similarity are recorded as image pair sequences. The feature point matching algorithm is applied to the image pairs in the image pair sequence to obtain the matched feature point pairs.

[0079] The feature point extraction and matching methods used in this invention have several options. Feature point extraction methods can employ traditional SIFT, or the neural network-based SuperPoint method (DeTone, Daniel, Tomasz Malisiewicz, and Andrew Rabinovich. "Superpoint: Self-supervised interest point detection and description." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2018), etc. Feature point matching methods can employ traditional KNN, or the neural network-based SuperGlue method (Sarlin, Paul-Edouard, et al. "Superglue: Learning feature matching with graph neural networks." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020), etc. In the experiments conducted in this invention, both SuperPoint and SuperGlue methods were used. Since the results of feature point extraction and matching affect the reconstruction effect, the most suitable method can be adopted in practical applications based on the actual situation of the input image.

[0080] Step 3: Refine the multi-view view, then input the refined multi-view view and camera pose into the NeRF network for iterative training. Optimize the NeRF network parameters and camera pose by minimizing the loss function until the loss function is minimized or the preset number of iterations is reached. Finally, the NeRF network outputs the 3D surface reconstruction result of the multi-view view.

[0081] The loss function includes the cross-union ratio (CUI) loss of a Gaussian mixture of two rays with a triangular correspondence, as detailed below:

[0082] Loss = λ·Loss img +(1-λ)·Loss IoU

[0083]

[0084] Where λ is a hyperparameter set during training, used to adjust the weights of the two loss terms, and Loss... IoU This represents the cross-union ratio (CUI) loss of the Gaussian mixture distribution for two rays with a triangular correspondence. For each camera ray, based on several sampling points obtained from volume rendering and the volume density value output by the NeRF network, the Gaussian mixture distribution on that ray can be calculated in 3D space. Each sampling point in the sampling point set S input to the refined NeRF network during volume rendering is used as the mean of each Gaussian distribution, and the corresponding volume density value δ estimated by the NeRF network is used as the weight w of each Gaussian distribution. Considering image conditions and computational resources, the top n sampling points with the highest volume density values ​​are selected to calculate the Gaussian mixture distribution corresponding to the camera ray. The calculation process requires discretization of the 3D space. The formula for calculating the Gaussian distribution after discretization is as follows:

[0085]

[0086] Where v represents the coordinates of a point in three-dimensional space, S i S represents the three-dimensional coordinates of the i-th sampling point on the camera ray. i ∈S, that is, the mean of the Gaussian function, σ MoG w represents the standard deviation of a Gaussian distribution. i Let represent the weight corresponding to the i-th sampling point, and n represent the number of sampling points. In the implementation, considering both image conditions and computational resources, the top n sampling points with the highest volume density values ​​are selected, and the Gaussian mixture distribution corresponding to the camera ray is calculated.

[0087] Considering that as the number of iterations increases, the surface reconstruction results will gradually approach the true value, i.e., the Gaussian mixture distribution will gradually become more concentrated, and the visualized surface will become steeper. To make the discretized Gaussian distribution closer to the true distribution while reducing computational overhead and improving computational efficiency, this invention proposes a scheme to automatically adjust the discretization resolution. Uniform discretization is performed only within the three-dimensional coordinate range of the top n sampling points with the highest volume density values, and the results are included in the MoG, ignoring values ​​outside the range. Thanks to the layered sampling strategy in existing methods, as the number of iterations increases, the sampling points obtained by fine sampling will become increasingly densely distributed around the volume density peak (i.e., around the coordinates of the true surface in three-dimensional space). Therefore, the discretization range will increasingly converge towards the peak, and values ​​outside the range will increasingly converge towards 0. The automatic resolution adjustment scheme proposed in this invention can help the discretized calculation results maximize the fit to the distribution function, retain peak information as much as possible, and avoid additional computational overhead with the increase of the number of iterations.

[0088] For two rays with a triangular correspondence, i.e., two rays whose origin is a pair of feature points obtained by feature point extraction and matching from multiple viewpoints, there are two corresponding Gaussian mixture distributions, MoG1 and MoG2. The intersection-over-union (IoU) of these two Gaussian mixture distributions in 3D space is the loss term for the intersection-over-union of the Gaussian mixture distributions of the two rays with a triangular correspondence.

[0089]

[0090] Where V is the set of three-dimensional points obtained by discretization sampling in three-dimensional space.

[0091] Example 3:

[0092] As an optional implementation, existing methods randomly sample pixels on the image as the camera ray origin point set O. This embodiment, based on Embodiment 2, calculates the loss... img A subset K', randomly sampled from the feature point set K obtained by the feature point extraction algorithm, is added to the algorithm. O and K' are then merged to form the camera ray origin set O'. This design allows NeRF to improve the rendering performance of feature points while uniformly improving the performance of all pixels in each training iteration, thus enhancing the scene rendering effect in details. Figure 3 Table 1 shows multi-view images under poor lighting conditions (significant changes in lighting). Figure 4 Based on the reconstruction results using the method of Example 3, it can be seen that the reconstruction results of the present invention can generally retain more details and have higher camera correction accuracy within the same time period.

[0093] Table 1. Camera pose correction effect in Example 3

[0094]

[0095] Example 4:

[0096] As an optional implementation, this embodiment, based on Embodiment 1, utilizes a high-frequency function to map the input coordinates to a high-dimensional space to enable the MLP to better fit data containing high-frequency variations (such as color, geometry, etc.). The encoding function is as follows:

[0097] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp))

[0098] Where L is the dimension parameter, and p represents the input variable to be encoded, including the three-dimensional coordinate X and the two-dimensional ray direction d. In practice, different L values ​​are specified for X and d to obtain different encoding functions γ(X) and γ(d), and position encoding is performed accordingly.

[0099] Example 5:

[0100] Corresponding to the aforementioned embodiment of a three-dimensional surface reconstruction and camera pose self-correction method based on neural radiation fields, the present invention also provides an embodiment of a three-dimensional surface reconstruction and camera pose self-correction system based on neural radiation fields.

[0101] The present invention provides a three-dimensional surface reconstruction and camera pose self-correction system based on neural radiation field, comprising one or more processors for implementing a three-dimensional surface reconstruction and camera pose self-correction method based on neural radiation field as described in the above embodiments.

[0102] An embodiment of the three-dimensional surface reconstruction and camera pose self-correction system based on neural radiation field of the present invention can be applied to any device with data processing capability, such as a computer or other equipment or device.

[0103] The system implementation can be achieved through software, hardware, or a combination of both. Taking software implementation as an example, as a logical system, it is formed by the processor of any data-processing device that hosts the system reading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, the hardware structure of any data-processing device hosting the 3D surface reconstruction and camera pose self-correction system based on neural radiation fields of this invention includes at least a processor, memory, a network interface, and non-volatile memory. In addition, the data-processing device hosting the system in the embodiment may also include other hardware depending on the actual function of that data-processing device, which will not be elaborated further.

[0104] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0105] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for three-dimensional surface reconstruction and camera pose self-correction based on neural radiation fields, characterized in that, include: Obtain a set of multi-view views to be reconstructed; Camera pose prediction is performed on multi-viewpoints to obtain the estimated camera pose for each view. The multi-view view is processed to a higher level of detail. The processed multi-view view and camera pose are then input into the NeRF network for iterative training. The NeRF network parameters and camera pose are optimized by minimizing the loss function until the loss function is minimized or the preset number of iterations is reached. Finally, the NeRF network outputs the 3D surface reconstruction result of the multi-view view. In each iteration of training, the multi-viewpoint is processed with varying degrees of fineness based on the magnitude of the loss function, as shown below: in, This represents an adjustment parameter for the level of detail in processing multi-viewpoints, where n represents the number of training iterations. This represents the loss function of the NeRF network. These are hyperparameters fixed during training; the initial value of the camera pose is the estimated camera pose. The loss function includes the loss between the volume rendering synthetic image of the 3D surface reconstruction results based on the multi-view output of the NeRF network, the loss between the real image and the Gaussian mixture cross-union ratio loss of two rays with triangular correspondence; the Gaussian mixture cross-union ratio loss of two rays with triangular correspondence is expressed as follows: Where V is the set of three-dimensional points obtained by discretization sampling in three-dimensional space; v represents the coordinates of a point in three-dimensional space; Let represent the three-dimensional coordinates of the i-th sampling point on the camera ray, i.e., the mean of the Gaussian function. S is the set of sample points input to the NeRF network during volume rendering; This represents the standard deviation of a Gaussian distribution. The weight corresponding to the i-th sampling point is represented by the volume density value estimated by the NeRF network corresponding to the i-th sampling point; n represents the number of sampling points. and These are Gaussian mixtures of two rays with triangular correspondence; the origin of the two rays with triangular correspondence is the feature point pair obtained by feature point extraction and matching from multiple viewpoints.

2. The method according to claim 1, characterized in that, The loss between the volume rendering synthesized image and the real image in the 3D surface reconstruction result based on the multi-view view output of the NeRF network. It is expressed as follows: Where R represents the camera ray set, This represents the actual color value of each pixel in a real multi-view view. The volume rendering synthesized image represents the predicted color value of each pixel in the 3D surface reconstruction result of the multi-view view output by a NeRF-based coarse network on sparse sampling. The volume rendering synthesized image represents the predicted color value of each pixel in the multi-view 3D surface reconstruction result of the NeRF-based fine network output on fine sampling.

3. The method according to claim 2, characterized in that, In the camera ray cluster, each camera ray is represented as: , where o is the origin of the ray, o∈O', O' is the set of camera ray origin points O', d represents the ray direction calculated from the camera parameters, t represents the distance between the sampled point on the ray and the ray origin, and the set of camera ray origin points O' includes the set of pixels sampled on the image and the subset K' sampled from feature point pairs obtained by feature point extraction and matching of multi-view views.

4. The method according to claim 1 or 3, characterized in that, The specific method for extracting and matching the feature point pairs is as follows: The trained image matching model is used to extract the visual features of each image from the multi-view view and map them into feature vectors. The similarity between images is measured by calculating the Euclidean distance or cosine similarity between the feature vectors of different images. Images with high similarity are recorded as image pair sequences. Feature point extraction and matching algorithms are applied to the image pairs in the image pair sequence to obtain matching feature point pairs.

5. The method according to claim 4, characterized in that, The feature point extraction algorithm is the SIFT algorithm and the SuperPoint method based on neural networks; the matching algorithm is the KNN algorithm and the SuperGlue method based on neural networks.

6. The method according to claim 1, characterized in that, The method for refining the multi-view image is as follows: after performing Gaussian filtering on the original view, downsampling is performed, and then interpolation is used to obtain the original size image. The refinement of the generated image is adjusted by adjusting the parameters of the Gaussian kernel and adjusting the downsampling ratio accordingly based on the Gaussian kernel parameters. Alternatively, an image generation function can be implicitly modeled using a convolutional neural network with the same input and output dimensions. The convolutional neural network and the NeRF network can be trained iteratively and synchronously, and the original view can be used to supervise the image generation effect. As the training iterations increase, a gradually refined image can be obtained.

7. The method according to claim 1, characterized in that, The NeRF network structure is MLP.

8. The method according to claim 7, characterized in that, The MLP network includes a positional coding layer, and the coding function of the positional coding layer is represented as follows: Where L is the dimension parameter. This represents the input variable to be encoded, which is either a three-dimensional coordinate X or a two-dimensional ray direction d.

9. A three-dimensional surface reconstruction and camera pose self-correction system based on neural radiation fields, characterized in that, The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements a method for three-dimensional surface reconstruction and camera pose self-correction based on neural radiation fields as described in any one of claims 1-8.

Images