Nerve radiation field-based three-dimensional reconstruction method and equipment applying same

Abstract

The invention discloses a three-dimensional reconstruction method based on a neural radiation field, and the method comprises the steps: carrying out the motion structure recovery of an input image, carrying out the sampling of light emitted by a camera, and obtaining the color residual error of the image, and the number of features, matched with the feature points of a sparse initial point cloud, in the image; if the number of the image matching features is less than a first threshold value and the color residual error is greater than a second threshold value, marking the image as a strong noise image; otherwise, marking the image as a weak noise image; when the proportion of the strong noise image exceeds a third threshold value, the weight of the strong noise image is set to be zero in the loss function, training of the neural radiation field network is carried out, and the color on the observation view angle with the weight being zero is updated; and when the proportion of the strong noise image does not exceed a third threshold value, setting the loss function as a robust self-adaptive loss function, and performing training of the neural radiation field network to realize updating of the color on the corresponding view angle. According to the method disclosed by the invention, the visualization effect of rendering is improved while the additional calculation amount is hardly increased.

Description

Technical Field

[0001] The present invention relates to a three-dimensional reconstruction method based on a neural radiation field and a device applying the method, belonging to the technical field of computer vision. Background Art

[0002] Understanding the 3D world from a 2D scene is a fundamental problem in computer vision, and Neural Radiance Fields (NeRF) excels at synthesizing new views given multiple views of a static scene. However, when the scene contains distractors that are not persistent or displaced in the acquired 2D image, such as moving objects, shading changes, and shadows, artifacts appear as view-dependent effects or "floaters."

[0003] Furthermore, high-quality 3D reconstruction requires dense viewpoint data as a training foundation. However, while building our own dataset, we found that for most real-world applications, particularly emergency response scenarios and information collection tasks such as building rescue and repair, acquiring dense scene views is not only costly but also time-consuming to render. Some terrains, such as rainforests, mountains, riverbanks, and hills, are difficult to access from certain angles. Even using drones, they are subject to interference from flora and fauna, no-fly zones, and signal issues. Synthesizing new viewpoints from a limited number of 2D sparse viewpoint images is an extremely challenging task. This stems from the fact that these sparse viewpoints provide insufficient constraints for 3D reconstruction. Directly applying NeRF to such sparse viewpoint scenes often results in significant performance degradation. Firstly, volume rendering methods do not correctly constrain the scene geometry to consist of empty space and opaque surfaces. Secondly, given the extremely limited number of input views, the appearance and geometry of the reconstructed scene often exhibit a high degree of uncertainty. While these methods may learn renderings that closely match the training views, they are essentially overfitting the model to the training data.

[0004] Therefore, it is necessary to conduct more in-depth research on existing 3D reconstruction methods to solve the above problems. Summary of the Invention

[0005] To overcome the above problems, we conducted in-depth research and proposed a 3D reconstruction method based on neural radiation field, which includes the following steps:

[0006] S1. Perform motion structure recovery on the input image to obtain the sparse point cloud of the scene and the camera pose information corresponding to the input image, and sample on the light rays from the camera;

[0007] S2. Render the color of the two-dimensional pixel corresponding to the light based on the sampled data, and obtain the color residual of the image based on the rendered pixel color and the color in the original image;

[0008] Get the number of features in the image that match the feature points of the sparse initial point cloud;

[0009] S3. If the number of image matching features is less than the first threshold and the color residual is greater than the second threshold, extract the image and mark the image as a strong noise image;

[0010] Otherwise, the image is marked as a weak noise image;

[0011] S4. When the proportion of strong noise images exceeds a third threshold, the weight of the strong noise images is set to zero in the loss function, the neural radiation field network is trained, and the color at the observation angle with the weight of zero is updated;

[0012] When the proportion of strong noise images does not exceed the third threshold, the loss function is set to a robust adaptive loss function, and the neural radiation field network is trained to achieve color updates at the corresponding viewing angle;

[0013] S5. Use the trained neural radiation field network to recognize the input image and obtain the color image of the object under different viewing angles.

[0014] In a preferred embodiment, before S1, there is a step S0 of preprocessing the input:

[0015] If the input is an image, no preprocessing is required;

[0016] If the input is in video format, split the video into image frames.

[0017] In a preferred embodiment, in S4, when the proportion of strong noise images exceeds the third threshold, the loss function Set to:

[0018]

[0019] Among them, r represents any ray starting from the camera, each ray corresponds to a pixel in the image, R represents the set of r, C(r) represents the real pixel color value corresponding to the ray r, represents the rendered color predicted along ray r by coarse sampling, represents the rendered color predicted along ray r by fine sampling, represents the square L2 norm, ω represents the binarized weight function, I N Represents the current input image, S CF Represents a collection of strongly noisy images.

[0020] In a preferred embodiment, in S4, when the proportion of strong noise images does not exceed the third threshold, the loss function f(x, α, c) is set to:

[0021]

[0022] Among them, α is a hyperparameter, x represents the sum of the pixel losses in a single image, and c is a hyperparameter.

[0023] In a preferred embodiment, during the training process, the value of α continuously changes from a larger value to a smaller value.

[0024] In a preferred embodiment, depth information of the image is also obtained in step S0, and the depth information is used as a supervisory signal for training the neural radiation field network in S4.

[0025] In a preferred embodiment, in S2, for feature points in the image that are not matched with the sparse initial point cloud, their depth information is combined to guide sampling.

[0026] The present invention also discloses a three-dimensional reconstruction device based on neural radiation field, which is used to collect input images of objects to be three-dimensionally reconstructed, including a turntable and a camera arm.

[0027] The turntable is a rotatable platform for holding samples to be reconstructed;

[0028] The camera arm is provided with a plurality of cameras, which are fixed on the side of the turntable and are used to shoot samples on the turntable.

[0029] The present invention also provides an electronic device, comprising:

[0030] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the above methods.

[0031] The present invention also provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any one of the above methods.

[0032] The beneficial effects of the present invention include:

[0033] (1) The number of feature points matched to the 3D point cloud and the color residual are used as joint constraints, and the interference of transient objects is removed by weight update, which improves the rendering visualization effect while almost increasing the amount of additional calculations;

[0034] (2) Through multi-branch optimization and loss update methods, the performance can be maintained in the presence or absence of interference. Even in the absence of interference, not only does the performance not decline, but the rendering quality can also be partially improved.

[0035] (3) Utilize the depth prior to simultaneously constrain its loss and sampling space, thereby improving the rendering quality of the algorithm in sparse 3D reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic flow chart of a three-dimensional reconstruction method based on neural radiation fields according to a preferred embodiment of the present invention;

[0037] Figure 2 A schematic structural diagram of a three-dimensional reconstruction device based on neural radiation field according to a preferred embodiment of the present invention;

[0038] Figure 3 The comparison results of the rendering effects in Experimental Example 2 are shown. DETAILED DESCRIPTION

[0039] The present invention will be described in further detail below with reference to the accompanying drawings and examples, through which the features and advantages of the present invention will become more clearly understood.

[0040] The word "exemplary" is used exclusively herein to mean "serving as an example, example, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0041] According to the present invention, a three-dimensional reconstruction method based on neural radiation field is provided. Figure 1 As shown, the following steps are included:

[0042] S1. Perform motion structure recovery on the input image to obtain the sparse point cloud of the scene and the camera pose information corresponding to the input image, and sample on the light rays from the camera;

[0043] S2. Render the color of the two-dimensional pixel corresponding to the light based on the sampled data, and obtain the color residual of the image based on the rendered pixel color and the color in the original image;

[0044] Get the number of features in the image that match the feature points of the sparse initial point cloud;

[0045] S3. If the number of image matching features is less than the first threshold and the color residual is greater than the second threshold, extract the image and mark the image as a strong noise image;

[0046] Otherwise, the image is marked as a weak noise image;

[0047] S4. When the proportion of strong noise images exceeds a third threshold, the weight of the strong noise images is set to zero in the loss function, the neural radiation field network is trained, and the color at the observation angle with the weight of zero is updated;

[0048] When the proportion of strong noise images does not exceed the third threshold, the loss function is set to a robust adaptive loss function, and the neural radiation field network is trained to achieve color updates at the corresponding viewing angle;

[0049] S5. Use the trained neural radiation field network to recognize the input image and obtain the color image of the object under different viewing angles.

[0050] Preferably, before S1, there is a step S0 of preprocessing the input:

[0051] If the input is an image, no preprocessing is required;

[0052] If the input is in video format, split the video into image frames.

[0053] In the present invention, the video splitting method is not limited, and those skilled in the art may use any known method, such as using FFMPEG (Fast Forward Moving Picture Experts Group) for splitting.

[0054] In S1, motion structure recovery is a commonly used method in 3D reconstruction. It recovers the 3D structure and camera motion of the scene through a series of 2D images taken from different perspectives. This process involves feature point detection and matching, camera pose estimation, 3D point reconstruction and global optimization. Those skilled in the art can adopt any existing known motion structure recovery method according to actual needs, which is not limited in the present invention.

[0055] For example, feature point detection uses SIFT detector. SIFT feature is a local image feature that is robust to common interferences in images such as translation, rotation, scale change, brightness change, etc.

[0056] The 3D reconstruction process was performed using the Levenberg-Marquardt method.

[0057] The global optimization process is performed using a graph optimization framework, such as Bundle Adjustment.

[0058] According to a preferred embodiment of the present invention, a ray marching method is used to generate light rays starting from the camera.

[0059] In S2, based on the spatial coordinates and viewing direction in the sampled data, a multilayer perceptron is used to obtain the volume density at the corresponding location. Based on the predicted volume density, volume rendering is used to render the color of the 2D pixel corresponding to the light. This process is identical to the traditional NeRF (Neural Radiance Fields) process and will not be further described here.

[0060] Furthermore, in S2, based on the color of the rendered pixel and the color in the original image, a color residual of the pixel is obtained, and the color residual of each pixel is summed to obtain the color residual of the image.

[0061] In S3, preferably, the number of feature points in each image that match the sparse initial point cloud is counted, and the images are sorted accordingly, and images with a number of image matching features less than a first threshold are extracted; the images are sorted according to the color residuals of the extracted images, and images with residuals greater than a second threshold are extracted and marked as strong noise images.

[0062] In S3, the first threshold and the second threshold can be specific values ​​or percentages of images in all images. For example, when the first threshold is set to 50%, it means that images with a small number of image matching features account for 50% of the total number of images, and images with a small number of image matching features are extracted; when the second threshold is set to 20%, it means that images with high color residuals account for 20% of the total number of images.

[0063] In the present invention, there is no restriction on the specific values ​​of the first threshold and the second threshold, and those skilled in the art can freely set them according to actual needs.

[0064] In S4, there is no limitation on the specific value of the third threshold, and those skilled in the art can freely set it according to actual needs. Preferably, it is set to 1%.

[0065] When the proportion of strong noise images exceeds the third threshold, the loss function Set to:

[0066]

[0067] Among them, r represents any ray starting from the camera, each ray corresponds to a pixel in the image, R represents the set of r, C(r) represents the real pixel color value corresponding to the ray r, represents the rendered color predicted along ray r by coarse sampling, represents the rendered color predicted along ray r by fine sampling, represents the square L2 norm, ω represents the binarized weight function, I N Represents the current input image, S CF Represents a collection of strongly noisy images.

[0068] When the proportion of strong noise images does not exceed the third threshold, the loss function f(x,α,c) is set to:

[0069]

[0070] Among them, α is a hyperparameter, x represents the sum of the pixel losses in a single image, and c is a hyperparameter.

[0071] In this paper, α is a hyperparameter that controls the robustness of the loss function. By adjusting the value of α, the loss function can exhibit different characteristics when dealing with residuals of different sizes. For example, when α is close to 2, the loss function is close to the L2 loss and is more sensitive to residuals; when α is close to 0, the loss function is close to the Cauchy loss and is more robust to residuals.

[0072] Preferably, the loss form after α takes different values ​​is as follows:

[0073]

[0074] The loss approaches L2 loss in the limit of α close to 2; when α = 1, the loss is a smooth form of L1 loss; when α approaches zero from the negative direction, this produces Cauchy loss; when α = -2, the function is approximately German-Mcclure loss.

[0075] In a preferred embodiment, during the training process, the value of α changes from a larger value to a smaller value, so that the loss function is more sensitive to the residual in the early stage of training to facilitate rapid convergence, and more robust to the residual in the later stage of training to avoid overfitting.

[0076] In the present invention, through the above constraints and loss optimization, artifacts can be significantly reduced and image rendering quality can be improved, greatly improving robustness.

[0077] The inventors also found that, generally speaking, high-quality three-dimensional reconstruction requires dense perspective data as a training basis. However, in the process of building their own data sets, for most real-world application needs, especially rescue scenarios in emergency response and information collection work represented by various types of building rescue and repair, obtaining dense scene views is not only costly, but also time-consuming during the rendering process. This results in significant performance degradation when limited 2D sparse perspective images are used to synthesize new perspectives. On the one hand, the volume rendering method does not correctly constrain the scene geometry to consist of blank space and opaque surfaces; on the other hand, the appearance and geometry of the reconstructed scene often show a high degree of uncertainty.

[0078] Preferably, depth information of the image is also obtained in step S0, and the depth information is used as a supervisory signal for training the neural radiation field network in S4.

[0079] By introducing depth images and utilizing depth prior to simultaneously constrain its loss and sampling space, the rendering quality of the algorithm in sparse 3D reconstruction is improved.

[0080] The depth image can be obtained by a depth sensor or an RGB-D camera, or by processing the image using Depth-Anything-V2, which is not limited in the present invention.

[0081] Depth-Anything-V2 is a depth estimation model whose core principle is based on the expansion and optimization of the DINOv2-DPT architecture. This model achieves more precise capture and prediction of image depth information.

[0082] Unlike traditional NeRF, which requires coarse and fine sampling for feature point matching during structure-from-motion recovery, this method performs only coarse sampling, eliminating fine sampling. Furthermore, in S2, the depth information of unmatched feature points in the image that do not match the sparse initial point cloud is used to guide sampling.

[0083] Furthermore, although the depth image provides the distance information from each pixel on the image to the camera plane, due to factors such as noise, occlusion and camera calibration error, the actual depth value often has a certain degree of uncertainty.

[0084] In the present invention, an uncertainty estimation value σ is set for each pixel in the RGB image. Dj (u,v), is used to quantify this uncertainty.

[0085] Preferably, the uncertainty estimate is the difference between the depth in the depth image and the depth actually captured by the sensor.

[0086] Preferably, the uncertainty estimate is obtained through a convolutional neural network, the input of the convolutional neural network is an RGB image and a depth image, and the output is the uncertainty estimate.

[0087] Preferably, when training the convolutional neural network, the dataset includes RGB, depth maps, and ground truth camera trajectories. Both the RGB image and the depth map are 640×480 PNG images. The RGB and depth images are pre-aligned using the OpenNI driver, ensuring a 1:1 pixel correspondence between the RGB and depth images. Depth-Anything-V2 is then used to estimate the depth of the images in the dataset, and the difference between the estimated depth and the depth captured by the sensor is recorded as an uncertainty annotation.

[0088] Furthermore, the following steps are performed to sample the feature points in the image that are not matched with the sparse initial point cloud:

[0089] Get the sampling range;

[0090] Use Gaussian distribution to generate sampling points within the sampling range.

[0091] in,

[0092] The sampling range is expressed as:

[0093] t n =max(0,D j (u,v)-k·σ Dj (u,v))

[0094] t f =D j (u,v)+k·σ Dj (u,v)

[0095] Among them, t n Indicates the upper bound of the sampling range, t f Indicates the next level of the sampling range, (u, v) indicates the coordinates of the pixel point, D j (u, v) represents the depth value of the pixel, and k is a hyperparameter used to control the width of the sampling range.

[0096] In the process of generating sampling points, the mean μ of the Gaussian distribution is set to:

[0097] μ=D j (u,v)

[0098] The standard deviation σ is set to:

[0099]

[0100] Where N is the desired number of sampling points.

[0101] Generally speaking, in NeRF, the number of coarse sampling points is 64 and the number of fine sampling points is 128. However, in a preferred embodiment of the present invention, since the sampling point positions can be obtained based on depth information, the fine sampling process is omitted, saving computing power. More preferably, N is set to 64.

[0102] In a preferred embodiment of the present invention, the standard deviation is divided by In this way, while keeping the total uncertainty level unchanged, the uncertainty of each sampling point gradually decreases as the number of sampling points increases, thus achieving convergence.

[0103] According to the present invention, the color and transparency are predicted based on the above-mentioned optimized sampling points and their corresponding light directions. After iterative optimization of parameters, the accuracy and authenticity of the reconstructed scene can be gradually improved.

[0104] In a preferred embodiment, in step S4, sparse 3D point cloud is also used as a supervisory signal when training the neural radiation field network.

[0105] Sparse 3D point clouds can provide the neural radiance field network with additional information about the scene geometry, reducing the reliance on the depth estimator. It can also utilize the noise in the sparse point cloud, which not only reduces the impact on rendering quality but also provides additional loss constraints, thereby helping the model learn the representation of the scene more accurately.

[0106] The sparse 3D point cloud is obtained by motion structure recovery in step S1, and the supervision is depth supervision.

[0107] In NeRF, light will terminate at a certain point when traveling in a certain direction. This point largely determines the color and density of the light. According to the above preferred embodiment of the present invention, sparse 3D point cloud is used as a supervisory signal. Through deep supervision, the estimation of the light termination distribution can be optimized, thereby optimizing the color of the volume rendering and the geometric shape of the object.

[0108] Furthermore, for any input image j, each sparse 3D point cloud keypoint x obtained from motion structure recovery is i , its depth D ij It can be obtained by projecting the key points back to the camera coordinate system and taking the Z value. Ideally, the ray termination distribution should match the depth information provided by the sparse point cloud in the scene, but in practice, this depth value will be affected by various noise sources, such as feature point matching errors, SfM optimization errors, etc.

[0109] Preferably, in the deep supervision process, the loss function is set to:

[0110]

[0111] in, represents the deep supervision loss, D ij is the point x in the sparse 3D point cloud in the input image j i The depth, σ i Indicates that when point x is detected i The average reprojection error on the view of , S(t) is the termination distribution of the light, that is, the probability density of the light terminating at position t.

[0112] Among them, the depth D ij is a random variable whose distribution is normal distribution N(D ij ,σ i ), σ i Represents the average reprojection error over the views where keypoints are detected.

[0113] The total loss function is It can be expressed as:

[0114]

[0115] Among them, λ D is a hyperparameter used to balance the weight between color loss and depth supervision loss.

[0116] Various embodiments of the methods described above in the present invention may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] The present invention also discloses a three-dimensional reconstruction device based on neural radiation field, which is used to collect input images of an object to be three-dimensionally reconstructed, including a turntable 1 and a camera arm 2. Figure 2 As shown,

[0118] The turntable 1 is a rotatable platform for holding the sample to be reconstructed;

[0119] The camera arm 2 is provided with a plurality of cameras 21 , which are fixed to the side of the turntable 1 and are used to photograph the samples on the turntable 1 .

[0120] In the present invention, by controlling the rotation angle of the turntable, multi-angle scanning of the target object can be achieved, thereby obtaining more comprehensive appearance visual information.

[0121] In the present invention, the camera 21 may be a visible light camera, or an infrared camera or a camera of other wavelengths.

[0122] In a preferred embodiment, the camera 21 includes a depth sensor and multiple monocular cameras, and simultaneously obtains an RGB image and a depth image of the object to be three-dimensionally reconstructed.

[0123] Preferably, the rotating shaft of the turntable is vertically arranged, and the multiple cameras 21 are arranged vertically, so that the cameras can capture images of the sample at different positions.

[0124] In the present invention, there is no limitation on the rotation mode of the turntable, which can be realized by mechanical structures such as a motor and a reducer, or by manual rotation, as long as the rotation can be realized.

[0125] In a preferred embodiment, the upper end of the camera arm has an arc-shaped structure, and part of the camera 2 is arranged in the arc-shaped position, so that the camera has multiple shooting angles.

[0126] The device of the present invention uses a self-rotating turntable, which can not only greatly improve the collection efficiency but also guarantee the collection accuracy to a great extent compared with the traditional manual collection method. The most important thing is that light has a great impact on data collection. On the one hand, light can affect the appearance and surface texture of the object, which directly affects the accuracy and quality of three-dimensional reconstruction. However, traditional data collection is mostly done by manual or equipment scanning around the object. Since it is difficult to maintain consistent lighting conditions at all angles of the object in most collection environments and to control the lighting of the object, it is difficult to obtain more accurate data. In the process of automatically collecting data from the object using a turntable, consistent lighting conditions can be maintained by controlling parameters such as the intensity and direction of the light source, thereby avoiding errors caused by inconsistent manual operation. At the same time, the device can maintain consistent lighting conditions by recording parameters such as the intensity and direction of the light source, thereby achieving data repeatability and facilitating subsequent analysis and comparison. It provides a good foundation for the next stage of three-dimensional reconstruction.

[0127] In a preferred embodiment, the three-dimensional reconstruction device further comprises a base 3, preferably a movable base.

[0128] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0129] Example

[0130] Example 1

[0131] Performing 3D reconstruction includes the following steps:

[0132] S1. Perform motion structure recovery on the input image to obtain the sparse point cloud of the scene and the camera pose information corresponding to the input image, and sample on the light rays from the camera;

[0133] S2. Render the color of the two-dimensional pixel corresponding to the light based on the sampled data, and obtain the color residual of the image based on the rendered pixel color and the color in the original image;

[0134] Get the number of features in the image that match the feature points of the sparse initial point cloud;

[0135] S3. If the number of image matching features is less than the first threshold and the color residual is greater than the second threshold, extract the image and mark the image as a strong noise image;

[0136] Otherwise, the image is marked as a weak noise image;

[0137] S4. When the proportion of strong noise images exceeds a third threshold, the weight of the strong noise images is set to zero in the loss function, the neural radiation field network is trained, and the color at the observation angle with the weight of zero is updated;

[0138] When the proportion of strong noise images does not exceed the third threshold, the loss function is set to a robust adaptive loss function, and the neural radiation field network is trained to achieve color updates at the corresponding viewing angle;

[0139] S5. Use the trained neural radiation field network to recognize the input image and obtain the color image of the object under different viewing angles.

[0140] In S3, the number of feature points in each image that match the sparse initial point cloud is counted and sorted accordingly, extracting images with fewer than a first threshold of matching features. The images are sorted based on the color residuals of the extracted images, and images with residuals greater than a second threshold are extracted and marked as strongly noisy images. The first and second thresholds are the percentages of the image in all images, set to 50%, and the second threshold is set to 20%.

[0141] In S4, the third threshold is set to 1%. When the proportion of strong noise images exceeds the third threshold, the loss function Set to:

[0142]

[0143] When the proportion of strong noise images does not exceed the third threshold, the loss function f(x,α,c) is set to:

[0144]

[0145] During the training process, the α value changes from a larger value to a smaller value.

[0146] During the experiment, the datasets NeRF_Shadow and NeRF_Synthetic were used. The datasets include five scene categories: Krusty Krab, Conference Room, Rupan, UCAS, and Tiguan.

[0147] Among them, NeRF_Synthetic is a public dataset, and NeRF_Shadow is a self-made dataset. The dataset scenes contain videos and pictures with and without transient object interference, which makes it easier to compare the robustness and noise reduction effects of the algorithm. The dataset was obtained by shooting with a mobile phone and has a resolution of 1920×1080.

[0148] Example 2

[0149] The same experiment as Example 1 was conducted, except that three scenes were selected in NeRF_Shadow and NeRF_Synthetic respectively, and 2, 5, and 10 images were input into each scene, and their corresponding pose information was reconstructed and input;

[0150] In step S0, the depth information of the image is also obtained, and in step S4, the depth information is used as a supervisory signal for training the neural radiation field network;

[0151] In S2, for the feature points in the image that are not matched with the sparse initial point cloud, their depth information is combined to guide sampling, and an uncertainty estimate is set for each pixel in the RGB image. The uncertainty estimate is the difference between the depth in the depth image and the depth actually collected by the sensor.

[0152] The sampling of feature points in the image that are not matched with the sparse initial point cloud is achieved by the following steps:

[0153] Get the sampling range;

[0154] Use Gaussian distribution to generate sampling points within the sampling range.

[0155] The sampling range is expressed as:

[0156] t n =max(0,D j (u,v)-k·σ Dj (u,v))

[0157] t f =D(u,v)+k·σ Dj (u,v)

[0158] In the process of generating sampling points, the mean μ of the Gaussian distribution is set to:

[0159] μ=D j (u,v)

[0160] The standard deviation σ is set to:

[0161]

[0162] In step S4, sparse 3D point cloud is also used as a supervisory signal when training the neural radiation field network.

[0163] The total loss function Expressed as:

[0164]

[0165] Comparative Example 1

[0166] The same experiment as in Example 1 was performed, except that the NeRF method, Instant-NGP method, NeRF-W method, Mip-NeRF method, and Robust-NeRF method were used.

[0167] Comparative Example 2

[0168] The same experiment as Example 2 was performed, except that the NeRF method, Instant-NGP method, NeRF-W method, Mip-NeRF method, and Robust-NeRF method were used.

[0169] Experimental Example 1

[0170] Comparing the results in Example 1 and Comparative Example 1, PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) were used as evaluation indicators, respectively.

[0171] PSNR and SSIM are commonly used metrics for evaluating NeRF models, measuring image quality and similarity, respectively. Table 1 shows the comparison results of Peak Signal-to-Noise Ratio (PSNR) on the NeRF_Shadow dataset, and Table 2 shows the comparison results of Peak Signal-to-Noise Ratio (PSNR) on the NeRF_Syntheti dataset. Table 3 shows the comparison results of SSIM on the NeRF_Shadow dataset, and Table 4 shows the comparison results of SSIM on the NeRF_Syntheti dataset.

[0172] Table 1 Peak signal-to-noise ratio in the NeRF_Shadow dataset

[0173] Model / Scene Krusty Krab Meeting Room Like a rock UCAS Tiguan mean Comparative Example 1-NeRF 21.08 19.77 16.06 22.36 17.01 19.26 Comparative Example 1-Instant-NGP 23.21 20.95 19.74 23.39 20.58 21.57 Comparative Example 1-NeRF-W 25.16 25.42 19.28 26.25 26.67 24.56 Comparative Example 1-Mip-NeRF 23.13 22.37 20.86 25.23 24.82 23.68 Comparative Example 1-Robust-NeRF 26.79 26.31 18.25 29.54 25.37 25.25 Example 1 30.40 28.30 21.99 31.11 26.39 27.64

[0174] Table 2 Peak signal-to-noise ratio in the NeRF_Syntheti dataset

[0175] Model / Scene Chair drum set Banyan Tree hot dog Lego mean Comparative Example 1-NeRF 29.72 25.64 31.83 36.18 31.23 30.92 Comparative Example 1-Instant-NGP 31.02 24.18 32.33 34.86 32.77 31.03 Comparative Example 1-NeRF-W 31.21 27.23 30.71 32.08 31.79 30.60 Comparative Example 1-Mip-NeRF 32.63 30.44 33.42 34.50 34.25 33.05 Comparative Example 1-Robust-NeRF 29.10 25.31 31.32 33.70 32.45 30.38 Example 1 36.98 28.30 36.50 37.11 38.04 35.39

[0176] Table 3 Structural similarity in the NeRF_Shadow dataset

[0177]

[0178]

[0179] Table 4 Structural similarity in the NeRF_Synthetic dataset

[0180] Model / Scene Chair drum set Banyan Tree hot dog Lego mean Comparative Example 1-NeRF 0.967 0.925 0.964 0.974 0.961 0.958 Comparative Example 1-Instant-NGP - - - - - - Comparative Example 1-NeRF-W 0.931 0.811 0.874 0.933 0.949 0.900 Comparative Example 1-Mip-NeRF 0.988 0.945 0.984 0.988 0.984 0.978 Comparative Example 1-Robust-NeRF 0.753 0.653 0.792 0.852 0.835 0.777 Example 1 0.952 0.940 0.962 0.970 0.986 0.962

[0181] As can be seen from Table 1, the method in Example 1 has the highest average peak signal-to-noise ratio. Except for the Tiguan car scene, which is slightly lower than NeRF-W by 0.28dB, the PSNR values ​​of other scenes in the NeRF_Shadow dataset are higher than those of other methods. This shows that the overall performance of the algorithm in interference scenes is better than that of other methods. From the perspective of the mean, compared with the robust model NeRF-W and Robust-NeRF model for transient object interference, its mean is improved by 3.08dB and 2.39dB respectively, indicating that it has achieved better results in noise processing in the scene. At the same time, it is not difficult to find that Robust-NeRF is 2.61dB lower than Mip-NeRF in the scene Ru Pan (better image acquisition), while the method in Example 1 is 1.13dB higher than Mip-NeRF, which reflects that it can maintain good results when facing various noise levels in real acquisition scenes.

[0182] As shown in Table 2, the method in Example 1 still achieved the highest average PSNR across all scenarios, at 35.39. However, the robust models NeRF-W and Robust-NeRF presented different results on the NeRF_Shadow dataset, with NeRF-W performing worse than Instant-NGP and Robust-NeRF performing worse than Mip-NeRF.

[0183] As shown in Table 3, the method in Example 1 outperforms other algorithms in both single-scenario and average performance, demonstrating superior performance. Compared to other algorithms, its average SSIM values ​​improve by 2.34% to 10.08%. Since the Instant-NGP packaging tool does not output relevant SSIM values, they are omitted here.

[0184] As can be seen from Table 4, the method in Example 1 is second only to Mip-NeRF and slightly higher than the original NeRF. This is partly due to the lack of noise in the image, and partly due to the multi-scale representation method introduced by Mip-NeRF, which achieves multi-scale sampling by rendering anti-aliased cone frustums. This method effectively reduces aliasing and further improves image quality. The method in Example 1 mainly relies on an adaptive loss function for updates in noise-free scenes, so there is no significant performance drop in weak noise scenes, and there is a slight improvement compared to the original NeRF. However, Robust-NeRF still shows a significant performance drop in weak noise scenes.

[0185] Experimental Example 2

[0186] Compare the results in Example 2 and Comparative Example 2, where Table 5 shows the average PSNR of each scene rendered under sparse input. Figure 3 A two-dimensional display image of the rendering effect.

[0187] Table 5 Average PSNR under sparse input

[0188]

[0189] As can be seen from Table 5, under the same sparse input setting, the PSNR value of the method in Example 2 is higher than that of other methods, which directly reflects its advantages in preserving image details and reducing noise.

[0190] from Figure 2 It can be seen that when two images are input, the method of Example 2 can only render a clear effect in some scenes. Although there are still some blurs and distortions in many details, the rendering quality has been significantly improved compared to the extremely blurry rendering effect of NeRF. Especially on the real dataset NeRF_Shadow, due to the lack of carefully designed sampling conditions in the laboratory like NeRF_Synthetic, the NeRF rendering effect makes it extremely difficult to obtain information, and it is even more difficult to make a difference in downstream tasks. In addition to some details, the method in Example 2 has a better presentation of the shape information of objects and the co-constructed structure of scenes and objects, which can reflect the advantages of three-dimensional models over two-dimensional images. With the increase in the number of input images, when 5 and 10 images are input, the rendering quality of both NeRF and the method in Example 2 has been significantly improved. However, there are still many artifacts or floating objects in the NeRF model, and the method in Example 2 can produce more refined and realistic rendering effects, and the details are clearer and more accurate, and in multi-view Figure 1 This also verifies the effectiveness of the deep supervision module in the method of Example 2 for rendering effects.

[0191] The present invention has been described above with reference to preferred embodiments, but these embodiments are merely exemplary and serve only as illustrations. On this basis, various replacements and improvements can be made to the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction method based on neural radiation field, characterized in that: The following steps are involved: S1. Perform motion structure recovery on the input image to obtain the sparse point cloud of the scene and the camera pose information corresponding to the input image, and perform sampling on the light rays from the camera; S2. Render the color of the two-dimensional pixel corresponding to the light based on the sampled data, and obtain the color residual of the image based on the rendered pixel color and the color in the original image; Get the number of features in the image that match the feature points of the sparse initial point cloud; S3. If the number of image matching features is less than the first threshold and the color residual is greater than the second threshold, extract the image and mark the image as a strong noise image; Otherwise, the image is marked as a weak noise image; S4. When the proportion of strong noise images exceeds a third threshold, the weight of the strong noise images is set to zero in the loss function, the neural radiation field network is trained, and the color at the observation angle with the weight of zero is updated; When the proportion of strong noise images does not exceed the third threshold, the loss function is set to a robust adaptive loss function, and the neural radiation field network is trained to achieve color updates at the corresponding viewing angle; S5. Use the trained neural radiation field network to recognize the input image and obtain the color image of the object under different viewing angles.

2. The three-dimensional reconstruction method based on neural radiation field according to claim 1, characterized in that: Before S1, there is also step S0, which preprocesses the input: If the input is an image, no preprocessing is required; If the input is in video format, split the video into image frames.

3. The three-dimensional reconstruction method based on neural radiation field according to claim 1, characterized in that: In S4, when the proportion of strong noise images exceeds the third threshold, the loss function Set to: Where r represents any ray starting from the camera, each ray corresponds to a pixel in the image, R represents the set of r, and c(r) represents the real pixel color value corresponding to the ray r. represents the rendered color predicted along ray r by coarse sampling, represents the rendered color predicted along ray r by fine sampling, represents the square L2 norm, ω represents the binarized weight function, I N Represents the current input image, S CF Represents a collection of strongly noisy images.

4. The three-dimensional reconstruction method based on neural radiation field according to claim 1, characterized in that: In S4, when the proportion of strong noise images does not exceed the third threshold, the loss function f(x,α,c) is set to: Among them, α is a hyperparameter, x represents the sum of the pixel losses in a single image, and c is a hyperparameter.

5. The three-dimensional reconstruction method based on neural radiation field according to claim 4, characterized in that: During the training process, the α value changes from a larger value to a smaller value.

6. The three-dimensional reconstruction method based on neural radiation field according to claim 1, characterized in that: In step S0, depth information of the image is also obtained, and in step S4, the depth information is used as a supervisory signal for training the neural radiation field network.

7. The three-dimensional reconstruction method based on neural radiation field according to claim 1, characterized in that: In S2, for the feature points in the image that are not matched with the sparse initial point cloud, their depth information is combined to guide sampling.

8. A three-dimensional reconstruction device based on neural radiation field, used to collect input images of an object to be three-dimensionally reconstructed, characterized in that: Including turntable 1 and camera arm 2, The turntable 1 is a rotatable platform for holding the sample to be reconstructed; The camera arm 2 is provided with a plurality of cameras 21 , which are fixed to the side of the turntable 1 and are used to photograph the samples on the turntable 1 .

9. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images