A method for three-dimensional reconstruction based on neural radiation field and a device applying the method

Abstract

The application discloses a three-dimensional reconstruction method based on a neural radiance field, comprising: performing motion structure recovery on an input image, sampling on a light ray from a camera, and obtaining color residuals of the image and a feature quantity of the image matched with feature points of a sparse initial point cloud; if the feature quantity of the image is less than a first threshold value and the color residuals are greater than a second threshold value, the image is marked as a strong noise image; otherwise, the image is marked as a weak noise image; when a proportion of the strong noise image exceeds a third threshold value, a weight of the strong noise image in a loss function is set to zero, a neural radiance field network is trained, and colors in an observation angle with the zero weight are updated; when the proportion of the strong noise image does not exceed the third threshold value, the loss function is set to a robust adaptive loss function, the neural radiance field network is trained, and colors in a corresponding angle are updated. The method disclosed by the application improves the visual effect of rendering while almost not increasing additional calculation.

Description

Technical Field

[0001] This invention relates to a three-dimensional reconstruction method based on neural radiation fields and a device for applying this method, belonging to the field of computer vision technology. Background Technology

[0002] Understanding the three-dimensional world from a two-dimensional scene is a fundamental problem in computer vision, and Neural Radiance Fields (NeRF) excels at synthesizing new views given multiple views of a static scene. However, when the scene contains interfering elements that do not persist or shift in the acquired two-dimensional image, such as moving objects, changes in brightness, and shadows, artifacts will appear as view-dependent effects or "floating objects."

[0003] Furthermore, high-quality 3D reconstruction requires dense viewpoint data as a training foundation. However, during the construction of our own dataset, we found that for most real-world applications, especially emergency response rescue scenarios and information gathering work represented by various building rescue and repair efforts, acquiring dense scene views is not only costly but also extremely time-consuming during rendering. For some terrains such as rainforests, mountains, riverbanks, and hills, certain angles are difficult to reach, and even using drones is subject to interference from flora and fauna, no-fly zones, and signal issues. Synthesizing new perspectives using extremely limited 2D sparse viewpoint images is a highly challenging task, stemming from the fact that these sparse perspectives provide insufficient constraints for 3D reconstruction. Directly applying NeRF to such sparse viewpoint scenarios often leads to significant performance degradation. On the one hand, volume rendering methods do not correctly constrain the scene geometry to consist of empty spaces and opaque surfaces. On the other hand, when faced with an extremely limited number of input views, the appearance and geometry of the reconstructed scene often exhibit high uncertainty. Although it may learn rendering effects that highly match the training views, this is essentially an overfitting of the model to the training data.

[0004] Therefore, it is necessary to conduct more in-depth research on existing 3D reconstruction methods to solve the above problems. Summary of the Invention

[0005] To overcome the above problems, in-depth research was conducted, and a three-dimensional reconstruction method based on neural radiation fields was proposed, including the following steps:

[0006] S1. Perform motion structure restoration on the input image to obtain sparse point cloud of the scene and camera pose information corresponding to the input image, and sample along the light ray emanating from the camera.

[0007] S2. Based on the sampled data, render the color of the two-dimensional pixel corresponding to the light, and obtain the color residual of the image based on the color of the rendered pixel and the color in the original image;

[0008] Obtain the number of features in the image that match the feature points of the sparse initial point cloud;

[0009] S3. If the number of matching features in the image is less than the first threshold and the color residual is greater than the second threshold, then the image is extracted; the image is then marked as a noisy image.

[0010] Otherwise, label the image as a weak noise image;

[0011] S4. When the proportion of strong noise images exceeds the third threshold, the weight of strong noise images is set to zero in the loss function, the neural radiation field network is trained, and the color of the observation view with zero weight is updated.

[0012] When the proportion of strong noise images does not exceed the third threshold, the loss function is set to a robust and adaptive loss function, and the neural radiation field network is trained to update the color at the corresponding viewpoint.

[0013] S5. The trained neural radiation field network is used to identify the input image and obtain the color images of the object from different perspectives.

[0014] In a preferred embodiment, before S1, there is a step S0: preprocessing the input.

[0015] If the input is an image, no preprocessing is required;

[0016] If the input is in video format, the video will be split into image frames.

[0017] In a preferred embodiment, in S4, when the proportion of strong noise images exceeds a third threshold, the loss function... Set to:

[0018]

[0019] Where r represents any ray of light originating from the camera, each ray corresponds to a pixel in the image, R represents the set of r, and C(r) represents the actual pixel color value corresponding to ray r. This represents the rendered color predicted by coarse sampling along the ray r. This represents the rendered color predicted by fine sampling along the ray r. Let I denote the squared L2 norm, ω denote the binarized weighting function, and I N S represents the currently input image. CF This represents a set of images with strong noise.

[0020] In a preferred embodiment, in S4, when the proportion of noisy images does not exceed a third threshold, the loss function f(x,α,c) is set as follows:

[0021]

[0022] Where α is a hyperparameter, x represents the sum of pixel loss in a single image, and c is a hyperparameter.

[0023] In a preferred embodiment, during training, the α value continuously changes from a larger value to a smaller value.

[0024] In a preferred embodiment, depth information of the image is also obtained in step S0, and the depth information is used as a supervision signal for training the neural radiation field network in step S4.

[0025] In a preferred embodiment, in S2, sampling is guided by the depth information of unmatched feature points in the image that do not match the sparse initial point cloud.

[0026] This invention also discloses a three-dimensional reconstruction device based on neural radiation fields, used to acquire input images of an object to be reconstructed in three dimensions, including a turntable and a camera arm.

[0027] The turntable is a rotatable platform used to hold the sample to be reconstructed;

[0028] The camera arm is equipped with multiple cameras, which are fixed to the side of the turntable and used to photograph the samples on the turntable.

[0029] The present invention also provides an electronic device, comprising:

[0030] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods described above.

[0031] The present invention also provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in any of the preceding claims.

[0032] The beneficial effects of this invention include:

[0033] (1) The number of feature points matched to the three-dimensional point cloud and the color residual are constrained together, and the interference of transient objects is removed by weight update, which improves the visualization effect of rendering while hardly increasing the amount of additional computation.

[0034] (2) Through multi-branch optimization and loss update method, performance can still be maintained in the face of interference or no interference. Even in the face of no interference scene, not only will its performance not decline, but its rendering quality can also be partially improved.

[0035] (3) By utilizing depth priors to simultaneously constrain its loss and sampling space, the rendering quality of the algorithm in sparse 3D reconstruction is improved. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of a three-dimensional reconstruction method based on neural radiation fields according to a preferred embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of a three-dimensional reconstruction device based on a neural radiation field according to a preferred embodiment of the present invention.

[0038] Figure 3 The comparison results of the rendering effects in Experiment Example 2 are shown. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present invention will become clearer and more apparent.

[0040] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.

[0041] According to the present invention, a three-dimensional reconstruction method based on neural radiation fields is provided, such as... Figure 1 As shown, it includes the following steps:

[0042] S1. Perform motion structure restoration on the input image to obtain sparse point cloud of the scene and camera pose information corresponding to the input image, and sample along the light ray emanating from the camera.

[0043] S2. Based on the sampled data, render the color of the two-dimensional pixel corresponding to the light, and obtain the color residual of the image based on the color of the rendered pixel and the color in the original image;

[0044] Obtain the number of features in the image that match the feature points of the sparse initial point cloud;

[0045] S3. If the number of matching features in the image is less than the first threshold and the color residual is greater than the second threshold, then the image is extracted; the image is then marked as a noisy image.

[0046] Otherwise, label the image as a weak noise image;

[0047] S4. When the proportion of strong noise images exceeds the third threshold, the weight of strong noise images is set to zero in the loss function, the neural radiation field network is trained, and the color of the observation view with zero weight is updated.

[0048] When the proportion of strong noise images does not exceed the third threshold, the loss function is set to a robust and adaptive loss function, and the neural radiation field network is trained to update the color at the corresponding viewpoint.

[0049] S5. The trained neural radiation field network is used to identify the input image and obtain the color images of the object from different perspectives.

[0050] Preferably, before S1, there is a step S0: preprocessing the input.

[0051] If the input is an image, no preprocessing is required;

[0052] If the input is in video format, the video will be split into image frames.

[0053] In this invention, the video splitting method is not limited, and those skilled in the art can use any known method, such as FFMPEG (Fast Forward Moving Picture Experts Group) for splitting.

[0054] In S1, motion structure recovery is a common method in 3D reconstruction. It recovers the 3D structure of the scene and camera motion through a series of 2D images taken from different perspectives. This process involves feature point detection and matching, camera pose estimation, 3D point reconstruction and global optimization. Those skilled in the art can use any known motion structure recovery method according to actual needs, and no limitation is made in this invention.

[0055] For example, feature point detection uses a SIFT detector. SIFT features are a type of local image feature that is robust to common image disturbances such as translation, rotation, scale changes, and brightness transformations.

[0056] The 3D reconstruction process was performed using the Levenberg-Marquardt method.

[0057] The global optimization process uses a graph optimization framework, such as Bundle Adjustment.

[0058] According to a preferred embodiment of the present invention, a ray stepping method is used to generate light rays originating from the camera.

[0059] In S2, based on the spatial coordinates and viewing direction in the sampled data, the volume density at the corresponding location is obtained through a multilayer perceptron. Based on the predicted volume density, the color of the two-dimensional pixel corresponding to the light is rendered using a volume rendering method. This process is the same as the traditional NeRF (Neural Radiance Fields) process and will not be elaborated here.

[0060] Furthermore, in S2, the color residual of the pixel is obtained based on the color of the rendered pixel and the color in the original image, and the colors of each pixel are summed to obtain the color residual of the image.

[0061] In S3, preferably, the number of feature points that match the sparse initial point cloud in each image is counted, and the images are sorted accordingly. Images with fewer than a first threshold of matching feature points are extracted. Images are sorted according to the color residual of the extracted images, and images with a residual greater than a second threshold are extracted and marked as strong noise images.

[0062] In S3, the first threshold and the second threshold can be specific values ​​or percentages of the images in all images. For example, when the first threshold is set to 50%, it means that images with fewer matching features account for 50% of the total number of images, and images with fewer matching features are extracted. When the second threshold is set to 20%, it means that images with high color residuals account for 20% of the total number of images.

[0063] In this invention, there are no restrictions on the specific values ​​of the first threshold and the second threshold, and those skilled in the art can set them freely according to actual needs.

[0064] In S4, the specific value of the third threshold is not limited, and those skilled in the art can set it freely according to actual needs. Preferably, it is set to 1%.

[0065] When the proportion of noisy images exceeds the third threshold, the loss function... Set to:

[0066]

[0067] Where r represents any ray of light originating from the camera, each ray corresponds to a pixel in the image, R represents the set of r, and C(r) represents the actual pixel color value corresponding to ray r. This represents the rendered color predicted by coarse sampling along the ray r. This represents the rendered color predicted by fine sampling along the ray r. Let I denote the squared L2 norm, ω denote the binarized weighting function, and I N S represents the currently input image. CF This represents a set of images with strong noise.

[0068] When the proportion of noisy images does not exceed the third threshold, the loss function f(x,α,c) is set as follows:

[0069]

[0070] Where α is a hyperparameter, x represents the sum of pixel loss in a single image, and c is a hyperparameter.

[0071] In this invention, α is a hyperparameter that controls the robustness of the loss function. By adjusting the value of α, the loss function can exhibit different characteristics when dealing with residuals of different sizes. For example, when α is close to 2, the loss function is close to L2 loss and is more sensitive to residuals; when α is close to 0, the loss function is close to Cauchy loss and is more robust to residuals.

[0072] Preferably, the loss forms for different values ​​of α are as follows:

[0073]

[0074] The loss approaches L2 loss as α approaches the limit of 2; when α = 1, the loss is a smooth form of L1 loss; when α approaches zero from the negative direction, this produces Cauchy loss; when α = -2, the function approximates German-Mcclure loss.

[0075] In a preferred embodiment, during training, the α value changes continuously from a large value to a small value, making the loss function more sensitive to the residuals in the early stage of training so as to achieve rapid convergence, and more robust to the residuals in the later stage of training to avoid overfitting.

[0076] In this invention, through the above-mentioned constraints and loss optimization, artifacts can be significantly reduced and image rendering quality can be improved, greatly enhancing robustness.

[0077] The inventors also discovered that, generally speaking, high-quality 3D reconstruction requires dense viewpoint data as a training basis. However, in the process of building their own datasets, for most real-world application needs, especially for rescue scenarios in emergency response and information collection work represented by various building rescue and repair, acquiring dense scene views is not only costly but also time-consuming during rendering. This results in significant performance degradation when synthesizing new viewpoints from limited 2D sparse viewpoint images. On the one hand, the volume rendering method does not correctly constrain the scene geometry to consist of blank spaces and opaque surfaces; on the other hand, the appearance and geometry of the reconstructed scene often exhibit a high degree of uncertainty.

[0078] Preferably, in step S0, depth information of the image is also obtained, and in step S4, the depth information is used as a supervision signal for training the neural radiation field network.

[0079] By introducing depth images and utilizing depth priors to simultaneously constrain their loss and sampling space, the rendering quality of the algorithm in sparse 3D reconstruction is improved.

[0080] The depth image can be acquired by a depth sensor or an RGB-D camera, or it can be obtained by processing the image using Depth-Anything-V2. No limitation is imposed in this invention.

[0081] Depth-Anything-V2 is a depth estimation model whose core principle is based on an extension and optimization of the DINOv2-DPT architecture. This model enables more refined capture and prediction of image depth information.

[0082] In traditional NeRF, feature point matching during the motion structure recovery process requires two steps: coarse sampling and fine sampling. Unlike traditional NeRF, this invention does not perform fine sampling, but only coarse sampling. Furthermore, in S2, for unmatched feature points in the image that do not match the sparse initial point cloud, their depth information is used to guide sampling.

[0083] Furthermore, although depth images provide distance information from each pixel in the image to the camera plane, the actual depth values ​​obtained often have a certain degree of uncertainty due to factors such as noise, occlusion, and camera calibration errors.

[0084] In this invention, an uncertainty estimate σ is set for each pixel in the RGB image. Dj (u,v) is used to quantify this uncertainty.

[0085] Preferably, the uncertainty estimate is the difference between the depth in the depth image and the depth actually acquired by the sensor.

[0086] Preferably, the uncertainty estimate is obtained through a convolutional neural network, wherein the input of the convolutional neural network is an RGB image and a depth image, and the output is the uncertainty estimate.

[0087] Preferably, during the training of the convolutional neural network, the dataset includes RGB images, depth maps, and ground truth camera trajectories. Both the RGB images and the 640×480 depth maps are PNG images. The RGB and depth images have been pre-aligned using the OpenNI driver, meaning the pixels in the RGB and depth images are already in a 1:1 correspondence. Then, Depth-Anything-V2 is used to perform depth estimation on the images in the dataset, and the difference between this estimation and the depth obtained from the sensor is recorded as an uncertainty label.

[0088] Furthermore, the sampling of unmatched feature points in the image that do not correspond to the sparse initial point cloud is achieved through the following steps:

[0089] Obtain the sampling range;

[0090] Sampling points are generated within the sampling range using a Gaussian distribution.

[0091] in,

[0092] The sampling range is represented as follows:

[0093] t n =max(0,D j (u,v)-k·σ Dj (u,v))

[0094] t f =D j (u,v)+k·σ Dj (u,v)

[0095] Among them, t n t represents the upper bound of the sampling range. f The lower bound of the sampling range is represented by (u,v), where (u,v) represents the coordinates of the pixel. j (u,v) represents the depth value of a pixel, and k is a hyperparameter used to control the width of the sampling range.

[0096] During the generation of sampling points, the mean μ of the Gaussian distribution is set as follows:

[0097] μ=D j (u,v)

[0098] The standard deviation σ is set as follows:

[0099]

[0100] Where N is the desired number of sampling points.

[0101] Generally, NeRF uses 64 coarse sampling points and 128 fine sampling points. However, in the preferred embodiment of this invention, since the sampling point positions can be obtained based on depth information, the fine sampling process is omitted, saving computational power. More preferably, N=64.

[0102] In a preferred embodiment of the present invention, the standard deviation is divided by This allows the uncertainty of each sampling point to gradually decrease as the number of sampling points increases, while keeping the total level of uncertainty constant, thus achieving convergence.

[0103] According to the present invention, based on the above-optimized sampling points and their corresponding light directions, color and transparency are predicted. Through iterative optimization of parameters, the accuracy and realism of the reconstructed scene can be gradually improved.

[0104] In a preferred embodiment, in step S4, sparse 3D point clouds are also used as supervision signals when training the neural radiation field network.

[0105] Sparse 3D point clouds can provide neural radiation field networks with additional information about scene geometry, reducing reliance on depth estimators and leveraging noise in sparse point clouds to reduce the impact on rendering quality while providing additional loss constraints, thereby helping the model learn scene representations more accurately.

[0106] The sparse 3D point cloud is obtained by motion structure recovery in step S1, and the supervision is depth supervision.

[0107] In NeRF, light rays travel in a certain direction and terminate at a certain point. This point largely determines the color and density of the light rays. According to the preferred embodiment of the present invention, sparse 3D point clouds serve as a supervisory signal. By supervising the depth, the estimation of the light ray termination distribution can be optimized, thereby optimizing the color of the volume rendering and the geometry of the object.

[0108] Furthermore, for any input image j, each sparse 3D point cloud keypoint x obtained from the motion structure recovery i Its depth D ij The depth value can be obtained by projecting the key points back to the camera coordinate system and taking the Z value. Ideally, the ray termination distribution should match the depth information provided by the sparse point cloud in the scene. However, in reality, this depth value is affected by a variety of noise sources, such as feature point matching errors and SfM optimization errors.

[0109] Preferably, during deep supervision, the loss function is set as follows:

[0110]

[0111] in, D represents the loss of deep supervision. ij For the sparse 3D point cloud in the input image j, the midpoint x i depth, σ i This indicates that point x was detected. i The average reprojection error on the view, S(t) is the termination distribution of the light ray, that is, the probability density of the light ray terminating at position t.

[0112] Wherein, depth D ij Let N(D) be a random variable whose distribution is a normal distribution. ij ,σ i ), σ i This represents the average reprojection error on the view where key points were detected.

[0113] Then the total loss function It can be represented as:

[0114]

[0115] Where, λ D It is a hyperparameter used to balance the weights between color loss and depth supervision loss.

[0116] Various embodiments of the methods described above in this invention can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] This invention also discloses a three-dimensional reconstruction device based on neural radiation fields, used to acquire input images of an object to be reconstructed in three dimensions, including a turntable 1 and a camera arm 2, such as... Figure 2 As shown,

[0118] The turntable 1 is a rotatable platform used to hold the sample to be reconstructed;

[0119] The camera arm 2 is equipped with multiple cameras 21, which are fixed to the side of the turntable 1 and used to photograph the samples on the turntable 1.

[0120] In this invention, by controlling the rotation angle of the turntable, multi-angle scanning of the target object can be achieved, thereby obtaining more comprehensive visual information about its appearance.

[0121] In this invention, the camera 21 can be a visible light camera, an infrared camera, or a camera in other wavelength bands.

[0122] In a preferred embodiment, the camera 21 includes a depth sensor and multiple monocular cameras to simultaneously acquire RGB images and depth images of the object to be reconstructed in 3D.

[0123] Preferably, the rotating shaft of the turntable is vertically arranged, and the multiple cameras 21 are arranged vertically, so that the cameras can capture images of the sample from different positions.

[0124] In this invention, the rotation method of the turntable is not limited; it can be achieved through mechanical structures such as motors and reducers, or it can be rotated manually. Any method that enables rotation is acceptable.

[0125] In a preferred embodiment, the upper end of the camera arm has an arc-shaped structure, and some of the cameras 21 are positioned in the arc-shaped position, so that the cameras have multiple shooting angles.

[0126] The device in this invention employs a self-rotating turntable, which significantly improves acquisition efficiency and largely guarantees acquisition accuracy compared to traditional manual methods. Most importantly, lighting has a significant impact on data acquisition. On one hand, lighting affects the appearance and surface texture of an object, directly influencing the accuracy and quality of 3D reconstruction. However, traditional data acquisition often involves manual or automated scanning around the object. Since it's difficult to maintain consistent lighting conditions across different angles of the object in most acquisition environments and to control the lighting, accurate data is hard to obtain. The turntable, however, automatically acquires data by controlling parameters such as the intensity and direction of the light source to maintain consistent lighting conditions, thus avoiding errors caused by inconsistent human operation. Simultaneously, the device maintains consistent lighting conditions by recording parameters such as the intensity and direction of the light source, ensuring data repeatability and facilitating subsequent analysis and comparison. This provides a solid foundation for the next stage of 3D reconstruction.

[0127] In a preferred embodiment, the three-dimensional reconstruction device further includes a base 3, preferably a movable base.

[0128] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0129] Example

[0130] Example 1

[0131] Performing 3D reconstruction includes the following steps:

[0132] S1. Perform motion structure restoration on the input image to obtain sparse point cloud of the scene and camera pose information corresponding to the input image, and sample along the light ray emanating from the camera.

[0133] S2. Based on the sampled data, render the color of the two-dimensional pixel corresponding to the light, and obtain the color residual of the image based on the color of the rendered pixel and the color in the original image;

[0134] Obtain the number of features in the image that match the feature points of the sparse initial point cloud;

[0135] S3. If the number of matching features in the image is less than the first threshold and the color residual is greater than the second threshold, then the image is extracted; the image is then marked as a noisy image.

[0136] Otherwise, label the image as a weak noise image;

[0137] S4. When the proportion of strong noise images exceeds the third threshold, the weight of strong noise images is set to zero in the loss function, the neural radiation field network is trained, and the color of the observation view with zero weight is updated.

[0138] When the proportion of strong noise images does not exceed the third threshold, the loss function is set to a robust and adaptive loss function, and the neural radiation field network is trained to update the color at the corresponding viewpoint.

[0139] S5. The trained neural radiation field network is used to identify the input image and obtain the color images of the object from different perspectives.

[0140] In step S3, the number of feature points matching the sparse initial point cloud in each image is counted, and the images are sorted accordingly. Images with fewer than a first threshold of matching feature points are extracted. Images are then sorted based on their color residuals, and images with residuals greater than a second threshold are extracted and marked as noisy images. The first and second thresholds represent the percentage of images in all images, set to 50% and 20% respectively.

[0141] In S4, the third threshold is set to 1%. When the proportion of strong noise images exceeds the third threshold, the loss function... Set to:

[0142]

[0143] When the proportion of noisy images does not exceed the third threshold, the loss function f(x,α,c) is set as follows:

[0144]

[0145] During training, the α value continuously changes from a larger value to a smaller value.

[0146] The experiment used the NeRF_Shadow and NeRF_Synthetic datasets, which included five scene categories: Krusty Krab, meeting room, Raspberry Pi, UCAS, and Tiguan.

[0147] Among them, NeRF_Synthetic is a public dataset, and NeRF_Shadow is a self-made dataset. The dataset contains videos and images with and without transient object interference, which facilitates the comparison of the robustness and noise reduction effect of the algorithm. The dataset was obtained by shooting with a mobile phone and has a resolution of 1920×1080.

[0148] Example 2

[0149] The same experiment as in Example 1 was conducted, except that three scenes were selected for NeRF_Shadow and NeRF_Synthetic respectively, and 2, 5 and 10 images were input for each scene. Their corresponding pose information was reconstructed and input.

[0150] In step S0, depth information of the image is also obtained, and in step S4, the depth information is used as a supervision signal for training the neural radiation field network.

[0151] In S2, for feature points in the image that do not match the sparse initial point cloud, sampling is guided by their depth information, and an uncertainty estimate is set for each pixel in the RGB image. The uncertainty estimate is the difference between the depth in the depth image and the depth actually acquired by the sensor.

[0152] The following steps are used to sample feature points in the image that do not match the sparse initial point cloud:

[0153] Obtain the sampling range;

[0154] Sampling points are generated within the sampling range using a Gaussian distribution.

[0155] The sampling range is represented as follows:

[0156] t n =max(0,D j (u,v)-k·σ Dj (u,v))

[0157] t f =D(u,v)+k·σ Dj (u,v)

[0158] During the generation of sampling points, the mean μ of the Gaussian distribution is set as follows:

[0159] μ=D j (u,v)

[0160] The standard deviation σ is set as follows:

[0161]

[0162] In step S4, sparse 3D point clouds are also used as supervision signals when training the neural radiation field network.

[0163] Total loss function Represented as:

[0164]

[0165] Comparative Example 1

[0166] The same experiments as in Example 1 were conducted, except that the NeRF method, Instant-NGP method, NeRF-W method, Mip-NeRF method, and Robust-NeRF method were used.

[0167] Comparative Example 2

[0168] The same experiments as in Example 2 were conducted, except that the NeRF method, Instant-NGP method, NeRF-W method, Mip-NeRF method, and Robust-NeRF method were used.

[0169] Experimental Example 1

[0170] Comparing the results of Example 1 and Comparative Example 1, PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) were used as evaluation metrics, respectively.

[0171] PSNR and SSIM are used to measure image quality and similarity, respectively, and are commonly used metrics for evaluating NeRF models. Table 1 shows the comparison results of PSNR on the NeRF_Shadow dataset, Table 2 shows the comparison results of PSNR on the NeRF_Syntheti dataset, Table 3 shows the comparison results of SSIM on the NeRF_Shadow dataset, and Table 4 shows the comparison results of SSIM on the NeRF_Syntheti dataset.

[0172] Table 1 shows the peak signal-to-noise ratio in the NeRF_Shadow dataset.

[0173] Comparative Example 1-NeRF 21.08 19.77 16.06 22.36 17.01 19.26 Comparative Example 1 - Instant-NGP 23.21 20.95 19.74 23.39 20.58 21.57 Comparative Example 1-NeRF-W 25.16 25.42 19.28 26.25 26.67 24.56 Comparative Example 1-Mip-NeRF 23.13 22.37 20.86 25.23 24.82 23.68 Comparative Example 1 - Robust-NeRF 26.79 26.31 18.25 29.54 25.37 25.25 Example 1 30.40 28.30 21.99 31.11 26.39 27.64

[0174] Table 2 shows the peak signal-to-noise ratio in the NeRF_Syntheti dataset.

[0175] Comparative Example 1-NeRF 29.72 25.64 31.83 36.18 31.23 30.92 Comparative Example 1 - Instant-NGP 31.02 24.18 32.33 34.86 32.77 31.03 Comparative Example 1-NeRF-W 31.21 27.23 30.71 32.08 31.79 30.60 Comparative Example 1-Mip-NeRF 32.63 30.44 33.42 34.50 34.25 33.05 Comparative Example 1 - Robust-NeRF 29.10 25.31 31.32 33.70 32.45 30.38 Example 1 36.98 28.30 36.50 37.11 38.04 35.39

[0176] Table 3 shows the structural similarity in the NeRF_Shadow dataset.

[0177]

[0178]

[0179] Table 4 shows the structural similarity in the NeRF_Synthetic dataset.

[0180] Comparative Example 1-NeRF 0.967 0.925 0.964 0.974 0.961 0.958 Comparative Example 1 - Instant-NGP - - - - - - Comparative Example 1-NeRF-W 0.931 0.811 0.874 0.933 0.949 0.900 Comparative Example 1-Mip-NeRF 0.988 0.945 0.984 0.988 0.984 0.978 Comparative Example 1 - Robust-NeRF 0.753 0.653 0.792 0.852 0.835 0.777 Example 1 0.952 0.940 0.962 0.970 0.986 0.962

[0181] As shown in Table 1, the method in Example 1 has the highest average peak signal-to-noise ratio (PSNR). Except for a slight decrease of 0.28 dB compared to NeRF-W in the Tiguan car scene, the PSNR values ​​in other scenes of the NeRF_Shadow dataset are higher than other methods. This indicates that the algorithm in this paper performs better than other methods in disturbed scenes. In terms of mean values, compared to the robust models NeRF-W and Robust-NeRF, which are designed for transient object interference, the mean values ​​are improved by 3.08 dB and 2.39 dB respectively, indicating that it achieves good results in noise processing in the scene. It is also easy to see that Robust-NeRF decreases by 2.61 dB compared to Mip-NeRF in scenes like Pan (where image acquisition is good), while the method in Example 1 improves by 1.13 dB compared to Mip-NeRF. This reflects its ability to maintain good performance under various noise levels in real-world acquisition scenarios.

[0182] As shown in Table 2, the method in Example 1 still has the highest average PSNR across all scenarios, at 35.39. However, the robust models NeRF-W and Robust-NeRF exhibit different results compared to the NeRF_Shadow dataset; NeRF-W performs worse than Instant-NGP, and Robust-NeRF performs worse than Mip-NeRF.

[0183] As shown in Table 3, the method in Example 1 outperforms other algorithms in both individual scenarios and average performance, demonstrating superior performance. Compared to other algorithms, its average SSIM value is improved by 2.34% to 10.08%. Since the Instant-NGP wrapper tools do not output the relevant SSIM values, they are omitted here.

[0184] As shown in Table 4, the method in Example 1 is second only to Mip-NeRF and slightly better than the original NeRF. This is partly due to the lack of noise in the image, and partly due to the multi-scale representation method introduced by Mip-NeRF, which achieves multi-scale sampling by rendering anti-aliased truncated cones. This method effectively reduces aliasing and further improves image quality. In contrast, the method in Example 1 mainly relies on an adaptive loss function for updates in noise-free scenes, thus showing no significant performance degradation in weakly noisy scenes and even a slight improvement over the original NeRF. Robust-NeRF, on the other hand, still shows a significant performance degradation in weakly noisy scenes.

[0185] Experimental Example 2

[0186] Comparing the results of Example 2 and Comparative Example 2, Table 5 shows the average PSNR for rendering each scene under sparse input. Figure 3 A two-dimensional display image for rendering effects.

[0187] Table 5. Average PSNR under Sparse Input

[0188]

[0189] As can be seen from Table 5, under the same sparse input settings, the PSNR value of the method in Example 2 is higher than that of other methods, which directly reflects its advantages in preserving image details and reducing noise.

[0190] from Figure 2 As can be seen, with only two input images, the method in Example 2 can only render clear effects in some scenes. Although some blurring and distortion still exist in many details, the rendering quality is significantly improved compared to the extremely blurry rendering effect of NeRF. Especially on the real dataset NeRF_Shadow, due to the lack of carefully designed sampling conditions in the laboratory like NeRF_Synthetic, the NeRF rendering effect makes it extremely difficult to obtain information, and even more difficult to make a difference in downstream tasks. In contrast, the method in Example 2, except for some details, presents the shape information of objects and the co-constructed structure of scene and objects better, demonstrating the advantage of 3D models over 2D images. With the increase of the number of input images, the rendering quality of both NeRF and the method in Example 2 is significantly improved when inputting 5 and 10 images. However, there are still many artifacts or floating objects in the NeRF model, while the method in Example 2 can produce more refined and realistic rendering effects, with clearer and more accurate details, and in multi-view... Figure 1 The consistency performance is excellent. This also verifies the effectiveness of the depth supervision module in Example 2 for rendering results.

[0191] The present invention has been described above with reference to preferred embodiments; however, these embodiments are merely exemplary and illustrative. Various substitutions and modifications can be made to the present invention based on these embodiments, all of which fall within the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction method based on neural radiation fields, characterized in that, Includes the following steps: S1. Perform motion structure restoration on the input image to obtain sparse point cloud of the scene and camera pose information corresponding to the input image, and sample along the light ray emanating from the camera. S2. Based on the sampled data, render the color of the two-dimensional pixel corresponding to the light, and obtain the color residual of the image based on the color of the rendered pixel and the color in the original image; Obtain the number of features in the image that match the feature points of the sparse initial point cloud; S3. If the number of matching features in the image is less than the first threshold and the color residual is greater than the second threshold, then the image is extracted; the image is then marked as a noisy image. Otherwise, label the image as a weak noise image; S4. When the proportion of strong noise images exceeds the third threshold, the weight of strong noise images is set to zero in the loss function, the neural radiation field network is trained, and the color of the observation view with zero weight is updated. When the proportion of strong noise images does not exceed the third threshold, the loss function is set to a robust and adaptive loss function, and the neural radiation field network is trained to update the color at the corresponding viewpoint. S5. The trained neural radiation field network is used to identify the input image and obtain the color images of the object from different perspectives. In S4, when the proportion of noisy images exceeds the third threshold, the loss function... Set to: in, This represents any ray of light originating from the camera, with each ray corresponding to a pixel in the image. express The set, Indicates light rays The corresponding real pixel color value, This indicates that the light is sampled coarsely along the ray. r Predicted rendering colors This indicates that fine sampling is performed along the light ray. Predicted rendering colors Denotes the squared L2 norm. The weight function represents the binarization. This represents the currently input image. This represents a set of images with strong noise.

2. The three-dimensional reconstruction method based on neural radiation field according to claim 1, characterized in that, Before S1, there is a step S0, which preprocesses the input: If the input is an image, no preprocessing is required; If the input is in video format, the video will be split into image frames.

3. The three-dimensional reconstruction method based on neural radiation field according to claim 1, characterized in that, In S4, when the proportion of noisy images does not exceed the third threshold, the loss function... Set to: in, For hyperparameters, This represents the sum of pixel losses in a single image. This is a hyperparameter.

4. The three-dimensional reconstruction method based on neural radiation field according to claim 3, characterized in that, During the training, The value changes continuously from large to small.

5. The three-dimensional reconstruction method based on neural radiation field according to claim 1, characterized in that, In step S0, depth information of the image is also obtained, and in step S4, the depth information is used as a supervision signal for training the neural radiation field network.

6. The three-dimensional reconstruction method based on neural radiation field according to claim 1, characterized in that, In S2, for feature points in the image that do not match the sparse initial point cloud, their depth information is used to guide sampling.

7. A three-dimensional reconstruction device based on neural radiation fields, used to acquire input images as described in any one of claims 1-6, characterized in that, Includes turntable 1 and camera arm 2, The turntable 1 is a rotatable platform used to hold the sample to be reconstructed; The camera arm 2 is equipped with multiple cameras 21, which are fixed to the side of the turntable 1 and used to photograph the samples on the turntable 1.

8. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

9. A computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images