Multi-view 3D reconstruction method based on deep residual and neural implicit surface learning
Through a multi-view 3D reconstruction method of deep residual and neural implicit surface learning, combined with ResNet-50 network and SDF function, the problem of high-fidelity reconstruction in large-scale complex scenarios is solved, and high-quality 3D surface reconstruction and fine geometric details are achieved. Capture improves the efficiency of computing resources utilization.
Patent Information
- Application Number
- CN202311857090.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-12-29
AI Technical Summary
Existing multi-view 3D reconstruction methods cannot achieve high-fidelity surface reconstruction in large-scale complex scenarios, especially in extreme weather and long-distance sparse view scenarios, and neural rendering performance is hindered by view sparsity and complex content.
Using a multi-view three-dimensional reconstruction method based on depth residual and neural implicit surface learning, deep features are extracted through the ResNet-50 network, combined with SDF function and Color function, photometric consistency and point constraints are introduced, and SDF network is optimized to achieve high-quality surfaces. reconstruction.
It realizes high-quality three-dimensional surface reconstruction of large-scale scenes, captures more refined 3D geometric details, significantly improves the accuracy of the three-dimensional surface, solves the geometric blur problem in neural rendering, and improves the utilization efficiency of computing resources.
Smart Images

Figure CN117830520B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a multi-view Figure 3 The invention discloses a three-dimensional reconstruction method, and belongs to the technical field of three-dimensional reconstruction. Background Art
[0002] The purpose of 3D reconstruction is to restore accurate scene geometry information from multiple images observed from different perspectives. The scene geometry information can be used for scene representation in virtual reality or environment mapping for autonomous robot navigation. At the same time, 3D reconstruction technology based on multiple views can also serve the fields of digital reconstruction of cultural relics, traffic accident analysis, and reproduction of other architectural sites.
[0003] Traditional multi-view Figure 3 The 3D reconstruction algorithm is a combination of Structure-from-Motion (SFM) and Multi-view Stereo (MVS). Although it has achieved good reconstruction results, it inevitably introduces cumulative errors to the geometric structure information reconstructed in the end due to the cumbersome steps. In addition, an inherent disadvantage of this traditional algorithm is that it cannot handle sparse blurred views, such as large areas with uniform colors, complex texture areas, or remote sensing scenes taken from a distance.
[0004] In order to reduce the deviation caused by traditional multi-view reconstruction methods, existing 3D reconstruction methods represent the scene geometry information as a neural implicit surface and use volume rendering to optimize the surface because volume rendering is more robust than surface rendering. The results show that the effect is good in indoor datasets (DTU) or some small outdoor scene datasets shot at close range (we list some data in the BlendedMVS dataset), which optimizes the deviation caused by traditional methods to a certain extent. However, using only the color information synthesized by volume rendering to optimize the surface structure of the scene still has certain difficulties in processing data in extreme weather conditions (cloudy or foggy, dark or daytime) and remote sensing scene data with sparse views at a long distance. In addition, existing methods encounter challenges when capturing complex details in large-scale scenes. The performance of neural rendering is obviously hindered by the sparsity of views and the complex content inherent in such scenes. Summary of the invention
[0005] The purpose of the present invention is to solve the problem that high-fidelity surface reconstruction cannot be achieved in large-scale complex scenes during the existing process of surface reconstruction through neural rendering of images, and to provide a multi-viewing method based on depth residual and neural implicit surface learning. Figure 3 Dimensional reconstruction method.
[0006] Multi-view based on deep residual and neural implicit surface learning Figure 3 The reconstruction method includes:
[0007] S1. Using a camera to acquire images of multiple perspectives to be three-dimensionally reconstructed;
[0008] S2, using the motion recovery structure SFM to perform image feature extraction, feature matching and sparse point cloud reconstruction on the image acquired by S1, generating a sparse point cloud with three-dimensional coordinate information and generating the camera's position information;
[0009] S3, assuming that the light emitted from the camera is incident on the scene surface, obtaining the three-dimensional coordinate information of the sampling point set on the scene surface;
[0010] S4, inputting the three-dimensional coordinates of the sparse point cloud obtained in S2, the position information of the camera, and the three-dimensional coordinate information of the sampling point set obtained in S3 into the SDF function to generate point cloud information represented by the SDF function;
[0011] S5, uses ResNet-50 convolutional block to extract the appearance information of the image obtained in S1;
[0012] S6, inputting the appearance information obtained by S5, the three-dimensional coordinates of the sparse point cloud obtained by S2, and the three-dimensional coordinate information of the sampling point set obtained by S3 into the Color function to generate the color value of the point cloud;
[0013] S7, setting constraints based on the SDF function point cloud information obtained in S4 and the color value of the sparse point cloud obtained in S7, and calculating the color loss function;
[0014] The weight constraint function is calculated based on the weight constraint condition using the SDF function point cloud information obtained by S4;
[0015] Set constraints based on the point cloud color information obtained by S6 and calculate the photometric consistency loss function;
[0016] Based on the SDF function point cloud information obtained by S4, the SDF loss function of the point is calculated according to the constraint condition that the surface point SDF=0;
[0017] S8, using the loss function obtained in S7 to reversely optimize the SDF network, convert the SDF into mesh and texture, and directly calculate the SDF value of each sampled voxel;
[0018] S9. Use the Matching Cubes algorithm to extract the zero isosurface from the SDF value obtained in S8, directly output the 3D surface, and complete the 3D reconstruction.
[0019] Preferably, the specific method of using the ResNet-50 convolution block in S5 to extract the appearance information of the image obtained in S1 includes:
[0020] Extract shallow features of the appearance in the image, input the image into the ResNet-50 convolutional block, output a 1×1×256 feature tensor, and complete the appearance embedding;
[0021] The ResNet-50 convolutional block introduces an inverted residual structure as a bottleneck layer in the residual structure to reduce the number of parameters;
[0022] The bottleneck layer first passes through a 1×1 convolution kernel, then a 3×3 convolution kernel, and finally a 1×1 convolution kernel;
[0023] The 256-dimensional input first passes through a 1×1×64 convolutional layer, then a 3×3×64 convolutional layer, and finally a 1×1×256 convolutional layer. Each convolutional layer is obtained using the ReLU activation function.
[0024] Preferably, the constraints of the three-dimensional reconstruction include color constraints, weight constraints, point constraints and photometric consistency constraints.
[0025] Preferably, the specific method of calculating the weight constraint function according to the weight constraint condition based on the SDF function point cloud information obtained in S4 in S7 includes:
[0026] Find two adjacent sampling points near the surface along the ray emitted from the camera. The SDF values of the two sampling points satisfy:
[0027] f(p(t i ))·f(p(t i+1 ))<0
[0028] Among them, p(t i ) represents the first sampling point t between two adjacent sampling points i The initial point set, f(p(t i )) represents p(t i ), p(t i+1 ) represents the other sampling point t between two adjacent sampling points i+1 The initial point set, f(p(t i+1 )) represents p(t i+1 )’s SDF value;
[0029] The first intersection point of the ray and the surface is approximated by linear interpolation as:
[0030]
[0031]
[0032] in, represents the point set obtained by linear interpolation, t j+1 It means p(t j+1) point, f(p(t j+1 )) represents p(t j+1 ) point SDF value, t j It means p(t j ) point, f(p(t j )) represents p(t j ) point SDF value;
[0033] v represents the unit direction vector of the ray;
[0034] Will Add to p(t i ), we get a new point set P:
[0035]
[0036] Represents the point set obtained after interpolation;
[0037] Use the newly generated point set P to re-render the volume and generate the final color:
[0038]
[0039] Among them, w(t i ) represents the weight of the initial point set, c(t i ) represents the pixel value of the initial point set, Represents the weight of the point set after linear interpolation, It represents the pixel value of the point set after linear interpolation, and n represents the number of points;
[0040] Color deviation ΔC final for:
[0041]
[0042] Among them, c(t * ) represents the pixel value of the intersection of the ray and the surface, ε interp represents the deviation caused by linear interpolation, ε weight-final represents weight bias;
[0043] Weight Constrained Loss:
[0044]
[0045] Preferably, the specific method of setting constraint conditions based on the point cloud color information obtained in S6 and calculating the photometric consistency loss function in S7 includes:
[0046] Take a small part S of the surface, and the pixel of S on the projection of the source view is q. Then, except for the occluded part, the image blocks of the small part S of the surface corresponding to different source views are geometrically consistent;
[0047] With reference image pixel I r The camera coordinate system is used to represent S, and we have:
[0048]
[0049] Where d represents the normal direction, n T Indicates that the SDF network is Normals calculated by automatic differentiation of points;
[0050] Let a point x in the reference image i The pixel values of x are located using the unit matrix H, so that i Connecting with the corresponding points x in other images, we get the relationship:
[0051]
[0052] Among them, K r and K i represents the camera internal calibration matrix, R r and R i represents the rotation matrix, t i and t r Represents the translation vector, corresponding to the source view I i and I r ;H i X represents the source image pixel, H i Represents the identity matrix corresponding to the pixel at a certain point in the source image;
[0053] Take the rendered image as the reference image and calculate the normalized cross-correlation between the reference image and the source view:
[0054]
[0055] Among them, Cov() represents covariance, Var() represents variance, NCC(X,H i X) represents the NCC score between the sampled patch and the corresponding patches on all source images, X represents the sampled patch pixel value,
[0056] Get the best four NCC scores. The NCC score range is -1 to 1, where -1 represents irrelevant and 1 represents relevant. The larger the NCC score, the better.
[0057] Compute the photometric consistency loss for the corresponding view
[0058]
[0059] Where i represents the number of NCC scores.
[0060] Preferably, the specific method of calculating the SDF loss function of the point in S7 based on the SDF function point cloud information obtained in S4 according to the constraint condition of setting the surface point SDF=0 includes:
[0061] Point Constraint Loss
[0062]
[0063] Among them, SFM reconstructs the sparse 3D point P0, the direct supervision of P0 is f(p), P represents the spatial position of the point, and N represents the number of points.
[0064] Preferably, the three-dimensional reconstruction in S7 includes calculating a total loss function, and the specific method thereof includes:
[0065] Total loss function is the color constraint loss Weight Constrained Loss Point Constraint Loss and photometric consistency constraint loss The weighted sum of:
[0066]
[0067] Among them, λ represents the weight of the weight constraint loss function, α represents the weight of the photometric consistency constraint loss function, and β represents the weight of the point constraint loss function.
[0068] Preferably, the SDF function is:
[0069] The surface S of the scene is represented as:
[0070] S = {p∈R 3 |f(p)=0}
[0071] Where f(p) represents the function where f(p) = 0 on the surface of the observed object, p represents the spatial position of a point, and R represents the signed distance;
[0072] This function is represented by a neural network, which is named SDF network in NeuS. The SDF network function is associated with NeRF in NeuS, and the loss function of NeRF is used to optimize the surface representation SDF network.
[0073] The multi-view algorithm based on deep residual and neural implicit surface learning proposed in this paper Figure 3 The dimensional reconstruction method has the following advantages:
[0074] 1. The ResNet-50 network can be used to extract deep features from images, improve the ability of neural surface rendering, and achieve high-quality 3D surface reconstruction of large-scale scenes;
[0075] 2. Embed appearance information into the neural radiation field to learn accurate posture information and locate 3D surfaces in a coarse-to-fine manner; also add surface rendering to improve the original single rendering framework, making the rendering process close to unbiased, and reversely optimize the SDF network by reducing the difference between the rendered color and the real color of the surface;
[0076] 3. Interpolate nearby points on the surface and optimize their weights to capture finer 3D geometric details in the reconstruction result;
[0077] 4. The introduction of photometric consistency and point constraints significantly improves the accuracy of 3D surfaces; the introduction of multi-view stereo matching to constrain geometry solves the geometric ambiguity problem that exists when only color information is used to optimize the scene geometry structure;
[0078] 5. Automatically sample the geometric regions of interest to effectively achieve coarse 3D geometric positioning and make efficient use of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0079] Figure 1 The multi-view algorithm based on deep residual and neural implicit surface learning of the present invention is Figure 3 Principle block diagram of the dimensional reconstruction method;
[0080] Figure 2 This is the principle block diagram of the appearance embedding algorithm;
[0081] Figure 3 This is the ResNet-50 network structure diagram;
[0082] Figure 4 It is the DVGO geometric coarse sampling strategy;
[0083] Figure 5 It is the Res-NeuS geometric coarse sampling strategy;
[0084] Figure 6 It is the qualitative surface reconstruction result of indoor small scene data;
[0085] Figure 7 It is the quantitative surface reconstruction result of large-scale scenes in the BlendedMVS dataset;
[0086] Figure 8 is the directional surface reconstruction result of the ablation study. DETAILED DESCRIPTION
[0087] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0088] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0089] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.
[0090] Embodiment 1:
[0091] Combine the following Figure 1-Figure 3 This embodiment describes a multi-view algorithm based on depth residual and neural implicit surface learning. Figure 3 The reconstruction method includes:
[0092] S1. Using a camera to acquire images of multiple perspectives to be three-dimensionally reconstructed;
[0093] S2, using the motion recovery structure SFM to perform image feature extraction, feature matching and sparse point cloud reconstruction on the image acquired by S1, generating a sparse point cloud with three-dimensional coordinate information and generating the camera's position information;
[0094] S3, assuming that the light emitted from the camera is incident on the scene surface, obtaining the three-dimensional coordinate information of the sampling point set on the scene surface;
[0095] S4, inputting the three-dimensional coordinates of the sparse point cloud obtained in S2, the position information of the camera, and the three-dimensional coordinate information of the sampling point set obtained in S3 into the SDF function to generate point cloud information represented by the SDF function;
[0096] S5, uses ResNet-50 convolutional block to extract the appearance information of the image obtained in S1;
[0097] S6, inputting the appearance information obtained by S5, the three-dimensional coordinates of the sparse point cloud obtained by S2, and the three-dimensional coordinate information of the sampling point set obtained by S3 into the Color function to generate the color value of the point cloud;
[0098] S7, setting constraints based on the SDF function point cloud information obtained in S4 and the color value of the sparse point cloud obtained in S7, and calculating the color loss function;
[0099] The weight constraint function is calculated based on the weight constraint condition using the SDF function point cloud information obtained by S4;
[0100] Set constraints based on the point cloud color information obtained by S6 and calculate the photometric consistency loss function;
[0101] Based on the SDF function point cloud information obtained by S4, the SDF loss function of the point is calculated according to the constraint condition that the surface point SDF=0;
[0102] S8, using the loss function obtained in S7 to reversely optimize the SDF network, convert the SDF into mesh and texture, and directly calculate the SDF value of each sampled voxel;
[0103] S9. Use the Matching Cubes algorithm to extract the zero isosurface from the SDF value obtained in S8, directly output the 3D surface, and complete the 3D reconstruction.
[0104] Furthermore, the specific method of using the ResNet-50 convolution block in S5 to extract the appearance information of the image obtained in S1 includes:
[0105] Extract shallow features of the appearance in the image, input the image into the ResNet-50 convolutional block, output a 1×1×256 feature tensor, and complete the appearance embedding;
[0106] The ResNet-50 convolutional block introduces an inverted residual structure as a bottleneck layer in the residual structure to reduce the number of parameters;
[0107] The bottleneck layer first passes through a 1×1 convolution kernel, then a 3×3 convolution kernel, and finally a 1×1 convolution kernel;
[0108] The 256-dimensional input first passes through a 1×1×64 convolutional layer, then a 3×3×64 convolutional layer, and finally a 1×1×256 convolutional layer. Each convolutional layer is obtained using the ReLU activation function.
[0109] In this implementation, the total number of parameters is: 256×1×1×64+64×3×3×64+64×1×1×256=69632.
[0110] Furthermore, the constraints of 3D reconstruction include color constraints, weight constraints, point constraints and photometric consistency constraints.
[0111] Furthermore, the specific method of calculating the weight constraint function according to the weight constraint condition using the SDF function point cloud information obtained in S4 in S7 includes:
[0112] Find two adjacent sampling points near the surface along the ray emitted from the camera. The SDF values of the two sampling points satisfy:
[0113] f(p(t i ))·f(p(t i+1 ))<0
[0114] Among them, p(t i ) represents the first sampling point t between two adjacent sampling points i The initial point set, f(p(t i )) represents p(t i ), p(t i+1 ) represents the other sampling point t between two adjacent sampling points i+1 The initial point set, f(p(t i+1 )) represents p(t i+1 )’s SDF value;
[0115] The first intersection point of the ray and the surface is approximated by linear interpolation as:
[0116]
[0117]
[0118] in, represents the point set obtained by linear interpolation, t j+1 It means p(t j+1 ) point, f(p(t j+1 )) represents p(t j+1 ) point SDF value, t j It means p(t j ) point, f(p(t j )) represents p(t j ) point SDF value;
[0119] v represents the unit direction vector of the ray;
[0120] Will Add to p(t i ), we get a new point set P:
[0121]
[0122] Represents the point set obtained after interpolation;
[0123] Use the newly generated point set P to re-render the volume and generate the final color:
[0124]
[0125] Among them, w(t i ) represents the weight of the initial point set, c(t i ) represents the pixel value of the initial point set, Represents the weight of the point set after linear interpolation, It represents the pixel value of the point set after linear interpolation, and n represents the number of points;
[0126] Color deviation ΔC final for:
[0127]
[0128] Among them, c(t * ) represents the pixel value of the intersection of the ray and the surface, ε interp represents the deviation caused by linear interpolation, ε weight-final represents weight bias;
[0129] Weight Constrained Loss:
[0130]
[0131] Furthermore, the specific method of setting constraint conditions based on the point cloud color information obtained in S6 and calculating the photometric consistency loss function in S7 includes:
[0132] Take a small part S of the surface, and the pixel of S on the projection of the source view is q. Then, except for the occluded part, the image blocks of the small part S of the surface corresponding to different source views are geometrically consistent;
[0133] With reference image pixel I r The camera coordinate system is used to represent S, and we have:
[0134]
[0135] Where d represents the normal direction, n T Indicates that the SDF network is Normals calculated by automatic differentiation of points;
[0136] Let a point x in the reference image i The pixel values of x are located using the unit matrix H, so that i Connecting with the corresponding points x in other images, we get the relationship:
[0137]
[0138] Among them, K r and K i represents the camera internal calibration matrix, R r and R i represents the rotation matrix, t i and t r Represents the translation vector, corresponding to the source view I i and I r ;H i X represents the source image pixel, H iRepresents the identity matrix corresponding to the pixel at a certain point in the source image;
[0139] Take the rendered image as the reference image and calculate the normalized cross-correlation between the reference image and the source view:
[0140]
[0141] Among them, Cov() represents covariance, Var() represents variance, NCC(X,H i X) represents the NCC score between the sampled patch and the corresponding patches on all source images, X represents the sampled patch pixel value,
[0142] Get the best four NCC scores. The NCC score range is -1 to 1, where -1 represents irrelevant and 1 represents relevant. The larger the NCC score, the better.
[0143] Compute the photometric consistency loss for the corresponding view
[0144]
[0145] Where i represents the number of NCC scores.
[0146] Furthermore, in S7, the specific method of calculating the SDF loss function of the point according to the constraint condition of setting the surface point SDF=0 based on the SDF function point cloud information obtained in S4 includes:
[0147] Point Constraint Loss
[0148]
[0149] Among them, SFM reconstructs the sparse 3D point P0, the direct supervision of P0 is f(p), P represents the spatial position of the point, and N represents the number of points.
[0150] Furthermore, the three-dimensional reconstruction in S7 includes calculating a total loss function, and the specific method thereof includes:
[0151] Total loss function is the color constraint loss Weight Constrained Loss Point Constraint Loss and photometric consistency constraint loss The weighted sum of:
[0152]
[0153] Among them, λ represents the weight of the weight constraint loss function, α represents the weight of the photometric consistency constraint loss function, and β represents the weight of the point constraint loss function.
[0154] Furthermore, the SDF function is:
[0155] The surface S of the scene is represented as:
[0156] S = {p∈R 3 |f(p)=0}
[0157] Where f(p) represents the function where f(p) = 0 on the surface of the observed object, p represents the spatial position of a point, and R represents the signed distance;
[0158] This function is represented by a neural network, which is named SDF network in NeuS. The SDF network function is associated with NeRF in NeuS, and the loss function of NeRF is used to optimize the surface representation SDF network.
[0159] In this paper, a multi-view deep learning algorithm based on deep residual and neural implicit surface learning is proposed. Figure 3 Dimensional reconstruction method for high-fidelity surface reconstruction of complex scenes from multiple views.
[0160] The SDF network is used to locate the zero level set of the three-dimensional surface, and the volume rendering color network is forward optimized through image appearance embedding. Surface rendering is also added to improve the original single rendering framework, making the rendering process close to unbiased, and the SDF network is reversely optimized by reducing the difference between the rendered color and the real color of the surface. At the same time, in order to solve the geometric ambiguity problem existing in optimizing the scene geometry structure using only color information, multi-view stereo matching is introduced to constrain the geometry. In addition, in order to efficiently utilize computing resources and view dependence, a coarse sampling method of the point cloud of interest is designed to automatically filter out the point cloud input that is not of interest.
[0161] In the present invention, the deviation of volume rendering is analyzed; an appearance embedding optimization color field function is proposed; surface rendering and volume rendering are combined to make the rendering result close to unbiased; a multi-view stereo matching mechanism is introduced to constrain the three-dimensional geometric structure; and a new geometric coarse sampling strategy is proposed.
[0162] Multi-view surface reconstruction is a complex process that aims to restore the accurate geometric surface of a 3D object from images acquired from multiple viewpoints. Early image-based photogrammetry techniques used a method called volume occupancy grid to represent the scene. This method works by inspecting each cube (or voxel) and marking it as occupied if it meets strict color consistency conditions between the corresponding projected image pixels. However, the feasibility of this method is limited by the assumption of photometric consistency, as in practical applications, color consistency cannot be maintained due to automatic exposure variations and the presence of non-Lambertian materials.
[0163] Subsequent methods use 3D point clouds generated by multi-view stereo techniques and then perform dense surface reconstruction. However, reliance on point cloud quality often leads to loss of surface details or the presence of noise since point clouds are usually sparse. Recently, learning-based methods have attempted to improve the point cloud generation process by training neural networks. These methods learn image features and construct cost volumes to improve the quality and density of point clouds, but they are still limited by the resolution of the cost volume and cannot fully recover the geometric details of complex objects.
[0164] Surface rendering: The color of the cast light depends only on the color of the intersection of the light and the scene geometry. This makes the gradient back-propagated only to the local area near the intersection. Therefore, surface-based reconstruction methods have difficulty reconstructing complex objects with severe self-occlusion and sudden depth changes, and require object masks as supervision.
[0165] Volume rendering: is an image-based rendering method that renders a 3D scalar field into a 2D image by casting rays along a 3D volume. Like NeRF, it renders images by integrating the color of the sampled points of each ray, which can handle objects with sudden depth changes and synthesize high-quality images. However, it is difficult to extract high-fidelity surfaces from the learned implicit field because the dense scene representation is difficult to constrain its level set. Therefore, more direct surface modeling is the preferred photogrammetric surface reconstruction problem.
[0166] Neural implicit fields are a new way to represent the geometry of objects. During the reconstruction process, a neural network is usually trained to fit an implicit function that accepts 3D coordinates as input and outputs surface feature values, such as distance or color. This implicit function can be regarded as an implicit representation of the entire 3D surface. For example, [11,34-41], in order to more accurately define the scene representation of the 3D surface, implicit functions such as occupancy grids [32,33] or signed distance functions are better than simple density fields.
[0167] NeuS is a typical neural implicit surface reconstruction method that applies volume rendering methods [1,14-18] to learn implicit SDF representations. It is worth noting that simply applying standard volume rendering methods to density values related to SDF will lead to serious deviations in the reconstructed geometric surface, that is, the pixel weights when the volume density is the largest are not on or near the surface of the object. Therefore, NeuS constructs a new expression of volume density function and weight function to meet the requirement that when the volume density is the same, the weight assigned to the pixel at different distances from the camera should be different.
[0168] In NeuS, SDF-based volume rendering is very beneficial for recovering surfaces from 2D images, especially for some small indoor scene datasets. However, it is still challenging to achieve high-quality 3D surface reconstruction for some outdoor scenes with low visibility and remote sensing scenes, because the view sparsity of these scene features can cause severe geometric deformation or distortion. In addition, the biases existing in the current volume rendering paradigm (biases caused by sampling operations and weight biases) are greatly amplified when applied to such scenes.
[0169] Based on NeRE and its extension NeuS.
[0170] The surface S of the scene is represented as follows:
[0171] S = {p∈R 3 |f(p)=0}
[0172] Here f(p) is a function where f(p)=0 only on the surface of the observed object. This function can be represented by a neural network, which is named SDF network in NeuS. In NeuS, the SDF network function is associated with NeRF, so that the loss function of NeRF can be used to optimize the surface representation SDF network.
[0173] The volume rendering formula of the classic NeRF is:
[0174]
[0175] where p(t) = o + tv,
[0176] To accurately describe the volume density, the volume density must be maximum on or near the surface (that is, when f(x) = 0, σ(x) also reaches its maximum value), so NeuS redefined the expression of volume density as φ s (u) = se -su / (1+e -su )where u=f(x), where the volume density expression φ s (f(x)) is called S-density, and when it is put into the rendering formula, we get
[0177]
[0178] where p(t) = o + tv
[0179] Let w(t) = T(t)φ s(f(p(t))), and the w(t) function must satisfy the second property: when the volume density is the same, the weight assigned to the pixel at the point should be different when the distance from the camera is different, otherwise there will be ambiguity. In other words, the influence of T(t) must be taken into account, so the weight function is normalized:
[0180]
[0181] Let w(t) = T(t)ρ(t), and Therefore, T(t) and ρ(t) are solved, and NeuS now completes the perfect combination of NeRF and surface reconstruction.
[0182] NeuS scene representation is a pair of multi-layer perceptrons (MLPs). The first MLP receives sparse 3D points and camera position information x, outputs S-density, and a feature vector, which is sent to the second MLP with the 2D viewing direction d, outputting color. This architecture ensures that different colors are output when viewed from different angles, using color to constrain the geometric structure, but the underlying shape representation is only a function of position. Therefore, only the feature encoding corresponding to the sparse 3D points is considered and the interval length between sample points (sampling error) is ignored.
[0183] In the volume rendering process, it is assumed that when approaching the surface along the ray, the color weight distribution used to calculate the color integral reaches the maximum value at the position where SDF is 0, that is, the surface position. This situation is usually used for data sets with simple geometry, such as DTU data sets. However, in the face of some data sets with low visibility, long-distance sparse views and complex geometry, the position with the maximum color weight often deviates from the position where SDF is 0. Therefore, the color weight deviation will definitely weaken the geometric constraint ability.
[0184] We set C S is the color where the light intersects the surface of the object, C V is the color of the view after rendering. Represents the geometric surface. For neural rendering, SDF is often obtained through inference through an MLP network, and the color field is obtained through another MLP network, which can be mathematically expressed as:
[0185] sdf(p)=F Θ (p)
[0186]
[0187] At the same time, the composite view color after rendering is written in discrete form as:
[0188]
[0189] Assume that the initial intersection point of the ray and the surface is p(t * ), sdf(t * )=0, the color of the surface in rendering can be expressed as:
[0190] C S =c(t * )
[0191] For synthesizing a new view, the goal is to make the color of the synthesized view consistent with the target color
[0192]
[0193] p(t j ) is the distance from p(t * ) the nearest sampling point, ε sample represents the deviation caused by the sampling operation, ε weight Represents the deviation caused by volume rendering weighting.
[0194] For most neural rendering pipelines, the geometric information of objects is only constrained by the color loss of a single view in each iteration, and the consistency of different views in the direction of geometric optimization cannot be guaranteed, resulting in geometric ambiguity. When the input views become complex and sparse, the ambiguity of this geometric information will lead to inaccurate geometry. For remote sensing scenes with sparse images, this inherent ambiguity is undoubtedly a challenging problem.
[0195] Given a set of multi-view images with known camera poses, the goal is to reconstruct the surface without mask supervision by combining the advantages of neural rendering and volume rendering. The three-dimensional spatial field of the target is represented by a signed distance, and the corresponding surface is extracted using the zero level set of the SDF. During the rendering process, the goal is to optimize the signed distance function. A fused rendering method is proposed to eliminate the above-mentioned color weight bias and geometric bias: it not only combines the unbiased characteristics of surface rendering and the long-distance spatial receptive field method of volume rendering, but also introduces shallow appearance embedding of the image to constrain the ambiguity of the three-dimensional geometric shape, and finally introduces display SDF optimization to achieve geometric consistency. Methods such as Figure 1 shown.
[0196] Aiming at the unity bias of feature extraction from different views and considering that the data may be captured in different environments, shallow appearance features are extracted from each image to forward optimize the color field function, such as Figure 2 shown.
[0197] During the model training process, considering that the shallow features gradually disappear after repeated convolutions under normal circumstances, ResNet50 continuously adds the previous shallow features during backward training, which enriches the global nature of the features. Therefore, the useful features extracted from the image are input into ResNet50, and the output is a [1×1×256] feature tensor, which is passed to the second MLP neural network to complete the appearance embedding.
[0198] As shown in Table 1, this is the image network convolution result diagram:
[0199] Table 1
[0200]
[0201] Neus is used as the baseline and appearance embedding is added to the baseline. Both methods are evaluated on the BlendedMVS dataset to evaluate their mesh reconstruction effectiveness and view synthesis performance respectively.
[0202] Performance of NeuS and appearance-only embedding on the BlendedMVS dataset. Compared with NeuS, adding appearance embedding alone can significantly reduce surface noise and improve reconstruction accuracy.
[0203] Table 2 shows the quantitative results of the reconstructed surface.
[0204] Table 2
[0205]
[0206] Table 3 shows the quantitative indicators of the rendering results.
[0207] Table 3
[0208] method PSNR↑ SSIM↑ LPIPS↓ Baseline 28.24 0.909 0.077 Appearance Embed 33.99 0.982 0.039
[0209] In order to eliminate the deviation ε caused by the sampling operation sample , first find two adjacent sampling points near the surface along the ray emitted from the camera, and the SDF value satisfies:
[0210] f(p(t i ))·f(p(t i+1 ))<0
[0211] The first intersection of the ray and the surface is approximated by linear interpolation as
[0212]
[0213]
[0214] Then, the point set obtained by linear interpolation Add to the initial point set P(t i ), we get a new point set P = p(t * )∪P(t i ), use the newly generated point set P to re-render the volume to generate the final color:
[0215]
[0216] Then the color deviation becomes:
[0217]
[0218] After interpolation, although we get ε interp , ε interp is the deviation caused by linear interpolation, which is larger than the original sampling error ε sample At least two orders of magnitude smaller.
[0219] At the same time, the weight error is reduced accordingly to regularize the weight distribution:
[0220]
[0221] It is to delete abnormal weight distributions, such as those that are far away from the surface but have large weight values, thereby indirectly promoting the weight distribution to converge to the surface. Theoretically, when the weight tends to by The weight error ε is the center weight will tend to 0, and the final volume rendering result will produce the same result as the surface rendering.
[0222] Photometric consistency loss and point constraints are introduced to show the 3D representation of supervised SDF.
[0223] Take a small part S of the surface, whose pixel on the projection of the source view is q, then except for the occluded part, the image blocks of the small part S of the surface corresponding to different source views should be geometrically consistent. r The camera coordinate system is used to represent S, and we have:
[0224]
[0225] At this time, a certain point x in the reference image i The pixel value of can be located using a homography matrix H to associate it with the corresponding point x in the other image:
[0226]
[0227] Where K r and K i is the camera intrinsic calibration matrix, Rr and R i is the rotation matrix, t i and t r is the translation vector, corresponding to the source view I i and other views I r .
[0228] To measure the photometric consistency of different views, the normalized cross-correlation between the reference image and the source view is used.
[0229]
[0230] Where Cov represents covariance and Var represents variance. The rendered image is used as the reference image and the NCC scores between the sampled patch and the corresponding patches on all source images are calculated. To handle occlusion, the best four calculated NCC scores are found for each sampled patch and used to calculate the photometric consistency loss for the corresponding view:
[0231]
[0232] In the previous data processing process, images with known camera poses are required. The position information of these images is estimated by motion recovery structure (SFM). SDF also reconstructs sparse 3D points. Although these points will inevitably have noise, they have a certain accuracy. Due to the end-to-end architecture, the noise can be corrected to a certain extent. Therefore, these sparse 3D points P0 are used to directly supervise f(p).
[0233]
[0234] A scene is usually dominated by unoccupied space, which usually requires more computational resources and view dependence in the process of reconstructing the target scene. Based on this fact, the coarse 3D region of interest should be found efficiently before fine reconstruction. This will greatly reduce the number of query points on each ray in the later fine reconstruction stage.
[0235] When processing input data, the structure-from-motion method (SFM) is used to obtain the camera pose and sparse 3D point cloud. DVGO automatically selects the point cloud of interest based on the closest and farthest points where the rays emitted by each camera intersect the scene point cloud, such as Figure 4 and Figure 5 As shown, Figure 4 is the DVGO geometric coarse sampling strategy, Figure 5It is the Res-NeuS geometric coarse sampling strategy. Since the 3D point cloud area selected by DVGO is too large and limited, it is not accurate enough to locate the fine structure of the scene. Therefore, a new point cloud automatic filtering method is defined. The point cloud center is found through the camera pose information, and the average distance from the center to the camera position is calculated. The point cloud area around the center 360° is selected as the point cloud area of interest with the average distance as the radius. The radius r of the surrounding area is defined according to the camera shooting method (surrounding scene or long-distance panoramic coverage), as follows: Figure 5 shown.
[0236] The total loss is defined as the weighted sum of the individual losses:
[0237]
[0238] In this paper, the surface is reconstructed from 16 scenes of the BlendedMVS dataset to evaluate this method. The BlendedMVS dataset has scenes for remote sensing and other scenes with different categories of objects. The image resolution of these scenes is 768×576, and the number of views ranges from 56 to 333. The reconstructed surface is evaluated on the BlendedMVS dataset using the chamfer distance in 3D space, and the visual effect of the reconstructed surface is shown for the DTU dataset.
[0239] The distance metric is used to evaluate the accuracy of the three-dimensional reconstructed surface. The chamfer distance in 3D space is mainly used for reconstruction work and is defined as follows:
[0240]
[0241]
[0242] In the above formula, S1 is the ground truth sampling point, and S2 is the reconstructed surface sampling point. The chamfer distance from S1 to S2 is used as the evaluation index of reconstruction accuracy (Acc). On the contrary, the harmful distance from S2 to S is considered as the evaluation index of reconstruction completeness (Comp). The total score is defined as the average of accuracy and completeness. The smaller the distance, the better the reconstruction effect.
[0243] Furthermore, the performance of view synthesis similar to NeRE is evaluated by image quality assessment metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learning-Perceptual Image Patch Similarity (LPIPS).
[0244] The proposed method is compared with the most advanced learning-based method NeuS and the traditional reconstruction method colmap, and a comparative analysis is conducted from the perspective of the reconstruction effect and evaluation indicators of the models.
[0245] Assuming that the area to be reconstructed is within a sphere, 2048 rays are sampled per batch and the data is trained on a single NVIDIA GeForce RTX 4090 GPU. The model is trained for 50,000 iterations in about 4 hours to meet the memory requirements. After the network is trained, a mesh is extracted from the SDF within the predefined bounding box using the Marching Cube algorithm, with a volume size of 512.
[0246] First, several reconstruction methods are used to test some indoor scenes in the DTU dataset and some outdoor small scene data in the BlendedMVS dataset, and the reconstruction results are compared. Figure 6 As shown, the test results show that the method proposed in this application is much better than the baselines.
[0247] Since the method proposed in this application is good at reconstructing small scenes, the method proposed in this application is applied to some large scenes with low visibility and remote sensing scenes with sparse feature views in the BlendedMVS dataset, and the reconstruction results are compared and analyzed. Figure 7 From Table 4, we can see that the surface reconstructed by COLMAP is noisy, while NeuS only uses color constraints, which leads to serious deformation, distortion and holes in the geometric surface structure. In contrast, the method proposed in this application can not only reconstruct accurate geometric structures but also remove smooth surface noise. For example, it can reconstruct the geometric structure of scene7 (low visibility) and restore the depth mutation of scene8.
[0248] Table 4
[0249] method NeuS COLMAP This application Scene 3 0.125 0.119 0.117 Scene 4 0.00128 0.0425 0.000810 Scene 5 0.297 1.75 0.0877 Scene 6 1.18 0.165 0.581 Scene 7 3.475 0.385 0.469 Scene 8 0.285 0.242 0.237 Scene 9 2.11 1.89 1.84 Scene 10 0.425 0.87 0.421 Scene 11 1.32 0.692 0.510 Scene 12 0.990 0.921 0.102 Scene 13 0.121 0.234 0.061 Scene 14 0.540 0.370 0.319 Scene 15 1.306 0.454 0.311 Scene 16 0.213 0.232 0.120 average value 0.884 0.597 0.369
[0250] Use the dome church data in the BlindedMVS dataset for ablation experiments, take NeuS as the baseline, and add these modules in turn, such as Figure 8 As shown in the figure, the Baseline geometry is distorted, the surface produces a lot of noise, and the reconstructed area is incomplete. Model A reconstructs the complete area, but the surface noise is still large. Model B not only reconstructs the complete area, but also significantly optimizes the geometry. Model C improves the geometry, and the error is similar to that of Model B. In contrast, the Full model shows excellent results in both reconstructing accurate geometry and reducing surface noise.
[0251] Table 5 shows the quantitative results of the ablation model.
[0252] Table 5
[0253] method Appearance Constraints Weight Constraint Geometric constraints Completeness Baseline 0.2134 Model-A √ √ 0.1725 Model-B √ √ 0.1289 Model-C √ √ 0.1275 Full Model √ √ √ 0.1203
[0254] In summary, appearance embedding tends to capture scene details, geometric constraints improve the quality of geometric reconstruction to a certain extent, and weight constraints effectively improve model accuracy.
[0255] In summary, this application proposes a Res-NeuS neural surface photogrammetry reconstruction method. Res-NeuS achieves surface reconstruction accuracy for large-scale scenes. Res-NeuS unlocks the ability of ResNet-50 to extract deep image information for neural surface reconstruction modeled as SDF. This application shows that Res-NeuS can skillfully reconstruct complex scene structures with excellent fidelity in both object-centric capture and a wide range of remote sensing scenes. This effectiveness allows detailed large-scale reconstruction from multi-view images. In order to mitigate randomness and ensure sufficient detail sampling, this application uses long training iterations.
[0256] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.
Claims
1. A multi-view 3D reconstruction method based on deep residual and neural implicit surface learning, characterized in that: It includes: S1. Using a camera to acquire images of multiple perspectives to be three-dimensionally reconstructed; S2, using the motion recovery structure SFM to perform image feature extraction, feature matching and sparse point cloud reconstruction on the image acquired by S1, generating a sparse point cloud with three-dimensional coordinate information and generating the camera's position information; S3, assuming that the light emitted from the camera is incident on the scene surface, obtaining the three-dimensional coordinate information of the sampling point set on the scene surface; S4, inputting the three-dimensional coordinates of the sparse point cloud obtained in S2, the position information of the camera, and the three-dimensional coordinate information of the sampling point set obtained in S3 into the SDF function to generate point cloud information represented by the SDF function; S5, uses ResNet-50 convolutional block to extract the appearance information of the image obtained in S1; S6, inputting the appearance information obtained by S5, the three-dimensional coordinates of the sparse point cloud obtained by S2, and the three-dimensional coordinate information of the sampling point set obtained by S3 into the Color function to generate the color value of the point cloud; S7, setting constraints based on the SDF function point cloud information obtained in S4 and the color value of the sparse point cloud obtained in S7, and calculating the color loss function; The SDF function point cloud information obtained by S4 is used to calculate the weight constraint loss according to the weight constraint condition; Set constraints based on the point cloud color information obtained by S6 and calculate the photometric consistency loss function; Based on the SDF function point cloud information obtained by S4, the SDF loss function of the point is calculated according to the constraint condition that the surface point SDF=0; S8, using the loss function obtained in S7 to reversely optimize the SDF network, convert the SDF into mesh and texture, and directly calculate the SDF value of each sampled voxel; S9, using the Matching Cubes algorithm to extract the zero isosurface from the SDF value obtained in S8, directly output the 3D surface, and complete the 3D reconstruction; The SDF function is: The surface S of the scene is represented as: S={p∈R 3 |f(p)=0} Where f(p) represents the function where f(p) = 0 on the surface of the observed object, p represents the spatial position of a point, and R represents the signed distance; This function is represented by a neural network, which is named SDF network in NeuS, and the SDF network function is associated with NeRF in NeuS; Use NeRF's loss function to optimize the surface representation SDF network; S7 The three-dimensional reconstruction includes calculating a total loss function, and the specific method thereof includes: Total loss function is the color constraint loss Weight Constrained Loss Point Constraint Loss and photometric consistency constraint loss The weighted sum of: Among them, λ represents the weight of the weight constraint loss function, α represents the weight of the photometric consistency constraint loss function, and β represents the weight of the point constraint loss function.
2. The multi-view 3D reconstruction method based on depth residual and neural implicit surface learning according to claim 1, characterized in that: The specific method of using the ResNet-50 convolution block to extract the appearance information of the image obtained in S1 in S5 includes: Extract shallow features of the appearance in the image, input the image into the ResNet-50 convolutional block, output a 1×1×256 feature tensor, and complete the appearance embedding; The ResNet-50 convolutional block introduces an inverted residual structure as a bottleneck layer in the residual structure to reduce the number of parameters; The bottleneck layer first passes through a 1×1 convolution kernel, then a 3×3 convolution kernel, and finally a 1×1 convolution kernel; The 256-dimensional input first passes through a 1×1×64 convolutional layer, then a 3×3×64 convolutional layer, and finally a 1×1×256 convolutional layer; Each convolutional layer is obtained using the Relu activation function.
3. The multi-view 3D reconstruction method based on depth residual and neural implicit surface learning according to claim 1, characterized in that: The constraints of 3D reconstruction include color constraints, weight constraints, point constraints and photometric consistency constraints.
4. The multi-view 3D reconstruction method based on depth residual and neural implicit surface learning according to claim 3, characterized in that: The specific method of calculating the weight constraint loss according to the weight constraint condition using the SDF function point cloud information obtained in S4 in S7 includes: Find two adjacent sampling points near the surface along the ray emitted from the camera. The SDF values of the two sampling points satisfy: f(p(t i ))·f(p(t i+1 ))<0 Among them, p(t i ) represents the first sampling point t between two adjacent sampling points i The initial point set, f(p(t i )) represents p(t i ), p(t i+1 ) represents the other sampling point t between two adjacent sampling points i+1 The initial point set, f(p(t i+1 )) represents p(t i+1 )’s SDF value; The first intersection point of the ray and the surface is approximated by linear interpolation as: in, represents the point set obtained by linear interpolation, t j+1 It means p(t j+1 ) point, f(p(t j+1 )) represents p(t j+1 ) point SDF value, t j It means p(t j ) point, f(p(t j )) represents p(t j ) point SDF value; v represents the unit direction vector of the ray; Will Add to p(t i ), we get a new point set P: Represents the point set obtained after interpolation; Use the newly generated point set P to re-render the volume and generate the final color: Among them, w(t i ) represents the weight of the initial point set, c(t i ) represents the pixel value of the initial point set, Represents the weight of the point set after linear interpolation, Represents the pixel value of the point set after linear interpolation, and n represents the number of points; Color deviation ΔC final for: Among them, c(t * ) represents the pixel value of the intersection of the ray and the surface, ε interp represents the deviation caused by linear interpolation, ε weight-final represents weight bias; Weight Constrained Loss:
5. The multi-view 3D reconstruction method based on depth residual and neural implicit surface learning according to claim 4, characterized in that: The specific method of setting constraint conditions based on the point cloud color information obtained in S6 and calculating the photometric consistency loss function in S7 includes: Take a small part S of the surface, and the pixel of S on the projection of the source view is q. Then, except for the occluded part, the image blocks of the small part S of the surface corresponding to different source views are geometrically consistent; With reference image pixel I r The camera coordinate system is used to represent S, and we have: Where d represents the normal direction, n T Indicates that the SDF network is Normals calculated by automatic differentiation of points; Let a point x in the reference image i The pixel values of x are located using the unit matrix H, so that i Associated with corresponding points x in other images, Get the relationship: Among them, K r and K i represents the camera internal calibration matrix, R r and R i represents the rotation matrix, u i and u r Represents the translation vector, corresponding to the source view I i and I r ;H i X represents the source image pixel, H i Represents the identity matrix corresponding to the pixel at a certain point in the source image; Take the rendered image as the reference image and calculate the normalized cross-correlation between the reference image and the source view: Among them, Cov() represents covariance, Var() represents variance, NCC(X, H i X) represents the NCC score between the sampled patch and the corresponding patches on all source images, X represents the sampled patch pixel value, Get the best four NCC scores. The NCC score range is -1 to 1, where -1 represents irrelevant and 1 represents relevant. The larger the NCC score, the better. Compute the photometric consistency loss for the corresponding view Where i represents the number of NCC scores.
6. The multi-view 3D reconstruction method based on depth residual and neural implicit surface learning according to claim 5, characterized in that: S7: The specific method of calculating the SDF loss function of the point using the SDF function point cloud information obtained in S4 according to the constraint condition of setting the surface point SDF=0 includes: Point Constraint Loss Among them, SFM reconstructs the sparse 3D point P0, and the direct supervision of P0 is f(p), where p represents the spatial position of the point and N represents the number of points.
