A three-dimensional reconstruction method and system based on deep learning

Abstract

The application discloses a kind of three-dimensional reconstruction method and system based on deep learning, belong to computer vision and deep learning technical field.For the existing neural implicit method is poor in weak texture area and non-lambertian surface reconstruction result;Traditional geometric supervision relies on noisy data, resulting in incomplete surface;Global and local feature fusion efficiency is low in optimization process, by using camera to obtain the multi-view image of target object;Using SfM model obtains the camera parameters and sparse point cloud corresponding to multi-view image;Build three-dimensional reconstruction model based on neural radiation field, and obtain reconstruction result;Evaluate the reconstruction result, and comprehensively evaluate the reconstruction result by performance index.The application solves the problem of fuzzy reconstruction and poor accuracy of target object in weak texture area;Overcome the problem of data labeling, data acquisition is simple, without introducing additional effective supervision;Weak texture area can be converted into digital model permanently stored in computer for cultural relics heritage.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and deep learning, and particularly relates to a three-dimensional reconstruction method and system based on deep learning. BACKGROUND

[0002] Three-dimensional reconstruction of a scene using multi-view image sequences has been an important research direction in computer vision and graphics, and is widely used in cultural heritage protection, autonomous driving, digital twin, virtual reality, etc. The multi-view stereo vision (MVS) three-dimensional reconstruction method uses the photometric consistency in the photo sequence to predict the depth map of the scene and fuse it into a point cloud, but there are problems such as depth estimation error accumulation, point cloud holes, and the need for certain calibration data. However, when dealing with scenes with weak texture regions or non-Lambertian surfaces, the multi-view stereo vision (MVS) three-dimensional reconstruction method has difficulty finding features in the image sequence in the region, and cannot use photometric consistency to predict the depth map, resulting in missing predicted depth maps in the region and holes in the reconstruction results.

[0003] The three-dimensional reconstruction method based on neural implicit representation (such as NeuS, VolSDF) represents the scene as a neural radiance field, uses color loss in the image sequence, and optimizes the geometry and appearance of the scene through differentiable volume rendering, which better avoids the problem of using photometric consistency to predict the depth map in the multi-view stereo vision (MVS) three-dimensional reconstruction method. At the same time, using neural implicit representation, the reconstructed three-dimensional surface is dense and has no holes. However, due to the lack of obvious texture features in weak texture or non-Lambertian surface regions, the color difference is not obvious, resulting in a relatively smooth reconstruction result in the region based on neural implicit representation. Subsequent methods mainly improve the three-dimensional reconstruction method based on neural implicit representation. MonoSDF uses the depth and normal vector prior information of the scene to regularize and constrain the neural radiance field, MVSDF uses the predicted depth map in multi-view stereo vision (MVS) to supervise the optimization of the neural radiance field, RegSDF and TUVR use the directional point cloud generated in multi-view stereo vision (MVS) to guide the optimization of the neural radiance field. The above methods all introduce regularization prior information based on neural implicit representation to further optimize the reconstruction result in weak texture or non-Lambertian surface regions, but these prior information requires additional computational consumption, making it difficult to balance efficiency and accuracy. SUMMARY

[0004] In view of the problem of poor reconstruction result in weak texture region and non-Lambertian surface in the existing three-dimensional reconstruction method, the present application provides a three-dimensional reconstruction method and system based on deep learning. It is suitable for high-precision surface reconstruction in complex scenes, especially for three-dimensional modeling of textureless regions and non-Lambertian surface objects.

[0005] In order to achieve the above object, the following technical solutions are adopted in the present application:

[0006] A three-dimensional reconstruction method based on deep learning, the method comprising the following steps:

[0007] Step 1: acquiring a multi-view image sequence {I i ……I N} of a target scene by using a camera;

[0008] Step 2: inputting all the acquired multi-view images into a general SfM algorithm COLMAP, performing feature extraction, feature matching and sparse reconstruction on the multi-view image sequence, and obtaining corresponding camera parameters (camera intrinsic parameters K and camera extrinsic parameters R i , t i ) and a sparse point cloud set X

[0009] Step 3: encrypting the sparse point cloud X i by using a k-nearest neighbor interpolation algorithm, projecting the encrypted point cloud into the multi-view image for denoising, checking the pixel color consistency, and finally obtaining an effective point cloud;

[0010] For each sparse point x i ∈X i , find its k nearest neighbors to obtain {x i1 ,...,x ik}, and calculate the centroid of the above k nearest neighbors as the interpolated point cloud x p ;

[0011]

[0012] Wherein, X ij represents the jth nearest neighbor of x i .

[0013] After obtaining the interpolated point cloud x p , project it into the multi-view image, check the pixel color consistency, and filter out part of the interpolated points, the formula being:

[0014]

[0015] Wherein, L photo represents the color consistency loss, X p is the interpolated point cloud set, C() represents the pixel color, and H() represents the projection matrix.

[0016] Step 4, build a dynamic neural implicit three-dimensional reconstruction model based on neural radiance field, input the multi-view image sequence and its corresponding camera parameters into the dynamic neural implicit three-dimensional reconstruction model for training, use the effective point cloud obtained in step 3 and multi-view consistency for supervision and constraint, and introduce a curriculum learning optimization strategy to optimize the training of neural radiance field, gradually obtain the three-dimensional reconstruction model of the target object, and realize the three-dimensional reconstruction of the target object;

[0017] The neural implicit three-dimensional reconstruction model based on neural radiance field represents the scene as a neural radiance field, the input is a continuous 5D coordinate (the spatial position x and the view direction d are obtained from the multi-view image sequence and the camera parameters), and the output is the corresponding body density σ(x) and color c. The surface of the scene is represented as a signed distance field SDF, which is fitted by a multi-layer perception MLP. The color loss depends on the input multi-view image sequence, and the color network and the signed distance field SDF network are optimized simultaneously to obtain a signed distance field SDF network with good visual perception.

[0018] The neural implicit three-dimensional reconstruction model includes a color network and a signed distance field SDF network. The dynamic neural implicit three-dimensional reconstruction model, based on the neural implicit three-dimensional reconstruction model based on neural radiance field, emphasizes the addition of "dynamic", that is, the introduction of an adaptive supervision module and a curriculum learning optimization strategy, to improve the three-dimensional reconstruction accuracy in weak texture areas or non-Lambertian surfaces.

[0019] The adaptive supervision module is to supervise and constrain the dynamic neural implicit three-dimensional reconstruction model using the effective point cloud obtained in step 3 and the multi-view photometric consistency, specifically:

[0020] Suppose the effective point cloud obtained in step 3 is accurately located on the surface of the scene, to constrain the signed distance field SDF network, that is, to constrain the SDF of the effective point cloud to be 0, that is, f(x p ) = 0;

[0021] The multi-view photometric consistency is that the projection area of the same point cloud in multiple views should have similar brightness or color. To fully detect the multi-view photometric consistency of the point cloud in the weak texture area, for all point clouds X i , find a 3x3 image block P centered on it, and calculate the structural similarity SSIM of the image block to constrain the multi-view photometric consistency, the formula is:

[0022]

[0023] Where L patch represents the multi-view photometric consistency loss, P k and represent the local features of the image block P, and SSIM represents the structural similarity, the formula is:

[0024]

[0025] where, μ x and μ y represent the mean of image blocks x and y, and C1 and C2 are constants to prevent the denominator from being 0.

[0026] The course learning optimization strategy is: starting from the training mechanism of deep learning, dynamically adjusting the supervision weight between the priori region and the non-priori region of the scene, and realizing the stage optimization of the training process. That is, the adaptive supervision module mentioned above supervises the effective region of the model, and the supervision weight is adjusted stage by stage to gradually train the priori and non-priori regions of the scene:

[0027] The stage optimization includes three parts: only using SfM point cloud to supervise the signed distance field SDF network, jointly supervising the point cloud and multi-view photometric consistency, and globally optimizing the surface details;

[0028] During the training process, the weight of the non-priori region pixel points is gradually increased, that is, the difficulty of training is gradually increased, and the formula is:

[0029]

[0030] where, L SDF represents the loss of the signed distance field SDF network; L sup represents the loss of the geometric priori region; L unsup represents the loss of the non-geometric priori region; V represents the number of observation angles, that is, the number of camera shooting angles; and v represents one of the observation angles.

[0031] The stage training uses the weight ρ t to control:

[0032]

[0033] where, t represents the training time, T init represents the number of iterations of the priori region, T anneal represents the annealing frequency, T total represents the entire training process, and ρ min is set to 0.2 to ensure that the priori region always exists during the training process.

[0034] Step 5, evaluate the reconstruction result, extract the surface grid through the Marching Cubes algorithm, and quantitatively evaluate the reconstruction accuracy based on the chamfer distance:

[0035] Extract the surface grid through the Marching Cubes algorithm, and quantitatively evaluate the reconstruction accuracy based on the chamfer distance (Chamfer Distance):

[0036]

[0037] S1, S2 of the above formula respectively represent two groups of 3D point clouds, the first term represents the sum of the minimum distances from any point x in S1 to S2, and the second term represents the sum of the minimum distances from any point y in S2 to S1;

[0038] If the distance is large, it indicates that the two groups of point clouds are quite different; if the distance is small, it indicates that the reconstruction effect is good.

[0039] A three-dimensional reconstruction system based on deep learning, the system comprising an image data acquisition module, an image data preprocessing module, a dynamic neural implicit three-dimensional reconstruction module, and a reconstruction performance verification module;

[0040] The image data acquisition module is configured to capture a multi-view image sequence of a scene using a camera;

[0041] The image data preprocessing module is configured to calculate camera poses and sparse point clouds from the multi-view image sequence using SfM;

[0042] The dynamic neural implicit three-dimensional reconstruction module, based on a neural implicit three-dimensional reconstruction model based on neural radiance fields, focuses on adding "dynamicity", introducing an adaptive supervision module and a curriculum learning optimization strategy, to improve the three-dimensional reconstruction accuracy in weak texture areas or non-Lambertian surfaces;

[0043] The reconstruction performance verification module uses the Marching Cubes algorithm to extract surface meshes, and quantitatively evaluates the reconstruction accuracy based on the chamfer distance.

[0044] Compared with the prior art, the present application has the following advantages:

[0045] The present application provides a three-dimensional reconstruction method and system based on deep learning. In view of the fact that existing methods generally cannot generate accurate geometric surfaces in textureless areas and non-Lambertian surfaces, an adaptive geometric supervision strategy is introduced, the SfM point cloud after encryption and denoising is used to directly locate the zero level set of the SDF network, and multi-view photometric consistency is used to improve the geometric surface. A curriculum learning-based geometric perception iterative optimization strategy is also introduced, which dynamically adjusts the weight between the priori and non-priori regions of the SDF network, so that the surface reconstruction is both accurate and effective. It can be applied to high-precision surface reconstruction in textureless areas and non-Lambertian surface scenes, and supports high-fidelity reconstruction of complex scenes such as cultural heritage (e.g. bronze wares, porcelain wares). BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative labor based on the embodiments of the present application also belong to the protection scope of the present application.

[0047] Figure 1 The overall flowchart of the three-dimensional reconstruction method based on deep learning provided by the present application is shown in the figure.

[0048] Figure 2 The neural implicit three-dimensional reconstruction model network structure based on neural radiance field in the three-dimensional reconstruction method based on deep learning provided by the present application is shown in the figure.

[0049] Figure 3 The module schematic diagram of the three-dimensional reconstruction method based on deep learning provided by the present application is shown in the figure.

[0050] Figure 4 The schematic diagram of the point cloud encryption module in the three-dimensional reconstruction method based on deep learning provided by the present application is shown in the figure.

[0051] Figure 5 The module schematic diagram of the curriculum learning strategy in the three-dimensional reconstruction method based on deep learning provided by the present application is shown in the figure.

[0052] Figure 6 The reconstruction visualization result comparison chart of different methods of the three-dimensional reconstruction method based on deep learning provided by the present application on part of the scene of the DTU data set is shown in the figure.

[0053] Figure 7 The reconstruction visualization result comparison chart of the three-dimensional reconstruction method based on deep learning provided by the present application on weak texture and non-Lambertian scene is shown in the figure.

[0054] Figure 8 The reconstruction visualization result chart of the three-dimensional reconstruction method based on deep learning provided by the present application on cultural relic model is shown in the figure. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor also belong to the protection scope of the present application.

[0056] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0057] Reference Figure 1 Figure 2 For an embodiment of the present application, a three-dimensional reconstruction method based on deep learning is provided, comprising the following steps:

[0058] Step 1: Obtain a multi-view image sequence {I i ……I N} of a target scene using a camera.

[0059] Step 2: input all the obtained multi-view images into a general SfM algorithm COLMAP, perform feature extraction, feature matching and sparse reconstruction on the multi-view image sequence, and obtain corresponding camera parameters (camera intrinsic parameters K and camera extrinsic parameters R i , t i ) and a sparse point cloud set X

[0060] Step 3, use k-nearest neighbor interpolation algorithm to encrypt the sparse point cloud X i , and project the encrypted point cloud into the multi-view image for denoising, check its pixel color consistency, and finally obtain the effective point cloud.

[0061] Step 4, build a dynamic neural implicit three-dimensional reconstruction model based on neural radiance field, input the multi-view image sequence and its corresponding camera parameters into the dynamic neural implicit three-dimensional reconstruction model for training. Use the effective point cloud obtained in step 3 and the multi-view consistency for supervision and constraint, and introduce a course learning optimization strategy to optimize the training of the neural radiance field, gradually obtain the three-dimensional reconstruction model of the target object, and realize the three-dimensional reconstruction of the target object.

[0062] Step 5, evaluate the reconstruction result, extract the surface mesh through the Marching Cubes algorithm, and quantitatively evaluate the reconstruction accuracy based on the chamfer distance.

[0063] In a preferred embodiment of the present invention, in step 1, the acquisition of the scene image sequence mainly involves using a camera to capture the target scene from multiple perspectives, resulting in a multi-view image sequence of the target scene. The DTU benchmark dataset is used for the experiment. The DTU dataset is an indoor object dataset captured by an industrial robotic arm equipped with adjustable brightness lights, containing 128 scenes in a controlled laboratory environment. The models were captured using a structured light scanner. Under seven different lighting conditions, each scene was scanned at the same 49 or 64 camera positions, generating 343 or 448 RGB images with a resolution of 1200×1600 pixels.

[0064] In a preferred embodiment of the present invention, in step 2, the SfM algorithm uses COLMAP, inputting a multi-view image sequence of a scene to obtain the corresponding camera parameters (camera intrinsic parameter K and camera extrinsic parameter R). i , t i ) and sparse point cloud set

[0065] As a preferred embodiment of the present invention, the specific steps for encrypting and denoising the sparse point cloud in step 3 are as follows: the encryption process is described in detail below. Figure 4 For each sparse point x i Find its k nearest neighbors to get {x i1 ,...,x ik} Calculate the centroids of the above k nearest neighbors as the point cloud x generated by interpolation. p ;

[0066]

[0067] Among them, X ij x represents i The j-th nearest neighbor.

[0068] Obtain the interpolated point cloud x p Then, it is projected onto a multi-view image, the pixel color consistency is checked, and some interpolation points are filtered out. The formula is:

[0069]

[0070] Among them, L photo X represents the loss of color consistency. p This is the set of interpolated point clouds, where C() represents the pixel color and H() represents the projection matrix.

[0071] The dynamic neural implicit three-dimensional reconstruction module in step 4 is improved on the basis of the neural implicit three-dimensional reconstruction model based on neural radiance fields, introduces an adaptive supervision module and a curriculum learning optimization strategy, inputs multi-view images and corresponding camera parameters into the dynamic neural implicit three-dimensional reconstruction model for training, uses the effective point cloud obtained in step 3 and multi-view consistency for supervision and constraint, and introduces a curriculum learning optimization strategy to optimize the training of the neural radiance field, so as to gradually obtain a three-dimensional reconstruction model of the target object and realize three-dimensional reconstruction of the target object.

[0072] Reference Figure 2 The neural implicit three-dimensional reconstruction model based on neural radiance fields represents a scene as a neural radiance field, inputs continuous 5D coordinates (spatial position x and viewing direction d obtained from a multi-view image sequence and camera parameters), and outputs corresponding body density σ(x) and color c. The surface of the scene is represented as a signed distance field SDF, and a multi-layer perception MLP is used to fit it. The color loss of the input multi-view image sequence is used to optimize the color network and the signed distance field SDF network simultaneously to obtain a signed distance field SDF network with good visual perception.

[0073] The neural implicit three-dimensional reconstruction model includes a color network and a signed distance field SDF network. The dynamic neural implicit three-dimensional reconstruction model is improved on the basis of the neural implicit three-dimensional reconstruction model based on neural radiance fields, and focuses on increasing the dynamic nature, that is, introducing an adaptive supervision module and a curriculum learning optimization strategy to improve the three-dimensional reconstruction accuracy in weak texture areas or non-Lambertian surfaces.

[0074] The adaptive supervision module uses the effective point cloud obtained in step 3 and multi-view photometric consistency to supervise the dynamic neural implicit three-dimensional reconstruction model, specifically:

[0075] The effective point cloud obtained in step 3 is located on the surface of the scene, which is used to constrain the signed distance field SDF network, that is, the SDF of the effective point cloud is 0, that is, f(x p ) = 0;

[0076] Multi-view photometric consistency is that the projection area of the same point cloud in multiple views should have similar brightness or color. In order to fully detect the multi-view photometric consistency of the point cloud in the weak texture area, for all point clouds X i , find a 3x3 image block P centered on it, and calculate the structural similarity SSIM of the image block to perform multi-view photometric consistency constraint, the formula is:

[0077]

[0078] Where L patch represents the multi-view photometric consistency loss, P k and where P represents the local features of the image patch, SSIM represents the structural similarity, and the formula is:

[0079]

[0080] where μ x and μ y represent the mean values of the image patches x and y, C1 and C2 are constants to prevent the denominator from being zero.

[0081] The course learning optimization strategy is: starting from the training mechanism of deep learning, dynamically adjusting the supervision weight between the priori region and the non-priori region of the scene, and realizing the stage optimization of the training process. That is, the adaptive supervision module mentioned above supervises the effective area of the model, adjusts the supervision weight in stages, and gradually trains the priori and non-priori regions of the scene:

[0082] The stage optimization refers to Figure 5 , which includes three parts: using SfM point cloud to supervise the signed distance field SDF network, jointly supervising the point cloud and multi-view photometric consistency, and globally optimizing the surface details.

[0083] During the training process, the weight of the non-priori region pixels is gradually increased, that is, the difficulty of training is gradually increased, and the formula is:

[0084]

[0085] where L SDF represents the loss of the signed distance field SDF network; L sup represents the loss of the geometric priori region; L unsup represents the loss of the non-geometric priori region; V represents the number of observation angles, that is, the number of camera shooting angles; and v represents one of the observation angles.

[0086] The stage training uses the weight ρ t to control:

[0087]

[0088] where t represents the training time, T init represents the number of iterations of the priori region, T anneal represents the annealing frequency, T total represents the entire training process, and ρ min is set to 0.2 to ensure that the priori region always exists during the training process.

[0089] The neural implicit three-dimensional reconstruction model based on neural radiance field in step 4, the SDF network is coded by an 8-layer MLP with a 256-dimensional hidden layer and a skip layer, and the color network is coded by a 4-layer MLP with a 256-dimensional hidden layer. The batch size is set to 512, the number of sampled rays per batch is 512, the number of uniformly sampled points of the coarse network is 64, and the number of non-uniformly sampled points of the fine network is 128. Each scene is iteratively trained for 300,000 times. The above model is deployed on an NVIDIA RTX4090 Ti GPU server, and is deployed using the PyTorch framework, the PyCharm compiler and the Python 3.8 language.

[0090] In step 5, the reconstruction result is evaluated, the surface mesh is extracted by the Marching Cubes algorithm, and the reconstruction accuracy is quantitatively evaluated based on the chamfer distance;

[0091] The surface mesh is extracted by the Marching Cubes algorithm, and the reconstruction accuracy is quantitatively evaluated based on the chamfer distance (Chamfer Distance):

[0092]

[0093] The above formula S1, S2 respectively represents two groups of 3D point clouds, the first term represents the sum of the minimum distances from any point x in S1 to S2, and the second term represents the sum of the minimum distances from any point y in S2 to S1;

[0094] If the distance is large, it means that the two groups of point clouds are quite different; if the distance is small, it means that the reconstruction effect is good.

[0095] This embodiment selects 15 models from the DTU dataset for comparison. We compare the proposed method with the traditional three-dimensional reconstruction algorithm COLMAP and the deep learning-based three-dimensional reconstruction methods IDR, VolSDF, NeuS, NeuralWarp and PG-NeuS. The reconstruction results of each algorithm are shown in Table 1, and the visual comparison is shown in Figure 6 Compared with the current three-dimensional reconstruction method with better performance, the visual comparison of the reconstruction results of the weak texture and non-Lambertian surface area is as follows Figure 7 The reconstruction results of the cultural relic model of our invention are as follows Figure 8 The proposed method improves the effect by 60% compared with the traditional reconstruction algorithm COLMAP, and improves the effect by 36% compared with the deep learning-based reconstruction algorithm NeuS.

[0096] Table 1 Quantitative comparison results of chamfer distance of different algorithms on DTU dataset

[0097]

[0098] A three-dimensional reconstruction system based on deep learning, the system comprising an image data acquisition module, an image data preprocessing module, a dynamic neural implicit three-dimensional reconstruction module, and a reconstruction performance verification module;

[0099] The image data acquisition module is configured to capture a multi-view image sequence of a scene using a camera.

[0100] The image data preprocessing module is configured to calculate camera poses and sparse point clouds from the multi-view image sequence using SfM.

[0101] The dynamic neural implicit three-dimensional reconstruction module, based on a neural implicit three-dimensional reconstruction model based on neural radiance fields, focuses on adding "dynamicity", introduces an adaptive supervision module and a curriculum learning optimization strategy, and is used to improve the three-dimensional reconstruction accuracy in weak texture areas or non-Lambertian surfaces.

[0102] The reconstruction performance verification module uses the Marching Cubes algorithm to extract surface meshes, and quantitatively evaluates the reconstruction accuracy based on the chamfer distance.

[0103] The contents not described in detail in the specification of the present application belong to the prior art known to those skilled in the art. Although the above describes the specific embodiments of the present application in order to facilitate those skilled in the art to understand the present application, it should be clear that the present application is not limited to the scope of the specific embodiments, and it is obvious to those skilled in the art that all the inventions utilizing the concept of the present application are within the scope of the present application.

Claims

1. A 3D reconstruction method based on deep learning, characterized in that, The method includes the following steps: Step 1: Use a camera to acquire a multi-view image sequence of the target scene. }; Step 2: Input all acquired multi-view images into the general SfM algorithm COLMAP to perform feature extraction, feature matching, and sparse reconstruction on the multi-view image sequence, obtaining the corresponding camera parameters and sparse point cloud set. ; Step 3: Use the k-nearest neighbor interpolation algorithm to process the sparse point cloud. Encryption is performed, and the encrypted point cloud is projected onto a multi-view image for denoising. The pixel color consistency is checked, and finally a valid point cloud is obtained. Step 4: Build a dynamic neural implicit 3D reconstruction model based on neural radiation field. Input multi-view image sequences and their corresponding camera parameters into the dynamic neural implicit 3D reconstruction model for training. Use the effective point cloud and multi-view consistency obtained in Step 3 for supervision and constraint. At the same time, introduce a course learning optimization strategy to optimize the training of neural radiation field, and gradually obtain the 3D reconstruction model of the target object to realize the 3D reconstruction of the target object. The neural implicit 3D reconstruction model based on neural radiation fields represents the scene as a neural radiation field, with continuous 5D coordinates as input and the corresponding volume density as output. and color The scene surface is represented as a symbolic distance field (SDF) and fitted using a multilayer perceptron (MLP). The color network and the symbolic distance field (SDF) network are simultaneously optimized based on the color loss of the input multi-view image sequence to obtain a symbolic distance field (SDF) with good visual perception. The neural implicit 3D reconstruction model includes a color network and a symbolic distance field (SDF) network; the dynamic neural implicit 3D reconstruction model is based on the neural implicit 3D reconstruction model of the neural radiation field, and introduces an adaptive supervision module and a course learning optimization strategy to improve the 3D reconstruction accuracy in weakly textured regions or non-Lambertian surfaces. The adaptive supervision module utilizes the effective point cloud and multi-view photometric consistency obtained in step 3 to supervise and constrain the dynamic neural implicit 3D reconstruction model, specifically: Assuming the effective point cloud obtained in step 3 lies on the scene surface, we constrain the symbolic distance field SDF network, i.e., constrain the effective point cloud. ;Right now ; Multi-view photometric consistency means that the projection areas of the same point cloud should have similar brightness or color in multiple viewpoints. To fully detect the multi-view photometric consistency of point clouds in weakly textured regions, for all point clouds... Find a 3×3 image patch centered on it. And calculate the structural similarity of image patches. To constrain photometric consistency across multiple viewing angles, the formula is: ， in, This indicates the loss of photometric consistency across multiple viewing angles. and Represents image blocks Local features, The formula for representing structural similarity is: ， in, and Represents image blocks and The mean, and To prevent constants with a denominator of 0; The course learning optimization strategy is as follows: the adaptive supervision module supervises the effective region of the model, the supervision weights are adjusted in stages, and the training scenarios include both prior and non-prior regions. The phased optimization includes three parts: using only SfM point cloud to supervise the symbolic range field SDF network, joint point cloud supervision and multi-view photometric consistency, and global optimization of surface details; During training, the weights of pixels in regions without prior knowledge are gradually increased, using the following formula: ， in, This represents the loss of an SDF network with a long symbolic distance. This indicates a loss with a geometric prior region; This represents the loss for regions without geometric priors. Indicates the viewing angle, i.e., the number of camera angles; This indicates one perspective of observation; Phase training utilizes weights To control: ， in, Indicates the training time. This indicates the number of iterations with a priori region. Indicates the annealing frequency. This indicates the entire training process. Set it to 0.2 to ensure that a prior region always exists during training; Step 5: Evaluate the reconstruction results. Extract the surface mesh using the Marching Cubes algorithm and quantitatively evaluate the reconstruction accuracy based on the chamfer distance.

2. The deep learning-based 3D reconstruction method according to claim 1, characterized in that, The specific operation of step 3 is as follows: For each sparse point Find its k nearest neighbors to get Calculate the centroids of the k nearest neighbors as the points generated by interpolation. : ， in, express The The nearest neighbor; Obtain the interpolated point cloud Then, it is projected onto a multi-view image, the pixel color consistency is checked, and some interpolation points are filtered out. The formula is: ， in, Indicates loss of color consistency. This is the set of interpolated point clouds. Indicates the color of a pixel. This represents the projection matrix.

3. The deep learning-based 3D reconstruction method according to claim 2, characterized in that, The specific operation of step 5 is as follows: The surface mesh was extracted using the Marching Cubes algorithm, and the reconstruction accuracy was quantitatively evaluated based on the chamfer distance. ， in, , These represent two sets of 3D point clouds, with the first item representing... any point in arrive The sum of the minimum distances, the second term represents any point in arrive The sum of the minimum distances.

4. A three-dimensional reconstruction system based on deep learning, characterized in that, The system is used to perform the steps in the deep learning-based three-dimensional reconstruction method according to any one of claims 1 to 3. The system includes an image data acquisition module, an image data preprocessing module, a dynamic neural implicit three-dimensional reconstruction module, and a reconstruction performance verification module. The image data acquisition module uses a camera to capture a sequence of multi-view images of a scene. The image data preprocessing module uses SfM to calculate the camera pose and sparse point cloud from a multi-view image sequence; The dynamic neural implicit 3D reconstruction module, based on the neural implicit 3D reconstruction model based on neural radiation field, introduces an adaptive supervision module and a course learning optimization strategy to improve the 3D reconstruction accuracy in weakly textured regions or non-Lambertian surfaces. The reconstruction performance verification module uses the Marching Cubes algorithm to extract the surface mesh and quantitatively evaluates the reconstruction accuracy based on the chamfer distance.

Images