Sparse photo synthesis 3D model reconstruction method based on deep neural network

Abstract

The invention discloses a sparse photo synthesis 3D model reconstruction method based on a deep neural network. The method comprises the following steps of S1, obtaining a sparse image set and performing preprocessing; s2, performing feature extraction to obtain an image feature tensor; s3, constructing a three-dimensional cost body corresponding to the main view under a 1 / 16 spatial resolution hierarchy; s4, calculating scores of the auxiliary views relative to the main view, and selecting an optimal auxiliary view image; s5, performing cost aggregation on the three-dimensional cost body by using a three-dimensional convolutional neural network; s6, calculating depth variance, and constructing a new three-dimensional cost body; s7, repeating the step S6, and generating a dense depth map under a 1 / 4 resolution hierarchy; and S8, completing reconstruction of the 3D model based on the dense depth map. According to the method, the cascade neural network and the visual angle self-adaptive strategy are adopted, high-quality three-dimensional reconstruction under the sparse image condition is achieved, and the method has the advantages of being high in efficiency, excellent in precision and high in robustness.

Description

Technical Field

[0001] The present invention relates to the field of 3D reconstruction, and in particular to a method for reconstructing a sparse photo synthesis 3D model based on a deep neural network. Background Art

[0002] With the continuous development of computer vision technology, 3D reconstruction methods based on image sequences have been widely used in fields such as autonomous driving, intelligent manufacturing, cultural relic protection, virtual reality, etc. Traditional 3D reconstruction methods usually rely on structured light, laser scanning, or multi-view geometry and other means for scene modeling. Among them, the 3D reconstruction technology based on multi-view images is particularly prominent, because it can achieve high-precision restoration of the scene geometric structure under the premise of only requiring conventional RGB image input, and has the significant advantages of low cost and strong adaptability.

[0003] In recent years, with the introduction of deep learning, especially convolutional neural networks, 3D reconstruction tasks have made breakthrough progress in key modules such as feature extraction, depth estimation, and voxel aggregation. Representative methods such as MVSNet, R-MVSNet, etc. construct a 3D cost volume between viewpoints and use a 3D convolutional neural network for aggregation and regression, effectively improving the robustness and accuracy of multi-view depth estimation, reducing memory overhead, and making it possible to reconstruct 3D high-resolution images.

[0004] However, most of the above-mentioned 3D reconstruction methods based on deep neural networks are based on the premise assumption of "dense image input and sufficient view coverage". In real application scenarios, especially in tasks such as drone aerial photography, mobile terminals, security monitoring, cultural relic photography, or historical image restoration, often only a very small number of image samples (such as 2-3 images) can be obtained, and the view coverage is incomplete and the overlapping area is limited. Under such sparse image input conditions, existing methods generally have problems such as decreased reconstruction accuracy, discontinuous model output, and incomplete occlusion areas.

[0005] First of all, most current 3D reconstruction networks use fixed image pairs to participate in constructing the cost volume, and lack an effective selection strategy for auxiliary views. Under sparse images, how to select the most informative auxiliary viewpoints becomes a key factor affecting the reconstruction effect. The lack of a dynamic view selection mechanism will result in a large number of redundant or invalid auxiliary images being included in the aggregation calculation, which not only affects the calculation efficiency but also reduces the geometric expression ability of the cost volume.

[0006] Secondly, existing methods often construct voxels based on a fixed depth hypothesis range, ignoring the depth uncertainty between different image regions. In areas with insufficient image information or complex structures, a fixed depth range may lead to a large number of invalid sampling or loss of key depth intervals, thereby affecting subsequent voxel aggregation and depth estimation accuracy. Although some methods have introduced a coarse-to-fine hierarchical design, they have not yet achieved a truly adaptive depth range adjustment mechanism.

[0007] Furthermore, most of the common 3D cost volume aggregation methods currently use direct stacking or simple fusion of image features, lacking explicit modeling of factors such as feature importance and geometric visibility. Under sparse image conditions, this blind aggregation can easily introduce noise, leading to feature distortion and matching failure, which in turn affects the quality of the depth map. Especially in occluded areas or background areas with a single texture, the reconstruction results of existing methods are prone to problems such as holes, jitter, or error boundary diffusion.

[0008] In addition, in the post-reconstruction processing stage, most methods only perform single-viewpoint depth regression on the main view image, and fail to effectively utilize the information complementarity between different perspectives. In the point cloud reconstruction stage, if the point clouds generated from different perspectives lack a unified coordinate alignment and fusion strategy, it will often lead to model structure breaks, inconsistent accuracy, and even deformation problems. At the same time, for the case of inconsistent point cloud density, the existing methods lack an effective sparse-dense fusion mechanism, which affects the structural integrity and surface continuity of the final three-dimensional model.

[0009] In summary, although multi-view 3D reconstruction technology based on deep learning has made significant progress in recent years, it still faces many challenges under sparse image input conditions, including the lack of information-driven auxiliary view selection mechanism, the lack of adaptive depth hypothesis construction strategy based on depth uncertainty perception, the lack of cost volume aggregation method driven by the joint fusion of features and view visibility, and the lack of 3D model generation mechanism with a balance between coordinate consistency and point cloud density.

[0010] Therefore, there is an urgent need to propose a 3D reconstruction method that is structurally reasonable, computationally efficient and adaptable for sparse image input conditions, which can complete the task of reconstructing 3D models with high quality, structural continuity and accurate texture restoration under extremely few image inputs and non-ideal shooting conditions. Summary of the invention

[0011] One purpose of the present invention is to propose a sparse photo synthesis 3D model reconstruction method based on deep neural network. The present invention adopts cascade neural network and perspective adaptation strategy, and realizes high-quality 3D model reconstruction under sparse image input conditions through multi-scale depth estimation, feature-guided aggregation and point cloud fusion methods. Compared with traditional methods that rely on dense image input, the present invention has the advantages of low requirement on the number of images, high reconstruction accuracy, strong adaptability and high computational efficiency.

[0012] A method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network according to an embodiment of the present invention includes the following steps: S1. Obtain a set of sparse images, preprocess them, and extract the camera parameters of each image; S2. Extract features from the preprocessed set of sparse images to obtain image feature tensors at 1 / 16, 1 / 8, and 1 / 4 spatial resolution levels; S3. At the 1 / 16 spatial resolution level, construct a three-dimensional cost volume corresponding to the main view image according to the initial depth hypothesis range of the main view image and in combination with the camera parameters; S4. Calculate the image feature similarity score and geometric visibility score of the auxiliary view relative to the main view, synthesize a view score according to a weighted fusion strategy, and select the optimal auxiliary view image; S5. Fuse the image feature tensors of the main view image and the selected auxiliary view image, and input them into a three-dimensional convolutional neural network based on the CasMVSNet structure to perform cost aggregation on the three-dimensional cost volume, and generate an initial depth map of the main view image at the 1 / 16 spatial resolution level; S6. Calculate the depth variance according to the initial depth map, adaptively adjust the depth hypothesis range, construct a new three-dimensional cost volume, and generate a depth map at the 1 / 8 resolution level; S7. Repeat step S6 to generate a dense depth map at the 1 / 4 resolution level; S8. Generate a three-dimensional dense point cloud based on the dense depth map, and perform registration and fusion to complete the reconstruction of the 3D model.

[0013] Optionally, the set of sparse images includes one main view image and at least one auxiliary view image.

[0014] Optionally, the preprocessing includes image alignment, size normalization, image enhancement, and distortion correction.

[0015] Optionally, step S3 specifically includes: S31. At the 1 / 16 spatial resolution level, construct an initial depth hypothesis range for the main view image and perform uniform discretization to obtain a depth hypothesis set: ; where D represents the depth hypothesis set, represents the kth depth value, , respectively represent the minimum and maximum depth values, and M represents the number of depth levels; S32. For each pixel coordinate and each depth value in the feature tensor of the main view image, back-project them to three-dimensional points in the main view camera coordinate system using the main view camera parameters: ; Wherein, represents the three-dimensional point corresponding to the pixel at the k-th depth, represents the inverse operation of the main view camera parameters, x and y represent the pixel positions on the feature tensor, and -1 represents the "inverse" operation; S33. Map the three-dimensional points to each auxiliary view image plane, and calculate the two-dimensional projection coordinates on the auxiliary view through the camera parameters of the auxiliary view image: ; Wherein, represents the camera parameters of the i-th auxiliary view image, represents the rotation matrix, represents the translation vector, represents the transpose of the rotation matrix of the main view image, represents the translation vector of the main view image, and represent the two-dimensional projection coordinates of the three-dimensional point in the i-th auxiliary view image, represents the scale factor of the projection; S34. Use bilinear interpolation to sample the feature tensor of the auxiliary view image at the two-dimensional projection coordinate position to obtain the reprojection feature tensor; S35. Calculate the feature difference cost term between the feature tensor of the main view image and the reprojection feature tensor of each auxiliary view image: ; Wherein, represents the cost value corresponding to the pixel position of the i-th auxiliary view, represents the L1 norm, represents the feature tensor of the main view image, represents the reprojection feature tensor of the auxiliary view image; S36. Aggregate and stack all auxiliary view images and cost values to construct a three-dimensional cost volume: ; Wherein, represents the aggregated cost value of the main view image at the pixel position (x, y) and the depth value , Aggregate represents the aggregation operation, and N represents the number of auxiliary view images.

[0016] Optionally, the specific content of S4 includes: S41. For each auxiliary view image, calculate the image feature similarity score with the main view image: ; where, represents the image feature similarity score between the auxiliary view image and the main view image, and represent the height and width of the feature map of the main view image respectively, and represent the feature tensors of the main view image and the auxiliary view image respectively, represents the L2 norm; S42. In the depth hypothesis set at the current spatial resolution level, according to the pixel back-projection and projection conversion relationship, count the number of projected pixel points of the valid pixels in the main view image in the auxiliary view image, and calculate the geometric visibility score: ; where, represents the geometric visibility score of the auxiliary view image, represents the number of valid pixels projected into the auxiliary view image from the main view image under the depth hypothesis set; S43. Perform weighted fusion on the image feature similarity score and the geometric visibility score to obtain the view score: ; where, represents the view score, represents the fusion weight coefficient, which describes the weight of the feature similarity score in the view score; S44. Dynamically adjust the fusion weight coefficient according to the sparsity: ; where, represents the adjusted fusion weight coefficient, represents the adjustment constant, and min represents taking the minimum value; S45. Sort the view scores of all auxiliary views, and select at least one auxiliary view image with the highest score.

[0017] Optionally, the S5 specifically includes: S51. Fuse the main view image feature tensor and the selected auxiliary view image along the channel dimension to generate a fused feature tensor; S52. Input the fused feature tensor into a three-level cascaded 3D convolutional neural network module constructed based on the CasMVSNet structure, and complete 3D cost volume aggregation at the 1 / 16 spatial resolution level to generate an aggregated cost volume. The CasMVSNet is a cascaded multi-view stereo vision network based on deep learning; S53. Perform Softmax normalization on the aggregated cost volume along the depth dimension to obtain the matching probability distribution at each pixel position under the depth hypothesis set at the 1 / 16 spatial resolution level; S54. Generate an initial depth map by performing weighted averaging based on the matching probability distribution and the corresponding depth values: ; where D(x, y) represents the initial depth map of the main view image, represents the k-th depth value, represents the matching probability distribution under the k-th depth hypothesis, and M represents the number of depth levels.

[0018] Optionally, the three-dimensional cost volume aggregation specifically includes: Guided feature generation stage: Perform layer-by-layer downsampling on the fused feature tensor to obtain low-resolution guided features, medium-resolution guided features, and high-resolution guided features respectively; Three-stage cost aggregation stage: After fusing the three-dimensional cost volume and the low-resolution guided features, input them into the first three-dimensional convolutional module, output the first cost volume, after fusing the first cost volume and the medium-resolution guided features, input them into the second three-dimensional convolutional module, output the second cost volume, and after fusing the second cost volume and the high-resolution guided features, input them into the third three-dimensional convolutional module, and output the aggregated cost volume.

[0019] Optionally, the aggregated cost volume has the same spatial and depth dimensions as the original three-dimensional cost volume.

[0020] Optionally, S6 specifically includes: S61. Calculate the depth variance according to the matching probability distribution after cost volume normalization: ; where represents the depth variance at pixel (x, y), represents the matching probability distribution under the k-th depth hypothesis, represents the k-th depth value, and D(x, y) represents the initial depth map of the main view image; S62. Calculate the depth search range extension amount according to the depth variance and a preset coefficient: ; where represents the depth search range extension amount at pixel (x, y), represents the preset coefficient, represents the depth standard deviation at pixel (x, y); S63. Centering on the initial depth map of the front view image, dynamically determine the depth hypothesis range at the 1 / 8 resolution level according to the depth search range expansion amount: ; where R(x, y) represents the depth hypothesis range of pixel (x, y) at the 1 / 8 resolution level, and D(x, y) represents the initial depth map of the front view image; S64. Calculate the depth variance and the adaptive depth search range for each pixel on the entire front view image respectively, and construct the global depth hypothesis range at the 1 / 8 resolution level; S65. Reconstruct the depth hypothesis set and the three-dimensional cost volume of the front view image to generate the depth map at the 1 / 8 resolution level.

[0021] Optionally, the specific steps of S8 include: S81. Obtain the dense depth map, and in combination with the camera parameters of the front view image, back-project the position of each pixel in the front view image and the corresponding depth value into the camera coordinate system to calculate the three-dimensional spatial position coordinates; S82. Bind the three-dimensional spatial coordinates corresponding to each pixel position with the color information in the front view image to generate a three-dimensional dense point cloud containing geometric positions and color attributes; S83. Generate three-dimensional dense point clouds for the front view image and the auxiliary view image respectively, and attach the corresponding external camera parameters, where the external parameters include the rotation matrix and the translation vector corresponding to each view; S84. Register the three-dimensional dense point clouds corresponding to all views through the external camera parameters and map them into the same coordinate system; S85. Perform fusion and optimization processing on the registered three-dimensional dense point clouds to generate a fused point cloud with continuous structure and strong spatial consistency; S86. Perform a mesh reconstruction operation based on the fused point cloud to construct a three-dimensional topological structure, and form a complete 3D model through surface connection.

[0022] The beneficial effects of the present invention are: First of all, the present invention proposes a three-dimensional model reconstruction method based on a deep neural network for sparse image input conditions, which can effectively overcome the problems faced by the prior art, such as incomplete three-dimensional structure, decreased depth estimation accuracy, and failed reconstruction in occluded areas, in the case of limited number of images and insufficient view coverage. By gradually optimizing the depth estimation process at multiple scale space resolution levels and combining an adaptive view selection mechanism with a depth hypothesis adjustment strategy, the present invention significantly improves the three-dimensional modeling ability under low image input density.

[0023] Secondly, in the initial stage, the present invention introduces preprocessing steps such as image alignment, enhancement, and normalization, enhancing the matching feasibility of the input image. The multi-scale feature extraction module is utilized to achieve multi-layer perception of image semantics and structural details, laying a solid feature foundation for subsequent voxel construction and aggregation. In the depth estimation stage, the present invention not only uses a cascaded 3D convolutional aggregation network based on CasMVSNet to refine the cost volume layer by layer but also combines a guiding mechanism that fuses features and perspective similarity scores, making the constructed 3D cost volume more reasonable in terms of semantic expression, geometric consistency, and visibility trade-off, thereby enhancing the reliability of voxel matching and the accuracy of depth map generation.

[0024] In addition, the present invention constructs an adaptive depth hypothesis adjustment mechanism based on depth variance, which can dynamically expand or contract the search range according to the depth uncertainty of different image regions, thereby reducing redundant calculations while ensuring the estimation coverage rate and improving the generalization ability of the model in different scenarios. Compared with the traditional strategy of using a fixed depth range, the present invention can also maintain good depth regression stability in complex structures or texture-sparse regions.

[0025] Finally, in the 3D point cloud generation and model reconstruction stage, the present invention completes the dense point cloud back-projection by constructing a pixel-level dense depth map and combining image color information. At the same time, unified camera extrinsic parameters are used for multi-viewpoint cloud registration and fusion processing, achieving significant improvements in both spatial consistency and structural continuity. The final 3D mesh model generated by the surface reconstruction algorithm not only retains geometric details but also has better density balance and topological integrity, making it suitable for 3D reconstruction applications under various low-image-number conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention. In the drawings: Figure 1 is a flowchart of a sparse photo synthesis 3D model reconstruction method based on a deep neural network proposed by the present invention; Figure 2 is a schematic structural diagram of a network based on the CasMVSNet structure of a sparse photo synthesis 3D model reconstruction method based on a deep neural network proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] Now, the present invention will be further described in detail with reference to the drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0028] Reference Figure 1-2, A method for reconstructing a sparse photo synthesis 3D model based on a deep neural network, comprising the following steps: S1. Obtain a sparse image set and perform preprocessing, and extract the camera parameters of each image; S2. Extract features from the preprocessed sparse image set to obtain image feature tensors at 1 / 16, 1 / 8, and 1 / 4 spatial resolution levels; S3. At the 1 / 16 spatial resolution level, according to the initial depth hypothesis range of the main view image and in combination with the camera parameters, construct a three-dimensional cost volume corresponding to the main view; S4. Calculate the image feature similarity score and geometric visibility score of the auxiliary view relative to the main view, synthesize the view score according to the weighted fusion strategy, and select the optimal auxiliary view image; S5. Fuse the image feature tensors of the main view image and the selected auxiliary view image, and input them into a three-dimensional convolutional neural network based on the CasMVSNet structure to perform cost aggregation on the three-dimensional cost volume, and generate an initial depth map of the main view image at the 1 / 16 spatial resolution level; S6. Calculate the depth variance according to the initial depth map, adaptively adjust the depth hypothesis range, construct a new three-dimensional cost volume, and generate a depth map at the 1 / 8 resolution level; S7. Repeat step S6 to generate a dense depth map at the 1 / 4 resolution level; S8. Generate a three-dimensional dense point cloud based on the dense depth map, and perform registration and fusion to complete the reconstruction of the 3D model.

[0029] Through constructing a complete three-dimensional reconstruction process for sparse images, the present invention realizes a full-process closed loop of image preprocessing, multi-scale feature extraction, hierarchical depth estimation, view selection, cost volume aggregation, and point cloud reconstruction under the deep neural network architecture, significantly improving the modeling integrity, stability, and accuracy under the condition of sparse image input.

[0030] In this embodiment, the sparse image set includes a main view image and at least one auxiliary view image.

[0031] By introducing the setting of the main view and at least one auxiliary view in the sparse image set, the present invention enables the network to have higher context understanding ability, ensuring effective depth inference and spatial modeling can still be achieved under the condition of the minimum image input amount.

[0032] In this embodiment, the preprocessing includes image alignment, size normalization, image enhancement, and distortion correction.

[0033] By introducing preprocessing operations such as image alignment, size normalization, enhancement, and distortion correction, the present invention effectively improves the geometric consistency between images and the quality of feature matching, and enhances the adaptability and robustness of the network to input sparse images.

[0034] In this embodiment, S3 specifically includes: S31. At the 1 / 16 spatial resolution level, construct an initial depth hypothesis range for the main view image and perform uniform discretization to obtain a depth hypothesis set: ; where D represents the depth hypothesis set, represents the k-th depth value, , represent the minimum and maximum depth values respectively, and M represents the number of depth levels; S32. For each pixel coordinate and each depth value in the feature tensor of the main view image, back-project them to three-dimensional points in the main view camera coordinate system using the main view camera parameters: ; where, represents the three-dimensional point corresponding to the pixel at the k-th depth, represents the inverse operation of the main view camera parameters, x and y represent the pixel positions on the feature tensor, and -1 represents the "inverse" operation; S33. Map the three-dimensional points to each auxiliary view image plane and calculate the two-dimensional projection coordinates on the auxiliary view through the camera parameters of the auxiliary view image: ; where, represents the camera parameters of the i-th auxiliary view image, represents the rotation matrix, represents the translation vector, represents the transpose of the rotation matrix of the main view image, represents the translation vector of the main view image, and represent the two-dimensional projection coordinates of the three-dimensional point in the i-th auxiliary view image, represents the scale factor of the projection; S34. Use bilinear interpolation to sample the feature tensor of the auxiliary view image at the two-dimensional projection coordinate positions to obtain a reprojection feature tensor; S35. Calculate the feature difference cost term between the feature tensor of the main view image and the reprojection feature tensor of each auxiliary view image: ; where, Represents the cost value corresponding to the pixel position of the i-th auxiliary view, represents the L1 norm, represents the feature tensor of the main view image, represents the reprojection feature tensor of the auxiliary view image; S36. Aggregate and stack all auxiliary view images and cost values to construct a three-dimensional cost volume: ; where, represents the aggregated cost value of the main view image at the pixel position (x, y) and depth value Aggregate represents the aggregation operation, and N represents the number of auxiliary view images.

[0035] In the process of constructing the cost volume, the present invention introduces a pixel-level back-projection, auxiliary view reprojection, and feature difference aggregation mechanism, performs spatial transformation based on an accurate camera model, and significantly improves the geometric accuracy and structural expression ability of the three-dimensional cost volume.

[0036] In this embodiment, the specific steps of S4 are as follows: S41. For each auxiliary view image, calculate the image feature similarity score with the main view image: ; where, represents the image feature similarity score between the auxiliary view image and the main view image, and respectively represent the height and width of the feature map of the main view image, and respectively represent the feature tensors of the main view image and the auxiliary view image, represents the L2 norm; S42. In the depth hypothesis set at the current spatial resolution level, according to the pixel back-projection and projection conversion relationship, count the number of projected pixel points of the valid pixels in the main view image in the auxiliary view image, and calculate the geometric visibility score: ; where, represents the geometric visibility score of the auxiliary view image, represents the number of valid pixels in the main view image projected into the auxiliary view image under the depth hypothesis set; S43. Weightedly fuse the image feature similarity score and the geometric visibility score to obtain the view score: ; where, represents the view score, represents the fusion weight coefficient, describing the weight of the feature similarity score in the perspective score; S44. Dynamically adjust the fusion weight coefficient according to the sparsity degree: ; Among them, represents the adjusted fusion weight coefficient, represents the adjustment constant, and min represents taking the minimum value; S45. Sort the perspective scores of all auxiliary views, and select at least one auxiliary view image with the highest score.

[0037] Through comprehensively considering the image feature similarity and geometric visibility score, and dynamically adjusting the perspective fusion weight, the present invention realizes an optimal selection strategy for auxiliary views adapting to the change of image sparsity, effectively improving the utilization rate of perspective information and the overall quality of 3D reconstruction.

[0038] In this embodiment, the S5 specifically includes: S51. Fuse the main view image feature tensor and the selected auxiliary view image along the channel dimension to generate a fused feature tensor; S52. Input the fused feature tensor into a three-level cascaded 3D convolutional neural network module constructed based on the CasMVSNet structure, and complete 3D cost volume aggregation at the 1 / 16 spatial resolution level to generate an aggregated cost volume; S53. Perform Softmax normalization processing on the aggregated cost volume along the depth dimension to obtain the matching probability distribution of each pixel position under the depth hypothesis set at the 1 / 16 spatial resolution level; S54. Perform weighted average according to the matching probability distribution and the corresponding depth value to generate an initial depth map: ; Among them, D(x,y) represents the initial depth map of the main view image, represents the k-th depth value, represents the matching probability distribution under the k-th depth hypothesis, and M represents the number of depth levels.

[0039] By inputting the fused feature tensor into a three-level cascaded convolutional network based on the CasMVSNet structure, the present invention realizes staged cost volume aggregation and depth estimation, enhancing the network's expression ability for sparse images and improving the matching accuracy and detail retention effect of the initial depth map.

[0040] In this embodiment, the 3D cost volume aggregation specifically includes: Guided feature generation stage: Perform layer-by-layer downsampling on the fused feature tensor to obtain low-resolution guided features, medium-resolution guided features, and high-resolution guided features respectively; Three-stage cost aggregation stage: After fusing the three-dimensional cost volume and the low-resolution guiding features, input them into the first three-dimensional convolutional module to output the first cost volume. After fusing the first cost volume and the medium-resolution guiding features, input them into the second three-dimensional convolutional module to output the second cost volume. After fusing the second cost volume and the high-resolution guiding features, input them into the third three-dimensional convolutional module to output the aggregated cost volume.

[0041] Based on the initial depth map matching probability distribution, the present invention adaptively calculates the depth variance and adjusts the search range, realizes the modeling and response to the depth uncertainty of different image regions, and significantly improves the accuracy and stability of depth estimation at subsequent high-resolution levels.

[0042] In this embodiment, the aggregated cost volume has the same spatial and depth dimensions as the original three-dimensional cost volume.

[0043] The present invention constructs a cost volume by adaptively sampling the pixel-level depth interval at the 1 / 8 resolution level, and combines a hierarchical convolutional network to complete the depth map generation, realizing a more refined and low-redundancy depth estimation process, and improving the reliability of the middle-level structure modeling.

[0044] In this embodiment, S6 specifically includes: S61. Calculate the depth variance according to the matching probability distribution after normalizing the cost volume: ; Among them, represents the depth variance at pixel (x, y), represents the matching probability distribution under the k-th depth hypothesis, represents the k-th depth value, and D(x, y) represents the initial depth map of the main view image; S62. Calculate the depth search range extension amount according to the depth variance and a preset coefficient: ; Among them, represents the depth search range extension amount at pixel (x, y), represents the preset coefficient, represents the depth standard deviation at pixel (x, y); S63. Centering on the initial depth map of the main view image, dynamically determine the depth hypothesis range at the 1 / 8 resolution level according to the depth search range extension amount: ; Among them, R(x, y) represents the depth hypothesis range of pixel (x, y) at the 1 / 8 resolution level, and D(x, y) represents the initial depth map of the main view image; S64. Calculate the depth variance and the adaptive depth search range for each pixel on the entire front view image respectively, and construct the depth hypothesis range at the 1 / 8 resolution level globally; S65. Reconstruct the depth hypothesis set and the three-dimensional cost volume of the front view image to generate a depth map at the 1 / 8 resolution level.

[0045] Through the repeated depth variance-driven adaptive search and inference mechanism, the present invention completes the generation of a dense depth map at the 1 / 4 resolution, ensuring that the final output has higher depth continuity and boundary fidelity, providing an accurate geometric basis for the subsequent three-dimensional point cloud back-projection.

[0046] In this embodiment, the specific steps of S8 are as follows: S81. Obtain the dense depth map, and in combination with the camera parameters of the front view image, back-project the position of each pixel in the front view image and the corresponding depth value into the camera coordinate system to calculate the three-dimensional spatial position coordinates; S82. Bind the three-dimensional spatial coordinates corresponding to each pixel position with the color information in the front view image to generate a three-dimensional dense point cloud containing geometric positions and color attributes; S83. Generate three-dimensional dense point clouds for the front view image and the auxiliary view images respectively, and attach the corresponding external camera parameters, where the external parameters include the rotation matrix and the translation vector corresponding to each view; S84. Register the three-dimensional dense point clouds corresponding to all views through the external camera parameters and map them into the same coordinate system; S85. Perform fusion optimization processing on the registered three-dimensional dense point clouds to generate a fused point cloud with continuous structure and strong spatial consistency; S86. Perform a mesh reconstruction operation based on the fused point cloud to construct a three-dimensional topological structure and form a complete 3D model through surface connection.

[0047] By back-projecting the dense depth map to generate a point cloud and combining the camera poses to complete multi-view registration and fusion optimization, the present invention finally constructs a high-quality three-dimensional model with continuous structure and unified coordinates, significantly improving the reconstruction integrity and spatial consistency under sparse image conditions.

[0048] Example 1: To verify the feasibility of the present invention in implementation, the present invention is applied to a digital restoration task of a historical and cultural building. Due to geographical location restrictions and on-site control, the shooting conditions are extremely limited. Only 3 - 4 images can be obtained from a fixed position by a mobile phone or a portable drone. There are obvious problems such as sparse images, discontinuous shooting perspectives, and uneven image quality, belonging to a typical sparse image three-dimensional reconstruction scenario.

[0049] Traditional structured light scanning and lidar technologies cannot be applied in this scenario. On the one hand, due to high costs and large equipment volume, and on the other hand, they are also affected by objective factors such as environmental occlusion and lighting changes. Conventional MVS methods based on dense multi-view images perform extremely poorly under the condition of insufficient image quantity, and there are obvious holes, misalignments, and surface jitters in the reconstruction results. Therefore, in this embodiment, the sparse photo-based three-dimensional reconstruction method based on a deep neural network proposed by the present invention is applied to this task.

[0050] During the on-site acquisition process, only one portable drone was used to take three view images in a triangular layout in front of the main hall of the ancient building. The image resolution was 5472×3648. After shooting, the images were aligned, distortion corrected, and size standardized by the image preprocessing module described in the present invention, and the internal and corresponding external parameters of the images were extracted. Through multi-scale image feature extraction and cascaded depth estimation, the present invention generated an initial depth map at the 1 / 16 resolution level, and then through two-level depth variance calculation and adaptive adjustment of the search range, a dense depth map was output at the 1 / 4 level. Finally, a complete three-dimensional point cloud was constructed and a mesh model was generated through the processes of back-projection, registration, and fusion.

[0051] In the comparison with common three-dimensional reconstruction methods, the present invention demonstrates significant advantages. To ensure the credibility of the experimental results, COLMAP (traditional SfM+MVS method), MVSNet (classical depth MVS method), CasMVSNet (original cascaded method), and the method of the present invention were used for comparison during on-site reconstruction, and the reconstruction time, point cloud density, reconstruction accuracy (compared with lidar data), surface integrity, and occlusion area restoration rate were tested under the same input image quantity and computing resource limitations.

[0052] From the actual operation data, it can be seen that under the condition of the same input of 3 sparse images, the model generated by the COLMAP method has serious geometric distortions, there are a large number of holes in the MVSNet model, and although CasMVSNet has improvements, it is still not ideal in visibility selection and depth boundary transition. The present invention effectively retains the structural details and restores the continuity of the occluded areas by using view selection, depth variance-driven hierarchical search, and guided feature aggregation.

[0053] Table 1 Comparison table of three-dimensional reconstruction performance between the present invention and existing methods under sparse image conditions ; From the experimental comparison results shown in Table 1, it can be seen that under the same sparse image input conditions (only 3 high-resolution images), the present invention shows technical effects that are significantly superior to existing mainstream methods in multiple key performance indicators.

[0054] In terms of reconstruction accuracy, the average reconstruction error of the present invention is 0.85 cm, far lower than 3.42 cm of COLMAP, 2.05 cm of MVSNet, and 1.24 cm of CasMVSNet, indicating that the present invention can still stably output a high-precision geometric structure under low image redundancy conditions and has excellent depth regression ability.

[0055] In terms of the reduction rate of occluded areas, the present invention reaches 88.3%, which is more than 14 percentage points higher than that of CasMVSNet, significantly alleviating the problem that traditional methods are prone to reconstruction failure in areas with insufficient visibility, and fully demonstrating the innovation of the present invention in terms of view selection and feature guidance mechanism.

[0056] From the perspective of the point cloud density index, the point cloud density output by the present invention reaches 1,860 points per square meter, which is much higher than that of traditional algorithms such as MVSNet and COLMAP, and is also about 30% higher than that of CasMVSNet. This shows that the present invention can not only identify the main structure but also effectively capture detailed features, providing a rich geometric basis for subsequent model surface reconstruction.

[0057] In terms of efficiency, the entire reconstruction process of the present invention only takes 14 minutes, which is better than 22 minutes of MVSNet and 16 minutes of CasMVSNet, indicating that the present invention has been optimized in terms of algorithm structure and computing resource scheduling and has higher operating efficiency.

[0058] In terms of resource occupancy, the present invention controls the GPU video memory usage within 3.5 GB, which is significantly lower than that of MVSNet and CasMVSNet, and improves the hardware adaptation ability on the premise of ensuring accuracy.

[0059] In addition, in terms of the surface integrity score, the present invention reaches 0.89, which is better than 0.73 of CasMVSNet and 0.58 of MVSNet, indicating that the present invention performs excellently in terms of texture area, edge details, and continuity, and can generate a high-quality three-dimensional model with a complete structure and clear boundaries.

[0060] The above is only the preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A method for reconstructing a sparse photo synthesis 3D model based on a deep neural network, characterized in that, It includes the following steps: S1. Obtain a sparse image set, perform preprocessing, and extract the camera parameters of each image; S2. Extract features from the preprocessed sparse image set to obtain image feature tensors at 1 / 16, 1 / 8, and 1 / 4 spatial resolution levels; S3. At the 1 / 16 spatial resolution level, according to the initial depth hypothesis range of the main view image and in combination with the camera parameters, construct a three-dimensional cost volume corresponding to the main view; S4. Calculate the image feature similarity score and geometric visibility score of the auxiliary view relative to the main view, synthesize the view score according to the weighted fusion strategy, and select the optimal auxiliary view image; S5. Fuse the image feature tensors of the main view image and the selected auxiliary view image, and input them into a three-dimensional convolutional neural network based on the CasMVSNet structure to perform cost aggregation on the three-dimensional cost volume, and generate an initial depth map of the main view image at the 1 / 16 spatial resolution level; S6. Calculate the depth variance according to the initial depth map, adaptively adjust the depth hypothesis range, construct a new three-dimensional cost volume, and generate a depth map at the 1 / 8 resolution level; S7. Repeat step S6 to generate a dense depth map at the 1 / 4 resolution level; S8. Generate a three-dimensional dense point cloud based on the dense depth map, and perform registration and fusion to complete the reconstruction of the 3D model.

2. The method for reconstructing a 3D model by sparse photo synthesis based on a deep neural network according to claim 1, characterized in that The sparse image set includes one main view image and at least one auxiliary view image.

3. A method for reconstructing a sparse photo synthesis 3D model based on a deep neural network according to claim 1, characterized in that, The preprocessing includes image alignment, size normalization, image enhancement, and distortion correction.

4. A method for reconstructing a 3D model by sparse photo synthesis based on a deep neural network according to claim 1, characterized in that, The specific content of S3 includes: S31. At the 1 / 16 spatial resolution level, construct an initial depth hypothesis range for the main view image and perform uniform discretization to obtain a depth hypothesis set; S32. For each pixel coordinate and each depth value in the feature tensor of the main view image, back-project them to three-dimensional points in the main view camera coordinate system using the main view camera parameters; S33. Map the three-dimensional points to each auxiliary view image plane, and calculate the two-dimensional projection coordinates on the auxiliary view through the camera parameters of the auxiliary view image; S34. Use bilinear interpolation to sample the feature tensor of the auxiliary view image at the two-dimensional projection coordinate position to obtain a reprojection feature tensor; S35. Calculate the feature difference cost term between the feature tensor of the main view image and the reprojection feature tensors of each auxiliary view image; S36. Aggregate and stack all auxiliary view images and cost values to construct a three-dimensional cost volume.

5. A method for reconstructing a 3D model by sparse photo synthesis based on a deep neural network according to claim 1, characterized in that, The specific content of S4 includes: S41. For each auxiliary view image, calculate the image feature similarity score with the main view image; S42. In the depth hypothesis set at the current spatial resolution level, according to the pixel back-projection and projection conversion relationship, count the number of projected pixel points of the valid pixels in the main view image in the auxiliary view image, and calculate the geometric visibility score; S43. Perform weighted fusion on the image feature similarity score and the geometric visibility score to obtain the view score; S44. Dynamically adjust the fusion weight coefficient according to the sparsity; S45. Sort the view scores of all auxiliary views, and select at least one auxiliary view image with the highest score.

6. A method for reconstructing a 3D model by sparse photo synthesis based on a deep neural network according to claim 1, characterized in that The S5 specifically includes: S51, fusing the main view image feature tensor and the selected auxiliary view image according to the channel dimension to generate a fused feature tensor; S52, inputting the fused feature tensor into a three-level cascaded three-dimensional convolutional neural network module built based on the CasMVSNet structure, completing three-dimensional cost volume aggregation at a 1 / 16 spatial resolution level, and generating an aggregated cost volume; S53, performing Softmax normalization processing on the aggregated cost volume along the depth dimension to obtain a matching probability distribution of each pixel position under a depth hypothesis set at a 1 / 16 spatial resolution level; S54, performing weighted averaging based on the matching probability distribution and the corresponding depth value to generate an initial depth map: ； Among them, D(x, y) represents the initial depth map of the front view image, represents the k-th depth value, represents the matching probability distribution under the k-th depth hypothesis, and M represents the number of depth levels.

7. A method for reconstructing a 3D model by sparse photo synthesis based on a deep neural network according to claim 6, characterized in that, The three-dimensional cost volume aggregation specifically includes: Guided feature generation stage: The fused feature tensor is downsampled layer by layer to obtain low-resolution guided features, medium-resolution guided features, and high-resolution guided features respectively; Three-stage cost aggregation stage: the three-dimensional cost volume and the low-resolution guided features are fused and input into the first-layer three-dimensional convolution module to output the first-layer cost volume; the first-layer cost volume and the medium-resolution guided features are fused and input into the second-layer three-dimensional convolution module to output the second-layer cost volume; the second-layer cost volume and the high-resolution guided features are fused and input into the third-layer three-dimensional convolution module to output the aggregated cost volume.

8. A method for reconstructing a 3D model by sparse photo synthesis based on a deep neural network according to claim 7, characterized in that, The aggregated cost volume has the same spatial and depth dimensions as the original 3D cost volume.

9. A method for reconstructing a 3D model by sparse photo synthesis based on a deep neural network according to claim 1, characterized in that, The S6 specifically includes: S61. Calculate the depth variance based on the normalized matching probability distribution of the cost volume: ； Among them, represents the depth variance at pixel (x, y), represents the matching probability distribution under the k-th depth hypothesis, represents the k-th depth value, and D(x, y) represents the initial depth map of the main view image; S62, calculating the depth search range extension amount according to the depth variance and the preset coefficient: ； Among them, represents the depth search range expansion amount at pixel (x, y), represents a preset coefficient, represents the depth standard deviation at pixel (x, y); S63, taking the initial depth map of the main view image as the center, dynamically determining the depth assumption range at the 1 / 8 resolution level according to the depth search range extension amount: ； Where R(x,y) represents the depth assumption range of pixel (x,y) at 1 / 8 resolution level, and D(x,y) represents the initial depth map of the main view image; S64, calculating the depth variance and the adaptive depth search range for each pixel on the entire main view image, and constructing a global depth hypothesis range at a 1 / 8 resolution level; S65. Reconstruct the depth hypothesis set and the three-dimensional cost volume of the main view image to generate a depth map at a 1 / 8 resolution level.

10. A method for reconstructing a 3D model by sparse photo synthesis based on a deep neural network according to claim 1, characterized in that, The S8 specifically includes: S81, obtaining a dense depth map, combining the camera parameters of the main view image, back-projecting the position of each pixel in the main view image and the corresponding depth value into the camera coordinate system, and calculating the three-dimensional space position coordinates; S82, binding the three-dimensional space coordinates corresponding to each pixel position with the color information in the primary view image to generate a three-dimensional dense point cloud including geometric positions and color attributes; S83, generating three-dimensional dense point clouds for the main view image and the auxiliary view image respectively, and adding corresponding camera extrinsic parameters, wherein the extrinsic parameters include a rotation matrix and a translation vector corresponding to each view; S84, registering the three-dimensional dense point clouds corresponding to all views through camera extrinsic parameters and mapping them to the same coordinate system; S85. Perform fusion optimization processing on the registered three-dimensional dense point cloud to generate a fused point cloud with continuous structure and strong spatial consistency; S86. Perform mesh reconstruction operations based on the fused point cloud to construct a three-dimensional topological structure and form a complete 3D model through surface connection.

Images