A 3D model reconstruction method based on sparse photo synthesis based on deep neural network

Abstract

The present invention discloses a method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network, comprising the following steps: S1, obtaining a sparse image set and preprocessing it; S2, performing feature extraction to obtain an image feature tensor; S3, constructing a three-dimensional cost volume corresponding to the main view at a 1 / 16 spatial resolution level; S4, calculating the scores of the auxiliary views relative to the main view and selecting the optimal auxiliary view image; S5, using a three-dimensional convolutional neural network to perform cost aggregation on the three-dimensional cost volume; S6, calculating the depth variance and constructing a new three-dimensional cost volume; S7, repeating step S6 to generate a dense depth map at a 1 / 4 resolution level; S8, completing the reconstruction of the 3D model based on the dense depth map. The present invention employs a cascaded neural network and a viewpoint adaptation strategy to achieve high-quality three-dimensional reconstruction under sparse image conditions, with the advantages of high efficiency, excellent accuracy, and strong robustness.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction, and in particular to a sparse photo synthesis 3D model reconstruction method based on a deep neural network. Background Art

[0002] With the continuous development of computer vision technology, 3D reconstruction methods based on image sequences have been widely used in autonomous driving, intelligent manufacturing, cultural relics protection, virtual reality and other fields. Traditional 3D reconstruction methods usually rely on structured light, laser scanning or multi-view geometry to model the scene. Among them, 3D reconstruction technology based on multi-view images is particularly prominent because it can achieve high-precision restoration of the scene geometric structure with only conventional RGB image input, and has the significant advantages of low cost and strong adaptability.

[0003] In recent years, with the introduction of deep learning, especially convolutional neural networks, 3D reconstruction tasks have made breakthrough progress in key modules such as feature extraction, depth estimation and voxel aggregation. Representative methods such as MVSNet and R-MVSNet have effectively improved the robustness and accuracy of multi-view depth estimation by constructing three-dimensional cost volumes between viewpoints and using three-dimensional convolutional neural networks for aggregation and regression, reducing memory overhead and making 3D reconstruction of high-resolution images possible.

[0004] However, most of these deep neural network-based 3D reconstruction methods are based on the assumption of dense image input and sufficient view coverage. In real-world applications, particularly in drone aerial photography, mobile devices, security monitoring, cultural relic photography, or historical image restoration, only a very small number of image samples (e.g., 2-3) are available, with incomplete view coverage and limited overlap. Under these sparse image input conditions, existing methods commonly suffer from reduced reconstruction accuracy, discontinuous model output, and incomplete occluded areas.

[0005] First, most current 3D reconstruction networks use fixed image pairs to construct the cost volume, but lack an effective selection strategy for auxiliary views. In the case of sparse images, how to choose the most informative auxiliary view becomes a key factor affecting the reconstruction effect. The lack of a dynamic view selection mechanism will result in a large number of redundant or invalid auxiliary images being included in the aggregation calculation, which not only affects the computational efficiency, but also reduces the geometric expression ability of the cost volume.

[0006] Secondly, existing methods often construct voxels based on a fixed depth hypothesis range, ignoring the depth uncertainty between different image regions. In areas with insufficient image information or complex structures, a fixed depth range may lead to a large number of invalid samples or loss of key depth intervals, thereby affecting subsequent voxel aggregation and depth estimation accuracy. Although some methods have introduced a coarse-to-fine hierarchical design, they have not yet achieved a truly adaptive depth range adjustment mechanism.

[0007] Furthermore, most of the common three-dimensional cost volume aggregation methods currently use direct stacking or simple fusion of image features, lacking explicit modeling of factors such as feature importance and geometric visibility. Under sparse image conditions, this blind aggregation easily introduces noise, resulting in feature distortion and matching failure, which in turn affects the quality of the depth map. Especially in occluded areas or background areas with a single texture, the reconstruction results of existing methods are prone to problems such as holes, jitter, or error boundary diffusion.

[0008] In addition, in the post-reconstruction processing stage, most methods only perform single-viewpoint depth regression on the main view image, and fail to effectively utilize the information complementarity between different perspectives. In the point cloud reconstruction stage, if the point clouds generated from different perspectives lack a unified coordinate alignment and fusion strategy, it often leads to model structure fractures, inconsistent accuracy, and even deformation problems. At the same time, for the case of inconsistent point cloud density, existing methods lack an effective sparse-dense fusion mechanism, which affects the structural integrity and surface continuity of the final three-dimensional model.

[0009] In summary, although multi-view 3D reconstruction technology based on deep learning has made significant progress in recent years, it still faces many challenges under sparse image input conditions, including the lack of an information-driven auxiliary view selection mechanism, the lack of an adaptive depth hypothesis construction strategy based on depth uncertainty perception, the lack of a cost volume aggregation method driven jointly by fusion features and view visibility, and the lack of a 3D model generation mechanism that balances coordinate consistency and point cloud density.

[0010] Therefore, there is an urgent need to propose a 3D reconstruction method that is structurally sound, computationally efficient, and adaptable for sparse image input conditions, which can complete the task of reconstructing 3D models with high quality, structural continuity, and accurate texture restoration under extremely few image inputs and non-ideal shooting conditions. Summary of the Invention

[0011] One purpose of the present invention is to propose a sparse photo synthesis 3D model reconstruction method based on deep neural network. The present invention adopts cascade neural network and view adaptation strategy, and realizes high-quality 3D model reconstruction under sparse image input conditions through multi-scale depth estimation, feature-guided aggregation and point cloud fusion methods. Compared with traditional methods that rely on dense image input, the present invention has the advantages of low requirement on the number of images, high reconstruction accuracy, strong adaptability and high computational efficiency.

[0012] A method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network according to an embodiment of the present invention includes the following steps:

[0013] S1, obtain a sparse image set and preprocess it, and extract the camera parameters of each image;

[0014] S2. Perform feature extraction on the preprocessed sparse image set to obtain image feature tensors at 1 / 16, 1 / 8, and 1 / 4 spatial resolution levels;

[0015] S3, at the 1 / 16 spatial resolution level, based on the initial depth assumption range of the main view image and combined with the camera parameters, construct the 3D cost volume corresponding to the main view;

[0016] S4. Calculate the image feature similarity score and geometric visibility score of the auxiliary view relative to the main view, synthesize the view angle score according to the weighted fusion strategy, and select the optimal auxiliary view image;

[0017] S5. The image feature tensors of the main view image and the selected auxiliary view image are fused and input into a 3D convolutional neural network based on the CasMVSNet structure to perform cost aggregation on the 3D cost volume to generate an initial depth map of the main view image at a 1 / 16 spatial resolution level.

[0018] S6. Calculate the depth variance based on the initial depth map, adaptively adjust the depth hypothesis range, construct a new 3D cost volume, and generate a depth map at a 1 / 8 resolution level.

[0019] S7, repeat step S6 to generate a dense depth map at a 1 / 4 resolution level;

[0020] S8. Generate a three-dimensional dense point cloud based on the dense depth map, and perform registration and fusion to complete the reconstruction of the 3D model.

[0021] Optionally, the sparse image set includes one primary view image and at least one auxiliary view image.

[0022] Optionally, the preprocessing includes image alignment, size standardization, image enhancement and distortion correction.

[0023] Optionally, the S3 specifically includes:

[0024] S31. At the 1 / 16 spatial resolution level, construct an initial depth hypothesis range for the main view image and perform uniform discretization to obtain a depth hypothesis set:

[0025] ;

[0026] Where D represents the set of depth hypotheses, represents the kth depth value, 、 Represent the minimum and maximum depth values ​​respectively, and M represents the number of depth levels;

[0027] S32. For each pixel coordinate and each depth value in the feature tensor of the main view image, back-project the main view camera parameters to a three-dimensional point in the main view camera coordinate system:

[0028] ;

[0029] in, represents the 3D point corresponding to the pixel at the kth depth, Represents the inverse operation of the main view camera parameters, x and y represent the pixel position on the feature tensor, and -1 represents the "inverse" operation;

[0030] S33. Map the three-dimensional point to each auxiliary view image plane, and calculate the two-dimensional projection coordinates on the auxiliary view using the camera parameters of the auxiliary view image:

[0031] ;

[0032] in, represents the camera parameters of the i-th auxiliary view image, represents the rotation matrix, represents the translation vector, represents the transpose of the main view image rotation matrix, represents the translation vector of the main view image, and represents the two-dimensional projection coordinates of the three-dimensional point in the i-th auxiliary view image, represents the scale factor of the projection;

[0033] S34. Sampling the feature tensor of the auxiliary view image at the two-dimensional projection coordinate position using a bilinear interpolation method to obtain a reprojected feature tensor;

[0034] S35. Calculate the feature difference cost term of the feature tensor of the main view image and the reprojected feature tensor of each auxiliary view image:

[0035] ;

[0036] in, represents the cost value of the pixel position corresponding to the i-th auxiliary view, represents the L1 norm, represents the feature tensor of the main view image, Reprojected feature tensor representing the auxiliary view image;

[0037] S36. Aggregate and stack all auxiliary view images and cost values ​​to construct a three-dimensional cost volume:

[0038] ;

[0039] in, Indicates the main view image at pixel position (x, y) and depth value Aggregate represents the aggregation operation, and N represents the number of auxiliary view images.

[0040] Optionally, the S4 specifically includes:

[0041] S41. For each auxiliary view image, calculate the image feature similarity score between it and the main view image:

[0042] ;

[0043] in, represents the image feature similarity score between the auxiliary view image and the main view image, and Represent the height and width of the feature map of the main view image, and Represent the feature tensors of the main view image and the auxiliary view image respectively, represents the L2 norm;

[0044] S42. In the depth hypothesis set at the current spatial resolution level, based on the pixel back-projection and projection transformation relationship, count the number of projected pixels of valid pixels in the primary view image in the auxiliary view image, and calculate the geometric visibility score:

[0045] ;

[0046] in, represents the geometric visibility score of the auxiliary view image, represents the number of valid pixels in the primary view image that are projected into the auxiliary view image under the depth hypothesis set;

[0047] S43. Perform weighted fusion of the image feature similarity score and the geometric visibility score to obtain a viewing angle score:

[0048] ;

[0049] in, represents the viewing angle score, Represents the fusion weight coefficient, which describes the weight of the feature similarity score in the view score;

[0050] S44. Dynamically adjust the fusion weight coefficient according to the sparsity level:

[0051] ;

[0052] in, represents the adjusted fusion weight coefficient, Indicates the adjustment constant, min indicates the minimum value;

[0053] S45 . Sort the viewing angle scores of all auxiliary views, and select at least one auxiliary view image with the highest score.

[0054] Optionally, the S5 specifically includes:

[0055] S51, fusing the main view image feature tensor and the selected auxiliary view image according to the channel dimension to generate a fused feature tensor;

[0056] S52, inputting the fused feature tensor into a three-level cascaded three-dimensional convolutional neural network module built based on the CasMVSNet structure, completing three-dimensional cost volume aggregation at a 1 / 16 spatial resolution level, and generating an aggregated cost volume. The CasMVSNet is a cascaded multi-view stereo vision network based on deep learning;

[0057] S53, performing Softmax normalization processing on the aggregated cost volume along the depth dimension to obtain a matching probability distribution of each pixel position under a depth hypothesis set at a 1 / 16 spatial resolution level;

[0058] S54. Perform weighted averaging based on the matching probability distribution and the corresponding depth values ​​to generate an initial depth map:

[0059] ;

[0060] Where D(x,y) represents the initial depth map of the main view image, represents the kth depth value, represents the matching probability distribution under the k-th depth hypothesis, and M represents the number of depth levels.

[0061] Optionally, the three-dimensional cost volume aggregation specifically includes:

[0062] Guided feature generation stage: The fused feature tensor is downsampled layer by layer to obtain low-resolution guided features, medium-resolution guided features, and high-resolution guided features respectively;

[0063] Three-stage cost aggregation stage: the three-dimensional cost volume and the low-resolution guided features are fused and input into the first-layer three-dimensional convolution module, and the first-layer cost volume is output. The first-layer cost volume and the medium-resolution guided features are fused and input into the second-layer three-dimensional convolution module, and the second-layer cost volume is output. The second-layer cost volume and the high-resolution guided features are fused and input into the third-layer three-dimensional convolution module, and the aggregated cost volume is output.

[0064] Optionally, the aggregated cost volume has the same spatial and depth dimensions as the original 3D cost volume.

[0065] Optionally, the S6 specifically includes:

[0066] S61. Calculate the depth variance based on the normalized matching probability distribution of the cost volume:

[0067] ;

[0068] in, represents the depth variance at pixel (x,y), represents the matching probability distribution under the k-th depth hypothesis, represents the kth depth value, and D(x,y) represents the initial depth map of the main view image;

[0069] S62. Calculate the depth search range extension amount according to the depth variance and the preset coefficient:

[0070] ;

[0071] in, Indicates the depth search range extension at pixel (x,y), Indicates the preset coefficient, Indicates the depth standard deviation at pixel (x,y);

[0072] S63. Taking the initial depth map of the main view image as the center, dynamically determine the depth hypothesis range at the 1 / 8 resolution level according to the depth search range extension amount:

[0073] ;

[0074] Where R(x,y) represents the depth hypothesis range of pixel (x,y) at 1 / 8 resolution level, and D(x,y) represents the initial depth map of the main view image.

[0075] S64, calculating the depth variance and the adaptive depth search range for each pixel in the entire primary view image, and constructing a global depth hypothesis range at a 1 / 8 resolution level;

[0076] S65. Reconstruct the depth hypothesis set and the three-dimensional cost volume of the main view image to generate a depth map at a 1 / 8 resolution level.

[0077] Optionally, the S8 specifically includes:

[0078] S81. Obtain a dense depth map, combine it with the camera parameters of the main view image, back-project the position of each pixel in the main view image and the corresponding depth value into the camera coordinate system, and calculate the three-dimensional space position coordinates;

[0079] S82, binding the three-dimensional spatial coordinates corresponding to each pixel position with the color information in the primary view image to generate a three-dimensional dense point cloud containing geometric positions and color attributes;

[0080] S83, generating three-dimensional dense point clouds for the main view image and the auxiliary view image respectively, and adding corresponding camera extrinsic parameters, wherein the extrinsic parameters include the rotation matrix and translation vector corresponding to each view;

[0081] S84, registering the three-dimensional dense point clouds corresponding to all views using camera extrinsic parameters and mapping them to the same coordinate system;

[0082] S85, performing fusion optimization processing on the registered three-dimensional dense point cloud to generate a fused point cloud with continuous structure and strong spatial consistency;

[0083] S86. Perform mesh reconstruction based on the fused point cloud to construct a three-dimensional topological structure and form a complete 3D model through surface connection.

[0084] The beneficial effects of the present invention are:

[0085] First, the present invention proposes a three-dimensional model reconstruction method based on deep neural networks for sparse image input conditions, which can effectively overcome the problems faced by existing technologies such as incomplete three-dimensional structure, reduced depth estimation accuracy, and failure to reconstruct occluded areas when the number of images is limited and the viewing angle coverage is insufficient. By gradually optimizing the depth estimation process at multi-scale spatial resolution levels and combining an adaptive view selection mechanism with a depth hypothesis adjustment strategy, the present invention significantly improves the three-dimensional modeling capability under low image input density.

[0086] Secondly, the present invention introduces preprocessing steps such as image alignment, enhancement and normalization in the initial stage, which enhances the matching feasibility of the input image, and uses a multi-scale feature extraction module to realize multi-layer perception of image semantics and structural details, laying a solid feature foundation for subsequent voxel construction and aggregation. In the depth estimation link, the present invention not only uses a cascaded three-dimensional convolutional aggregation network based on CasMVSNet to refine the cost volume layer by layer, but also combines the guidance mechanism of fusion features and perspective similarity scoring, so that the constructed three-dimensional cost volume is more reasonable in terms of semantic expression, geometric consistency and visibility trade-offs, thereby improving the reliability of voxel matching and the accuracy of depth map generation.

[0087] In addition, the present invention constructs an adaptive depth hypothesis adjustment mechanism based on depth variance, which can dynamically expand or shrink the search range according to the depth uncertainty of different image regions, thereby reducing redundant calculations while ensuring the estimated coverage, and improving the generalization ability of the model in different scenarios. Compared with the traditional strategy of using a fixed depth range, the present invention can also maintain good depth regression stability in complex structures or sparse texture areas.

[0088] Finally, during the 3D point cloud generation and model reconstruction phase, the present invention constructs a dense pixel-level depth map and combines it with image color information to perform dense point cloud back-projection. Furthermore, it uses unified camera extrinsics for multi-view point cloud registration and fusion, achieving significant improvements in both spatial consistency and structural continuity. The resulting 3D mesh model, generated through a surface reconstruction algorithm, retains geometric details while exhibiting improved density balance and topological integrity, making it suitable for a variety of 3D reconstruction applications requiring low image counts. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0090] Figure 1 This is a flowchart of a 3D model reconstruction method based on deep neural network synthesized from sparse photos proposed by the present invention;

[0091] Figure 2 This is a structural diagram of a network based on the CasMVSNet structure for a sparse photo synthesis 3D model reconstruction method based on a deep neural network proposed in the present invention. DETAILED DESCRIPTION

[0092] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0093] refer to Figure 1-2 A 3D model reconstruction method based on sparse photo synthesis based on deep neural network includes the following steps:

[0094] S1, obtain a sparse image set and preprocess it, and extract the camera parameters of each image;

[0095] S2. Perform feature extraction on the preprocessed sparse image set to obtain image feature tensors at 1 / 16, 1 / 8, and 1 / 4 spatial resolution levels;

[0096] S3, at the 1 / 16 spatial resolution level, based on the initial depth assumption range of the main view image and combined with the camera parameters, construct the 3D cost volume corresponding to the main view;

[0097] S4. Calculate the image feature similarity score and geometric visibility score of the auxiliary view relative to the main view, synthesize the view angle score according to the weighted fusion strategy, and select the optimal auxiliary view image;

[0098] S5. The image feature tensors of the main view image and the selected auxiliary view image are fused and input into a 3D convolutional neural network based on the CasMVSNet structure to perform cost aggregation on the 3D cost volume to generate an initial depth map of the main view image at a 1 / 16 spatial resolution level.

[0099] S6. Calculate the depth variance based on the initial depth map, adaptively adjust the depth hypothesis range, construct a new 3D cost volume, and generate a depth map at a 1 / 8 resolution level.

[0100] S7, repeat step S6 to generate a dense depth map at a 1 / 4 resolution level;

[0101] S8. Generate a three-dimensional dense point cloud based on the dense depth map, and perform registration and fusion to complete the reconstruction of the 3D model.

[0102] By constructing a complete sparse image 3D reconstruction process, the present invention realizes a closed-loop process of image preprocessing, multi-scale feature extraction, hierarchical depth estimation, viewpoint selection, cost volume aggregation and point cloud reconstruction under the deep neural network architecture, significantly improving the modeling integrity, stability and accuracy under sparse image input conditions.

[0103] In this embodiment, the sparse image set includes one main view image and at least one auxiliary view image.

[0104] By introducing the setting of a main view and at least one auxiliary view in a sparse image set, the present invention enables the network to have a higher context understanding ability, ensuring that effective deep reasoning and spatial modeling can still be achieved with a minimum amount of image input.

[0105] In this embodiment, the preprocessing includes image alignment, size standardization, image enhancement and distortion correction.

[0106] By introducing preprocessing operations such as image alignment, size standardization, enhancement and distortion correction, the present invention effectively improves the geometric consistency and feature matching quality between images, and enhances the adaptability and robustness of the network to sparse input images.

[0107] In this embodiment, S3 specifically includes:

[0108] S31. At the 1 / 16 spatial resolution level, construct an initial depth hypothesis range for the main view image and perform uniform discretization to obtain a depth hypothesis set:

[0109] ;

[0110] Where D represents the set of depth hypotheses, represents the kth depth value, 、 Represent the minimum and maximum depth values ​​respectively, and M represents the number of depth levels;

[0111] S32. For each pixel coordinate and each depth value in the feature tensor of the main view image, back-project the main view camera parameters to a three-dimensional point in the main view camera coordinate system:

[0112] ;

[0113] in, represents the 3D point corresponding to the pixel at the kth depth, Represents the inverse operation of the main view camera parameters, x and y represent the pixel position on the feature tensor, and -1 represents the "inverse" operation;

[0114] S33. Map the three-dimensional point to each auxiliary view image plane, and calculate the two-dimensional projection coordinates on the auxiliary view using the camera parameters of the auxiliary view image:

[0115] ;

[0116] in, represents the camera parameters of the i-th auxiliary view image, represents the rotation matrix, represents the translation vector, represents the transpose of the main view image rotation matrix, represents the translation vector of the main view image, and represents the two-dimensional projection coordinates of the three-dimensional point in the i-th auxiliary view image, represents the scale factor of the projection;

[0117] S34. Sampling the feature tensor of the auxiliary view image at the two-dimensional projection coordinate position using a bilinear interpolation method to obtain a reprojected feature tensor;

[0118] S35. Calculate the feature difference cost term of the feature tensor of the main view image and the reprojected feature tensor of each auxiliary view image:

[0119] ;

[0120] in, represents the cost value of the pixel position corresponding to the i-th auxiliary view, represents the L1 norm, represents the feature tensor of the main view image, Reprojected feature tensor representing the auxiliary view image;

[0121] S36. Aggregate and stack all auxiliary view images and cost values ​​to construct a three-dimensional cost volume:

[0122] ;

[0123] in, Indicates the main view image at pixel position (x, y) and depth value Aggregate represents the aggregation operation, and N represents the number of auxiliary view images.

[0124] The present invention introduces pixel-level back-projection, auxiliary view reprojection and feature difference aggregation mechanisms in the cost volume construction process, performs spatial transformation based on an accurate camera model, and significantly improves the geometric accuracy and structural expression ability of the three-dimensional cost volume.

[0125] In this embodiment, the S4 specifically includes:

[0126] S41. For each auxiliary view image, calculate the image feature similarity score between it and the main view image:

[0127] ;

[0128] in, represents the image feature similarity score between the auxiliary view image and the main view image, and Represent the height and width of the feature map of the main view image, and Represent the feature tensors of the main view image and the auxiliary view image respectively, represents the L2 norm;

[0129] S42. In the depth hypothesis set at the current spatial resolution level, based on the pixel back-projection and projection transformation relationship, count the number of projected pixels of valid pixels in the primary view image in the auxiliary view image, and calculate the geometric visibility score:

[0130] ;

[0131] in, represents the geometric visibility score of the auxiliary view image, represents the number of valid pixels in the primary view image that are projected into the auxiliary view image under the depth hypothesis set;

[0132] S43. Perform weighted fusion of the image feature similarity score and the geometric visibility score to obtain a viewing angle score:

[0133] ;

[0134] in, represents the viewing angle score, Represents the fusion weight coefficient, which describes the weight of the feature similarity score in the view score;

[0135] S44. Dynamically adjust the fusion weight coefficient according to the sparsity level:

[0136] ;

[0137] in, represents the adjusted fusion weight coefficient, Indicates the adjustment constant, min indicates the minimum value;

[0138] S45 . Sort the viewing angle scores of all auxiliary views, and select at least one auxiliary view image with the highest score.

[0139] The present invention realizes an optimal auxiliary view selection strategy that adapts to changes in image sparsity by comprehensively considering image feature similarity and geometric visibility scores and dynamically adjusting the perspective fusion weights, thereby effectively improving the utilization of perspective information and the overall quality of 3D reconstruction.

[0140] In this embodiment, the S5 specifically includes:

[0141] S51, fusing the main view image feature tensor and the selected auxiliary view image according to the channel dimension to generate a fused feature tensor;

[0142] S52, inputting the fused feature tensor into a three-level cascaded three-dimensional convolutional neural network module built based on the CasMVSNet structure, completing three-dimensional cost volume aggregation at a 1 / 16 spatial resolution level, and generating an aggregated cost volume;

[0143] S53, performing Softmax normalization processing on the aggregated cost volume along the depth dimension to obtain a matching probability distribution of each pixel position under a depth hypothesis set at a 1 / 16 spatial resolution level;

[0144] S54. Perform weighted averaging based on the matching probability distribution and the corresponding depth values ​​to generate an initial depth map:

[0145] ;

[0146] Where D(x,y) represents the initial depth map of the main view image, represents the kth depth value, represents the matching probability distribution under the k-th depth hypothesis, and M represents the number of depth levels.

[0147] The present invention realizes staged cost volume aggregation and depth estimation by inputting the fused feature tensor into a three-level cascade convolutional network based on the CasMVSNet structure, thereby enhancing the network's ability to express sparse images and improving the matching accuracy and detail preservation effect of the initial depth map.

[0148] In this embodiment, the three-dimensional cost volume aggregation specifically includes:

[0149] Guided feature generation stage: The fused feature tensor is downsampled layer by layer to obtain low-resolution guided features, medium-resolution guided features, and high-resolution guided features respectively;

[0150] Three-stage cost aggregation stage: the three-dimensional cost volume and the low-resolution guided features are fused and input into the first-layer three-dimensional convolution module, and the first-layer cost volume is output. The first-layer cost volume and the medium-resolution guided features are fused and input into the second-layer three-dimensional convolution module, and the second-layer cost volume is output. The second-layer cost volume and the high-resolution guided features are fused and input into the third-layer three-dimensional convolution module, and the aggregated cost volume is output.

[0151] The present invention adaptively calculates the depth variance and adjusts the search range based on the initial depth map matching probability distribution, thereby modeling and responding to the depth uncertainty of different image regions, significantly improving the accuracy and stability of depth estimation in subsequent high-resolution layers.

[0152] In this embodiment, the aggregated cost volume has the same spatial and depth dimensions as the original 3D cost volume.

[0153] The present invention uses pixel-level depth interval adaptive sampling to construct the cost volume at the 1 / 8 resolution level, and combines it with a hierarchical convolutional network to complete the depth map generation, realizing a more refined and low-redundancy depth estimation process and improving the reliability of mid-level structure modeling.

[0154] In this embodiment, S6 specifically includes:

[0155] S61. Calculate the depth variance based on the normalized matching probability distribution of the cost volume:

[0156] ;

[0157] in, represents the depth variance at pixel (x,y), represents the matching probability distribution under the k-th depth hypothesis, represents the kth depth value, and D(x,y) represents the initial depth map of the main view image;

[0158] S62. Calculate the depth search range extension amount according to the depth variance and the preset coefficient:

[0159] ;

[0160] in, Indicates the depth search range extension at pixel (x,y), Indicates the preset coefficient, Indicates the depth standard deviation at pixel (x,y);

[0161] S63. Taking the initial depth map of the main view image as the center, dynamically determine the depth hypothesis range at the 1 / 8 resolution level according to the depth search range extension amount:

[0162] ;

[0163] Where R(x,y) represents the depth hypothesis range of pixel (x,y) at 1 / 8 resolution level, and D(x,y) represents the initial depth map of the main view image.

[0164] S64, calculating the depth variance and the adaptive depth search range for each pixel in the entire primary view image, and constructing a global depth hypothesis range at a 1 / 8 resolution level;

[0165] S65. Reconstruct the depth hypothesis set and the three-dimensional cost volume of the main view image to generate a depth map at a 1 / 8 resolution level.

[0166] The present invention completes dense depth map generation at 1 / 4 resolution through a repeated depth variance-driven adaptive search and inference mechanism, ensuring that the final output has higher depth continuity and boundary fidelity, and providing an accurate geometric basis for subsequent three-dimensional point cloud back projection.

[0167] In this embodiment, S8 specifically includes:

[0168] S81. Obtain a dense depth map, combine it with the camera parameters of the main view image, back-project the position of each pixel in the main view image and the corresponding depth value into the camera coordinate system, and calculate the three-dimensional space position coordinates;

[0169] S82, binding the three-dimensional spatial coordinates corresponding to each pixel position with the color information in the primary view image to generate a three-dimensional dense point cloud containing geometric positions and color attributes;

[0170] S83, generating three-dimensional dense point clouds for the main view image and the auxiliary view image respectively, and adding corresponding camera extrinsic parameters, wherein the extrinsic parameters include the rotation matrix and translation vector corresponding to each view;

[0171] S84, registering the three-dimensional dense point clouds corresponding to all views using camera extrinsic parameters and mapping them to the same coordinate system;

[0172] S85, performing fusion optimization processing on the registered three-dimensional dense point cloud to generate a fused point cloud with continuous structure and strong spatial consistency;

[0173] S86. Perform mesh reconstruction based on the fused point cloud to construct a three-dimensional topological structure and form a complete 3D model through surface connection.

[0174] The present invention generates a point cloud by back-projecting the dense depth map, and completes multi-view registration and fusion optimization in combination with the camera posture, ultimately constructing a high-quality three-dimensional model with continuous structure and unified coordinates, significantly improving the reconstruction integrity and spatial consistency under sparse image conditions.

[0175] Example 1:

[0176] To verify the feasibility of the present invention in practice, the present invention was applied to a digital restoration task of a historical and cultural building. Due to geographical restrictions and on-site controls, the shooting conditions for this task were extremely limited. Only 3 to 4 images could be acquired from a fixed position using a mobile phone or a portable drone. There were obvious problems of image sparsity, discontinuous shooting angles, and uneven image quality, which was a typical sparse image 3D reconstruction scenario.

[0177] Traditional structured light scanning and lidar technologies cannot be applied in this scenario, on the one hand due to their high cost and large equipment size, and on the other hand due to objective factors such as environmental occlusion and lighting changes. Conventional MVS methods based on dense multi-view images perform extremely poorly when the number of images is insufficient, and the reconstruction results have obvious holes, misalignments and surface jitter. Therefore, this embodiment applies the sparse photo synthesis 3D reconstruction method based on deep neural networks proposed in the present invention to this task.

[0178] During the on-site acquisition process, only a portable drone was used to capture three view images in a triangulated manner in front of the main hall of the ancient building. The image resolution was 5472×3648. After shooting, the images were aligned, distortion corrected, and size-normalized using the image preprocessing module described in the present invention. The image intrinsic parameters and corresponding extrinsic parameters were also extracted. After multi-scale image feature extraction and cascaded depth estimation, the present invention generated an initial depth map at a 1 / 16 resolution level. Then, through two-level depth variance calculation and adaptive adjustment of the search range, a dense depth map was output at a 1 / 4 level. Finally, a complete three-dimensional point cloud was constructed and a mesh model was generated using the back-projection, registration, and fusion processes.

[0179] In comparison with common 3D reconstruction methods, our method demonstrates significant advantages. To ensure the credibility of the experimental results, we compared our method with COLMAP (a traditional SfM+MVS method), MVSNet (a classic deep MVS method), and CasMVSNet (a primitive cascade method). We tested reconstruction time, point cloud density, reconstruction accuracy (compared to LiDAR data), surface integrity, and occlusion recovery under the same number of input images and computational resource constraints.

[0180] Actual operation data shows that under the same three sparse image input conditions, the model generated by the COLMAP method has serious geometric distortions, and the MVSNet model has a large number of regional holes. Although CasMVSNet has made improvements, it is still not ideal in visibility selection and depth boundary transition. The present invention uses perspective selection, depth variance-driven hierarchical search and guided feature aggregation to effectively preserve structural details and restore the continuity of occluded areas.

[0181] Table 1 Comparison of 3D reconstruction performance between the present invention and existing methods under sparse image conditions

[0182] ;

[0183] From the experimental comparison results shown in Table 1, it can be seen that under the same sparse image input conditions (only 3 high-resolution images), the present invention shows technical effects that are significantly superior to existing mainstream methods in multiple key performance indicators.

[0184] In terms of reconstruction accuracy, the average reconstruction error of the present invention is 0.85cm, which is much lower than COLMAP's 3.42cm, MVSNet's 2.05cm, and CasMVSNet's 1.24cm. This shows that the present invention can still stably output high-precision geometric structures under low image redundancy conditions and has excellent depth regression capabilities.

[0185] In terms of the restoration rate of occluded areas, the present invention reaches 88.3%, which is more than 14 percentage points higher than CasMVSNet. This significantly alleviates the problem of traditional methods easily failing to reconstruct in areas with insufficient visibility, fully demonstrating the innovation of the present invention in terms of perspective selection and feature guidance mechanism.

[0186] From the point cloud density index, the point cloud density output by the present invention reaches 1860 points / square meter, which is much higher than that of traditional algorithms such as MVSNet and COLMAP, and is also improved by about 30% compared with CasMVSNet. This shows that the present invention can not only identify the main structure, but also effectively capture the detailed features, providing a rich geometric foundation for subsequent model surface reconstruction.

[0187] In terms of efficiency, the entire reconstruction process of the present invention takes only 14 minutes, which is better than the 22 minutes and 16 minutes of MVSNet and CasMVSNet, respectively. This shows that the present invention has been optimized in terms of algorithm structure and computing resource scheduling, and has higher operating efficiency.

[0188] In terms of resource usage, the present invention controls the GPU memory usage at 3.5GB, which is significantly reduced compared with MVSNet and CasMVSNet, and improves the hardware adaptability while ensuring accuracy.

[0189] In addition, in terms of surface integrity score, the present invention reaches 0.89, which is better than 0.73 of CasMVSNet and 0.58 of MVSNet. This shows that the present invention performs superiorly in texture area, edge details and continuity, and can generate high-quality 3D models with complete structure and clear boundaries.

[0190] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A 3D model reconstruction method based on sparse photo synthesis based on deep neural network, characterized in that: The steps include: S1, obtain a sparse image set and preprocess it, and extract the camera parameters of each image; S2. Perform feature extraction on the preprocessed sparse image set to obtain image feature tensors at 1 / 16, 1 / 8, and 1 / 4 spatial resolution levels; S3, at the 1 / 16 spatial resolution level, based on the initial depth assumption range of the main view image and combined with the camera parameters, construct the 3D cost volume corresponding to the main view; S4. Calculate the image feature similarity score and geometric visibility score of the auxiliary view relative to the main view, synthesize the view angle score according to the weighted fusion strategy, and select the optimal auxiliary view image; S5. The image feature tensors of the main view image and the selected auxiliary view image are fused and input into a 3D convolutional neural network based on the CasMVSNet structure to perform cost aggregation on the 3D cost volume to generate an initial depth map of the main view image at a 1 / 16 spatial resolution level. S6. Calculate the depth variance based on the initial depth map, adaptively adjust the depth hypothesis range, construct a new 3D cost volume, and generate a depth map at a 1 / 8 resolution level. S7, repeat step S6 to generate a dense depth map at a 1 / 4 resolution level; S8. Generate a 3D dense point cloud based on the dense depth map, perform registration and fusion, and complete the reconstruction of the 3D model; The S4 specifically includes: S41. For each auxiliary view image, calculate an image feature similarity score between the auxiliary view image and the main view image; S42. In the depth hypothesis set at the current spatial resolution level, based on the pixel back-projection and projection transformation relationship, count the number of projected pixels of valid pixels in the primary view image in the auxiliary view image, and calculate the geometric visibility score; S43, performing weighted fusion of the image feature similarity score and the geometric visibility score to obtain a viewing angle score; S44, dynamically adjust the fusion weight coefficient according to the sparsity level; S45 . Sort the viewing angle scores of all auxiliary views, and select at least one auxiliary view image with the highest score.

2. The method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network according to claim 1, characterized in that: The sparse image set includes one primary view image and at least one auxiliary view image.

3. The method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network according to claim 1, characterized in that: The preprocessing includes image alignment, size normalization, image enhancement and distortion correction.

4. The method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network according to claim 1, characterized in that: The S3 specifically includes: S31. At a spatial resolution level of 1 / 16, construct an initial depth hypothesis range for the main view image and perform uniform discretization to obtain a depth hypothesis set; S32. Back-project each pixel coordinate and each depth value in the feature tensor of the primary view image to a three-dimensional point in the primary view camera coordinate system using the primary view camera parameters; S33, mapping the three-dimensional point to each auxiliary view image plane, and calculating the two-dimensional projection coordinates on the auxiliary view using the camera parameters of the auxiliary view image; S34. Sampling the feature tensor of the auxiliary view image at the two-dimensional projection coordinate position using a bilinear interpolation method to obtain a reprojected feature tensor; S35, calculating a feature difference cost term between the feature tensor of the main view image and the reprojected feature tensor of each auxiliary view image; S36: Aggregate and stack all auxiliary view images and cost values ​​to construct a three-dimensional cost volume.

5. The method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network according to claim 1, characterized in that: The S5 specifically includes: S51, fusing the main view image feature tensor and the selected auxiliary view image according to the channel dimension to generate a fused feature tensor; S52, inputting the fused feature tensor into a three-level cascaded three-dimensional convolutional neural network module built based on the CasMVSNet structure, completing three-dimensional cost volume aggregation at a 1 / 16 spatial resolution level, and generating an aggregated cost volume; S53, performing Softmax normalization processing on the aggregated cost volume along the depth dimension to obtain a matching probability distribution of each pixel position under a depth hypothesis set at a 1 / 16 spatial resolution level; S54. Perform weighted averaging based on the matching probability distribution and the corresponding depth values ​​to generate an initial depth map: ； in, represents the initial depth map of the main view image, Indicates the Depth values, Indicates the The matching probability distribution under the depth hypothesis, Indicates the number of depth levels.

6. The method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network according to claim 5, characterized in that: The three-dimensional cost volume aggregation specifically includes: Guided feature generation stage: The fused feature tensor is downsampled layer by layer to obtain low-resolution guided features, medium-resolution guided features, and high-resolution guided features respectively; Three-stage cost aggregation stage: the three-dimensional cost volume and the low-resolution guided features are fused and input into the first-layer three-dimensional convolution module, and the first-layer cost volume is output. The first-layer cost volume and the medium-resolution guided features are fused and input into the second-layer three-dimensional convolution module, and the second-layer cost volume is output. The second-layer cost volume and the high-resolution guided features are fused and input into the third-layer three-dimensional convolution module, and the aggregated cost volume is output.

7. The method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network according to claim 6, characterized in that: The aggregated cost volume has the same spatial and depth dimensions as the original 3D cost volume.

8. The method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network according to claim 1, characterized in that: The S6 specifically includes: S61. Calculate the depth variance based on the normalized matching probability distribution of the cost volume: ； in, Represents pixels The depth variance at Indicates the The matching probability distribution under the depth hypothesis, Indicates the Depth values, represents the initial depth map of the main view image, Indicates the number of depth levels; S62. Calculate the depth search range extension amount according to the depth variance and the preset coefficient: ； in, Represents pixels The depth search range extension at Indicates the preset coefficient, Represents pixels The standard deviation of the depth at S63. Taking the initial depth map of the main view image as the center, dynamically determine the depth hypothesis range at the 1 / 8 resolution level according to the depth search range extension amount: ； in, Represents pixels The depth hypothesis range at 1 / 8 resolution level, An initial depth map representing the main view image; S64, calculating the depth variance and the adaptive depth search range for each pixel in the entire primary view image, and constructing a global depth hypothesis range at a 1 / 8 resolution level; S65. Reconstruct the depth hypothesis set and the three-dimensional cost volume of the main view image to generate a depth map at a 1 / 8 resolution level.

9. The method for reconstructing a 3D model by synthesizing sparse photos based on a deep neural network according to claim 1, characterized in that: The S8 specifically includes: S81. Obtain a dense depth map, combine it with the camera parameters of the main view image, back-project the position of each pixel in the main view image and the corresponding depth value into the camera coordinate system, and calculate the three-dimensional space position coordinates; S82, binding the three-dimensional spatial coordinates corresponding to each pixel position with the color information in the primary view image to generate a three-dimensional dense point cloud containing geometric positions and color attributes; S83, generating three-dimensional dense point clouds for the main view image and the auxiliary view image respectively, and adding corresponding camera extrinsic parameters, wherein the extrinsic parameters include the rotation matrix and translation vector corresponding to each view; S84, registering the three-dimensional dense point clouds corresponding to all views using camera extrinsic parameters and mapping them to the same coordinate system; S85, performing fusion optimization processing on the registered three-dimensional dense point cloud to generate a fused point cloud with continuous structure and strong spatial consistency; S86. Perform mesh reconstruction based on the fused point cloud to construct a three-dimensional topological structure and form a complete 3D model through surface connection.

Images