A multi-view three-dimensional reconstruction method based on a four-fold depth and wave type depth geometric enhancement MVSNet network

Abstract

The application discloses a kind of based on fourfold depth and wave type depth geometric enhancement MVSNet network's multi-view three-dimensional reconstruction method, belong to computer vision in multi-view three-dimensional reconstruction technical field, its network model basic flow is fourfold depth rough estimation, fourfold depth refinement, depth optimization based on efficient gauss-newton module.The application predicts fourfold depth and constructs wave type depth map, reduces the depth interpolation deviation in depth fusion stage, to significantly improve the quality of reconstructed point cloud, although having time-consuming double 3D CNN structure, but still can quickly predict depth map as a whole.The method disclosed by the application adopts progressive refinement type architecture, effectively avoids the error propagation problem inherent in cascade architecture, is superior to other depth learning MVS methods with similar architecture in terms of computational efficiency and overall reconstruction quality, thus providing an effective technical solution for high-precision, high-integrity, high-efficiency three-dimensional reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of multi-view computer vision. Figure 3 In the field of 3D reconstruction technology, specifically, it relates to a multi-view MVSNet network based on quadruple depth and wave-like depth geometry enhancement. Figure 3 Dimensional reconstruction method. Background Technology

[0002] Multi-View Stereo (MVS), a key fundamental task in computer vision, has undergone extensive and in-depth research over the past few decades. The MVS method aims to reconstruct the dense 3D geometry of a target scene from 2D RGB images with known camera parameters. Its underlying logic is to establish pixel-level correspondences between reference and source views. Human perception of the 3D world is more intuitive, enabling the MVS method to play an indispensable role in numerous fields such as national defense, smart cities, intelligent transportation, cultural relic protection, disaster relief, agriculture and forestry, and virtual / augmented reality.

[0003] Based on the different representations of the reconstructed 3D scene, MVS methods can be divided into several categories: voxel-based, point diffusion-based, patch-based, and depth map-based. Among these, the last category, depth map-based methods, decouple the MVS task into depth prediction and depth fusion, making them suitable for large-scale scene reconstruction and thus highly favored for their flexibility and efficiency. Recently, some excellent tools and algorithms have emerged in this field, with Gipuma, COLMAP, and ACMM being typical examples.

[0004] However, traditional depth map-based MVS methods have significant shortcomings. When reconstructing target scenes with varying illumination, weak or no texture, reflective surfaces, or repetitive patterns (collectively referred to as challenging scenes), traditional MVS methods employ hand-designed similarity metrics, which easily lead to point cloud holes and blurring. Furthermore, the engineered regularization methods used in traditional MVS cannot eliminate the negative impact of invisible pixels (such as pixels in occluded areas and out-of-view regions), thus reducing the point cloud quality in these invisible areas.

[0005] With the development of deep learning, more and more scholars are turning their attention to using Convolutional Neural Networks (CNNs) to solve the Multiview View (MVS) problem. In 2018, Yao et al. first proposed the end-to-end trainable MVS network MVSNet to predict depth, and then used general techniques for depth fusion. MVSNet has become the de facto foundational pipeline followed by the vast majority of current MVS networks, including four key steps: feature extraction using 2D CNNs, cost aggregation including a differentiable homography transformation step, cost volume regularization using 3D CNNs, and depth regression including a soft-argmin step. The first and third steps utilize the powerful semantic information enrichment and noise filtering capabilities of CNNs, respectively, enabling the learning-based MVS method to significantly improve the accuracy and completeness of 3D reconstruction results compared to traditional MVS methods. Furthermore, leveraging the efficient computing power of the Graphics Processing Unit (GPU) in computer graphics cards, the reconstruction efficiency of the learning-based MVS method is greatly improved compared to traditional methods. Due to these advantages, the learning-based MVS method has rapidly become the preferred method for multiview mapping. Figure 3 The field of 3D reconstruction is a research hotspot, and related research is dedicated to solving various problems existing in the MVSNet pipeline.

[0006] For example, research on addressing the excessive memory and computational burden of 3D CNNs has mainly proceeded in two directions. On the one hand, recursive methods (such as R-MVSNet, D2HC-RMVSNet, AA-RMVSNet, etc.) propose using recursive convolutional neural networks to replace 3D CNNs, reducing 3D processing to 2D. Although this comes at the cost of increased runtime, it successfully overcomes the memory bottleneck in high-resolution reconstruction. On the other hand, multi-stage cascaded methods (such as CasMVSNet, CVP-MVSNet, UCS-MVSNet, etc.) are based on residual estimation and infer depth maps in a coarse-to-fine manner. Although various cascaded frameworks differ in their principles for determining the overall range and sampling interval of depth assumptions in the refinement stage, they all significantly reduce the memory and time costs required by learning-based MVS methods. In addition, there are many targeted and representative research works on enhancing the diversity and expressiveness of 2D features, promoting cost aggregation for visibility perception, and designing different "probability-depth" derivation algorithms.

[0007] Nevertheless, learning-based MVS methods still face some challenges in achieving high-precision, high-completeness, and high-efficiency 3D reconstruction: (1) There are still significant limitations when reconstructing non-ideal scenes and invisible areas; (2) The complex network structures introduced to improve prediction accuracy lead to longer running time and increased memory burden; (3) Most current methods adopt multi-stage cascaded frameworks to improve efficiency, but the error propagation problem inherent in such frameworks may have a potential negative impact on reconstruction quality. Summary of the Invention

[0008] In view of this, the present invention proposes a multi-view MVSNet network based on quadruple depth and wave-shaped depth geometry enhancement. Figure 3 The proposed 3D reconstruction method employs a basic network process consisting of four-fold coarse depth estimation, four-fold depth refinement, and depth optimization based on efficient Gaussian-Newton modules. This method aims to address the shortcomings of existing learning-based MVS methods by improving the reconstruction quality of deep learning MVS methods based on a progressively refined architecture from a deep fusion perspective. Simultaneously, careful design ensures efficient network runtime, thus providing a practical technical solution for achieving high-precision, high-completeness, and high-efficiency 3D reconstruction.

[0009] A multi-view MVSNet network based on quadruple depth and wave-shaped depth geometry enhancement Figure 3 The 3D reconstruction method includes the following steps:

[0010] Step S1: Acquire a set of multi-view images of a scene. Each time, select one image as a reference image and select (N-1) source images with the smallest angle between the images. Use these N images as input. Construct an efficient multi-scale information-rich feature extractor. Using the full-resolution reference image and its corresponding source image as input, obtain two different sets of quarter-resolution scale feature maps. and and a set of half-resolution scale feature maps They are respectively used for four-fold coarse depth estimation, four-fold depth refinement, and depth optimization based on efficient Gaussian-Newton modules;

[0011] Step S2: Through differentiable homography transformation, a reference feature map is generated. and N-1 source feature maps N feature maps The reference camera's view frustum is mapped onto a forward parallel plane partitioned based on several depth assumptions, thereby generating N feature volumes. The inter-group correlation similarity between the reference feature volume and N-1 source feature volumes is calculated to construct N-1 similarity volumes. The similarity volumes are fed into the visibility-aware view aggregation module based on epipolar transform to generate a unified matching cost volume.

[0012] Step S3: Use a dual 3D CNN regularized cost body to obtain a dual two-channel probability body; calculate a quadruple depth map and a quadruple confidence map from the dual two-channel probability body; use a multiple loss function to constrain the distribution of the quadruple depth map near the base truth value; use a wave-like selection strategy to calculate a coarse wave-like depth map with wave-like depth geometry from the quadruple depth map, and calculate a wave-like confidence map corresponding to the coarse wave-like depth map from the quadruple confidence map, thus completing the coarse quadruple depth estimation;

[0013] Step S4: Construct a quadruple depth refinement module. Determine a unified initial depth range for the refinement module based on a coarse wavy depth map, using N feature maps. Repeat steps S2 and S3 to obtain the refined wave-shaped depth map and confidence map;

[0014] Step S5: Use the efficient multi-scale information-rich feature extractor described in step S1 to extract the features beforehand. By directly calculating and removing the extra feature extraction network used in the traditional Gaussian-Newton module, an efficient Gaussian-Newton module is constructed. This module is used to upsample and optimize the wavy depth map after quarter-resolution refinement to obtain the final half-resolution depth map.

[0015] Step S6: Replace the reference image and repeat the above steps until the depth maps corresponding to all images of the target scene are obtained.

[0016] Furthermore, the efficient multi-scale information-rich feature extractor described in step S1 can extract information-rich features, and the specific process is as follows:

[0017] First, multiple sets of convolutional layers are used to progressively collect semantic information from the input image. Each set of convolutional layers contains a convolution operation with a stride of 2 to reduce the resolution level. Then, bilinear interpolation is used to upsample the low-resolution intermediate features containing semantic information, and these upsampled features are added to the original high-resolution features containing spatial information. Finally, several convolutional layers are applied again to filter features while changing the feature scale. Through this iterative "top-down-bottom-up" processing, high-level semantic information and low-level spatial information are gathered into the information-rich features output by the feature extraction network.

[0018] Furthermore, the efficient multi-scale information-rich feature extractor described in step S1 is highly efficient, and its specific implementation is as follows:

[0019] First, the multi-scale filtering convolutional blocks extract features at different resolution levels for subsequent stages. Therefore, the additional feature networks originally required for later stages can be eliminated, improving computational efficiency and reducing memory burden at the overall network inference level. The efficient multi-scale information-rich feature extractor itself also employs an efficient image processing method. Most previous MVS methods used feature pyramid networks to process different view images sequentially. The efficient multi-scale information-rich feature extractor, through batch processing, merges different views of the same batch of data into a single batch size, thereby extracting information-rich features from all input images at once. This efficient image processing method reduces network runtime at the cost of some memory consumption.

[0020] Furthermore, the visibility-aware view aggregation module based on the epipolar transformer described in step S2 calculates the visibility-aware weights by combining the cross-attention concept of the transformer with a filtering network. The specific process is as follows:

[0021] Reference features Treat the source features as query vectors The source feature volume W obtained by differentiable homography transformation i If treated as a key vector, then the similarity volume S, which is a value vector, is assigned to it. i The visibility-aware weights can be calculated using a cross-attention mechanism. A 2D CNN is then used to further filter intermediate results to enhance robustness. The visibility-aware weights w i The calculation is as follows:

[0022]

[0023] In the formula, i = 1, ..., N-1 represents the corresponding source view number, Φ represents the 2D CNN filtering network, and C represents the number of feature channels. These view weights are also directly used for quadruple depth refinement.

[0024] Furthermore, the process of using a dual 3D CNN regularization cost body to obtain a dual two-channel probability body in step S3 is as follows:

[0025] The cost volume is fed into a cost volume filtering network composed of dual 3D CNNs. At the end of each 3D CNN, a convolutional unit with two output channels is added. The output dual two-channel volume is then subjected to a softmax operation, which is the dual two-channel probability volume.

[0026] Furthermore, the specific process for calculating the quadruple depth map and quadruple confidence map from the dual two-channel probability volume in step S3 is as follows:

[0027] First, a deep regression approach is used to analyze the dual two-channel probability volume. The dual two-channel depth map was calculated. That is, each two-channel depth map is calculated separately for all depth assumptions d. j The corresponding two-channel probability sum:

[0028]

[0029] In the formula, d j (j = 0, 1, ..., D-1, where D is the total number of depth hypothesis planes) represents the depth hypothesis; then, by extracting the maximum and minimum values ​​along the channel dimension, a quadruple depth map can be obtained from the dual two-channel depth map: and

[0030] The confidence score is used to measure the quality of depth estimation and can be calculated as the sum of the probabilities of the four closest depth hypotheses. Therefore, in a similar manner as described above, the dual two-channel confidence score map can be calculated from the dual two-channel probability volume, and then the quadruple confidence score map can be obtained from the dual two-channel confidence score map.

[0031] Furthermore, the process of using multiple loss functions to constrain the distribution of the quadratic depth map near the base ground truth in step S3 is as follows:

[0032] First, the L1 loss function is used to constrain the four-fold depth maps individually, making each one smooth and close to the ground truth depth. Then, a loss function is constructed to constrain each pair of depths— and —Symmetrically distributed above and below the fundamental truth depth plane, i.e., the fundamental truth depth D g lie in and Between; finally, for the final wave-shaped depth map D 1 The subpixel accuracy is also constrained using the L1 loss function.

[0033] Furthermore, in step S3, a wave-like selection strategy is adopted to calculate a coarse depth map with wave-like depth geometry from the quadruple depth map, and a wave-like confidence map corresponding to the wave-like depth map is calculated from the quadruple confidence map. The specific process is as follows:

[0034] First, a coordinate mask is calculated to select depth values ​​from different depth maps at different coordinate positions in the coarse wavy depth map. Then, the mean maps of the four intermediate values ​​of the quadruple depth map and the overall mean map are calculated. These five mean maps are very close to the base ground truth depth map. Finally, depth values ​​are selected from the quadruple depth map and the five mean maps in a specific order, so that the depth within each wavy depth cell fluctuates slightly up and down along the base ground truth depth surface, rather than all being on one side of the base ground truth depth surface, thus forming a coarse wavy depth map D^1. Compared with single-sided cells, wavy depth cells have smaller depth interpolation deviations in the depth fusion stage when the depth prediction deviation is the same, thus generating higher quality point clouds.

[0035] Furthermore, the specific process for determining the unified initial depth range of the refinement module based on the coarse wavy depth map in step S4 is as follows:

[0036] The maximum and minimum mean values ​​of the quadruple depth map are calculated separately. The difference between the maximum and minimum mean values ​​is then used as a standard to extend a range along the depth direction to both sides, forming the initial depth range. Since the quadruple depth map can be seen as a repeated measurement of the coarse depth prediction, the larger the difference between the maximum and minimum mean values, the less reliable the coarse depth prediction is. Therefore, the corresponding initial depth range during refinement should be wider to ensure that the underlying true depth is still included within the initial depth range. Overall, the initial depth range of the refinement module is smaller than the predefined depth range used in the coarse estimation.

[0037] By filtering and fusing the final depth map of all reference images, a dense point cloud of the scene corresponding to this set of multi-view images can be obtained, thus completing the multi-view process. Figure 3 Reconstruction.

[0038] The beneficial effects of this invention are: based on the various problems faced by existing technologies, this invention proposes a multi-view MVSNet network based on quadruple depth and wave-shaped depth geometry enhancement. Figure 3This invention proposes a 3D depth reconstruction method. The network model of this method features a progressively refined architecture, effectively avoiding the error propagation problem inherent in cascaded architectures. The basic process involves four-fold coarse depth estimation, four-fold depth refinement, and depth optimization based on an efficient Gaussian-Newton module. First, this invention constructs an efficient multi-scale information-rich feature extractor, whose information-rich features extracted in a shorter runtime help the network predict more accurate depth maps for challenging scenes. Second, this invention constructs a visibility-aware view aggregation module based on an epipolar transform, which utilizes the idea of ​​cross-attention and robust visibility-aware weights calculated by a filtering network to improve the reconstruction quality of invisible regions. Then, this invention proposes predicting a four-fold depth map and calculating a coarse depth map with a wavy geometry from it. This geometry reduces interpolation bias in the depth fusion stage. Furthermore, this invention constructs a four-fold depth refinement module, which determines a unified initial depth range based on the coarse wavy depth map, achieving refinement of the coarse wavy depth map. The above complete four-fold depth and wavy depth geometry-related design significantly improves the accuracy and completeness of the reconstructed point cloud. Finally, this invention constructs an efficient Gaussian-Newton module to optimize the depth map and improve the output resolution, thus significantly enhancing the reconstruction quality. Training and testing on the DTU dataset experimentally validates the multi-view capabilities of the MVSNet network proposed in this invention, which utilizes quadruple depth and wavy depth geometry enhancement. Figure 3 The 3D reconstruction method can significantly improve the quality of reconstructed point clouds. Although it has a time-consuming dual 3D CNN structure, it can still complete depth map prediction at a very fast speed overall. Attached Figure Description

[0039] Figure 1 For multi-view MVSNet networks based on quadruple depth and wave-shaped depth geometry enhancement Figure 3 Flowchart of 3D reconstruction method;

[0040] Figure 2 For multi-view MVSNet networks based on quadruple depth and wave-shaped depth geometry enhancement Figure 3 A visualization of the point cloud reconstruction results of the entire scene in the DTU dataset evaluation set using the 3D reconstruction method;

[0041] Figure 3 A comparison chart of point cloud visualization results for reconstructing challenging scenes from the DTU dataset using different methods;

[0042] Figure 4 A comparison chart of GPU memory consumption, runtime, and overall reconstruction quality for different methods;

[0043] Figure 5 Ablation experiment diagrams for four-fold depth coarse estimation, four-fold depth refinement, and depth optimization based on efficient Gaussian-Newton modules. Detailed Implementation

[0044] To more clearly illustrate the intended objectives, technical means, and effects of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings and specific examples.

[0045] It should be noted that, unless otherwise specified, all technical and scientific terms used in this invention have the same common meaning in the technical field to which this invention pertains. Experimental conditions not specifically mentioned in the following embodiments employ conventional techniques in the field. It should be understood that the described embodiments are only some examples of the present invention, and not all examples. All other embodiments obtained by those skilled in the art based on these embodiments without inventive effort are within the scope of protection of this invention.

[0046] Example 1

[0047] To address the problems existing in current technologies, this invention improves the reconstruction quality of deep learning MVS methods based on progressively refined architectures from a deep fusion perspective. Through careful design, it ensures efficient network runtime and provides a practical technical solution to achieve high-precision, high-completeness, and high-efficiency 3D reconstruction. This invention proposes a multi-view MVSNet network based on quadruple depth and wave-shaped depth geometry enhancement. Figure 3 A 3D reconstruction method. The network model of this method has a progressively refined architecture. Its basic process consists of four-fold depth coarse estimation, four-fold depth refinement, and depth optimization based on efficient Gaussian-Newton modules, such as... Figure 1 As shown. This multi-view MVSNet network is based on quadruple depth and wave-shaped depth geometry enhancement. Figure 3 The 3D reconstruction method, with the following specific steps:

[0048] Step S1: Acquire a set of multi-view images of a scene. Each time, select one image as a reference image and select (N-1) source images with the smallest angle between the images. Use these N images as input. Construct an efficient multi-scale information-rich feature extractor. Using the full-resolution reference image and its corresponding source image as input, obtain two different sets of quarter-resolution scale feature maps. and and a set of half-resolution scale feature maps They are respectively used for four-fold coarse depth estimation, four-fold depth refinement, and depth optimization based on efficient Gauss-Newton modules.

[0049] Step S1.1: Obtain a set of multi-view images of a certain scene. Each time, select one image as a reference image and select (N-1) source images with the smallest angle between the images. Use these N images as input.

[0050] For example, when using the DTU public dataset, a set of multi-view images of a certain scene is selected from its training or evaluation set for network training or evaluation. Based on the predefined image angle scores in its matching record file, the (N-1) source images with the best matching degree can be selected for each image sequentially. In this embodiment, the value is N = 3.

[0051] Step S1.2: Construct an efficient multi-scale information-rich feature extractor, using a full-resolution reference image and its corresponding source image as input, to obtain two different sets of quarter-resolution scale feature maps. and and a set of half-resolution scale feature maps They are respectively used for four-fold coarse depth estimation, four-fold depth refinement, and depth optimization based on efficient Gauss-Newton modules.

[0052] First, multiple sets of convolutional layers are used to progressively collect semantic information from the input image. Each set of convolutional layers contains a convolution operation with a stride of 2 to reduce the resolution level. Then, bilinear interpolation is used to upsample the low-resolution intermediate features containing semantic information, and these upsampled features are added to the original high-resolution features containing spatial information. Finally, several convolutional layers are applied again to filter features while changing the feature scale. Through this iterative "top-down-bottom-up" processing, high-level semantic information and low-level spatial information are gathered into the information-rich features output by the feature extraction network.

[0053] Multi-scale filtering convolutional blocks extract features at different resolution levels for subsequent stages, thus eliminating the need for additional feature networks in later stages. This improves computational efficiency and reduces memory burden at the overall network inference level. The efficient multi-scale information-rich feature extractor itself also employs an efficient image processing method. Most previous MVS methods used feature pyramid networks to process different view images sequentially. The efficient multi-scale information-rich feature extractor, through batch processing, merges different views of the same batch of data into a single batch size, thereby extracting information-rich features from all input images at once. This efficient image processing method reduces network runtime at the cost of increased memory consumption.

[0054] This efficient, multi-scale, information-rich feature extractor takes a full-resolution input image... In the process, features at different scales are extracted. and These are used for four-fold coarse depth estimation, four-fold depth refinement, and depth optimization based on an efficient Gaussian-Newton module, respectively. In this embodiment, the values ​​are: W×H = 640×512 for the DTU training set; W×H = 1600×1152 for the DTU evaluation set; and C = 32 feature channels.

[0055] Step S2: Through differentiable homography transformation, a reference feature map is generated. and N-1 source feature maps N feature maps The reference camera's view frustum is mapped onto a forward parallel plane partitioned based on several depth assumptions, thereby generating N feature volumes. The inter-group correlation similarity between the reference feature volume and N-1 source feature volumes is calculated to construct N-1 similarity volumes. The similarity volumes are then fed into the visibility-aware view aggregation module based on epipolar transformers to generate a unified matching cost volume.

[0056] Step S2.1: Through differentiable homography transformation, a reference feature map is generated. and N-1 source feature maps N feature maps The data is mapped onto the forward parallel plane in the reference camera's view frustum, which is divided based on several depth assumptions, thereby generating N feature volumes.

[0057] The reference feature volume W0 can be easily copied along the depth dimension. The source feature body W is obtained. i Source features are required Perform a differentiable homography transformation, or differentiable torsion for short. Specifically, given the corresponding camera parameters... Pixel p in the reference view r Calculate the corresponding source view pixel p s :

[0058]

[0059] In the formula, d j (j = 0, 1, ..., D-1, where D is the total number of depth hypothesis planes) represents the depth hypothesis, i.e., the forward parallel planes divided based on several depth hypotheses. This is achieved by using a predefined depth range (d... min ,d max Within ) , d is obtained using a uniform sampling method across the inverse depth range. j :

[0060] 1 / d j =1 / d min -j / (D-1)(1 / d min -1 / d max )

[0061] In this embodiment, the relevant values ​​are: for the DTU dataset, (d min ,d max ) = (425mm, 935mm); for training, D = 48; for evaluation, D = 96.

[0062] Use the calculated sampling grid coordinates to analyze the source feature map. Differentiable bilinear sampling is performed to twist the source feature map onto several depth hypothesis planes, thus obtaining the source feature volume W. i .

[0063] Step S2.2: Calculate the inter-group correlation similarity between the reference feature body and N-1 source feature bodies, and construct N-1 similarity volumes.

[0064] In the field of deep learning-based MVS, feature maps are first transformed into feature volumes or similarity volumes, and then the cost volume is calculated.

[0065] By uniformly dividing feature channel C into G groups, the inter-group correlation similarity S between feature volumes can be calculated. i for:

[0066] S i (d j )=G / C· <W0,W i (d j )>

[0067] In the formula, <·,·> represent the inner product operation. In this embodiment, the value is G = 4.

[0068] Step S2.3: Feed the similarity volume into the visibility-aware view aggregation module based on epipolar transform to generate a unified matching cost volume.

[0069] In the transformer, the original cross-attention mechanism between the query vector Q, the key vector K, and the value vector V is as follows:

[0070]

[0071] Where C represents the number of channels in Q and K. Similarly, the reference feature will be used... Treat it as a query vector and match it along the epipolar line to the key vector W. i The cross-attention mechanism can be used to calculate the similarity volume S used to enhance the value vector. i Attention weights. A 2D CNN is used here to further filter intermediate results to enhance robustness. Finally, the visibility-aware weights w i The calculation is as follows:

[0072]

[0073] In the formula, i = 1, ..., N-1 represents the corresponding source view number, Φ represents the 2D CNN filtering network, and C represents the number of feature channels. These view weights are also directly used for quadruple depth refinement.

[0074] The unified matching cost C is calculated through a weighted sum:

[0075]

[0076] In the formula S i For similarity volume, roles are value vectors.

[0077] Step S3: Use a dual 3D CNN regularization cost body to obtain a dual two-channel probability body; calculate a quadruple depth map and a quadruple confidence map from the dual two-channel probability body; use a multiple loss function to constrain the distribution of the quadruple depth map near the base truth value; use a wave-like selection strategy to calculate a coarse wave-like depth map with wave-like depth geometry from the quadruple depth map, and calculate a wave-like confidence map corresponding to the coarse wave-like depth map from the quadruple confidence map, thus completing the coarse quadruple depth estimation.

[0078] Step S3.1: Use a dual 3D CNN regularization cost body to obtain a dual two-channel probability body.

[0079] The cost volume is fed into a cost volume filtering network composed of two U-Net-based multi-scale 3D CNNs, which utilize the powerful noise filtering capabilities of CNNs to regularize the cost volume. At the end of each 3D CNN, a convolutional unit with two output channels is added. The output two-channel volume is then subjected to a softmax operation, which is the two-channel probability volume P.

[0080] Step S3.2: Calculate the quadruple depth map and quadruple confidence map from the dual two-channel probability volume.

[0081] In the field of deep learning-based MVS, the depth map is calculated from the probabilistic volume using depth regression:

[0082]

[0083] Based on this, the present invention realizes the calculation of dual-channel input by adding three symbols, Q, a, and b, to represent dual channels.

[0084] First, a deep regression approach is used to analyze the dual two-channel probability volume. The dual two-channel depth map was calculated. That is, each two-channel depth map is calculated separately for all depth assumptions d. j The corresponding two-channel probability sum:

[0085]

[0086] In the formula, d j(j = 0, 1, ..., D-1, where D is the total number of depth hypothesis planes) represents the depth hypothesis; then, by extracting the maximum and minimum values ​​along the channel dimension, a quadruple depth map can be obtained from the dual two-channel depth map: and

[0087] The confidence score is used to measure the quality of depth estimation and can be calculated as the sum of the probabilities of the four closest depth hypotheses. Therefore, in a similar manner as described above, the dual two-channel confidence score map can be calculated from the dual two-channel probability volume, and then the quadruple confidence score map can be obtained from the dual two-channel confidence score map.

[0088] Step S3.3: Use multiple loss functions to constrain the distribution of the quadruple depth map near the base ground truth.

[0089] First, the L1 loss function is used to constrain the four-fold depth maps individually, making each one smooth and close to the ground truth depth. Then, a loss function is constructed to constrain each pair of depths— and —Symmetrically distributed above and below the fundamental truth depth plane, i.e., the fundamental truth depth D g lie in and between:

[0090]

[0091] When the depth of the fundamental truth value is D g Not located and In between, It must be greater than Therefore, the L1 loss function makes Increase until the underlying truth depth D g lie in and In between, at this time Less than Subsequently, the L1 loss function prompts Shrink but maintain the underlying truth depth D g lie in and Even if and From both sides, approaching the fundamental truth depth D g .

[0092] Finally, the final wave-shaped depth map D 1 The sub-pixel accuracy is also constrained using the l1 loss function.

[0093] Step S3.4: Using a wave-like selection strategy, a coarse wave-like depth map with wave-like depth geometry is calculated from the quadruple depth map, and a wave-like confidence map corresponding to the coarse wave-like depth map is calculated from the quadruple confidence map, thus completing the quadruple depth coarse estimation.

[0094] First, a coordinate mask is calculated to select depth values ​​from different depth maps at different coordinate positions in the coarse wavy depth map. Then, the mean maps of the four intermediate values ​​of the quadruple depth map and the overall mean map are calculated; these five mean maps closely approximate the base true depth map. Finally, depth values ​​are selected sequentially from the quadruple depth map and the five mean maps in a specific order, ensuring that the depth within each wavy depth unit fluctuates slightly up and down along the base true depth surface, rather than remaining entirely on one side of the base true depth surface, thus forming the coarse wavy depth map D. 1 Compared with single-sided cells, wave-shaped depth cells have smaller depth interpolation deviations in the depth fusion stage when the depth prediction deviation is the same, thus generating higher quality point clouds.

[0095] Step S4: Construct a quadruple depth refinement module. Determine a unified initial depth range for the refinement module based on a coarse wavy depth map, using N feature maps. Repeat steps S2 and S3 to obtain the refined wave-shaped depth map and confidence map.

[0096] First, the maximum and minimum mean values ​​of the quadruple depth map are calculated separately. The difference between the maximum and minimum mean values ​​is then calculated. Using this difference as a standard, an interval is extended outwards along the depth direction to both sides as the initial depth range. Since the quadruple depth map can be seen as a repeated measurement of the coarse depth prediction, the larger the difference between the maximum and minimum mean values, the less reliable the coarse depth prediction is. Therefore, the corresponding initial depth range during refinement should be wider to ensure that the underlying true depth is still included within the initial depth range. Overall, the initial depth range of the refinement module is wider than the predefined depth range (d) used in the coarse estimation. min ,d max It needs to be smaller.

[0097] Then, given an initial depth range and N feature maps... Repeat steps S2 and S3 to obtain the refined wavy depth map D. 2 With confidence plot.

[0098] Step S5: Use the efficient multi-scale information-rich feature extractor described in step S1 to extract the features beforehand. By directly calculating and removing the extra feature extraction network used in the traditional Gaussian-Newton module, an efficient Gaussian-Newton module is constructed. This module is used to upsample and optimize the wavy depth map after quarter-resolution refinement to obtain the final half-resolution depth map.

[0099] The traditional Gauss-Newton module uses the Gauss-Newton iterative optimization algorithm, aiming to minimize the error function at pixel p:

[0100]

[0101] In the formula, p i′ This represents the reprojected pixel of p. and These represent the half-resolution reference feature and source feature extracted from the feature pyramid, respectively. During this error minimization process, the optimal depth value of pixel p is determined, thus achieving depth map optimization. Clearly, employing information-rich features can enhance the difference between the source and reference features, making the final depth value obtained through the iterative optimization process more accurate.

[0102] Therefore, by using the efficient multi-scale information-rich feature extractor described in step S1, the pre-extracted features are... On the one hand, this information can be used to enrich features and promote the depth optimization process; on the other hand, the feature pyramid used in the traditional Gaussian-Newton module can be omitted, thereby reducing memory burden and running time, thus constructing an efficient Gaussian-Newton module. This efficient Gaussian-Newton module can refine the wavy depth map D at quarter resolution. 2 Upsampling and optimization are performed to obtain the final half-resolution depth map D. 3 .

[0103] Step S6: Replace the reference image and repeat the above steps until the depth maps corresponding to all images of the target scene are obtained.

[0104] A supervised learning strategy is employed, using the loss described in step S3 and the L1 loss between the effective values ​​of the final depth map and the effective values ​​of the base ground truth depth map to train the network model. Next, the trained network model is used to evaluate each scene, predicting the corresponding depth maps for all its reference images. Through depth map filtering and fusion, a dense point cloud is generated, thereby achieving multi-view capability. Figure 3 3D reconstruction. Multi-view reconstruction based on the MVSNet network with quadruple depth and wave-like depth geometry enhancement. Figure 3 The visualization results of the 3D reconstruction method on the evaluation set of the DTU dataset for reconstructing point clouds of the entire scene are as follows: Figure 2 As shown

[0105] As described above, the embodiments of the present invention employ a multi-view MVSNet network based on quadruple depth and wave-shaped depth geometry enhancement. Figure 3 A 3D reconstruction method is used to reconstruct a dense 3D point cloud of a target scene from 2D RGB images with multiple known camera parameters.

[0106] Example 2

[0107] Experiment setup details

[0108] The DTU dataset is a large-scale indoor multi-view stereo vision dataset containing 124 different scenes. Each scene was scanned from 49 or 64 viewpoints under seven different lighting conditions. The dataset was acquired in a controlled laboratory environment with fixed camera trajectories. To ensure fair comparisons, we followed the ground-value depth maps and dataset partitioning used in previous methods. The official MATLAB evaluation code is used to calculate the distance metrics for the point clouds, namely average accuracy (Accent.) and average integrity (Comp.), in millimeters. The quality of the reconstructed point cloud is measured by overall quality, also known as total quality or overall reconstruction quality, calculated using the following formula:

[0109] Overall = (Acc. + Comp.) / 2

[0110] The experimental software environment configuration used in this embodiment of the invention is as follows: the operating system is Ubuntu 20.04; the GPU is NVIDIA GeForce RTX 3090; the deep learning framework used is PyTorch 1.8.1, CUDA 11.7 and cuDNN 8.6.0.

[0111] The specific experimental settings of this invention embodiment are as follows: the total number of training epochs is 16; the batch size is 4; the optimizer is Adam; the initial learning rate is 0.001, and the learning rate is reduced to half of its original value at the 8th, 10th and 12th epochs.

[0112] Comparison experiment with other network models

[0113] First, the proposed method is quantitatively compared with traditional MVS methods, learning-based progressive optimization MVS methods, and learning-based cascaded MVS methods (see Table 1). The results show that the proposed method outperforms other learning-based methods in terms of completeness and overall quality, especially in progressive optimization methods with similar architectures. Specifically, compared with the recent state-of-the-art learning-based progressive refinement MVS method—ACINR-MVSNet—the proposed method achieves a relative improvement of 24.18% in completeness and 8.96% in overall reconstruction quality.

[0114] Table 1. Results of the reconstruction quality comparison experiment (lower distance index means better reconstruction quality).

[0115]

[0116]

[0117] To verify the robustness of this algorithm, three challenging scenarios—Scan24, Scan75, and Scan77—were selected from the DTU dataset. Reconstruction was performed using R-MVSNet, ACINR-MVSNet, and the proposed method, respectively, and qualitative comparisons were conducted. The results are as follows: Figure 3 As shown, the reconstruction results of R-MVSNet are optimized through its additionally designed full point cloud post-processing steps, which are not included in most other methods (including the method proposed in this invention); ACINR-MVSNet is one of the most recent state-of-the-art learning-based progressive refinement MVS methods. Figure 3 As can be seen, because the method proposed in this invention employs quadruple depth prediction and a wave-shaped depth map, it reconstructs more complete 3D point clouds for R-MVSNet and ACINR-MVSNet in the aforementioned challenging scenarios. This finding is also supported by the comparison results of the integrity indices in Table 1.

[0118] Table 2 compares the GPU memory and runtime required to predict a depth map for different learning-based MVS methods. The table also lists quantitative evaluation metrics for the reconstruction quality of each method, as well as network configurations or experimental settings that directly impact efficiency, including the resolution of the predicted depth map and the number of depth hypothesis planes.

[0119] Table 2. Results of the efficiency comparison experiment.

[0120]

[0121] As shown in Table 2 and Figure 4 As shown, although the proposed method employs a time-consuming dual 3D CNN network to predict quadruple depth, the application of an efficient multi-scale information-enriched feature extractor and an efficient Gaussian-Newton module still allows for relatively fast depth map prediction, even rivaling the efficiency of classic cascaded methods. Specifically, compared to MVSNet, the proposed method reconstructs a depth map that is almost nine times larger, using only 2GB more memory, resulting in a 63.81% reduction in runtime and a 33.98% improvement in reconstruction quality.

[0122] The above experimental results verify the effectiveness of the MVSNet network proposed in this invention, which is based on quadruple depth and wave-shaped depth geometry enhancement, in multi-view applications. Figure 3Effectiveness in 3D point cloud reconstruction. This method not only significantly improves the quality of reconstructed point clouds but also consumes less memory and has faster inference speed compared to other progressive refinement methods.

[0123] ablation experiment

[0124] In the ablation experiments, a network model with 3 training views was used for unified evaluation, and the number of evaluation views was 5. To verify the effectiveness of the proposed quadruple depth and wavy geometry, the quadruple depth refinement module, and the efficient Gaussian-Newton module in the method, the efficient Gaussian-Newton module and the quadruple depth refinement module were gradually removed from the network of the proposed method, and finally, the quadruple depth enhancement MVS module was replaced with a regular MVS module for ablation experiments. For fair comparison, the depth maps predicted by different ablation network models were upsampled to the same resolution as the input image before depth fusion. The quantitative comparison results are shown in Table 3. With the addition of the quadruple depth enhancement MVS module, the quadruple depth refinement module, and the efficient Gaussian-Newton module to the method, the reconstructed point cloud showed a significant performance improvement in terms of completeness and overall accuracy.

[0125] Table 3 Ablation Experiment Results

[0126]

[0127] Qualitative comparison results as follows Figure 5 As shown, with the addition of the quadruple depth enhancement MVS module, the quadruple depth refinement module, and the efficient Gaussian-Newton module to the method, the depth map becomes more accurate, the point cloud becomes denser, and finer details are captured. These improvements further demonstrate the effectiveness of our quadruple depth design, quadruple depth optimization module, and efficient Gaussian-Newton optimization module.

[0128] in conclusion

[0129] In summary, this invention proposes a multi-view MVSNet network based on quadruple depth and wave-shaped depth geometry enhancement. Figure 3This invention proposes a 3D depth reconstruction method. The network model of this method features a progressively refined architecture, effectively avoiding the error propagation problem inherent in cascaded architectures. Its basic process includes four-fold coarse depth estimation, four-fold depth refinement, and depth optimization based on an efficient Gaussian-Newton module. First, this invention designs an efficient multi-scale information-rich feature extractor, capable of extracting information-rich features in a short runtime, thereby helping the network predict depth maps more accurately in challenging scenarios. Second, this invention introduces a visibility-aware view aggregation module based on an epipolar transform, which utilizes a cross-attention mechanism and robust visibility-aware weights calculated by a filtering network to significantly improve the reconstruction quality of invisible regions. Then, this invention proposes predicting a four-fold depth map and calculating a coarse depth map with a wavy geometry from it. This geometry reduces interpolation bias in the depth fusion stage. Furthermore, this invention constructs a four-fold depth refinement module, refining the coarse wavy depth map by determining a unified initial depth range, thereby significantly improving the accuracy and completeness of the reconstructed point cloud. The above complete four-fold depth and wavy depth geometry-related design significantly improves the accuracy and completeness of the reconstructed point cloud. Finally, this invention develops an efficient Gaussian-Newton module to optimize the depth map and improve the output resolution, thereby significantly improving the reconstruction quality. Training and testing on the DTU dataset, experiments verify the multi-view performance of the MVSNet network based on quadruple depth and wavy depth geometry enhancement proposed in this invention. Figure 3 The 3D reconstruction method can significantly improve the quality of reconstructed point clouds. Although this method uses a time-consuming dual 3D CNN structure, it can still complete depth map prediction at a relatively fast speed overall.

[0130] It will be readily understood by those skilled in the art that the above-mentioned preferred solutions can be freely combined and superimposed without conflict. The above description provides a further detailed explanation of the present invention, but it should not be construed as limiting the specific implementation of the invention to these descriptions. For those skilled in the art, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multi-view three-dimensional reconstruction method based on a four-fold depth and wave-type depth geometry enhanced MVSNet network, characterized in that, The method comprises the following steps: Step S1: Obtain a set of multi-view images of a certain scene, select one as a reference image each time and select the (N-1) source images with the smallest image angle for it, and take the N images as input; construct an efficient multi-scale information-rich feature extractor, take the full-resolution reference image and its corresponding source image as input, and obtain two different quarter-resolution scale feature maps and and a set of half-resolution scale feature maps respectively for fourfold depth rough estimation, fourfold depth refinement, depth optimization based on efficient Gauss-Newton module; Step S2: Through differentiable homography transformation, a reference feature map is generated. and N-1 source feature maps N feature maps The reference camera's view frustum is mapped onto a forward parallel plane partitioned based on several depth assumptions, thereby generating N feature volumes. The inter-group correlation similarity between the reference feature volume and N-1 source feature volumes is calculated to construct N-1 similarity volumes. The similarity volumes are fed into the visibility-aware view aggregation module based on epipolar transform to generate a unified matching cost volume. Step S3: obtaining a double two-channel probability volume by using a double 3D CNN regularized cost volume; calculating a four-depth map and a four-confidence map from the double two-channel probability volume; using a multiple loss function to constrain the distribution of the four-depth map around the basic ground truth; using a wavy selection strategy to calculate a rough wavy depth map with a wavy depth geometry from the four-depth map and a wavy confidence map corresponding to the rough wavy depth map from the four-confidence map, thereby completing the four-depth rough estimation; Step S4: constructing a four-fold depth refinement module, determining a unified initial depth range of the refinement module based on the coarse wavy depth map, and using N feature maps Steps S2 and S3 are repeated to obtain a refined wavy depth map and a confidence map. Step S5: using the high-efficiency multi-scale information-rich feature extractor described in step S1 to extract the high-efficiency multi-scale information-rich feature of the wave-shaped depth map in step S4 Direct calculation, remove the additional feature extraction network used in the traditional Gauss-Newton module, and construct a high-efficiency Gauss-Newton module. Using this module, the quarter-resolution refined wave-shaped depth map is up-sampled and optimized to obtain a half-resolution final depth map. Step S6: replacing the reference image and repeating the above steps until the depth maps corresponding to all the images of the target scene are obtained.

2. The multi-view three-dimensional reconstruction method of the MVSNet network based on four-depth and wavy depth geometry enhancement according to claim 1, characterized in that: The high-efficiency multi-scale information-rich feature extractor in step S1 can extract information-rich features, and the specific process is as follows: First, a plurality of convolution layers are used to gradually collect semantic information from the input image, wherein each group of convolution layers includes a convolution operation with a step of 2 for reducing the resolution level; then, the low-resolution intermediate features containing semantic information are up-sampled by using a bilinear interpolation, and the up-sampled features are added to the original high-resolution features containing spatial information; finally, a plurality of convolution layers are applied again to change the feature scale while filtering the features; through this repeated "top-down-bottom-up" processing process, high-level semantic information and low-level spatial information are collected into the information-rich features output by the feature extraction network.

3. The multi-view three-dimensional reconstruction method of the MVSNet network based on four-depth and wavy depth geometry enhancement according to claim 1, characterized in that: The visibility-aware view aggregation module based on the epipolar transformer in step S2 calculates the visibility-aware weight by combining the cross-attention idea of the transformer and the filtering network, and the specific process is as follows: Reference features Source features Transformed source features W i As key vectors, the similarity volumes S i The visibility-aware weights w i are calculated as follows: In the formula, i = 1, …, N-1 represents the corresponding source view number, Φ represents a 2D CNN filtering network, and C represents the number of feature channels; the view weight is also directly used for four-depth refinement.

4. The multi-view three-dimensional reconstruction method of the MVSNet network based on four-depth and wavy depth geometry enhancement according to claim 1, characterized in that: In step S3, the double two-channel probability volume is obtained by feeding the cost volume into a cost volume filtering network composed of double 3D CNNs, and the specific process is as follows: A convolution unit with an output channel of 2 is added at the end of each 3D CNN, and a double two-channel volume is output after a softmax operation, thereby obtaining a double two-channel probability volume.

5. The multi-view three-dimensional reconstruction method of the MVSNet network based on four-depth and wavy depth geometry enhancement according to claim 1, characterized in that: In step S3, the four-depth map and the four-confidence map are calculated from the double two-channel probability volume, and the specific process is as follows: First, the dual two-channel depth maps are computed from the dual two-channel probability volumes using a deep regression approach i.e. each two-channel depth map is computed as the corresponding two-channel probability sum over all depth hypotheses d j :​ where d j (j = 0, 1, …, D - 1, D is the total number of depth hypotheses) represents the depth hypotheses; then, by extracting the maximum and minimum values along the channel dimension, a quad-depth map can be obtained from the dual two-channel depth map: and The confidence is used to measure the quality of depth estimation, which can be calculated as the probability sum of the four closest depth hypotheses, so in a similar manner as described above, the dual two-channel confidence map can be calculated from the dual two-channel probability volume, and then the four-confidence map can be obtained from the dual two-channel confidence map.

6. The multi-view three-dimensional reconstruction method based on the MVSNet network with four-depth and wave-depth geometric enhancement according to claim 1, wherein: In step S3, the multi-loss function is used to constrain the distribution of the four-depth map around the basic ground truth, and the specific process is as follows: First, L1 loss function is used to constrain the four depth maps respectively, so that they are smooth and close to the ground truth depth; then, a loss function is constructed to constrain each pair of depth and to be symmetrically distributed above and below the ground truth depth plane, i.e. the ground truth depth D g is located between and ; finally, the sub-pixel accuracy of the rough wave-shaped depth map D 1 is also constrained by the L1 loss function.

7. The multi-view three-dimensional reconstruction method based on the MVSNet network with four-depth and wave-depth geometric enhancement according to claim 1, wherein: In step S3, the wave selection strategy is used to calculate the rough wave-depth map with wave-depth geometry from the four-depth map, and the wave-confidence map corresponding to the rough wave-depth map is calculated from the four-confidence map, so as to complete the rough estimation of the four-depth, and the specific process is as follows: First, a coordinate mask map is calculated for selecting depth values in different depth maps at different coordinate positions of the coarse wavy depth map; then, four intermediate value mean maps and an overall mean map of the four depth maps are calculated, and the five mean maps are very close to the underlying true value depth map; finally, depth values are selected from the four depth maps and the five mean maps in a specific order, so that the depth in each wavy depth unit slightly fluctuates up and down along the underlying true value depth surface, rather than being entirely on one side of the underlying true value depth surface, thereby forming a coarse wavy depth map D 1 ​ Compared with the single-sided unit, the wave-depth unit has smaller depth interpolation bias in the depth fusion stage under the same depth prediction bias, thereby generating a higher quality point cloud.

8. The multi-view three-dimensional reconstruction method based on the MVSNet network with four-depth and wave-depth geometric enhancement according to claim 1, wherein: In step S4, the unified initial depth range of the refinement module is determined based on the rough wave-depth map, and the specific process is as follows: The maximum mean map and the minimum mean map of the four-depth map are calculated, the difference between the maximum mean map and the minimum mean map is calculated, and then the difference is used as a standard to extend one difference interval to both sides along the depth direction as an initial depth range.

Images