Multi-view stereo 3d reconstruction method based on depth perception

Abstract

The application relates to a multi-view stereo 3D reconstruction method based on deep perception, and relates to multi-view 3D reconstruction. Through a multi-scale feature extraction framework, high-discriminative features are extracted from images of multiple perspectives in combination with a feature pyramid and a double-branch deep fusion network. Subsequently, the two-dimensional features are mapped to a three-dimensional depth hypothesis plane by using a differentiable homography transformation, a cost volume is constructed, and the feature correlation between views is emphasized by using a grouping aggregation strategy. A 3D U-Net combined with a depth collaborative attention mechanism is used to regularize the cost volume, so that mis-matching and redundancy are reduced. The depth map estimation is regarded as a classification task, the depth hypothesis probability is calculated by using Softmax, the optimal depth value is selected by using Argmax, and a multi-perspective depth map is generated. The whole process is supervised by using a cross-entropy loss to train the model, high-precision 3D reconstruction is realized, and the precision and efficiency of 3D reconstruction are effectively improved. The method has higher precision and robustness in 3D reconstruction of complex scenes.

Description

Technical Field

[0001] This invention relates to multi-view Figure 3 This involves 3D reconstruction, particularly a multi-view stereo 3D reconstruction method based on depth perception. Background Technology

[0002] 3D reconstruction technology is a core research area in computer vision, aiming to recover the 3D structure of a scene. With the rapid development of artificial intelligence and computer vision technologies, 3D reconstruction has been widely applied in various fields such as virtual reality, autonomous driving, game development, precision agriculture, and medical diagnosis. However, despite significant progress in many applications, 3D reconstruction technology still faces numerous challenges when dealing with complex scenes.

[0003] Multi-view stereo (MVS) is an important method for 3D reconstruction, aiming to recover the 3D structure and texture information of a scene from 2D images from multiple viewpoints. Traditional 3D reconstruction methods mainly rely on hand-designed similarity metrics and engineering regularization methods to recover the 3D point cloud of the scene through multi-view feature matching. Typical reconstruction algorithms such as Colmap, Gipuma, and OpenMVS often perform poorly when dealing with complex scenes with weak textures, repeated textures, and reflections, resulting in incomplete or inaccurate reconstruction results.

[0004] With the introduction of deep learning technology, the field of 3D reconstruction has ushered in new breakthroughs. In 2018, Yao Yao et al. proposed the first end-to-end multi-view stereo reconstruction framework, MVSNet. This network first uses a convolutional neural network (CNN) for feature extraction, more effectively obtaining high-level image features. Then, it encodes features through camera parameters and uses differentiable projection transformation to project the features into 3D space to form a cost volume. Finally, it employs a variance-based multi-view aggregation method to adapt to input from any view and uses a 3D convolutional neural network structure for noise filtering and depth map generation. Although this method can extract more discriminative features from large-scale data, the large memory footprint and high computational cost of the 3D CNN structure make it difficult to apply to the reconstruction of real-world scenes. Subsequent research has improved MVSNet in many ways. For example, CasMVSNet and CVP-MVSNet propose a coarse-to-fine depth inference structure, which gradually reduces the number of hypothetical planes by generating depth maps at multiple resolutions, thereby improving accuracy and efficiency. Other methods, such as R-MVSNet and AA-RMVSNet, use recurrent neural networks (RNNs) to segment the 3D cost volume into a series of cost maps along the depth dimension before processing, thereby avoiding the use of 3D CNNs and reducing memory usage.

[0005] While deep learning-based multi-view reconstruction methods have made some progress in accuracy and efficiency, many current coarse-to-fine methods only utilize the coarse-level depth map to guide depth assumptions, failing to fully leverage the depth information from the coarse depth map itself. This results in insufficient accuracy and completeness in depth estimation. Furthermore, during cost volume regularization, the sharing of visual similarity among features at different spatial locations leads to similar confidence scores being assigned to the same location at different depth levels, causing ambiguity in depth estimation and affecting the final reconstruction quality. Therefore, learning-based multi-view stereo 3D reconstruction methods still have significant room for improvement. Summary of the Invention

[0006] The purpose of this invention is to address the problems of insufficient accuracy and completeness in multi-view stereo 3D reconstruction methods, such as the failure to fully utilize depth information in coarse depth maps and feature mismatch during the cost volume regularization stage. This invention proposes a depth-aware multi-view stereo network-based 3D reconstruction method. The method employs a dual-branch deep fusion network and a depth collaborative attention module to fully utilize depth information during multi-view feature extraction and cost volume regularization, thereby improving the accuracy of depth estimation and the quality of 3D reconstruction.

[0007] This invention constructs a novel depth-aware multi-view stereo 3D reconstruction framework, providing an innovative solution for 3D reconstruction technology. Compared with most existing methods, this framework demonstrates significant advantages in improving 3D reconstruction quality, especially in reconstructing complex scenes, effectively enhancing reconstruction accuracy and robustness. It provides strong technical support for a wide range of applications, particularly 3D reconstruction in large-scale and complex environments.

[0008] This invention includes the following steps:

[0009] 1) Multi-view feature extraction: Based on a multi-scale coarse-to-fine feature extraction framework, combined with a feature pyramid and a dual-branch deep fusion network, image features with high discriminative power are extracted from N+1 images from different viewpoints; four feature maps are obtained, with sizes of 1, 1 / 2, 1 / 4, and 1 / 8 of the original image, respectively; the dual-branch deep fusion network includes a feature extraction branch and a deep information fusion branch, both of which adopt an encoder-decoder structure and enhance feature fusion through skip connections;

[0010] 2) Cost body construction: Based on differentiable homography transformation, the two-dimensional source feature maps of different views are mapped to the three-dimensional discrete depth hypothesis plane to obtain N source feature bodies; the similarity between the source feature bodies and the reference features is calculated using a grouping and aggregation strategy to aggregate the source feature bodies and finally construct the cost body C; the grouping and aggregation strategy includes grouping similarity calculation and cross-attention mechanism to emphasize the correlation between features of different views;

[0011] 3) Cost volume regularization: 3D U-Net combined with a deep collaborative attention mechanism is used to regularize the cost volume, reducing mismatches and redundant information in the cost volume; the deep collaborative attention mechanism includes one-dimensional sequence segmentation, one-dimensional convolution processing of different sizes, group normalization and function activation, as well as self-attention calculation, to enhance the feature representation capability of the cost volume.

[0012] 4) Depth map inference: Treat the depth map estimation task as a classification task. Calculate the probability of each depth hypothesis using the Softmax function and select the depth value with the highest probability using the Argmax function. Finally, obtain the multi-view depth map of the scene and use cross-entropy loss to supervise the training of the model.

[0013] Furthermore, the method also includes the following steps: after obtaining the multi-view depth map of the scene, the consistency between the depth maps is calculated by combining the corresponding RGB image and the camera projection matrix using the Open3D library; and the depth information of multiple views is aligned to generate a point cloud model of the scene; through this process, an accurate three-dimensional model of the scene is reconstructed from the two-dimensional images of multiple views.

[0014] In step 1), the encoder part of the feature extraction branch includes a convolutional block, two basic residual blocks and three downsampling layers; the input of the depth information fusion branch is formed by stitching together the fused depth map and the coarse depth map.

[0015] In step 2), the grouping aggregation strategy involves stacking reference feature maps along the depth hypothesis plane to obtain a reference feature body, and then using a grouping strategy with the source feature body to reduce the number of channels in the feature body. Finally, similarity and attention weights are calculated to obtain the final cost body.

[0016] In step 3), the encoder part of the 3D U-Net is implemented by stacking convolutional blocks and downsampling layers to downsample the cost volume multiple times; the decoder part is formed by stacking convolutional blocks and upsampling layers, and the cost volumes of the same scale enhanced by the deep collaborative attention module are added together through skip connections.

[0017] In step 3), the deep collaborative attention mechanism performs average pooling on the input along the width, height and depth dimensions to obtain three one-dimensional sequences. Then, it uses one-dimensional convolution of different sizes, group normalization and function activation to obtain the attention corresponding to each channel. Finally, the weighted cost body is calculated.

[0018] In step 4), the cross-entropy loss calculation method considers the differences between the total number of pixels in the image, the true distribution, and the model's predicted distribution. The total loss function of the four stages of the model is obtained by weighting with different weights.

[0019] Compared with the prior art, the advantages of the present invention are as follows:

[0020] 1) In the multi-view feature extraction stage, this invention combines a feature pyramid and a dual-branch deep fusion network for feature extraction, fully utilizing the depth prior information from the coarse stage, enabling the network to learn the rich implicit scene edge and geometric information. This method can achieve more accurate and robust reconstruction for challenging regions, such as scene reflections, repetitive textures, and weak textures. In contrast, conventional coarse-to-fine deep learning methods only use the depth map from the coarse stage to guide the range of depth assumptions, losing the rich geometric information inherent in the depth map itself, resulting in lower reconstruction accuracy.

[0021] 2) In the cost body construction stage, this invention introduces a group similarity aggregation strategy. Compared with traditional variance-based or learning-based aggregation strategies, this strategy can aggregate source feature bodies more efficiently and accurately, significantly improving the robustness and accuracy of the model.

[0022] 3) During the cost volume regularization stage, the original cost volume calculated from image features may be contaminated by noise due to the presence of non-Lambertian surfaces and object occlusion. This noise may cause features at different keypoints to be mismatched, resulting in similar confidence levels at different depths of the cost volume, affecting the accuracy of depth estimation. Since the similarity confidence problem occurs in different depth assumption layers at the same pixel location, some current attention methods only focus on width, height, and channel dimensions, which has certain limitations. This invention designs a depth collaborative attention module that uses information from the entire cost volume to calculate attention weights to enhance or weaken the similarity confidence at different depths, thus alleviating the feature mismatch problem. Attached Figure Description

[0023] Figure 1 This is a framework diagram of the multi-view stereoscopic reconstruction based on depth perception of the present invention;

[0024] Figure 2 This is a diagram of the dual-branch deep fusion network structure of the present invention;

[0025] Figure 3 This is a diagram of the regularized network structure of the present invention;

[0026] Figure 4 This is a structural diagram of the deep collaborative attention module of the present invention;

[0027] Figure 5 This is a partial reconstruction result of the present invention on the DTU dataset;

[0028] Figure 6 This is a comparison of the reconstruction results of the present invention with those of other methods. Detailed Implementation

[0029] To make the technical solution and advantages of the present invention clearer, the following embodiments will be used to further illustrate the present invention in conjunction with the accompanying drawings.

[0030] This invention proposes a multi-view stereo 3D reconstruction method based on depth perception. The overall network architecture is detailed below. Figure 1 The specific implementation includes the following steps:

[0031] S1. Multi-view Feature Extraction: The method combines a feature pyramid and a two-branch deep fusion network to extract features from... Highly discriminative image features are extracted from images from different perspectives. Based on a multi-scale, coarse-to-fine feature extraction framework, feature maps at four scales are obtained, with sizes of 1, 1 / 2, 1 / 4, and 1 / 8 of the original image, respectively.

[0032] S2. Cost Body Construction: Based on differentiable homography transformation, the two-dimensional source feature maps of different views are mapped to the three-dimensional discrete depth hypothesis plane to obtain N source feature bodies. The similarity between the source feature bodies and the reference features is calculated by using a grouping and aggregation strategy to aggregate the source feature bodies, and finally the cost body C is constructed.

[0033] S3. Cost Volume Regularization: 3D U-Net combined with a deep collaborative attention mechanism is used to regularize the cost volume, reducing mismatches and redundant information in the cost volume.

[0034] S4. Depth Map Inference: Treat the depth map estimation task as a classification task. Calculate the probability of each depth hypothesis using the Softmax function and select the depth value with the highest probability using the Argmax function to obtain the accurate depth map. The training of the model is supervised by cross-entropy loss.

[0035] Step S1 is as follows:

[0036] Given a scenario Zhang Multi-view Images ,in As a reference image, These images are used as source images and input into a feature extraction network for feature extraction. The overall network structure of this invention comprises four stages, progressing from coarse to fine. In the coarsest stage (stage 0), the feature extraction network uses a feature pyramid to extract multi-scale features. Subsequent stages employ a dual-branch deep fusion network. The basic structure of the dual-branch deep fusion network is as follows: Figure 2 As shown, it includes two main branches: a feature extraction branch and a depth information fusion branch. This design can simultaneously utilize the color information of the image and the spatial information of the depth image, effectively promoting the full fusion of image features and depth information, and providing stronger feature representation capabilities for depth estimation tasks.

[0037] The feature extraction branch employs an encoder-decoder network structure with symmetrical skip connections. The encoder consists of a convolutional block, two basic residual blocks, and three downsampling layers. The RGB image is concatenated with the normalized depth map, and after the initial convolutional layer, image features are extracted through multiple downsampling layers and basic residual blocks. Next, the decoder upsamples the feature map through a series of deconvolutional layers, ultimately generating a feature map of the same size as the input image. The specific calculation method is shown in formula (1):

[0038] (1)

[0039] in, Indicates the first Depth map after phase upsampling Indicates the first Reference image of the stage, This indicates a splicing operation. This represents the feature extraction operation of the feature extraction branch.

[0040] The structure of the deep information fusion branch is similar to that of the feature extraction branch, also employing an encoder-decoder structure. The input to this branch consists of the fused depth map. and coarse depth map The network is constructed by splicing features together. The encoder fuses features from corresponding levels in the feature extraction branches through skip connections, thereby enhancing information interaction between branches and improving the network's ability to perceive image texture and depth information. Finally, the output of the depth information fusion branch is combined with the original features extracted from the coarsest stage via FPN. The values ​​are summed, and a refined depth map is finally obtained through a convolutional layer. The calculation method is shown in formula (2).

[0041] (2)

[0042] in, This represents the convolution operation. This represents an element addition operation. The feature extraction operation represents the deep information fusion branch; the final input image. After passing through the feature extraction network, four sets of feature maps at different scales were obtained. , , , The feature map sizes are respectively , , , .

[0043] Step S2 is as follows:

[0044] After obtaining multi-scale feature maps from the multi-view feature extraction step, based on the depth hypothesis... Given the input camera parameters, the position of the corresponding pixel in the source image is calculated in the reference image through differentiable homography transformation, achieving feature matching between the reference and source images. and in the The corresponding pixel in the source view In the A deep hypothesis The feature mapping is defined as follows:

[0045] (3)

[0046] in, and Representing the reference perspective and the first Camera intrinsic parameter matrix from each source viewpoint and From the reference point to the first The rotation and translation matrices of each source viewpoint. Through this process, The source image was transformed to... Individual feature body .

[0047] Next, a grouping similarity aggregation strategy is used to combine the features after feature mapping. The source feature volumes are aggregated into a three-dimensional cost volume, thereby combining the geometric information of multi-view images into a unified three-dimensional structure, where each location contains a cost value under a certain depth assumption. Specifically, the reference feature map... Stacking along the depth assumption plane yields the reference feature volume. and reference feature body With source feature body By employing a grouping strategy, the number of channels in the feature body is reduced, resulting in... ,in, Indicates the number of groups. and This represents the reference feature body and the source feature body after grouping. Calculated according to formula (4). and The similarity is calculated, and the average is calculated based on the number of channels:

[0048] (4)

[0049] in, Indicates the first The reference feature of the group Indicates the first Group 1 in The source feature body of each view, This represents the element-wise product.

[0050] Under the same depth assumption, the contribution of pixels corresponding to different views to the final cost volume is different. Therefore, the attention weights of different source feature volumes are calculated through the cross-attention mechanism according to formula (5):

[0051] (5)

[0052] in, Indicates the first The view in the first Attention score under the layer depth assumption plane This represents the similarity between the reference feature map and the source feature map under the depth hypothesis plane at the th layer. This is the temperature coefficient.

[0053] Finally, based on the aggregation weights corresponding to each source feature body... The final cost body was calculated. The calculation formula is as follows:

[0054] (6)

[0055] The group similarity aggregation strategy effectively improves computational efficiency and generates attention based on the contribution of each source view under different depth assumptions, emphasizing the correlation of features of different views and generating the final cost volume, which lays the foundation for subsequent cost volume regularization.

[0056] Step S3 is as follows:

[0057] Due to the presence of non-Lambertian surfaces and object occlusion, the original cost volume calculated from image features... It may be susceptible to noise contamination. The presence of this noise can lead to erroneous depth information during deep inference. Therefore, the regularization step aims to refine the cost volume to generate a probability volume for deep inference. .

[0058] The regularized network structure adopted in this invention is as follows: Figure 3 As shown, the overall structure is a 3D U-Net with an encoding and decoding mechanism. The encoder is implemented by stacking convolutional blocks and downsampling layers, downsampling the cost volume three times to generate the original scale. , , The cost volume is composed of stacked convolutional blocks and upsampling layers, and the cost volumes of the same scale, which have been enhanced by the deep collaborative attention module, are added together through skip connections.

[0059] Figure 4Given a deep collaborative attention architecture graph, and an input... Average pooling is performed along the width, height, and depth dimensions to obtain three one-dimensional sequences: Each sequence is divided into four sub-features, using It is represented by the following formula:

[0060] (7)

[0061] (8)

[0062] (9)

[0063] in, Then, these sub-features are processed using one-dimensional convolutions of different sizes to obtain... These sub-features are then concatenated and normalized through grouping. After the function is activated, the attention corresponding to each channel is obtained:

[0064] (10)

[0065] (11)

[0066] (12)

[0067] in, , and These represent the weights in the width, height, and depth dimensions, respectively. express function, This indicates a grouping normalization operation. This indicates the splicing operation. The weighted cost body can be calculated using formula (13). :

[0068] (13)

[0069] Next, regarding Applying 7×7×7 average pooling to reduce the cost volume size and then performing grouping normalization yields the following results. Then, grouped convolution techniques are used to generate query, key, and value matrices to compute self-attention.

[0070] (14)

[0071] (15)

[0072] (16)

[0073] (17)

[0074] in, , , These represent the functions that generate the query, key, and value mappings, respectively. Indicates size is

[0075] 1×1×1 grouped convolution.

[0076] Finally, the attention score for the channel dimension is calculated according to formula (19). and with The cost of attention enhancement is obtained by weighting. :

[0077] (18)

[0078] (19)

[0079] Cost body After passing through the cost volume regularization network that incorporates deep collaborative attention in this step, a refined cost volume with noise filtered out is obtained. .

[0080] Step S4 is as follows:

[0081] This invention treats the depth estimation task as a classification task and adopts a "winner-takes-all" strategy to obtain the final depth map. First, the fine cost volume is calculated according to formula (20). Applying the Softmax function along the depth dimension yields the probability volume. The probability body contains the corresponding weights for all depth hypotheses:

[0082] (20)

[0083] in, It is the corresponding cost body The value of the position, This indicates the number of layers assumed for depth.

[0084] Then, the depth hypothesis value with the highest probability is calculated according to formula (21) and used as the final estimated depth value for that pixel:

[0085] (twenty one)

[0086] Specifically, at pixel position Above, from all possible depth assumptions Select a depth value This value makes the probability Maximum. The final result. It is the most likely depth value for that pixel.

[0087] Finally, this invention employs cross-entropy loss to supervise the training of the model, and the calculation method is shown in formula (22):

[0088] (twenty two)

[0089] in, This represents the total number of pixels in the image. Represents the true distribution. In the case of one-hot encoding, the value only corresponds to the actual depth. The value of one item is 1, and the value of the others is 0. The total loss function of the four stages of the model is obtained by weighting the values ​​with different weights. The specific calculation method is shown in formula (23):

[0090] (twenty three)

[0091] in, This represents the total number of stages. This represents the weighting coefficient for the first stage. This represents the cross-entropy loss in the first stage. In this invention, Set to 4, All are set to 1.

[0092] Furthermore, after obtaining the multi-view depth maps of the scene, the corresponding RGB images and the camera's projection matrix are combined, and the consistency between the depth maps is calculated using the Open3D library. The depth information from multiple views is then aligned to generate a point cloud model of the scene.

[0093] The depth-sensing-based multi-view stereo 3D reconstruction method designed in this invention was trained and tested on the publicly available indoor dataset DTU. Figure 5 The reconstruction results of the reconstruction method of the present invention on the DTU test set are shown. The depth-aware multi-view stereo 3D reconstruction method of the present invention can still achieve excellent 3D reconstruction results when facing challenging scenes, such as areas with low texture, reflection and occlusion and other features that are not obvious.

[0094] Figure 6 This paper presents a comparison of point cloud models generated by the present invention with those generated by some existing 3D reconstruction methods. By comparing the point cloud models of different methods, the advantages of the present invention in reconstruction effect can be intuitively demonstrated.

[0095] Table 1 shows the performance differences between the 3D reconstruction method of this invention (Ours) and various existing 3D reconstruction methods (Gipuma, COLMAP, MVSNet, etc.) in terms of accuracy, completeness, and overall error. Bold text indicates the best performance, and underlined text indicates the second best. Evaluation metrics include accuracy, completeness, and overall error; lower values ​​indicate better model performance.

[0096] Table 1 Quantitative Comparison of 3D Reconstruction Results

[0097]

[0098] In the comparison process, accuracy is examined by considering the average distance between the 3D point cloud reconstructed by the method of this invention and the real point cloud. Completeness is assessed by evaluating the average distance between the real point cloud and the point cloud reconstructed by this invention. Overall error is the average of accuracy and completeness, serving as an important indicator for comprehensively evaluating the model's performance. The accuracy of this invention is 0.327 mm, placing it at a relatively high level in the table, indicating that this invention can accurately capture the shape and position of objects during reconstruction with minimal error. The completeness of this invention is 0.301 mm, ranking first in Table 1, significantly better than all other methods, indicating that this invention can relatively completely preserve information from the real scene during reconstruction, reducing information loss. The overall error of this invention is 0.314 mm, also ranking first in Table 1, better than all other methods, indicating that this invention has high overall performance in 3D reconstruction.

[0099] While some methods are comparable to this invention in terms of accuracy (such as CasMVSNet), the advantages of this invention are more pronounced in terms of completeness and overall error. Based on the data in the table and... Figure 5 and 6 The demonstration shows that the present invention can still achieve excellent 3D reconstruction results when facing challenging scenarios (such as areas with low texture, indistinct reflections, and occlusions). This indicates that the present invention has high applicability and robustness.

[0100] Experimental data show that, compared with traditional frameworks, the reconstruction method of this invention can not only capture the details of the target object better, but also maintain high reconstruction quality in complex scenes, achieve more complete and accurate 3D reconstruction, and surpass existing technologies in multiple indicators, demonstrating better performance.

[0101] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A multi-view stereoscopic reconstruction method based on depth perception, characterized in that... Includes the following steps: 1) Multi-view feature extraction: Based on a multi-scale coarse-to-fine feature extraction framework, combined with feature pyramids and a dual-branch deep fusion network, highly discriminative image features are extracted from N+1 images from different viewpoints; Four feature maps at different scales are obtained, with sizes of 1, 1 / 2, 1 / 4, and 1 / 8 of the original image, respectively. The dual-branch deep fusion network includes a feature extraction branch and a depth information fusion branch, both of which adopt an encoder-decoder structure and enhance feature fusion through skip connections. The encoder part of the feature extraction branch includes one convolutional block, two basic residual blocks, and three downsampling layers. The input of the depth information fusion branch is formed by concatenating the fused depth map and the coarse depth map. 2) Cost volume construction: Based on differentiable homography transformation, the two-dimensional source feature maps of different views are mapped to the three-dimensional discrete depth hypothesis plane to obtain N source feature volumes; A grouping and aggregation strategy is used to calculate the similarity between the source feature body and the reference feature body, so as to aggregate the source feature body and finally construct the cost body C. The grouping and aggregation strategy includes grouping similarity calculation and cross-attention mechanism to emphasize the correlation between features of different views. A grouping similarity aggregation strategy is used to combine the features after feature mapping. The source feature volumes are aggregated into a three-dimensional cost volume, thereby combining the geometric information of multi-view images into a unified three-dimensional structure, where each location contains a cost value under a certain depth assumption; specifically, the reference feature map is used. Stacking along the depth assumption plane yields the reference feature volume. and reference feature body With source feature body By employing a grouping strategy, the number of channels in the feature body is reduced, resulting in... ,in, Indicates the number of groups. and Represents the reference feature body and source feature body after grouping; calculated according to the following formula. and The similarity is calculated, and the average is calculated based on the number of channels: in, Indicates the first The reference feature of the group Indicates the first Group 1 in The source feature body of each view, Indicates element-wise product; Under the same depth assumption, pixels corresponding to different views contribute differently to the final cost volume. Therefore, the attention weights of different source feature volumes are calculated using a cross-attention mechanism according to the following formula: in, Indicates the first The view in the first Layer depth assumes attention weights in the plane. This represents the similarity between the reference feature map and the source feature map under the depth hypothesis plane at the th layer. Temperature coefficient; Finally, based on the aggregation weights corresponding to each source feature body The final cost body was calculated. The calculation formula is as follows: The group similarity aggregation strategy effectively improves computational efficiency and generates attention based on the contribution of each source view under different depth assumptions, emphasizing the correlation of features of different views and generating the final cost volume; 3) Cost Volume Regularization: 3D U-Net combined with a deep collaborative attention mechanism is used to regularize the cost volume, reducing mismatches and redundant information. The encoder part of the 3D U-Net is implemented by stacking convolutional blocks and downsampling layers to downsample the cost volume multiple times. The decoder part is constructed by stacking convolutional blocks and upsampling layers, and the cost volumes of the same scale enhanced by the deep collaborative attention module are added together through skip connections. The deep collaborative attention mechanism includes: average pooling of the input along the width, height, and depth dimensions to obtain three one-dimensional sequences; dividing each sequence into four sub-features; processing them with one-dimensional convolutions of different sizes; concatenating them; and then obtaining the attention corresponding to each channel through group normalization and function activation. Finally, the weighted cost volume is calculated to enhance the feature representation capability of the cost volume. 4) Depth map inference: Treat the depth map estimation task as a classification task. Calculate the probability of each depth hypothesis using the Softmax function and select the depth value with the highest probability using the Argmax function. Finally, obtain the multi-view depth map of the scene and use cross-entropy loss to supervise the training of the model.

2. The multi-view stereoscopic reconstruction method based on depth perception as described in claim 1, characterized in that... It also includes the following steps: after obtaining the multi-view depth map of the scene, combine the corresponding RGB image and the camera's projection matrix, and use the Open3D library to calculate the consistency between the depth maps; The depth information from multiple perspectives is then aligned to generate a point cloud model of the scene. This process reconstructs an accurate 3D model of the scene from 2D images from multiple perspectives.

3. The multi-view stereoscopic reconstruction method based on depth perception as described in claim 1, characterized in that... In step 2), the grouping aggregation strategy involves stacking reference feature maps along the depth hypothesis plane to obtain a reference feature body, and then using a grouping strategy with the source feature body to reduce the number of channels in the feature body. Finally, similarity and attention weights are calculated to obtain the final cost body.

4. The multi-view stereoscopic reconstruction method based on depth perception as described in claim 1, characterized in that... In step 2), the specific steps for constructing the cost body are as follows: According to the depth hypothesis Given the input camera parameters, the position of the corresponding pixel in the source image is calculated in the reference image through differentiable homography transformation, achieving feature matching between the reference and source images; for a given pixel in the reference view... and in the The corresponding pixel in the source view In the A deep hypothesis The feature mapping is defined as follows: in, and Representing the reference viewpoint and the first Camera intrinsic parameter matrix from each source viewpoint and From the reference point to the first The rotation and translation matrices from each source perspective; through this process, The source image was transformed to... Individual feature body .

5. The multi-view stereoscopic reconstruction method based on depth perception as described in claim 1, characterized in that... In step 4), the method for calculating the cross-entropy loss considers the differences between the total number of pixels in the image, the true distribution, and the model's predicted distribution. The model's total loss function is obtained by weighting the data with different weights.

6. The multi-view stereoscopic reconstruction method based on depth perception as described in claim 1, characterized in that... In step 4), the specific steps for depth map inference are as follows: Treating the depth map estimation task as a classification task, the fine cost volume is calculated according to the following formula. Applying the Softmax function along the depth dimension yields the probability volume. The probability body contains the corresponding weights for all depth hypotheses: in, It is the corresponding cost body The value of the position, Indicates the number of layers in the depth hypothesis; Then, the depth hypothesis with the highest probability is calculated according to the following formula, and used as the final estimated depth value for that pixel: Specifically, at pixel position Above, from all depth assumptions Select a depth value This value makes the probability The maximum, the final result It is the depth value of that pixel; Finally, the model is trained under the supervision of cross-entropy loss, as shown in the following equation: in, This represents the total number of pixels in the image. Represents the true distribution. In the case of one-hot encoding, the value only corresponds to the actual depth. The term is 1, and the others are 0; the total loss function of the four stages of the model is obtained by weighting the terms with different weights, as shown in the following formula: in, This represents the total number of stages. This represents the weighting coefficient for the first stage. This represents the cross-entropy loss in the first stage.

Images