Large-scale multi-view stereo matching method based on edge enhancement and adaptive view angle

Abstract

The large-scale multi-view stereo matching method based on edge enhancement and adaptive view angle comprises the following steps: step (1): a part of the data set of three-dimensional reconstruction of high-resolution aerial images is taken as a training sample, and another part is taken as a test sample; step (2): a lightweight edge-enhanced adaptive multi-view aggregation matching model is constructed; step (3): the overall loss function L of the absolute depth estimation task, the multi-view aggregation weighted task and the edge region perception task in the model is set; step (4): the model constructed in step (2) is trained; step (5): the test sample is input into the lightweight edge-enhanced adaptive multi-view aggregation matching model containing weight parameters, and the depth value and the confidence score of each pixel in the test sample are predicted and output. The application strengthens the contribution efficiency of the source view to the depth estimation task of the reference view, improves the accuracy of large-scale aerial image depth estimation and the reconstruction accuracy of the edge part, and reduces the resource occupation.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image technology, specifically relating to a large-scale multi-view stereo matching method based on edge enhancement and adaptive perspective. Background Technology

[0002] Depth estimation for large-scale aerial scenes aims to estimate the depth of large-scale scenes using drone aerial images. With the development of deep learning technology, multi-view stereo matching networks can produce satisfactory results in near-range object reconstruction. However, in tasks involving Earth surface reconstruction from multi-view aerial images, the large scene scale, wide depth dynamic range, and high image data volume lead to high GPU memory requirements, making them unsuitable for large-scale and high-resolution scene reconstruction. Furthermore, 3D scene reconstruction from tilted aerial images faces new challenges, such as feature inconsistencies due to noise and occlusion, and scale variability of objects observed in different views.

[0003] In deep learning models, the Cas-MVSNet (Cascade Multi-View Stereo Network) model is currently the mainstream baseline for MVSNet, possessing the ability to perform depth estimation for large-scale scenes. However, because the Cas-MVSNet model (DOI: 10.1109 / cvpr42600.2020.00257) is mainly applied to depth estimation tasks in small-scale synthetic scenes, it does not address the high resource requirements for urban scene reconstruction and the occlusion and deformation issues of tilted views, resulting in poor depth estimation performance and efficiency for large-scale aerial scenes. Summary of the Invention

[0004] To overcome the shortcomings of the existing technology, the present invention aims to provide a large-scale multi-view stereo matching method based on edge enhancement and adaptive perspective. This method enhances the contribution efficiency of the source view to the depth estimation task of the reference view, improves the accuracy of depth estimation and edge reconstruction of large-scale aerial images, and reduces resource consumption.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A large-scale multi-view stereo matching method based on edge enhancement and adaptive perspective includes the following steps;

[0007] Step (1): Reconstruct the UAV aerial images with a ground resolution of 0.1m to obtain a dataset. The dataset includes the images and the ground truth depth maps of the images.

[0008] One part of the image is used as training samples, and the other part is used as test samples;

[0009] The input unit of the network is an image pair consisting of a reference view and four auxiliary reference views for depth estimation;

[0010] Step (2): Construct a lightweight edge-enhanced adaptive multi-view aggregation matching model using Cas-MVSNet as the backbone. By constructing different cost bodies to form weights for different source views, the contributions of different source views are combined.

[0011] Step (3): Define the overall loss function L for the absolute depth estimation task, multi-view aggregation weighting task, and edge region perception task in the lightweight edge enhancement adaptive multi-view aggregation matching model:

[0012] Step (4): Train the lightweight edge-enhanced adaptive multi-view aggregation matching model constructed in step (2):

[0013] Step (5): Input the test sample into a lightweight edge-enhanced adaptive multi-view aggregation matching model containing weight parameters, and predict the depth value and confidence score of each pixel in the test sample.

[0014] The specific steps of step (2) in constructing the lightweight edge-enhanced adaptive multi-view aggregation matching model are as follows:

[0015] (2a): Input training samples and obtain three levels of pyramid feature maps from the backbone network and feature pyramid network of Cas-MVSNet, with resolutions from low to high as F1, F2, and F3;

[0016] (2b): Using the pyramid feature map F3 as the input to the edge detection branch, the image edge detection map E is obtained. img ;

[0017] (2c): Constructing the pairwise view weight module;

[0018] (2d): Using the pyramid feature map F1 as input to the pairwise view weighting module, the contribution matrix W of the source view is obtained. s Paired view depth estimation diagram D s ;

[0019] (2e): The contribution matrix W of the source view s Multiplying the pyramid feature maps F1, F2, and F3 respectively yields the pyramid cost volume, which is a pairwise view-weighted weighted volume.

[0020] (2f): Pyramid cost volume weighted by paired views As input to the cyclic regularization and regression modules, the pyramid probability volume is obtained. A pyramid depth estimation map is obtained by using a winner-take-all strategy. Will Spatial resolution unified to With the same spatial resolution, we obtain the upsampled depth estimation map.

[0021] (2g): Depth estimation map Image edge detection map E img As input to the depth refinement module for edge enhancement, a refined depth estimation map D is obtained. refine ;

[0022] In step (2b), the pyramid feature map F3 is used as the input to the edge detection branch to obtain the image edge detection map E. img Its implementation is as follows:

[0023] The pyramid feature map F3 is passed through an edge feature extraction layer, followed by a ReLU activation layer to obtain the image edge features. Image edge features After passing through an edge detection layer, followed by a sigmoid activation layer, the image edge detection map E is obtained. img .

[0024] In (2c), the construction of the pairwise view weight module is implemented as follows:

[0025] (2c1) Pyramid Feature Map F1: An image consisting of a source view and four auxiliary reference views used for depth estimation. The first-level features obtained from the input feature extraction network can be divided into a pyramid feature map F1 based on the reference view and the source views. ref Pyramid feature map F1 with four source views s The pyramid feature map F1 of each of the four source views. s The pyramid feature map F1 is implemented with differentiable homography warped to the reference view. ref Above, the feature body V1 of each source view is obtained. s The pyramid feature map F1 of the reference view ref The feature body V1 of the reference view is obtained by copying it D times in the depth dimension. ref The feature body V1 of each of the four source views. s Feature V1 of the reference view ref The dot product operation is performed on the corresponding pixel features to obtain the paired view cost volume C1;

[0026] (2c2) Apply 2D convolutional regularization to the pairwise view cost volume C1, as follows:

[0027] A network layer consisting of a 2D convolutional layer, a batch normalization layer, and a ReLU activation layer is defined as a batch normalized convolutional layer; a network layer consisting of a 2D deconvolutional layer, a batch normalization layer, and a ReLU activation layer is defined as a batch normalized deconvolutional layer; and a network layer consisting of a batch normalization convolutional layer with a stride of 2 and a batch normalization convolutional layer with a stride of 1 is defined as a downsampling batch normalized convolutional layer. The 2D convolutional regularization operation passes C1 sequentially through a batch normalized convolutional layer and three downsampling batch normalized convolutional layers to obtain results at different resolutions. Will After one batch normalized deconvolution layer and Fusion Will After one batch normalized deconvolution layer and Fusion Will After one batch normalized deconvolution layer and Fusion Will After performing regression through a 2D convolutional layer and a softmax layer, a probability volume P1 is obtained. Then, the probability volume P1 is subjected to soft argmin along the depth dimension to obtain the estimated depth map D for the reference view. s The contribution matrix W of different views is obtained by normalizing the pixel-level probability volumes of different source views. s .

[0028] In (2f), the cyclic regularization and regression module is implemented as follows:

[0029] Pyramid cost volume weighted by paired views As input to the recurrent regularization and regression module, two state variables, state1 and state2, are initialized as zero tensors, and their size is weighted by the pyramid cost volume C of the paired views. agg Correspondingly, the pyramid cost volumes weighted by paired views are sequentially passed through 3x3 convolutional layers with ReLU activation, followed by GRU units based on 3x3 convolutional kernels, and the results are used to update state1. Then, they are passed through downsampled 3x3 convolutional layers with ReLU activation, followed by GRU units based on 3x3 convolutional kernels, and the results are used to update state2. Finally, the results from upsampled convolutional layers are added to the updated state1, and then passed through ReLU activation layers and upsampled convolutional layers to obtain the regularized cost volume. The regularized cost volume is passed through the regression layer and the activation layer respectively to obtain the pyramid probability volume. A pyramid depth estimation map is obtained by using a winner-take-all strategy. Will Spatial resolution unified to With the same spatial resolution, we obtain the upsampled depth estimation map.

[0030] In the (2g) section, the depth estimation map Image edge detection map E img As input to the depth refinement module for edge enhancement, a refined depth estimation map D is obtained. refine Its implementation is as follows:

[0031] Depth estimation map Image edge detection map E img As input to the depth refinement module for edge enhancement, it passes through three convolutional layers with batch normalization and ReLU activation to extract deep geometric edge features. Deep geometric edge features After passing through an edge detection layer, followed by a sigmoid activation layer, the geometric edge detection map E is obtained. geo With image edge detection map E img The geometric-visual fusion edge map E is obtained through OR operation, which incorporates deep geometric edge features. After Tanh activation, the means of the two Gaussian distributions in the depth dimension are obtained. After Sigmoid activation and truncation, the standard deviations of the two Gaussian distributions in the depth dimension are obtained. In the edge regions identified in the geometric-visual fusion edge map E, Gaussian distribution discrimination and mean residual refinement are performed to obtain the refined depth estimation map D. refine .

[0032] Step (3) specifically involves:

[0033] (3a) The existing SmoothL1Loss function is set as the loss function for the absolute depth estimation task in the lightweight edge-enhanced adaptive multi-view aggregation matching model, using L k express;

[0034] (3b) The existing SmoothL1Loss function is set as the loss function for the multi-view aggregation weighting task in the lightweight edge-enhanced adaptive multi-view aggregation matching model, using L s express;

[0035] (3c) Set the existing SmoothL1Loss function as the loss function for the edge region perception task in the lightweight edge-enhanced adaptive multi-view aggregation matching model, using L edge express;

[0036] (3d) Loss function L of the absolute depth estimation task k The loss function L for multi-view aggregation weighted tasks s The loss function L for edge region perception tasksedge Let the loss function L of the lightweight edge-enhanced adaptive multi-view aggregation matching model be:

[0037] L = L k +L s +L edge .

[0038] In (3a), the existing SmoothL1Loss function is set as the loss function L for the absolute depth estimation task in the lightweight edge-enhanced adaptive multi-view aggregation matching model. k It is represented as follows:

[0039]

[0040] Among them, L k D represents the loss in stage k. k (p) represents the depth estimate at pixel p in the k-th stage, D gt (p) represents the ground truth value at depth p pixels, and l(x,y) represents the smoothing L1 squared loss function. x and y represent the depth estimate and the true depth, respectively.

[0041] In (3b), the existing SmoothL1Loss function is set as the loss function L for the multi-view aggregation weighting task in the lightweight edge-enhanced adaptive multi-view aggregation matching model. s It is represented as follows:

[0042]

[0043] Among them, L s D represents the loss between the depth estimate and the true depth value of the reference view when s is used as the source view in the paired view weighting module. s (p) represents the depth estimate at pixel p when s is the source view, D gt (p) represents the ground truth value at depth p pixels, and l(x,y) represents the smoothing L1 squared loss function. x and y represent the depth estimate and the true depth, respectively.

[0044] In (3c), the existing SmoothL1Loss function is set as the loss function L for the edge region perception task in the lightweight edge enhancement adaptive multi-view aggregation matching model. edge It is represented as follows:

[0045]

[0046] Among them, L edge Let E(p) represent the edge loss, where E(p) is the probability estimate that pixel p is an edge.gt (p) represents the true probability that pixel p is an edge, and l(x,y) represents the smoothing L1 squared loss function. x and y represent the edge detection probability estimate and the true edge probability, respectively.

[0047] Step (4) specifically involves:

[0048] (4a) Set the learning rate to 0.001, the optimizer to RMSProp, the batch training data size to 1, and the number of training rounds to 16;

[0049] (4b) Input the training samples obtained in step (1) into the lightweight edge-enhanced adaptive multi-view aggregation matching model, use the optimizer RMSProp to optimize the overall loss function L in (3d), update the weight parameters, and when the number of training rounds reaches 16, the trained lightweight edge-enhanced adaptive multi-view aggregation matching model is obtained.

[0050] The beneficial effects of this invention are:

[0051] First, this invention takes into account the problem that different views contribute differently to the depth estimation of the reference view. It introduces a pairwise view weighting module, establishes a cost body for each pair of views, and determines the contribution matrix of different views to the reference view based on the probability distribution obtained from regression, thereby improving the accuracy of depth estimation of tilted aerial images.

[0052] Second, this invention takes into account the problem of poor depth estimation results in edge regions, and introduces an edge-enhancing depth refinement network to perform bimodal Gaussian distribution modeling and residual calculation on the geometric-visual edge part, thereby optimizing the depth estimation of edge regions.

[0053] Third, this invention addresses the problem of excessive resource requirements for cost volumes in large-scale scenarios using 3D convolutional regularized regression. It employs cyclic regularized regression on the cost volume to effectively reduce resource requirements. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0055] Figure 2 Aerial images of large-scale scenes used for simulation.

[0056] Figure 3 For the purpose of using the present invention to Figure 2 Simulation results image of depth estimation. Detailed Implementation

[0057] The present invention will now be described in further detail with reference to the accompanying drawings.

[0058] Reference Figure 1This embodiment is based on a large-scale multi-view stereo matching method with edge enhancement and adaptive perspective. The implementation steps are as follows:

[0059] Step 1: Construct training and testing samples for depth estimation of large-scale scene aerial images.

[0060] We obtained the publicly available large-scale urban scene 3D reconstruction dataset OMVS, which includes 36,900 images with a width of 768*384 pixels and ground truth depth maps for each image. 20% of these images are used as reference views, and 80% are used as source views for depth estimation of auxiliary reference views. 22,960 images were used as the training set, and 13,940 images were used as the test set.

[0061] According to the Cas-MVSNet settings, the number of viewpoints is set to 5, including one reference view as the main target image for depth estimation, and four source views selected based on structure from motion (patent number: CN113052880B) to assist the reference view in depth estimation.

[0062] The input unit of the network is an image pair consisting of a reference view and four auxiliary reference views for depth estimation;

[0063] Step 2: A lightweight edge-enhanced adaptive multi-view aggregation matching model is constructed using Cas-MVSNet as the backbone. By constructing different cost volumes to form weights for different source views, the contributions of different source views are efficiently combined. Furthermore, the error of the final depth estimation map is reduced through unique edge processing.

[0064] 2.1) Input training samples and obtain three levels of pyramid feature maps from the backbone network and feature pyramid network of Cas-MVSNet, with resolutions from low to high as F1, F2, and F3.

[0065] 2.2) Using the pyramid feature map F3 as input to the edge detection branch, the image edge detection map E is obtained. img :

[0066] The pyramid feature map F3 is passed through an edge feature extraction layer, followed by a ReLU activation layer to obtain the image edge features. Image edge features After passing through an edge detection layer, followed by a sigmoid activation layer, the image edge detection map E is obtained. img ;

[0067] 2.3) Constructing the pairwise view weight module:

[0068] 2.3.1) Pyramid feature map F1 of each source view s The pyramid feature map F1 is implemented with differentiable homography warped to the reference view. ref Above, the feature body V1 of each source view is obtained. s The pyramid feature map F1 of the reference view ref The feature body V1 of the reference view is obtained by copying it D times in the depth dimension. ref The feature body V1 of each source view s Feature V1 of the reference view ref The dot product operation is performed on the corresponding pixel features to obtain the paired view cost volume C1;

[0069] 2.3.2) Apply 2D convolutional regularization to the paired view cost volume C1, as follows:

[0070] A network layer consisting of a 2D convolutional layer, a batch normalization layer, and a ReLU activation layer is defined as a batch normalized convolutional layer; a network layer consisting of a 2D deconvolutional layer, a batch normalization layer, and a ReLU activation layer is defined as a batch normalized deconvolutional layer; and a network layer consisting of one batch normalized convolutional layer with a stride of 2 and one batch normalized convolutional layer with a stride of 1 is defined as a downsampling batch normalized convolutional layer. The 2D convolutional regularization operation is sequentially passed through batch normalized convolutional layers and three downsampling batch normalized convolutional layers to obtain intermediate results at different resolutions. Will After one batch normalized deconvolution layer and Fusion Will After one batch normalized deconvolution layer and Fusion Will After one batch normalized deconvolution layer and Fusion Will The probability volume P1 is obtained by regression after one 2D convolutional layer and one softmax layer, and the contribution matrix W of the current view is obtained. s Depth map D estimated from reference view s The contribution matrix W of different views is obtained by normalizing the probability volumes of different source views at the pixel level. s .

[0071] 2.4) Using the pyramid feature map F1 as input to the pairwise view weighting module, the contribution matrix W of the source view is obtained. s Paired view depth estimation diagram D s ;

[0072] 2.5) The contribution matrix W of the source view sMultiplying the pyramid feature maps F1, F2, and F3 respectively yields the pyramid cost volume, which is a pairwise view-weighted weighted volume.

[0073] 2.6) Pyramid cost volume weighted by paired views As input to the recurrent regularization and regression modules, the pyramid depth estimation map is obtained. Will Spatial resolution unified to With the same spatial resolution, we obtain the upsampled depth estimation map. The loop regularization and regression modules are implemented as follows:

[0074] The pyramid cost volume C weighted by paired views agg As input to the recurrent regularization and regression module, two state variables (state1 and state2) are initialized as zero tensors, and their size is weighted by the pyramid cost volume C of the paired views. agg Corresponding. The pyramid cost volume C weighted by paired views. agg The result of the first pass through a 3x3 convolutional layer with ReLU activation and a GRU unit with a 3x3 convolutional kernel is used to update state1. Then, after passing through a downsampled 3x3 convolutional layer with ReLU activation and a GRU unit with a 3x3 convolutional kernel, the result is used to update state2. The result of the upsampled convolutional layer is added to state1, and then passed through a ReLU activation layer and an upsampled convolutional layer to obtain the regularized cost body. The regularized cost volume is passed through the regression layer and the activation layer respectively to obtain the pyramid probability volume.

[0075] 2.7) Depth estimation map Image edge detection map E img As input to the depth refinement module for edge enhancement, a refined depth estimation map D is obtained. refine Its implementation is as follows:

[0076] Depth estimation map Image edge detection map E img As input to the depth refinement module for edge enhancement, it passes through three convolutional layers with batch normalization and ReLU activation to extract deep geometric edge features. Deep geometric edge features After passing through an edge detection layer, followed by a sigmoid activation layer, the geometric edge detection map E is obtained. geo With image edge detection map E img The geometric-visual fusion edge map E is obtained through OR operation. Deep geometric edge features are then processed. After Tanh activation, the means of the two Gaussian distributions in the depth dimension are obtained. After Sigmoid activation and truncation, the standard deviations of the two Gaussian distributions in the depth dimension are obtained. In the edge regions identified in the geometric-visual fusion edge map E, Gaussian distribution discrimination and mean residual refinement are performed to obtain the refined depth estimation map D. refine .

[0077] Step 3: Define the overall loss function L for the absolute depth estimation task, multi-view aggregation weighting task, and edge region perception task in the lightweight edge-enhanced adaptive multi-view aggregation matching model. The correct iteration of the model is achieved by introducing ground truth supervision for each module.

[0078] 3.1) The existing SmoothL1Loss function is set as the loss function L for the absolute depth estimation task in the lightweight edge-enhanced adaptive multi-view aggregation matching model. k It is represented as follows:

[0079]

[0080] Among them, L k D represents the loss in stage k. k (p) represents the depth estimate at pixel p in the k-th stage, D gt (p) represents the ground truth value at depth p pixels, and l(x,y) represents the smoothing L1 squared loss function. x and y represent the depth estimate and the true depth, respectively.

[0081] 3.2) The existing SmoothL1Loss function is set as the loss function L for the multi-view aggregation weighting task in the lightweight edge-enhanced adaptive multi-view aggregation matching model. s It is represented as follows:

[0082]

[0083] Among them, L s D represents the loss between the depth estimate and the true depth value of the reference view when s is used as the source view in the paired view weighting module. s (p) represents the depth estimate at pixel p when s is the source view, D gt (p) represents the ground truth value at depth p pixels, and l(x,y) represents the smoothing L1 squared loss function. x and y represent the depth estimate and the true depth, respectively.

[0084] 3.3) The existing SmoothL1Loss function is set as the loss function L for the edge region perception task in the lightweight edge-enhanced adaptive multi-view aggregation matching model. edge It is represented as follows:

[0085]

[0086] Among them, L edge Let E(p) represent the edge loss, where E(p) is the probability estimate that pixel p is an edge. gt (p) represents the true probability that pixel p is an edge, and l(x,y) represents the smoothing L1 squared loss function. x and y represent the edge detection probability estimate and the true edge probability, respectively.

[0087] 3.4) Loss function L for the absolute depth estimation task k The loss function L for multi-view aggregation weighted tasks s The loss function L for edge region perception tasks edge Let the loss function L of the lightweight edge-enhanced adaptive multi-view aggregation matching model be:

[0088] L = L k +L s +L edge

[0089] Step 4: Train the lightweight edge-enhanced adaptive multi-view aggregation matching model constructed in Step 2.

[0090] 4.1) Set training parameters:

[0091] Set the learning rate to 0.001, the optimizer to RMSProp, the batch training data size to 1, and the number of training epochs to 16;

[0092] 4.2) Input the training samples from step 1 into the lightweight edge-enhanced adaptive multi-view aggregation matching model, use the optimizer RMSProp to optimize the overall loss function L in step 3, update the weight parameters, and when the number of training rounds reaches 16, the trained lightweight edge-enhanced adaptive multi-view aggregation matching model is obtained.

[0093] Step 5: Input the test samples from Step 1 into a lightweight edge-enhanced adaptive multi-view aggregation matching model containing weight parameters, and predict the depth value and confidence score of each pixel in the test samples.

[0094] The effects of this invention can be further illustrated by the following simulation experiments:

[0095] I. Simulation Conditions

[0096] The simulation is based on the publicly available OMVS dataset and is used to train and test a lightweight edge-enhanced adaptive multi-view aggregation matching model. The benchmark method used is the Cas-MVSNet model.

[0097] The processor used in the simulation is Core TM The system uses an i9-13900K x 32, 125.60GB of RAM, and a 24GB GeForce RTX 4090 GPU. The emulation platform is an Ubuntu 20.04 operating system, using the PyTorch deep learning framework and implemented in Python.

[0098] II. Simulation Content and Analysis

[0099] Using the present invention and the benchmark method (Cas-MVSNet model) respectively, for Figure 2 A depth estimation algorithm simulation was performed, and the simulation results are as follows: Figure 3 As shown, where Figure 3 (a) is a simulation result diagram using existing technology. Figure 3 (b) is a simulation result diagram of the present invention.

[0100] from Figure 3 The results in (a) show that the simulation results of the benchmark method have problems such as inaccurate depth estimation and unclear edges. Figure 3 (b) The present invention can optimize the contribution matching of adjacent views and make the depth estimation of edge parts more accurate.

[0101] III. Comparison and Analysis of Simulation Experiment Results

[0102] To verify the effectiveness of the present invention, depth estimation simulations were performed on the OMVS test dataset using the present invention and the benchmark model. The numerical results of the evaluation metrics for depth estimation were compared, as shown in Table 1.

[0103] Table 1 Comparison of the numerical results of the evaluation indicators detected by the present invention and the benchmark model.

[0104] benchmark model This invention Mean Absolute Error 0.177 0.118 Proportion with an error of less than 0.3m 92.2％ 95.3％ Proportion with an error less than 0.6m 97.0％ 97.9％

[0105] According to the comparison of the evaluation index values ​​of the present invention and the benchmark model in Table 1, it can be seen that the average absolute error of the present invention decreased from 0.177m to 0.118m, the proportion of errors less than 0.3m increased from 92.2% to 95.3%, and the proportion of errors less than 0.6m increased from 97.0% to 97.9%.

[0106] In summary, this invention introduces a pairwise view weighting module and an edge enhancement module into Cas-MVSNet to construct a lightweight adaptive multi-view aggregation matching model with edge enhancement. First, it obtains multi-level pyramid feature maps from the backbone network and feature pyramid network of Cas-MVSNet. Then, the feature map at the top of the pyramid is input into the pairwise view weighting module to obtain the contribution matrix of different source views, which is fused with the cost body construction step in the standard process to obtain a pairwise view weighted pyramid cost body. Simultaneously, the feature map at the bottom of the pyramid is processed through an image edge detection branch to obtain an image edge detection map. The pairwise view weighted pyramid cost body is then processed through a recurrent regularization and regression module to obtain a preliminary depth estimate. Finally, the preliminary depth estimate and the image edge detection map are input into the depth refinement network for edge enhancement, combined with geometric edge detection information to obtain a geometric-visual edge detection map. Gaussian residual refinement is then performed on the edge regions to obtain a refined depth estimation map. By differentiating the contributions of different views to the depth estimation of the reference view and optimizing the edge parts through geometric-visual dual-layer detection, the depth estimation accuracy of large-scale scene aerial images is improved.

Claims

1. A large-scale multi-view stereo matching method based on edge enhancement and adaptive perspective, characterized in that, Includes the following steps; Step (1): Reconstruct the UAV aerial images with a ground resolution of 0.1m to obtain a dataset. The dataset includes the images and the ground truth depth maps of the images. One part of the image is used as training samples, and the other part is used as test samples; The input unit of the network is an image pair consisting of a reference view and four auxiliary reference views for depth estimation; Step (2): Construct a lightweight edge-enhanced adaptive multi-view aggregation matching model using Cas-MVSNet as the backbone. By constructing different cost bodies to form weights for different source views, the contributions of different source views are combined. Step (3): Define the overall loss function L for the absolute depth estimation task, the multi-view aggregation weighting task, and the edge region perception task in the lightweight edge enhancement adaptive multi-view aggregation matching model; Step (4): Train the lightweight edge-enhanced adaptive multi-view aggregation matching model constructed in step (2): Step (5): Input the test sample into a lightweight edge-enhanced adaptive multi-view aggregation matching model containing weight parameters, and predict the depth value and confidence score of each pixel in the test sample. The specific steps of step (2) in constructing the lightweight edge-enhanced adaptive multi-view aggregation matching model are as follows: (2a): Input training samples and obtain three levels of pyramid feature maps from the backbone network and feature pyramid network of Cas-MVSNet, with resolutions from low to high as F1, F2, and F3. (2b): Using the pyramid feature map F3 as the input to the edge detection branch, the image edge detection map E is obtained. img ; (2c): Construct a view weight module outside the main branch; (2d): Using the pyramid feature map F1 as input to the pairwise view weighting module, the contribution matrix W of the source view is obtained. s Paired view depth estimation diagram D s ; (2e): The contribution matrix W of the source view s Multiplying the pyramid feature maps F1, F2, and F3 respectively yields the pyramid cost volume, which is a pairwise view-weighted weighted volume. (2f): Pyramid cost volume weighted by paired views As input to the cyclic regularization and regression modules, the pyramid probability volume is obtained. A pyramid depth estimation map is obtained by using a winner-take-all strategy. Will Spatial resolution unified to With the same spatial resolution, we obtain the upsampled depth estimation map. (2g): Depth estimation map Image edge detection map E img As input to the depth refinement module for edge enhancement, a refined depth estimation map D is obtained. refine ; In step (2b), the pyramid feature map F3 is used as the input to the edge detection branch to obtain the image edge detection map E. img Its implementation is as follows: The pyramid feature map F3 is passed through an edge feature extraction layer, followed by a ReLU activation layer to obtain the image edge features. Image edge features After passing through an edge detection layer, followed by a sigmoid activation layer, the image edge detection map E is obtained. img ; In the (2g) section, the depth estimation map Image edge detection map E img As input to the depth refinement module for edge enhancement, a refined depth estimation map D is obtained. refine Its implementation is as follows: Depth estimation map Image edge detection map E img As input to the depth refinement module for edge enhancement, it passes through three convolutional layers with batch normalization and ReLU activation to extract deep geometric edge features. Deep geometric edge features After passing through an edge detection layer, followed by a sigmoid activation layer, the geometric edge detection map E is obtained. geo With image edge detection map E img The geometric-visual fusion edge map E is obtained through OR operation, which incorporates deep geometric edge features. After Tanh activation, the means of the two Gaussian distributions in the depth dimension are obtained. After Sigmoid activation and truncation, the standard deviations of the two Gaussian distributions in the depth dimension are obtained. In the edge regions identified in the geometric-visual fusion edge map E, Gaussian distribution discrimination and mean residual refinement are performed to obtain the refined depth estimation map D. refine .

2. The large-scale multi-view stereo matching method based on edge enhancement and adaptive perspective as described in claim 1, characterized in that, In (2c), the construction of the pairwise view weight module is implemented as follows: (2c1) Pyramid Feature Map F1: An image consisting of a source view and four auxiliary reference views used for depth estimation. The first-level features obtained from the input feature extraction network can be divided into a pyramid feature map F1 based on the reference view and the source views. ref Pyramid feature map F1 with four source views s The pyramid feature map F1 of each of the four source views. s The pyramid feature map F1 is implemented with differentiable homography warped to the reference view. ref Above, the feature body V1 of each source view is obtained. s The pyramid feature map F1 of the reference view ref The feature body V1 of the reference view is obtained by copying it D times in the depth dimension. ref The feature body V1 of each of the four source views. s Feature V1 of the reference view ref The dot product operation is performed on the corresponding pixel features to obtain the paired view cost volume C1; (2c2) Apply 2D convolutional regularization to the pairwise view cost volume C1, as follows: A network layer consisting of a 2D convolutional layer, a batch normalization layer, and a ReLU activation layer is defined as a batch normalized convolutional layer; a network layer consisting of a 2D deconvolutional layer, a batch normalization layer, and a ReLU activation layer is defined as a batch normalized deconvolutional layer; and a network layer consisting of a batch normalization convolutional layer with a stride of 2 and a batch normalization convolutional layer with a stride of 1 is defined as a downsampling batch normalized convolutional layer. The 2D convolutional regularization operation passes C1 sequentially through a batch normalized convolutional layer and three downsampling batch normalized convolutional layers to obtain results at different resolutions. Will After one batch normalized deconvolution layer and Fusion Will After one batch normalized deconvolution layer and Fusion Will After one batch normalized deconvolution layer and Fusion Will After performing regression through a 2D convolutional layer and a softmax layer, a probability volume P1 is obtained. Then, the probability volume P1 is subjected to soft argmin along the depth dimension to obtain the estimated depth map D for the reference view. s The contribution matrix W of different views is obtained by normalizing the pixel-level probability volumes of different source views. s .

3. The large-scale multi-view stereo matching method based on edge enhancement and adaptive perspective as described in claim 1, characterized in that, In (2f), the cyclic regularization and regression module is implemented as follows: Pyramid cost volume weighted by paired views As input to the recurrent regularization and regression module, two state variables, state1 and state2, are initialized as zero tensors, and their size is weighted by the pyramid cost volume C of the paired views. agg Correspondingly, the pyramid cost volumes weighted by paired views are sequentially passed through 3x3 convolutional layers with ReLU activation, followed by GRU units based on 3x3 convolutional kernels, and the results are used to update state1. Then, they are passed through downsampled 3x3 convolutional layers with ReLU activation, followed by GRU units based on 3x3 convolutional kernels, and the results are used to update state2. Finally, the results from upsampled convolutional layers are added to the updated state1, and then passed through ReLU activation layers and upsampled convolutional layers to obtain the regularized cost volume. The regularized cost volume is passed through the regression layer and the activation layer respectively to obtain the pyramid probability volume.

4. The large-scale multi-view stereo matching method based on edge enhancement and adaptive perspective as described in claim 1, characterized in that, Step (3) specifically involves: (3a) The existing SmoothL1Loss function is set as the loss function for the absolute depth estimation task in the lightweight edge-enhanced adaptive multi-view aggregation matching model, using L k express; (3b) The existing SmoothL1Loss function is set as the loss function for the multi-view aggregation weighting task in the lightweight edge-enhanced adaptive multi-view aggregation matching model, using L s express; (3c) Set the existing SmoothL1Loss function as the loss function for the edge region perception task in the lightweight edge-enhanced adaptive multi-view aggregation matching model, using L edge express; (3d) Loss function L of the absolute depth estimation task k The loss function L for multi-view aggregation weighted tasks s The loss function L for edge region perception tasks edge Let the loss function L of the lightweight edge-enhanced adaptive multi-view aggregation matching model be: L=L k +L s +L edge 。 5. The large-scale multi-view stereo matching method based on edge enhancement and adaptive perspective as described in claim 4, characterized in that, In (3a), the existing SmoothL1Loss function is set as the loss function L for the absolute depth estimation task in the lightweight edge-enhanced adaptive multi-view aggregation matching model. k It is represented as follows: Among them, L k D represents the loss in stage k. k (p) represents the depth estimate at pixel p in the k-th stage, D gt (p) represents the ground truth value at depth p pixels, and l(x,y) represents the smoothing L1 squared loss function. x and y represent the depth estimate and the true depth value, respectively; In (3b), the existing SmoothL1Loss function is set as the loss function L for the multi-view aggregation weighting task in the lightweight edge-enhanced adaptive multi-view aggregation matching model. s It is represented as follows: Among them, L s D represents the loss between the depth estimate and the true depth value of the reference view when s is used as the source view in the paired view weighting module. s (p) represents the depth estimate at pixel p when s is the source view, D gt (p) represents the ground truth value at depth p pixels, and l(x,y) represents the smoothing L1 squared loss function. x and y represent the depth estimate and the true depth value, respectively; In (3c), the existing SmoothL1Loss function is set as the loss function L for the edge region perception task in the lightweight edge enhancement adaptive multi-view aggregation matching model. edge It is represented as follows: Among them, L edge Let E(p) represent the edge loss, where E(p) is the probability estimate that pixel p is an edge. gt (p) represents the true probability that pixel p is an edge, and l(x,y) represents the smoothing L1 squared loss function. x and y represent the edge detection probability estimate and the true edge probability, respectively.

6. The large-scale multi-view stereo matching method based on edge enhancement and adaptive perspective as described in claim 1, characterized in that, Step (4) specifically involves: (4a) Set the learning rate to 0.001, the optimizer to RMSProp, the batch training data size to 1, and the number of training rounds to 16; (4b) Input the training samples obtained in step (1) into the lightweight edge-enhanced adaptive multi-view aggregation matching model, use the optimizer RMSProp to optimize the overall loss function L in (3d), update the weight parameters, and when the number of training rounds reaches 16, the trained lightweight edge-enhanced adaptive multi-view aggregation matching model is obtained.

Images