Large-scale multi-view stereo matching method based on edge enhancement and adaptive view angle
By introducing edge enhancement and adaptive perspective methods into multi-view stereo matching network, the problem of poor depth estimation performance in large-scale aerial photography scenarios is solved, and the reconstruction accuracy and resource utilization efficiency of edge areas are improved.
Patent Information
- Application Number
- CN202411160562.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The existing multi-view stereo matching network has poor depth estimation performance and efficiency in large-scale aerial photography scenarios, especially in urban-level scene reconstruction and occlusion and deformation problems under sloping views.
Using a large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angles, we will build a lightweight edge enhancement adaptive multi-view aggregation matching model, introduce paired view weight modules and edge enhancement depth refinement networks to optimize depth estimation Resource occupancy of the task and reconstruction accuracy of edge areas.
The accuracy of large-scale aerial image depth estimation and the reconstruction accuracy of edge parts are improved, resource occupation is reduced, and the accuracy and efficiency of tilted aerial image depth estimation are improved.
Smart Images

Figure CN119068209A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of remote sensing images, and in particular relates to a large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle. Background Art
[0002] Large-scale aerial scene depth estimation aims to estimate the depth of large-scale scenes by using drone aerial images. With the development of deep learning technology, multi-view stereo matching networks can produce satisfactory results in the reconstruction of close-range objects. However, when applied to the task of reconstructing the earth's surface from multi-view aerial images, due to the large scene scale, large depth dynamic range, and high image data volume, the reconstruction task has high GPU memory requirements, which makes them less suitable for large-scale and high-resolution scene reconstruction. At the same time, the reconstruction of 3D scenes from oblique aerial images faces new challenges, such as inconsistent features due to noise and occlusion, and scale variability of objects observed in different views.
[0003] Among deep learning models, the Cas-MVSNet (Cascade Multi-View Stereo Network) model is the current mainstream baseline of MVSNet, which has the ability to perform depth estimation for large-scale scenes. However, since the Cas-MVSNet (DOI: 10.1109 / cvpr42600.2020.00257) model is mainly used for depth estimation tasks in small-scale synthetic scenes, it does not address the high resource requirements for urban-level scene reconstruction and the occlusion and deformation of oblique views, resulting in poor depth estimation performance and efficiency for large-scale aerial photography scenes. Summary of the invention
[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle, which enhances the contribution efficiency of the source view to the reference view depth estimation task, improves the accuracy of depth estimation of large-scale aerial images and the reconstruction accuracy of the edge part, and reduces resource occupancy.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is:
[0006] A large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle includes the following steps:
[0007] Step (1): reconstruct the drone aerial image with a ground resolution of 0.1 m to obtain a data set, which includes the image and the depth truth map of the image;
[0008] One part of the image is used as a training sample, and the other part is used as a test sample;
[0009] The input unit of the network is an image pair consisting of a reference view and four auxiliary reference views for source views for depth estimation;
[0010] Step (2): A lightweight edge-enhanced adaptive multi-view aggregation matching model is constructed with Cas-MVSNet as the backbone. Different cost volume composition weights are constructed for different source views to achieve the combination of contributions from different source views.
[0011] Step (3): Set the overall loss function L for the absolute depth estimation task, multi-view aggregation weighting task, and edge region perception task in the adaptive multi-view aggregation matching model with lightweight edge enhancement:
[0012] Step (4): Train the lightweight edge-enhanced adaptive multi-view aggregation matching model constructed in step (2):
[0013] Step (5): Input the test sample into the adaptive multi-view aggregation matching model with lightweight edge enhancement including weight parameters, and predict and output the depth value and confidence score of each pixel in the test sample.
[0014] The step (2) of constructing a lightweight edge-enhanced adaptive multi-view aggregation matching model is specifically as follows:
[0015] (2a): Input training samples and obtain three levels of pyramid feature maps from the backbone network and feature pyramid network of Cas-MVSNet, with resolutions from low to high being F1, F2, and F3;
[0016] (2b): Take the pyramid feature map F3 as the input of the edge detection branch to obtain the image edge detection map E img ;
[0017] (2c): Construct a pairwise view weight module;
[0018] (2d) The pyramid feature map F1 is used as the input of the paired view weight module to obtain the contribution matrix W of the source view. s With the paired view depth estimation map D s ;
[0019] (2e): The contribution matrix W of the source view s Multiply with the pyramid feature maps F1, F2, and F3 respectively to obtain the pyramid cost volume weighted by the paired views
[0020] (2f): Pyramid cost volume weighted by pairwise views As the input of the loop regularization and regression module, the pyramid probability volume is obtained. Using the winner-takes-all strategy to obtain the pyramid depth estimation graph Will The spatial resolution is unified to The same spatial resolution, get the upsampled depth estimation map
[0021] (2g): The depth estimation map And image edge detection figure E img As the input of the edge-enhanced depth refinement module, the refined depth estimation map D is obtained. refine ;
[0022] In (2b), the pyramid feature map F3 is used as the input of the edge detection branch to obtain the image edge detection map E img , which is implemented as follows:
[0023] The pyramid feature map F3 passes through an edge feature extraction layer and then passes through a ReLU activation layer to obtain the image edge features. The image edge features After an edge detection layer, a Sigmoid activation layer is used to obtain the image edge detection map E. img .
[0024] In (2c), the pairwise view weight module is constructed as follows:
[0025] (2c1) Pyramid feature map F1 An image pair consisting of a reference view and four auxiliary reference views for depth estimation of the source view is input to the feature extraction network to obtain the first-level features. It can be divided into a pyramid feature map F1 of the reference view according to the reference view and the source view. ref Pyramid feature map F1 with four source views s , the pyramid feature map F1 of each of the four source views s Implement differentiable homography warping to the pyramid feature map F1 of the reference view ref On the top, we get the feature volume V1 of each source view s , the pyramid feature map F1 of the reference view ref D times in the depth dimension to obtain the feature volume V1 of the reference view ref , the feature volume V1 of each source view in the four source views s Feature body V1 with reference view ref Perform dot product operation on the corresponding pixel features to obtain the paired view cost volume C1;
[0026] (2c2) A 2D convolution regularization operation is performed on the paired view cost volume C1, which is implemented as follows:
[0027] It is stipulated that the network layer composed of a two-dimensional convolution layer, a batch normalization layer, and a ReLU activation layer is a batch normalized convolution layer, and the network layer composed of a two-dimensional deconvolution layer, a batch normalization layer, and a ReLU activation layer is a batch normalized deconvolution layer. It is stipulated that the network layer composed of a batch normalized convolution layer with a step size of 2 and a batch normalized convolution layer with a step size of 1 is a downsampled batch normalized convolution layer; the 2D convolution regularization operation is to pass C1 through a batch normalized convolution layer and three layers of downsampled batch normalized convolution layers in turn to obtain results of different resolutions. Will After a batch normalization deconvolution layer and Fusion Will After a batch normalization deconvolution layer and Fusion Will After a batch normalization deconvolution layer and Fusion Will After a two-dimensional convolution layer and a Softmax layer, the probability body P1 is regressed. The probability body P1 is subjected to Soft argmin in the depth dimension to obtain the depth map D estimated for the reference view. s , according to the probability volume of different source views, the contribution matrix W of different views is obtained by normalizing the pixel level s .
[0028] In (2f), the cycle regularization and regression module is implemented as follows:
[0029] Pyramid cost volume weighting pairs of views As the input of the loop regularization and regression module, the two state variables state1 and state2 are initialized to zero tensors and the pyramid cost volume C with the same size as the pairwise view weights agg Correspondingly, the pyramid cost body weighted by paired views is passed through a 3*3 convolution layer with ReLU activation and a GRU unit based on a 3*3 convolution kernel, and the result is used to update state1; the result is passed through a downsampling 3*3 convolution layer with ReLU activation and a GRU unit based on a 3*3 convolution kernel, and the result is used to update state2. The result obtained through the upsampling convolution layer is added to the updated state1, and then the regularized cost body is obtained through the ReLU activation layer and the upsampling convolution layer. The regularized cost volume is passed through the regression layer and the activation layer to obtain the pyramid probability volume Using the winner-takes-all strategy to obtain the pyramid depth estimation graph Will The spatial resolution is unified to The same spatial resolution, get the upsampled depth estimation map
[0030] In (2g), the depth estimation map And image edge detection figure E img As the input of the edge-enhanced depth refinement module, the refined depth estimation map D is obtained. refine , which is implemented as follows:
[0031] The depth estimation map And image edge detection figure E img As the input of the edge enhancement deep refinement module, it passes through three layers of convolutional layers with batch normalization and ReLU activation to extract deep geometric edge features. Deep geometric edge features After an edge detection layer, a Sigmoid activation layer is used to obtain the geometric edge detection map E. geo , and the image edge detection graph E img The geometric-visual fusion edge map E is obtained by OR operation, and the deep geometric edge features are After Tanh activation, the mean of the two Gaussian distributions in the depth dimension is obtained. After Sigmoid activation and truncation, the standard deviation of the two Gaussian distributions in the depth dimension is obtained. Gaussian distribution discrimination and mean residual refinement are performed in the part judged as the edge in the geometric-visual fusion edge map E to obtain the refined depth estimation map D. refine .
[0032] The step (3) is specifically:
[0033] (3a) The existing SmoothL1Loss function is set as the loss function of the absolute depth estimation task in the adaptive multi-view aggregation matching model with lightweight edge enhancement, and L k express;
[0034] (3b) The existing SmoothL1Loss function is set as the loss function of the multi-view aggregation weighted task in the lightweight edge-enhanced adaptive multi-view aggregation matching model, and L s express;
[0035] (3c) The existing SmoothL1Loss function is set as the loss function of the edge region perception task in the adaptive multi-view aggregation matching model with lightweight edge enhancement, and L edge express;
[0036] (3d) The loss function L of the absolute depth estimation task k , the loss function L for the multi-view aggregation weighted task s , the loss function L of the edge area perception taskedge , the loss function L of the adaptive multi-view aggregation matching model with lightweight edge enhancement is set as:
[0037] L=L k +L s +L edge .
[0038] In (3a), the existing SmoothL1Loss function is set as the loss function L of the absolute depth estimation task in the adaptive multi-view aggregation matching model with lightweight edge enhancement. k , which is expressed as follows:
[0039]
[0040] Among them, L k represents the loss of the kth stage, D k (p) represents the depth estimation value of p pixels in the kth stage, D gt (p) represents the true depth value at p pixels, l(x,y) represents the smooth L1 square loss function, x and y represent the depth estimate and the true depth respectively.
[0041] In (3b), the existing SmoothL1Loss function is set as the loss function L of the multi-view aggregation weighted task in the adaptive multi-view aggregation matching model with lightweight edge enhancement. s , which is expressed as follows:
[0042]
[0043] Among them, L s Denotes the loss between the depth estimate and the ground-truth of the reference view when s is used as the source view in constructing the paired view weight module, D s (p) represents the depth estimation value at pixel p when s is used as the source view, D gt (p) represents the true depth value at p pixels, l(x,y) represents the smooth L1 square loss function, x and y represent the depth estimate and the true depth respectively.
[0044] In (3c), the existing SmoothL1Loss function is set as the loss function L of the edge region perception task in the adaptive multi-view aggregation matching model with lightweight edge enhancement. edge , which is expressed as follows:
[0045]
[0046] Among them, L edge represents edge loss, E(p) represents the probability estimate of pixel p being judged as an edge, and Egt (p) represents the true probability value of p pixels being judged as edges, l(x,y) represents the smooth L1 square loss function, x and y represent the estimated value of edge map detection probability and the true value of edge map probability respectively.
[0047] The step (4) is specifically:
[0048] (4a) Set the learning rate to 0.001, the optimizer to use RMSProp, the batch training data size to 1, and the number of training rounds to 16;
[0049] (4b) The training samples obtained in step (1) are input into the lightweight edge-enhanced adaptive multi-view aggregation matching model, and the optimizer RMSProp is used to optimize the overall loss function L in (3d), and the weight parameters are updated. When the number of training rounds reaches 16, the trained lightweight edge-enhanced adaptive multi-view aggregation matching model is obtained.
[0050] Beneficial effects of the present invention:
[0051] First, the present invention takes into account the problem that different views contribute differently to the depth estimation of the reference view. It introduces a paired view weight module, establishes cost bodies for paired views respectively, and determines the contribution matrix of different views to the reference view according to the probability distribution obtained by regression, thereby improving the accuracy of depth estimation of oblique aerial images.
[0052] Second, the present invention takes into account the problem of poor depth estimation results in edge areas, introduces an edge-enhanced depth refinement network, performs bimodal Gaussian distribution modeling and residual operations on the geometric-visual edge parts, and optimizes the depth estimation of edge areas.
[0053] Third, the present invention takes into account the problem of excessive resource requirements of cost body through three-dimensional convolution regularization regression in large-scale scenarios, and adopts cyclic regularization regression for the cost body to effectively reduce resource requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is a flow chart for implementing the present invention.
[0055] Figure 2 Aerial image of the large-scale scene used for simulation.
[0056] Figure 3 In order to use the present invention Figure 2 Image of simulation results for depth estimation. DETAILED DESCRIPTION
[0057] The present invention will be further described in detail below in conjunction with the accompanying drawings.
[0058] Reference Figure 1In this embodiment, a large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle is implemented in the following steps:
[0059] Step 1: Construct large-scale scene aerial image depth estimation training samples and test samples.
[0060] Obtain the public urban-level large-scale scene 3D reconstruction dataset OMVS, which includes 36,900 images with a width of 768*384 pixels and the depth truth map of each image, of which 20% are used as reference views and 80% are used as source views for auxiliary reference view depth estimation. 22,960 images are used as training sets and 13,940 images are used as test sets.
[0061] According to the Cas-MVSNet setting, the number of views is set to 5, including one reference view as the main depth estimation target image and four source views selected according to structure from motion (patent number: CN113052880B) to assist the reference view in depth estimation;
[0062] The input unit of the network is an image pair consisting of a reference view and four auxiliary reference views for source views for depth estimation;
[0063] Step 2: A lightweight edge-enhanced adaptive multi-view aggregation matching model is constructed with Cas-MVSNet as the backbone. By constructing different cost volume composition weights for different source views, the contributions of different source views are efficiently combined. The error of the final depth estimation map is reduced through unique processing of edges.
[0064] 2.1) Input training samples and obtain three levels of pyramid feature maps from the backbone network and feature pyramid network of Cas-MVSNet, with resolutions from low to high being F1, F2, and F3;
[0065] 2.2) Take the pyramid feature map F3 as the input of the edge detection branch to obtain the image edge detection map E img :
[0066] The pyramid feature map F3 passes through an edge feature extraction layer and then passes through a ReLU activation layer to obtain the image edge features. The image edge features After an edge detection layer, a Sigmoid activation layer is used to obtain the image edge detection map E. img ;
[0067] 2.3) Constructing paired view weight module:
[0068] 2.3.1) The pyramid feature map F1 of each source view s Implement differentiable homography warping to the pyramid feature map F1 of the reference view ref On the top, we get the feature volume V1 of each source view s , the pyramid feature map F1 of the reference view ref D times in the depth dimension to obtain the feature volume V1 of the reference view ref , the feature volume V1 of each source view s Feature body V1 with reference view ref Perform dot product operation on the corresponding pixel features to obtain the paired view cost volume C1;
[0069] 2.3.2) A 2D convolution regularization operation is performed on the paired view cost volume C1, which is implemented as follows:
[0070] The network layer consisting of a 2D convolution layer, a batch normalization layer, and a ReLU activation layer is defined as a batch normalization convolution layer, and the network layer consisting of a 2D deconvolution layer, a batch normalization layer, and a ReLU activation layer is defined as a batch normalization deconvolution layer. The network layer consisting of a batch normalization convolution layer with a stride of 2 and a batch normalization convolution layer with a stride of 1 is defined as a downsampled batch normalization convolution layer. The 2D convolution regularization operation passes through the batch normalization convolution layer and the three downsampled batch normalization convolution layers in turn to obtain intermediate results of different resolutions. Will After a batch normalization deconvolution layer and Fusion Will After a batch normalization deconvolution layer and Fusion Will After a batch normalization deconvolution layer and Fusion Will After a two-dimensional convolution layer and a Softmax layer, the probability body P1 is obtained and the contribution matrix W of the current view is s , the depth map D estimated for the reference view s ; According to the probability volume of different source views, the pixel level normalization is performed to obtain the contribution matrix W of different views s .
[0071] 2.4) Take the pyramid feature map F1 as the input of the paired view weight module to obtain the contribution matrix W of the source view s With the paired view depth estimation map D s ;
[0072] 2.5) The contribution matrix W of the source view sMultiply with the pyramid feature maps F1, F2, and F3 respectively to obtain the pyramid cost volume weighted by the paired views
[0073] 2.6) Pyramid cost volume weighted by paired views As the input of the loop regularization and regression module, the pyramid depth estimation map is obtained Will The spatial resolution is unified to The same spatial resolution, get the upsampled depth estimation map The loop regularization and regression modules are implemented as follows:
[0074] The pyramid cost volume C of the weighted pairwise views agg As the input of the loop regularization and regression module, the two state variables (state1 and state2) are initialized to zero tensors and the pyramid cost volume C with the same size as the pairwise view weights agg Corresponding. Pyramid cost volume C weighted by pairwise views agg The result obtained by sequentially passing through the 3*3 convolution layer with ReLU activation and the GRU unit based on the 3*3 convolution kernel is used to update the state state1. The result obtained by passing through the downsampling 3*3 convolution layer with ReLU activation and the GRU unit based on the 3*3 convolution kernel is used to update the state2. The result obtained by the upsampling convolution layer is added to state1, and then the regularized cost volume is obtained by passing through the ReLU activation layer and the upsampling convolution layer. The regularized cost volume is passed through the regression layer and the activation layer to obtain the pyramid probability volume
[0075] 2.7) The depth estimation map And image edge detection figure E img As the input of the edge-enhanced depth refinement module, the refined depth estimation map D is obtained. refine , which is implemented as follows:
[0076] The depth estimation map And image edge detection figure E img As the input of the edge enhancement deep refinement module, it passes through three layers of convolutional layers with batch normalization and ReLU activation to extract deep geometric edge features. Deep geometric edge features After an edge detection layer, a Sigmoid activation layer is used to obtain the geometric edge detection map E. geo , and the image edge detection graph E img The geometric-visual fusion edge map E is obtained by OR operation. After Tanh activation, the mean of the two Gaussian distributions in the depth dimension is obtained. After Sigmoid activation and truncation, the standard deviation of the two Gaussian distributions in the depth dimension is obtained. Gaussian distribution discrimination and mean residual refinement are performed in the part judged as the edge in the geometric-visual fusion edge map E to obtain the refined depth estimation map D. refine .
[0077] Step 3: Set the overall loss function L of the absolute depth estimation task, multi-view aggregation weighted task, and edge area perception task in the lightweight edge-enhanced adaptive multi-view aggregation matching model, and realize the correct iteration of the model by introducing the true value supervision of each module.
[0078] 3.1) Set the existing SmoothL1Loss function as the loss function L for the absolute depth estimation task in the adaptive multi-view aggregation matching model with lightweight edge enhancement k , which is expressed as follows:
[0079]
[0080] Among them, L k represents the loss of the kth stage, D k (p) represents the depth estimation value of p pixels in the kth stage, D gt (p) represents the true depth value at p pixels, l(x,y) represents the smooth L1 square loss function, x and y represent the depth estimate and the true depth respectively.
[0081] 3.2) Set the existing SmoothL1Loss function as the loss function L of the multi-view aggregation weighted task in the lightweight edge-enhanced adaptive multi-view aggregation matching model s , which is expressed as follows:
[0082]
[0083] Among them, L s Denotes the loss between the depth estimate and the ground-truth of the reference view when s is used as the source view in constructing the paired view weight module, D s (p) represents the depth estimation value at pixel p when s is used as the source view, D gt (p) represents the true depth value at p pixels, l(x,y) represents the smooth L1 square loss function, x and y represent the depth estimate and the true depth respectively.
[0084] 3.3) Set the existing SmoothL1Loss function as the loss function L for edge region perception task in the adaptive multi-view aggregation matching model with lightweight edge enhancement edge , which is expressed as follows:
[0085]
[0086] Among them, L edge represents edge loss, E(p) represents the probability estimate of pixel p being judged as an edge, and E gt (p) represents the true probability value of p pixels being judged as edges, l(x,y) represents the smooth L1 square loss function, x and y represent the estimated value of edge map detection probability and the true value of edge map probability respectively.
[0087] 3.4) Loss function L of the absolute depth estimation task k , the loss function L for the multi-view aggregation weighted task s , the loss function L of the edge area perception task edge , the loss function L of the adaptive multi-view aggregation matching model with lightweight edge enhancement is set as:
[0088] L=L k +L s +L edge
[0089] Step 4: Train the lightweight edge-enhanced adaptive multi-view aggregation matching model constructed in step 2.
[0090] 4.1) Set training parameters:
[0091] Set the learning rate to 0.001, the optimizer to use RMSProp, the batch training data size to 1, and the number of training rounds to 16;
[0092] 4.2) Input the training samples in step 1 into the lightweight edge-enhanced adaptive multi-view aggregation matching model, use the optimizer RMSProp to optimize the overall loss function L in step 3, update the weight parameters, and when the number of training rounds reaches 16, obtain the trained lightweight edge-enhanced adaptive multi-view aggregation matching model.
[0093] In step 5, the test sample in step 1 is input into the adaptive multi-view aggregation matching model with lightweight edge enhancement including weight parameters, and the depth value and confidence score of each pixel in the output test sample are predicted.
[0094] The effect of the present invention can be further illustrated by the following simulation experiment:
[0095] 1. Simulation conditions
[0096] The simulation is based on the public OMVS dataset, and the training and testing of the lightweight edge-enhanced adaptive multi-view aggregation matching model are performed. The benchmark method used is the Cas-MVSNet model.
[0097] The processor used for the simulation is Core TM i9-13900K×32, memory is 125.60GB, GPU is 24G GeForceRTX4090, the simulation platform is Ubuntu20.04 operating system, using Pytorch deep learning framework, implemented in Python language.
[0098] 2. Simulation content and analysis
[0099] The present invention and the benchmark method (Cas-MVSNet model) are used to Figure 2 The depth estimation algorithm is simulated, and the simulation results are as follows Figure 3 As shown, Figure 3 (a) is the simulation result diagram using the prior art. Figure 3 (b) is a diagram of the simulation results of the present invention.
[0100] from Figure 3 The results of (a) show that the simulation results of the baseline method have problems such as inaccurate depth estimation and unclear edges. Figure 3 (b) The present invention can optimize the matching of adjacent view contributions, and the depth estimation of the edge part is more accurate.
[0101] 3. Comparison and analysis of simulation experimental results
[0102] To verify the effectiveness of the present invention, the present invention and the benchmark model are used to perform depth estimation simulation on the OMVS test data set, and the numerical results of the evaluation indicators of the depth estimation are compared, as shown in Table 1.
[0103] Table 1 Comparison of numerical results of evaluation indicators detected by the present invention and the benchmark model
[0104] Benchmark Model The present invention Mean absolute error 0.177 0.118 Error less than 0.3m ratio 92.2% 95.3% Error less than 0.6m ratio 97.0% 97.9%
[0105] According to the comparison of the numerical results of the evaluation indicators of the present invention and the benchmark model in Table 1, it can be seen that the average absolute error of the present invention is reduced from 0.177m to 0.118m, the proportion of errors less than 0.3m is increased from 92.2% to 95.3%, and the proportion of errors less than 0.6m is increased from 97.0% to 97.9%.
[0106] In summary, the present invention introduces a paired view weight module and an edge enhancement module in Cas-MVSNet to construct an adaptive multi-view aggregation matching model with lightweight edge enhancement. It first obtains pyramid feature maps of multiple levels from the backbone network and feature pyramid network of Cas-MVSNet; then the feature map at the top of the pyramid is input into the paired view weight module to obtain the contribution matrix of different source views, and is fused with the cost body construction step in the standard process to obtain the paired view weighted pyramid cost body; at the same time, the feature map at the bottom of the pyramid is passed through the image edge detection branch to obtain the image edge detection map; the paired view weighted pyramid cost body is passed through the cyclic regularization and regression module to obtain the preliminary estimated depth; finally, the preliminary estimated depth and the image edge detection map are input into the edge enhanced depth refinement network, and the geometric edge detection information is combined to obtain the geometric-visual edge detection map, and the edge area is refined by Gaussian residual operation to obtain the refined depth estimation map. By making differentiated designs for the contributions of different views to the depth estimation of the reference view, and performing geometric-visual double-layer detection on the edge part and then optimizing it, the depth estimation accuracy of large-scale scene aerial images is improved.
Claims
1. A large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle, characterized in that: The steps include: Step (1): reconstruct the drone aerial image with a ground resolution of 0.1 m to obtain a data set, which includes the image and the depth truth map of the image; One part of the image is used as a training sample, and the other part is used as a test sample; The input unit of the network is an image pair consisting of a reference view and four auxiliary reference views for source views for depth estimation; Step (2): A lightweight edge-enhanced adaptive multi-view aggregation matching model is constructed with Cas-MVSNet as the backbone. Different cost volume composition weights are constructed for different source views to achieve the combination of contributions from different source views. Step (3): setting the overall loss function L of the absolute depth estimation task, the multi-view aggregation weighting task and the edge region perception task in the adaptive multi-view aggregation matching model with lightweight edge enhancement; Step (4): Train the lightweight edge-enhanced adaptive multi-view aggregation matching model constructed in step (2): Step (5): Input the test sample into the adaptive multi-view aggregation matching model with lightweight edge enhancement including weight parameters, and predict and output the depth value and confidence score of each pixel in the test sample.
2. The large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle according to claim 1, characterized in that: The step (2) of constructing a lightweight edge-enhanced adaptive multi-view aggregation matching model is specifically as follows: (2a): Input training samples and obtain three levels of pyramid feature maps from the backbone network and feature pyramid network of Cas-MVSNet, with resolutions from low to high being F1, F2, and F3; (2b): Take the pyramid feature map F3 as the input of the edge detection branch to obtain the image edge detection map E img ; (2c): Build a pairwise view weight module outside the main branch; (2d) The pyramid feature map F1 is used as the input of the paired view weight module to obtain the contribution matrix W of the source view. s With the paired view depth estimation map D s ; (2e): The contribution matrix W of the source view s Multiply with the pyramid feature maps F1, F2, and F3 respectively to obtain the pyramid cost volume weighted by the paired views (2f): Pyramid cost volume weighted by pairwise views As the input of the loop regularization and regression module, the pyramid probability volume is obtained. Using the winner-takes-all strategy to obtain the pyramid depth estimation graph Will The spatial resolution is unified to The same spatial resolution, get the upsampled depth estimation map (2g): The depth estimation map And image edge detection figure E img As the input of the edge-enhanced depth refinement module, the refined depth estimation map D is obtained. refine .
3. The large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle according to claim 2, characterized in that: In (2b), the pyramid feature map F3 is used as the input of the edge detection branch to obtain the image edge detection map E img , which is implemented as follows: The pyramid feature map F3 passes through an edge feature extraction layer and then passes through a ReLU activation layer to obtain the image edge features. The image edge features After an edge detection layer, a Sigmoid activation layer is used to obtain the image edge detection map E. img .
4. The large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle according to claim 2, characterized in that: In (2c), the pairwise view weight module is constructed as follows: (2c1) Pyramid feature map F1 An image pair consisting of a reference view and four auxiliary reference views for source view depth estimation is input to the feature extraction network to obtain the first-level features. It can be divided into a pyramid feature map F1 of the reference view according to the reference view and the source view. ref Pyramid feature map F1 with four source views s , the pyramid feature map F1 of each of the four source views s Implement differentiable homography warping to the pyramid feature map F1 of the reference view ref On the top, we get the feature volume V1 of each source view s , the pyramid feature map F1 of the reference view ref D times in the depth dimension to obtain the feature volume V1 of the reference view ref , the feature volume V1 of each source view in the four source views s Feature body V1 with reference view ref Perform dot product operation on the corresponding pixel features to obtain the paired view cost volume C1; (2c2) A 2D convolution regularization operation is performed on the paired view cost volume C1, which is implemented as follows: It is stipulated that the network layer composed of a two-dimensional convolution layer, a batch normalization layer, and a ReLU activation layer is a batch normalized convolution layer, and the network layer composed of a two-dimensional deconvolution layer, a batch normalization layer, and a ReLU activation layer is a batch normalized deconvolution layer. It is stipulated that the network layer composed of a batch normalized convolution layer with a step size of 2 and a batch normalized convolution layer with a step size of 1 is a downsampled batch normalized convolution layer; the 2D convolution regularization operation is to pass C1 through a batch normalized convolution layer and three layers of downsampled batch normalized convolution layers in turn to obtain results of different resolutions. Will After a batch normalization deconvolution layer and Fusion Will After a batch normalization deconvolution layer and Fusion Will After a batch normalization deconvolution layer and Fusion Will After a two-dimensional convolution layer and a Softmax layer, the probability body P1 is regressed. The probability body P1 is subjected to Soft argmin in the depth dimension to obtain the depth map D estimated for the reference view. s , according to the probability volume of different source views, the contribution matrix W of different views is obtained by normalizing the pixel level s .
5. The large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle according to claim 2, characterized in that: In (2f), the cycle regularization and regression module is implemented as follows: Pyramid cost volume weighting pairs of views As the input of the loop regularization and regression module, the two state variables state1 and state2 are initialized to zero tensors and the pyramid cost volume C with the same size as the pairwise view weights agg Correspondingly, the pyramid cost body weighted by paired views is passed through a 3*3 convolution layer with ReLU activation and a GRU unit based on a 3*3 convolution kernel, and the result is used to update state1; the result is passed through a downsampling 3*3 convolution layer with ReLU activation and a GRU unit based on a 3*3 convolution kernel, and the result is used to update state2. The result obtained through the upsampling convolution layer is added to the updated state1, and then the regularized cost body is obtained through the ReLU activation layer and the upsampling convolution layer. The regularized cost volume is passed through the regression layer and the activation layer to obtain the pyramid probability volume 6. The large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle according to claim 2, characterized in that: In (2g), the depth estimation map And image edge detection figure E img As the input of the edge-enhanced depth refinement module, the refined depth estimation map D is obtained. refine , which is implemented as follows: The depth estimation map And image edge detection figure E img As the input of the edge enhancement deep refinement module, it passes through three layers of convolutional layers with batch normalization and ReLU activation to extract deep geometric edge features. Deep geometric edge features After an edge detection layer, a Sigmoid activation layer is used to obtain the geometric edge detection map E. geo , and the image edge detection graph E img The geometric-visual fusion edge map E is obtained by OR operation, and the deep geometric edge features are After Tanh activation, the mean of the two Gaussian distributions in the depth dimension is obtained. After Sigmoid activation and truncation, the standard deviation of the two Gaussian distributions in the depth dimension is obtained. Gaussian distribution discrimination and mean residual refinement are performed in the part judged as the edge in the geometric-visual fusion edge map E to obtain the refined depth estimation map D. refine .
7. The large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle according to claim 1, characterized in that: The step (3) is specifically: (3a) The existing SmoothL1Loss function is set as the loss function of the absolute depth estimation task in the adaptive multi-view aggregation matching model with lightweight edge enhancement, and L k express; (3b) The existing SmoothL1Loss function is set as the loss function of the multi-view aggregation weighted task in the lightweight edge-enhanced adaptive multi-view aggregation matching model, and L s express; (3c) The existing SmoothL1Loss function is set as the loss function of the edge region perception task in the adaptive multi-view aggregation matching model with lightweight edge enhancement, and L edge express; (3d) The loss function L of the absolute depth estimation task k , the loss function L for the multi-view aggregation weighted task s , the loss function L of the edge area perception task edge , the loss function L of the adaptive multi-view aggregation matching model with lightweight edge enhancement is set as: L=L k +L s +L edge .
8. The large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle according to claim 7, characterized in that: In (3a), the existing SmoothL1Loss function is set as the loss function L of the absolute depth estimation task in the adaptive multi-view aggregation matching model with lightweight edge enhancement. k , which is expressed as follows: Among them, L k represents the loss of the kth stage, D k (p) represents the depth estimation value of p pixels in the kth stage, D gt (p) represents the true depth value at p pixels, l(x,y) represents the smooth L1 square loss function, x and y represent the depth estimation value and the true depth value respectively; In (3b), the existing SmoothL1Loss function is set as the loss function L of the multi-view aggregation weighted task in the adaptive multi-view aggregation matching model with lightweight edge enhancement. s , which is expressed as follows: Among them, L s Denotes the loss between the depth estimate and the ground-truth of the reference view when s is used as the source view in constructing the paired view weight module, D s (p) represents the depth estimation value at pixel p when s is used as the source view, D gt (p) represents the true depth value at p pixels, l(x,y) represents the smooth L1 square loss function, x and y represent the depth estimation value and the true depth value respectively; In (3c), the existing SmoothL1Loss function is set as the loss function L of the edge region perception task in the adaptive multi-view aggregation matching model with lightweight edge enhancement. edge , which is expressed as follows: Among them, L edge represents edge loss, E(p) represents the probability estimate of pixel p being judged as an edge, and E gt (p) represents the true probability value of p pixels being judged as edges, l(x,y) represents the smooth L1 square loss function, x and y represent the estimated value of edge map detection probability and the true value of edge map probability respectively.
9. The large-scale multi-view stereo matching method based on edge enhancement and adaptive viewing angle according to claim 1, characterized in that: The step (4) is specifically: (4a) Set the learning rate to 0.001, the optimizer to use RMSProp, the batch training data size to 1, and the number of training rounds to 16; (4b) The training samples obtained in step (1) are input into the lightweight edge-enhanced adaptive multi-view aggregation matching model, and the optimizer RMSProp is used to optimize the overall loss function L in (3d), and the weight parameters are updated. When the number of training rounds reaches 16, the trained lightweight edge-enhanced adaptive multi-view aggregation matching model is obtained.
Citation Information
Patent Citations
A SFM sparse reconstruction method, system and application
CN113052880B
Steel structure deformation monitoring processing method and system based on BIM
CN117808964A
Unmanned aerial vehicle image scene semantic segmentation method based on depth prior
CN118314347A
3D Reconstruction Without 3D Convolutions
US20230360241A1
Cited By
Transparent object reconstruction and new view angle synthesis method based on 3DGS
CN119579796A
Multi-view aerial image depth map generation and earth surface three-dimensional reconstruction method
CN121304961A
