Multi-view three-dimensional reconstruction method based on multi-scale structure perception fusion and confidence weighted optimization

Abstract

The invention discloses a multi-view three-dimensional reconstruction method based on multi-scale structure perception fusion and confidence weighted optimization, and the method specifically comprises the following steps: 1, obtaining a multi-view data set, and dividing the data set into a training set and a test set; 2, constructing a depth estimation network, wherein the depth estimation network sequentially comprises a multi-scale feature extraction module, a structure perception and attention guidance feature fusion module, a confidence coefficient weighted feature enhancement module and a multi-stage depth estimation and refinement module; step 3, training the depth estimation network by using the training set in the step 1 to obtain a trained depth estimation model; and step 4, inputting the test set constructed in the step 1 into the model trained in the step 3 to obtain a predicted depth map of each reference view. The method solves the problems that an existing reconstruction method is insufficient in detail reduction in a weak texture region and fuzzy in boundary in a shielding and illumination variation region.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, image processing and 3D reconstruction technology, and relates to a multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence weighted optimization. Background Technology

[0002] With the rapid development of computer vision, remote sensing mapping, and UAV aerial photography technologies, 3D reconstruction technology has been widely applied in urban modeling, cultural relic protection, natural environment monitoring, and disaster assessment. This technology extracts geometric and texture information from images acquired from different perspectives to restore the spatial structure and surface features of a scene, thereby generating dense point clouds and high-precision 3D models. Compared to LiDAR systems that rely on dedicated ranging hardware, image-based 3D reconstruction offers advantages such as low cost, strong adaptability, and flexible deployment, making it particularly suitable for large-scale, rapid modeling tasks in complex environments captured by UAV aerial photography.

[0003] Currently, deep learning-based multi-view stereo (MVS) reconstruction has become a mainstream research and application direction. These methods typically utilize end-to-end neural network structures to perform feature extraction, depth estimation, and dense point cloud generation, with representative implementations such as MVSNet, R-MVSNet, and PatchMatchNet. While these methods offer improvements in data processing speed and automated system integration compared to traditional algorithms, the generated 3D models still suffer from issues such as missing details, blurred boundaries, and surface discontinuities. Specifically, in areas with weak texture, insufficient texture information makes it difficult to distinguish image features, leading to deviations in pixel matching results. This results in insufficient model detail reproduction, often causing local blurring and missing details, thus reducing overall modeling accuracy. In areas with occlusion or lighting variations, differences in visibility between different viewpoints cause noise, holes, or misalignment in the point cloud, affecting the structural consistency and geometric accuracy of the 3D model in complex scenes. Therefore, although existing methods have certain advantages in terms of efficiency and resource utilization, they are still difficult to guarantee high accuracy, detail integrity and structural stability in complex aerial photography scenes with missing textures and severe occlusion. There is an urgent need for an improved method with higher reconstruction quality and stronger boundary clarity. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-view stereo reconstruction method based on multi-scale structure perception fusion and confidence weighted optimization, which solves the problems of insufficient detail restoration in weak texture areas and blurred boundaries in occluded and lighting-changing areas in existing reconstruction methods.

[0005] The technical solution adopted in this invention is a multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-weighted optimization, which specifically includes the following steps: Step 1: Obtain the multi-view dataset and divide it into a training set and a test set; Step 2: Construct a deep estimation network, which includes a multi-scale feature extraction module, a structure-aware and attention-guided feature fusion module, a confidence-weighted feature enhancement module, and a multi-stage depth estimation and refinement module. Step 3: Use the training set from Step 1 to train the depth estimation network to obtain the trained depth estimation model. Step 4: Input the test set constructed in Step 1 into the model trained in Step 3 to obtain the predicted depth map of each reference view.

[0006] The invention is further characterized by: The specific process of step 2 is as follows: Step 2.1: Input the image into the multi-scale feature extraction module for processing to obtain features at different resolutions; Step 2.2: Input the features of different resolutions obtained in Step 2.1 into the structure-aware and attention-guided feature fusion module to obtain fused features at three scales; Step 2.3: Input the fused features of the three scales obtained in Step 2.2 into the confidence-weighted feature enhancement module for enhancement, and output the enhanced features.

[0007] The specific process of step 2.1 is as follows: Let the input image be denoted as H and W represent the height and width of the image, respectively. Preliminary layer-by-layer feature extraction is performed sequentially through convolutional layers, batch normalization, and the ReLU activation function, resulting in feature maps at different resolutions, which are the first-scale features. Second-scale features and third-scale features These correspond to feature representations at 1 / 2, 1 / 4, and 1 / 8 spatial resolutions, respectively. (1) (2) (3) in, This represents the convolution operation. This indicates a batch normalization operation. Represents the linear rectification activation function; Indicates shallow features; Indicates mid-level features; It represents deep features.

[0008] The specific process of step 2.2 is as follows: Step 2.2.1, for Interpolation and upsampling are performed to obtain the upsampled high-level features x. Perform a 1×1 convolution to align the channels and obtain the residual feature r. Calculate the sum of the two features, as shown in formula (4): (4) Step 2.2.2: The x and r obtained in formula (4) are used as inputs to the attention feature fusion module, and passed through two attention paths in parallel: Local attention branch: using 1×1 pointwise convolution and batch normalization, ReLU nonlinear mapping to model the inter-channel dependency relationship; Global attention branch: first extracting the global context through global average pooling, and then generating channel-level weights through pointwise convolution; the outputs of the two are added together and activated by Sigmoid to obtain the fused weights. The calculation process is shown in formula (5), and the final output fused feature is defined by formula (6): (5) (6) in, This represents the local attention channel response. Indicates the global attention channel response; It is a channel-wise adaptive weight, generated jointly by two branches: local attention and global attention. It is the Sigmoid activation function; It is element-wise multiplication; Step 2.2.2: Combine the features from step 2.2.1. The input is fed into the structure-aware enhancement module, where feature enhancement is performed through deformable convolutional layers. The output of a single deformable convolutional layer is defined as follows: (7) in, The value of the output feature map at position p; k is the index of a single sampling point in the convolution kernel; K is the total number of sampling points in the convolution kernel; The convolution weights are the values ​​corresponding to the k-th sampling point; Output feature location coordinates; The fixed offset of the k-th sampling point of the convolution kernel; Let be the offset learned by the network for the k-th sampling point. This is a dynamic weight mask constrained by the Sigmoid function.

[0009] In step 2.2.2, the structure-aware enhancement module consists of three deformable convolutional layers according to the above formula (7) and normalization and nonlinear activation sequentially stacked, specifically including: (1) a 1×1 convolution for channel adjustment; (2) a first deformable convolutional layer followed by BatchNorm and ReLU activation; (3) a second deformable convolutional layer followed by BatchNorm and ReLU activation; (4) a third deformable convolutional layer; the above steps (1) to (4) are combined as a whole operator, denoted as The stepwise fusion and enhancement process is represented by the following formulas (8) to (10). Finally, after processing by the SAFFM module, the fusion feature set of three scales is output as shown in the following formula (11): (8) (9) (10) (11) The specific process of step 2.3 is as follows: Step 2.3.1, will First, a pixel-level confidence weight map is obtained by performing 3×3 convolution, batch normalization, and ReLU activation, followed by 1×1 convolution and the Sigmoid function. : (12) Step 2.3.2, will The channel dimensions are adjusted using 1×1 convolutions, and then the data is passed through two layers of residual blocks and a channel-space joint attention module to obtain refined features. : (13) A single residual block is then defined as: (14) For the channel-space joint attention module, let Y be the input of CBAM, and the calculation process is shown in formula (15): (15) in, The channel attention weights are calculated as shown in formula (16): (16) in, This indicates adaptive global average pooling. For the Sigmoid function; The spatial attention weights are calculated as shown in formula (17): (17) in, This indicates that average pooling is performed on feature Z along the channel dimension, and the average response at each spatial location is output. This indicates that the feature Z takes the maximum value in the channel dimension; This means that the two are spliced ​​together on the channel dimension to form a feature map with two channels; Step 2.3.3: Calculate the confidence weight map obtained in Step 2.3.1. And the refined features obtained in step 2.3.2 Adaptive guided fusion is performed, and the confidence weight map is used. As a weighting factor, for refined features With original features By merging, an enhanced output can be obtained. : (18) in, This indicates element-wise multiplication, where i represents the index of the three scales; Step 2.3.4: Perform steps 2.3.1 to 2.3.3 above on the input features at the three scales respectively to obtain the enhanced feature set: (19) in, This is the enhanced feature set.

[0010] The specific process of step 3 is as follows: Step 3.1: Construct a confidence-weighted mask for each stage, which involves a total of N=4 stages of depth estimation. The predicted depth map output for each stage is as follows: (20) Wherein, the actual depth corresponding to the j-th stage is The corresponding confidence weighting plot is as follows: , representing the depth prediction confidence level of each pixel in the view at the i-th scale (high or low); construct a confidence-weighted mask for each stage. The definition is as follows: (twenty one) in, These are pixel coordinates; The confidence threshold; This represents the magnification factor for the low-confidence region. This is a valid pixel mask used to eliminate regions with no depth values; Confidence weighting plot generated for the CAFEM module; When the confidence value of a pixel is low, that is When the value is above the threshold τ, the weight is amplified to λ; Step 3.2: Construct a weighted loss function using the confidence-weighted mask obtained in Step 3.1. Within each stage, calculate the pixel-by-pixel error between the predicted depth and the true depth using a smoothed L1 loss, and then... After weighting, the loss in stage j is defined as: (twenty two) Step 3.3: Perform multi-stage joint optimization on the weighted losses obtained in Step 3.2 to obtain the overall optimization objective function: (twenty three) in, Let be the overall optimization objective function.

[0011] In step 3.2, the SmoothL1 loss function is defined as follows: (twenty four) in, Indicates the predicted depth value , Represents the true depth value .

[0012] The beneficial effects of this invention are as follows: 1. This invention proposes a feature fusion module based on multi-scale structure perception. By introducing deformable convolution and attention control mechanisms in the feature extraction stage, it achieves effective fusion of multi-scale information, which fully preserves the structural details of weak texture areas and makes the morphological characterization of complex boundary areas clearer, thereby improving the overall accuracy of 3D modeling.

[0013] 2. This invention proposes a confidence-weighted feature enhancement module that can automatically identify unstable prediction regions during training, and give stronger attention and optimization to occlusion and lighting change scenarios, thereby improving the prediction stability and boundary restoration ability of the model in complex scenarios.

[0014] 3. This invention designs a confidence-based weighted loss optimization mechanism. By using a confidence weight map to apply differentiated weights to different regions, regions with weak textures and large viewpoint differences receive stronger supervision, thereby improving the accuracy and consistency of the overall reconstruction effect.

[0015] 4. Compared with existing methods, this invention achieves high-precision, clear-boundary, and continuous 3D reconstruction results in complex scenes captured by UAV aerial photography while maintaining the lightweight network structure and high inference efficiency. It has good engineering application value and promotion prospects. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall process of the multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence weighted optimization of the present invention; Figure 2 This is a schematic diagram of the structure perception and attention-guided feature fusion module (SAFFM) in the multi-view stereo reconstruction method based on multi-scale structure perception fusion and confidence weighted optimization of the present invention; Figure 3 This is a schematic diagram of the confidence-weighted feature enhancement module (CAFEM) in the multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-weighted optimization of the present invention; Figure 4 This is a schematic diagram of the calculation process of the confidence-based weighted loss function (CAWLoss) in the multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-based weighted optimization of the present invention.

[0017] Figure 5 This is a stage-by-stage depth result image of the depth estimation network model in the multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence weighted optimization of the present invention; Figure 6 These are comparison images of point cloud reconstruction effects in different application scenarios; Figures 7(a) to 7(d) are the reconstruction point cloud effects of the multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence weighted optimization in complex UAV aerial photography scenarios. Detailed Implementation

[0018] The following detailed description is provided in conjunction with specific implementation methods.

[0019] Example 1 like Figure 1 As shown, the present invention provides a multi-view stereo reconstruction method based on multi-scale structure-aware fusion and confidence-weighted optimization, comprising the following steps: Step 1: Obtain multi-view images and corresponding camera pose parameters and matching pair information contained in the publicly available multi-view dataset DTU, construct training and testing sets to provide input data for subsequent depth estimation.

[0020] Step 2: Construct a depth estimation network. Input the data obtained in Step 1 into the depth estimation network, selecting one image as the reference view and the rest as source views. The network sequentially includes a multi-scale feature extraction module, a structure-aware and attention-guided feature fusion module, a confidence-weighted feature enhancement module, and a multi-stage depth estimation and thinning module to obtain the depth map results for each reference view.

[0021] Step 3: Train the depth estimation network using the DTU training set from Step 1. Optimize the network parameters using a confidence-weighted loss constraint. Continuously update the network parameters through backpropagation until the model converges, thus obtaining the trained depth estimation model.

[0022] Step 4: Input the DTU test set constructed in Step 1 and the BlendedMVS test set with drone aerial photography scenes into the model trained in Step 3 to obtain the predicted depth maps for each reference view. Then, fuse the multi-view depth results, use photometric consistency and geometric consistency constraints to filter out invalid depths, and project the valid depth information back into the 3D space. Combine the corresponding color information to generate a dense point cloud to obtain the complete 3D reconstruction result.

[0023] Example 2 Based on Example 1, each step is implemented according to the following process: The specific process of step 2 is as follows: Step 2.1: The input image is processed by the multi-scale feature extraction module to obtain features at different resolutions. Let the input image be denoted as... H and W represent the height and width of the image, respectively. Preliminary layer-by-layer feature extraction is performed sequentially through convolutional layers, batch normalization, and the ReLU activation function to obtain feature maps at different resolutions, including first-scale features. Second-scale features and third-scale features These correspond to feature representations at 1 / 2, 1 / 4, and 1 / 8 spatial resolutions, respectively. (1) (2) (3) in, This represents the convolution operation. This indicates a batch normalization operation. Represents the linear rectification activation function; Indicates shallow features; Indicates mid-level features; It represents deep features.

[0024] Step 2.2: Input the three scale features from Step 2.1 into the Structure-Aware and Attention-guided Feature Fusion Module (SAFFM). This module includes an Attentional Feature Fusion (AFF) module and a Structure-Aware Enhancement module, which are used to fuse feature information across different scales and enhance structural detail responses.

[0025] Step 2.2.1: In the process of cross-scale feature fusion, this method, based on the idea of ​​the Iterative Attentional Feature Fusion (iAFF) module proposed in 2023, designs an improved version of AFF to achieve this adaptive selection. Unlike the iterative feature update method of the original iAFF, the improved AFF uses a non-iterative feature fusion approach and simplifies and optimizes the weighting process of local and global attention. Specifically, as follows... Figure 2 As shown, given the input features initially extracted in step 2.1 (High-level semantic features after upsampling) and residual features (Shallow lateral connection characteristics), for Interpolation and upsampling are performed to obtain the upsampled high-level features x. Perform a 1×1 convolution to align the channels and obtain the residual feature r. Calculate the sum of the two features. The calculation process is shown in formula (4).

[0026] (4) like Figure 2 As shown in the AFF module below, x and r obtained from formula (4) are used as inputs to the attention feature fusion module, passing through two attention paths in parallel: Local Attention: 1×1 pointwise convolution, batch normalization, and ReLU nonlinear mapping are used to model the inter-channel dependencies, highlighting local differences. Global Attention: Global average pooling is used to extract the global context, and then pointwise convolution is used to generate channel-level weights, emphasizing global consistency. The outputs of the two are added together and activated by Sigmoid to obtain the fused weights. The calculation process is shown in formula (5). The final fused feature output is defined by formula (6): (5) (6) in, This represents the local attention channel response. Indicates the global attention channel response; It is a channel-wise adaptive weight, generated jointly by two branches: local attention and global attention. It is the Sigmoid activation function; It is element-wise multiplication.

[0027] Step 2.2.2: Combine the features from step 2.2.1. The input is further processed by the structure-aware enhancement module in step 2.2. This module introduces a deformable convolutional layer (DCN) to enhance the model's ability to model complex boundaries and weakly textured regions. The output of a single-layer DCN is defined as: (7) in, The value of the output feature map at position p; k is the index of a single sampling point in the convolution kernel; K is the total number of sampling points in the convolution kernel; The convolution weights are the values ​​corresponding to the k-th sampling point; Output feature location coordinates; The fixed offset of the k-th sampling point of the convolution kernel; Let be the offset learned by the network for the k-th sampling point. This is a dynamic weight mask constrained by the Sigmoid function.

[0028] The structure-aware enhancement module consists of three deformable convolutional layers (DCN) as described in formula (7) above, stacked with normalized and nonlinear activation sequences, specifically including: (1) A 1×1 convolution is used for channel adjustment; (2) The first layer DCN is connected to BatchNorm and ReLU activation; (3) The second layer DCN is connected to BatchNorm and ReLU activation; (4) Third layer DCN.

[0029] Treating the above combination of operations as a single operator, denoted as... Therefore, the structure-aware enhancement module can adaptively adjust the sampling point position during convolution and assign greater weight to key regions, thereby enhancing the model's ability to perceive complex boundaries and weakly textured regions.

[0030] This embodiment employs the aforementioned attention fusion and [other techniques] at the deep, intermediate, and shallow scales. The enhanced combination method achieves feature fusion and structural modeling step by step.

[0031] (8) (9) (10) Formulas (8) to (10) represent the stepwise fusion and enhancement process. Finally, after processing by the SAFFM module, step 2.2 outputs the fusion feature set at three scales: (11) Among them, the features at each scale not only preserve local details but also take into account global consistency, which can effectively alleviate the problems of insufficient feature representation in weak texture regions and unstable features in complex boundary regions, and provide more robust input for subsequent confidence-weighted feature enhancement modules and multi-stage depth estimation and refinement modules.

[0032] Step 2.3: The fused features from the three scales obtained in Step 2.2 are input into the Confidence-Aware Feature Enhancement Module (CAFEM) for further enhancement. This module uses confidence information to weight and modulate the features in the deep estimation network, thereby enhancing the model's response in occluded, weakly textured, and illumination-variable regions. The CAFEM structure is as follows: Figure 3 As shown, it consists of two paths: a confidence prediction branch and a refinement branch, as well as a final adaptive guided fusion mechanism. The specific processing steps are as follows: Step 2.3.1, as follows Figure 3 As shown in the medium confidence prediction branch, the features of formula (11) Taking the example as the input to this module, it first undergoes 3×3 convolution with batch normalization and ReLU activation, and then passes through 1×1 convolution with the Sigmoid function to obtain a pixel-level confidence weight map. : (12) This confidence weight map is used to measure the feature confidence at each location; the larger the value, the lower the confidence in that area.

[0033] Step 2.3.2, still using the third-scale input features of formula (11) For example, the data is simultaneously fed into the refinement branch. First, the channel dimensions are adjusted through a 1×1 convolution, and then the data passes through two layers of residual blocks and a channel-space joint attention module (CBAM) to obtain refined features. : (13) The calculation definition of a residual block is shown in formula (14). Let X be the input feature of the residual block, then a single residual block is defined as: (14) For the channel-space joint attention module, let Y be the input of CBAM, and the calculation process is shown in formula (15): (15) in, The channel attention weights are broadcast along the corresponding dimensions to the input feature shape. The calculation process is shown in formula (16): (16) in, This indicates adaptive global average pooling, used to extract the global response for each channel; This is the Sigmoid function. The spatial attention weights are broadcast along the corresponding dimensions to the shape of the input features. The calculation process is shown in formula (17): (17) in, This indicates that average pooling is performed on feature Z along the channel dimension. This indicates that the feature Z takes the maximum value in the channel dimension; This indicates that the two will be spliced ​​together on the channel dimension.

[0034] In this invention, the channel-space joint attention module with refined branches highlights significant channels through the channel attention mechanism and then combines it with the spatial attention mechanism to locate important regions, thereby improving the ability of features to distinguish complex structures and weak texture regions.

[0035] Step 2.3.3: Calculate the confidence weight map obtained in Step 2.3.1. And the refined features obtained in step 2.3.2 Perform adaptive guided fusion. Use the confidence weight map... As a weighting factor, for refined features With original features By merging, an enhanced output can be obtained. : (18) in, This indicates element-wise multiplication, where i represents the index of the three scales; the third scale is used as an example here. This mechanism ensures that in regions with low confidence, the model relies more on refined features, while in regions with high confidence, the original features are retained, achieving a dynamic balance.

[0036] The above processing is applied to the input features at the three scales respectively (the above process takes the third scale features as an example) to obtain the enhanced feature set: (19) In the depth estimation network proposed in Embodiment 2 of this invention, in addition to the above-mentioned multi-scale feature extraction module, structure-aware and attention-guided feature fusion module, and confidence-weighted feature enhancement module, a multi-stage depth estimation and refinement module is also included.

[0037] The multi-stage depth estimation and refinement module adopts the multi-stage progressive optimization strategy in the existing PatchMatchNet framework, sequentially including three sub-processes: depth initialization, propagation, and evaluation. Through multi-scale iterative refinement from coarse to fine, it achieves continuous optimization and accuracy improvement of the depth map. This module is used to refine the depth prediction results at different resolution stages to ensure that the output depth map has the same resolution as the reference image. Since this part belongs to existing well-known technology, this invention does not modify its structure and algorithm principles, and therefore will not elaborate further in this specification.

[0038] Example 3 Step 3, the confidence-aware weighted Smooth L1 Loss (CAWLoss) optimization, includes three steps: constructing a confidence-weighted mask, building a weighted loss function, and multi-stage joint optimization. The specific process is as follows: In the training process of the deep estimation network, in order to improve the model's learning ability in regions with weak texture, occlusion, and complex boundaries, this invention is based on the confidence weight map obtained in step S2.3.1. The supervision signals for multi-stage depth estimation are weighted and optimized. For example... Figure 4 As shown, its core idea is to assign stronger supervision weights to regions with low confidence and maintain normal weights to regions with high confidence, so that the model can adaptively focus on features in difficult regions during training.

[0039] Example 4 The specific process of step 3 is as follows: Step 3.1: Construct a confidence-weighted mask for each stage. The network consists of N=4 stages of depth estimation, with each stage outputting a predicted depth map: (20) The actual depth corresponding to stage j is The corresponding confidence weighting plot is as follows: , which represents the depth prediction confidence of each pixel in the view at the i-th scale, where a larger value indicates a more unstable prediction.

[0040] To reflect the confidence differences among pixels, this invention constructs a confidence-weighted mask for each stage. The definition is as follows: (twenty one) in, These are pixel coordinates; The confidence threshold; This represents the magnification factor for the low-confidence region. This is a valid pixel mask used to eliminate regions with no depth values; Confidence weighting plot generated for the CAFEM module.

[0041] When the confidence value of a pixel is low, that is When the weight exceeds the threshold τ, the weight is amplified to λ, thereby enhancing the supervision intensity of this region during the training process.

[0042] Step 3.2: Construct a weighted loss function using the confidence-weighted mask obtained in Step 3.1. Within each stage, calculate the pixel-by-pixel error between the predicted depth and the true depth using Smooth L1 Loss, and based on... Weighting is applied. The loss in stage j is defined as: (twenty two) The SmoothL1 loss function is defined as follows: (twenty three) in, Indicates the predicted depth value , Represents the true depth value The SmoothL1 loss measures the pixel-level difference between the two, maintaining smoothness when the error is small and suppressing the influence of outliers when the error is large, thereby improving the stability of the overall depth estimation.

[0043] Step 3.3: Perform multi-stage joint optimization on the weighted losses obtained in Step 3.2 to obtain the overall optimization objective function: (twenty four) This function imposes overall constraints on the multi-stage prediction results, enabling the network to optimize depth estimation accuracy layer by layer at different scales.

[0044] By applying higher loss weights to low-confidence regions, the model can learn more fully the deep features of weak textures, occlusions, and complex boundary regions during training; at the same time, using normal weights for high-confidence regions can effectively avoid overfitting.

[0045] Therefore, the confidence-weighted loss function of this invention significantly improves the model's ability to restore details and its reconstruction robustness while ensuring overall convergence stability.

[0046] Example 5 To verify the effectiveness and superiority of the method of this invention, multiple sets of experiments were conducted on the proposed multi-view stereo reconstruction method based on multi-scale structure-aware fusion and confidence-weighted optimization. The experimental results are as follows: Figure 5 As shown in Figures 7(a) to (d).

[0047] like Figure 5 The figure shows the stage-by-stage depth estimation results of the depth estimation network in the scan114 scene on the DTU dataset test set of the method described in this invention. This figure illustrates the multi-stage depth estimation process from low resolution to high resolution: from stage 3 to stage 0, the spatial resolution of the depth map gradually increases (from 64×80, 128×160, 256×320 to 512×640), the boundaries and details of the predicted results become clearer, and the depth distribution becomes more continuous and smooth. This result fully demonstrates that the Structure-Aware and Attention-Guided Feature Fusion Module (SAFFM) proposed in this invention can fully fuse local structural information and global semantic features across different scales, refining the depth estimation results stage by stage. Especially in areas with weak texture and detail, the model can effectively extract structural boundary information, significantly improving the continuity of depth prediction and edge reconstruction capabilities. Meanwhile, due to the introduction of confidence-weighted feature enhancement module (CAFEM) and confidence-based weighted loss function (CAWLoss) optimization mechanism in the network, the prediction results still maintain high stability in low confidence areas with occlusion and illumination changes, further improving the overall modeling accuracy and reliability.

[0048] like Figure 6 The image shows a comparison of the point cloud reconstruction results of the method of this invention under different scenarios (scan77, sacn75, sacn9) on the DTU dataset test set. The left column is the input image, the second column from the left is the reconstruction reference ground truth, the third column from the left is the result of the baseline method (PatchMatchNet), and the right column is the reconstruction result of this invention. Overall, under various complex objects and different lighting conditions, the point cloud generated by this invention is more continuous and complete in terms of object edges, texture transitions, and weakly textured surfaces, with a significant reduction in noise points. Specifically, in Figure 6In the first row, the edge region of the reconstructed object's handle is shown. The point cloud generated by this method has clear boundaries, significantly reduced noise, and edge contours that better match the actual shape. In the second row, the curved surface region of the red pepper, the point cloud generated by this method is tightly connected and has a smooth surface, without the holes and misalignments found in PatchMatchNet. On the sloping surface of the container in the first row and the weakly textured surface like the roof of the house in the third row, the point cloud generated by this method maintains good continuity, and structural details are more realistically reproduced. Compared to the traditional PatchMatchNet method, this invention, through structure-aware fusion and confidence-weighted optimization, not only maintains the accuracy of the overall geometric shape but also achieves higher boundary consistency and detail accuracy under varying lighting conditions and occlusion scenarios.

[0049] Example 6 Figure 7(a) shows the 3D reconstructed point cloud effect of the method of the present invention in the complex UAV aerial photography scene 5b7a3890fc8fcf6781e2593a in the BlendedMVS test set of the public dataset; Figures 7(b), 7(c), and 7(d) are magnified views of the area within the red box in Figure 7(a) from top to bottom and left to right, respectively. As can be seen from Figure 7(a), the present invention can maintain high geometric consistency and structural continuity in large-scale aerial photography scenes. The overall point cloud is densely distributed with clear outlines, and the roofs, walls, and ground boundaries of buildings are smooth and naturally connected. Roads and ground structures are all completely presented without obvious discontinuities or misalignments. As can be seen from the magnified results in Figure 7(b), the texture boundaries of the ground area are clear, the planar structure of regular areas such as the sports field remains intact, the point cloud is evenly distributed, and there are no obvious voids or fragmentation phenomena. In the area surrounding the building shown in Figure 7(c), the building facade and ground are smoothly connected, and the geometric structure at the partially occluded areas still maintains continuity. In the circular feature area shown in Figure 7(d), the curved surface outline is complete, the edge transition is natural, and the details of complex-shaped areas are accurately restored. In summary, the method of this invention can effectively overcome the reconstruction instability problems caused by weak textures, occlusion, and changes in lighting in complex scenes captured by UAV aerial photography, achieving high-precision 3D structure restoration. Compared with existing methods, the point cloud generated by this invention performs better in terms of detail integrity and boundary clarity, indicating that this method has good reliability and application value in complex real-world reconstruction.

[0050] In summary, the experimental results fully verify that the key problems raised in the background technology, namely the lack of details in weak texture areas, unclear reconstruction boundaries due to occlusion and lighting differences leading to unstable predictions, and insufficient reconstruction accuracy in complex scenes, have all been effectively improved, thus effectively enhancing the stability and accuracy of 3D reconstruction results in complex scenes.

[0051] This invention constructs a multi-view stereo reconstruction technology system for complex scenes captured by UAV aerial photography. It includes a multi-scale feature extraction module, a structure-aware and attention-guided feature fusion module, a confidence-weighted feature enhancement module, and a confidence-weighted loss optimization mechanism. Through the synergistic effect of these modules, a complete reconstruction process is formed, from image feature extraction and depth estimation to 3D point cloud generation. This system maintains detail integrity, clear boundaries, and surface continuity in areas with weak texture, occlusion, and varying illumination, improving the reconstruction accuracy and stability of the 3D model.

Claims

1. A multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-weighted optimization, characterized in that, Specifically, the steps include the following: Step 1: Obtain the multi-view dataset and divide it into a training set and a test set; Step 2: Construct a deep estimation network, which includes a multi-scale feature extraction module, a structure-aware and attention-guided feature fusion module, a confidence-weighted feature enhancement module, and a multi-stage depth estimation and refinement module. Step 3: Use the training set from Step 1 to train the depth estimation network to obtain the trained depth estimation model. Step 4: Input the test set constructed in Step 1 into the model trained in Step 3 to obtain the predicted depth map of each reference view.

2. The multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-weighted optimization according to claim 1, characterized in that, The specific process of step 2 is as follows: Step 2.1: Input the image into the multi-scale feature extraction module for processing to obtain features at different resolutions; Step 2.2: Input the features of different resolutions obtained in Step 2.1 into the structure-aware and attention-guided feature fusion module to obtain fused features at three scales; Step 2.3: Input the fused features of the three scales obtained in Step 2.2 into the confidence-weighted feature enhancement module for enhancement, and output the enhanced features.

3. The multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-weighted optimization according to claim 2, characterized in that, The specific process of step 2.1 is as follows: Let the input image be denoted as H and W represent the height and width of the image, respectively. Preliminary layer-by-layer feature extraction is performed sequentially through convolutional layers, batch normalization, and the ReLU activation function, resulting in feature maps at different resolutions, which are the first-scale features. Second-scale features and third-scale features These correspond to feature representations at 1 / 2, 1 / 4, and 1 / 8 spatial resolutions, respectively. （1） （2） （3） in, This represents the convolution operation. This indicates a batch normalization operation. Represents the linear rectification activation function; Indicates shallow features; Indicates mid-level features; It represents deep features.

4. The multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-weighted optimization according to claim 3, characterized in that, The specific process of step 2.2 is as follows: Step 2.2.1, for Interpolation and upsampling are performed to obtain the upsampled high-level features x. Perform a 1×1 convolution to align the channels and obtain the residual feature r. Calculate the sum of the two features, as shown in formula (4): （4） Step 2.2.2, take the x and r obtained in formula (4) as the input of the attention feature fusion module and pass through two attention paths in parallel: Local attention branch: use 1×1 pointwise convolution and batch normalization, ReLU nonlinear mapping to model the inter-channel dependency relationship; Global attention Branch: First, extract the global context through global average pooling, and then generate channel-level weights through pointwise convolution; The two outputs are added together and then activated by a Sigmoid function to obtain the fusion weights. The calculation process is shown in formula (5), and the final output fused feature is defined by formula (6): （5） （6） in, This represents the local attention channel response. Indicates the global attention channel response; It is a channel-wise adaptive weight, generated jointly by two branches: local attention and global attention. It is the Sigmoid activation function; It is element-wise multiplication; Step 2.2.2: Combine the features from step 2.2.

1. The input is fed into the structure-aware enhancement module, where feature enhancement is performed through deformable convolutional layers. The output of a single deformable convolutional layer is defined as follows: （7） in, The value of the output feature map at position p; k is the index of a single sampling point in the convolution kernel; K is the total number of sampling points in the convolution kernel; The convolution weights are the values ​​corresponding to the k-th sampling point; Output feature location coordinates; The fixed offset of the k-th sampling point of the convolution kernel; Let be the offset learned by the network for the k-th sampling point. This is a dynamic weight mask constrained by the Sigmoid function.

5. The multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-weighted optimization according to claim 4, characterized in that, In step 2.2.2, the structure-aware enhancement module consists of three deformable convolutional layers of the above formula (7) and normalization and nonlinear activation stacked in sequence, specifically including: (1) a 1×1 convolution for channel adjustment; (2) a first deformable convolutional layer connected to BatchNorm and ReLU activation; (3) a second deformable convolutional layer connected to BatchNorm and ReLU activation; and (4) a third deformable convolutional layer. Treat the above steps (1) to (4) as a single operator, denoted as The stepwise fusion and enhancement process is represented by the following formulas (8) to (10). Finally, after processing by the SAFFM module, the fusion feature set of three scales is output as shown in the following formula (11): （8） （9） （10） （11） Where F represents the final output feature after fusion.

6. The multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-weighted optimization according to claim 5, characterized in that, The specific process of step 2.3 is as follows: Step 2.3.1, will First, convolution, batch normalization, and ReLU activation are applied, followed by convolution and the Sigmoid function to obtain a pixel-level confidence weight map. : （12） Step 2.3.2, will The channel dimensions are adjusted by convolution, and then the data is passed through two layers of residual blocks and a channel-space joint attention module to obtain refined features. : （13） The calculation definition of the residual block is shown in formula (14). Let X be the input feature of the residual block, then a single residual block is defined as: （14） For the channel-space joint attention module, let Y be the input of CBAM, and the calculation process is shown in formula (15): （15） in, The channel attention weights are broadcast along the corresponding dimensions to the input feature shape. The calculation process is shown in formula (16): （16） The spatial attention weights are calculated as shown in formula (17): （17） in, This indicates that average pooling is performed on feature Z along the channel dimension, and the average response at each spatial location is output. This indicates that the feature Z takes the maximum value in the channel dimension; This means that the two are spliced ​​together on the channel dimension to form a feature map with two channels; Step 2.3.3: Calculate the confidence weight map obtained in Step 2.3.

1. And the refined features obtained in step 2.3.2 Adaptive guided fusion is performed, and the confidence weight map is used. As a weighting factor, for refined features With original features By merging, an enhanced output can be obtained. : （18） in, This indicates element-wise multiplication, where i represents the index of the three scales; Step 2.3.4: Perform steps 2.3.1 to 2.3.3 above on the input features at the three scales respectively to obtain the enhanced feature set: （19） in, This is the enhanced feature set.

7. The multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-weighted optimization according to claim 6, characterized in that, The specific process of step 3 is as follows: Step 3.1: Construct a confidence-weighted mask for each stage, which involves a total of N=4 stages of depth estimation. The predicted depth map output for each stage is as follows: （20） Wherein, the actual depth corresponding to the j-th stage is The corresponding confidence weighting plot is as follows: , indicating the depth prediction confidence of each pixel in the view at the i-th scale is low or high; Construct a confidence-weighted mask for each stage. The definition is as follows: （21） in, These are pixel coordinates; The confidence threshold; This represents the magnification factor for the low-confidence region. This is a valid pixel mask used to eliminate regions with no depth values; Confidence weighting plot generated for the CAFEM module; When the confidence value of a pixel is low, that is When the value is above the threshold τ, the weight is amplified to λ; Step 3.2: Construct a weighted loss function using the confidence-weighted mask obtained in Step 3.

1. Within each stage, calculate the pixel-by-pixel error between the predicted depth and the true depth using a smoothed L1 loss, and then... After weighting, the loss in stage j is defined as: （22） in, Indicates the predicted depth value. Indicates the actual depth value; Step 3.3: Perform multi-stage joint optimization on the weighted losses obtained in Step 3.2 to obtain the overall optimization objective function: （23） in, Let be the overall optimization objective function.

8. The multi-view stereo reconstruction method based on multi-scale structural perception fusion and confidence-weighted optimization according to claim 7, characterized in that, In step 3.2, the SmoothL1 loss function is defined as follows: （24） in, Indicates the predicted depth value , Represents the true depth value .

Images