MVS three-dimensional reconstruction method fusing frequency characteristics

Abstract

The present application relates to a kind of fusion frequency feature MVS three-dimensional reconstruction method, comprising: constructing MVS dataset;Wherein, the MVS dataset is the indoor MVS dataset containing several scans, each scan has several views under several different illumination conditions;HL-MVSNet model is constructed;Wherein, the HL-MVSNet model includes: feature extraction network, attention module, cost volume generation module;The MVS dataset is used to train the HL-MVSNet model, and MVS three-dimensional reconstruction model is obtained;Using the MVS three-dimensional reconstruction model, fusion frequency feature MVS three-dimensional reconstruction is carried out.The present application realizes the significant improvement in reconstruction accuracy and integrity.

Description

Technical Field

[0001] This invention relates to the field of multi-view stereo matching technology, and in particular to a MVS three-dimensional reconstruction method that integrates frequency features. Background Technology

[0002] Multi-view stereo matching (MVS) aims to reconstruct the 3D structure of a scene from a series of images. Due to its widespread application in autonomous driving, robotics, and virtual reality, MVS has gained increasing attention in recent years. While traditional methods have achieved good results, they still have some limitations, such as low reconstruction completeness due to occlusion, lighting variations, textureless regions, and non-Lambertian surfaces.

[0003] To overcome the aforementioned challenges, learning-based multi-view stereo matching (MVS) methods have been proposed. The first to be proposed is MVSNet, an end-to-end multi-view stereo matching method based on deep learning. MVSNet uses convolutional neural networks (CNNs) to extract features, more effectively capturing high-level image features. In the cost volume regularization stage, MVSNet introduces a variance-based multi-view aggregation method to adapt to input from any view. Furthermore, MVSNet employs a 3D CNN structure to filter the cost volume; however, this structure is memory-intensive, resulting in high costs in practical applications. To address the high memory consumption issue, R-MVSNet was proposed. This network uses gated units (GRUs) from recurrent neural networks instead of 3D CNNs, although efficiency is slightly reduced, it effectively reduces memory consumption. Cascade-MVSNet uses large depth intervals and fewer depth ranges for depth prediction on downsampled low-resolution images, then applies the prediction results to undownsampled high-resolution images, significantly reducing the model's memory requirements. AA-RMVSNet aggregates contextual features across multiple regions with different texture richness through an Intra-view AA module, effectively reducing feature loss. TransMVSNet introduces the transformer into the MVSNet network, effectively aggregating global contextual information between the reference and source images, thus significantly reducing feature loss.

[0004] Despite the great success of learning-based MVS methods, they still face challenges. Through studying the methods mentioned above, we found two main problems with current learning-based MVS methods: (1) Convolution has excellent performance in acquiring local features, but its limited receptive field also affects the perception of contextual information. This makes it difficult to perform robust depth estimation for challenging regions such as poor texture, repetitive patterns, and non-Lambertian surfaces in the MVS task. (2) In addition, although existing learning-based MVS methods have improved the long-range contextual relationships of images by introducing Transformers, thus solving the problem of accurate matching in low-texture, repetitive, specular, and reflective regions to some extent, they ignore the rich frequencies present in natural images. Encoding different frequencies plays a very important role in acquiring different information in images (such as lines, shapes, textures, and colors). This information also plays an irreplaceable role in the accurate matching of low-texture, repetitive, specular, and reflective regions in the MVS task. Summary of the Invention

[0005] The purpose of this invention is to propose a frequency-feature-integrated MVS 3D reconstruction method to address the problems existing in the prior art. This invention designs an end-to-end MVS network, HL-MVSNet. This network utilizes a context enhancement module (CEM) based on dilated convolution to enhance the contextual information of the image, and simultaneously employs a Transformer-based HL-Attention module to acquire information such as lines, shapes, textures, and colors of the image by encoding different frequencies. This enhances the long-range contextual relationships of the image and further solves the problem of accurate matching of low-texture, repetitive, specular, and reflective regions. HL-MVSNet achieves a significant improvement in reconstruction accuracy and completeness.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method for 3D reconstruction using MVS that integrates frequency features, comprising:

[0008] Construct an MVS dataset; wherein the MVS dataset is an indoor MVS dataset containing several scans, each scan having several views under several different lighting conditions;

[0009] Construct an HL-MVSNet model; wherein the HL-MVSNet model includes: a feature extraction network, an attention module, and a cost volume generation module;

[0010] The HL-MVSNet model is trained using the MVS dataset to obtain the MVS 3D reconstruction model;

[0011] Using the aforementioned MVS 3D reconstruction model, MVS 3D reconstruction with fused frequency features is performed.

[0012] Optionally, the feature extraction network includes: a feature extraction module and a context enhancement module;

[0013] The feature extraction module is used to extract three feature maps of different scales from the input image, including a low-resolution output and two high-resolution outputs;

[0014] The context enhancement module is used to fuse context information from two feature maps at two different scales with higher resolution to obtain a feature map after context enhancement processing.

[0015] Optionally, the context enhancement module includes: four dilated convolutional layers with different dilation rates and a 1×1 convolutional layer;

[0016] The outputs of each dilated convolution are concatenated together and passed through a 1×1 convolutional layer to fuse contextual information from different receptive fields; the outputs of the dilated convolutions are then added to the inputs to fuse coarse-grained and fine-grained features.

[0017] Optionally, the attention module is used to perform multi-head attention processing on the feature map extracted by the feature pyramid network of the fusion context enhancement module to obtain the feature map after the attention module has processed.

[0018] The attention module includes: a high-frequency path and a low-frequency path;

[0019] The high-frequency path is used to encode local details of the object;

[0020] The low-frequency path is used to encode the global structure of the object;

[0021] The high-frequency path is assigned (1-α)Nh attention heads, and the low-frequency path is assigned αNh attention heads; where Nh refers to the total number of attention heads.

[0022] Optionally, the high-frequency path captures fine-grained high frequencies through local window self-attention, and the local window is a non-overlapping window segmentation of the image;

[0023] The low-frequency path first performs average pooling on each window to obtain the key vector and value vector of the low-frequency path.

[0024] Optionally, the attention outputs of the high-frequency path and the low-frequency path are concatenated together as the final output of the attention module;

[0025] The attention outputs of both the high-frequency and low-frequency paths are calculated using the scaled dot product attention formula.

[0026] The scaling dot product attention formula is:

[0027]

[0028] Where Q is the query vector, K is the key vector, and V is the value vector, QK T To query the product of the transpose of the vector and the key vector, This is the scaling factor.

[0029] Optionally, the HL-MVSNet model further includes: a transit path module;

[0030] The transmission path module is used to align the output of the low-resolution feature map after passing through the attention module with the channel of the high-resolution feature map of the previous layer through a convolutional neural network, then upsample the output of the low-resolution feature map after passing through the attention module and add it to the high-resolution feature map, and finally fuse the added result through a convolutional neural network; wherein, the low-resolution feature map is the low-resolution output of the feature extraction module, and the high-resolution feature map is the high-resolution output of the feature extraction module.

[0031] Optionally, the cost body generation module is used to perform several homography transformations on the feature map obtained after processing by the feature extraction network and the attention module to generate the cost body;

[0032] The cost body is:

[0033]

[0034] Among them, C (d) (p) represents the cost volume, N represents the number of source and reference views, and d represents the depth. This represents the similarity between the reference view and the source view at pixel p at depth d.

[0035] Optionally, the HL-MVSNet model can be trained end-to-end using the focus loss function on the MVS dataset;

[0036] The focus loss function is:

[0037]

[0038] Where L represents the focus loss function, γ represents the focusing parameter, and P (d) (p) represents the predicted probability of the depth hypothesis d at pixel p. {pv} represents the depth value that is closest to the true value among all hypotheses, {pv} represents the subset of pixels with valid true values, and p represents a pixel.

[0039] The beneficial effects of this invention are as follows:

[0040] This invention proposes HL-MVSNet, a multi-view stereo network based on feature aggregation Transformer. A feature extraction module using fused dilated convolutions adaptively expands the receptive field; then, an Attention module aggregates global long-range context-aware information within and between images, making the aggregated dense features more suitable for reliable feature matching. Furthermore, a cost volume generation module employs visibility cost aggregation to suppress the adverse effects of occluded pixels. Extensive experiments on DTU and Tanks & Temples demonstrate that, compared to existing learning-based methods, the proposed method outperforms other methods and exhibits excellent generalization ability. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the HL-MVSNet model structure according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the context enhancement module according to an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the attention module in an embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] This embodiment proposes a 3D reconstruction method for MVS that integrates frequency features, including:

[0048] Construct an MVS dataset; where the MVS dataset is an indoor MVS dataset containing several scans, each scan having several views under several different lighting conditions;

[0049] Construct an HL-MVSNet model; the HL-MVSNet model includes: a feature extraction network, an attention module, and a cost volume generation module; the feature extraction network first extracts feature maps from the input image, then inputs these feature maps into the attention module to obtain new feature maps, performs several homography transformations on the new feature maps to obtain the cost volume, and inputs the cost volume into a 3DCNN (where 3DCNN is...). Figure 1 In the 3D convolutional neural network, the cost volume after homography transformation may contain noise. A multi-scale 3D CNN network, similar to a 3D version of UNet, is used to regularize the cost volume, optimizing the noise and obtaining the probability volume. The probability volume is then processed to obtain the depth map.

[0050] The HL-MVSNet model was trained using the MVS dataset to obtain the MVS 3D reconstruction model;

[0051] Using the MVS 3D reconstruction model, we perform MVS 3D reconstruction with fused frequency features.

[0052] The input to the MVS 3D reconstruction model is images of the same scene from different perspectives. These images are divided into reference views and source views, and the output is the depth map of each scene. Finally, the depth maps are fused together to form a point cloud according to the traditional MVS method.

[0053] Specifically, in this embodiment, HL-MVSNet employs the same coarse-to-fine cascaded structure as CasMVSNet. Throughout all stages, the depth assumption is that features are sampled uniformly across the depth range. The first stage acquires image features at low resolution and constructs a cost volume with a predetermined depth range but a large depth interval, while subsequent stages use higher spatial resolution, a narrower depth range, and smaller depth intervals. HL-MVSNet adds a context extraction module that fuses dilated convolutions and an HL-Attention module to the multi-view stereo network, and its network structure is as follows: Figure 1 As shown.

[0054] First, a feature pyramid network is used to extract multi-scale features from N images at three resolutions. Then, to ensure the high-resolution features have a matching receptive field, they are fed into a context enhancement module to expand the receptive field. Next, to further address the problem of accurate matching of low-texture, repetitive, specular, and reflective regions, the features are fed into an HL-Attention module to capture local details and global structure information of the image. Then, a differentiable homography transformation is used to warp the features of the source view to the reference camera coordinate system. A cost volume is then generated based on the similarity of the feature maps, and a 3D CNN is used to regularize the cost volume to generate a probability volume for depth map prediction. The HL-MVSNet network is trained end-to-end using a focus loss function, which treats depth estimation as a classification task, adjusting sample weights based on prediction accuracy. This makes the model pay more attention to misclassified and hard-classified samples, ensuring the stability and accuracy of the model's depth prediction. Finally, the obtained image depth maps are fused to obtain the point cloud of the scene.

[0055] Furthermore, the feature extraction network includes a feature extraction module and a context enhancement module;

[0056] The feature extraction module is used to extract feature maps of three different scales from the input image;

[0057] The context enhancement module is used to fuse contextual information from two feature maps at two different scales with higher resolution.

[0058] The output of the feature extraction network is a feature map after context enhancement; low-resolution feature maps are not processed.

[0059] The context enhancement module includes: four dilated convolutional layers with different dilation rates and a 1×1 convolutional layer;

[0060] The outputs of each dilated convolution are concatenated together and passed through a 1×1 convolutional layer to fuse contextual information from different receptive fields; the outputs of the dilated convolutions are then added to the inputs to fuse coarse-grained and fine-grained features.

[0061] Specifically, in deep learning-based MVS methods, feature pyramid networks are typically used to extract features from images at multiple scales. Feature pyramid networks extract three feature maps of different scales from the input image using a top-down approach, and then fuse high-level and low-level feature maps in a top-down manner to obtain richer feature representations with multi-scale information. The resolutions of the three feature maps obtained through the feature pyramid network are H / 4×W / 4, H / 2×W / 2, and H×W, respectively. However, the receptive field of the feature pyramid network cannot fully adapt to these three resolutions; higher-resolution images require larger receptive fields to obtain effective semantics, which limits the performance of the feature pyramid network.

[0062] Therefore, this embodiment designs a context enhancement module to enhance the contextual information of different receptive fields at high resolution. Specifically, as follows: Figure 1 As shown, for the middle and lower layers with higher resolution, after obtaining their feature maps, this embodiment inputs them into the context enhancement module to obtain richer context information. The context enhancement module is as follows: Figure 2 As shown, it includes four dilated convolutional layers with different dilation rates, set to rate, 2*rate, 3*rate, and 4*rate respectively. The specific rate value can be set according to the resolution. These separate convolutional layers can acquire multiple feature maps in different receptive fields. Furthermore, to finely merge multi-scale information, this embodiment concatenates the outputs of each dilated convolution and fuses the contextual information from different receptive fields using a 1×1 convolutional layer. Finally, to preserve the coarse-grained information of the initial input, this embodiment adds the output of the dilated convolution to the input to fuse coarse-grained and fine-grained features.

[0063] Furthermore, the attention module includes: high-frequency paths and low-frequency paths;

[0064] High-frequency paths are used to encode local details of an object;

[0065] Low-frequency paths are used to encode the global structure of objects;

[0066] High-frequency paths are assigned (1-α)Nh attention heads, and low-frequency paths are assigned α attention heads.

[0067] The high-frequency path captures fine-grained high frequencies through self-attention of local windows, and the local window is a non-overlapping window segmentation of the image;

[0068] The low-frequency path first performs average pooling on each window to obtain the input low-frequency signal. The keys and values ​​of the low-frequency path are obtained from the average pooled windows.

[0069] The attention outputs of the high-frequency path and the low-frequency path are concatenated together as the final output of the attention module;

[0070] The attention outputs for both high-frequency and low-frequency paths are calculated using the scaled dot product attention formula.

[0071] Specifically, natural images contain rich frequencies. High frequencies capture local details of objects (e.g., lines and shapes), while low frequencies encode global structure (e.g., texture and color). However, learning-based MVS methods rarely consider utilizing the high- and low-frequency information of images. This results in the rich contextual information of the extracted feature maps not being fully utilized, ultimately leading to less than ideal prediction quality, especially for low-texture and repetitive regions. Considering that transformers can effectively capture both local details and global structure of images, this embodiment designs a transformer-based HL-Attention module to capture high- and low-frequency information in images.

[0072] HL-Attention in the transformer retains the standard multi-head attention layer configuration, but it is separated into two paths. One path encodes high-frequency interactions through local self-attention with relatively high-resolution feature maps, while the other path encodes low-frequency interactions through global attention with downsampled feature maps. The high-frequency path is assigned (1-α)Nh attention heads, and the low-frequency path is assigned α attention heads. The total number of attention heads is the same as in the standard multi-head attention layer. This avoids doubling the computational cost by simultaneously assigning the same number of attention heads to both paths as in the standard multi-head attention layer. Here, α is set according to the situation, and Nh refers to the total number of attention heads.

[0073] For the high-frequency paths of HL-Attention, i.e. Figure 3 The upper-level path in the image mainly encodes the local details of the object. However, using global attention on the feature map may lead to information redundancy and increase meaningless computational costs. Therefore, the high-frequency path captures fine-grained high frequencies through local window self-attention (e.g., 2×2 window). The local window is a non-overlapping window segmentation of the image, which saves a lot of computational costs compared with time-consuming window shifting or multi-scale window segmentation.

[0074] For the low-frequency path of HL-Attention, i.e. Figure 3The lower-level paths in the low-frequency path primarily encode the global structure of the object. While directly using standard multi-head attention is effective for capturing low-frequency information in an image, it incurs significant computational costs. Since average pooling is a low-pass filter, the low-frequency path first performs average pooling on each window to obtain the input low-frequency signal. The keys and values ​​of the low-frequency path are obtained from the average-pooled windows. Thanks to the reduced key and value lengths, the complexity of the low-frequency path is significantly reduced.

[0075] Figure 3 Positional encoding in this context is used to supplement window position information. By providing position-related features to the model, positional encoding helps the model understand the relative or absolute positional relationships between windows in a sequence, thereby improving the model's performance.

[0076] Regardless of whether it's a high-frequency or low-frequency path, the attention output is calculated using the scaled dot product attention formula, as shown in formula n. Based on formula n, features are grouped into query Q, key K, and value V. Q retrieves relevant information from V based on the attention weight obtained from the dot product of Q and K for each V. The attention mechanism measures the feature similarity between Q and K and retrieves information from V based on the calculated weights. For high-frequency paths, QKV is obtained from the original feature map. For low-frequency paths, Q is the same as for high-frequency paths, both derived from the original feature map, while K and V are derived from the feature map after average pooling.

[0077]

[0078] After calculating the attention outputs of the high-frequency and low-frequency paths, HL-Attention concatenates the outputs of the two paths together as the final output.

[0079] Furthermore, the HL-MVSNet model also includes: a transit path module;

[0080] The pass-through path module is used to align the output of the low-resolution feature map through the attention module with the channel of the high-resolution feature in the previous layer through a convolutional neural network, then upsample it and add it to the high-resolution feature, and finally fuse the result through a convolutional neural network.

[0081] Specifically, since calculating the scaling dot product attention formula for image features at high resolution consumes a large amount of computational cost, how to effectively transfer features from low resolution to high resolution remains a problem. Therefore, this embodiment designs a transfer path module. Specifically, it first... Figure 1The output of the low-to-medium resolution feature map after passing through HL-Attention is aligned with the channel of the high-resolution feature in the previous layer through a convolutional neural network. After upsampling, it is added to the high-resolution feature. Finally, the result is fused through a convolutional neural network.

[0082] Furthermore, the cost body generation module is used to generate a cost body based on the similarity of the feature maps;

[0083] Specifically, this embodiment applies a differentiable homography transformation to align all source images to the reference view. Under the depth assumption d, pixel p in the reference view corresponds to the pixel p in the source view. Homography transformation between them is defined as

[0084]

[0085] Where R and t represent the rotation and translation between the two views. K0 and K are the essential matrices of the reference camera and the source camera, respectively. The feature map after homography transformation preserves the original resolution through bilinear interpolation. By discretizing the known depth space into D depth values, this embodiment is able to classify each pixel into one of these values.

[0086]

[0087] The pairwise feature relationship at position p is shown in the above formula, where, This represents the feature map of the homography transformation of the i-th source image at depth d. In this way, the number of channels is reduced to 1, alleviating memory consumption during regularization. To aggregate all N-1 pairs of feature correlations, this embodiment assumes that each pixel in the height and width dimensions of the 3D correlation has different saliency, but is consistent in the depth dimension. Therefore, this embodiment assigns a pixel-wise weighted map to the correlation with the maximum depth dimension. The aggregated correlation volume is defined as...

[0088]

[0089] Furthermore, using the MVS dataset, the HL-MVSNet model was trained end-to-end using the focus loss function;

[0090] Specifically, this embodiment applies a focus loss function, treating depth estimation as a classification task to enhance single-hotspot supervision of fuzzy regions. The focus loss for each depth estimation stage is as follows:

[0091]

[0092] In the formula P (d) (p) represents the predicted probability of the depth hypothesis d at pixel p. This represents the depth value that best approximates the true value among all hypotheses. {pv} represents the subset of pixels with valid true values. When the focus parameter γ = 0, the focus loss degenerates into cross-entropy loss. Empirically, γ = 2 is suitable for more complex scenes, while γ = 0 can produce sufficiently good results for relatively simple scenes.

[0093] Furthermore, this embodiment evaluates HL-MVSNet through extensive experiments. First, the details of the experimental setup are described, followed by comparisons with state-of-the-art methods. Finally, a series of ablation studies are conducted to validate the effectiveness of each component.

[0094] The dataset used in this embodiment:

[0095] The proposed dataset is evaluated on two benchmark datasets: DTU, Tanks and Temples, and BlendedMVS. The DTU dataset is an indoor MVS dataset containing 128 scans, each with 49 or 64 views under seven different lighting conditions. This embodiment divides the entire dataset into training, validation, and test sets in the manner of MVSNet. The Tanks and Temples dataset is typically used to validate performance on more complex outdoor reconstructions. It includes a mid-level set and a high-level set. The mid-level set contains eight scenes, and the high-level set contains six scenes. The Tanks and Temples dataset is evaluated by uploading the generated point clouds to the official website.

[0096] This embodiment first trains HL-MVSNet on the DTU training set for evaluation on the DTU test set. During training, the input images are N=5, each 640×512 pixels in size. The depth range is from 425mm to 935mm. At different stages, this embodiment samples hypothetical depths of 48, 32, and 8, with depth intervals of 4, 2, and 1, respectively. HL-MVSNet is implemented using the PyTorch framework and trained for 16 epochs using the Adam optimizer. The initial learning rate is 0.0001, halved after the 6th, 8th, and 10th epochs. Training is performed using a 4-way NVIDIA RTX3090 GPU with a batch size of 1.

[0097] The experimental results of this embodiment are as follows:

[0098] Comparative experiment:

[0099] To demonstrate the effectiveness of HL-MVSNet, quantitative testing experiments were conducted on the DTU dataset in this embodiment. The number of input images was N=5, and the image size was 1152×864. Depth filtering was performed through geometric and photometric constraints, and depth fusion using Gipuma was used to obtain the final 3D point cloud.

[0100] The DTU dataset proposes two evaluation metrics: accuracy and completeness, both of which were proposed by Seitz. Accuracy calculates the distance from the 3D reconstruction result to the real scene. This metric describes the quality of the reconstructed 3D point cloud result, and the calculation formula is shown in Equation (1). Here, S1 represents the set of all points in the point cloud reconstructed by the algorithm, and S2 represents the set of all points in the standard point cloud.

[0101]

[0102] The calculation of completeness is the opposite of the above calculation method. It describes the coverage of the real point cloud on the reconstructed point cloud, and is represented by the average of the nearest distances from each point in the standard point cloud to the reconstructed point cloud. The specific calculation formula is shown in formula (2). Among them, the definitions of S1 and S2 are the same as those in the accuracy calculation formula.

[0103]

[0104] The average of the two metrics is used to obtain the overall performance (OA) of the model. Both metrics are used to measure the degree of deviation between the reconstructed model and the real model; the smaller the value, the better the reconstruction effect.

[0105] The calculation method is shown in formula (3).

[0106]

[0107] Generalization experiment:

[0108] For the evaluation of the Tanks & Temples dataset, the model was fine-tuned using the DTU training set, and then quantitatively tested on the Tanks & Temples dataset to evaluate the generalization ability of HL-MVSNet. The input image size was 768×576, and the number of input views N=10. The evaluation on this dataset was performed by submitting the generated point cloud to the official website to obtain the F-score. The F-score is the harmonic mean of precision and recall; a higher value indicates better method performance.

[0109] ablation experiment

[0110] This embodiment conducts ablation studies to quantitatively analyze the effectiveness of the CEM and HL-Attention modules. The following four ablation studies were performed on the DTU test set using the same parameters as the generalization experiments: (a) Baseline based on a cascaded architecture, using a feature pyramid network for feature extraction and variance-based metrics, without adding any additional modules; (b) Baseline + CEM module; (c) Baseline + HL-Attention module; (d) Baseline + CEM + HL-Attention, i.e., HL-MVSNet.

[0111] This embodiment proposes a multi-view stereo network HL-MVSNet based on feature aggregation Transformer. Specifically, a feature extraction module with fused dilated convolutions adaptively expands the receptive field; then, an Attention module is used to aggregate global long-range context-aware information within and between images, making the aggregated dense features more suitable for reliable feature matching; furthermore, visibility cost aggregation is employed to suppress the adverse effects of occluded pixels. Extensive experiments on DTU and Tanks & Temples demonstrate that the proposed method outperforms many state-of-the-art learning-based methods and exhibits excellent generalization ability.

[0112] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for three-dimensional reconstruction using MVS that integrates frequency features, characterized in that, include: Construct an MVS dataset; wherein the MVS dataset is an indoor MVS dataset containing several scans, each scan having several views under several different lighting conditions; Construct an HL-MVSNet model; wherein the HL-MVSNet model includes: a feature extraction network, an attention module, and a cost volume generation module; The HL-MVSNet model is trained using the MVS dataset to obtain the MVS 3D reconstruction model; Using the aforementioned MVS 3D reconstruction model, MVS 3D reconstruction with fused frequency features is performed; The feature extraction network includes: a feature extraction module and a context enhancement module; The feature extraction module is used to extract three feature maps of different scales from the input image, including a low-resolution output and two high-resolution outputs; The context enhancement module is used to fuse context information of feature maps at two higher resolution scales to obtain a feature map after context enhancement processing. The context enhancement module includes: four dilated convolutional layers with different dilation rates and a 1 × 1 convolutional layer; The outputs of each dilated convolution are concatenated and passed through a 1 × 1 convolutional layer to fuse contextual information from different receptive fields; the outputs of the dilated convolutions are added to the inputs to fuse coarse-grained and fine-grained features. This is used to perform multi-head attention processing on the feature maps extracted by the feature pyramid network of the fusion context enhancement module, and obtain the feature maps after the attention module has processed them. The attention module includes: a high-frequency path and a low-frequency path; The high-frequency path is used to encode local details of the object; The low-frequency path is used to encode the global structure of the object; The high-frequency path allocation (1- Nh attention heads, low-frequency path allocation Nh attention heads; where Nh refers to the total number of attention heads.

2. The MVS three-dimensional reconstruction method based on fused frequency features according to claim 1, characterized in that, The high-frequency path captures fine-grained high frequencies through self-attention of local windows, and the local window is a non-overlapping window segmentation of the image. The low-frequency path first performs average pooling on each window to obtain the key vector and value vector of the low-frequency path.

3. The MVS three-dimensional reconstruction method based on fused frequency features according to claim 1, characterized in that, The attention outputs of the high-frequency path and the low-frequency path are concatenated together to form the final output of the attention module; The attention outputs of both the high-frequency and low-frequency paths are calculated using the scaled dot product attention formula. The scaling dot product attention formula is: Where Q is the query vector, K is the key vector, and V is the value vector. To query the product of the transpose of the vector and the key vector, This is the scaling factor.

4. The MVS three-dimensional reconstruction method based on fused frequency features according to claim 1, characterized in that, The HL-MVSNet model also includes: a transit path module; The transmission path module is used to align the output of the low-resolution feature map after passing through the attention module with the channel of the high-resolution feature map of the previous layer through a convolutional neural network, then upsample the output of the low-resolution feature map after passing through the attention module and add it to the high-resolution feature map, and finally fuse the added result through a convolutional neural network; wherein, the low-resolution feature map is the low-resolution output of the feature extraction module, and the high-resolution feature map is the high-resolution output of the feature extraction module.

5. The MVS three-dimensional reconstruction method based on fused frequency features according to claim 1, characterized in that, The cost body generation module is used to perform several homography transformations on the feature map obtained after processing by the feature extraction network and the attention module to generate the cost body. The cost body is: in, As a cost body, The number of source views and reference views. d For depth, This represents the similarity between the reference view and the source view at pixel p at depth d.

6. The MVS three-dimensional reconstruction method based on fused frequency features according to claim 1, characterized in that, Using the MVS dataset, the HL-MVSNet model is trained end-to-end using the focus loss function; The focus loss function is: in, Represents the focus loss function. Indicates the focus parameter, Representing the depth hypothesis In pixels The predicted probability at that location. This represents the depth value that is closest to the true value among all hypotheses. Let p represent a subset of pixels that have valid truth values.

Images