MVS three-dimensional reconstruction method fusing frequency characteristics

Abstract

The invention relates to an MVS three-dimensional reconstruction method fusing frequency characteristics. The method comprises the following steps: constructing an MVS data set; wherein the MVS data set is an indoor MVS data set comprising a plurality of scans, and each scan has a plurality of views under a plurality of different illumination conditions; an HL-MVSNet model is constructed; wherein the HL-MVSNet model comprises a feature extraction network, an attention module and a cost body generation module; the HL-MVSNet model is trained by means of the MVS data set, and an MVS three-dimensional reconstruction model is obtained; and utilizing the MVS three-dimensional reconstruction model to carry out MVS three-dimensional reconstruction fused with frequency characteristics. According to the method, the reconstruction accuracy and integrity are remarkably improved.

Description

Technical Field

[0001] The invention relates to the technical field of multi-view stereo matching, and in particular to an MVS three-dimensional reconstruction method integrating frequency features. Background Art

[0002] Multi-view stereo matching (MVS) aims to reconstruct the 3D structure of a scene from a sequence of images. MVS has attracted increasing attention in recent years due to its wide applications in autonomous driving, robotics, and virtual reality. Although traditional methods have achieved good results, they still have some limitations, such as incomplete reconstruction due to occlusions, illumination variations, textureless regions, and non-Lambertian surfaces.

[0003] To overcome the above challenges, learning-based multi-view stereo matching (MVS) methods have been proposed. The first to be proposed is MVSNet, an end-to-end multi-view stereo matching method based on deep learning. MVSNet uses convolutional neural network (CNN) to extract features and more effectively obtain high-level image features. In the cost volume regularization stage, MVSNet introduces a variance-based multi-view aggregation method to adapt to the input of any view. In addition, MVSNet also uses a 3D CNN structure to filter the cost volume. However, due to the high memory usage of this structure, the cost is relatively high in practical applications. To solve the problem of high memory usage, R-MVSNet has been proposed. This network uses the gated unit GRU in the recurrent neural network to replace 3DCNN. Although the efficiency is reduced, it effectively reduces the memory usage. Cascade-MVSNet uses large depth intervals and fewer depth intervals for depth prediction on downsampled low-resolution images, and then applies the prediction results to non-downsampled high-resolution images, which significantly reduces the memory requirements of the model. AA-RMVSNet aggregates context features in multiple regions of different texture richness through the Intra-view AA module, effectively reducing feature loss. TransMVSNet introduces transformer into the MVSNet network, effectively aggregating the global context information between the reference image and the source image, and effectively reducing feature loss.

[0004] Although learning-based MVS methods have achieved great success, they still face challenges. By studying the above methods, we found that the current learning-based MVS methods have two main problems: (1) Convolution has excellent performance in acquiring local features, but the limited receptive field of convolution also affects the perception of contextual information when acquiring features, which makes it difficult to perform robust depth estimation for challenging areas such as poor texture, repeated patterns, and non-Lambertian surfaces in the MVS task. (2) In addition, although existing learning-based MVS methods have introduced Transformer to enhance the long-range contextual connection of images, so that the problem of accurate matching in low-texture, repeated, specular, and reflective areas has been solved to a certain extent, it ignores the rich frequencies in natural images. Encoding different frequencies plays a very important role in acquiring different information of the image (such as lines, shapes, textures, and colors). This information also plays an irreplaceable role in the accurate matching of low-texture, repeated, specular, and reflective areas in the MVS task. Summary of the invention

[0005] The purpose of the present invention is to propose an MVS 3D reconstruction method integrating frequency features to solve the problems existing in the above-mentioned prior art. The present invention designs an end-to-end MVS network HL-MVSNet, which uses a context enhancement module (Context Enhancement Module, CEM) based on dilated convolution to enhance the context information of the image, and uses a Transformer-based HL-Attention module to obtain information such as lines, shapes, textures and colors of the image by encoding different frequencies of the image, thereby enhancing the remote contextual connection of the image and further solving the problem of accurate matching of low texture, repetitive, mirror and reflective areas. HL-MVSNet achieves significant improvements in reconstruction accuracy and completeness.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] An MVS three-dimensional reconstruction method integrating frequency features, comprising:

[0008] Constructing an MVS dataset; wherein the MVS dataset is an indoor MVS dataset comprising a plurality of scans, each of which has a plurality of views under a plurality of different lighting conditions;

[0009] Constructing a HL-MVSNet model; wherein the HL-MVSNet model includes: a feature extraction network, an attention module, and a cost volume generation module;

[0010] Using the MVS data set to train the HL-MVSNet model to obtain an MVS three-dimensional reconstruction model;

[0011] The MVS three-dimensional reconstruction model is used to perform MVS three-dimensional reconstruction integrating frequency features.

[0012] Optionally, the feature extraction network includes: a feature extraction module and a context enhancement module;

[0013] The feature extraction module is used to extract three feature maps of different scales from the input image, including a low-resolution output and two high-resolution outputs;

[0014] The context enhancement module is used to perform context information fusion on feature maps of two scales with higher resolutions to obtain feature maps after context enhancement processing.

[0015] Optionally, the context enhancement module includes: four atrous convolution layers with different dilation rates and one 1×1 convolution layer;

[0016] The output of each dilated convolution is concatenated together and passed through a 1×1 convolution layer to fuse the contextual information under different receptive fields; and the output of the dilated convolution is added to the input to fuse the coarse-grained and fine-grained features.

[0017] Optionally, the attention module is used to perform multi-head attention processing on the feature map extracted by the feature pyramid network of the fusion context enhancement module to obtain the feature map processed by the attention module;

[0018] The attention module includes: a high-frequency path and a low-frequency path;

[0019] The high frequency path is used to encode local details of the object;

[0020] The low-frequency path is used to encode the global structure of the object;

[0021] The high-frequency path is allocated (1-α)Nh attention heads, and the low-frequency path is allocated αNh attention heads; wherein Nh refers to the total number of attention heads.

[0022] Optionally, the high-frequency path captures fine-grained high frequencies through local window self-attention, and the local window is a non-overlapping window segmentation of the image;

[0023] The low-frequency path first performs average pooling on each window to obtain a key vector and a value vector of the low-frequency path.

[0024] Optionally, the attention outputs of the high-frequency path and the low-frequency path are concatenated together as the final output of the attention module;

[0025] The attention outputs of the high-frequency path and the low-frequency path are both calculated using the scaled dot product attention formula;

[0026] The scaled dot product attention formula is:

[0027]

[0028] Among them, Q is the query vector, K is the key vector, V is the value vector, QK T is the product of the query vector and the transpose of the key vector, is the scaling factor.

[0029] Optionally, the HL-MVSNet model further includes: a transfer path module;

[0030] The transfer path module is used to align the output of the low-resolution feature map after passing through the attention module with the channel of the previous layer of high-resolution features through a layer of convolutional neural network, then upsample the output of the low-resolution feature map after passing through the attention module and add it to the high-resolution feature map, and finally fuse the added result through a layer of convolutional neural network; wherein the low-resolution feature map is the low-resolution output of the feature extraction module, and the high-resolution feature map is the high-resolution output of the feature extraction module.

[0031] Optionally, the cost volume generation module is used to perform several homography transformations on the feature map obtained after being processed by the feature extraction network and the attention module to generate a cost volume;

[0032] The cost body is:

[0033]

[0034] Among them, C (d) (p) is the cost volume, N is the number of source views and reference views, d is the depth, is the similarity between the reference view and the source view at pixel p at depth d.

[0035] Optionally, using the MVS dataset, the HL-MVSNet model is trained end-to-end using a focal loss function;

[0036] The focal loss function is:

[0037]

[0038] Among them, L represents the focus loss function, γ represents the focus parameter, and P (d) (p) represents the predicted probability of depth hypothesis d at pixel p, represents the depth value closest to the true value among all hypotheses, {pv} represents the subset of pixels with valid true values, and p represents the pixel point.

[0039] The beneficial effects of the present invention are:

[0040] The present invention proposes a multi-view stereo network HL-MVSNet based on feature aggregation Transformer. The feature extraction module using fused dilated convolutions adaptively expands the receptive field; then it is executed through the Attention module to aggregate the global long-range contextual perception information within and between images, making the aggregated dense features more suitable for reliable feature matching; in addition, the cost volume generation module uses visibility cost aggregation to suppress the adverse effects of occluded pixels. A large number of experiments on DTU, Tanks & Temples show that compared with existing learning-based methods, the method proposed in the present invention outperforms other methods and has excellent generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 Schematic diagram of the HL-MVSNet model structure of an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of a context enhancement module according to an embodiment of the present invention;

[0044] Figure 3 Schematic diagram of an attention module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] This embodiment proposes an MVS 3D reconstruction method integrating frequency features, including:

[0048] Constructing an MVS dataset; wherein the MVS dataset is an indoor MVS dataset including a plurality of scans, each of which has a plurality of views under a plurality of different lighting conditions;

[0049] Construct the HL-MVSNet model; the HL-MVSNet model includes: feature extraction network, attention module, and cost volume generation module; the feature extraction network first extracts the input image into a feature map, then inputs the feature map into the attention module to obtain a new feature map, performs several homography transformations on the new feature map to obtain a cost volume, and inputs the cost volume into the 3DCNN (where 3DCNN is Figure 1 The cost body may contain noise after homography transformation. A multi-scale 3DCNN network similar to the 3D version of Unet is used to regularize the cost body, optimize the noise of the cost body and obtain the probability body. The probability body is obtained in ), and then the probability body is processed to obtain the depth map.

[0050] The HL-MVSNet model is trained using the MVS dataset to obtain the MVS 3D reconstruction model;

[0051] The MVS 3D reconstruction model is used to perform MVS 3D reconstruction integrating frequency features.

[0052] Among them, the input of the MVS 3D reconstruction model is images of the same scene from different perspectives. These images will be divided into reference views and source views. The output is the depth map of each scene. Finally, the point cloud is fused according to the depth map of the traditional MVS method.

[0053] Specifically, in this embodiment, HL-MVSNet adopts the same coarse-to-fine cascade structure as CasMVSNet. In all stages, the depth hypothesis is uniformly sampled from the depth range. The first stage acquires image features at low resolution and constructs a cost volume with a predetermined depth range but a larger depth interval, while subsequent stages use higher spatial resolution, narrower depth range, and smaller depth interval. HL-MVSNet adds the context extraction module fused with dilated convolution and the HL-Attention module to the multi-view stereo network, and its network structure is as follows: Figure 1 shown.

[0054] First, the feature pyramid network is used to extract multi-scale features of N images at three resolutions; then, in order to make the high-resolution features have a matching receptive field, the high-resolution features are sent to the context enhancement module to expand the receptive field; then, in order to further solve the problem of accurate matching of low-texture, repeated, mirror and reflection areas, the features are sent to the HL-Attention module to capture information such as local details and global structure of the image; then the features of the source view are warped to the reference camera coordinate system using a differentiable homography transformation; then the cost volume is generated based on the similarity of the feature map, and then the cost volume is regularized by 3D CNN to generate a probability volume for depth map prediction; the HL-MVSNet network uses the focal loss function for end-to-end training. The focal loss function regards depth estimation as a classification task, and adjusts the sample weights based on the prediction accuracy, so that the model pays more attention to samples with classification errors and difficult classification, ensuring the stability and accuracy of the model's predicted depth. Finally, the obtained image depth map is fused to obtain the point cloud of the scene.

[0055] Further, the feature extraction network includes: a feature extraction module and a context enhancement module;

[0056] A feature extraction module is used to extract feature maps of three different scales from the input image;

[0057] The context enhancement module is used to fuse the context information of the feature maps of two scales with higher resolution.

[0058] The output of the feature extraction network is a feature map after context enhancement, and the low-resolution feature map is not processed.

[0059] The context enhancement module includes: four atrous convolutional layers with different dilation rates and one 1×1 convolutional layer;

[0060] The output of each dilated convolution is concatenated together and passed through a 1×1 convolution layer to fuse the contextual information under different receptive fields; and the output of the dilated convolution is added to the input to fuse the coarse-grained and fine-grained features.

[0061] Specifically, in the MVS method based on deep learning, a feature pyramid network is usually used to extract multi-scale features from images. The feature pyramid network extracts three feature maps of different scales from the input image by introducing a top-down approach, and fuses the high-level feature map with the low-level feature map in a top-down manner to obtain a richer feature representation with multi-scale information. The resolutions of the three feature maps obtained by the feature pyramid network are H / 4×W / 4, H / 2×W / 2, and H×W, respectively. However, the receptive field of the feature pyramid network cannot fully adapt to the resolutions at these three scales. Images with higher resolutions require a larger receptive field to obtain effective semantics, which limits the performance of the feature pyramid network.

[0062] Therefore, this embodiment designs a context enhancement module to enhance the context information of different receptive fields at high resolution. Figure 1 As shown in FIG. 1 , for the middle and lower layers with higher resolution, after obtaining their feature maps, in order to obtain rich context information, this embodiment inputs them into the context enhancement module. Figure 2 As shown, it includes four hole convolution layers with different expansion rates, and the expansion rates are set to rate, 2*rate, 3*rate, and 4*rate respectively. The specific rate value can be set according to the rate. These separated convolution layers can obtain multiple feature maps in different receptive fields. In addition, in order to finely merge multi-scale information, this embodiment splices the output of each hole convolution together and uses a 1×1 convolution layer to fuse the contextual information under different receptive fields. Finally, in order to maintain the coarse-grained information of the initial input, this embodiment adds the output of the hole convolution to the input to fuse the coarse-grained and fine-grained features.

[0063] Furthermore, the attention module includes: a high-frequency path and a low-frequency path;

[0064] The high-frequency path is used to encode local details of the object;

[0065] The low-frequency path encodes the global structure of the object;

[0066] The high-frequency path is assigned (1-α)Nh attention heads, and the low-frequency path is assigned α attention heads.

[0067] The high-frequency path captures fine-grained high frequencies through local window self-attention, and the local window is a non-overlapping window segmentation of the image;

[0068] The low-frequency path first performs average pooling on each window to obtain the input low-frequency signal, and the key and value of the low-frequency path are obtained based on the window after average pooling.

[0069] The attention outputs of the high-frequency path and the low-frequency path are concatenated together as the final output of the attention module;

[0070] The attention outputs of both the high-frequency and low-frequency paths are calculated using the scaled dot-product attention formula;

[0071] Specifically, natural images contain a wealth of frequencies, with high frequencies capturing local details of objects (such as lines and shapes), and low frequencies encoding global structures (such as textures and colors). However, few learning-based MVS methods can take into account the use of high- and low-frequency information in images, resulting in the fact that the rich contextual information of the extracted feature maps is not fully utilized, which ultimately makes the prediction quality less than ideal, especially for low-texture areas and repeated areas. Considering that transformers can effectively capture local details and global structures of images, this embodiment designs a transformer-based HL-Attention module to capture high- and low-frequency information in images.

[0072] HL-Attention continues the standard multi-headed attention layer multi-headed setting in the transformer and is separated into two paths. One path encodes high-frequency interactions through local self-attention with relatively high-resolution feature maps, and the other path encodes low-frequency interactions through global attention with downsampled feature maps. The high-frequency path is allocated (1-α)Nh attention heads, and the low-frequency path is allocated α attention heads. The total number of attention heads is the same as the standard multi-headed attention layer. This avoids the two paths simultaneously allocating the same number of attention heads as the standard multi-headed attention layer, resulting in doubling the computational cost. Among them, α is set according to the situation, and Nh refers to the total number of attention heads.

[0073] For the high-frequency path of HL-Attention, that is Figure 3 The upper path in the

[15] mainly encodes the local details of the object. However, using global attention on the feature map may lead to information redundancy and increase meaningless computational costs. Therefore, the high-frequency path captures fine-grained high frequencies through local window self-attention (e.g., 2×2 window), and the local window is a non-overlapping window segmentation of the image, which saves a lot of computational costs compared to time-consuming operations such as window shifting or multi-scale window segmentation.

[0074] For the low-frequency path of HL-Attention, that is, Figure 3The main task of the lower path in is to encode the global structure of the object. Although directly using standard multi-head attention is effective for capturing the low-frequency information of the image, it will result in a large consumption of computational cost. Since average pooling is a low-pass filter, the low-frequency path first average pools each window to obtain the input low-frequency signal. The key and value of the low-frequency path are obtained according to the window after average pooling. Thanks to the shortened length of the key and value, the low-frequency path significantly reduces the complexity.

[0075] Figure 3 The positional encoding in is a position encoding, which is used to supplement the window position information. The position encoding helps the model understand the relative or absolute position relationship between windows in the sequence by providing position-related features to the model, thereby improving the performance of the model.

[0076] Regardless of the high-frequency path or the low-frequency path, the attention output is calculated by the scaled dot product attention formula, which is as follows: Formula n, according to which the features are grouped into query Q, key K, and value V, Q retrieves relevant information from V according to the attention weight obtained by the dot product of Q and K corresponding to each V, and the attention mechanism measures the feature similarity between Q and K, and retrieves information from V according to the calculated weight. QKV of the high-frequency path is obtained through the original feature map, Q of the low-frequency path is the same as the high-frequency path, and both come from the original feature map, while K and V come from the feature map after average pooling.

[0077]

[0078] After calculating the attention outputs of the high-frequency path and the low-frequency path, HL-Attention concatenates the outputs of the two paths as the final output.

[0079] Furthermore, the HL-MVSNet model also includes: a transfer path module;

[0080] The transfer path module is used to align the output of the low-resolution feature map through the attention module with the channel of the previous layer of high-resolution features through a layer of convolutional neural network, then add it to the high-resolution features after upsampling, and finally fuse the added results through a layer of convolutional neural network.

[0081] Specifically, since the scaled dot product attention formula for calculating image features at high resolution consumes a lot of computational cost, how to effectively transfer features from low resolution to high resolution is still a problem. Therefore, this embodiment designs a transfer path module. Specifically, first, Figure 1The output of the medium and low-resolution feature maps after HL-Attention is aligned with the channels of the previous layer of high-resolution features through a layer of convolutional neural network, and then upsampled and added to the high-resolution features. Finally, the added results are fused through a layer of convolutional neural network.

[0082] Further, a cost volume generation module is used to generate a cost volume according to the similarity of the feature graphs;

[0083] Specifically, this embodiment applies a differentiable homography to align all source images to the reference view. Under the depth assumption d, the pixel p on the reference view is aligned with the corresponding pixel on the source view. The homography transformation between them is defined as

[0084]

[0085] where R and t represent the rotation and translation between the two views. 0 and K are the essential matrices of the reference camera and the source camera. The feature map after homography transformation maintains the original resolution through bilinear interpolation. By discretizing the known depth space into D depth values, this embodiment can classify each pixel into one of these values.

[0086]

[0087] The pairwise feature relationship at position p is shown in the above formula, where: Represents the feature map of the homography transformation of the i-th source image at depth d. In this way, the number of channels is reduced to 1, alleviating memory consumption during regularization. In order to aggregate all N-1 pairs of feature correlation volumes, this embodiment considers that each pixel in the height and width dimensions of the 3D correlation volume has different significance, but is consistent in the depth dimension. Therefore, this embodiment assigns a pixel-by-pixel weight map to its maximum depth dimension correlation. The aggregated correlation volume is defined as

[0088]

[0089] Furthermore, the HL-MVSNet model is trained end-to-end using the MVS dataset and the focal loss function;

[0090] Specifically, this embodiment applies the focal loss function and regards depth estimation as a classification task to strengthen the single hotspot supervision of the blurred area. The focal loss of each depth estimation stage is as follows:

[0091]

[0092] Where P (d) (p) represents the predicted probability of depth hypothesis d at pixel p, Denotes the depth value closest to the true value among all hypotheses. {pv} denotes the subset of pixels with valid true values. When the focus parameter γ = 0, the focus loss degenerates into the cross entropy loss. According to experience, γ = 2 is suitable for more complex scenes, while γ = 0 can produce good enough results for relatively simple scenes.

[0093] Furthermore, this example evaluates HL-MVSNet through extensive experiments. We first describe the details of the experimental setup, then compare it with the state-of-the-art methods. Finally, we conduct a series of ablation studies to verify the effectiveness of each component.

[0094] The data set used in this example is:

[0095] The proposed one is evaluated on two benchmark datasets, including DTU, Tanks and Temples, and BlendedMVS. The DTU dataset is an indoor MVS dataset containing 128 scans, each with 49 or 64 views under 7 different lighting conditions. This embodiment divides the entire dataset into training, validation, and test sets in the manner of MVSNet. The Tanks and Temples dataset is typically used to verify more complex outdoor reconstruction performance. It includes an intermediate set and an advanced set. The intermediate set has 8 scenes and the advanced set has 6 scenes. The Tanks and Temples dataset is evaluated by uploading the generated point cloud to the official website.

[0096] This embodiment first trains HL-MVSNet on the DTU training set to evaluate it on the DTU test set. During the training phase, this embodiment inputs N=5 images of size 640×512. The depth ranges from 425mm to 935mm. At different stages, this embodiment samples the three hypothetical depths of 48, 32, and 8, respectively, with depth intervals of 4, 2, and 1, respectively. HL-MVSNet is implemented using the Pytorch framework and trained for 16 epochs using the Adam optimizer. The initial learning rate is 0.0001, and it is halved after the 6th, 8th, and 10th epochs. A 4-way NVIDIA RTX3090 is used for training, and the batchsize is set to 1.

[0097] The experimental results of this embodiment are:

[0098] Comparative experiment:

[0099] In order to prove the effectiveness of HL-MVSNet, this example conducts a quantitative test experiment on the DTU dataset. The number of input images N = 5, the image size is 1152 × 864, deep filtering is performed through geometric constraints and photometric constraints, and Gipuma's deep fusion is used to obtain the final 3D point cloud.

[0100] The DTU dataset proposes two evaluation indicators, namely accuracy and completeness. These two evaluation indicators were proposed by Seitz. Accuracy calculates the distance from the 3D reconstruction result to the real scene. This indicator describes the quality of the reconstructed 3D point cloud result. The calculation formula is shown in formula (1). Among them, S1 represents the set of all points in the point cloud reconstructed by the algorithm, and S2 represents the set of all points in the standard point cloud.

[0101]

[0102] The calculation of completeness is opposite to the above calculation method. It describes the coverage of the real point cloud on the reconstructed point cloud, and is expressed by the average value of the closest distance from each point in the standard point cloud to the reconstructed point cloud. The specific calculation formula is shown in formula (2). The definitions of S1 and S2 are the same as those in the accuracy calculation formula.

[0103]

[0104] The above two indicators are averaged to obtain the overall performance of the model (OA). Both indicators are used to measure the degree of deviation between the reconstructed model and the true model. The smaller the value, the better the reconstruction effect.

[0105] The calculation method is shown in formula (3).

[0106]

[0107] Generalization experiment:

[0108] Regarding the evaluation of the Tanks & Temples dataset, the model was fine-tuned using the DTU training set, and the fine-tuned model was used to perform quantitative tests on the Tanks & Temples dataset to evaluate the generalization ability of HL-MVSNet. The size of the input image is 768×576, and the number of input views is N=10. The evaluation of this dataset is to obtain the F-score by submitting the generated point cloud to the official website. The F-score is the harmonic mean of precision and recall, and the higher the value, the better the performance of the method.

[0109] Ablation experiment

[0110] This example conducts ablation studies to quantitatively analyze the effectiveness of the CEM module and the HL-Attention module. The following four groups of ablation studies are conducted on the DTU test set using the same parameters as the generalization experiment. (a) Baseline is based on a cascade architecture, uses a feature pyramid network for feature extraction and variance-based metrics, and does not add any additional modules; (b) Baseline + CEM module; (c) Baseline + HL-Attention module; (d) Baseline + CEM + HL-Attention, i.e., HL-MVSNet.

[0111] This embodiment proposes a multi-view stereo network HL-MVSNet based on feature aggregation Transformer. Specifically, the feature extraction module using fused hole convolution is used to adaptively expand the receptive field; then it is executed through the Attention module to aggregate the global long-range contextual perception information within and between images, making the aggregated dense features more suitable for reliable feature matching; in addition, the visibility cost aggregation is used to suppress the adverse effects of occluded pixels. A large number of experiments on DTU, Tanks & Temples show that compared with many advanced learning-based methods, the proposed method outperforms other methods and has excellent generalization ability.

[0112] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A MVS 3D reconstruction method integrating frequency features, characterized in that: include: Constructing an MVS dataset; wherein the MVS dataset is an indoor MVS dataset comprising a plurality of scans, each of which has a plurality of views under a plurality of different lighting conditions; Constructing a HL-MVSNet model; wherein the HL-MVSNet model includes: a feature extraction network, an attention module, and a cost volume generation module; Using the MVS data set to train the HL-MVSNet model to obtain an MVS three-dimensional reconstruction model; The MVS three-dimensional reconstruction model is used to perform MVS three-dimensional reconstruction integrating frequency features.

2. The MVS 3D reconstruction method integrating frequency features according to claim 1, characterized in that: The feature extraction network includes: a feature extraction module and a context enhancement module; The feature extraction module is used to extract three feature maps of different scales from the input image, including a low-resolution output and two high-resolution outputs; The context enhancement module is used to perform context information fusion on feature maps of two scales with higher resolutions to obtain feature maps after context enhancement processing.

3. The MVS 3D reconstruction method integrating frequency features according to claim 2, characterized in that: The context enhancement module includes: four hole convolution layers with different dilation rates and a 1×1 convolution layer; The output of each dilated convolution is concatenated together and passed through a 1×1 convolution layer to fuse the contextual information under different receptive fields; and the output of the dilated convolution is added to the input to fuse the coarse-grained and fine-grained features.

4. The MVS 3D reconstruction method integrating frequency features according to claim 2, characterized in that: The attention module is used to perform multi-head attention processing on the feature map extracted by the feature pyramid network of the fusion context enhancement module to obtain the feature map processed by the attention module; The attention module includes: a high-frequency path and a low-frequency path; The high frequency path is used to encode local details of the object; The low-frequency path is used to encode the global structure of the object; The high-frequency path is allocated (1-α)Nh attention heads, and the low-frequency path is allocated αNh attention heads; wherein Nh refers to the total number of attention heads.

5. The MVS 3D reconstruction method integrating frequency features according to claim 4, characterized in that: The high-frequency path captures fine-grained high frequencies through local window self-attention, and the local window is a non-overlapping window segmentation of the image; The low-frequency path first performs average pooling on each window to obtain a key vector and a value vector of the low-frequency path.

6. The MVS 3D reconstruction method integrating frequency features according to claim 4, characterized in that: The attention outputs of the high-frequency path and the low-frequency path are concatenated together as the final output of the attention module; The attention outputs of the high-frequency path and the low-frequency path are both calculated using the scaled dot product attention formula; The scaled dot product attention formula is: Among them, Q is the query vector, K is the key vector, V is the value vector, QK T is the product of the query vector and the transpose of the key vector, is the scaling factor.

7. The MVS 3D reconstruction method integrating frequency features according to claim 4, characterized in that: The HL-MVSNet model also includes: a transfer path module; The transfer path module is used to align the output of the low-resolution feature map after passing through the attention module with the channel of the previous layer of high-resolution features through a layer of convolutional neural network, then upsample the output of the low-resolution feature map after passing through the attention module and add it to the high-resolution feature map, and finally fuse the added result through a layer of convolutional neural network; wherein the low-resolution feature map is the low-resolution output of the feature extraction module, and the high-resolution feature map is the high-resolution output of the feature extraction module.

8. The MVS 3D reconstruction method integrating frequency features according to claim 4, characterized in that: The cost volume generation module is used to perform several homography transformations on the feature map obtained after being processed by the feature extraction network and the attention module to generate a cost volume; The cost body is: Among them, C (d) (p) is the cost volume, N is the number of source views and reference views, d is the depth, is the similarity between the reference view and the source view at pixel p at depth d.

9. The MVS 3D reconstruction method integrating frequency features according to claim 1, characterized in that: Using the MVS dataset, the HL-MVSNet model is trained end-to-end using a focal loss function; The focal loss function is: Among them, L represents the focus loss function, γ represents the focus parameter, and P (d) (p) represents the predicted probability of depth hypothesis d at pixel p, represents the depth value closest to the true value among all hypotheses, {pv} represents the subset of pixels with valid true values, and p represents the pixel point.

Images