Multi-view three-dimensional reconstruction system and method based on self-attention mechanism

Abstract

The invention provides a multi-view three-dimensional reconstruction system and method based on a self-attention mechanism, the system comprises a multi-scale feature extraction module, a matching cost body construction module, a depth sampling range estimation module, a Transform attention aggregation module and a loss function module, the multi-scale feature extraction module is used for extracting multi-scale features of an image; the matching cost body construction module is used for constructing a multi-view matching cost body; the depth sampling range estimation module is used for estimating a depth sampling range; the Transform attention aggregation module is used for aggregating the matching cost body; the loss function module is used for calculating the distance between the predicted depth distribution and the real depth map. According to the multi-view three-dimensional reconstruction system based on the self-attention mechanism provided by the invention, through the multi-scale feature extraction module, the matching cost body construction module, the depth sampling range estimation module, the Transform attention aggregation module and other modules, the problems existing in MVS reconstruction are effectively solved, and the integrity and accuracy of three-dimensional reconstruction are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and three-dimensional reconstruction, and in particular to a multi-view stereoscopic reconstruction system and method based on a self-attention mechanism. Background Art

[0002] Multi-view stereo image reconstruction has been a hot topic in the past decade. Remarkable progress has been made in reconstruction quality and efficiency based on the MVS network, and it has been widely used in autonomous driving, augmented reality, robotics and other fields. The goal of multi-view stereo vision is to reconstruct a three-dimensional system of a scene from a set of images captured by a camera from multiple viewpoints. The basis of the MVS method is to estimate the depth map from the input image, so the integrity and accuracy of the final point cloud inevitably depends on the integrity and accuracy of the calculated depth map.

[0003] Compared with the 2DCNN method, the cost volume pyramid based method reduces the depth range of the current stage by the prediction of the previous stage. With the gradual increase of volume resolution and the adaptive adjustment of depth interval, it outputs in a coarse-to-fine manner, saving memory and time while achieving better performance, and has become the preferred technique for utilizing high-resolution images. Although this method has achieved encouraging results, they often miss small objects and boundary areas with depth changes.

[0004] The single-stage end-to-end MVS architecture for deep learning to learn from images follows the idea that all view images contribute equally to the matching cost volume. For example, MVSNet and R-MVSNet use variance operations to fuse different source volumes, while DPSNet chooses mean operations. However, due to factors such as lighting, camera geometry, and scene content variability, images from different perspectives will produce different image capture features. Since the convolution operation in CNN can only capture local information and cannot establish long-distance connections of the global image, when processing complex scenes, there are often problems such as insufficient understanding of the overall image and ineffective use of depth information association, which limits the integrity and accuracy of the reconstruction results.

[0005] To this end, a multi-view stereo reconstruction system and method based on self-attention mechanism is proposed. Summary of the invention

[0006] The present invention aims to solve the problems raised in the background technology and provides a multi-view stereo reconstruction system and method based on the self-attention mechanism. Through dynamic scale feature extraction and Transformer attention mechanism, multi-view depth correlation is effectively established to achieve high-precision three-dimensional scene reconstruction, aiming to improve the integrity and accuracy of multi-view stereo reconstruction.

[0007] The specific technical solutions are as follows:

[0008] A multi-view stereo reconstruction system based on self-attention mechanism, comprising:

[0009] Multi-scale feature extraction module, used to extract multi-scale features of images;

[0010] Matching cost volume construction module, used to construct multi-view matching cost volume;

[0011] A depth sampling range estimation module, used to estimate the depth sampling range;

[0012] Transformer attention aggregation module, used to aggregate matching cost bodies;

[0013] The loss function module is used to calculate the distance between the predicted depth distribution and the true depth map.

[0014] The above-mentioned multi-view stereo reconstruction system based on the self-attention mechanism, wherein: the multi-scale feature extraction module includes a shallow feature extraction submodule and a global representation subnetwork, and multi-scale feature extraction is achieved through the fusion of convolutional neural network (CNN) and Transformer.

[0015] In the above-mentioned multi-view stereo reconstruction system based on the self-attention mechanism, the specific working process of the multi-scale feature extraction module is as follows:

[0016] Firstly, the shallow feature extraction submodule uses a convolutional neural network (CNN) to extract shallow features from the input multi-view images to obtain the preliminary feature representation of the image.

[0017] Then, the global representation sub-network uses a lightweight Transformer module to learn the global representation of shallow features. Through the Transformer's attention mechanism, the global similar feature correlation and long-distance dependency in the image are captured. At the same time, the feature pyramid network (FPN) is combined to perform multi-scale feature fusion to focus on global features and local texture details at the same time.

[0018] The local features extracted by the convolutional neural network (CNN) are then fused with the global features captured by the Transformer to obtain a multi-scale feature representation from coarse to fine, providing rich feature information for subsequent depth estimation and 3D reconstruction.

[0019] Among them: I0, and Denote the input and output of feature extraction, and use bicubic interpolation to adjust F (h) The size of the Transformer's positional encoding is adapted to different image scales. Under different resolutions, the visual Transformer backbone needs to meet two conditions, namely, an effective attention mechanism and robustness of position encoding at different scales. The output of the visual Transformer is directly added to the top-level features of the FPN encoder. First, the four multi-scale features are encoded with a resolution of 1 / 4 of the original image. Then FPN upsamples these multi-scale features, and can obtain coarse-to-fine features through the FPN decoder. From the original resolution To (H, W), the specific formula (1) is as follows:

[0020]

[0021] The above-mentioned multi-view stereo reconstruction system based on the self-attention mechanism, wherein: the matching cost volume construction module uses a plane scanning algorithm to construct a multi-view matching cost volume, and realizes the fusion of depth information through a cross-attention operation.

[0022] In the above-mentioned multi-view stereo reconstruction system based on the self-attention mechanism, the specific working process of the matching cost volume construction module is as follows:

[0023] Within a given sampling depth range, the plane scanning algorithm is used to construct multiple front-to-back parallel planes in the reference camera space, the features of the source view are mapped to the reference camera frustum, and a multi-view matching cost volume is constructed. Through the cross-attention operation, the association between the reference view cost volume and the source view cost volume is calculated to achieve effective fusion of depth information.

[0024] The above-mentioned multi-view stereo reconstruction system based on the self-attention mechanism, wherein: it also includes a cost volume regularization module, and the cost volume regularization module is used to regularize the constructed cost volume to smooth the noise and outliers in the depth estimation. This step helps to improve the quality and stability of the depth map.

[0025] The above-mentioned multi-view stereo reconstruction system based on the self-attention mechanism, wherein: the depth sampling range estimation module adopts an inverse depth strategy to estimate the depth sampling range to achieve coarse-to-fine depth map propagation. A flexible depth sampling strategy is adopted to ensure that the pixel points on the source view level line are distributed as evenly as possible through inverse depth sampling. In each stage, the depth prediction of the previous stage is used as the center, and a depth hypothesis is uniformly generated within the assumed depth range to achieve coarse-to-fine depth map propagation.

[0026] The above-mentioned multi-view stereo reconstruction system based on the self-attention mechanism, wherein: the Transformer attention aggregation module utilizes the Transformer's attention mechanism to aggregate the matching cost volume, constructs the deep 3D association between different perspectives, summarizes the values ​​through the epipolar attention score, and determines the final cost volume, thereby achieving high-precision depth map estimation.

[0027] The above-mentioned multi-view stereo reconstruction system based on the self-attention mechanism, wherein: the loss function module uses EMD distance (Earth Mover's Distance) as the loss function to measure the distance between the predicted depth distribution and the real depth map to ensure the accuracy of the depth prediction range, as follows:

[0028] The depth estimation is converted into depth-aware classification for solution, and the prediction distribution P is measured using EMD. i ∈R D Distribution P θ，i ∈R D The EMD distance between the predicted depth distribution and the real depth map is used as the loss function

[0029]

[0030] Among them, inf represents the extreme value, Π(P i , P θ，i ) is the marginal distribution P i and P θ，i The set of all possible distributions, whose marginal distribution is P i , P θ，i Satisfies the following function:

[0031] and

[0032] The present invention also provides a multi-view stereoscopic reconstruction method based on a self-attention mechanism, comprising the following steps:

[0033] S1: First, the shallow feature extraction submodule uses a convolutional neural network (CNN) to extract shallow features from multiple input view images to obtain preliminary feature representations of the images;

[0034] Then, the global representation sub-network uses a lightweight Transformer module to learn the global representation of shallow features. Through the Transformer's attention mechanism, the global similar feature correlation and long-distance dependency in the image are captured. At the same time, the feature pyramid network (FPN) is combined to perform multi-scale feature fusion to focus on global features and local texture details at the same time.

[0035] The local features extracted by the convolutional neural network (CNN) are then fused with the global features captured by the Transformer to obtain a multi-scale feature representation from coarse to fine, providing rich feature information for subsequent depth estimation and 3D reconstruction.

[0036] S2: In a given sampling depth interval, a plane scanning algorithm is used to construct multiple front-to-back parallel planes in the reference camera space, the features of the source view are mapped to the reference camera frustum, and a multi-view matching cost volume is constructed. Through the cross-attention operation, the association between the reference view cost volume and the source view cost volume is calculated to achieve effective fusion of depth information;

[0037] S3: The cost volume regularization module is used to regularize the constructed cost volume to smooth out the noise and outliers in the depth estimation;

[0038] S4: The depth sampling range estimation module uses the inverse depth strategy to estimate the depth sampling range and realize the propagation of the depth map from coarse to fine;

[0039] S5: The Transformer attention aggregation module uses the Transformer attention mechanism to aggregate the matching cost volumes, build deep 3D associations between different perspectives, summarize the values ​​through the epipolar attention scores, and determine the final cost volume, thereby achieving high-precision depth map estimation;

[0040] S6: The loss function module uses EMD distance (Earth Mover's Distance) as the loss function to measure the distance between the predicted depth distribution and the true depth map to ensure the accuracy of the depth prediction range.

[0041] The present invention has the following beneficial effects:

[0042] The multi-view stereo reconstruction system and method based on the self-attention mechanism provided by the present invention effectively solves the problems existing in MVS reconstruction through modules such as multi-scale feature extraction, matching cost volume construction, deep sampling range estimation, and Transformer attention aggregation, and improves the integrity and accuracy of three-dimensional reconstruction. This method has achieved excellent experimental results on multiple data sets and has high practical value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the architecture of a multi-view stereoscopic reconstruction system based on a self-attention mechanism provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of multi-scale feature extraction in a multi-view stereo reconstruction system based on a self-attention mechanism provided by an embodiment of the present invention;

[0045] Figure 3 The following are the demonstration images of the real point cloud of the DTU dataset scan1, the reconstruction using CascadeMvsnet, and the reconstruction using this method;

[0046] Figure 4 The following are the demonstration images of the real point cloud of the DTU dataset scan4, the reconstruction using CascadeMvsnet, and the reconstruction using this method;

[0047] Figure 5 The following are demonstration images of the real point cloud of the DTU dataset scan10, the reconstruction using CascadeMvsnet, and the reconstruction using this method. DETAILED DESCRIPTION

[0048] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.

[0049] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0050] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as a limitation on this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0051] In the description of the present invention, unless otherwise clearly specified and limited, if the term "connection" or the like appears to indicate the connection relationship between components, the term should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0052] Example

[0053] like Figure 1As shown, the multi-view stereo reconstruction system based on the self-attention mechanism provided in this embodiment inherits the advantages of the cascade cost body method of 3DCNN and Cascade-MVSNet in ucsnet, which fully utilizes the ability of 3DCNN to aggregate local context information and saves memory and time. Given a set of N images of different views, including a reference image I0∈R H×W×3 and the source image And their camera internal and external parameter matrices. This embodiment focuses on multi-scale feature extraction and deep correlation of cost volume aggregation under multi-view learning to obtain a larger adaptive receptive field, thereby obtaining higher accuracy and more complete depth results. The network architecture is as follows Figure 1 The main modules of the network proposed in this embodiment are: multi-scale feature extraction module, matching cost volume construction module, depth sampling range estimation module, Transformer attention aggregation module and loss function module.

[0054] Among them, the multi-scale feature extraction module is used to extract the multi-scale features of the image;

[0055] Among them, the matching cost volume construction module is used to construct the multi-view matching cost volume;

[0056] Wherein, the depth sampling range estimation module is used to estimate the depth sampling range;

[0057] Among them, the Transformer attention aggregation module is used to aggregate the matching cost body;

[0058] Among them, the loss function module is used to calculate the distance between the predicted depth distribution and the true depth map.

[0059] Specifically, in this embodiment: the multi-scale feature extraction module includes a shallow feature extraction sub-module and a global representation sub-network composed of a lightweight Transformer module, and a lightweight Transformer-CNN aggregation network is designed. By fusing the global representation of the Transformer and the local feature extraction of the FPN, it can simultaneously focus on the correlation of global similar features and the texture details of local features, and realize multi-scale feature extraction through the fusion of convolutional neural network (CNN) and Transformer.

[0060] Specifically, Figure 2 As shown, in this embodiment: the specific working process of the multi-scale feature extraction module is as follows:

[0061] Firstly, the shallow feature extraction submodule uses a convolutional neural network (CNN) to extract shallow features from the input multi-view images to obtain the preliminary feature representation of the image.

[0062] Then, the global representation sub-network uses a lightweight Transformer module to learn the global representation of shallow features. Through the Transformer's attention mechanism, the global similar feature correlation and long-distance dependency in the image are captured. At the same time, the feature pyramid network (FPN) is combined to perform multi-scale feature fusion to focus on global features and local texture details at the same time.

[0063] The local features extracted by the convolutional neural network (CNN) are then fused with the global features captured by the Transformer to obtain a multi-scale feature representation from coarse to fine, providing rich feature information for subsequent depth estimation and 3D reconstruction.

[0064] in: and Represented as the input and output of feature extraction, bicubic interpolation is used to adjust the size of the position encoding of the visual Transformer to adapt to different image scales. In order to meet F (h) Under different resolutions, the visual Transformer backbone needs to meet two conditions: The attention mechanism and the robustness of position encoding at different scales are used to add the output of the visual Transformer directly to the top-level features of the FPN encoder. First, the four multi-scale features are encoded with a resolution of 1 / 4 of the original image. Then FPN upsamples these multi-scale features, and can obtain coarse-to-fine features through the FPN decoder. From the original resolution To (H, W), the specific formula (1) is as follows:

[0065]

[0066] Specifically, in this embodiment: the matching cost volume construction module uses a plane scanning algorithm to construct a multi-view matching cost volume, and realizes the fusion of depth information through a cross-attention operation.

[0067] Among them, the specific working process of the matching cost body construction module is as follows:

[0068] Within a given sampling depth range, the plane scanning algorithm is used to construct multiple front-to-back parallel planes in the reference camera space, the features of the source view are mapped to the reference camera frustum, and a multi-view matching cost volume is constructed. Through the cross-attention operation, the association between the reference view cost volume and the source view cost volume is calculated to achieve effective fusion of depth information.

[0069] Specifically, the multi-view stereo reconstruction system based on the self-attention mechanism also includes a cost volume regularization module, which is used to regularize the constructed cost volume to smooth the noise and outliers in the depth estimation. This step helps to improve the quality and stability of the depth map.

[0070] Given Using plane scanning, multiple parallel planes are created in the reference camera space. Specifically, in the camera internal parameters and the transformation parameters from the source view to the reference view Under the condition of , the characteristics of the source view can be mapped into the reference camera frustum:

[0071]

[0072] Among them, d j is the j-th hypothesized depth of pixel pr in the reference feature, ps i , j is the pixel corresponding to the jth assumed depth in the i-th source feature. After the transformation operation, N-1 source view cost volumes are constructed Where D is the total number of assumed depths.

[0073] Compared with the method of directly using variance-based cost volume aggregation, the purpose of this embodiment is to construct the depth correlation of cost volume aggregation under different perspectives. First, Transformer uses the reference perspective cost volume as the query q feature to match the source perspective cost volume feature key k along the epipolar line, thereby enhancing the corresponding depth value v, generating attention guidance, and aggregating feature cost volumes from different views. Under the epipolar line constraint, the attention calculates the correlation between the reference perspective cost volume and the source perspective cost volume. First, the depth information is restored through the homography transformation. According to the transformation operation of equation (2), the assumed depth position of the query feature q is projected onto the epipolar line of the source image to obtain the source volume feature. Therefore, deep associations of query features are constructed through criss-cross attention operations.

[0074]

[0075] where v i ∈R C×D is achieved by superimposing along the depth dimension calculate.

[0076] Use grouped correlation to measure the visual similarity between reference features and source objects in an efficient way:

[0077]

[0078] in is the g-th group feature of vi. yes The g-th group feature of .

[0079] <.,.> is the inner product. Stacking along the channel dimension gives k i ∈R G×D Finally, the epipolar attention score A i The values ​​are summed up to determine the final cost volume:

[0080]

[0081] The above-mentioned multi-view stereo reconstruction system based on the self-attention mechanism, wherein: the depth sampling range estimation module adopts an inverse depth strategy to estimate the depth sampling range to achieve coarse-to-fine depth map propagation.

[0082] The depth sampling range estimation module adopts a flexible depth sampling strategy to ensure that the pixels on the source view level line are as evenly distributed as possible through inverse depth sampling. In each stage, the depth prediction of the previous stage is used as the center to uniformly generate depth hypotheses within the assumed depth range, realizing the propagation of depth maps from coarse to fine.

[0083] A single strategy is used to estimate the depth range at each stage, which causes the neighboring information of each pixel to be ignored and the pixel distance to be uneven. To solve this problem, the depth value is obtained by simply inverting the depth strategy without adding any neural network modules. Figure 1 As shown, the number of pyramid layers in the framework is flexible, such as training 4 different layers and evaluating at any number of stages. This embodiment sets four stages, where the input resolutions of the four stages are:

[0084]

[0085] In order to solve the problem that the distance between pixels corresponding to the epipolar line of the imaging plane is not uniform when sampling the depth space, and to make the pixels distributed on the source view level line as uniform as possible, in the first stage, a discrete inverse sampling method is used to select the depth hypothesis plane during depth sampling to achieve equidistant sampling in the pixel space. In order to achieve coarse-to-fine depth map propagation, the depth hypothesis of each stage is centered on the depth prediction of the previous stage, and a d is uniformly generated within the assumed depth range. j Assumption.

[0086]

[0087] where d min and d max are the minimum and maximum depth values ​​of the reference image, respectively.

[0088] Specifically, in this embodiment: the Transformer attention aggregation module utilizes the Transformer's attention mechanism to aggregate matching cost volumes, constructs deep 3D associations between different perspectives, summarizes the values ​​through the polar attention scores, and determines the final cost volume, thereby achieving high-precision depth map estimation.

[0089] Among them, the loss function module uses EMD distance (Earth Mover's Distance) as the loss function to measure the distance between the predicted depth distribution and the actual depth map to ensure the accuracy of the depth prediction range, as follows:

[0090] In order to ensure the accuracy of the depth prediction range, the depth estimation is converted into a depth-aware classification P i ∈R D Solution, P θ，i ∈R D The distance between the predicted distribution and the true distribution is measured, and the EMD distance between the predicted depth distribution and the true depth map is used as the loss function.

[0091]

[0092] Among them, inf represents the extreme value, ∏(P i , P θ，i ) is the marginal distribution P i and P θ，i The set of all possible distributions, whose marginal distribution is P i , P θ，i satisfy:

[0093] and

[0094]

[0095] The present invention also provides a multi-view stereoscopic reconstruction method based on a self-attention mechanism, comprising the following steps:

[0096] S1: First, the shallow feature extraction submodule uses a convolutional neural network (CNN) to extract shallow features from multiple input view images to obtain preliminary feature representations of the images.

[0097] Then, the global representation sub-network uses a lightweight Transformer module to learn the global representation of shallow features. Through the Transformer's attention mechanism, the global similar feature correlation and long-distance dependency in the image are captured. At the same time, the feature pyramid network (FPN) is combined to perform multi-scale feature fusion to focus on global features and local texture details at the same time.

[0098] The local features extracted by the convolutional neural network (CNN) are then fused with the global features captured by the Transformer to obtain a multi-scale feature representation from coarse to fine, providing rich feature information for subsequent depth estimation and 3D reconstruction.

[0099] S2: In a given sampling depth interval, a plane scanning algorithm is used to construct multiple front-to-back parallel planes in the reference camera space, the features of the source view are mapped to the reference camera frustum, and a multi-view matching cost volume is constructed. Through the cross-attention operation, the association between the reference view cost volume and the source view cost volume is calculated to achieve effective fusion of depth information;

[0100] S3: The cost volume regularization module is used to regularize the constructed cost volume to smooth out the noise and outliers in the depth estimation;

[0101] S4: The depth sampling range estimation module uses the inverse depth strategy to estimate the depth sampling range and realize the propagation of the depth map from coarse to fine;

[0102] S5: The Transformer attention aggregation module uses the Transformer attention mechanism to aggregate the matching cost volumes, build deep 3D associations between different perspectives, summarize the values ​​through the epipolar attention scores, and determine the final cost volume, thereby achieving high-precision depth map estimation;

[0103] S6: The loss function module uses EMD distance (Earth Mover's Distance) as the loss function to measure the distance between the predicted depth distribution and the true depth map to ensure the accuracy of the depth prediction range.

[0104] Experimental results and analysis

[0105] Dataset

[0106] The current MVS dataset provides true value data in point cloud or mesh format. MVS has the following commonly used training and evaluation datasets: DTU dataset It consists of 124 scanned scene data under 7 different light intensities as well as reference points and normal vector information. High-resolution RGB images of size 1600×1200. In addition to the camera pose containing the camera image, it provides ground truth point clouds with normal information. The dataset information used in the experiment is shown in Table 1:

[0107] Table 1 DTU dataset information

[0108]

[0109] Experimental environment

[0110] This example is based on the Pytorch framework. It is first trained on the DTU training set for 9 epocs and optimized with the Adam algorithm. The system is trained on a single NVIDIA RTX3090 graphics card with a batch size of 2 on each GPU. The number of depth samples in multiple stages is 8, 8, 8, and 4, respectively, with a depth range of 425-935 mm. The number of input images is set to 4 using the original image resolution. The batch size on each GPU is 2. The initial learning rate is 0.001, and it is decayed at the 6th, 8th, and 9th epochs, respectively, with a temperature parameter te=2.

[0111] Comparison results of DTU dataset

[0112] This embodiment uses official evaluation indicators to evaluate the proposed method on the DTU dataset. Thanks to the cost volume aggregation module mechanism of Transformer, the system of this embodiment is able to generate denser and more complete point clouds and retain more details. Quantitative comparisons are shown in Table 2. Accuracy and completeness are two official metrics. The accuracy metric is the distance between the 3D reconstructed point cloud and the real point cloud obtained by the structured light scanner, while the completeness metric is the distance between the real point cloud and the 3D reconstructed point cloud. Comprehensiveness is the average of accuracy and completeness, which represents the overall performance of the system. The system of this embodiment is competitive in terms of accuracy and completeness, and is generally much better than some known methods, as shown in Table 2:

[0113] Table 2 Comparison of main evaluation indicators with other methods (the lower the better)

[0114]

[0115] Figure 3-5 The figure shows a qualitative comparison of the scene reconstruction between the method of this embodiment and Cas-MVSNet on the instance of the DTU dataset. The real point cloud obtained by scanning the dataset is incomplete. Compared with the Cas-MVSNet method, the point cloud in the yellow box in the weak texture area is relatively dense and more complete. This effect is related to the multi-scale feature extraction module and Transformer realizing feature fusion through attention operation, which enhances the globality, retains more spatial information and captures more feature information.

[0116] Ablation experiment

[0117] This embodiment conducts extensive ablation studies to verify the enhancement effect brought by the proposed module. This embodiment uses the DTU training set to train the method of this embodiment, and performs all tests on the DTU evaluation set. The training setting is the same as the above experimental environment.

[0118] Table 3 Ablation experiment on DTU dataset (the lower the better)

[0119]

[0120]

[0121] in conclusion

[0122] The work of this embodiment is to complete the 3D scene reconstruction from multiple RGB images. In order to solve the problem of general feature extraction effect and poor correlation between cost bodies, a multi-scale feature extraction module is designed to focus on the global similar feature correlation and the texture details of local features. The cost body aggregation module is implemented through the attention mechanism to construct deep 3D associations from different perspectives. The model is generalized to various resolutions using efficient multi-scale training, and a smooth depth map and a clear point cloud without abnormalities are obtained. Experiments conducted on the DTU dataset show that the method of this embodiment is feasible. Further work may focus on reducing memory consumption and shortening running time.

[0123] In summary, the multi-view stereoscopic reconstruction system and method based on the self-attention mechanism provided in this embodiment has the following advantages:

[0124] The multi-view stereo reconstruction system and method based on the self-attention mechanism provided by the present invention effectively solves the problems existing in MVS reconstruction through modules such as multi-scale feature extraction, matching cost volume construction, deep sampling range estimation, and Transformer attention aggregation, and improves the integrity and accuracy of three-dimensional reconstruction. This method has achieved excellent experimental results on multiple data sets and has high practical value and promotion prospects.

[0125] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-view stereo reconstruction system based on self-attention mechanism, characterized in that: include: Multi-scale feature extraction module, used to extract multi-scale features of images; Matching cost volume construction module, used to construct multi-view matching cost volume; A depth sampling range estimation module, used to estimate the depth sampling range; Transformer attention aggregation module, used to aggregate matching cost bodies; The loss function module is used to calculate the distance between the predicted depth distribution and the true depth map.

2. The multi-view stereo reconstruction system based on self-attention mechanism according to claim 1, characterized in that: The multi-scale feature extraction module includes a shallow feature extraction submodule and a global representation subnetwork, and realizes multi-scale feature extraction through the fusion of convolutional neural network (CNN) and Transformer.

3. The multi-view stereo reconstruction system based on self-attention mechanism according to claim 2, characterized in that: The specific working process of the multi-scale feature extraction module is as follows: Firstly, the shallow feature extraction submodule uses a convolutional neural network (CNN) to extract shallow features from the input multi-view images to obtain the preliminary feature representation of the image. Then, the global representation sub-network uses a lightweight Transformer module to learn the global representation of shallow features. Through the Transformer's attention mechanism, the global similar feature correlation and long-distance dependency in the image are captured. At the same time, the feature pyramid network (FPN) is combined to perform multi-scale feature fusion to focus on global features and local texture details at the same time. The local features extracted by the convolutional neural network (CNN) are then fused with the global features captured by the Transformer to obtain a multi-scale feature representation from coarse to fine, providing rich feature information for subsequent depth estimation and 3D reconstruction.

4. The multi-view stereo reconstruction system based on self-attention mechanism according to claim 3, characterized in that: The matching cost volume construction module uses a plane scanning algorithm to construct a multi-view matching cost volume and realizes the fusion of depth information through a cross-attention operation.

5. The multi-view stereo reconstruction system based on self-attention mechanism according to claim 4, characterized in that: The specific working process of the matching cost body construction module is as follows: Within a given sampling depth range, the plane scanning algorithm is used to construct multiple front-to-back parallel planes in the reference camera space, the features of the source view are mapped to the reference camera frustum, and a multi-view matching cost volume is constructed. Through the cross-attention operation, the association between the reference view cost volume and the source view cost volume is calculated to achieve effective fusion of depth information.

6. The multi-view stereo reconstruction system based on self-attention mechanism according to claim 5, characterized in that: A cost volume regularization module is also included, and the cost volume regularization module is used to perform regularization processing on the constructed cost volume to smooth noise and outliers in the depth estimation.

7. The multi-view stereo reconstruction system based on self-attention mechanism according to claim 6, characterized in that: The depth sampling range estimation module adopts an inverse depth strategy to perform depth sampling range estimation to achieve coarse-to-fine depth map propagation.

8. The multi-view stereo reconstruction system based on self-attention mechanism according to claim 7, characterized in that: The Transformer attention aggregation module uses the Transformer's attention mechanism to aggregate matching cost volumes, construct deep 3D associations between different perspectives, summarize the values ​​through the epipolar attention scores, and determine the final cost volume, thereby achieving high-precision depth map estimation.

9. The multi-view stereo reconstruction system based on self-attention mechanism according to claim 8, characterized in that: The loss function module uses EMD distance as the loss function to measure the distance between the predicted depth distribution and the real depth map to ensure the accuracy of the depth prediction range, as follows: The depth estimation is converted into depth-aware classification for solution, and the prediction distribution P is measured using EMD. i ∈R D Distribution P θ，i ∈R D The distance between the predicted depth distribution and the real depth map is used as the loss function. Among them, inf represents the extreme value, Π(P i , P θ，i ) is the marginal distribution P i and P θ，i The set of all possible distributions, whose marginal distribution is P i , P θ，i Satisfies the following function:

10. A multi-view stereo reconstruction method based on self-attention mechanism according to claim 9, characterized in that: The steps include: S1: First, the shallow feature extraction submodule uses a convolutional neural network (CNN) to extract shallow features from multiple input view images to obtain preliminary feature representations of the images. Then, the global representation sub-network uses a lightweight Transformer module to learn the global representation of shallow features. Through the Transformer's attention mechanism, the global similar feature correlation and long-distance dependency in the image are captured. At the same time, the feature pyramid network (FPN) is combined to perform multi-scale feature fusion to focus on global features and local texture details at the same time. The local features extracted by the convolutional neural network (CNN) are then fused with the global features captured by the Transformer to obtain a multi-scale feature representation from coarse to fine, providing rich feature information for subsequent depth estimation and 3D reconstruction. S2: In a given sampling depth interval, a plane scanning algorithm is used to construct multiple front-to-back parallel planes in the reference camera space, the features of the source view are mapped to the reference camera frustum, and a multi-view matching cost volume is constructed. Through the cross-attention operation, the association between the reference view cost volume and the source view cost volume is calculated to achieve effective fusion of depth information; S3: The cost volume regularization module is used to regularize the constructed cost volume to smooth out the noise and outliers in the depth estimation; S4: The depth sampling range estimation module uses the inverse depth strategy to estimate the depth sampling range and realize the propagation of the depth map from coarse to fine; S5: The Transformer attention aggregation module uses the Transformer attention mechanism to aggregate the matching cost volumes, build deep 3D associations between different perspectives, summarize the values ​​through the epipolar attention scores, and determine the final cost volume, thereby achieving high-precision depth map estimation; S6: The loss function module uses EMD distance as the loss function to measure the distance between the predicted depth distribution and the true depth map to ensure the accuracy of the depth prediction range.

Images