A multi-view 3D reconstruction method and system based on ViT
By combining the feature extraction methods of ViT and FPN with the depth estimation algorithm of dynamic multi-scale training and temperature coefficient control, the problem of insufficient feature robustness in multi-view 3D reconstruction is solved, and high-precision reconstruction is achieved in reflective and weak texture environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2022-07-26
- Publication Date
- 2026-07-24
AI Technical Summary
Existing multi-view 3D reconstruction algorithms struggle to extract robust feature representations in reflective and weak texture conditions, resulting in insufficient reconstruction accuracy.
A multi-view 3D reconstruction method based on ViT is adopted, which combines ViT and Feature Pyramid Network (FPN) for feature extraction, and uses a depth estimation algorithm with dynamic multi-scale strategy and temperature coefficient control for depth map prediction and point cloud reconstruction.
It improves the robustness of features in reflective and weakly textured conditions, achieves more efficient training convergence and more accurate depth estimation, and has achieved excellent results, especially on the authoritative dataset Tanks & Temples.
Smart Images
Figure CN115423927B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of 3D visual reconstruction technology, and in particular to a multi-view 3D reconstruction method and system based on ViT. Background Technology
[0002] Multi-View Stereo (MVS) reconstruction achieves high-precision 3D reconstruction using 2D images from multiple perspectives. MVS typically involves several steps, such as camera pose determination, depth estimation, depth map fusion, and dense point cloud reconstruction. The core step is estimating the depth map for each image given the camera parameters, preparing for subsequent depth map fusion and dense point cloud reconstruction.
[0003] Many traditional algorithms, which match low-level features of 2D images, struggle to handle issues such as occlusion and lighting. In recent years, deep learning methods have played an increasingly important role in multi-view depth estimation.
[0004] The main steps of deep learning-based MVS include feature extraction, cost volume formulation, and cost volume regularization. Most current research focuses on the construction and learning of the cost volume. However, for deep learning-based MVS, the ability to extract robust feature representations is the most important factor determining the algorithm's final generalization ability.
[0005] However, most MVS algorithms use Feature Pyramid Network (FPN) to extract features, and it is still difficult to learn robust feature representations in situations such as reflectivity and weak texture.
[0006] In the field of computer vision, CNNs have become the dominant model for visual tasks. Subsequently, increasingly efficient structures have emerged, and computer vision and natural language processing are becoming more and more convergent. Using Transformers to complete visual tasks has become a new research direction to reduce structural complexity and explore scalability and training efficiency. The emergence of the Vision Transformer (ViT) cleverly applies the Transformer to image classification tasks. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a multi-view 3D reconstruction method and system based on ViT, which can improve the robustness of feature extraction in MVS, especially in cases of reflection and weak texture to obtain robust feature representation.
[0008] The objective of this invention can be achieved through the following technical solutions:
[0009] According to a first aspect of the present invention, a multi-view 3D reconstruction method based on ViT is provided, the method comprising the following steps:
[0010] Step S1: Obtain multi-view 2D images of the specified scene and the corresponding camera parameters, and divide them into training set and test set;
[0011] Step S2: Select reference images and their nearest neighbors from the training set as source images; construct a feature extraction model that integrates ViT and Feature Pyramid Network (FPN) to extract features from the reference images and source images respectively;
[0012] Step S3: Based on the homography changes of camera parameters, transform the source image features extracted in step S2 into the pose of the source image, construct the cost matrix, and optimize the depth map predicted by the cost matrix at different levels based on the classification loss function.
[0013] Step S4: Train the ViT-based multi-view reconstruction MVS model constructed in steps S2 to S3 using a dynamic multi-scale strategy;
[0014] Step S5: Select reference images and corresponding source images from the test set, and use the ViT-based multi-view reconstruction MVS model pre-trained in Step S4 to perform feature extraction, cost matrix construction and learning in sequence to predict the depth map; and use a depth fusion algorithm to perform deep fusion and point cloud reconstruction on the predicted multi-view depth map.
[0015] Preferably, the training set in step S1 is the DTU dataset, and the test set is the Tanks & Temples dataset; each image in the DTU dataset is equipped with a corresponding depth map, camera intrinsic parameters, and extrinsic parameters.
[0016] Preferably, step S2 includes the following sub-steps:
[0017] Step S21: Select reference images from the training set, and select the k nearest neighbor images of the reference images as source images;
[0018] Step S22: Perform ViT feature extraction and Convolutional Neural Network (CNN) feature extraction on the downsampled reference image and source image, respectively; wherein, the ViT feature extraction is either single-scale ViT feature extraction based on the MVSFormer-P model or multi-scale ViT feature extraction based on the MVSFormer-H model, and the corresponding output image features are F (p) and F (h) The features extracted using a convolutional neural network (CNN) are F.(0) ;
[0019] Step S23, in feature F (0) Based on this, the feature F extracted by single-scale ViT is superimposed. (p) or multi-scale ViT feature extraction feature F (h) The feature pyramid network (FPN) is used to fuse multi-stage features to obtain F. (l) .
[0020] Preferably, the MVSFormer-P model in step S2 includes a Transoformer pre-trained DINO module based on self-supervised distillation and an attention-gated linear unit (GLU) for dimensionality reduction of the features output by the DINO module.
[0021] The MVSFormer-H model includes a Transformer pre-trained Twins module based on multi-scale attention.
[0022] Preferably, the single-scale ViT feature extraction based on the MVSFormer-P model has its weights fixed and is not trained during training; the multi-scale ViT feature extraction based on the MVSFormer-H model fine-tunes the MVSFormer-H model during training.
[0023] Preferably, step S3 includes the following sub-steps:
[0024] 1) Initialize the depth interval of the model for each stage. Where, d j Let be the image depth corresponding to candidate plane j;
[0025] 2) Transform the features of the source image to the reference viewpoint using the following expression:
[0026]
[0027] In the formula, p is any coordinate point in the 2D reference image I0, and p′ is the coordinate point of coordinate point p after homography transformation; K0, K j For camera intrinsic parameters, R 0→i Let t be a rotation matrix. 0→i Let d be the offset vector. j Image depth;
[0028] 3) Calculate the similarity between the transformed source features and the reference features using the grouped pooling features to obtain the corresponding cost matrix; the similarity calculation expression is:
[0029]
[0030] in, These represent the reference features and source features after group pooling, respectively.
[0031] 4) Use a 2D CNN to obtain the weights for each pixel based on the similarity. Feature fusion is performed based on weights, expressed as follows:
[0032]
[0033] In the formula, C i For similarity, W i For pixel weights, d j Image depth;
[0034] 5) Input the fused feature C into the 3D U-net to obtain the final output feature.
[0035] 6) Using the cross-entropy loss function, the cost matrix of different levels is optimized hierarchically to predict the depth map. The expression for the predicted depth map is:
[0036]
[0037] In the formula, d j Image depth, The output feature obtained in step 5); D is the number of candidate depth planes.
[0038] Preferably, step S4 specifically involves: using gradient-accumulated dynamic batching combined with dynamic resolution training to divide the input of a large image into several smaller sub-batches, accumulating the gradient results of each sub-batch, and optimizing the averaged unified parameters; wherein the resolution is within a set range, and the aspect ratio of the image is randomly selected from the set range.
[0039] Preferably, in step S5, reference images and corresponding source images are selected from the test set, and the ViT-based multi-view reconstruction MVS model pre-trained according to steps S2-S4 is used to sequentially perform feature extraction, cost matrix construction and learning, and depth map prediction. Specifically, this includes the following sub-steps:
[0040] 1) Select reference images and source images, and use the multi-view reconstruction MVS model pre-trained in steps S2 to S4 to extract features and construct and learn the cost matrix in sequence.
[0041] 2) A temperature-controlled depth estimation algorithm is used to predict the final depth map. The specific expression is as follows:
[0042]
[0043] In the formula, softmax is the normalized exponential function, and t is the temperature coefficient used to control the expected degree of aggregation; when t=1, the above formula is equivalent to the regression expectation, and when t=∞, the above formula is equivalent to argmax. The output features obtained in step S3; d j Image depth.
[0044] Preferably, in step S5, a deep fusion algorithm is used to perform deep fusion and point cloud reconstruction on the predicted multi-view depth map. Specifically, this involves using a Gipuma-based deep fusion algorithm to perform deep fusion and point cloud reconstruction on the multi-view depth map, including the following sub-steps:
[0045] 1) After obtaining the depth map for each viewpoint, filter the depth map using a set confidence threshold;
[0046] 2) Project the 2D coordinate points onto the 3D space based on the depth map d0 from the reference viewpoint, and then project them onto the depth map d0 from the target viewpoint. i Project it back and calculate the reprojection consistency coefficient;
[0047] 3) When the number of viewing angles with a reprojection consistency coefficient less than the set value reaches the set value, retain the point cloud.
[0048] According to a second aspect of the present invention, a ViT-based multi-view 3D reconstruction system is provided, employing any one of the methods described above, the system comprising:
[0049] The feature extraction module is used to extract features from images using a deep network that integrates ViT and Feature Pyramid Network (FPN).
[0050] The cost matrix module is used to perform homography transformation on the extracted features, and to construct the cost matrix and optimize the depth map predicted by the cost matrix at different levels.
[0051] The dynamic multi-scale strategy training module is used for dynamic multi-scale training of the multi-view reconstruction MVS model constructed by the feature extraction module and the cost matrix module.
[0052] The multi-view 3D reconstruction module is used to perform deep fusion and point cloud reconstruction on the predicted multi-view depth map using a deep fusion algorithm.
[0053] Compared with the prior art, the present invention has the following advantages:
[0054] 1) This invention provides a multi-view 3D reconstruction method based on ViT, which enhances the robustness of 2D features by using pre-trained ViT to assist in feature encoding and extraction.
[0055] 2) The adoption of an efficient dynamic multi-scale training strategy accelerates training convergence;
[0056] 3) A depth estimation method based on temperature coefficient is adopted to obtain accurate depth estimation results while maintaining reliable confidence.
[0057] 4) The method of this invention has advantages in the reconstruction of weak textures and reflections, and has achieved the first place on the authoritative and challenging 3D reconstruction dataset - Tanks & Temples dataset. Attached Figure Description
[0058] Figure 1 This is a schematic diagram of the depth map estimation system of the present invention; wherein, Figure 1 (A) is the feature extraction module. Figure 1 (B) is the cost matrix module;
[0059] Figure 2 This is a schematic diagram illustrating the classification and regression results in multi-stage depth estimation; among them, Figure 2 (a) Schematic diagram of setting up multi-stage confidence for classification and regression; Figure 2 (b) is a schematic diagram of the regression and classification results of DTU scan77;
[0060] Figure 3 This diagram illustrates the results of MVS depth estimation using different pre-trained models; where, Figure 3 (a)~ Figure 3 (e) shows the depth estimation results for Baseline, Baselin+ResNet50, MVSFormer-P, MVSFOremer-H and Ground-Truth, respectively. Detailed Implementation
[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0062] Example
[0063] Step S1: Obtain multi-view 2D images of the specified scene and the corresponding camera parameters, and divide them into training set and test set;
[0064] The training set uses the DTU dataset, and the test set uses the Tanks & Temples dataset.
[0065] The DTU dataset consists of two sets of images: a training set containing 79 scenes, each with 49 different angles and 7 different lighting conditions per angle, totaling 27,097 images; and a test set containing 22 scenes, each with 49 different angles and 7 different lighting conditions per angle, but using only the brightest lighting condition, consisting of 1,078 images. Each image is accompanied by a corresponding depth map and camera intrinsic and extrinsic parameters.
[0066] The Tanks & Temples dataset consists of an intermediate set and an advanced set. The intermediate set includes eight scenes: "Family," "Francis," "Horse," "Lighthouse," "M60," "Panther," "Playground," and "Train." The advanced set includes six scenes: "Auditorium," "Ballroom," "Courtroom," "Museum," "Palace," and "Temple." Each scene contains 200-300 images. Each image does not have a corresponding depth map.
[0067] Step S2: Select one image from the training set as a reference image and its four nearest neighbor images as source images, and extract their depth features respectively, specifically as follows:
[0068] 1) Select one image as the reference image and four neighboring images as the source images;
[0069] During the source image selection process, images with an angle difference of approximately 5° from the reference image should be chosen whenever possible; the angle difference is calculated using camera extrinsic parameters.
[0070] 2) A feature extraction model combining ViT (Vision Transformer) and Feature Pyramid Network (FPN) is used to extract features from the selected reference and source images, such as... Figure 1 As shown in (A), after passing through the Feature Pyramid Network (FPN), image features are obtained in four stages.
[0071] ViT (Vision Transformer) includes single-scale ViT and multi-scale ViT; among them, single-scale ViT and multi-scale ViT use Transformer pre-trained DINO based on self-supervised distillation and Transformer pre-trained Twins based on multi-scale attention as pre-trained ViT, respectively, to construct MVSFormer-P and MVSFormer-H.
[0072] For MVSFormer-P, the output feature is F(p); for the Transformer pre-trained DINO based on self-supervised distillation, a gated linear unit (GLU) is used for feature dimensionality reduction. The calculation process of GLU is as follows:
[0073]
[0074] In the formula, F dino This represents the characteristics of DINO's output; A is DINO's attention feature, which has G groups. It is DINO attention feature concatenation, mean-averaging of G groups; H and W represent the height and width of the image; ConvBN is convolution + BatchNorm layer; Swish represents the activation function Swish(x) = x·sigmoid(x), ⊙ represents element-wise multiplication, [;] represents feature dimension concatenation, C p These are the features after dimensionality reduction;
[0075] Then, after using two layers of deconvolution to further upsample the features, F is obtained. (p) .
[0076] For MVSFormer-H, the output feature is F (h) The image features are fused using a Feature Pyramid Network (FPN) across four layers of different scales, with resolutions of (1 / 8, 1 / 16, 1 / 32, 1 / 64) of the original image resolution. The final F... (h) It can be written as:
[0077]
[0078] In the formula, H and W represent the height and width of the original image;
[0079] To address the issue of high memory consumption in single-scale ViT, its weights are fixed and not trained. In contrast, multi-scale ViT consumes less memory, so MVSFormer-H is fine-tuned during training to achieve better results.
[0080] Final feature F (0) Add F (p) or F (h) Furthermore, features at other scales are obtained as the output of feature extraction.
[0081] Step S3: Based on the homography changes of camera parameters, transform the source image features extracted in Step S2 into the pose of the source image, construct a cost matrix, and optimize the depth map predicted by the cost matrix at different levels based on the classification loss function; the specific process is as follows:
[0082] 1) For each stage model, initialize a set of depth intervals.
[0083] The superscript for level 1 has been omitted here for simplification.
[0084] 2) Transform the features of the original image to the reference viewpoint;
[0085] Suppose there is a point p with coordinates in a 2D reference image I0, and the camera intrinsic parameters K0 and K are known. j Rotation matrix R 0→i and offset vector t 0→i At a depth of d j The coordinates of the transformed point p′ on the plane are:
[0086]
[0087] 3) Use the features obtained from group pooling to calculate the similarity between the transformed source features and the reference features:
[0088]
[0089] In the formula, These represent the reference features and source features after group pooling, respectively.
[0090] 4) Use a set of 2D CNNs to obtain the weights for each pixel based on the similarity. And based on the weighted fusion features, the specific expression is:
[0091]
[0092] 5) The fused result will be used as input to 3D U-net to obtain the final output.
[0093] 6) Use classification loss to hierarchically optimize the cost matrix of different levels to predict the depth map.
[0094] The classification loss uses cross-entropy loss to optimize the final depth map, and argmax is used to obtain the final depth map output, as shown in the expression:
[0095]
[0096] In the formula, d j Image depth, The output features obtained in step S3 are represented by D, which represents the number of candidate depth planes.
[0097] Depth estimation can be viewed as a classification problem with D depth planes, where each pixel selects a target category from D candidate depth planes for classification. (Reference) Figure 2This method can achieve better confidence levels to filter out flying points.
[0098] Step S4: Train the MVS model using a dynamic multi-scale strategy:
[0099] The ViT model is very sensitive to the resolution of the input image, and training with the same resolution will lead to overfitting; while the test set of MVS is generally different high-resolution images (1200x1600~1080x1920).
[0100] This invention dynamically trains using images with a resolution range of 512 to 1280, with the aspect ratio of the images randomly selected from 0.67 to 0.8;
[0101] Since large image sizes tend to consume more GPU memory, typical dynamic size training requires optimization using the smallest batch size, based on the maximum resolution. However, small batch sizes often lead to unstable optimization and slow convergence.
[0102] This invention uses gradient accumulation-based dynamic batching combined with dynamic resolution training to achieve large batch optimization and faster convergence. The specific process is as follows: by dividing the large image input into several smaller sub-batches, accumulating the gradient results of each sub-batch, and optimizing the averaged unified parameters, the process of large batch optimization is simulated.
[0103] Step S5: Select reference images and corresponding source images from the test set, and use the ViT-based multi-view reconstruction MVS model pre-trained according to steps S2-S3 to perform feature extraction, cost matrix construction and learning in sequence to predict the depth map; and use a depth fusion algorithm to perform deep fusion and point cloud reconstruction on the predicted multi-view depth map.
[0104] Unlike steps S2 to S3, in this embodiment, four images are selected from the DTU dataset as source images, and nine images are selected from the Tanks & Temples dataset as source images.
[0105] Another difference is that a temperature-controlled depth estimation algorithm is used to predict depth maps, obtaining more accurate depth results through mathematical expectation while maintaining reliable confidence levels; the specific depth estimation expression is as follows:
[0106]
[0107] In the formula, softmax is the normalized exponential function, and t is the temperature coefficient used to control the expected degree of aggregation; when t=1, the above formula is equivalent to the regression expectation, and when t=∞, the above formula is equivalent to argmax. The output features obtained in step S3; d j Image depth.
[0108] In this embodiment, {t 1 , t 2 , t 3 , t 4} = {5, 2.5, 1.5, 1}. That is, in the early stages, the system tends to favor classification, while in the later stages, it tends to favor regression.
[0109] In this embodiment, the Gipuma deep fusion method is used to perform deep fusion and point cloud generation on multi-view depth maps, specifically including the following sub-steps:
[0110] 1) Filter the depth map for each viewpoint with a confidence threshold of 0.5;
[0111] 2) Project the 2D coordinate points onto the 3D space based on the depth map d0 from the reference viewpoint, and then project them onto the depth map d0 from the target viewpoint. i Project it back and calculate the reprojection consistency coefficient;
[0112] 3) When there are two or more viewpoints with a reprojection consistency coefficient of less than 0.1, the point cloud can be retained; finally, the target point cloud is obtained through this deep fusion method.
[0113] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A multi-view 3D reconstruction method based on ViT, characterized in that, The method includes the following steps: Step S1: Obtain multi-view 2D images of the specified scene and the corresponding camera parameters, and divide them into training set and test set; Step S2: Select reference images and their nearest neighbors from the training set as source images; construct a feature extraction model that integrates ViT and Feature Pyramid Network (FPN) to extract features from the reference images and source images respectively, including the following sub-steps: Step S21: Select reference images from the training set, and select the k nearest neighbor images of the reference images as source images; Step S22: Perform ViT feature extraction and Convolutional Neural Network (CNN) feature extraction on the downsampled reference image and source image, respectively; wherein, the ViT feature extraction is either single-scale ViT feature extraction based on the MVSFormer-P model or multi-scale ViT feature extraction based on the MVSFormer-H model, and the corresponding output image features are respectively and The features extracted using a convolutional neural network (CNN) are: ; Step S23, in the features Based on this, features extracted by single-scale ViT are superimposed. or features extracted by multi-scale ViT feature extraction Multi-stage features are obtained by using Feature Pyramid Network (FPN) fusion. ; Step S3: Based on the homography changes of camera parameters, transform the source image features extracted in step S2 into the pose of the source image, construct the cost matrix, and optimize the depth map predicted by the cost matrix at different levels based on the classification loss function. Step S4: Train the ViT-based multi-view reconstruction MVS model constructed in steps S2-S3 using a dynamic multi-scale strategy; Step S5: Select reference images and corresponding source images from the test set, and use the ViT-based multi-view reconstruction MVS model pre-trained in Step S4 to perform feature extraction, cost matrix construction and learning in sequence to predict the depth map; and use a depth fusion algorithm to perform deep fusion and point cloud reconstruction on the predicted multi-view depth map.
2. The multi-view 3D reconstruction method based on ViT according to claim 1, characterized in that, The training set in step S1 is the DTU dataset, and the test set is the Tanks & Temples dataset; each image in the DTU dataset is equipped with a corresponding depth map, camera intrinsic parameters, and extrinsic parameters.
3. The multi-view 3D reconstruction method based on ViT according to claim 1, characterized in that, The MVSFormer-P model in step S2 includes a Transformer pre-trained DINO module based on self-supervised distillation and an attention-gated linear unit (GLU) for dimensionality reduction of the features output by the DINO module. The MVSFormer-H model includes a Transformer pre-trained Twins module based on multi-scale attention.
4. The multi-view 3D reconstruction method based on ViT according to claim 3, characterized in that, The single-scale ViT feature extraction based on the MVSFormer-P model has its weights fixed during training and is not trained; the multi-scale ViT feature extraction based on the MVSFormer-H model has the MVSFormer-H model fine-tuned during training.
5. The multi-view 3D reconstruction method based on ViT according to claim 1, characterized in that, Step S3 includes the following sub-steps: 1) Initialize the depth interval of the model for each stage. ;in, Candidate plane Corresponding image depth; 2) Transform the features of the source image to the reference viewpoint using the following expression: In the formula, 2D reference image Any coordinate point in the, coordinate point The coordinates of the point should be transformed; For camera internal parameters, For rotation matrix, For offset vectors, Image depth; 3) Calculate the similarity between the transformed source features and the reference features using the grouped pooling features to obtain the corresponding cost matrix; the similarity calculation expression is: in, These represent the reference features and source features after group pooling, respectively. 4) Use a 2D CNN to obtain the weights for each pixel based on the similarity. And feature fusion is performed based on weights, expressed as: In the formula, For similarity, For pixel weights, Image depth; 5) Combine the features The input is fed into a 3D U-net to obtain the final output features. ; 6) Using the cross-entropy loss function, the cost matrix of different levels is optimized hierarchically to predict the depth map. The expression for the predicted depth map is: In the formula, Image depth, The output features obtained in step 5); The number of candidate depth planes.
6. The multi-view 3D reconstruction method based on ViT according to claim 1, characterized in that, Step S4 specifically involves: using gradient-accumulated dynamic batching combined with dynamic resolution training to divide the input of a large image into several smaller sub-batches, accumulating the gradient results of each sub-batch, and optimizing the averaged unified parameters; wherein the resolution is within a set range, and the aspect ratio of the image is randomly selected from the set range.
7. The multi-view 3D reconstruction method based on ViT according to claim 1, characterized in that, In step S5, reference images and corresponding source images are selected from the test set. The ViT-based multi-view reconstruction MVS model, pre-trained according to steps S2-S4, is used to sequentially perform feature extraction, cost matrix construction and learning, and depth map prediction. Specifically, this includes the following sub-steps: 1) Select reference images and source images, and use the multi-view reconstruction MVS model pre-trained in steps S2~S4 to perform feature extraction, cost matrix construction and learning in sequence; 2) A temperature-controlled depth estimation algorithm is used to predict the final depth map. The specific expression is as follows: In the formula, For normalized exponential functions, This is a temperature coefficient used to control the desired degree of polymerization; when The above formula is equivalent to the regression expectation, when The above expression is equivalent to argmax; The output features obtained in step S3; Image depth.
8. The multi-view 3D reconstruction method based on ViT according to claim 1, characterized in that, In step S5, a deep fusion algorithm is used to perform deep fusion and point cloud reconstruction on the predicted multi-view depth map. Specifically, the following sub-steps are included: A Gipuma-based deep fusion algorithm is used to perform deep fusion and point cloud reconstruction on the multi-view depth map. 1) After obtaining the depth map for each viewpoint, filter the depth map using a set confidence threshold; 2) Depth map based on reference viewpoint Projecting 2D coordinate points into 3D space, and then based on the depth map from the target's perspective. Project it back and calculate the reprojection consistency coefficient; 3) When the number of views with a reprojection consistency coefficient less than the set value reaches the set number, the point cloud is retained.
9. A multi-view 3D reconstruction system based on ViT, characterized in that, The system comprising the method according to any one of claims 1 to 8, wherein the system includes: The feature extraction module is used to extract features from images using a deep network that integrates ViT and Feature Pyramid Network (FPN). The cost matrix module is used to perform homography transformation on the extracted features, and to construct the cost matrix and optimize the depth map predicted by the cost matrix at different levels. The dynamic multi-scale strategy training module is used for dynamic multi-scale training of the multi-view reconstruction MVS model constructed by the feature extraction module and the cost matrix module. The multi-view 3D reconstruction module is used to perform deep fusion and point cloud reconstruction on the predicted multi-view depth map using a deep fusion algorithm.