Indoor scene three-dimensional reconstruction method and system based on feature pyramid fusing AdaVGGT and CNN features

Abstract

The invention discloses an indoor scene three-dimensional reconstruction method and system based on feature pyramid fusion AdaVGGT and CNN features. Obtaining an indoor scene three-dimensional image, and processing the image to obtain a plurality of local segments; inputting each local segment into a CNN branch and an AdaVGGT branch to obtain a multi-scale local feature and a global feature, and fusing the multi-scale local feature and the global feature to generate a multi-scale fusion feature; the multi-scale fusion features are respectively projected into three-dimensional voxel grids, a GRU network is used for fusion, and then MLP prediction is used to obtain three-dimensional reconstruction local segments; and generating a complete dense three-dimensional surface model according to each dense three-dimensional surface model local segment, and taking the complete dense three-dimensional surface model as a reconstructed three-dimensional scene. According to the method, by introducing an adapter and an extended feature fusion mechanism, the reconstruction consistency of a large-scale scene structure and a shielding region is improved, and the recovery quality of geometric boundaries and texture details in a complex scene is improved.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, specifically relating to a method and system for 3D reconstruction of indoor scenes based on feature pyramid fusion of AdaVGGT and CNN features. Background Technology

[0002] With the rapid development of applications such as virtual reality, augmented reality, robot navigation, and autonomous driving, recovering the 3D geometry of scenes from monocular videos or multi-view images has become an important research direction in the field of computer vision. Existing 3D reconstruction methods mainly include reconstruction methods based on traditional multi-view geometry and end-to-end reconstruction methods based on deep learning. The former relies on image feature matching and geometric optimization, and is relatively sensitive to changes in viewpoint, lighting differences, and occlusion; the latter uses deep neural networks to directly learn the 3D scene representation from image sequences, enabling higher automation and real-time performance.

[0003] Deep learning-based 3D reconstruction frameworks typically include modules for image feature extraction, voxel or depth volume construction, spatial fusion, and geometric prediction. In such frameworks, convolutional neural networks (CNNs) are widely used as the fundamental component for feature extraction due to their simple structure and computational efficiency. However, the local receptive field of convolutional operations limits their ability to model global dependencies, resulting in insufficient geometric consistency and global accuracy in reconstruction results in large-scale or structurally complex scenes. Furthermore, the features extracted by CNNs often emphasize local texture and depth gradient information, making it difficult to fully capture cross-viewpoint semantic relationships.

[0004] In recent years, the Transformer architecture has been widely adopted in computer vision tasks due to its global attention mechanism and adaptive feature modeling capabilities. The Vision Transformer (ViT) and its improved models can model long-range dependencies between features globally, thereby obtaining richer contextual semantic information. In particular, geometrically constrained Transformer models (such as VGGT) have demonstrated superior performance in multi-view geometric understanding and 3D scene analysis tasks, better fusing image and spatial information and providing new solutions for 3D reconstruction tasks.

[0005] However, directly using Transformer to replace traditional CNN for feature extraction still faces many challenges: First, the high computational complexity and large number of parameters of Transformer increase the burden of model training and inference; second, Transformer features differ from the original features of the reconstruction module in terms of dimension, scale, and distribution, and direct fusion can easily lead to feature mismatch or information redundancy; third, Transformer usually needs to be pre-trained on large-scale image data, and there is still a distribution gap between its feature space and the specific 3D reconstruction task, resulting in limited transfer effect. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides a method and system for 3D reconstruction of indoor scenes based on feature pyramid fusion of AdaVGGT and CNN features. By introducing an adapter and an extended feature fusion mechanism, it solves the problems of training burden caused by directly using Transformer and insufficient consistency and accuracy caused by using CNN to extract features.

[0007] To achieve the above objectives, the technical solution provided by the present invention is as follows: In a first aspect, the present invention provides a method for 3D reconstruction of indoor scenes based on feature pyramid fusion of AdaVGGT and CNN features, the method comprising: A structured light depth camera captures real indoor scenes, obtaining a 3D image sequence of the indoor scene and the camera's pose parameters; the 3D image sequence of the indoor scene and the camera's pose parameters are then preprocessed. From the preprocessed 3D image sequence of the indoor scene, key frames are selected based on relative translation and rotation thresholds to form multiple local segments; Each local segment is input into the CNN branch and the VGGT network branch with a lightweight adapter, respectively, to obtain multi-scale local features and global features. Multi-scale local and global features are used to generate multi-scale fused features through an extended feature fusion network based on feature pyramids; Multi-scale fused features are projected onto a three-dimensional voxel lattice, fused using a GRU network, and then predicted using an MLP to obtain local fragments of the three-dimensional reconstruction. A complete dense 3D surface model is generated from local fragments of each dense 3D surface model, and then used as the reconstructed 3D scene.

[0008] Optionally, the three-dimensional image sequence of the indoor scene includes RGB images and depth images.

[0009] Optionally, the CNN branch includes StemConv, first MBConv, second MBConv, third MBConv, fourth MBConv, fifth MBConv, and sixth MBConv connected in sequence; the fourth MBConv, fifth MBConv, and sixth MBConv are used to output feature maps of large, medium, and small resolutions, respectively. Optionally, the VGGT network branch that introduces lightweight adapters is a network structure obtained by inserting multi-convolutional adapter modules and one-dimensional deep convolutional adapter modules into the Frame Attention layer and Global Attention layer in each attention block of the VGGT large model, respectively.

[0010] Optionally, the multi-convolutional adapter module in each attention block is added in parallel with the MLP layer at the Frame Attention layer, and the one-dimensional deep convolutional adapter module is added in parallel with the MLP layer at the Global Attention layer.

[0011] Optionally, the implementation process of the multi-convolution adapter module is as follows: Input tensor After passing through a fully connected layer, projection is performed independently by token; Then, a ReLU layer is applied to activate the token, while simultaneously reshaping the input tensor into a 2D image shape. ; The reshaped tensor Input the three convolutional branches respectively; The outputs of the three convolutional branches are summed, and then passed through a fully connected layer to increase the dimension back to the original dimension, resulting in...

[0012] Will Input before adding convolution As a global residual, it is then reshaped back to the input tensor. Consistent shape is output .

[0013] Optionally, the implementation process of the one-dimensional depthwise convolution adapter module is as follows: Input tensor Reshape into 2D image shape And normalize the features of each token; Will After passing through a depthwise convolutional layer (dw), a GELU layer, and a 1D pointwise convolutional layer (pw) in sequence, we obtain... ; Will Input before adding convolution As a global residual, it is then reshaped back to the input tensor. Consistent shape is output .

[0014] Secondly, the present invention provides an indoor scene three-dimensional reconstruction system, comprising: The data acquisition and preprocessing module is responsible for capturing real indoor scenes using a structured light depth camera, acquiring 3D image sequences of the indoor scenes and camera pose parameters; preprocessing the 3D image sequences of the indoor scenes and camera pose parameters; and selecting keyframes from the preprocessed 3D image sequences of the indoor scenes based on relative translation and rotation thresholds to form multiple local segments. The multi-scale fusion feature extraction module is responsible for inputting each local segment into the CNN branch and the VGGT network branch with a lightweight adapter, respectively, to obtain multi-scale local features and global features. The multi-scale local features and global features are then used to generate multi-scale fusion features through an extended feature fusion network based on feature pyramids. The 3D scene reconstruction module is responsible for projecting multi-scale fused features onto a 3D voxel grid, fusing them using a GRU network, and then using MLP to predict local 3D reconstruction fragments. Based on the local fragments of each dense 3D surface model, a complete dense 3D surface model is generated and used as the reconstructed 3D scene.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described thereon.

[0016] Fourthly, the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described above.

[0017] The beneficial effects of this invention are: (1) In the feature extraction process, the present invention introduces a VGGT network branch with a lightweight adapter in addition to the CNN branch, which can capture global geometric relationships and long-range dependency information across perspectives. Compared with the traditional method that only relies on local convolutional features, it improves the model’s consistency in reconstructing large-scale scene structures and occluded areas.

[0018] (2) The present invention embeds a lightweight adapter module (i.e., a multi-convolution adapter module and a one-dimensional depth convolution adapter module) in the Transformer backbone VGGT. Only the adapter weights are fine-tuned, and the backbone parameters are kept frozen. This achieves low parameter quantity and high efficiency task-specific optimization, significantly reducing computational cost and memory usage. This design enables the present invention to complete high-quality end-to-end training and reconstruction under limited hardware conditions.

[0019] (3) The extended feature fusion network based on feature pyramid in this invention uses a multi-scale feature pyramid structure to fuse the local texture features of the convolutional branch (i.e., the CNN branch) with the global context features of the VGGT network branch with a lightweight adapter layer by layer, so that the output features have both local fineness and global integrity, thereby improving the recovery quality of geometric boundaries and texture details in complex scenes.

[0020] (4) The feature fusion module of the present invention can be seamlessly embedded into the existing reconstruction framework based on voxel fusion (such as TSDF fusion or cost volume construction) without changing its back-end geometric inference structure, thus combining engineering feasibility and versatility. Attached Figure Description

[0021] Figure 1 This is a standard ScanNet database instance scenario.

[0022] Figure 2 Here is a schematic diagram of the structure of the CNN branch MnasMulti, where (a) is the overall structure diagram, (b) is StemConv, and (c) is MBConv.

[0023] Figure 3 This is a schematic diagram of the structure of each attention block in AdaVGGT.

[0024] Figure 4 It is an embedded block structure diagram of the Adapter, where the Adapter represents a multi-convolution adapter module or a one-dimensional depth convolution adapter module.

[0025] Figure 5 These are structural diagrams of two types of adapters: (a) is a multi-convolution adapter module, and (b) is a one-dimensional depthwise convolution adapter module.

[0026] Figure 6 This is a schematic diagram of an extended feature fusion network structure based on feature pyramids.

[0027] Figure 7 This is a flowchart of the indoor scene three-dimensional reconstruction method provided in the embodiment of the present invention.

[0028] Figure 8 This is an example image of the reconstructed scene output. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. The flowcharts shown in the drawings are merely illustrative and do not necessarily include all contents and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the actual situation.

[0030] This embodiment provides a method for 3D reconstruction of indoor scenes based on feature pyramid fusion of AdaVGGT and CNN features. See appendix. Figure 7 It includes the following steps: S1, Data Acquisition and Preprocessing: This embodiment uses a structured light depth camera to acquire real indoor scenes. The camera resolution is 640×480, and the frame rate is 30 frames per second. The acquired raw data includes different scanned scenes. Each scene data contains a series of continuous indoor scene 3D image sequences, camera pose parameters, and dense 3D surface models. The indoor scene 3D image sequences include 640×480 resolution RGB images and 640×480 resolution depth maps. Preprocessing of the acquired data specifically includes image distortion correction, depth alignment, brightness normalization, and color balancing operations, as well as data augmentation (random flipping and color perturbation). The preprocessed data is used to construct training and testing sets. This embodiment uses the ScanNet public dataset as the training and testing sets, which uses its built-in 3D model TSDF as the dataset label. See Appendix. Figure 1 .

[0031] S2, Keyframe Selection and Segment Segmentation: From the preprocessed 3D image sequence of the indoor scene, keyframes are selected based on relative translation and rotation thresholds to form multiple local segments, specifically: To reduce redundant computation and enhance viewpoint coverage, this embodiment selects keyframes for the preprocessed indoor scene 3D image sequence based on the camera's inter-frame pose change threshold. When the relative translation between two frames exceeds 0.1 meters or the rotation angle exceeds 15°, the subsequent frame is designated as a keyframe. Nine consecutive keyframes constitute a local fragment, and each fragment corresponds to a reconstruction subtask. The fragment bounding volume (FBV) is calculated based on the view frustum of these keyframes, ensuring that the FBV encloses the view frustum of all keyframes. The FBV is cube-shaped and used to define the 3D reconstruction range of the fragment, thereby reducing unnecessary voxel computation.

[0032] S3, Dual-branch feature extraction: Each local segment is input into two pathways: a CNN branch and a VGGT network (AdaVGGT) branch with a lightweight adapter, respectively, to obtain multi-scale local features res1, res2, res3 and global features adavggt_feat; specifically including the following steps: S3.1, input the RGB image from the local segment into the CNN branch MnasMulti; MnasMulti is a custom class based on torchvision's mnasnet1_0, which is initialized with ImageNet pre-trained weights and participates in end-to-end training; its output includes feature maps of three resolutions: res1, res2, and res3, representing high resolution, medium resolution, and low resolution, respectively; For example, see Appendix Figure 2 The CNN branch MnasMulti described in (a) includes sequentially concatenated initial convolutions StemConv, first MBConv (3×3), second MBConv (3×3), third MBConv (5×5), fourth MBConv (3×3), fifth MBConv (5×5), and sixth MBConv (5×5); the fourth MBConv, fifth MBConv, and sixth MBConv are used to output large, medium, and small resolution feature maps res1, res2, and res3, respectively; See appendix Figure 2 The StemConv described in (b) includes 3×3 Conv2D, BatchNorm2D, and ReLU6 connected in series.

[0033] See appendix Figure 2 In (c), the first MBConv, the second MBConv, the third MBConv, the fourth MBConv, the fifth MBConv, and the sixth MBConv all use the same inverted residual block MBConv, which includes a 1×1 expansion layer, a k×k depthwise separable convolution, a channel attention module, a projection layer, and a residual connection, all connected in sequence.

[0034] S3.2, the RGB image in the local segment is input to the AdaVGGT branch of the VGGT network, which introduces a lightweight adapter.

[0035] The AdaVGGT network uses a pre-trained visual Transformer model, VGGT, as its backbone. For efficient parameter fine-tuning, see the appendix. Figure 3In this embodiment, a multiconvolutional adapter module (MultiConvAdapter) and a one-dimensional depthwise adapter module (DepthWise Adapter) are inserted into the frame attention layer and global attention layer, respectively, in each attention block of VGGT. The Transformer backbone parameters are frozen, and only the adapter weights are trained to reduce training costs and ensure stability.

[0036] There are 10 attention blocks in total. Within each attention block, the MultiConv Adapter module is added in parallel with the MLP layer at the Frame Attention layer, and the DepthWiseAdapter module is added in parallel with the MLP layer at the Global Attention layer. See Appendix. Figure 4 That is, let the input After passing through the adapter, the output is obtained. and at the adapter output Add a scaling factor before adding the output of the MLP layer. , get output :

[0037] in This is the proportionality coefficient; See appendix Figure 5 In section (a), the implementation process of the multi-convolution adapter module is as follows: A1, input tensor After passing through a fully connected layer, the token is projected independently to compress the dimensions and reduce the amount of computation. A2, then through a ReLU layer to activate the token, simultaneously reshaping the input tensor into a 2D image shape. Prepare to input the convolution; A3 will be the reshaped tensor. The input consists of three convolutional branches: branch1, branch2, and branch3. Branch1 contains a 2D convolutional block that performs only point convolutions to capture inter-channel relationships. Branch2 contains two 2D convolutional layers: the first layer has a 1x1 kernel, and the second layer has a 3x3 kernel, capturing medium-sized local features. Branch3 is similar to branch2, but the second layer has a 5x5 kernel and padding of 2, capturing larger local features. A4, After passing through three convolutional branches, the outputs of the three branches are summed, and then passed through a fully connected layer to increase the dimension back to the original dimension, resulting in... :

[0038] A5, will Input before adding convolution As a global residual, it is then reshaped back to the input tensor. Consistent shape is output .

[0039] See appendix Figure 5 In section (b), the implementation process of the one-dimensional depthwise convolution adapter module is as follows: B1 will input tensor Reshape into 2D image shape And normalize the features of each token to help stabilize subsequent convolutions; B2, will The process proceeds through the following three layers in sequence: First, a depthwise convolutional layer (dw), where each input channel is convolved independently without channel mixing; second, a GELU layer, using Gaussian error linear unit activation functions to provide smooth non-linearity; third, a 1D pointwise convolutional layer (pw), with a single kernel, mixing channel information. After these three layers, the result is... :

[0040] B3, will Input before adding convolution As a global residual, it is then reshaped back to the input tensor. Consistent shape is output .

[0041] In S3.3, the CNN branch extracts features at three scales: res1, res2, and res3. The VGGT network branch, which introduces a lightweight adapter, extracts global features: adavggt_feat.

[0042] S4, multi-scale local features res1, res2, res3, and global feature adavggt_feat are fused through an extended feature fusion network based on feature pyramids. See Appendix. Figure 6 Generate multi-scale fusion features, specifically: S4.1 Project the res1, res2, and res3 features extracted by the CNN onto a unified dimension to obtain res1_proj, res2_proj, and res3_proj, which prepares for the top-down path of the Feature Pyramid (FPN). S4.2, using the top-down path of FPN to progressively refine features, the global features adavggt_feat of the VGGT network branch with the lightweight adapter extracted in step S3 are fused with res3_proj through element-wise addition, and then smoothed using a 3x3 convolution to obtain p3:

[0043] After calculating p3, upsample p3, add the upsampled result to res2_proj and fuse them, then smooth it using a 3x3 convolution to obtain p2:

[0044] After calculating p2, upsample p2 in the same way, add the upsampled result to res1_proj and fuse them, then smooth it using a 3x3 convolution to obtain p1:

[0045] S4.3, Apply the final convolution to the FPN features and adjust the number of channels:

[0046] Output1, output2, and output3 are 2D convolutional layers used to adjust the number of channels. Then, p1, p2, and p3 are reshaped to obtain multi-scale 2D features p1_r, p2_r, and p3_r for the fragment, which are convenient for downstream sequence tasks.

[0047] S5, 3D Voxel Construction and TSDF Prediction: Multi-scale fused features are projected onto a 3D voxel lattice, fused using a GRU network, and then predicted using an MLP to obtain local 3D reconstructed fragments. Specifically... S5.1 Using the camera's pose parameters, the multi-scale 2D features of the fragment extracted in step S4 are back-projected along the ray into the 3D space within the FBV calculated in step S2. For each voxel, the visibility weight is calculated based on the number of visible views, and the features from different views are aggregated on an average basis to form a sparse 3D feature volume of three scales. These three 3D feature volumes are the inputs for subsequent 3D inference. S5.2, refine TSDF prediction in three levels: start with dense prediction at the coarse level, and gradually move to sparse representation at the fine level; at each level, first use sparse convolution to extract geometric features, then apply a GRU network to fuse the previous global hidden state, and finally use MLP to predict TSDF; upsample the TSDF predicted by the first two levels and concatenate it with the 3D feature volume of the next level as the input of the next level GRU network; the TSDF predicted by the third level will be used as the input of the next step.

[0048] S6, Global Fusion and Scene Reconstruction: Generates a complete dense 3D surface model based on local fragments of each dense 3D surface model, and uses it as the reconstructed 3D scene. Specifically: The TSDF results predicted for each fragment are transformed into the global coordinate system, and the corresponding parts of the global TSDF voxels are replaced. The fused global TSDF is updated incrementally over time to achieve incremental reconstruction. After all fragments are reconstructed, the Marching Cubes algorithm is used to extract isosurfaces and generate the final 3D scene mesh. This step achieves incremental updates to ensure real-time coherent reconstruction. See Appendix. Figure 8 .

[0049] For example, the loss function L of the method of the present invention is:

[0050] in For the loss of occupancy, This is the SDF loss. The calculation formula is:

[0051] Where BCE is the binary cross-entropy calculation; The predicted occupancy score represents the confidence level of the voxel within the cutoff distance; This indicates the actual occupancy label. The calculation formula is:

[0052] in This represents the signed distance from the predicted voxel to the nearest surface. This represents the signed distance from the actual voxel to the nearest surface.

[0053] Experimental evaluation criteria: In the extended feature fusion 3D reconstruction method based on adapter-enhanced Transformer, to comprehensively evaluate the reconstruction accuracy and efficiency of the algorithm in different scenarios, experiments were conducted on 100 test scenarios in the ScanNet dataset. Combining common 3D reconstruction evaluation metrics with some general metrics, the following were used as evaluation metrics in this experiment: Absolute Relative Error (AbsRel), Absolute Difference (AbsDiff), Root Mean Squared Error (RMSE), Accuracy (Accuracy), Completeness (Comp), Precision (Prec), Recall, and F1 Score. AbsRel, AbsDiff, and RMSE are 2D depth prediction metrics, while Acc, Comp, Prec, Recall, and F1 Score are 3D geometry reconstruction metrics. The main meanings and calculation methods of each metric are as follows, for reference in subsequent experimental result analysis: 2D depth prediction metrics:

[0054]

[0055]

[0056] in Let be the predicted depth of the i-th pixel. Let N be the ground truth depth of the i-th pixel, and N be the number of valid pixels. 3D geometric reconstruction metrics:

[0057]

[0058]

[0059]

[0060]

[0061] in This represents the predicted point cloud obtained from the reconstruction. Represents a true point cloud. Indicates the prediction point To real point cloud distance, This represents the distance threshold (0.05m in this embodiment). The value is lverson; it is 1 if the condition is true, and 0 otherwise.

[0062] Experimental results: This embodiment is an experiment on extended feature fusion 3D reconstruction based on an adapter-enhanced Transformer. The training batch size was set to 4, the initial learning rate was 0.001, the optimizer used was adamW, and the training epochs were 20. The experimental results are as follows: Table 1. Experimental results of different methods on the ScanNet dataset:

[0063] On the ScanNet dataset, this embodiment demonstrates that the extended feature fusion 3D reconstruction method based on adapter-enhanced Transformer proposed in this invention significantly reduces absolute difference and root mean square error compared to existing methods (such as Atlas, NeuralRecon, and VisFusion), while also showing significant improvements in accuracy, precision, and F1-Score.

[0064] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for 3D reconstruction of indoor scenes based on feature pyramid fusion of AdaVGGT and CNN features, characterized in that, The method includes: A structured light depth camera captures real indoor scenes, obtaining a 3D image sequence of the indoor scene and the camera's pose parameters; the 3D image sequence of the indoor scene and the camera's pose parameters are then preprocessed. From the preprocessed 3D image sequence of the indoor scene, key frames are selected based on relative translation and rotation thresholds to form multiple local segments; Each local segment is input into the CNN branch and the VGGT network branch with a lightweight adapter, respectively, to obtain multi-scale local features and global features. Multi-scale local and global features are used to generate multi-scale fused features through an extended feature fusion network based on feature pyramids; Multi-scale fused features are projected onto a three-dimensional voxel lattice, fused using a GRU network, and then predicted using an MLP to obtain local fragments of the three-dimensional reconstruction. A complete dense 3D surface model is generated from local fragments of each dense 3D surface model, and then used as the reconstructed 3D scene.

2. The method according to claim 1, characterized in that: The indoor scene 3D image sequence includes RGB images and depth images.

3. The method according to claim 1, characterized in that: The CNN branch includes StemConv, first MBConv, second MBConv, third MBConv, fourth MBConv, fifth MBConv, and sixth MBConv connected in sequence; the fourth MBConv, fifth MBConv, and sixth MBConv are used to output feature maps of large, medium, and small resolutions, respectively.

4. The method according to claim 1, characterized in that: The VGGT network branch that introduces lightweight adapters is a network structure obtained by inserting multi-convolutional adapter modules and one-dimensional deep convolutional adapter modules into the Frame Attention layer and Global Attention layer in each attention block of the VGGT large model, respectively.

5. The method according to claim 4, characterized in that: In each attention block, the multi-convolutional adapter module is added in parallel with the MLP layer at the FrameAttention layer, and the one-dimensional deep convolutional adapter module is added in parallel with the MLP layer at the GlobalAttention layer.

6. The method according to claim 4, characterized in that: The implementation process of the multi-convolution adapter module is as follows: Input tensor After passing through a fully connected layer, projection is performed independently by token; Then, a ReLU layer is applied to activate the token, while simultaneously reshaping the input tensor into a 2D image shape. ; The reshaped tensor Input the three convolutional branches respectively; The outputs of the three convolutional branches are summed, and then passed through a fully connected layer to increase the dimension back to the original dimension, resulting in... ; Will Input before adding convolution As a global residual, it is then reshaped back to the input tensor. Consistent shape is output .

7. The method according to claim 4, characterized in that: The implementation process of the one-dimensional depthwise convolution adapter module is as follows: Input tensor Reshape into 2D image shape And normalize the features of each token; Will After passing through a depthwise convolutional layer (dw), a GELU layer, and a 1D pointwise convolutional layer (pw) in sequence, we obtain... ; Will Input before adding convolution As a global residual, it is then reshaped back to the input tensor. Consistent shape is output .

8. A three-dimensional reconstruction system for an indoor scene for implementing the method of any one of claims 1-7, characterized in that, include: The data acquisition and preprocessing module is responsible for capturing real indoor scenes using a structured light depth camera, acquiring 3D image sequences of the indoor scenes and the camera's pose parameters; Preprocess the 3D image sequence of the indoor scene and the camera's pose parameters; select keyframes from the preprocessed 3D image sequence of the indoor scene based on relative translation and rotation thresholds to form multiple local segments; The multi-scale fusion feature extraction module is responsible for inputting each local segment into the CNN branch and the VGGT network branch with a lightweight adapter, respectively, to obtain multi-scale local features and global features. The multi-scale local features and global features are then used to generate multi-scale fusion features through an extended feature fusion network based on feature pyramids. The 3D scene reconstruction module is responsible for projecting multi-scale fused features onto a 3D voxel grid, fusing them using a GRU network, and then using MLP to predict local 3D reconstruction fragments. Based on the local fragments of each dense 3D surface model, a complete dense 3D surface model is generated and used as the reconstructed 3D scene.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed in the computer, it causes the computer to perform the method of any one of claims 1-7.

10. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-7.

Images