Method, device and storage medium for accelerating three-dimensional scene reconstruction

Abstract

The application discloses a three-dimensional scene reconstruction acceleration method and device and a storage medium, relates to the technical field of computer vision, and comprises the following steps: when input video to be reconstructed is forward propagated to a global attention layer of a Transformer backbone network, determining a target compression ratio according to a level identifier of a current layer and a mapping relationship between the level and a compression ratio; determining a query matrix, a key matrix and a value matrix of the current layer, sparsely sampling the key matrix and the value matrix according to the target compression ratio to obtain a target key matrix and a target value matrix; performing attention calculation according to the query matrix, the target key matrix and the target value matrix to obtain a context feature representation of the current layer, until all attention calculation is completed, inputting an output feature sequence of the Transformer backbone network to a prediction head to obtain a multi-view reconstruction result of the video to be reconstructed, and guaranteeing the accurate completion of the multi-view three-dimensional reconstruction task while improving the efficiency.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, device and storage medium for accelerating 3D scene reconstruction. Background Technology

[0002] In recent years, end-to-end feedforward 3D reconstruction models based on the Transformer architecture (represented by the Vision Geometry Grounded Transformer, VGGT) have made significant progress. These models, pre-trained on large-scale datasets, can directly output multi-view reconstruction results from multi-view image inputs through a single forward propagation, achieving efficient mapping from 2D images to 3D structures. This paradigm avoids the cumbersome iterative optimization and complex preprocessing required by traditional Structure from Motion (SfM) and Multi-View Stereo (MVS) methods, significantly improving inference efficiency while maintaining high reconstruction accuracy. This demonstrates enormous application potential in fields with high real-time and ease-of-use requirements, such as augmented reality navigation and rapid digitization of large-scale scenes. However, in practical deployments, especially when processing long input sequences, the computational complexity and memory consumption of the attention mechanism, upon which the Transformer-based feedforward 3D reconstruction models rely for their superior performance, increase quadratically with the length of the input sequence. This leads to a sharp increase in computation time and resource consumption when the model processes long videos, high-resolution images, or large-scale image sets, resulting in a significant decrease in reconstruction efficiency and making it difficult to meet the demand for efficient video 3D reconstruction in real-world scenarios.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a method, device, and storage medium for accelerating 3D scene reconstruction, aiming to solve the technical problem of low processing efficiency caused by the high computational complexity of attention in multi-view image reconstruction of 3D geometry tasks using the Transformer architecture in the prior art.

[0005] To achieve the above objectives, this application proposes a method for accelerating 3D scene reconstruction, the method comprising:

[0006] When the input video to be reconstructed is forward propagated to the global attention layer of the Transformer backbone network in the 3D reconstruction model, the target compression ratio corresponding to the current layer is determined according to the layer identifier of the current layer and the mapping relationship between the layer and the compression ratio.

[0007] A linear mapping is performed on the input feature sequence of the current layer to generate a query matrix, a key matrix, and a value matrix. According to the target compression ratio, sparse sampling is performed on the key matrix and the value matrix in the sequence dimension to obtain the target key matrix and the target value matrix.

[0008] Attention calculation is performed based on the query matrix, target key matrix, and target value matrix corresponding to the current layer to obtain the context feature representation of the current layer, until all attention calculations of the Transformer backbone network are completed;

[0009] The output feature sequence of the Transformer backbone network is input into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction result of the video to be reconstructed.

[0010] In one embodiment, prior to the step of forward propagating the input video to be reconstructed to the global attention layer of the Transformer backbone network in the 3D reconstruction model, the method further includes:

[0011] Each video frame in the video to be reconstructed is input into the three-dimensional reconstruction model, and the image encoder of the three-dimensional reconstruction model is used to extract features from each video frame to obtain the initial feature vector sequence corresponding to the video to be reconstructed.

[0012] The initial feature vector sequence is input into the Transformer backbone network of the 3D reconstruction model for forward propagation.

[0013] In one embodiment, before the step of determining the target compression ratio corresponding to the current layer based on the layer identifier and the mapping relationship between the layer and the compression ratio, the method further includes:

[0014] Obtain a test dataset, which includes multiple test videos;

[0015] The test video is input into the trained 3D reconstruction model for forward inference, and the attention matrix output by each global attention layer in the 3D reconstruction model is extracted.

[0016] The effective rank value corresponding to the global attention layer is obtained by calculating the effective rank based on the attention matrix.

[0017] The effective rank values ​​are mapped according to rules to generate the compression ratio corresponding to the global attention layer;

[0018] Based on the compression ratio corresponding to the global attention layer and the hierarchical identifier of the global attention layer, the mapping relationship between the hierarchical level and the compression ratio is constructed.

[0019] In one embodiment, the step of sparsely sampling the key matrix and the value matrix along the sequence dimension according to the target compression ratio to obtain the target key matrix and the target value matrix includes:

[0020] A non-uniform sparse sampling algorithm that preserves time structure is used to perform frame-level sparse sampling on the key matrix in the sequence dimension according to the target compression ratio to obtain the compressed target key matrix;

[0021] A non-uniform sparse sampling algorithm that preserves time structure is used to perform frame-level sparse sampling on the value matrix in the sequence dimension according to the target compression ratio to obtain the compressed target value matrix.

[0022] In one embodiment, the step of using a time-structure-preserving non-uniform sparse sampling algorithm to perform frame-level sparse sampling on the key matrix in the sequence dimension according to the target compression ratio to obtain the compressed target key matrix includes:

[0023] The number of row vectors to be retained in the key matrix is ​​determined based on the target compression ratio;

[0024] The key matrix is ​​divided into first frame groups according to the number of video frames, the number of row vectors, and the first frame group includes multiple first row vectors, the first row vectors being the row vectors in the key matrix;

[0025] Within each first frame group, the information entropy of each first row vector is calculated, and the first row vector corresponding to the maximum information entropy value is determined as the first target row vector corresponding to the first frame group.

[0026] The target key matrix is ​​reconstructed based on the first target row vector corresponding to each of the first frame groups.

[0027] In one embodiment, the Transformer backbone network of the 3D reconstruction model consists of multiple alternating global attention layers and intra-frame attention layers.

[0028] In one embodiment, when propagating to each of the intra-frame attention layers of the Transformer backbone network, the input feature sequence of the intra-frame attention layer is grouped according to video frames to obtain the input features corresponding to each video frame;

[0029] Perform self-attention processing on each of the input features to obtain the self-attention processing results for each of the video frames;

[0030] The self-attention processing results corresponding to each video frame are concatenated, and the corresponding concatenation results are used as the input feature sequence of the next global attention layer.

[0031] In one embodiment, the step of inputting the output feature sequence of the Transformer backbone network into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction result of the video to be reconstructed includes:

[0032] The output feature sequence of the Transformer backbone network is input into the first prediction sub-network in the prediction head. The camera pose information and depth information of each video frame are predicted through the first prediction sub-network to obtain the depth map and camera pose parameters corresponding to each video frame.

[0033] The output feature sequence of the Transformer backbone network, the depth map of each video frame, and the camera pose parameters are input into the second prediction sub-network in the prediction head. The second prediction sub-network predicts the point cloud information of each video frame to obtain the point cloud feature map corresponding to each video frame.

[0034] Based on the camera pose parameters and point cloud feature maps of each video frame, a 3D scene point cloud model corresponding to the video to be reconstructed is generated, and the 3D scene point cloud model, the camera pose parameters, and the depth map are used as the multi-view reconstruction result.

[0035] Furthermore, to achieve the above objectives, this application also proposes a three-dimensional scene reconstruction acceleration device, which includes:

[0036] The compression ratio determination module is used to determine the target compression ratio corresponding to the current layer based on the layer identifier of the current layer and the mapping relationship between the layer and the compression ratio when the input video to be reconstructed is forward propagated to the global attention layer of the Transformer backbone network in the 3D reconstruction model.

[0037] The compression module is used to perform linear mapping on the input feature sequence of the current layer to generate a query matrix, a key matrix, and a value matrix. According to the target compression ratio, the key matrix and the value matrix are sparsely sampled in the sequence dimension to obtain the target key matrix and the target value matrix.

[0038] The global attention processing module is used to perform attention calculations based on the query matrix, the target key matrix, and the target value matrix corresponding to the current layer to obtain the context feature representation of the current layer, until all attention calculations of the Transformer backbone network are completed.

[0039] The reconstruction module is used to input the output feature sequence of the Transformer backbone network into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction result of the video to be reconstructed.

[0040] In addition, to achieve the above objectives, this application also proposes a three-dimensional scene reconstruction acceleration device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the three-dimensional scene reconstruction acceleration method described above.

[0041] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the three-dimensional scene reconstruction acceleration method described above.

[0042] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the three-dimensional scene reconstruction acceleration method described above.

[0043] The one or more technical solutions proposed in this application have at least the following technical effects: The video to be reconstructed is input into a 3D reconstruction model for forward propagation. When propagating to each global attention layer of the Transformer backbone network in the 3D reconstruction model, the target compression ratio corresponding to the current layer is determined based on the layer identifier and the mapping relationship between the layer and the compression ratio. This achieves differentiated compression at different layers, balancing compression efficiency and feature information preservation, avoiding limited efficiency improvement or feature loss caused by indiscriminate compression, and laying the foundation for subsequent accurate compression. A linear mapping is performed on the input feature sequence of the current layer to generate a query matrix, a key matrix, and a value matrix. According to the target compression ratio, sparse sampling is performed on the key matrix and value matrix along the sequence dimension to obtain the target key matrix and target value matrix. The sequence length of the key matrix and value matrix is ​​reduced to the reciprocal of the original length of the target compression ratio, thereby reducing the computational cost of subsequent attention processing. Next, attention calculations are performed based on the query matrix, target key matrix, and target value matrix corresponding to the current layer to obtain the contextual feature representation of the current layer. This process continues until all attention calculations in the Transformer backbone network are completed. The contextual feature representation of each global attention layer has the same dimension as the input feature sequence and can be directly passed layer by layer in the Transformer backbone network. Finally, after completing all attention calculations in the Transformer backbone network, the output feature sequence of the Transformer backbone network is input into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction results of the video to be reconstructed. This application reduces the computational complexity of global attention at each layer by performing sequential dimensionality sparse compression on the key and value matrices before calculating global attention in the forward propagation of the 3D reconstruction model, based on a pre-defined compression ratio at each layer, while preserving the integrity of the query matrix. Attention calculation is then performed, and the reconstruction result is finally output through the prediction head. The entire process does not change the original forward propagation logic of the model. Without compromising the reconstruction capability of the 3D reconstruction model, the computational complexity of global attention at each layer is directly reduced, and the overall model processing efficiency is improved by layering these reductions. This solves the technical problem of low processing efficiency caused by the high computational complexity of attention in the Transformer architecture 3D reconstruction model, and ensures the accurate completion of multi-view 3D reconstruction tasks while improving efficiency. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating an embodiment of the method for accelerating 3D scene reconstruction according to this application.

[0047] Figure 2 A schematic diagram of the architecture of a three-dimensional reconstruction model provided in this application;

[0048] Figure 3 A scene illustration of a context-compression-based 3D reconstruction acceleration method provided in this application;

[0049] Figure 4 A comparative diagram of subjective 3D reconstruction results on the VGGT model using the 3D scene reconstruction acceleration method provided in this application;

[0050] Figure 5 A comparative diagram of subjective results of scene-level 3D reconstruction on three cutting-edge pedestal models using the 3D scene reconstruction acceleration method provided in this application;

[0051] Figure 6 This is a schematic diagram of the module structure of the 3D scene reconstruction acceleration device according to an embodiment of this application;

[0052] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the 3D scene reconstruction acceleration method in the embodiments of this application. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0054] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of this application embodiment is as follows: When the input video to be reconstructed is forward propagated to the global attention layer of the Transformer backbone network in the 3D reconstruction model, the target compression ratio corresponding to the current layer is determined according to the layer identifier of the current layer and the mapping relationship between the layer and the compression ratio; a linear mapping is performed on the input feature sequence of the current layer to generate a query matrix, a key matrix, and a value matrix, and sparse sampling is performed on the key matrix and the value matrix in the sequence dimension according to the target compression ratio to obtain the target key matrix and the target value matrix; attention calculation is performed according to the query matrix, the target key matrix, and the target value matrix corresponding to the current layer to obtain the context feature representation of the current layer, until all attention calculations of the Transformer backbone network are completed; the output feature sequence of the Transformer backbone network is input into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction result of the video to be reconstructed.

[0056] In this embodiment, for ease of description, the three-dimensional scene reconstruction acceleration system will be used as the execution subject for the following description.

[0057] Recovering the three-dimensional geometry of a scene from a multi-view image sequence is a fundamental and crucial task in the field of computer vision. This technology aims to transform two-dimensional images or videos with limited perspectives into measurable, interactive, and analyzable three-dimensional digital models, providing core spatial perception and understanding capabilities for numerous downstream applications such as Augmented Reality (AR), robot navigation, autonomous driving, digital twins, and cultural heritage protection.

[0058] Early 3D reconstruction techniques primarily relied on classic Structure from Motion (SfM) and Multi-View Stereo (MVS) approaches. SfM aims to recover sparse 3D point clouds and camera parameters from unordered images, while MVS performs dense matching based on known or estimated camera parameters to recover surface geometry. These methods are built upon rigorous geometric and optimization theories and typically require complex feature extraction, matching, and iterative bundle adjustment operations for specific scenes. Although they can achieve high reconstruction accuracy, their processes are cumbersome, have limited automation, and incur huge computational costs. They heavily depend on manual intervention and high-performance computing equipment, making it difficult to handle large-scale data or meet real-time requirements. Their generalization ability is also limited by the robustness of hand-designed features.

[0059] To overcome the bottlenecks of traditional methods, with the rapid development of deep learning technology, research paradigms have begun to evolve towards data-driven neural network methods. These methods utilize large-scale, diverse datasets for training, enabling the network to learn the implicit mapping relationship from images to 3D geometry, thus freeing it from dependence on traditional iterative optimization processes. A series of feedforward neural reconstruction methods, represented by Dense and Unconstrained Stereo 3D Reconstruction (Dust3R), are typical examples of this direction. These methods employ a feedforward neural network to directly predict a preliminary 3D representation aligned with the input image pixels from the forward propagation. These preliminary predictions are then integrated to generate the final 3D model. Compared to traditional methods, early feedforward reconstruction methods significantly simplified the reconstruction process, substantially reduced computational costs, and improved the model's generalization ability to different scenes by leveraging prior knowledge learned from data. This effectively enhanced generalization ability provides a more efficient and scalable solution for the practical application of multi-view 3D geometry reconstruction, and lays the core technical roadmap of "feedforward and unification" for subsequent technological development.

[0060] Following the technical route of "feedforward and unification," end-to-end feedforward reconstruction models based on the Transformer architecture have made progress in recent years, with the Vision Geometry Grounded Transformer (VGGT) as a representative example. These models construct a unified neural network and, through pre-training on large-scale datasets, can directly map multi-view 2D images to 3D geometric attributes in a single end-to-end forward propagation from multi-view image inputs. This avoids the complex iterative optimization and step-by-step operations of traditional SfM / MVS processes, achieving a significant improvement in inference efficiency while maintaining high reconstruction accuracy.

[0061] VGGT-type end-to-end feedforward reconstruction models, with their strong real-time performance, excellent generalization ability, and convenient deployment, have reshaped the application landscape of 3D reconstruction technology, opening up new application possibilities in many real-world scenarios with stringent requirements for efficiency and ease of use. In real-time interactive and mobile 3D perception scenarios, their feedforward computation characteristics are well-suited for achieving real-time 3D understanding on mobile and edge devices such as smartphones and edge detection devices, supporting applications such as augmented reality (AR) navigation and real-time mobile scanning modeling. In the field of rapid digitization of large-scale scenes, they can efficiently process massive amounts of image or video data collected during urban digital twins and infrastructure inspections, quickly generating initial geometric models of scenes and providing real-time data support for subsequent scene analysis and fault detection. In the field of intelligent video understanding, the dense geometric correspondences output by the model can provide rich 3D prior information for advanced visual tasks such as motion segmentation and dynamic object tracking in videos, driving the development of video understanding technology towards a more geometric and physical direction.

[0062] However, VGGT-based end-to-end feedforward reconstruction models using the Transformer architecture face significant challenges in their large-scale practical application. Their superior 3D geometric reasoning and cross-view information fusion capabilities rely on the global attention mechanism of the Transformer architecture. This mechanism allows each element in the sequence (corresponding to features of each image) to interact with all other elements during computation, establishing a global contextual relationship. This powerful global modeling capability ensures accurate restoration of geometric consistency across multiple views. However, the inherent characteristics of this global attention mechanism also lead to a quadratic increase in computational complexity and memory consumption with the length of the input sequence, i.e., O(N²). The computational overhead increases dramatically with the length of the image or video frame sequence. When processing high-dimensional inputs such as long videos, the computation time and memory consumption increase exponentially. For example, reconstructing a 30-second standard frame rate video can take approximately 20 minutes, completely failing to meet the requirements of real-time or near-real-time applications.

[0063] Meanwhile, long input sequences can also trigger a second bottleneck: performance degradation. Existing models lack intelligent filtering and compression mechanisms for long-range contextual information. Redundant or low-quality information in the sequence can lead to the accumulation of errors layer by layer during the transmission between network layers. This not only reduces the accuracy of the reconstructed geometry (such as increased depth estimation error, increased point cloud noise, and deterioration of structural integrity and smoothness), but also significantly reduces the stability and consistency of the reconstruction output, limiting the reliable application of the model in high-precision, long-sequence 3D reconstruction tasks.

[0064] This application provides a solution that, by inputting the video frame to be reconstructed into the model for forward propagation, obtains the target compression ratio specific to the current layer based on the mapping relationship between the layer identifier and the compression ratio when entering each global attention layer, achieving differentiated compression at different layers according to redundancy. Then, based on the target compression ratio, sparse sampling is performed on the key and value matrices generated from the input feature sequence of the current layer along the sequence dimension, reducing the sequence length involved in attention calculation, thus obtaining the target key and target value matrices and directly reducing the matrix dot product computation. Attention calculation is then performed using the complete query matrix and the compressed target key and target value matrices, resulting in a lightweight but information-complete contextual feature representation, significantly reducing the computational complexity of each attention layer from quadratic. After completing all attention calculations in the Transformer backbone network, the output feature sequence of the Transformer backbone network is input into the prediction head, which outputs the multi-view reconstruction results. By using layered differential compression and asymmetric attention computation, the computational load is reduced layer by layer without changing the model structure or losing reconstruction accuracy, thereby improving the overall inference speed of the 3D reconstruction model. This effectively solves the technical problem of low processing efficiency caused by the complexity of attention computation in multi-view 3D geometric reconstruction. It achieves an order-of-magnitude improvement in processing efficiency while outputting high-quality multi-view reconstruction results.

[0065] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a 3D scene reconstruction acceleration device capable of achieving the above functions. The following description uses a 3D scene reconstruction acceleration system as an example to illustrate this embodiment and the subsequent embodiments.

[0066] Based on this, embodiments of this application provide a method for accelerating 3D scene reconstruction, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the three-dimensional scene reconstruction acceleration method of this application.

[0067] In this embodiment, the method for accelerating 3D scene reconstruction includes steps 101-104:

[0068] Step 101: When the input video to be reconstructed is forward propagated to the global attention layer of the Transformer backbone network in the 3D reconstruction model, the target compression ratio corresponding to the current layer is determined according to the layer identifier of the current layer and the mapping relationship between the layer and the compression ratio.

[0069] Specifically, the video to be reconstructed is the original video requiring multi-view 3D reconstruction, consisting of a series of video frames. It provides the multi-view observations needed for reconstruction. The more frames in the video to be reconstructed, the richer the potential geometric information, and the longer the sequence length. The 3D reconstruction model is a feedforward end-to-end 3D reconstruction model based on the Transformer architecture, such as VGGT. It can output pose, depth, point cloud, etc., through a single forward propagation. The 3D reconstruction model includes an image encoder, a Transformer backbone network (composed of multiple attention layers), and a prediction head. The Transformer backbone network contains multiple global attention layers, which are the core computational modules within the 3D reconstruction model used to fuse information between all input video frames. The current layer is a global attention layer currently performing computation. The layer identifier of the current layer refers to the sequence number of this global attention layer in the Transformer stack structure, such as layer 1, layer 5, layer 10. The layer identifier serves as the index key for querying compression strategies. The mapping relationship between layers and compression ratios is a pre-established correspondence table. For example: layer 1 → compression ratio 1 / 10, layer 5 → compression ratio 1 / 30, layer 10 → compression ratio 1 / 90.

[0070] In some embodiments, during the offline phase, for a 3D reconstruction model (such as VGGT), forward inference can be performed using a representative multi-view test set, and the attention matrix of each global attention layer in its Transformer backbone network can be extracted. Subsequently, a redundancy metric (such as stable rank) is calculated based on the attention matrix of each global attention layer, and a compression ratio is assigned to each global attention layer accordingly—the higher the redundancy, the larger the compression ratio. Finally, a layer identifier-compression ratio mapping table is generated and stored. During the online inference phase, when the input video frame is encoded and enters the Transformer backbone network, and propagates layer by layer to any global attention layer, the layer identifier (such as layer number, layer type label, etc.) of the current layer (current global attention layer) is obtained, and the pre-built and stored mapping relationship between layers and compression ratios is invoked. The layer identifier is used to accurately match the mapping relationship to determine the target compression ratio corresponding to the current global attention layer, where a high redundancy global attention layer corresponds to a higher compression ratio, and a low redundancy global attention layer corresponds to a lower compression ratio. This target compression ratio provides a clear proportional basis for subsequent compression operations on the key and value matrices in the sequence dimension, guiding the sparse sampling intensity of the key and value matrices. This enables hierarchical adaptive allocation of computational resources, significantly reducing overall computational overhead while ensuring the quality of deep semantic fusion. Furthermore, this process is executed independently in each global attention layer of the Transformer backbone network, ensuring that the compression ratio of each layer is precisely adapted to its own feature characteristics.

[0071] Optionally, before the step of forward propagating the input video to be reconstructed to the global attention layer of the Transformer backbone network in the 3D reconstruction model, the following steps are also included:

[0072] Each video frame in the video to be reconstructed is input into the 3D reconstruction model. The image encoder of the 3D reconstruction model extracts features from each video frame to obtain the initial feature vector sequence corresponding to the video to be reconstructed.

[0073] The initial feature vector sequence is input into the Transformer backbone network of the 3D reconstruction model for forward propagation.

[0074] Specifically, the image encoder can be a Vision Transformer (ViT) or a Convolutional Neural Network (CNN) backbone network, responsible for encoding each video frame in the video to be reconstructed, converting it into a high-dimensional feature vector, which is used to transform the image into high-dimensional features. The initial feature vector sequence is a sequence composed of the feature vectors of all video frames in frame order, which serves as the input to the Transformer backbone network.

[0075] For example, the input video to be reconstructed is parsed to obtain multiple consecutively arranged video frames. All video frames are then input into a 3D reconstruction model based on the Transformer architecture in chronological order. The 3D reconstruction model calls the built-in image encoder to perform feature extraction operations on each video frame. Through encoding processes such as convolution, pooling, and high-dimensional mapping, the two-dimensional pixel-level video frames are transformed into high-dimensional feature vectors containing scene geometric features and semantic information. Then, all feature vectors are sequentially concatenated according to the original chronological order of the video frames to generate an initial feature vector sequence corresponding to the video to be reconstructed. This initial feature vector sequence retains the spatiotemporal structure information of the multi-view input. Subsequently, the initial feature vector sequence is input into the Transformer backbone network of the 3D reconstruction model to initiate the forward propagation process of the features. The feature data is progressively carried along the hierarchical structure of the Transformer backbone network. When the feature data propagates to any global attention layer, the layer identifier of that layer is obtained, and the mapping relationship between the layer and the compression ratio, which has been pre-constructed and stored offline, is retrieved. A precise matching query is performed in the mapping relationship through the layer identifier to determine the target compression ratio corresponding to the current global attention layer. This determination process is completed independently in each global attention layer of the Transformer backbone network to ensure that the compression ratio of each layer is precisely adapted to its own feature redundancy characteristics, providing a clear basis for the sparse sampling of the subsequent key matrix and value matrix.

[0076] Step 102: Perform linear mapping on the input feature sequence of the current layer to generate a query matrix, a key matrix, and a value matrix. Then, according to the target compression ratio, perform sparse sampling on the key matrix and the value matrix in the sequence dimension to obtain the target key matrix and the target value matrix.

[0077] Specifically, the input feature sequence of the current layer is the feature vector sequence that enters the current global attention layer, X. (l) ∈ ,in The sequence length of the input feature sequence for the current layer. The feature dimension of the feature vector. The key matrix is ​​the matrix generated by the current layer after performing a linear projection on the input feature sequence, with a shape of [sequence length]. Feature Dimension The rows correspond to the sequence dimension, and the columns correspond to the feature dimension. The value matrix is ​​a matrix generated from the same source as the key matrix, with the same shape, and is used for feature information fusion in attention calculation. The query matrix is ​​a matrix generated after the current global attention layer performs a linear mapping on the input feature vector sequence, with a shape of [sequence length]. Feature Dimension The query matrix can be used to capture fine-grained requirements for the features of the current frame, and is used to calculate attention weights with the key matrix. The sequence dimension is the dimension corresponding to the row vectors in the matrix, which can represent the number of feature units in the video frame (i.e., the sequence length), and is the core dimension affecting the computational complexity of attention. The target key matrix and target value matrix are lightweight key and value matrices obtained after sparse sampling, with the sequence dimension length reduced to 1 / k of the original length (k is the target compression ratio), while the feature dimension remains unchanged.

[0078] In some embodiments, based on the input feature sequence of the current global attention layer, the corresponding key matrix and value matrix are generated through linear projection. Then, according to the determined target compression ratio, sparse sampling is performed on the key matrix and value matrix in the sequence dimension. Feature row vectors with high information content are retained, redundant row vectors are removed, and the feature dimension remains unchanged. Finally, the target key matrix and target value matrix with shortened sequence length are obtained. Without destroying the feature expressive ability, the size of the matrix involved in attention calculation is reduced, the amount of computation and memory consumption are reduced, thereby improving the inference speed of the model when processing long sequence videos.

[0079] Optionally, the steps of performing a linear mapping on the input feature sequence of the current layer to generate the query matrix, key matrix, and value matrix include:

[0080] Using the key projection parameters within the current layer, a linear mapping is performed on the input feature sequence to generate a key matrix;

[0081] Using the value projection parameters within the current layer, a linear mapping is performed on the input feature sequence to generate a value matrix;

[0082] Using the query projection parameters within the current layer, a linear mapping is performed on the input feature sequence to generate a query matrix.

[0083] Specifically, the key projection parameters, value projection parameters, and query projection parameters are learnable parameters built into the current global attention layer, containing a weight matrix and a bias vector, used to map the input feature sequence to the attention feature dimension. Linear mapping is an operation that transforms the input feature sequence from its original dimension to the dimension required for attention computation through matrix multiplication and bias concatenation.

[0084] For example, the current global attention layer calls the built-in independent key projection parameters, value projection parameters, and query projection parameters to perform linear mapping operations on the input feature sequence. Through matrix multiplication combined with bias stacking, the input feature sequence is accurately mapped from the original dimension to the preset feature dimension required for attention calculation, generating a sequence with shape [sequence length]. Feature Dimension The key matrix, value matrix, and query matrix of [sequence length] Feature Dimension ] represents the number of frame feature units in the input feature sequence of the current layer, i.e., the sequence length.

[0085] Optionally, according to the target compression ratio, the steps of sparsely sampling the key matrix and value matrix generated by the current layer based on the input feature sequence of the current layer in the sequence dimension to obtain the target key matrix and target value matrix include:

[0086] Using the key projection parameters within the current layer, a linear mapping is performed on the input feature sequence to generate a key matrix; and using the value projection parameters within the current layer, a linear mapping is performed on the input feature sequence to generate a value matrix.

[0087] A non-uniform sparse sampling algorithm that preserves time structure is adopted to perform frame-level sparse sampling on the key matrix in the sequence dimension according to the target compression ratio to obtain the compressed target key matrix;

[0088] A non-uniform sparse sampling algorithm that preserves time structure is adopted to perform frame-level sparse sampling on the value matrix in the sequence dimension according to the target compression ratio, so as to obtain the compressed target value matrix.

[0089] Specifically, frame-level sparse sampling involves retaining or discarding feature units on a whole-frame basis (such as retaining the feature vectors of all image blocks corresponding to a certain frame), thereby ensuring the geometric consistency within a single frame and avoiding local structural fragmentation caused by block-level sampling.

[0090] For example, to initiate a non-uniform sparse sampling algorithm that preserves temporal structure, the number of frame feature units to be retained (i.e., the sequence dimension retention length) is first determined based on the target compression ratio. / k, where k is the target compression ratio), and then, according to the original temporal order of the video frames, the sequence dimension of the key matrix is ​​divided into several consecutive temporal window subsequences. Within each subsequence, row vectors with a high proportion of core geometric features are selected by calculating feature information entropy, and row vectors corresponding to redundant and low-quality information are removed, completing the frame-level sparse sampling of the key matrix to obtain the target key matrix. Simultaneously, using the same temporal structure to retain the non-uniform sparse sampling algorithm, and following the sampling positions and selection rules completely consistent with the key matrix, synchronous frame-level sparse sampling is performed on the value matrix in the sequence dimension to ensure that the correspondence between the sequence dimensions of the key matrix and the value matrix is ​​not disrupted. After sampling, the feature dimensions of the target key matrix and the target value matrix are... While maintaining the same length, the sequence length is synchronously reduced to 1 / k of the original length, ultimately forming a lightweight and information-complete target key matrix and target value matrix, providing efficient input for subsequent attention calculations. After generating the standard key matrix and value matrix through linear mapping, the sequence dimension is selectively reduced according to the target compression ratio, directly reducing the attention computation complexity from O(N²) to O((N / k)²), thus reducing computational overhead and memory consumption. A non-uniform sampling algorithm that preserves temporal structure is adopted, which avoids random sampling from disrupting the temporal continuity of video frames and prioritizes the preservation of core features, avoiding motion breaks or structural distortions caused by random or block-level sampling. This significantly improves inference efficiency while ensuring the accuracy and stability of the 3D reconstruction results.

[0091] Step 103: Perform attention calculation based on the query matrix, target key matrix, and target value matrix corresponding to the current layer to obtain the context feature representation of the current layer, until all attention calculations of the Transformer backbone network are completed.

[0092] Specifically, attention calculation is the core operation based on the Transformer attention mechanism. It calculates the similarity (attention weight) between the query matrix and the target key matrix, and then uses this weight to perform a weighted summation on the target value matrix to achieve cross-frame feature fusion. The contextual features are represented as the feature sequence output after attention calculation, which integrates the core contextual information across frames with the features of the current frame. It is the final feature output after the current layer is processed and is used to pass to the next layer or the backbone network output.

[0093] In some embodiments, the current global attention layer invokes built-in query projection parameters to perform a linear mapping operation on the input feature vector sequence. Through matrix multiplication and bias superposition, the features are mapped from the original dimension to the preset feature dimension required for attention computation, generating a shape of [ , The query matrix of ] (where The number of frame feature units in the input feature vector sequence. (Feature dimension). Since the query matrix is ​​used to capture the fine-grained requirements and local geometric details of the current frame features, it is not compressed in any sequence dimension to ensure that the model's perception accuracy of the current features is not affected. The attention calculation process is initiated by first performing a dot product operation on the transpose of the query matrix and the target key matrix to obtain a similarity matrix representing the strength of the feature association between the current frame and cross-frame features; then, this similarity matrix is ​​Softmax normalized to transform it into an attention weight matrix in the 0-1 interval, highlighting the contribution ratio of core features; finally, the normalized attention weight matrix and the target value matrix are multiplied to complete the weighted fusion of cross-frame core features, generating a preliminary feature matrix. This preliminary feature matrix, after layer normalization and residual connection processing, forms the contextual feature representation of the current layer, which integrates key cross-frame contextual information while retaining the fine-grained features of the current frame, allowing seamless transfer to the next layer or as an intermediate output of the Transformer backbone network. The above process is executed sequentially in each global attention layer of the Transformer backbone network until all global attention layers and intra-frame attention layers in the Transformer backbone network have alternately completed all attention calculations, forming a high-dimensional output feature sequence that combines global temporal correlation and local detailed features. Through an asymmetric computation mode combining a complete query matrix and a lightweight key / value matrix, the computational complexity is significantly reduced while fully preserving the perception of input details. This effectively avoids the loss of geometric information caused by query matrix compression, thus ensuring the accuracy, stability, and temporal consistency of the 3D reconstruction results while significantly improving inference efficiency.

[0094] Step 104: Input the output feature sequence of the Transformer backbone network into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction results of the video to be reconstructed.

[0095] Specifically, the output feature sequence of the Transformer backbone network is the final feature sequence obtained after layer-by-layer compression, asymmetric attention computation, and multi-layer contextual feature fusion through all global attention layers of the Transformer backbone network, preserving the geometric and temporal information between video frames. The 3D reconstruction model is a feedforward 3D reconstruction model based on the Transformer architecture, which includes an image encoder, a Transformer backbone network, and a prediction head, used to directly output 3D geometric information from multi-view videos. The prediction head is the decoding network module at the end of the 3D reconstruction model, used to decode and regress high-dimensional abstract features into specific 3D geometric quantities, typically including depth prediction branches and camera pose prediction branches. The multi-view reconstruction result is the final 3D geometric result output by the 3D reconstruction model, which may include depth maps, camera intrinsic and extrinsic parameters, relative pose, scene point clouds, etc.

[0096] In some embodiments, the output feature sequence obtained after the Transformer backbone network completes multi-layer global attention calculation and feature fusion is input into the prediction head of the 3D reconstruction model. The prediction head performs convolution, linear mapping, and regression decoding on the high-dimensional feature sequence to parse the scene's depth information and viewpoint pose information. Combining the depth information and viewpoint pose information, differentiable back projection and cross-view... Figure 1 Consistent fusion generates globally consistent 3D point clouds, ultimately outputting multi-view 3D reconstruction results, including depth maps, camera poses, and 3D point clouds corresponding to the video to be reconstructed, completing the entire video 3D reconstruction process. The abstract features extracted by the Transformer backbone network are transformed into usable 3D geometric information. Without changing the model's output structure, it fully inherits the accelerated feature flow, seamlessly integrating the accelerated scheme of hierarchical sparse sampling and asymmetric attention computation into the reconstruction process. This improves inference efficiency while ensuring the accuracy and completeness of the 3D reconstruction results.

[0097] Optionally, the steps of inputting the output feature sequence of the Transformer backbone network into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction results of the video to be reconstructed include:

[0098] The output feature sequence of the Transformer backbone network is input into the first prediction sub-network in the prediction head. The camera pose information and depth information of each video frame are predicted by the first prediction sub-network to obtain the depth map and camera pose parameters corresponding to each video frame.

[0099] The output feature sequence of the Transformer backbone network, the depth map of each video frame, and the camera pose parameters are input into the second prediction sub-network in the prediction head. The second prediction sub-network predicts the point cloud information of each video frame and obtains the point cloud feature map corresponding to each video frame.

[0100] Based on the camera pose parameters and point cloud feature maps of each video frame, a 3D scene point cloud model corresponding to the video to be reconstructed is generated, and the 3D scene point cloud model, camera pose parameters, and depth map are used as the multi-view reconstruction results.

[0101] Specifically, the first prediction sub-network is the branch network in the prediction head responsible for preliminary geometric parameter prediction, used to directly regress camera pose and depth information from high-dimensional features. Video frames are each image frame in the video to be reconstructed, serving as the basic processing unit for 3D reconstruction. Camera pose information consists of key parameters characterizing the camera's shooting angle, including intrinsic and extrinsic parameters (such as rotation matrix, translation vector, focal length, etc.), used to determine the spatial positional relationships between frames. Depth information is the distance information from each pixel in the video frame to the camera's imaging plane, providing the foundational data for constructing the 3D structure. The depth map is the depth information presented in image form, with pixel values ​​corresponding to the actual distance at that location, serving as the core carrier for 2D-to-3D mapping. Camera pose parameters are a set of specific numerical values ​​representing the camera pose, providing a basis for spatial coordinate transformation for point cloud fusion. The second prediction sub-network is the branch network in the prediction head responsible for generating point cloud information, requiring the combination of high-dimensional features, depth maps, and camera pose parameters to achieve dense point cloud prediction. The point cloud feature map is graph-structured data fusing 3D coordinates and high-dimensional features, corresponding to the dense 3D point information of each frame. The 3D scene point cloud model is a complete 3D scene model formed by integrating the point cloud information of all video frames and calibrating and fusing them with spatial coordinates. It is the core result of multi-view reconstruction. The multi-view reconstruction result is a complete output set including the 3D scene point cloud model, camera pose parameters, and depth map, which can be directly used for downstream 3D applications (such as modeling, visualization, analysis, etc.).

[0102] For example, the output feature sequence obtained after multi-layer global attention computation, feature fusion, and hierarchical sparse sampling acceleration by the Transformer backbone network is input into the prediction head of the 3D reconstruction model. The prediction head adopts a two-level progressive decoding structure, which sequentially completes the prediction of basic geometric parameters, the generation of dense point cloud features, and the fusion of 3D scene models. In the first stage, the output feature sequence of the Transformer backbone network is input into the first prediction sub-network in the prediction head. This first prediction sub-network performs geometric decoding on the high-dimensional abstract features through multi-layer convolution, linear mapping, and regression operations, and directly predicts the camera pose information and depth information corresponding to each video frame. The camera pose information includes rotation matrix, translation vector, and camera intrinsic parameters, etc., and the depth information is the distance from each pixel to the camera imaging plane. This generates the depth map and quantized camera pose parameters corresponding to each video frame, thus completing the extraction of the basic spatial constraints required for 3D reconstruction. In the second stage, the output feature sequence of the Transformer backbone network, the depth map corresponding to each video frame, and the camera pose parameters are fed into the second prediction sub-network in the prediction head. Under the spatial constraints of depth and pose, this second prediction sub-network further integrates high-dimensional semantic and geometric features. Through dense prediction and coordinate mapping operations, it outputs the point cloud information corresponding to each video frame, forming a point cloud feature map containing 3D coordinates and feature information. Further, based on the camera pose parameters of each video frame, the point cloud feature maps corresponding to each frame are uniformly transformed to the world coordinate system. Through coordinate alignment, deduplication, filtering, and fusion operations, a complete 3D scene point cloud model corresponding to the video to be reconstructed is generated. Finally, the generated 3D scene point cloud model, the camera pose parameters of each video frame, and the depth map are combined to form a complete multi-view reconstruction result and output. This implementation process, through step-by-step decoding, constraint transfer, and spatial fusion, can stably and efficiently transform accelerated abstract features into usable 3D geometric results. Without sacrificing reconstruction accuracy, it significantly improves the inference efficiency of the model in processing long video sequences, adapting to practical application needs such as real-time 3D reconstruction and large-scale scene digitization.

[0103] Through a progressive decoding approach using two-level predictive subnetworks, the first predictive subnetwork quickly obtains depth maps and camera pose parameters, providing fundamental constraints for 3D spatial localization. The second predictive subnetwork then fuses feature, depth, and pose information to generate a high-precision point cloud feature map. Finally, based on the camera pose, multi-frame point cloud alignment and fusion are performed to obtain a unified 3D scene point cloud model. While ensuring reconstruction accuracy and output integrity, this approach seamlessly integrates the aforementioned hierarchical sparse sampling and asymmetric attention acceleration mechanisms. It significantly reduces computational load and improves inference speed while stably outputting high-quality 3D geometric results, thus solving the technical problems of low efficiency and difficulty in reconstructing long sequences in existing models.

[0104] refer to Figure 2 , Figure 2 This is a schematic diagram of the architecture of a 3D reconstruction model provided in this application. After the video to be reconstructed is extracted into an initial feature vector sequence corresponding to the reconstructed video by the image encoder, it is propagated forward to the Transformer backbone network of the 3D reconstruction model. The Transformer backbone network is composed of alternating stacked global attention layers and intra-frame attention layers. When the features are propagated to each global attention layer, the mapping relationship between the layer and the compression ratio, which is pre-constructed offline, is called according to the layer identifier of the current layer to determine the target compression ratio suitable for the current layer. Then, a linear mapping is performed on the input feature sequence of the current layer to generate a complete query matrix, key matrix, and value matrix. According to the target compression ratio, a non-uniform sparse sampling algorithm that preserves the temporal structure is used to perform frame-level sampling on the key matrix and value matrix in the sequence dimension (frame dimension) to obtain the target key matrix and target value matrix. This sampling process prioritizes the preservation of frame features with high information entropy while maintaining the integrity of the video temporal structure.

[0105] Next, attention calculation is performed based on the complete query matrix, the compressed target key matrix, and the target value matrix: First, a similarity matrix is ​​obtained by performing a dot product operation on the query matrix and the transpose of the target key matrix. This matrix is ​​then normalized using Softmax to generate attention weights. These weights are then weighted and summed with the target value matrix to output the contextual feature representation of the current layer, which incorporates the global context. The contextual feature representation generated by the global attention layer is then passed to the adjacent intra-frame attention layer. After single-frame local feature enhancement, it continues to propagate to the next global attention layer, repeating the adaptive compression and attention calculation process until all attention calculations in the Transformer backbone network are completed. Finally, the feature sequence output by the Transformer backbone network is input into the prediction head of the 3D reconstruction model. After decoding, the multi-view 3D reconstruction result of the video to be reconstructed is output.

[0106] Based on the 3D scene reconstruction acceleration method provided in this application, the video to be reconstructed is input into a 3D reconstruction model for forward propagation. When propagating to each global attention layer of the Transformer backbone network in the 3D reconstruction model, the target compression ratio corresponding to the current layer is determined according to the layer identifier and the mapping relationship between the layer and the compression ratio. This achieves differentiated compression at different layers, balancing compression efficiency and feature information preservation, avoiding the limited efficiency improvement or feature loss caused by indiscriminate compression, and laying the foundation for subsequent accurate compression. A linear mapping is performed on the input feature sequence of the current layer to generate a query matrix, a key matrix, and a value matrix. According to the target compression ratio, sparse sampling is performed on the key matrix and value matrix along the sequence dimension to obtain the target key matrix and target value matrix. The sequence length of the key matrix and value matrix is ​​reduced to the reciprocal of the original length of the target compression ratio, thereby reducing the computational cost of subsequent attention processing. Next, attention calculations are performed based on the query matrix, target key matrix, and target value matrix corresponding to the current layer to obtain the contextual feature representation of the current layer. This process continues until all attention calculations in the Transformer backbone network are completed. The contextual feature representation of each global attention layer has the same dimension as the input feature sequence and can be directly passed layer by layer in the Transformer backbone network. Finally, after completing all attention calculations in the Transformer backbone network, the output feature sequence of the Transformer backbone network is input into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction results of the video to be reconstructed. This application reduces the computational complexity of global attention at each layer by performing sequential dimensionality sparse compression on the key and value matrices before calculating global attention in the forward propagation of the 3D reconstruction model, based on a pre-defined compression ratio at each layer, while preserving the integrity of the query matrix. Attention calculation is then performed, and the reconstruction result is finally output through the prediction head. The entire process does not change the original forward propagation logic of the model. Without compromising the reconstruction capability of the 3D reconstruction model, the computational complexity of global attention at each layer is directly reduced, and the overall model processing efficiency is improved by layering these reductions. This solves the technical problem of low processing efficiency caused by the high computational complexity of attention in the Transformer architecture 3D reconstruction model, and ensures the accurate completion of multi-view 3D reconstruction tasks while improving efficiency.

[0107] In some embodiments, before the step of determining the target compression ratio corresponding to the current layer based on the layer identifier and the mapping relationship between the layer and the compression ratio, the method further includes:

[0108] Obtain the test dataset, which includes multiple test videos;

[0109] The test video is input into the trained 3D reconstruction model for forward inference, and the attention matrix output by each global attention layer in the 3D reconstruction model is extracted.

[0110] The effective rank value of the global attention layer is obtained by calculating the effective rank based on the attention matrix.

[0111] Perform rule mapping on the effective rank values ​​to generate the compression ratio corresponding to the global attention layer;

[0112] Based on the compression ratio corresponding to the global attention layer and the hierarchical identifier of the global attention layer, a mapping relationship between the hierarchical level and the compression ratio is constructed.

[0113] Specifically, the test dataset is a representative collection of multi-view videos, such as those from Common Objects in 3D (CO3D) and RealEstate10K, used to simulate real-world reconstruction scenarios. These videos are not used for model training but are instead used to analyze model behavior and evaluate the feature redundancy of each global attention layer. The trained 3D reconstruction model is a feedforward 3D reconstruction model based on the Transformer architecture, such as VGGT, which has already undergone parameter training and possesses basic 3D reconstruction capabilities. Pixel-aligned implicit parameterized prediction is used for this purpose. ), arbitrary depth models for 3D scene reconstruction (Depth-Anything in 3D, Depth-Anything-3), etc.

[0114] Effective rank is a natural generalization of the traditional algebraic rank concept to numerical computation and deep learning. Given a matrix, the algebraic rank is defined as the number of non-zero singular values, essentially treating all non-zero singular values ​​equally and simply counting them as "present" or "absent." However, in the actual computation of deep neural networks, the magnitude of singular values ​​directly reflects the information contribution of the corresponding components to the overall behavior of the matrix: larger singular values ​​carry the main geometric and semantic information, while extremely small singular values ​​often originate from noise, rounding errors, or numerical redundancy. Effective rank, by weighted summation of the number of singular values, incorporates the "importance" of each non-zero component into the rank measurement, thus more precisely characterizing the true density of effective information in the matrix. Within this conceptual framework, algebraic rank can be seen as a degenerate form of effective rank—a special case where all non-zero singular values ​​are assigned equal weight (weight 1). When a matrix contains a large number of singular values ​​with small but non-zero magnitudes, the algebraic rank will artificially inflate the information capacity, while the effective rank can reasonably attenuate this through a weighting mechanism, thus accurately reflecting the nature of information redundancy. The effective rank has various specific implementations; this application uses the stable rank as the core metric for calculating redundancy. The definition of the stable rank is:

[0115]

[0116] in for matrix, Representing its singular values, this index is divided by the largest singular value. Using the squares of normalized singular values ​​as weights aligns with the core idea of ​​effective rank weighted summation, while also offering significant advantages in terms of metric rationality and computational efficiency. Metric rationality stems primarily from the long-tail characteristic of the singular value distribution in attention matrices—a few singular values ​​at the head dominate the matrix's main information energy, while a large number of tiny singular values ​​at the tail originate from numerical noise and computational redundancy. Stable rank assigns higher weights to the principal components at the head by weighting the squares of the singular values, while adaptively attenuating the noise components at the tail. Compared to algebraic rank, which treats all non-zero singular values ​​indiscriminately, stable rank accurately characterizes the essential difference between numerically full rank and informationally redundant values, exhibiting superior discriminative ability in deep learning scenarios where long-tail phenomena are prevalent in attention matrices.

[0117] The computational efficiency is mainly due to the fact that both the algebraic rank and the entropy-based effective rank implementation require complete singular value decomposition, resulting in a computational complexity of O(n log n). When dealing with long input sequences, computational overhead is significant. The stable rank rule, however, does not require decoupling of all singular subspaces; it only needs to compute the Frobenius norm and spectral norm of the matrix, thus reducing the complexity to [a much lower level]. This advantage significantly reduces the computational cost of performing systematic redundancy analysis on each global attention layer in the feedforward 3D reconstruction model, thus supporting efficient redundancy calibration in long sequence scenarios. Therefore, this application can use stable rank as a quantitative indicator of the redundancy of each attention layer. Based on this indicator, the information density of different global attention layers can be accurately calibrated, thereby dynamically and adaptively allocating differentiated context compression strategies to each global attention layer, achieving the best balance between computational efficiency and reconstruction accuracy.

[0118] This application calculates the effective rank using stable rank, which is calculated by dividing the sum of the squares of all singular values ​​of the matrix by the square of the largest singular value (i.e., the square of the Frobenius norm divided by the square of the spectral norm). Compared to the traditional algebraic rank (number of non-zero singular values), stable rank can distinguish the contribution of singular values ​​of different sizes through weighting, more accurately characterizing the actual effective information content. The smaller the value, the more concentrated the information and the higher the redundancy. The lower the calculated effective rank value, the higher the matrix redundancy and the larger the compressible space.

[0119] As an example, a test dataset with diverse scenarios is obtained, containing multiple multi-view test videos to represent the input distribution in real-world applications. Subsequently, the test videos from the dataset are sequentially fed into a pre-trained 3D reconstruction model for forward inference. Hook functions are inserted into each global attention layer of the Transformer backbone network, and the output attention matrix A∈R is extracted. N×N (N is the sequence length). Next, singular value decomposition is performed on each attention matrix A to obtain the singular value sequence { , ,…, …, }, and based on this, calculate the effective rank of the global attention layer. The formula for calculating the stable rank SRank(A) of the global attention layer A is:

[0120]

[0121] in, Let Frobenius norm be the global attention layer A. The spectral norm (i.e., the maximum singular value) of the global attention layer A ), Here are the singular values ​​of the global attention layer A. The formula for calculating the stable rank SRank(A) reflects redundancy by measuring the concentration of matrix energy on principal components. The smaller the stable rank value, the more concentrated the matrix information of the attention matrix output by the global attention layer is in a few principal directions, the higher the redundancy, and the larger the compressibility space. For attention matrices from multiple test videos of the same layer, the statistical average (or median) of their stable ranks can be calculated as the final, robust, and effective rank value of the global attention layer.

[0122] Next, using a pre-defined rule mapping function, the effective rank value calculated for each global attention layer is converted into a specific compression ratio k. The mapping rule can be monotonically decreasing, meaning that layers with smaller effective rank values ​​(higher redundancy) are assigned larger k values ​​(indicating a more aggressive compression strategy); conversely, layers with larger effective rank values ​​(information-dense) are assigned smaller k values ​​(adopting a conservative compression strategy). Optionally, for global attention layers with high redundancy, the value of k can range from 30 to 90; for global attention layers with low redundancy, the value of k can range from 10 to 30. Finally, the layer identifier (i.e., its depth position index) of each global attention layer is paired with the corresponding compression ratio k to construct a complete mapping relationship between layers and compression ratios. This mapping relationship is directly invoked in the subsequent inference stage, allowing each global attention layer to adopt an adaptive compression ratio based on its own redundancy. This minimizes the computational load of global attention without significantly reducing the accuracy of 3D reconstruction, thereby improving the inference speed of the model when processing long video sequences.

[0123] For example, this application uses effective rank information obtained from a given calibration dataset (e.g., a dataset consisting of several scene sequences of 30 frames, with the first frame taken every 10 frames) to perform hierarchical adaptive calibration of the compression ratio k of the global attention layer: First, for each global attention layer, the effective rank of all its attention heads is calculated, and the layer is divided into three categories: low-rank, medium-rank, and high-rank layers according to the effective rank threshold. If the effective rank of all attention heads in the current layer is less than 10, it is determined to be a low-rank layer, and the default compression ratio k=30 is used; if there is an attention head with an effective rank greater than 10 but less than 100, it is determined to be a medium-rank layer, and the default compression ratio k=15 is used; if there is an attention head with an effective rank greater than 100, it is determined to be a high-rank layer, and the default compression ratio k=10 is used. This hierarchical calibration rule implements the adaptive logic that the higher the redundancy, the greater the compression ratio. Stronger compression is used in low-rank layers (high feature redundancy) to maximize acceleration, while gentler compression is used in high-rank layers (high feature information density) to ensure reconstruction accuracy, thereby achieving an optimal balance between inference speed and 3D reconstruction quality.

[0124] By employing offline statistics, effective rank quantization, and adaptive mapping, a compression ratio matching the feature redundancy is assigned to each global attention layer. This achieves higher compression for highly redundant layers and lower compression for low-redundancy layers, ensuring that key features are not lost while significantly reducing the computational cost of attention. Simultaneously, the mapping relationships are directly used during inference without increasing online computational overhead. This makes the entire hierarchical sparse sampling acceleration scheme interpretable, adaptive, and practical, thereby solving the technical problems of high computational cost and slow inference speed in Transformer 3D reconstruction models with long video sequences.

[0125] Furthermore, to address the technical issues of high computational complexity and low efficiency in processing long video sequences in existing Transformer-based 3D reconstruction models, this application proposes a hierarchical adaptive sparse attention mechanism to replace the original computationally expensive global attention calculation, based on a quantitative evaluation of the redundancy of the global attention matrix at each layer. This hierarchical adaptive sparse attention mechanism is encapsulated as a plug-and-play general acceleration module, balancing acceleration performance with broad applicability.

[0126] The core of this hierarchical adaptive sparse attention mechanism lies in constructing an asymmetric compression logic linked to feature redundancy. The specific implementation process is as follows: During the model inference phase, when the feature sequence propagates to each global attention layer of the Transformer backbone network, the complete query matrix is ​​first preserved without any sequence-dimensional compression. This ensures the model can continuously maintain fine-grained perception of input features, avoiding the loss of geometric details and feature association information due to query matrix compression. Subsequently, the mapping relationship between layers and compression ratios generated offline is invoked. Based on the layer identifier of the current layer, the corresponding target compression ratio k is matched. This target compression ratio k is obtained by mapping the effective rank (redundancy quantification index) of the current layer's attention matrix using rules; higher redundancy results in a larger compression ratio. Next, a non-uniform sparse sampling algorithm that preserves temporal structure is employed to sparsely sample the key and value matrices generated from the input feature sequence in the current layer along the sequence dimension (frame dimension). This sampling algorithm, while ensuring a relatively uniform temporal distribution of sampling points and preserving the temporal structure of the video frames, prioritizes the retention of row vectors corresponding to frames with high information content by calculating feature entropy, eliminating redundant and low-quality row vectors. This reduces the sequence length of the key and value matrices to the reciprocal of the target compression ratio. Finally, attention calculation is performed based on the complete key matrix and the compressed key matrix (i.e., the target key matrix) and value matrix (i.e., the target value matrix). The complexity of the attention calculation is significantly reduced from O(N²) (where N is the number of frames in the input video) of the original global attention to O(N² / k), greatly reducing computational overhead while ensuring the accuracy of feature fusion.

[0127] To enhance the versatility of the hierarchical adaptive sparse attention mechanism, this application further designs it as an independent plug-and-play acceleration module. This module is encapsulated as a neural network layer conforming to a standard attention interface, with input and output formats fully compatible with conventional global attention layers. It can directly replace the original global attention layers in various Transformer-based feedforward 3D reconstruction models, such as VGGT. This design allows for seamless integration with various Transformer-based feedforward 3D reconstruction models, such as Depth-Anything-3, without requiring modifications to other network structures. In practical deployment, users only need to perform an offline analysis on the 3D reconstruction model: input the test dataset into the 3D reconstruction model for forward inference, extract the attention matrices of each global attention layer, and generate a mapping relationship between the layers and compression ratios adapted to the 3D reconstruction model through effective rank calculation and rule mapping. During subsequent model inference, simply load this mapping relationship, and the acceleration module will automatically enable adaptive compression. No modifications to any pre-trained weights of the target model or additional fine-tuning are required, achieving seamless integration with the 3D reconstruction model. This design enables lossless transfer of this application to various Transformer-based feedforward 3D reconstruction models, significantly improving the model's processing efficiency for long video sequences without sacrificing 3D reconstruction accuracy, and greatly expanding its application scope.

[0128] In some embodiments, the step of using a time-structure-preserving non-uniform sparse sampling algorithm to perform frame-level sparse sampling on the key matrix along the sequence dimension according to the target compression ratio to obtain the compressed target key matrix includes:

[0129] The number of row vectors to be retained in the key matrix is ​​determined based on the target compression ratio;

[0130] The key matrix is ​​divided into first frame groups according to the number of video frames. Each first frame group includes multiple first row vectors, which are the row vectors in the key matrix.

[0131] Within each first frame group, the information entropy of each first row vector is calculated, and the first row vector corresponding to the maximum information entropy is determined as the first target row vector corresponding to the first frame group.

[0132] The target key matrix is ​​reconstructed based on the first target row vector corresponding to each first frame group.

[0133] Specifically, the target compression ratio is the current layer compression factor k determined by the preceding offline calibration, used to calculate the total number of row vectors to be retained, N′=N / k. The number of row vectors to be retained is the number of rows N′ in the compressed target key matrix, which directly determines the number of subsequent groups. The first frame group is a set of row vectors divided according to the temporal order of video frames. Each frame group contains several consecutive row vectors, the total number of which is consistent with the number of row vectors to be retained. The first row vector is the row vector corresponding to a single video frame in the key matrix, and is the basic unit of frame-level sampling. Information entropy is an indicator of the richness of feature information in row vectors. The higher the information entropy value, the more core the geometric / semantic information contained in the frame features.

[0134] As an example, the number of row vectors N′ to be retained in the key matrix is ​​determined based on the target compression ratio k. If the original sequence length of the key matrix is ​​N (corresponding to N video frames) and the target compression ratio is k, then the number of row vectors to be retained is N / k, i.e., N′ = N / k. Then, according to the original temporal order of the video frames, the N row vectors of the key matrix are evenly divided into N / k first frame groups. Each first frame group contains k consecutive first row vectors (i.e., row vectors corresponding to k consecutive video frames), ensuring that the frame group division strictly follows the temporal order and does not disrupt the video's temporal structure. Next, within each first frame group, the information entropy of all first row vectors is calculated, for example, by measuring their texture richness and discriminativeness using the L2 norm or feature variance. The row vector with the highest information entropy is selected as the first target row vector corresponding to that frame group, prioritizing the preservation of core geometric and semantic information. Finally, according to the temporal order of each first frame group, all first target row vectors are sequentially arranged and reconstructed to obtain the target key matrix. While meeting the target compression ratio, a group selection strategy is adopted to prioritize the key vectors corresponding to the video frames with the richest information in each group, while preserving the temporal continuity of the video. This approach significantly reduces the sequence length while maximizing the retention of contextual information valuable for 3D geometric reconstruction.

[0135] In some embodiments, the step of using a time-structure-preserving non-uniform sparse sampling algorithm to perform frame-level sparse sampling on the value matrix along the sequence dimension according to the target compression ratio to obtain the compressed target value matrix includes:

[0136] The value matrix is ​​divided into second frame groups according to the number of video frames. Each second frame group includes multiple second row vectors, which are the row vectors in the value matrix.

[0137] Within each second frame group, the information entropy of each second row vector is calculated, and the second row vector corresponding to the maximum information entropy is determined as the second target row vector corresponding to the second frame group.

[0138] The target value matrix is ​​reconstructed based on the second target row vector corresponding to each second frame group.

[0139] The specific implementation process of frame-level sparse sampling of the value matrix is ​​strictly synchronized with the key matrix to ensure the semantic consistency of the attention mechanism. As an example, using the number of row vectors to be retained (calculated from the target compression ratio) determined during key matrix sparse sampling, all second row vectors of the value matrix (corresponding to row vectors in a single video frame) are uniformly divided into second frame groups, consistent with the number of row vectors to be retained, according to the original temporal order of the video frames. Each second frame group contains several consecutive second row vectors, ensuring that the frame group division strictly follows temporal logic and does not disrupt the video's temporal structure. Subsequently, within each second frame group, the information entropy of each second row vector is calculated, for example, using its L2 norm or eigenvariance as a surrogate indicator of information content. The second row vector with the largest information entropy is selected as the second target row vector corresponding to that frame group, thus ensuring that the retained value vectors come from the most discriminative local observations. Finally, the number of second target row vectors are concatenated according to the original temporal order to reconstruct the compressed target value matrix. By using the same frame grouping rules, sampling positions, and screening criteria as the key matrix sampling, the sequential dimension correspondence between the key matrix and the value matrix is ​​ensured to remain intact. While achieving lightweighting of the value matrix according to the target compression ratio, the integrity of the temporal structure and the effectiveness of features are also taken into account. This ensures the accuracy of subsequent attention calculations, reduces computational overhead, and avoids the decrease in reconstruction accuracy caused by the loss of core fusion information.

[0140] In some embodiments, the Transformer backbone network of the 3D reconstruction model consists of multiple alternating global attention layers and intra-frame attention layers.

[0141] Specifically, alternating arrangement refers to the structure in which the global attention layer and the intra-frame attention layer are connected in series and stacked in a fixed order, such as: global attention layer → intra-frame attention layer → global attention layer → intra-frame attention layer... The intra-frame attention layer is an attention layer that performs feature interaction within a single frame. It does not establish inter-frame correlations and focuses on modeling fine features such as local textures, edges, and geometric details within a single frame image.

[0142] In this application, the Transformer backbone network of the 3D reconstruction model is constructed using an alternating stacked structure of global attention layers and intra-frame attention layers. During feature forward propagation, the initial feature vector sequence corresponding to the reconstructed video first enters the first global attention layer, where inter-frame feature association and context fusion are performed across the entire sequence to capture long-distance cross-frame geometric correspondences. Subsequently, the output feature sequence is fed into the adjacent intra-frame attention layer, where local feature interaction and detail enhancement are performed only within each video frame to strengthen the expression of texture and geometric features within a single frame. After completion, the feature sequence is fed into the next global attention layer for deeper cross-frame information fusion, and then the subsequent intra-frame attention layers further refine the single-frame features. The alternating and progressive processing of the global attention layer and intra-frame attention layer allows the Transformer backbone network to simultaneously complete global temporal modeling and local feature enhancement at each layer, ultimately outputting a highly expressive feature sequence that integrates global context and single-frame details.

[0143] In some embodiments, when propagating to each intra-frame attention layer of the Transformer backbone network, the input feature sequence of the intra-frame attention layer is grouped according to video frames to obtain the input features corresponding to each video frame.

[0144] Perform self-attention processing on each input feature to obtain the self-attention processing results for each video frame;

[0145] The self-attention processing results corresponding to each video frame are concatenated, and the concatenated results are used as the input feature sequence for the next global attention layer.

[0146] Specifically, the input feature sequence is the feature tensor that enters the attention layer of the current frame.

[0147] As an example, when the feature sequence propagates to the intra-frame attention layer of the Transformer backbone network, it first groups the input feature sequence of the intra-frame attention layer based on the temporal partitioning rules of the video frames. Taking a single video frame as an independent unit, the continuous input feature sequence is split into input features corresponding to each video frame, ensuring that each group of input features contains only the high-dimensional feature information of a single frame and does not mix in feature data from other frames. Subsequently, for the input features corresponding to each video frame, self-attention processing is performed independently. By calculating the correlation weights between feature points within a single frame feature, local texture, edge details, and geometric structure features are enhanced and optimized, achieving deep interaction and refined expression of features within a single frame, resulting in self-attention processing results specific to each video frame. Finally, according to the temporal order of the original video frames, the self-attention processing results corresponding to all video frames are concatenated sequentially to reconstruct a complete feature sequence with the same length as the input feature sequence. This concatenated feature sequence is then directly used as the input feature sequence for the next global attention layer, providing a high-quality feature foundation enhanced with local details for subsequent long-distance inter-frame dependency modeling and global context fusion. By employing a process of grouping, independent processing of single frames, and temporal stitching, the expression of local features in a single frame is enhanced while avoiding redundant calculations between frames. This forms a complementary collaborative mode with the global attention layer, which enhances local details and models global correlations. This not only improves the completeness and accuracy of feature expression but also reasonably controls computational overhead, providing structural support for balancing 3D reconstruction accuracy and inference efficiency.

[0148] refer to Figure 3 , Figure 3This diagram illustrates the context-compression-based 3D reconstruction acceleration method provided in this application. Each video frame from the video to be reconstructed is input into the 3D reconstruction model. The image encoder of the 3D reconstruction model extracts features from each video frame to obtain an initial feature vector sequence corresponding to the video to be reconstructed. This initial feature vector sequence is then input into the Transformer backbone network of the 3D reconstruction model for forward propagation. When propagating to each global attention layer of the Transformer backbone network in the 3D reconstruction model, the target compression ratio corresponding to the current layer is determined based on the layer identifier and the mapping relationship between the layer and the compression ratio. According to the target compression ratio, the key matrix and value matrix generated by the current layer based on the input feature sequence of the current layer are sparsely sampled along the sequence dimension to obtain the target key matrix and target value matrix. The query matrix generated by the current layer based on the input feature vector sequence is determined as the target query matrix. Attention calculation is performed based on the target query matrix, target key matrix, and target value matrix to obtain the context feature representation of the current layer. The Transformer backbone network of the 3D reconstruction model consists of multiple alternating global attention layers and intra-frame attention layers. When propagating to each intra-frame attention layer of the Transformer backbone network, the input feature sequence of the intra-frame attention layer is grouped according to video frames to obtain the input features corresponding to each video frame; self-attention processing is performed on each input feature to obtain the self-attention processing result corresponding to each video frame; the self-attention processing results corresponding to each video frame are concatenated, and the corresponding concatenated result is used as the input feature sequence of the next global attention layer. The output feature sequence of the Transformer backbone network is input into the first prediction subnetwork in the prediction head. The first prediction subnetwork predicts the camera pose and depth information of each predicted video frame, obtaining the depth map and camera pose parameters corresponding to each predicted video frame. The output feature sequence of the Transformer backbone network, the depth map and camera pose parameters of each predicted video frame are input into the second prediction subnetwork in the prediction head. The second prediction subnetwork predicts the point cloud information of each predicted video frame, obtaining the point cloud feature map corresponding to each predicted video frame. Based on the camera pose parameters and point cloud feature maps of each predicted video frame, a 3D scene point cloud model corresponding to the video to be reconstructed is generated. The 3D scene point cloud model, camera pose parameters and depth map are used as the multi-view reconstruction result.

[0149] Furthermore, to systematically verify the effectiveness, versatility, and performance advantages of the 3D scene reconstruction acceleration method provided in this application in practical applications, this application deployed the 3D scene reconstruction acceleration method provided in this application on several mainstream visual geometry base models, including VGGT, π³ (Pi-cubed), and Depth-Anything-3 (DA3). Experiments were conducted on core tasks such as camera pose estimation, depth prediction, and 3D point cloud reconstruction, with the original versions of each model and the recently proposed Fast Visual Geometry TransformerFastVGGT serving as the main comparison baselines. Evaluation was conducted on two standard multi-view reconstruction datasets: the Seven Scenes Dataset (7-Scenes) and the NYU RGB-D multi-view dataset (NRGBD). The reconstruction quality was comprehensively measured using three metrics: accuracy (Accuracy, Acc↓), completeness (Comp↓), and normal consistency (Normal Consistency, NC↑), while the end-to-end inference time (Time↓) was also recorded. As shown in Table 1, the 3D scene reconstruction acceleration method provided in this application demonstrates excellent acceleration effect and accuracy preservation capability on different models. Table 1 shows the comparison of point cloud reconstruction results on 7-Scenes and NRGBD datasets.

[0150] Table 1: Comparison of point cloud reconstruction results on 7-Scenes and NRGBD datasets

[0151]

[0152] Compared to using the original VGGT for inference, when the 3D scene reconstruction acceleration method provided in this application is deployed on VGGT (Ours+VGGT), it can bring about a 10x improvement in inference speed, while the reconstruction accuracy is significantly better than the original model (i.e., VGGT) and FastVGGT. Compared to using the original DA3 for inference, when the 3D scene reconstruction acceleration method provided in this application is deployed on the Depth-Anything-3 model (i.e., Ours+DA3), it can achieve about an 8x speedup while maintaining comparable accuracy. Compared to using the original... Reasoning is performed, when the 3D scene reconstruction acceleration method provided in this application is deployed... Model (i.e., Ours+) When implemented, it can achieve more than 12 times speedup while completely maintaining its original model (i.e. The method provides high-precision reconstruction capabilities. Compared to the dedicated acceleration method FastVGGT, the 3D scene reconstruction acceleration method provided in this application achieves comprehensive superiority in both speed and accuracy, and can be seamlessly transferred to Transformer 3D reconstruction models with different architectures, fully verifying its cross-model architecture versatility and providing an efficient and lossless acceleration solution for various visual geometric base models.

[0153] To further evaluate the performance of the 3D scene reconstruction acceleration method provided in this application on camera pose estimation tasks, systematic experiments were conducted on two challenging dynamic and indoor scene datasets: the TUM Dynamic Scenes Dataset (TUM-dynamics) from the Technical University of Munich and ScanNet, a semantically annotated indoor scene 3D reconstruction dataset. Absolute trajectory error (ATE), relative translation error (RPEtrans), and relative rotation error (RPErot) were used as the core evaluation metrics. The results are shown in Table 2, which compares the pose estimation results on the TUM-dynamics and ScanNet datasets.

[0154] Table 2: Comparison of pose estimation results on TUM-dynamics

[10] and ScanNet

[11] datasets

[0155]

[0156] Compared to using the original VGGT for inference, when the 3D scene reconstruction acceleration method provided in this application is deployed on VGGT (Ours+VGGT), the inference time is reduced from 993.7 seconds to 98.9 seconds (approximately a 10-fold speedup), while the ATE decreases from 0.0183 to 0.0157. RPE-t and RPE-r also improve simultaneously, with key error indicators decreasing by 14%–60% overall. This indicates that the compression mechanism effectively filters out noise interference and enhances geometric consistency. Compared to using the original DA3 for inference, when the 3D scene reconstruction acceleration method provided in this application is deployed on Depth-Anything-3 (Ours+DA3), it can maintain its high accuracy advantage while compressing the inference time from 870.8 seconds to 111.8 seconds (approximately a 7–8-fold speedup), with stable or even slightly improved pose error indicators. Compared to using the original... The model performs inference, and when the 3D scene reconstruction acceleration method provided in this application is deployed... Model (Ours+) On ScanNet, this method achieves a speedup of over 12 times (862.2s → 71.2s), while ATE, RPE-t, and RPE-r remain almost unchanged. In contrast, while existing dedicated acceleration methods like FastVGGT offer some speedup, their pose accuracy on ScanNet is significantly inferior to our method (e.g., ATE: 0.0780 vs. 0.0634). Experimental results fully demonstrate that our hierarchical adaptive sparse attention mechanism significantly reduces computational overhead while stably preserving the core geometric features required for 3D reconstruction and camera pose estimation, providing an efficient and accurate acceleration solution for various visual geometric models.

[0157] To evaluate the performance of the 3D scene reconstruction acceleration method provided in this application on video depth estimation tasks, systematic experiments were conducted on the Bonn and KITTI standard datasets, which cover complex indoor and outdoor scenes. Absolute relative error (Abs Rel↓) and threshold accuracy (δ<1.25↑) were used as the core evaluation metrics. As shown in Table 3, the 3D scene reconstruction acceleration method provided in this application performed well on VGGT, Depth-Anything-3 (DA3), and... Significant inference acceleration has been achieved on various mainstream models, while the depth estimation accuracy remains basically unchanged or even slightly improved.

[0158] Table 3: Comparison of video depth estimation results on the Bonn and KITTI datasets

[0159]

[0160] On the Bonn dataset (containing long video sequences), the 3D scene reconstruction acceleration method provided in this application outperforms VGGT, DA3, and... This achieved speedups of approximately 9.6 times (552.4s → 57.3s), 6.9 times (493.4s → 71.1s), and 10.7 times (413.1s → 38.8s), respectively, with Abs Rel and δ < 1.25 indices highly consistent with the original model. Particularly noteworthy is the significant improvement compared to using the original... The model performs inference, and when the 3D scene reconstruction acceleration method provided in this application is deployed... Model (Ours+) When using the 3D scene reconstruction acceleration method provided in this application, the speedup is not only over 10 times faster, but also further reduces Abs Rel from 0.0329 to 0.0308, indicating that its adaptive compression mechanism can effectively filter out redundant or low-quality frames, thereby improving the robustness of depth estimation. In contrast, on the KITTI dataset, since about 61.5% of the test sequences are less than 200 frames long, the model does not fully expose the computational bottleneck of long sequences, so the speedup is relatively small (e.g., VGGT only speeds up by about 2.9 times). However, deploying the 3D scene reconstruction acceleration method provided in this application on VGGT for inference still outperforms using FastVGGT for inference, and the depth accuracy remains intact. The above results fully demonstrate that the 3D scene reconstruction acceleration method provided in this application can not only efficiently process long video inputs and significantly reduce computational overhead, but also accurately preserve the high-information context that is crucial to depth estimation through a non-uniform sparse sampling strategy that preserves temporal structure, thereby significantly improving inference efficiency while maintaining or even improving the quality and consistency of depth prediction.

[0161] refer to Figure 4 , Figure 4 This diagram illustrates the comparison of subjective 3D reconstruction results on the VGGT model. The diagrams represent the original VGGT model, the dedicated acceleration method FastVGGT, and the 3D scene reconstruction acceleration method (Ours+VGGT) proposed in this application. In terms of inference time, the original VGGT model takes 1146 seconds to reconstruct, FastVGGT takes 263 seconds (a reduction of approximately 77%), and the 3D scene reconstruction acceleration method proposed in this application takes only 90 seconds (a reduction of approximately 92%), significantly outperforming the dedicated acceleration method in terms of acceleration. Regarding reconstruction results, all three sets of results completely restore the overall structure and geometric contours of the scene. The 3D scene reconstruction acceleration method (Ours+VGGT) proposed in this application achieves maximum acceleration while maintaining a more uniform point cloud density distribution. The integrity of details such as walls and doors is essentially consistent with the original VGGT model and is superior to the FastVGGT solution. As can be seen from the local details marked in the box below, the 3D scene reconstruction acceleration method of this application retains richer texture and edge information, has less point cloud noise, and the clarity of scene details is comparable to the original model, without geometric distortion or loss of details caused by sparse sampling.

[0162] refer to Figure 5 , Figure 5 This diagram illustrates a comparison of subjective scene-level 3D reconstruction results using the acceleration scheme of this application on three cutting-edge pedestal models: VGGT, Depth-Anything-3, and π³. (a), (c), and (e) represent the reconstruction results of the original versions of each model, while (b), (d), and (f) represent the corresponding results after embedding the acceleration module provided in this application. Figure 4In this context, "TS" represents the Time Saving ratio. From a time efficiency perspective, the acceleration scheme in this application achieves a 92% time compression on the VGGT model and a 90% time compression on the Depth-Anything-3 model. The model also achieved a 92% time compression, resulting in an inference speedup of approximately 7-12 times, fully validating the versatility and efficient acceleration capabilities of this method across model architectures. In terms of reconstruction quality, all three accelerated results fully preserved the scene structure and detail representation of the original model: on the VGGT model, the accelerated point cloud density was more uniform, and geometric details such as indoor tables, chairs, and walls were clearly discernible, without distortion caused by sparse sampling; on the Depth-Anything-3 model, the scene integrity after acceleration was consistent with the original model, and furniture outlines and texture information were effectively preserved; on the π³ model, the accelerated point cloud had less noise, and the clarity of scene edges and local features was even better than the original version. These subjective comparison results intuitively demonstrate that the hierarchical adaptive sparse attention mechanism of this application can accurately preserve the core geometric and semantic information required for 3D reconstruction while significantly reducing computational overhead, demonstrating excellent performance in balancing acceleration and reconstruction accuracy, and providing an efficient and lossless acceleration solution for various Transformer-based visual geometric models.

[0163] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the three-dimensional scene reconstruction acceleration method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0164] This application also provides a 3D scene reconstruction acceleration device; please refer to... Figure 6 The 3D scene reconstruction acceleration device includes:

[0165] Compression ratio determination module 601 is used to determine the target compression ratio corresponding to the current layer based on the layer identifier of the current layer and the mapping relationship between the layer and the compression ratio when the input video to be reconstructed is forward propagated to the global attention layer of the Transformer backbone network in the 3D reconstruction model.

[0166] Compression module 602 is used to perform linear mapping on the input feature sequence of the current layer to generate a query matrix, a key matrix and a value matrix, and to perform sparse sampling on the key matrix and the value matrix in the sequence dimension according to the target compression ratio to obtain the target key matrix and the target value matrix.

[0167] The global attention processing module 603 is used to perform attention calculation based on the query matrix, target key matrix and target value matrix corresponding to the current layer to obtain the context feature representation of the current layer until all attention calculations of the Transformer backbone network are completed.

[0168] The reconstruction module 604 is used to input the output feature sequence of the Transformer backbone network into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction results of the video to be reconstructed.

[0169] The 3D scene reconstruction acceleration device provided in this application, employing the 3D scene reconstruction acceleration method in the above embodiments, can solve the technical problem of low processing efficiency caused by the high computational complexity of attention in multi-view image reconstruction of 3D geometry tasks using the Transformer architecture in the prior art. Compared with the prior art, the beneficial effects of the 3D scene reconstruction acceleration device provided in this application are the same as those of the 3D scene reconstruction acceleration method provided in the above embodiments, and other technical features in the 3D scene reconstruction acceleration device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0170] This application provides a 3D scene reconstruction acceleration device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the 3D scene reconstruction acceleration method in the above embodiment 1.

[0171] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing a 3D scene reconstruction acceleration device according to embodiments of this application. The 3D scene reconstruction acceleration device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The 3D scene reconstruction acceleration device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0172] like Figure 7As shown, the 3D scene reconstruction acceleration device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the 3D scene reconstruction acceleration device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the 3D scene reconstruction acceleration device to communicate wirelessly or wiredly with other devices to exchange data. Although a 3D scene reconstruction acceleration device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0173] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0174] The 3D scene reconstruction acceleration device provided in this application, employing the 3D scene reconstruction acceleration method in the above embodiments, can solve the technical problem of low processing efficiency caused by the high computational complexity of attention in multi-view image reconstruction of 3D geometry tasks using the Transformer architecture in the prior art. Compared with the prior art, the beneficial effects of the 3D scene reconstruction acceleration device provided in this application are the same as those of the 3D scene reconstruction acceleration method provided in the above embodiments, and other technical features in this 3D scene reconstruction acceleration device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0175] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0176] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0177] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the three-dimensional scene reconstruction acceleration method in the above embodiments.

[0178] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0179] The aforementioned computer-readable storage medium may be included in the 3D scene reconstruction acceleration device; or it may exist independently and not be assembled into the 3D scene reconstruction acceleration device.

[0180] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the 3D scene reconstruction acceleration device, the 3D scene reconstruction acceleration device: when the input video to be reconstructed is forward-propagated to the global attention layer of the Transformer backbone network in the 3D reconstruction model, determines the target compression ratio corresponding to the current layer based on the layer identifier and the mapping relationship between the layer and the compression ratio; performs linear mapping on the input feature sequence of the current layer to generate a query matrix, a key matrix, and a value matrix, and performs sparse sampling on the key matrix and the value matrix in the sequence dimension according to the target compression ratio to obtain the target key matrix and the target value matrix; performs attention calculation based on the query matrix, the target key matrix, and the target value matrix corresponding to the current layer to obtain the context feature representation of the current layer, until all attention calculations of the Transformer backbone network are completed; and inputs the output feature sequence of the Transformer backbone network into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction result of the video to be reconstructed.

[0181] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0182] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0183] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0184] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described 3D scene reconstruction acceleration method. This solves the technical problem of low processing efficiency caused by the high computational complexity of attention in multi-view image reconstruction of 3D geometry tasks using the Transformer architecture in the prior art. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the 3D scene reconstruction acceleration method provided in the above embodiments, and will not be repeated here.

[0185] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the three-dimensional scene reconstruction acceleration method described above.

[0186] The computer program product provided in this application can solve the technical problem of low processing efficiency caused by the high computational complexity of attention in the 3D geometry reconstruction task of the Transformer architecture in the prior art. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the 3D scene reconstruction acceleration method provided in the above embodiments, and will not be repeated here.

[0187] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for accelerating 3D scene reconstruction, characterized in that, The method for accelerating 3D scene reconstruction includes: When the input video to be reconstructed is forward propagated to the global attention layer of the Transformer backbone network in the 3D reconstruction model, the target compression ratio corresponding to the current layer is determined according to the layer identifier of the current layer and the mapping relationship between the layer and the compression ratio. A linear mapping is performed on the input feature sequence of the current layer to generate a query matrix, a key matrix, and a value matrix. According to the target compression ratio, sparse sampling is performed on the key matrix and the value matrix in the sequence dimension to obtain the target key matrix and the target value matrix. Attention calculation is performed based on the query matrix, target key matrix, and target value matrix corresponding to the current layer to obtain the context feature representation of the current layer, until all attention calculations of the Transformer backbone network are completed; The output feature sequence of the Transformer backbone network is input into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction result of the video to be reconstructed; Prior to the step of determining the target compression ratio corresponding to the current layer based on the layer identifier and the mapping relationship between the layer and the compression ratio, the method further includes: Obtain a test dataset, which includes multiple test videos; The test video is input into the trained 3D reconstruction model for forward inference, and the attention matrix output by each global attention layer in the 3D reconstruction model is extracted. The effective rank value corresponding to the global attention layer is obtained by calculating the effective rank based on the attention matrix. The effective rank values ​​are mapped according to rules to generate the compression ratio corresponding to the global attention layer; Based on the compression ratio corresponding to the global attention layer and the hierarchical identifier of the global attention layer, the mapping relationship between the hierarchical level and the compression ratio is constructed; The step of performing sparse sampling on the key matrix and the value matrix according to the target compression ratio to obtain the target key matrix and the target value matrix includes: A non-uniform sparse sampling algorithm that preserves time structure is used to perform frame-level sparse sampling on the key matrix in the sequence dimension according to the target compression ratio to obtain the compressed target key matrix; A non-uniform sparse sampling algorithm that preserves time structure is used to perform frame-level sparse sampling on the value matrix in the sequence dimension according to the target compression ratio to obtain the compressed target value matrix.

2. The method for accelerating 3D scene reconstruction as described in claim 1, characterized in that, Before the step of forward propagating the input video to be reconstructed to the global attention layer of the Transformer backbone network in the 3D reconstruction model, the method further includes: Each video frame in the video to be reconstructed is input into the three-dimensional reconstruction model. The image encoder of the three-dimensional reconstruction model extracts features from each video frame to obtain the initial feature vector sequence corresponding to the video to be reconstructed. The initial feature vector sequence is input into the Transformer backbone network of the 3D reconstruction model for forward propagation.

3. The method for accelerating 3D scene reconstruction as described in claim 1, characterized in that, The non-uniform sparse sampling algorithm employing time structure preservation, which performs frame-level sparse sampling on the key matrix along the sequence dimension according to the target compression ratio to obtain the compressed target key matrix, includes the following steps: The number of row vectors to be retained in the key matrix is ​​determined based on the target compression ratio; The key matrix is ​​divided into first frame groups according to the number of video frames, the number of row vectors, and the first frame group includes multiple first row vectors, the first row vectors being the row vectors in the key matrix; Within each first frame group, the information entropy of each first row vector is calculated, and the first row vector corresponding to the maximum information entropy value is determined as the first target row vector corresponding to the first frame group. The target key matrix is ​​reconstructed based on the first target row vector corresponding to each of the first frame groups.

4. The method for accelerating 3D scene reconstruction as described in claim 1, characterized in that, The Transformer backbone network of the 3D reconstruction model consists of multiple alternating global attention layers and intra-frame attention layers.

5. The method for accelerating 3D scene reconstruction as described in claim 4, characterized in that, Also includes: When propagating to each intra-frame attention layer of the Transformer backbone network, the input feature sequence of the intra-frame attention layer is grouped according to video frames to obtain the input features corresponding to each video frame. Perform self-attention processing on each of the input features to obtain the self-attention processing results for each of the video frames; The self-attention processing results corresponding to each video frame are concatenated, and the corresponding concatenation results are used as the input feature sequence of the next global attention layer.

6. The method for accelerating 3D scene reconstruction as described in claim 1, characterized in that, The step of inputting the output feature sequence of the Transformer backbone network into the prediction head of the 3D reconstruction model to obtain the multi-view reconstruction result of the video to be reconstructed includes: The output feature sequence of the Transformer backbone network is input into the first prediction sub-network in the prediction head. The camera pose information and depth information of each video frame are predicted through the first prediction sub-network to obtain the depth map and camera pose parameters corresponding to each video frame. The output feature sequence of the Transformer backbone network, the depth map of each video frame, and the camera pose parameters are input into the second prediction sub-network in the prediction head. The second prediction sub-network predicts the point cloud information of each video frame to obtain the point cloud feature map corresponding to each video frame. Based on the camera pose parameters and point cloud feature maps of each video frame, a 3D scene point cloud model corresponding to the video to be reconstructed is generated, and the 3D scene point cloud model, the camera pose parameters, and the depth map are used as the multi-view reconstruction result.

7. A three-dimensional scene reconstruction acceleration device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the three-dimensional scene reconstruction acceleration method as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the three-dimensional scene reconstruction acceleration method as described in any one of claims 1 to 6.

Images