A query-driven retrieval based streaming 3D reconstruction method, device and medium

Abstract

The present application relates to the field of computer vision, in particular to a stream three-dimensional reconstruction method and device based on query-driven retrieval and medium, video input is improved StreamVGGT model; when each frame arrives, the mark sequence is extracted through the image encoder, and is input into the Transformer aggregation layer; the Transformer aggregation layer is based on the attention similarity between the current frame query in the first layer global attention mechanism and the key in the historical frame key-value cache, and dynamically retrieves the most relevant historical frame set; in each subsequent layer of the Transformer aggregation layer, the key-value pair of the selected historical frame is used for causal attention calculation; based on the features output by the Transformer aggregation layer, the three-dimensional point cloud, the depth map and the camera pose are generated through the multi-task prediction head. Compared with the prior art, the present application can realize faithful long sequence reconstruction under fixed memory budget.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a streaming 3D reconstruction method and system based on deep learning. Background Technology

[0002] The need for frame-by-frame reconstruction of 3D scenes from continuous video streams (i.e., long-context streaming 3D reconstruction) is rapidly growing in applications such as autonomous driving, augmented reality, and embodied intelligence. Traditional 3D reconstruction pipelines rely on structure of motion recovery (SfM) and multi-view stereo vision (MVS), which are computationally expensive and sensitive to noise. In recent years, feedforward-based methods such as VGGT have achieved impressive generalization capabilities through end-to-end dense geometry prediction, but the quadratic complexity of their global attention mechanism limits their ability to process long sequences.

[0003] To address the problem of processing long sequences, researchers have proposed several streaming architecture schemes: (1) Explicit spatial memory methods (such as Spann3R and Point3R) anchor past observations in three-dimensional space, but storage requirements grow unbounded.

[0004] (2) Cyclic state compression methods (such as CUT3R and TTT3R) use fixed-length states, but there is a catastrophic forgetting problem.

[0005] (3) Causal attention methods (such as StreamVGGT) are based on the VGGT architecture and achieve streaming 3D reconstruction through a causal attention mechanism. They use a KV cache to store the key-value states of historical frames for frame-by-frame inference. After the image encoder processes the current frame, this method uses a causal attention block to perform cross-frame feature aggregation using the KV cache of historical frames. Finally, it outputs 3D point clouds, depth maps, and camera poses through a task-specific prediction head. Streaming inference is achieved through KV cache under causal attention, achieving competitive quality on short sequences. However, its KV cache grows linearly with the number of frames, leading to GPU memory overflow on long sequences; and there is a lot of redundant information in the KV cache (the average cosine similarity of the key descriptors of historical frames is about 0.98), which leads to a decrease in reconstruction quality. Summary of the Invention

[0006] The purpose of this invention is to solve the GPU memory overflow problem caused by the unbounded growth of KV cache with the sequence length in current streaming 3D reconstruction, and to provide a streaming 3D reconstruction method, device and medium based on query-driven retrieval.

[0007] The objective of this invention can be achieved through the following technical solutions: As a first aspect of the present invention, a streaming 3D reconstruction method based on query-driven retrieval is provided, comprising the following steps: Input the video into the improved StreamVGGT model; At the arrival of each frame, a marker sequence is extracted by the image encoder and input into the Transformer aggregation layer; The Transformer aggregation layer dynamically retrieves the most relevant set of historical frames based on the attention similarity between the current frame query and the historical frame keys in the KV cache in the first-layer global attention mechanism. In each subsequent layer of the Transformer aggregation layer, causal attention is calculated using the key-value pairs of the selected historical frames. Based on the features output by the Transformer aggregation layer, 3D point clouds, depth maps, and camera poses are generated by a multi-task prediction head.

[0008] As a preferred technical solution, the dynamic retrieval of the most relevant set of historical frames is specifically as follows: Using linear projection in the first-layer global attention mechanism, query vector, key vector, and value vector are computed for each tag in the current frame; The query vector and key vector of the image patch location are subjected to mean pooling to obtain frame-level query descriptors and key descriptors. The key value and key descriptor of the current frame are cached. The relevance score between the current frame and each historical frame is obtained by performing an inner product operation between the query descriptor of the current frame and the key descriptors of all cached historical frames. Given a frame budget, the first frame is retained as a coordinate reference, and the remaining frames are selected based on their relevance scores using a segmented sampling strategy to generate the final frame selection set.

[0009] As a preferred technical solution, the segmented sampling strategy is as follows: An adaptive threshold is calculated based on the correlation scores of all historical frames. The subsequence of frames whose correlation scores continuously exceed the adaptive threshold is defined as a high-correlation segment. The peak value of the relevance score within each segment is taken as the importance measure of that segment. The optional frame budget is initially allocated according to the proportion of the peak value of each segment. The allocation results are truncated and adjusted to ensure that each segment is allocated at least one frame and does not exceed the actual number of frames in that segment. Within each segment, the peak frame with the highest correlation is selected first, and then the remaining quota frames are sampled evenly over the time span of that segment. Summarize the sampling results of all segments. When the total exceeds the budget, retain the budgeted number of frames in descending order of relevance score. When the total is insufficient, supplement the budgeted number of frames from the historical frames that were never selected in descending order of relevance score.

[0010] As a preferred technical solution, the adaptive threshold is the mean of the correlation scores plus a set percentage of the standard deviation.

[0011] As a preferred technical solution, adjacent highly correlated segments with an interval less than a set frame number threshold are merged.

[0012] As a preferred technical solution, the method organizes a KV cache based on the predicted camera pose and periodically compresses over-represented regions, as follows: The camera pose and key descriptor predicted based on the current frame are recorded in the spatial memory module, the 3D spatial position and optical axis orientation of the camera are extracted and stored, and a historical bounding box is constructed based on all observed camera positions. After processing a fixed number of frames, a spatial distribution evaluation is performed to calculate the average number of frames in all occupied areas of the historical bounding box. Regions with more frames than the average frame count are identified as over-represented regions, and a certain proportion of these frames are retained through uniform step sampling; under-represented regions with no more than the average frame count remain unchanged.

[0013] As a preferred technical solution, the construction of the historical bounding box is specifically as follows: The scene space is divided into a uniform three-dimensional grid; the three-dimensional spatial position of the camera in each frame is quantized through the grid to obtain the position index of the spatial grid; the optical axis orientation is quantized into several azimuth intervals to obtain the azimuth index; the position index and the azimuth index together form a composite spatial key, and the pose perception space of each frame is assigned to the corresponding spatial region; the historical bounding box only grows and does not shrink with the arrival of new frames.

[0014] As a preferred technical solution, the deleted frames in the over-representation region are handled by tombstone marking: the key-value tensors occupying GPU memory are released, but the metadata information of the brain pose and descriptor is retained.

[0015] As a second aspect of the present invention, a query-driven retrieval-based streaming 3D reconstruction apparatus is provided, comprising a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the query-driven retrieval-based streaming 3D reconstruction method as described above.

[0016] As a third aspect of the present invention, a storage medium is provided having a program stored thereon, which, when executed, implements the query-driven retrieval-based streaming 3D reconstruction method as described above.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1) This invention designs a query-driven retrieval framework for the field of streaming 3D reconstruction, transforming the context construction problem of VGGT into a retrieval problem, enabling each frame to dynamically select its most relevant keyframes from the complete history for attention calculation, thus achieving faithful long sequence reconstruction under a fixed memory budget.

[0018] 2) This invention discovers that the similarity of query-key in the first layer of global attention of VGGT is already an effective indicator of frame relevance. High-quality frame selection can be achieved without additional learning modules. The entire framework does not require training and can be directly applied to existing pre-trained models.

[0019] 3) The segmented sampling strategy proposed in this invention is analogous to the retrieval diversity in recommendation systems. It identifies multiple highly relevant segments and allocates frame budgets according to the peak importance ratio. This ensures retrieval diversity and effectively avoids the single-peak concentration problem caused by naive Top-K selection.

[0020] 4) The pose-aware spatial memory mechanism designed in this invention enables long-term memory management of pose perception, constraining the memory usage of the KV cache while maintaining the diversity of spatial coverage. By utilizing the estimated camera pose, the KV states are organized according to spatial regions, periodically compressing over-represented regions while preserving the unique perspective of sparsely covered regions, thus achieving bounded memory usage and diverse spatial coverage on arbitrarily long sequences. Attached Figure Description

[0021] Figure 1 This is an overall flowchart of the query-driven retrieval-based streaming 3D reconstruction method of the present invention.

[0022] Figure 2 This is a schematic diagram of the query driver frame selection and segmented sampling strategy of the present invention; a) segment identification, b) proportional quota allocation, c) intra-segment sampling, and d) budget adjustment.

[0023] Figure 3 This is a two-step pipeline flowchart for the pose-aware spatial memory of the present invention; a) spatial region allocation, b) periodic compression. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0025] Example 1 This invention proposes a query-driven retrieval-based streaming 3D reconstruction method, which transforms the context construction problem of VGGT into a retrieval problem. It uses the attention similarity between the current frame query and the historical frame key as a relevance indicator for dynamic frame selection, and ensures the diversity of retrieval through a segmented sampling strategy. At the same time, it designs a pose-aware spatial memory mechanism to achieve effective management of long-term KV cache, thereby achieving high-quality long-sequence streaming 3D reconstruction under a fixed memory budget.

[0026] The overall framework of the method comprises the following main components: a query-driven frame selection mechanism, which utilizes query-key similarity in the first layer of global attention in the Transformer aggregation layer of VGGT for dynamic frame retrieval; a segmented sampling strategy, which identifies multiple highly relevant segments and allocates frame budgets proportionally to ensure diversity; and a pose-aware spatial memory, which organizes KV caches based on the estimated camera pose and periodically compresses overrepresented regions. The overall method requires no additional training and can be run directly on the pre-trained checkpoints of the existing StreamVGGT.

[0027] like Figure 1 As shown, the main steps to achieve streaming 3D reconstruction are: Step 1: Query-driven frame selection mechanism. The core innovation of this method lies in replacing the complete KV cache with a retrieval-based dynamic frame selection, enabling each frame to select the most relevant keyframes from history according to its own perspective requirements. The specific steps are as follows: Step 1.1: After the current frame undergoes intra-frame attention processing in the image encoder and the first layer of the Transformer aggregation layer, a preliminary labeled feature representation is obtained. Using linear projection in the first layer of the global attention mechanism, a query vector, key vector, and value vector are calculated for each label in the current frame.

[0028] Step 1.2: Since the tag sequence contains two types of tags: special tags (used for camera parameter prediction and information aggregation) and image patch tags, and only image patch tags encode frame-specific visual content, mean pooling is performed only on the query vector and key vector at the image patch location to obtain frame-level query descriptors and key descriptors. The query descriptor of the current frame is used to retrieve historical frames, while the key descriptor of the current frame is cached for use in subsequent frame retrievals.

[0029] Step 1.3: Perform an inner product operation between the query descriptor of the current frame and the key descriptors of all historical frames to obtain the relevance score between the current frame and each historical frame, and take the average value over all attention heads as the basis for measuring the degree of visual relevance between frames.

[0030] Step 1.4: Given a fixed number of frames, always retain the first frame as the coordinate reference. The remaining frames are selected based on their relevance scores using a segmented sampling strategy (Step 2) to generate the final frame selection set. Frame selection is performed globally at the frame level to maintain the integrity of the intra-frame label structure, and the selection is performed only once at the first layer. The selection result is shared and reused across all Transformer layers, avoiding the additional computational overhead of repeated selection at each layer.

[0031] Step 1.5: In each subsequent Transformer layer, causal attention is calculated using only the key-value pairs of the selected frame, thus completing cross-frame information aggregation within a fixed memory budget. After processing by all layers, the aggregated features are used by the prediction heads of each task to generate prediction results such as 3D point clouds, depth maps, and camera poses. The estimated camera pose and key descriptors of the current frame are then recorded in the spatial memory module (Step 3) for use in subsequent frames.

[0032] Step 2: Segmented sampling strategy.

[0033] Simply sorting by score and selecting the top few frames (i.e., Top-K selection) easily concentrates all selected frames near a single high similarity peak, leading to viewpoint redundancy and ignoring other valuable segments in the history. Inspired by the idea of ​​retrieval diversity in recommender systems, this method designs a segmented sampling strategy. By identifying multiple highly relevant segments and allocating frame budgets proportionally, it ensures the diversity of retrieval results. Figure 2 As shown, the specific steps are as follows: Step 2.1: Segmentation Identification. First, an adaptive threshold is calculated based on the relevance scores of all historical frames. This threshold is equal to the mean of the relevance scores plus a certain percentage of the standard deviation. The subsequence of frames whose relevance scores continuously exceed this threshold is defined as a high-relevance segment. At the same time, adjacent segments that are too close together are merged to avoid excessive fragmentation, resulting in several independent high-relevance segments.

[0034] Step 2.2: Proportional Quota Allocation. The peak value of the correlation score within each segment is taken as the importance metric for that segment. The optional frame budget is initially allocated proportionally to the peak values ​​of each segment. The allocation results are then truncated and adjusted to ensure that each segment receives at least one frame (ensuring that every detected relevant region contributes) and that the allocation does not exceed the actual number of frames for that segment (preventing oversampling).

[0035] Step 2.3: Intra-segment sampling. Within each segment, the peak frame with the highest correlation is selected first, and then the remaining quota frames are sampled evenly over the time span of that segment, taking into account both the selection of the optimal frame and the diversity of time coverage within the segment.

[0036] Step 2.4: Budget Adjustment. After summarizing the sampling results of all segments, the actual total may deviate from the preset frame budget. When the total exceeds the budget, frames are retained in the budgeted quantity according to their relevance scores from high to low; when the total is insufficient, historical frames that were never selected are supplemented to the budgeted quantity according to their relevance scores from high to low. Both cases ensure that exactly the preset number of frames are selected in the end, while maintaining diverse coverage of multiple segments.

[0037] This invention identifies multiple highly relevant segments by using adaptive thresholds and allocates frame budgets according to the proportion of peak importance. Combined with uniform sampling within segments and global budget adjustment, it ensures retrieval diversity.

[0038] Step 3: Pose-Aware Spatial Memory. In practical applications, camera trajectories may densely revisit certain scene regions while sparsely covering other regions, causing the key-value cache to accumulate a large number of redundant frames in frequently observed areas. This method maintains a pose-aware spatial memory module, using the estimated camera pose to spatially organize the cache state and periodically compressing over-represented regions, achieving bounded memory management. Figure 3 As shown, the specific steps are as follows: Step 3.1: Spatial Region Assignment. After each frame passes through the camera pose prediction head, its 3D spatial position and optical axis orientation are extracted and stored. A historical bounding box is constructed based on all observed camera positions, dividing the scene space into a uniform 3D grid. The position of each frame is quantized to obtain the index of its spatial cell, and the viewing orientation is further quantized into several azimuth intervals. The position index and azimuth index together form a composite spatial key, assigning each frame to the corresponding spatial region. In practical applications, a 3x3x3 spatial grid and four azimuth intervals are used, generating a maximum of 108 different regions. The historical bounding box only grows and does not shrink with the arrival of new frames, ensuring that the region division remains stable even after a frame is deleted.

[0039] Step 3.2: Periodic Compression. Spatial distribution evaluation is triggered after processing a fixed number of frames. First, the average number of frames in all occupied regions is calculated, and regions with more frames than the average are identified as overrepresented regions. For each overrepresented region, a certain percentage of frames are retained using uniform stride sampling, with a deletion ratio set at 50%. The retained frames are evenly distributed across the time span of the region to maintain the diversity of spatial coverage. Deleted frames are handled using a tombstone marking method: the key-value tensors occupying GPU memory are released, but lightweight metadata information (such as pose and descriptors) is retained to keep the spatial index structure stable and not affect the retrieval of subsequent frames. In addition, a safety protection mechanism is set up to skip the compression operation when the number of active key-value caches is below a minimum ratio threshold to prevent over-pruning from affecting the reconstruction quality. Underrepresented regions with no more than the average number of frames remain unchanged.

[0040] This invention achieves bounded management of long sequence KV buffers based on a composite spatial region division of three-dimensional position and viewpoint direction, and a periodic compression mechanism based on uniform stride sampling.

[0041] Step 4: Output 3D prediction results. After the above query-driven frame selection, segmented sampling, and pose-aware spatial memory management, the system outputs complete 3D prediction results for each frame, including 3D point cloud reconstruction, depth map estimation, and camera pose estimation, which can directly support downstream application tasks such as scene-level 3D reconstruction, video depth estimation, and camera pose estimation.

[0042] Example 2 As a second aspect of the present invention, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the above-described query-driven retrieval-based streaming 3D reconstruction method. In addition to the processors, memory, and interfaces described above, any data processing device in the embodiments may also include other hardware depending on the actual function of the data processing device, which will not be elaborated further.

[0043] Example 3 As a third aspect of the present invention, this application also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the query-driven retrieval-based streaming 3D reconstruction method described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0044] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A streaming 3D reconstruction method based on query-driven retrieval, characterized in that the steps are as follows: include: Input the video into the improved StreamVGGT model; At the arrival of each frame, a marker sequence is extracted by the image encoder and input into the Transformer aggregation layer; The Transformer aggregation layer dynamically retrieves the most relevant set of historical frames based on the attention similarity between the current frame query and the historical frame keys in the KV cache in the first-layer global attention mechanism. In each subsequent layer of the Transformer aggregation layer, causal attention is calculated using the key-value pairs of the selected historical frames. Based on the features output by the Transformer aggregation layer, 3D point clouds, depth maps, and camera poses are generated by a multi-task prediction head.

2. The streaming 3D reconstruction method based on query-driven retrieval according to claim 1, characterized in that, The dynamic retrieval of the most relevant set of historical frames is as follows: Using linear projection in the first-layer global attention mechanism, query vector, key vector, and value vector are computed for each tag in the current frame; The query vector and key vector of the image patch location are subjected to mean pooling to obtain frame-level query descriptors and key descriptors. The key value and key descriptor of the current frame are cached. The relevance score between the current frame and each historical frame is obtained by performing an inner product operation between the query descriptor of the current frame and the key descriptors of all cached historical frames. Given a frame budget, the first frame is retained as a coordinate reference, and the remaining frames are selected based on their relevance scores using a segmented sampling strategy to generate the final frame selection set.

3. The streaming 3D reconstruction method based on query-driven retrieval according to claim 2, characterized in that, The segmented sampling strategy is as follows: An adaptive threshold is calculated based on the correlation scores of all historical frames. The subsequence of frames whose correlation scores continuously exceed the adaptive threshold is defined as a high-correlation segment. The peak value of the relevance score within each segment is taken as the importance measure of that segment, and the optional frame budget is initially allocated according to the proportion of the peak value of each segment. The allocation results are truncated and adjusted to ensure that each segment is allocated at least one frame and does not exceed the actual number of frames in that segment; Within each segment, the peak frame with the highest correlation is selected first, and then the remaining quota frames are sampled evenly over the time span of that segment. Summarize the sampling results of all segments. When the total exceeds the budget, retain the budgeted number of frames in descending order of relevance score. When the total is insufficient, supplement the budgeted number of frames from the historical frames that were never selected in descending order of relevance score.

4. The streaming 3D reconstruction method based on query-driven retrieval according to claim 3, characterized in that, The adaptive threshold is the mean of the correlation scores plus a set percentage of the standard deviation.

5. The streaming 3D reconstruction method based on query-driven retrieval according to claim 3, characterized in that, Adjacent highly correlated segments with an interval less than a set frame number threshold are merged.

6. The streaming 3D reconstruction method based on query-driven retrieval according to claim 1, characterized in that, The method organizes a key-value cache based on the predicted camera pose and periodically compresses over-represented regions, as detailed below: The camera pose and key descriptor predicted based on the current frame are recorded in the spatial memory module, the 3D spatial position and optical axis orientation of the camera are extracted and stored, and a historical bounding box is constructed based on all observed camera positions. After processing a fixed number of frames, a spatial distribution evaluation is performed to calculate the average number of frames in all occupied areas of the historical bounding box. Regions with more frames than the average frame count are identified as over-represented regions, and a certain proportion of these frames are retained through uniform step sampling; under-represented regions with no more than the average frame count remain unchanged.

7. The streaming 3D reconstruction method based on query-driven retrieval according to claim 6, characterized in that, The construction of the history bounding box is as follows: The scene space is divided into a uniform three-dimensional grid; the three-dimensional spatial position of the camera in each frame is quantized through the grid to obtain the position index of the spatial grid; the optical axis orientation is quantized into several azimuth intervals to obtain the azimuth index; the position index and the azimuth index together form a composite spatial key, and the pose perception space of each frame is assigned to the corresponding spatial region; the historical bounding box only grows and does not shrink with the arrival of new frames.

8. The streaming 3D reconstruction method based on query-driven retrieval according to claim 6, characterized in that, Frames deleted in the over-representation region are handled using a tombstone marking method: the key-value tensors occupying GPU memory are released, but the metadata information of the skull pose and descriptor is retained.

9. A query-driven retrieval-based streaming 3D reconstruction device, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the query-driven retrieval-based streaming 3D reconstruction method as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the query-driven retrieval-based streaming 3D reconstruction method as described in any one of claims 1-8.

Images