An intra-frame attention optimization method and system for visual transformer based on interval merging

Abstract

This invention relates to a method and system for optimizing intra-frame attention in visual Transformers based on interval merging, belonging to the field of computer vision 3D reconstruction technology. Addressing the problem of decreased reconstruction accuracy caused by existing token merging techniques neglecting 3D geometric constraints, this invention employs an interval-triggered mechanism to perform token merging during the intra-frame attention stage of multi-view Transformers such as VGGT, maintaining sequence integrity during the global attention stage. It dynamically allocates the merging ratio for each frame through a two-parameter scoring model that integrates token feature discreteness and image gradient complexity. Furthermore, it maintains spatial topological continuity by combining sliding window local constraints and a Camera / Register Token protection mechanism. This invention requires no modification to the model backbone network, embedding itself in a plug-and-play manner. While significantly reducing inference latency and memory usage, it effectively avoids depth jumps and structural distortions, ensuring the accuracy of camera pose estimation, depth prediction, and point cloud reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision 3D reconstruction technology, specifically relating to a visual Transformer intra-frame attention optimization method and system based on interval merging. Background Technology

[0002] In recent years, the Visual Transformer (ViT) has been used in multi-view applications. Figure 3 Breakthroughs have been made in the field of 3D reconstruction. Multi-view Transformer models, represented by VGGT, employ an alternating structure of intra-frame attention and global attention to achieve high-precision camera pose estimation, depth prediction, and dense point cloud reconstruction. However, existing lightweight techniques have fundamental shortcomings when adapting to this type of model, failing to balance inference efficiency and 3D geometric accuracy.

[0003] Current mainstream token merging techniques generally employ global indiscriminate merging or forced merging strategies per frame. Their initial design is solely for 2D vision tasks such as image classification and semantic segmentation, completely ignoring the rigid constraints of 3D reconstruction on spatial topological continuity, geometric validity of positional encoding, and consistency of cross-view matching. Applying such methods directly to multi-view Transformers leads to three fatal problems: First, global merging easily merges tokens that are spatially distant and have similar textures, resulting in distorted positional encoding weights and loss of physical geometric meaning. Second, cross-region merging disrupts the spatial topological relationships of a single frame, causing blurred object edges and loss of depth transition information. Third, global token sequence compression prevents global attention from capturing long-distance geometric dependencies, ultimately leading to incorrect depth estimation, scene structural distortion, large-area holes in the reconstruction result, or even complete collapse.

[0004] While the sliding window mechanism has been applied in 2D models such as the Swing Transformer, existing solutions are not combined with strategies such as intra-frame / global attention separation, frame interval triggering, and special token protection. This makes them unsuitable for VGGT alternating attention structures and fails to address the unique geometric constraints of 3D reconstruction. Furthermore, the single-interval merging strategy lacks local spatial constraints, still resulting in cross-regional erroneous merging and failing to fundamentally solve the accuracy degradation problem.

[0005] Furthermore, existing token merging strategies often employ fixed merging thresholds, fixed merging ratios, or fixed frame interval parameters, applying the same compression intensity to different frames throughout the multi-view sequence. However, in practical 3D reconstruction tasks, the structural and texture complexity of different input frames varies significantly: frames with simple textures and high feature redundancy can withstand stronger token merging, while frames containing rich textures, edge structures, or local geometric changes are prone to depth boundary blurring, camera estimation shift, and increased 3D point cloud errors if the same compression intensity is applied. Therefore, how to adaptively adjust the token merging ratio based on the feature distribution of a single frame and the image texture complexity is a key challenge in existing multi-view reconstruction strategies. Figure 3 The problem of lightweight methods for reconstructing Transformers has not yet been effectively solved.

[0006] In summary, existing technologies struggle to dynamically adjust the token merging strength based on the content complexity of the multi-view input sequence without compromising 3D geometric information or ensuring reconstruction accuracy. Therefore, there is an urgent need for an adaptive lightweight optimization scheme that fits the alternating attention structure of the multi-view Transformer. Summary of the Invention

[0007] The purpose of this invention is to address the problem of decreased reconstruction accuracy caused by neglecting 3D geometric constraints in existing token merging techniques. It provides a method and system for optimizing intra-frame attention in visual Transformers based on interval merging. This invention is not a simple superposition of token merging, sliding window, and interval sampling techniques, but rather an innovative five-constraint integrated solution that addresses the core structural features of VGGT-type multi-view Transformers with alternating intra-frame and global attention, as well as the rigid constraints of spatial topology, position encoding, and cross-view matching in 3D reconstruction tasks. This solution resolves the technical contradiction between existing lightweight techniques and the inability to balance inference efficiency and 3D geometric accuracy.

[0008] The core innovation of this invention lies in constructing an intra-frame token merging mechanism based on a dual-parameter adaptive complexity score. This mechanism calculates the token feature complexity index and image gradient complexity index of the input frame separately, and then normalizes and fuses them using two adjustable weight parameters to obtain an adaptive complexity score for each frame. Subsequently, based on this complexity score, the token merging ratio or retention budget for that frame is dynamically determined, resulting in stronger merging for frames with simple textures and high redundancy, and more conservative merging for frames with rich textures and complex structures. This scheme further combines intra-frame attention-specific merging, complete global attention preservation, and a special token enforcement protection mechanism to maintain the geometric consistency required for 3D reconstruction while reducing computational complexity.

[0009] 1. Dual-index complexity estimation: The token complexity index and texture complexity index are calculated based on the feature variance of the intra-frame Patch Token and the gradient intensity of the input image, respectively;

[0010] 2. Two-parameter adaptive fusion: The two types of complexity indicators mentioned above are normalized and weighted by using two adjustable weight parameters λv and λg to obtain the frame-level merging complexity score.

[0011] 3. Adaptive merging ratio allocation: Dynamically determine the token merging ratio, maximum merging ratio, or minimum number of tokens to be retained for different frames based on the frame-level complexity score;

[0012] 4. Intra-frame dedicated merging and global integrity preservation: Token merging is performed only during the intra-frame attention phase, while the complete token sequence is preserved during the global attention phase;

[0013] 5. Special Token Protection Mechanism: Special tokens such as Camera Token and Register Token are not included in the merging process to maintain the stability of camera prediction and global features;

[0014] 6. Optional local spatial constraints: In some embodiments, token merging can be restricted to a single frame local window to further reduce the risk of cross-regional erroneous merging.

[0015] This invention requires no modification to the model backbone network or additional training. It can be embedded into mainstream multi-view Transformer models such as VGGT in a plug-and-play manner. While significantly reducing computational complexity and memory usage, it fully preserves the spatial structure, geometric consistency and cross-view matching foundation required for 3D reconstruction, achieving the dual optimization of inference efficiency and reconstruction accuracy.

[0016] To achieve the above objectives, the technical solution of the present invention is: a visual Transformer intra-frame attention optimization method based on interval merging, comprising:

[0017] Initialize the aggregator module and configure the token merging control parameters, including:

[0018] A. Whether to enable the intra-frame token merging switch, which controls the start and stop of the optimization strategy;

[0019] B. Token merging similarity threshold, used to determine the degree of similarity between tokens;

[0020] C. Frame merging interval and frame merging start offset are used to control the trigger position of token merging;

[0021] D. The token feature complexity weight parameter λv is used to adjust the proportion of the token feature variance index in the adaptive scoring.

[0022] E. Image gradient complexity weight parameter λg, used to adjust the proportion of image gradient index in adaptive scoring;

[0023] F. An adaptive merging ratio set, including low-complexity merging ratio, medium-complexity merging ratio, and high-complexity merging ratio;

[0024] G. Maximum merging ratio and minimum number of retained tokens are used to prevent excessive merging from causing a decrease in feature expressiveness;

[0025] The input multi-frame images are preprocessed by converting the images into PatchTokens through Patch Embedding, and Camera Token and Register Token are added to the beginning of the sequence. At the same time, position encoding is added to all tokens to enable the model to have spatial awareness.

[0026] Maintain two independent token sequences, including an intra-frame token sequence for intra-frame attention and a global token sequence for global attention;

[0027] During the alternating attention process, global self-attention calculation is performed on the global token sequence to maintain sequence integrity and prevent merging; intra-frame self-attention calculation is performed on the intra-frame token sequence, and the current frame is judged to meet the merging trigger condition based on the frame merging interval and frame merging start offset; when the merging trigger condition is met, the adaptive complexity score Si of the current frame is calculated, and the token merging ratio Ri of the current frame is determined based on Si; based on the token merging similarity threshold and the token merging ratio Ri, the token merging operation is performed on the intra-frame token sequence, and the position encoding is updated synchronously.

[0028] The intra-frame attention output features and the global attention output features are concatenated and fused to obtain the final output features.

[0029] The method further includes: performing a 3D visual parameter inference task based on the merged and optimized features, inputting the fused features into the corresponding prediction head, and completing tasks such as camera pose estimation, depth map prediction, and dense point cloud reconstruction.

[0030] Furthermore, the calculation of the adaptive complexity score Si for the current frame includes:

[0031] Calculate the token feature complexity index Vi of the current frame Patch Token;

[0032] Calculate the gradient complexity index Gi of the input image in the current frame;

[0033] Normalize Vi and Gi respectively to obtain normalized V'i and G'i;

[0034] The adaptive complexity score Si is calculated using the formula Si=(λv·V′i+λg·G′i) / (λv+λg+ε), where ε is a stabilizing term to prevent the denominator from being zero. By adjusting λv and λg, the model can be made to pay more attention to changes in feature distribution or image texture in different application scenarios.

[0035] Furthermore, for the i-th frame of the input image, the token feature complexity index Vi is first calculated based on the frame's Patch Token features. Specifically, variance, standard deviation, or dispersion statistics are calculated for all Patch Tokens in the frame along the feature dimension or token dimension to characterize the complexity of the feature distribution within the frame. The larger Vi is, the more dispersed the token representation in the frame is, and the richer the structural information it contains. Simultaneously, the image gradient complexity index Gi is calculated based on the original or preprocessed image of the i-th frame. Specifically, the mean gradient magnitude in the horizontal and vertical directions, Sobel gradient, differential gradient, or other edge strength statistics can be used to characterize the complexity of the texture and edge structure of the frame. The larger Gi is, the richer the texture or boundary information contained in the frame.

[0036] Furthermore, the token feature complexity index Vi is calculated in the following way:

[0037] Extract the mean variance of all Patch Tokens in the current frame, the mean feature dispersion or standard deviation of the tokens as the token feature complexity index Vi.

[0038] Furthermore, the gradient complexity metric Gi is calculated in the following way:

[0039] Calculate the horizontal gradient of the input image in the current frame. gradient in the x and vertical directions y, calculate the gradient in the horizontal direction gradient in the x and vertical directions The mean value of the gradient magnitude of y is used as the gradient complexity index Gi.

[0040] Furthermore, determining the token merging ratio Ri of the current frame based on Si includes:

[0041] Preset low-complexity merging ratio R low Medium complexity merging ratio R mid Merging ratios of high-complexity components R high , where Rhigh >R mid >R low ;

[0042] preset a first complexity threshold T1 and a second complexity threshold T2;

[0043] if Si<T1, then set Ri=R high ;

[0044] if T1≤Si<T2, then set Ri=R mid ;

[0045] if Si≥T2, then set Ri=R low .

[0046] Further, performing a token merging operation on the intra-frame token sequence includes:

[0047] performing L2 normalization on the token features;

[0048] calculating a cosine similarity matrix between tokens;

[0049] screening similar token groups according to a token merging similarity threshold;

[0050] performing weighted averaging on the tokens with the similarity as a weight to generate a new aggregated token;

[0051] mandatorily protecting the Camera Token and the Register Token from participating in merging;

[0052] setting a minimum reserved token number and a maximum merging ratio to prevent feature degradation caused by excessive merging.

[0053] Further, the token merging operation further includes a sliding window local constraint step:

[0054] restoring the intra-frame token sequence into a two-dimensional feature grid corresponding to the spatial structure of the input image;

[0055] setting a sliding window of a fixed size, and performing the merging operation only on spatially adjacent tokens falling within the same sliding window;

[0056] controlling the sliding window to move on the feature grid according to a preset step length until the entire frame of image is traversed.

[0057] Further, the token merging control parameters further include the maximum merging ratio and the minimum reserved token number;

[0058] When performing a token merging operation, the actual merging ratio of the current frame is limited to the maximum merging ratio, and the total number of tokens after merging is ensured to be no less than the minimum number of tokens to be retained.

[0059] Furthermore, the synchronous update of the location code includes:

[0060] Record the merged source mapping relationship for each new aggregated token;

[0061] Based on the merge mapping relationship, the position codes of the merged tokens are weighted and averaged to generate the position codes of the new tokens;

[0062] Keep the location encoding of the Camera Token and Register Token unchanged.

[0063] The present invention also provides a visual Transformer intra-frame attention optimization system based on interval merging, which performs the method described above, including:

[0064] The parameter configuration module is used to initialize and store token merging control parameters;

[0065] The Token generation module is used to perform Patch Embedding on the input image, generate Patch Tokens, and add Camera Tokens, Register Tokens, and location codes.

[0066] The dual-Token maintenance module independently maintains the intra-Token sequence for intra-frame attention and the global Token sequence for global attention.

[0067] The intra-frame attention processing module is used to perform intra-frame self-attention calculation and trigger token merging according to the interval strategy;

[0068] The complexity scoring module is used to calculate the token feature dispersion index and image gradient complexity index of the current frame, and to calculate the adaptive complexity score Si through a fusion formula.

[0069] The Token merging module is used to determine the merging ratio based on Si, and to perform token filtering, weighted aggregation and token protection according to the similarity threshold.

[0070] The global attention processing module is used to perform a complete global self-attention computation on the global token sequence;

[0071] The location code update module is used to synchronously update the location code according to the token merging relationship;

[0072] The result fusion module is used to concatenate and fuse intra-frame attention output features with global attention output features.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] This invention introduces an interval-based token merging mechanism during the intra-frame attention stage of the visual Transformer, significantly reducing the computational complexity of the model, improving inference speed, and substantially reducing memory consumption. This allows the model to be efficiently deployed on various hardware platforms, such as low-memory GPUs and edge computing devices. This invention performs token merging only on intra-frame attention, while maintaining the complete token sequence for global attention. This achieves lightweight inference while maximizing the preservation of 3D geometric information and long-range feature dependencies, ensuring that the output accuracy of tasks such as camera pose estimation, depth estimation, and 3D point cloud reconstruction remains consistent with the original model. The merging strategy employed in this invention is highly flexible and controllable, supporting multi-level parameter adjustments for merging on / off, similarity threshold, frame merging interval, starting offset, maximum merging ratio, and minimum number of retained tokens, adapting to different computing power conditions, accuracy requirements, and application scenarios. During the merging process, the model forcibly protects the CameraToken and Register. Key special tokens such as Token are not included in the merging process, effectively preventing the loss of core information and ensuring stable and reliable model output. Furthermore, this invention adopts a plug-and-play design, requiring no modification to the original backbone network structure. It can be directly embedded into mainstream multi-view Transformer models such as VGGT, resulting in low engineering implementation costs, strong portability, and high versatility, making it widely applicable to multi-view systems. Figure 3 It is used in various multi-frame visual tasks based on alternating attention, such as 3D reconstruction, AR / VR, autonomous driving, robot vision, and intelligent security monitoring.

[0075] Furthermore, this invention constructs a two-parameter adaptive adjustment mechanism using the token feature complexity weight λv and the image gradient complexity weight λg. This allows the token merging strength to no longer depend on a fixed threshold or fixed ratio, but rather to dynamically change according to the feature distribution and texture structure of the input frame. For simple frames with high redundancy, this invention can increase the merging ratio to further reduce computation; for frames with complex textures or rich structural information, this invention can reduce the merging ratio to protect depth boundaries, camera pose, and 3D point cloud structure, thereby achieving a more stable balance between efficiency and accuracy under different scenes and different viewpoint inputs.

[0076] In summary, this invention innovatively optimizes the computational cost of the model by utilizing the alternating attention mechanism of the visual Transformer. Combined with dynamically adjustable token merging and merging intervals, it can be configured according to specific needs, demonstrating excellent performance in computer vision reconstruction tasks across various environments. Experimental results show that its performance meets expectations, providing a dynamic and rapid visual reconstruction method. Attached Figure Description

[0077] Figure 1 This forms the overall framework of the method model of the present invention.

[0078] Figure 2 The images show the visual reconstruction effect of the method model of this invention and a comparison image of similar models.

[0079] Figure 3 This is a schematic diagram of a sliding window. Detailed Implementation

[0080] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0081] Example 1

[0082] like Figure 1 As shown, a visual Transformer intra-frame attention optimization method based on interval merging is described, with the following specific steps:

[0083] Step 1: Initialize the aggregator module and configure parameters

[0084] Before model inference begins, the intra-frame token merging module is initialized by setting the following key parameters in the model code:

[0085] enable_token_merge_in_frame: True indicates that intra-frame token merging is enabled;

[0086] token_merge_threshold: 0.90, means that only tokens with a similarity higher than 0.90 will be merged;

[0087] token_merge_frame_interval: 3, indicating that merging is performed every 3 frames;

[0088] token_merge_frame_offset: 0 indicates that the merging is triggered starting from frame 0.

[0089] By setting the parameters as described above, the model can achieve significant inference acceleration and memory savings while maintaining high accuracy.

[0090] Step 2: Multi-frame image input and initial token generation

[0091] Input a multi-frame image sequence with shape [B,S,3,H,W], where B is the batch size, S is the number of frames, 3 is the number of channels, and H and W are the image height and width, respectively.

[0092] First, the images are standardized. Then, Patch Embedding is performed, dividing each frame into fixed-size image patches. A linear projection layer maps these patches to D-dimensional feature vectors, forming an initial Patch Token sequence. Camera Tokens and Register Tokens are added to the front of the Patch Tokens to carry camera parameter information and enhance feature stability, respectively. Simultaneously, a two-dimensional positional code is generated for each Token, injecting spatial location information into the feature sequence to ensure the model possesses spatial structure awareness.

[0093] Step 3: Alternating attention processing and intra-frame spacing merging

[0094] This step is the core optimization process. The model maintains two independent tokens: Intra-frame tokens: used for intra-frame attention, which can be merged; Global tokens: used for global attention, which are not merged.

[0095] 3.1 Intra-frame self-attention calculation

[0096] Multi-layer intra-frame self-attention and feedforward network computation is performed on the intra-frame token to complete the interaction and extraction of local features within a single frame, and to obtain preliminary intra-frame features.

[0097] 3.2 Determine whether to perform interval merging

[0098] Determine whether the merging condition is met based on the global frame index:

[0099]

[0100]

[0101] When the merging interval is 3, merging is only performed on frames 0, 3, 6, 9..., and the merging step is skipped for the remaining frames, and the calculation proceeds directly to the next frame. The interval merging strategy avoids feature degradation caused by continuous merging, achieving the best balance between accuracy and efficiency.

[0102] 3.3 Cosine Similarity-Based Token Weighting

[0103] 3.3.1 When the merger conditions are met, the following merger process shall be executed:

[0104] L2 normalization is applied to token features to improve the stability of similarity calculation; cosine similarity is calculated between a set of tokens (A, B). ;

[0105]

[0106] Highly similar token groups are selected based on a threshold and marked as the set to be merged. A weighted average of the same group of tokens is calculated using similarity as the weight to generate a new aggregated token. Camera tokens and Register tokens are strictly protected and do not participate in any merging. A minimum number of tokens to retain and a maximum merging ratio are set to avoid excessive merging that could reduce expressive power. After merging, the number of tokens within a frame is significantly reduced, and the computational cost of self-attention is greatly reduced.

[0107] 3.3.2 Sliding Window Protection Mechanism

[0108] In dense vision tasks such as 3D reconstruction, multi-view stereo matching, and Neural Radiation Field (NeRF) based on the Transformer architecture, image tokenization and hierarchical token merging are core operations for balancing model computational complexity and feature representation capabilities. The standard computational complexity of the Transformer self-attention module is O(n²d), where n is the token sequence length and d is the token feature dimension. A quadratic increase in sequence length directly leads to a sharp rise in computational overhead and memory usage in 3D scene modeling tasks. Therefore, reducing sequence length through token merging has become a core technical path for the Transformer architecture to adapt to high-resolution 3D vision tasks.

[0109] The current mainstream token merging paradigm is based on global similarity merging methods, represented by Token Merging (ToMe). Its core theoretical logic is to iteratively merge token pairs with the highest semantic similarity globally by calculating the cosine similarity of token pairs in a high-dimensional feature space, thereby downsampling the token sequence while minimizing the loss of global semantic information. However, this method is designed only for global semantic understanding tasks such as image classification, completely ignoring the semantic-geometric coupling characteristics of token features in 3D vision tasks. This results in an unavoidable fundamental flaw, the core of which can be demonstrated from the following theoretical dimensions:

[0110] First, the cosine similarity metric essentially only represents the directional consistency of token features in a high-dimensional semantic space, completely omitting the geometric positional information of the tokens in the image plane and 3D space. From the underlying principle of feature encoding, the feature vector of an image token is a coupled encoding of visual semantic information (texture, color, material) and geometric positional information. The Transformer self-attention mechanism itself possesses permutation invariance—if the positional encoding is removed, and the spatial arrangement of the input token sequence is completely shuffled, the output of the self-attention module will not change. This means that positional encoding is the sole information anchor point for the Transformer architecture to establish spatial geometric relationships and learn the 3D scene structure, and it is also the core basis for epipolar geometric constraints and triangulation to solve 3D coordinates in multi-view stereo matching.

[0111] Traditional global similarity merging methods completely break this core anchor point: when two tokens located in completely different positions in 3D space, with high cosine similarity only due to similar texture and color (e.g., homogeneous white blocks in the upper left and lower right corners of a single frame image, corresponding to the projections of the foreground wall and the background ceiling in a 3D scene, respectively), are forcibly merged, the merging operation is usually implemented using a feature-weighted average, and the corresponding positional codes are also simultaneously weighted averaged. This directly leads to the merged token positional codes becoming a weighted result of two spatially unrelated and geometrically discontinuous positions, completely losing their true physical geometric meaning, resulting in the fundamental error described in the original text: "the positional codes are incorrectly recorded."

[0112] Second, global similarity merging severely disrupts the spatial topological relationships of tokens within a single frame, leading to edge confusion and 3D reconstruction collapse in dense prediction tasks. From the perspective of local continuity priors and multi-view geometry in digital image processing, spatially adjacent tokens in natural images correspond to continuous object surfaces in a 3D scene, exhibiting strong geometric correlation and semantic continuity. However, tokens spatially distant, even with similar semantic features, correspond to completely independent rays and depth regions in 3D space, lacking any geometric continuity. The global matching logic of traditional merging methods completely violates this fundamental prior, resulting in a complete disruption of the spatial arrangement of tokens within a single frame. Cross-regional erroneous merging directly triggers two fatal problems: firstly, edge confusion: image edges correspond to object contours and depth transition regions in a 3D scene, forming the core geometric boundaries for 3D reconstruction. Global merging easily merges tokens on both sides of the edge, belonging to different depth layers but with similar features, causing the depth transition information of the edge to be blurred and the semantic boundaries between foreground and background to be destroyed, ultimately resulting in object contour adhesion and loss of edge details in the reconstruction result; secondly, 3D reconstruction collapse: in multi-view... Figure 3In 3D reconstruction tasks, the network needs to use the positional correspondence of tokens in different views to complete feature matching and triangulation based on epipolar geometric constraints, thereby reconstructing the point cloud and depth information of the 3D scene. When the positional encoding of tokens within a single frame is incorrectly merged or the spatial topology is disrupted, cross-view feature matching will result in global correspondence errors. The network will be unable to solve for 3D coordinates that conform to the real physical laws, ultimately leading to completely disordered depth estimation, distorted scene geometry, and large-area holes or even complete failure in the reconstruction results, i.e., reconstruction collapse.

[0113] To address the fundamental shortcomings of traditional global similarity merging methods, this invention proposes a sliding window token merging mechanism based on local constraints. Its core design draws on the local window attention theory validated by dense visual models such as the Swing Transformer. By using spatial neighborhood constraints, it fundamentally eliminates erroneous merging across regions and fully preserves the spatial topological relationship of the tokens and the geometric validity of their positional encoding.

[0114] like Figure 3 As shown, the core implementation logic and theoretical basis of the sliding window merging mechanism of the present invention are as follows: First, the one-dimensional flattened token sequence is restored to a two-dimensional feature grid corresponding to the spatial structure of the input image according to its two-dimensional spatial coordinates in a single frame image, ensuring that the sequence order of the tokens corresponds one-to-one with the spatial position; then, a sliding window of a fixed size (such as M×M token size) is set to strictly limit the candidate set of the merging operation to the internal spatial range of a single sliding window, and the merging operation is only performed on tokens that are spatially adjacent within the window; when the tokens in a single window are merged, the window slides to the next spatial position according to a preset step size, and the merging process is repeated until the token downsampling of the entire frame image is completed.

[0115] This mechanism addresses the core shortcomings of traditional methods at the fundamental level, and its theoretical advantages can be summarized in three points:

[0116] 1. Fundamentally eliminates erroneous merging across spatial regions: The hard constraint of the sliding window ensures that tokens that are spatially distant (such as color blocks in the upper left and lower right corners of an image) will never enter the same merging candidate set. This completely avoids the problem of merging non-adjacent tokens caused by similar global features, and ensures that the merging operation always conforms to the prior knowledge of the local continuity of natural images.

[0117] 2. Fully preserve the physical geometric meaning of position encoding: The merging operation is only performed between spatially adjacent Tokens. The position encoding of the merged Token is the weighted average of the positions within the local neighborhood, and the result still corresponds to the real spatial center position within the window, with clear physical geometric meaning, which provides a reliable position anchor for cross-view feature matching and 3D coordinate solving, and fundamentally avoids erroneous recording of position encoding.

[0118] 3. Effectively protect edge structures and scene geometric continuity: The local constraint of the sliding window ensures that the merging operation does not cross the depth jump edges of the scene. Tokens in the edge area are only merged within their respective local windows, which fully preserves the feature specificity and geometric boundary information of edges, and solves the edge blurring problem of traditional methods; at the same time, stable position encoding and spatial topological relations ensure the accuracy of multi-view matching, completely avoiding the 3D reconstruction collapse problem caused by position disorder.

[0119] In addition, from the theoretical dimension of computational efficiency, the traditional global similarity merging needs to calculate the cosine similarity of all Token pairs, with a time complexity of O(n²); while the sliding window merging of the present invention only needs to calculate the similarity of M² Tokens in each window, and the overall time complexity is reduced to O(n·M²), where the window size M is much smaller than the total Token number n of the full image, which further reduces the computational overhead of the merging operation while realizing geometric structure protection.

[0120] 3.4 Determination of Adaptive Merging Ratio

[0121] Dynamically determine the Token merging ratio R for the i-th frame according to the frame-level complexity score Si i . A lower complexity score indicates higher Token redundancy of this frame, so a larger merging ratio can be adopted; a higher complexity score indicates that this frame contains more texture, boundary or structural information, so a smaller merging ratio should be adopted to reduce the impact on 3D geometry prediction.

[0122] In a segmented implementation, two complexity thresholds T1 and T2 may be set, and three levels of merging ratios R are set high , R mid and R low , where R high >R mid >R low . When Si<T1, the merging ratio of this frame is set to Rhigh; when T1≤Si<T2, the merging ratio of this frame is set to R mid ; when Si≥T2, the merging ratio of this frame is set to R low .

[0123] In another continuous implementation, the merging ratio can also be determined according to the following formula:

[0124] Ri=R max -Si γ ·(R max -R min )

[0125] Among them, R max R represents the maximum consolidation ratio. min This represents the minimum merging ratio, and γ represents the complexity response curve adjustment parameter. This continuous mapping enables smoother inter-frame merging intensity variations.

[0126] After determining the merging ratio Ri, the model determines the number of tokens that can be merged in the current frame based on Ri, the minimum number of tokens to be retained, and the similarity threshold, and performs token merging during the intra-frame attention stage.

[0127] 3.5 Location Encoding Synchronous Update: Token merging changes the sequence length and spatial correspondence, therefore, the location encoding is updated synchronously: the merging source of each token is recorded according to the merging mapping relationship; a weighted average of the location encodings of the merged tokens is performed to generate a new token location encoding; the location encodings of special tokens remain unchanged; and the token sequence and location encoding are dimensionally aligned to ensure the validity of subsequent calculations.

[0128] 3.6 Global Attention Calculation

[0129] Global self-attention computation is performed on the complete global token throughout the process, without any compression, preserving long-distance dependencies between frames and global geometric structure to ensure 3D reconstruction accuracy. This embodiment provides an optimization system corresponding to the method, including the following modules: A parameter configuration module is responsible for loading, storing, and adjusting all parameters for intra-frame token merging, supporting dynamic switching, real-time adjustment, and configuration saving. A token generation module completes functions such as image normalization, patch embedding, special token addition, and position encoding generation. A dual-token maintenance module independently maintains intra-frame tokens and global tokens, ensuring that the two sets of tokens do not interfere with each other during computation, merging, and updating. An intra-frame attention processing module performs intra-frame self-attention computation and determines whether to trigger token merging based on an interval strategy. A token merging module performs token filtering, weighted aggregation, special token protection, and merging strength control based on cosine similarity. A global attention processing module performs global self-attention computation on the complete token sequence, maintaining global information without loss. A position encoding update module synchronously updates position information according to the token merging relationship to ensure spatial consistency. A result fusion module concatenates and fuses intra-frame features and global features, outputting an optimized feature representation for downstream tasks. This system is integrated into existing multi-view Transformer models in a plug-in manner, achieving efficient inference acceleration without modifying the backbone network.

[0130] 3.7 Intra-frame and global feature fusion

[0131] The intra-frame attention output and the global attention output are concatenated along the channel dimension to form a fusion feature that combines high efficiency and high precision: concat_inter=torch.cat([frame_inter,global_inter],dim=-1)

[0132] Step 4: Additional 3D Parameter Task Inference

[0133] The fused features are input into each task head to complete the final output: Camera pose estimation: predict camera intrinsic parameters, extrinsic parameters, rotation matrix and translation vector through the camera pose head; Depth estimation: output dense depth map through the depth head; 3D point cloud reconstruction: generate 3D point cloud coordinates through the point cloud head to complete scene reconstruction.

[0134] Example 2

[0135] Based on Example 1, this embodiment further provides a two-parameter adaptive merging ratio adjustment method based on token feature complexity and image gradient complexity.

[0136] For the input multi-view image sequence, let the Patch Token set corresponding to the i-th frame be Xi={xi,1,xi,2,...,xi,N}, where N is the number of Patch Tokens, and xi,j is the feature vector of the j-th Patch Token in the i-th frame. First, the Token feature complexity Vi of this frame is calculated. Vi can be represented by the mean variance of the Patch Tokens on the feature dimension, the feature dispersion between Tokens, or the mean standard deviation, for example:

[0137] Vi=Mean(Var(Xi))

[0138] where Var denotes performing variance statistics on the Patch Token features, and Mean denotes taking the average. This indicator is used to measure the dispersion degree of the Token feature distribution within the i-th frame.

[0139] At the same time, the image gradient complexity Gi is calculated for the input image Ii of the i-th frame. Gi can be represented by the mean of the magnitudes of the horizontal gradient and the vertical gradient of the image:

[0140] Gi=Mean(| x Ii|+| y Ii|)

[0141] where ∂x and ∂y respectively represent the gradient operators of the image in the horizontal direction and the vertical direction. This indicator is used to measure the complexity of image textures, edges and local structures.

[0142] To avoid inconsistency in the dimensions of different indicators, Vi and Gi of all frames in the same input sequence are normalized respectively to obtain V'i and G'i. Then, the two adjustable weight parameters λv and λg are used to fuse the two:

[0143] Si=(λv·V'i+λg·G'i) / (λv+λg+ε)

[0144] where Si is the adaptive complexity score of the i-th frame, λv represents the weight of the Token feature complexity, λg represents the weight of the image gradient complexity, and ε is a numerical stability term.

[0145] The Token merging ratio Ri of the i-th frame is dynamically allocated according to Si. When Si is low, it indicates that the redundancy of this frame is high, and a higher merging ratio can be adopted; when Si is high, it indicates that this frame contains more structural information or texture information, and the merging ratio should be reduced to protect the reconstruction quality. In one embodiment, a three-stage mapping rule is adopted:

[0146] If Si<T1, then Ri=Rhigh;

[0147] If T1≤Si<T2, then Ri=Rmid;

[0148] If Si≥T2, then Ri=Rlow;

[0149] where Rhigh, Rmid, and Rlow respectively represent a high, medium, and low merging ratio, and Rhigh>Rmid>Rlow. In this manner, the present invention can dynamically adjust the Token merging intensity according to the content of an input frame, reducing the amount of calculation while lowering the information loss of complex frames.

[0150] Performance Comparison Experiment

[0151] A comparative experiment is conducted on a public multi-view dataset, and the results are as Figure 2 shown. Compared with the original VGGT and FastVGGT, in tasks such as depth estimation, point cloud reconstruction, and camera pose prediction, the present invention has significantly reduced total inference time, markedly decreased memory usage, and substantially the same precision as the original model.

[0152] Comparison of different merging strategies:

[0153] No merging: the maximum number of Tokens, the slowest speed, and the highest memory usage;

[0154] Per-frame merging: the fastest speed, but with precision loss;

[0155] Interval merging: optimal balance between efficiency and precision.

[0156] Experiments prove that the present invention can achieve dual optimization of calculation amount and memory usage on the premise of maintaining 3D reconstruction quality.

[0157] Parameter Tuning and Engineering Suggestions

[0158] To facilitate practical deployment, the optimal parameter ranges are provided as follows:

[0159] Similarity threshold: 0.90–0.95, balancing efficiency and precision;

[0160] Frame merging interval: 2–3 frames, suitable for most real-time scenarios;

[0161] Maximum merging ratio: 10%–15%, to avoid feature degradation;

[0162] Minimum number of reserved Tokens: ≥128, to ensure basic representation capability.

[0163] Users can freely adjust the parameters according to the computing power platform, real-time performance requirements, and precision targets to achieve the optimal effect.

[0164] Experimental Design and Result Analysis

[0165] Experimental Parameter Setting

[0166] 1. Hardware and software environment (local)

[0167] Hardware: GPU: RTX 5060 (16GB VRAM), CPU: AMD Ryzen 9 8945HX, RAM: 32GB.

[0168] Software: PyTorch 2.8.0, CUDA 12.8 / cuDNN 8.0, Python 3.10.

[0169] Comparative test

[0170] This experiment selected several comparative models with identical task output capabilities and parameter settings for a control experiment. All models were able to simultaneously complete camera pose estimation, 3D point cloud numerical calculation, and depth map numerical calculation tasks. The total inference time in the experiment included all preprocessing stages such as data loading, image normalization, and patch embedding, as well as the forward inference stage. Since the preprocessing flow and computational cost of the three models were basically the same, the time differences were mainly concentrated in the inference time of the three core tasks: camera pose estimation, point cloud numerical calculation, and depth numerical calculation. Therefore, this experiment used the time of the three core tasks and the total time as the main evaluation indicators for comparative analysis.

[0171] The experiment used the same set of multi-view image sequences with the same number and scene as a unified input to ensure consistency and fairness in the comparison conditions. The selected comparison models included the classic VGGT baseline model and the lightweight FastVGGT model. The FastVGGT model employs a unified global attention layer merging strategy, which contrasts sharply with the intra-frame interval merging scheme proposed in this invention. This effectively verifies the impact of different token merging mechanisms on model efficiency and performance, providing significant reference value and comparative significance. Table 1 shows the comparison results of the inference time of the proposed method and the two aforementioned comparison models in the same scene. The experiment recorded and statistically analyzed the individual inference time and total time of the three models for camera pose estimation, point cloud generation, and depth estimation tasks. Experimental results show that the intra-frame attention optimization method based on interval merging proposed in this invention significantly outperforms the VGGT baseline model in both single-task time and total time in multi-task joint inference scenarios, achieving a more significant inference acceleration effect while maintaining high-precision output.

[0172] Table 1 shows the experimental results compared with other 3D reconstruction methods.

[0173]

[0174] In this experiment, the model's parameters were set as follows: cossim threshold = 0.90, and merging interval k = 2. This means that within the same group of tokens, if the cosine similarity is greater than 0.90, they are merged into a single token, and a label is added to indicate its relative position in the image. Using the merged token in the calculation significantly reduces computation time. However, if every intra-frame attention layer is merged, the computational accuracy will be greatly reduced. Therefore, we set the merging interval k = 2, which allows the number of alternating attention layers between two adjacent merged layers to be two, meaning complete merging is unnecessary. This approach maintains controllable degradation in reconstruction and inference accuracy while significantly improving inference speed.

[0175] Dynamic parameter experiment

[0176] This experiment will test the dynamic parameter adjustment mechanism of the model. The adjustable parameters are the merging threshold (cossim) and the merging interval (k). Under the condition of setting the same merging threshold (Cossim=0.90), the impact of different merging intervals on various experimental data was explored. As shown in Table 2, the experimental results clearly demonstrate that different merging intervals have a significant impact on performance.

[0177] Table 2 shows the impact of different thresholds on model performance at the same merging interval (k=2).

[0178]

[0179] As shown in Table 2, under the same merging threshold, the different values ​​of k determine which layers of intra-frame attention need to be merged. The smaller the value of k, the more layers are merged for the same number of layers. When k=1, precision is achieved using Token. Therefore, while dynamically adjusting the two parameters, we strive to find a balance point. After extensive experimentation, this model recommends a merging threshold of Cossim=0.95 and a merging interval of k=2.

[0180] Table 3. Impact of different merging thresholds on model performance at the same merging interval (k=2).

[0181]

[0182] As shown in Table 3, different merging thresholds have a significant impact on the multi-task performance of the model under the same merging interval. With a gradually decreasing merging threshold, the number of tokens within the intra-frame attention of the same layer is greatly reduced, thus significantly improving the model's processing speed. Without merging...

[0183] Two-parameter adaptive ablation experiment

[0184] To verify the effectiveness of the two-parameter adaptive adjustment mechanism, this experiment compares different complexity scoring strategies under the same multi-view image input, the same hardware environment, and the same basic model parameters. The experimental comparison objects include: an adaptive strategy using only the token feature complexity index, an adaptive strategy using only the image gradient complexity index, and the proposed dual-parameter joint adaptive strategy of token feature complexity and image gradient complexity.

[0185] Specifically, when using only the token feature complexity metric, λv is set to 1 and λg to 0; when using only the image gradient complexity metric, λv is set to 0 and λg to 1; and when using the two-parameter joint adaptive strategy, both λv and λg are set to non-zero values, for example, λv=0.5 and λg=0.5. All three strategies were tested under the same merging threshold, the same merging interval, the same maximum merging ratio, and the same minimum number of tokens retained, as shown in Table 4.

[0186] Table 4. Design of experimental indices for ablation using a two-parameter adaptive strategy.

[0187]

[0188] Experimental results show that using the token feature complexity metric alone can adjust the merging ratio according to the dispersion of features within the model, but it is insufficient in responding to changes in texture edges in the input image. Using the image gradient complexity metric alone can better reflect image texture and edge strength, but it cannot fully reflect the differences in deep token feature distribution. This invention, by jointly adjusting the two weight parameters λv and λg, can simultaneously consider both the feature complexity within the model and the texture complexity of the input image, thus forming a more reasonable merging ratio distribution between different input frames. This achieves a more stable inference acceleration effect while ensuring the quality of 3D reconstruction.

[0189] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A visual Transformer intra-frame attention optimization method based on interval merging, characterized in that, include: Initialize the aggregator module and configure the token merging control parameters. The token merging control parameters include at least the frame merging interval, frame merging start offset, token merging similarity threshold, token feature complexity weight parameter λv, and image gradient complexity weight parameter λg. The input multi-frame images are preprocessed by converting the images into Patch Tokens through Patch Embedding, and Camera Token and Register Token are added to the beginning of the sequence. At the same time, position encoding is added to all tokens. Maintain two independent token sequences, including an intra-frame token sequence for intra-frame attention and a global token sequence for global attention; During the alternating attention process, global self-attention calculation is performed on the global token sequence to maintain sequence integrity and prevent merging; intra-frame self-attention calculation is performed on the intra-frame token sequence, and the current frame is judged to meet the merging trigger condition based on the frame merging interval and frame merging start offset; when the merging trigger condition is met, the adaptive complexity score Si of the current frame is calculated, and the token merging ratio Ri of the current frame is determined based on Si; based on the token merging similarity threshold and the token merging ratio Ri, the token merging operation is performed on the intra-frame token sequence, and the position encoding is updated synchronously. The intra-frame attention output features and the global attention output features are concatenated and fused to obtain the final output features.

2. The visual Transformer intra-frame attention optimization method based on interval merging according to claim 1, characterized in that, The calculation of the adaptive complexity score Si for the current frame includes: Calculate the token feature complexity index Vi of the current frame Patch Token; Calculate the gradient complexity index Gi of the input image in the current frame; Normalize Vi and Gi respectively to obtain normalized V'i and G'i; The adaptive complexity score Si is calculated using the formula Si=(λv·V′i+λg·G′i) / (λv+λg+ε), where ε is a stabilizing term to prevent the denominator from being zero.

3. The visual Transformer intra-frame attention optimization method based on interval merging according to claim 2, characterized in that, The token feature complexity index Vi is calculated in the following way: Extract the mean variance of all Patch Tokens in the current frame, the mean feature dispersion or standard deviation of the tokens as the token feature complexity index Vi.

4. The visual Transformer intra-frame attention optimization method based on interval merging according to claim 2, characterized in that, The gradient complexity metric Gi is calculated in the following way: Calculate the horizontal gradient of the input image in the current frame. gradient in the x and vertical directions y, calculate the gradient in the horizontal direction gradient in the x and vertical directions The mean value of the gradient magnitude of y is used as the gradient complexity index Gi.

5. The visual Transformer intra-frame attention optimization method based on interval merging according to claim 1, characterized in that, The determination of the token merging ratio Ri of the current frame based on Si includes: Preset low-complexity merging ratio R low Medium complexity merging ratio R mid Merging ratios of high-complexity components R high , where R high >R mid >R low ; Preset a first complexity threshold T1 and a second complexity threshold T2; If Si < T1, set Ri = R high ; If T1≤Si<T2, then set Ri=R mid ; If Si ≥ T2, then set Ri = R. low .

6. The visual Transformer intra-frame attention optimization method based on interval merging according to claim 1, characterized in that, Perform a token merging operation on the intra-frame token sequence, including: L2 normalization is applied to the token features; Calculate the cosine similarity matrix between tokens; Filter similar token groups based on token merging similarity threshold; A new aggregated token is generated by weighting the tokens based on their similarity. The Camera Token and Register Token are protected from being merged.

7. The visual Transformer intra-frame attention optimization method based on interval merging according to claim 6, characterized in that, The token merging operation also includes a sliding window local constraint step: The intra-frame token sequence is restored to a two-dimensional feature grid corresponding to the spatial structure of the input image. Set a fixed-size sliding window and only perform a merge operation on spatially adjacent tokens that fall within the same sliding window; The sliding window is controlled to move on the feature grid at a preset step size until the entire frame of the image is traversed.

8. The visual Transformer intra-frame attention optimization method based on interval merging according to claim 1, characterized in that, The token merging control parameters also include the maximum merging ratio and the minimum number of tokens to retain; When performing a token merging operation, the actual merging ratio of the current frame is limited to the maximum merging ratio, and the total number of tokens after merging is ensured to be no less than the minimum number of tokens to be retained.

9. The visual Transformer intra-frame attention optimization method based on interval merging according to claim 1, characterized in that, The synchronous update of the location code includes: Record the merged source mapping relationship for each new aggregated token; Based on the merge mapping relationship, the position codes of the merged tokens are weighted and averaged to generate the position codes of the new tokens; Keep the location encoding of the Camera Token and Register Token unchanged.

10. A visual Transformer intra-frame attention optimization system based on interval merging, performing the method as described in any one of claims 1-9, characterized in that, include: The parameter configuration module is used to initialize and store token merging control parameters; The Token generation module is used to perform Patch Embedding on the input image, generate Patch Tokens, and add Camera Tokens, Register Tokens, and location codes. The dual-Token maintenance module independently maintains the intra-Token sequence for intra-frame attention and the global Token sequence for global attention. The intra-frame attention processing module is used to perform intra-frame self-attention calculation and trigger token merging according to the interval strategy; The complexity scoring module is used to calculate the token feature dispersion index and image gradient complexity index of the current frame, and to calculate the adaptive complexity score Si through a fusion formula. The Token merging module is used to determine the merging ratio based on Si, and to perform token filtering, weighted aggregation and token protection according to the similarity threshold. The global attention processing module is used to perform a complete global self-attention computation on the global token sequence; The location code update module is used to synchronously update the location code according to the token merging relationship; The result fusion module is used to concatenate and fuse intra-frame attention output features with global attention output features.

Images