A visual token removal method for video understanding model inference acceleration

Abstract

The application provides a visual token removal method for video understanding model inference acceleration. The method is based on visual attention contraction phenomenon. By analyzing the distribution and change of attention weight of visual token in the inference pre-filling stage, the importance of visual token is dynamically evaluated, redundant tokens are screened and removed, thereby reducing the calculation overhead and improving the inference efficiency. The method comprises the following steps: frame sampling and region division are performed on the video input to obtain image blocks with space-time structure; visual features are extracted and visual tokens are generated by using a visual encoder and a projector; the visual tokens, system tokens and text tokens are spliced to form a multi-modal input; the importance of the visual tokens is evaluated by tracking the attention change trend of the text to the visual; and the unimportant visual tokens are accelerated to be identified by grouping and aggregation, and are gradually removed in the inference. While maintaining the output quality, the method significantly reduces the inference time and memory occupation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a visual token removal method for accelerating inference in video understanding models. Background Technology

[0002] With the rapid development of artificial intelligence technology, visual language models have become a hot research topic at the intersection of computer vision and natural language processing. These models, by combining visual encoders and language models, have achieved the understanding and generation of multimodal inputs and have demonstrated strong performance in image-text and video-text tasks. Especially in video understanding tasks, VLMs have been widely applied to action recognition, video summarization, and video content analysis. However, compared to image input, existing technologies have revealed many shortcomings in terms of processing efficiency and computational resource requirements in order to comprehensively acquire the greater amount of visual information in video input.

[0003] In video understanding tasks, Visual Models (VLMs) need to sample the video and divide each frame into multiple independent image patches, which are then transformed into visual token input models. As the number of video frames increases, the computational complexity quadratically increases with the number of visual tokens, leading to a significant increase in computational overhead, especially in long video tasks where computation and storage become bottlenecks.

[0004] To address the problem of excessive visual tokens, some optimization methods have been proposed, but all have significant limitations:

[0005] Visual encoder optimization: The number of visual tokens is reduced by adjusting the structure of the visual encoder, but the influence of different text inputs is not taken into account, and additional training is usually required on the model used, which increases the development cost and makes it difficult to be compatible with other existing models.

[0006] Token merging strategy: Merging visual tokens into a more compact representation to reduce computational complexity. However, this approach is currently highly dependent on specific model architectures, limiting its versatility and potentially leading to the loss of key information due to over-compression.

[0007] Directly discarding tokens: Some methods randomly or directly remove some tokens based on simple rules, but fail to make full use of the attention mechanism to dynamically evaluate the importance of tokens, resulting in an uncontrollable decline in model performance.

[0008] Attention mechanisms and their corresponding structures are important components of VLMs, but existing methods often fail to fully utilize attention mechanisms to evaluate the importance of visual tokens, resulting in a large number of tokens containing redundant information participating in the computation, which increases the inference time and memory consumption of the model.

[0009] In long video understanding tasks, video sequences contain a large amount of redundant information (such as minimal content changes between adjacent frames and highly similar background image blocks). Existing technologies have failed to effectively identify and remove this redundant information, resulting in inference efficiency that is difficult to meet the needs of practical applications. Furthermore, some optimization methods may compromise the model's output quality by excessively discarding information.

[0010] To address the above shortcomings, this invention proposes a visual token removal method based on the phenomenon of visual attention contraction. By dynamically filtering unimportant visual tokens, it significantly improves inference efficiency while ensuring the stability and versatility of model performance. Summary of the Invention

[0011] The purpose of this invention is to solve the technical problems mentioned in the background section and to provide a visual token removal method for accelerating inference in video understanding models.

[0012] The solution of the present invention to achieve the above objective is as follows:

[0013] A visual token removal method for accelerating inference in video understanding models is disclosed. Based on the phenomenon of visual attention contraction, the method dynamically evaluates the importance of visual tokens by analyzing the attention weight distribution at different levels during the inference process of a video visual language model, and gradually removes redundant visual tokens. The method includes the following steps:

[0014] a) Video input processing: The input video V is sampled into several frames, and each frame is divided into multiple independent image blocks to obtain the initial visual features;

[0015] b) Visual token generation: via visual encoder f ve Further extract visual features and use a projector f proj Mapped to the visual token space, the formula for generating visual tokens is:

[0016] X v =f proj (f ve (V));

[0017] Among them, X v This represents a visual token, where V represents the input video.

[0018] c) Multimodal input concatenation: This involves concatenating the system token X... s Visual Token X v and text token X t By concatenating the elements in sequence, we obtain the final input X. The concatenation formula is:

[0019] X = Concat(X) s X v X t );

[0020] d) Attention Weight Analysis: During the inference pre-filling stage, the attention weights of text tokens to visual tokens are calculated using an attention mechanism, and the importance of each visual token is calculated based on these attention weights. The formula is as follows:

[0021]

[0022] Among them, X v，i Represents the i-th visual token, Attention j，i This represents the attention weight of the j-th text token to the i-th visual token, where T is the number of text tokens;

[0023] e) Visual Token Grouping and Aggregation: Group visual tokens by time and space dimensions, and calculate the importance of each group using an aggregation formula.

[0024]

[0025] Where G represents the visual token group, Importance(X) v，i ) represents the attention of a single visual token within a group;

[0026] f) Dynamic token removal: Setting the starting layer L start and the end layer L end Based on the importance assessment results, visual token groups with lower values ​​after weighted summation of attention weights are gradually removed;

[0027] g) Inference optimization: After removing redundant visual tokens, the remaining visual tokens are combined with text tokens to complete the inference calculation, thereby improving inference efficiency.

[0028] On the other hand, the attention weight analysis is based on the multi-layer attention weight of the inference pre-filling stage, tracks the trend of attention change of text tokens to visual tokens, identifies visual tokens whose attention is continuously decreasing and removes them.

[0029] On the other hand, the visual token grouping includes time-dimension grouping and spatial-dimension grouping, and the attention weights after grouping are aggregated and calculated through time and spatial dimensions respectively.

[0030] On the other hand, the visual token removal operation is a progressive removal, removing r% of the current visual tokens each time, and the specific percentage r can be dynamically adjusted according to task requirements.

[0031] On the other hand, the execution conditions for the visual token removal operation include:

[0032] a) Triggered when no token removal occurs across multiple consecutive multi-level blocks;

[0033] b) Triggered when a certain percentage of visual token groups show a sustained decrease in attention level.

[0034] On the other hand: the token removal operation in the inference pre-filling stage is configured with a start layer and an end layer, with the start layer set to the Lth layer. start The end layer is set to the Lth layer. end The initial layer is designed to prevent the premature loss of key visual information, while the final layer is designed to ensure stable model output.

[0035] On the other hand, the video frames are divided into fixed-size image blocks, with each image block serving as a visual token containing temporal and spatial dimension information.

[0036] On the other hand, the method is compatible with existing KV caching mechanisms, token removal only occurs during the inference pre-filling stage, and the retained cache can be reused in subsequent generation processes.

[0037] On the other hand, the removal operation is based on maintaining the spatiotemporal structural integrity of the video sequence, and the removed tokens will not destroy the temporal continuity and spatial distribution characteristics between video frames.

[0038] On the other hand, the method is applicable to tasks such as video content analysis, action recognition, and video summarization, and can significantly reduce inference time and memory usage while maintaining output accuracy.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] This invention significantly reduces model computational overhead by dynamically evaluating the importance of visual tokens during the inference pre-filling stage using an attention mechanism and progressively removing unimportant visual tokens. This method strikes a balance between inference quality and efficiency, reducing inference time by 40%-70% by removing visual tokens, while simultaneously reducing the memory usage required for visual tokens during inference by more than 50%, thus meeting the demands for high-efficiency processing in practical applications.

[0041] The attention-weight-based dynamic filtering method can accurately identify key information and avoid discarding too many important visual tokens, thereby reducing computational complexity while maintaining the accuracy and stability of the model output. Compared with existing methods, this invention can achieve a dual improvement in performance and efficiency in video understanding tasks such as action recognition, video content analysis, and summary extraction.

[0042] This method requires no significant adjustments to the visual encoder or model structure and can be applied plug-and-play to various existing video visual language models. Furthermore, by flexibly setting the start and end layers for token removal and adjusting the dynamic removal ratio, the method is applicable to task types of varying difficulty, demonstrating good versatility and scalability.

[0043] This invention performs token filtering during the inference pre-filling stage, avoiding interference with the KV caching mechanism and thus preserving the efficient reuse capability of the KV cache. This design not only improves model inference speed but also reduces the complexity of practical applications and deployments.

[0044] This invention is applicable to tasks such as video content analysis, action recognition, and video summarization extraction. It demonstrates significant efficiency and performance advantages in various video understanding scenarios, providing an efficient and reliable technical solution for large-scale video data processing. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart of a visual token removal method for accelerating inference in a video understanding model, as described in an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the internal process of a visual token removal method for accelerating inference in a video understanding model according to an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings and several examples.

[0050] Example 1:

[0051] Video Input Processing: The input video V is sampled into several frames. For example, for an 80-second video clip, the model is set to sample 16 frames, divided evenly, i.e., one frame every 5 seconds. Each frame is further divided into fixed-size image patches, such as a single frame being divided into 12 × 12 = 144 image patches. Each image patch serves as a visual token, resulting in a total of 16 × 144 = 2304 visual tokens. The mathematical formula for generating visual tokens is:

[0052] Xv =f proj (f ve (V));

[0053] Among them, f ve This represents the visual encoder, responsible for extracting image patch features; f proj This represents a projector used to map visual features into a space aligned with a language model.

[0054] Multimodal input concatenation: System token X s The default setting for each model, along with the text token X. t Together they are generated by a language model word segmenter, and then combined with the visual token X. v By concatenating these sequences together, we obtain the complete input sequence:

[0055] X = Concat(X) s ,X v ,X t );

[0056] Attention Weight Analysis: During the pre-filling phase of inference, an attention mechanism is used to calculate the attention weights of text tokens on visual tokens. For the i-th visual token, its importance is defined as:

[0057]

[0058] Among them, Attention j，i Let T represent the attention weight of the j-th text token to the i-th visual token, where T is the total number of text tokens.

[0059] Visual token grouping and aggregation: Visual tokens are grouped according to temporal and spatial dimensions. For example, visual tokens within the same frame in the temporal dimension are considered a group, initially totaling 16 groups. In the spatial dimension, visual tokens of image patches at the same location are grouped together, corresponding to different regions in the video, initially totaling 144 groups. Each visual token belongs to both a temporal group and a spatial group. The importance of each group of tokens is aggregated using the following formula:

[0060]

[0061] Dynamic token removal: Set the starting layer L based on the distribution of attention weights. start and the end layer L endIf we set them as the 2nd layer and the 4th layer from the bottom respectively, during model inference, when the layers fall within this range, we record the attention values ​​of the most recent layers for each token group, i.e., the corresponding recent attention change trend, such as the most recent 3 layers. We then perform a weighted summation of the attention weights of each group in the most recent three layers, removing visual token groups with relatively low attention weights. The removal strategy is to remove r% of the current visual tokens each time, where the percentage r can be dynamically adjusted according to the task type and difficulty.

[0062] Inference optimization: By removing redundant visual tokens, computational overhead is reduced, and inference efficiency is optimized. The retained visual tokens are then combined with text tokens, allowing for multiple dynamic visual token removals to ultimately complete the inference computation. Since token removal only occurs during the pre-filling stage of model inference, it is compatible with commonly used key-value caching acceleration mechanisms. Key-value cache values ​​are obtained during the pre-filling stage and can be reused in the subsequent content generation stage.

[0063] Example 2:

[0064] In a video understanding multiple-choice question-answering task, the input is a 300-second video clip, and the model uses a total of 32 frames sampled. The implementation process is as follows: Each input frame is preprocessed into 12×12=144 image blocks, for a total of 32×144=4608 blocks. Visual tokens are then processed using formula X. v =f proj (f ve (V)) encoding is generated.

[0065] Attention Analysis and Token Removal: Each frame and location in the video is grouped according to both temporal and spatial dimensions. The attention value of the visual tokens aggregated in each group is calculated. For a 32-layer model, during the inference phase from layers 2 to 28, the changes in the attention values ​​of each token group over the most recent layers are recorded and weighted. Token removal is performed every three layers or when more than 30% of the token groups experience a continuous decrease in attention value. Based on the attention weight value obtained from the current layer's weighting, the 30% of the most recently acquired text tokens with lower attention values ​​are removed, group by group. Multiple dynamic visual token removals are performed throughout the inference process.

[0066] Inference results: After gradually removing unimportant visual tokens, the total inference time decreased by more than 50%, the memory usage of the corresponding visual tokens was about 45% of the original, and the model's performance and accuracy on the corresponding task were still maintained at more than 99%.

[0067] Example 3:

[0068] In a video content classification task, the input is a 60-second video clip, and 8 frames are sampled. Each frame is divided into 144 image blocks, generating a total of 8 × 144 = 1152 visual tokens.

[0069] Visual token X generated by splicing v With system token X s and text token X t This forms the input sequence X.

[0070] Dynamic token filtering: Visual tokens are grouped in the time and space dimensions, namely 8 time groups and 144 space groups.

[0071] The removal rate is set to 50%. Attention weights are calculated through aggregation and weighting in layers 2-24 of the model to determine when dynamic visual token removal should be performed. For a specific sample, removal is determined at layers 5 and 13 based on the temporal dimension, and at layers 9, 15, 18, and 22 based on the spatial dimension. The number of temporal groups decreases to 4 and 2 respectively, and the number of spatial groups decreases to 72, 36, 18, and 9 respectively. The number of visual tokens gradually decreases during inference, and in the final stage, only 2 × 9 = 18 visual tokens are retained.

[0072] Inference performance: After the token is removed, the inference speed is improved by more than 120%, and the accuracy drops by less than 0.4%.

[0073] Example 4:

[0074] In the video content annotation task, the input is a 10-minute video. Initial visual information is obtained by sampling and segmenting the video into blocks. Subsequent encoding and projection processing yields a total of 8 × 196 = 1568 visual tokens. These visual tokens are then grouped according to the spatiotemporal dimensions.

[0075] Attention filtering and removal: The attention weights of each group are aggregated. If more than 30% of the frames show a continuous decline in attention weight, token removal is determined. The bottom 30% of time groups (frames) in importance are selected, and all visual tokens for these groups are removed. Visual tokens for the remaining groups are retained.

[0076] During subsequent model inference propagation, attention tracking and visual token removal are continuously performed.

[0077] Annotation generation: The optimized output generates annotations that cover the main content, reduces inference time by about 45%, and decreases the overall evaluation score (out of 5) by only 0.1 points.

[0078] Example 5:

[0079] This method performs token filtering and removal during the inference pre-filling stage, and the retained visual and text tokens optimize the efficiency of the subsequent generation stage through a KV caching mechanism.

[0080] Caching optimization: During the model inference pre-filling stage, the input sequence of each layer, i.e., the key and value of each token, is stored. Attention weights can be calculated from the key and value, which are used to dynamically calculate the importance of the tokens.

[0081] When a visual token is removed, the removed visual tokens will not be included in the inference propagation of subsequent layers of the model. The retained visual tokens will be directly concatenated with the system tokens and text tokens to form the input of the next layer, thereby reducing the storage overhead of the KV cache.

[0082] Inference results: The model pre-filling inference time is reduced by about 50%, the storage overhead of the visual token KV cache is reduced by more than 40%, and the cached values ​​are directly used in subsequent generation while still having an acceleration effect. The generation results are consistent with the inference results without removing the tokens.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual token removal method for accelerating inference in video understanding models, characterized in that: The method is based on the phenomenon of visual attention contraction, that is, less important visual information tends to attract less attention during the model reasoning process. By analyzing the attention weight distribution of different layers in the video visual language model during the reasoning process, the importance of visual tokens is dynamically evaluated and redundant visual tokens are gradually removed. The method includes the following steps: a) Video Input Processing: This involves processing the input video... The sampling is divided into several frames, and each frame is divided into multiple independent image blocks to obtain the initial visual features; b) Visual token generation: via visual encoder Further extract visual features and use a projector Mapped to the visual token space, the formula for generating visual tokens is: ； in, Indicates a visual token. This refers to the input video; c) Multimodal input concatenation: This involves concatenating the system token. Visual tokens and text tokens Concatenate the sequences in order to obtain the complete input sequence. The splicing formula is: ； d) Attention Weight Analysis: During the inference pre-filling stage, the attention weights of text tokens to visual tokens are calculated using an attention mechanism, and the importance of each visual token is calculated based on these attention weights. The formula is as follows: ； in, Indicates the first A visual token, Attention Indicates the first The text token pair of the first Attention weights for each visual token Total number of text tokens; e) Visual Token Grouping and Aggregation: Group visual tokens by time and space dimensions, and calculate the importance of each group using an aggregation formula: ； in, Represents a visual token group. This represents the attention given to a single visual token within a group; f) Dynamic token removal: Setting the starting layer and end layer Based on the importance assessment results, visual token groups with lower attention weight values ​​after weighted summation are gradually removed; g) Inference optimization: After removing redundant visual tokens, the remaining visual tokens are combined with text tokens, and multiple dynamic visual token removals can be performed to finally complete the inference calculation, thereby improving inference efficiency.

2. The visual token removal method for accelerating inference in video understanding models according to claim 1, characterized in that: The attention weight analysis is based on the multi-layer attention weight in the inference pre-filling stage, tracks the trend of attention change of text tokens to visual tokens, identifies visual tokens whose attention is continuously decreasing, and performs visual token removal operations.

3. The visual token removal method for accelerating inference in video understanding models according to claim 2, characterized in that: The visual token grouping includes time-dimension grouping and spatial-dimension grouping, and the attention weights after grouping are aggregated and calculated through time and spatial dimensions respectively.

4. The visual token removal method for accelerating inference in video understanding models according to claim 3, characterized in that: The visual token removal operation is a progressive removal, where each removal operation processes a limited number of visual tokens. Visual tokens, specific proportions It can be dynamically adjusted according to task requirements.

5. The visual token removal method for accelerating inference in video understanding models according to claim 4, characterized in that: The conditions for executing the visual token removal operation include: a) Triggered when no token removal occurs across multiple consecutive multi-level blocks; b) Triggered when more than 30% of the token group attention value continues to decline.

6. The visual token removal method for accelerating inference in video understanding models according to claim 4, characterized in that: The token removal operation in the inference pre-filling phase has a start layer and an end layer, with the start layer set to the first... The end layer is set to the first layer. The initial layer is designed to prevent the premature loss of key visual information, while the final layer is designed to ensure stable model output.

7. The visual token removal method for accelerating inference in video understanding models according to claim 1, characterized in that: The input video sample is divided into several frames, and after being divided equally, it is sampled. Each frame image is further divided into image blocks of a fixed size. Each image block is used as a visual token, and the visual token contains information in the temporal and spatial dimensions.

8. The visual token removal method for accelerating inference in video understanding models according to claim 1, characterized in that: The method is compatible with existing KV caching mechanisms, and token removal only occurs during the inference pre-filling phase.

9. The visual token removal method for accelerating inference in video understanding models according to claim 1, characterized in that: The visual token removal operation is based on maintaining the spatiotemporal structural integrity of the video sequence. The removed tokens will not disrupt the temporal continuity and spatial distribution characteristics between video frames.

10. A visual token removal method for accelerating inference in video understanding models according to claim 1, characterized in that: The method is applicable to video understanding tasks such as video content analysis, action recognition, and video summarization extraction, and can significantly reduce inference time and memory usage while maintaining output accuracy.

Images