Visual token removal method for video understanding model reasoning acceleration

Abstract

The invention provides a visual token removal method for video understanding model reasoning acceleration, which is based on a visual attention contraction phenomenon, dynamically evaluates the importance of visual tokens and screens and removes redundant tokens by analyzing the distribution and change of attention weights of the visual tokens in a reasoning pre-filling stage. Therefore, calculation overhead is reduced and reasoning efficiency is improved. The method comprises the following steps: carrying out frame sampling and region division on video input to obtain image blocks with space-time structures; the method comprises the following steps: extracting visual features by using a visual encoder and a projector and generating a visual token; splicing the visual token with the system token and the text token to form multi-modal input; tracking the attention change trend of the text to the vision to evaluate the importance of the visual token; the recognition of unimportant visual tokens is accelerated through a grouping and aggregation mode, and the unimportant visual tokens are gradually removed in reasoning. While the output quality is maintained, the method significantly reduces the reasoning time and memory occupation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a visual token removal method for accelerating video understanding model reasoning. Background Art

[0002] With the rapid development of artificial intelligence technology, visual language models have become a hot research topic at the intersection of computer vision and natural language processing. These models achieve the understanding and generation of multimodal inputs by combining visual encoders and language models, and show strong performance in image-text tasks and video-text tasks. Especially in video understanding tasks, VLMs have been widely used in video understanding tasks such as action recognition, video summary generation, and video content analysis. However, compared with image input, in order to fully obtain more visual information in video input, existing technologies have exposed many deficiencies in processing efficiency and computing resource requirements:

[0003] In video understanding tasks, VLMs need to sample the video and divide each frame into multiple independent image patches, which are converted into visual tokens and input into the model. As the number of video frames increases, the computational complexity is quadratic to the number of visual tokens, resulting in a significant increase in computational overhead, especially in long video tasks, where computational and storage requirements face bottlenecks.

[0004] In order to solve the problem of too many visual tokens, some optimization methods have been proposed, but they all have obvious limitations:

[0005] Visual encoder optimization: The number of visual tokens is reduced by adjusting the visual encoder structure, but the impact of different text inputs is not considered, and additional training is usually required on the model used, which increases development costs and makes it difficult to be compatible with other existing models.

[0006] Token merging strategy: merge visual tokens into more compact representations to reduce computational complexity. However, this approach currently relies heavily on specific model architectures, which limits its versatility and may cause loss of key information due to over-compression.

[0007] Directly discarding tokens: Some methods directly remove some tokens randomly or based on simple rules, but fail to make full use of the attention mechanism to dynamically evaluate the importance of tokens, resulting in an uncontrollable decline in model performance.

[0008] The attention mechanism and its corresponding structure are important components of VLMs, but existing methods usually fail to fully utilize the attention mechanism to evaluate the importance of visual tokens, resulting in a large number of tokens containing redundant information being involved in the calculation, increasing the inference time and memory consumption of the model.

[0009] In the long video understanding task, the video sequence contains a lot of redundant information (such as the content of adjacent frames has little change, the content of background image blocks is highly similar, etc.). The existing technology fails to effectively identify and remove this redundant information, resulting in inference efficiency that is difficult to meet the needs of actual applications. At the same time, some optimization methods may damage the output quality of the model due to excessive information discarding.

[0010] In view of the above shortcomings, the present invention proposes a visual token removal method based on the phenomenon of visual attention contraction, which significantly improves the reasoning efficiency by dynamically screening unimportant visual tokens while ensuring the stability and versatility of the model performance. Summary of the invention

[0011] The purpose of the present invention is to solve the technical problems raised in the above-mentioned background technology and to provide a visual token removal method for accelerating video understanding model reasoning.

[0012] The scheme of the present invention to achieve the above object is:

[0013] A visual token removal method for accelerating the reasoning of a video understanding model, the method is based on the phenomenon of visual attention contraction, analyzes the attention weight distribution of different levels of a video visual language model during the reasoning process, dynamically evaluates the importance of visual tokens, and gradually removes redundant visual tokens; the method comprises the following steps:

[0014] a) Video input processing: The input video V is sampled into several frames, each frame is divided into multiple independent image blocks, and the initial visual features are obtained;

[0015] b) Visual token generation: through the visual encoder f ve Further extract visual features and pass them through the projector f proj Mapped to the visual token space, the generation formula of the visual token is:

[0016] X v =f proj (f ve (V));

[0017] Among them, X v represents the visual token, V represents the input video;

[0018] c) Multimodal input concatenation: transform the system token X s 、Visual TokenX v and the text token X t Connect them in order to get the final input X. The connection formula is:

[0019] X=Concat(X s , X v , X t );

[0020] d) Attention weight analysis: In the inference pre-filling stage, the attention mechanism is used to calculate the attention weight of the text token to the visual token, and the importance of each visual token is calculated based on the attention weight. The formula is:

[0021]

[0022] Among them, X v，i represents the i-th visual token, Attention j，i represents the attention weight of the jth text token to the ith visual token, and T is the number of text tokens;

[0023] e) Visual token grouping and aggregation: Visual tokens are grouped by time and space dimensions, and the importance of the groups is calculated using the aggregation formula:

[0024]

[0025] Among them, G represents the visual token group, Importance(X v，i ) represents the attention of a single visual token within a group;

[0026] f) Dynamic token removal: Set the starting layer L start and the end layer L end , and according to the importance evaluation results, gradually remove the visual token groups with lower values ​​after weighted sum of attention weights;

[0027] g) Reasoning optimization: After removing redundant visual tokens, the retained visual tokens are combined with text tokens to complete the reasoning calculation, thereby improving the reasoning efficiency.

[0028] On the other hand: The attention weight analysis is based on the multi-layer attention weights in the inference pre-filling stage, tracks the attention change trend of text tokens to visual tokens, identifies visual tokens whose attention continues to decline and removes them.

[0029] On the other hand: the visual token grouping includes time dimension grouping and space dimension grouping, and the attention weights after grouping are aggregated and calculated by time and space dimensions respectively.

[0030] On the other hand: the visual token removal operation is a progressive removal, each time removing r% of the current visual tokens, and the specific ratio r can be dynamically adjusted according to task requirements.

[0031] On the other hand: the execution conditions of the visual token removal operation include:

[0032] a) Triggered when no token removal is performed in consecutive multiple layers;

[0033] b) Triggered when a certain percentage of the visual token group's attention level drops continuously.

[0034] On the other hand: the token removal operation in the inference pre-filling phase is set with a start layer and an end layer, and the start layer is set to the Lth layer. start layer, the end layer is set to L end The setting of the starting layer ensures that key visual information is not lost prematurely, and the design of the ending layer ensures the stable output of the model.

[0035] On the other hand: the video frame is divided into image blocks of fixed size, each image block serves as a visual token, containing time dimension and space dimension information.

[0036] On the other hand: the method is compatible with the existing KV cache mechanism, token removal only occurs in the inference pre-filling phase, and the retained cache can be reused in subsequent generation processes.

[0037] On the other hand: the removal operation is based on the premise of maintaining the integrity of the spatiotemporal structure of the video sequence, and the removed tokens will not destroy the temporal continuity and spatial distribution characteristics between video frames.

[0038] On the other hand: the method is suitable for tasks such as video content analysis, action recognition, video summary extraction, etc., and can significantly reduce inference time and memory usage while maintaining output accuracy.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention significantly reduces the model computational overhead by using the attention mechanism to dynamically evaluate the importance of visual tokens in the inference pre-filling stage and gradually removing unimportant visual tokens. This method can balance the inference quality and efficiency, removing visual tokens to reduce the inference time by 40%-70%, while reducing the memory required for visual tokens in inference by more than 50%, meeting the demand for efficient processing in practical applications.

[0041] The dynamic screening method based on attention weights can accurately identify key information and avoid discarding too many important visual tokens, thereby reducing the computational complexity while maintaining the accuracy and stability of the model output. Compared with existing methods, the present invention can achieve both performance and efficiency improvements in video understanding tasks such as action recognition, video content analysis, and summary extraction.

[0042] This method does not require major adjustments to the visual encoder or model structure, and can be plug-and-play applied to a variety of existing video visual language models. At the same time, by flexibly setting the start and end layers of token removal and adjusting the dynamic removal ratio, the method is applicable to tasks of different difficulty levels and has good versatility and scalability.

[0043] The present invention performs token screening in the inference pre-filling stage to avoid interfering with the KV cache mechanism, thereby retaining the efficient reuse capability of the KV cache. This design not only improves the model inference speed, but also reduces the complexity of practical application and deployment.

[0044] The present invention is suitable for tasks such as video content analysis, action recognition, and video summary extraction. It demonstrates significant efficiency and performance advantages in a variety of video understanding scenarios, and provides an efficient and reliable technical solution for large-scale video data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 It is a flowchart of a visual token removal method for accelerating video understanding model reasoning in an embodiment of the present invention.

[0047] Figure 2 It is a schematic diagram of the internal flow of a model of a visual token removal method for accelerating video understanding model reasoning in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings and multiple embodiments.

[0050] Embodiment 1:

[0051] Video input processing: The input video V is sampled into several frames. For example, for an 80-second video clip, the model is set to sample 16 frames, and the sampling is evenly divided, that is, one frame every 5 seconds. Each frame image is further divided into image blocks (patches) of fixed size, such as a single frame image is divided into 12×12=144 image blocks. Each image block is used as a visual token, totaling 16×144=2304 visual tokens. The mathematical formula for generating visual tokens is:

[0052] Xv =f proj (f ve (V));

[0053] Among them, f ve represents the visual encoder, which is responsible for extracting image block features; f proj A representation projector that maps visual features into a space that is aligned with the language model.

[0054] Multimodal input concatenation: system token X s The default setting for each model, with the text token X t Together with the language model word segmenter, it is generated and combined with the visual token X v Put them together to get the complete input sequence:

[0055] X=Concat(X s ,X v ,X t );

[0056] Attention weight analysis: In the pre-filling phase of inference, the attention weight of the textual token to the visual token is calculated using the attention mechanism. For the i-th visual token, its importance is defined as:

[0057]

[0058] Among them, Attention j，i represents the attention weight of the j-th text token to the i-th visual token, and T is the total number of text tokens.

[0059] Visual token grouping and aggregation: Visual tokens are grouped by time and space. For example, visual tokens of the same frame in the time dimension are considered as a group, with an initial total of 16 groups. Visual tokens of image blocks at the same position in the spatial dimension are grouped, corresponding to various regions in the video, with an initial total of 144 groups. Each visual token is in both a time group and a space group. The importance of each group of tokens is aggregated using the following formula:

[0060]

[0061] Dynamic token removal: Set the starting layer L according to the distribution of attention weights start and the end layer L end, such as setting them to the 2nd layer and the 4th layer from the end. During the model inference process, when the number of layers is within this range, we will record the attention values ​​of the most recent layers of each token group, that is, the corresponding recent attention change trend, such as the most recent 3 layers. The attention weights of each group in the most recent three layers are weighted summed, and the visual token groups with relatively low attention weights are removed. The removal strategy is to remove the current r% of visual tokens each time, and the ratio r can be dynamically adjusted according to the task type and difficulty.

[0062] Reasoning optimization: By removing redundant visual tokens, reducing computing overhead and optimizing reasoning efficiency, the retained visual tokens are combined with text tokens to perform multiple dynamic visual token removals to complete the reasoning calculation. Since token removal is only performed in the pre-filling stage of model reasoning, it is compatible with the currently commonly used KV cache acceleration mechanism. The KV cache value is obtained in the pre-filling stage and can be reused in the stage of generating subsequent content.

[0063] Embodiment 2:

[0064] In a video comprehension multiple choice question answering task, the input is a 300-second video clip, and the model setting used samples a total of 32 frames. The implementation process is as follows: Each input frame image is preprocessed and each frame is divided into 12×12=144 image blocks, a total of 32×144=4608, and the visual tokens are calculated by formula X v =f proj (f ve (V))Encoding is generated.

[0065] Attention analysis and token removal: According to the time dimension and space dimension, each frame and each position in the video is grouped together, and the attention value of the visual token aggregation of each group is calculated. For a 32-layer model, the attention value changes of the most recent layers of each token group are recorded and tracked during the inference stage from the 2nd layer to the 28th layer, and the weighted sum is performed. The token removal operation is performed every three layers or when the attention value of more than 30% of the token groups decreases continuously. According to the attention weight value obtained by the weighting of the current layer, the 30% current visual tokens with lower attention values ​​than the recently obtained text tokens are removed in units of token groups. Multiple dynamic visual token removals will be performed throughout the inference process.

[0066] Inference results: After gradually removing unimportant visual tokens, the total inference time is reduced by more than 50%, the memory usage of the corresponding visual tokens is about 45% of the original, and the performance of the model on the corresponding task, that is, the accuracy, can still be maintained at more than 99% of the original.

[0067] Embodiment 3:

[0068] In a video content classification task, the input is a 60-second video clip, and 8 frames are sampled. Each frame is divided into 144 image blocks per frame, and a total of 8×144=1152 visual tokens are generated.

[0069] Concatenate the generated visual tokens X v With system token X s and the text token X t , forming the input sequence X

[0070] Dynamic token filtering: Visual tokens are grouped accordingly in the time dimension and the space dimension, i.e., 8 time groups and 144 space groups.

[0071] The removal ratio is set to 50%, and the attention weights are aggregated and weighted to determine when to perform dynamic visual token removal in layers 2-24 of the model. For example, for a specific sample, it is determined to be removed in the time dimension at layers 5 and 13, and in the space dimension at layers 9, 15, 18, and 22. The number of time groups is reduced to 4 and 2, respectively, and the number of space groups is reduced to 72, 36, 18, and 9, respectively. The number of visual tokens gradually decreases during the reasoning process, and only 2×9=18 visual tokens are retained in the final stage.

[0072] Inference performance: After the token is removed, the inference speed is increased by more than 120%, and the accuracy rate drops within 0.4%.

[0073] Embodiment 4:

[0074] In the video content annotation task, the input is a 10-minute video. The initial visual information is obtained by sampling and segmentation, followed by encoding and projection processing, and a total of 8×196=1568 visual tokens are obtained. The corresponding visual tokens are grouped according to the spatiotemporal dimension.

[0075] Attention screening and removal: Aggregate the attention weights of each group, and if more than 30% of the frames have a continuously decreasing attention weight, it is determined that tokens should be removed. The top 30% of the time groups, i.e. frames, ranked by importance are screened out, and the entire group of visual tokens is removed, while the visual tokens of the remaining groups are retained.

[0076] During the subsequent model inference propagation process, attention tracking and visual token removal are continuously performed.

[0077] Annotation generation: The annotations generated by the optimized output cover the main content, the reasoning time is shortened by about 45%, and the overall evaluation score (out of 5 points) is reduced by only 0.1 points.

[0078] Embodiment 5:

[0079] This method performs token screening and removal in the inference pre-filling stage, and the retained visual tokens and text tokens are used to optimize the efficiency of the subsequent generation stage through the KV cache mechanism.

[0080] Cache optimization: During the model inference pre-filling phase, the input sequence of each layer, that is, the key and value of each token, are stored. The key and value can be used to calculate the attention weight, which is used to dynamically calculate the importance of the token.

[0081] When it is determined that visual tokens are to be removed, the filtered and removed visual tokens will not enter the reasoning propagation of subsequent layers of the model, and the retained visual tokens are directly concatenated with the system tokens and text tokens again to form the input of the next layer, thereby reducing the storage overhead of the KV cache.

[0082] Inference results: The model pre-filling inference time is reduced by about 50%, the storage overhead of the KV cache of the visual token is reduced by more than 40%, and the cached values ​​are directly used in subsequent generation while still having an acceleration effect. The generation results are consistent with the inference results without removing the tokens.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual token removal method for accelerating video understanding model inference, characterized by: The method is based on the phenomenon of visual attention shrinkage, that is, unimportant visual information in the model reasoning process tends to attract less attention. By analyzing the attention weight distribution of different layers in the video visual language model during the reasoning process, the importance of visual tokens is dynamically evaluated, and redundant visual tokens are gradually removed. The method comprises the following steps: a) Video input processing: The input video V is sampled into several frames, each frame is divided into multiple independent image blocks, and the initial visual features are obtained; b) Visual token generation: through the visual encoder f ve Further extract visual features and pass them through the projector f proj Mapped to the visual token space, the generation formula of the visual token is: X v =f proj (f ve (V)); Among them, X v represents the visual token, V represents the input video; c) Multimodal input concatenation: transform the system token X s 、Visual TokenX v and the text token X t Connect them in order to get the final input X. The connection formula is: X=Concat(X s ,X v ,X t ); d) Attention weight analysis: In the inference pre-filling stage, the attention mechanism is used to calculate the attention weight of the text token to the visual token, and the importance of each visual token is calculated based on the attention weight. The formula is: Among them, X v，i represents the i-th visual token, Attention j，i represents the attention weight of the jth text token to the ith visual token, and T is the number of text tokens; e) Visual token grouping and aggregation: Visual tokens are grouped by time and space dimensions, and the importance of each group is calculated using the aggregation formula: Among them, G represents the visual token group, Importance(X v,i ) represents the attention of a single visual token within a group; f) Dynamic token removal: Set the starting layer L start and the end layer L end , and according to the importance evaluation results, gradually remove the visual token groups with lower values ​​after weighted sum of attention weights; g) Reasoning optimization: After removing redundant visual tokens, the retained visual tokens are combined with text tokens to complete the reasoning calculation, thereby improving the reasoning efficiency.

2. A visual token removal method for accelerating video understanding model reasoning according to claim 1, characterized in that: The attention weight analysis is based on the multi-layer attention weights in the inference pre-filling stage, tracks the changing trend of the attention of text tokens to visual tokens, identifies the visual tokens whose attention continues to decline and removes them.

3. The visual token removal method for accelerating video understanding model reasoning according to claim 1, characterized in that: The visual token grouping includes time dimension grouping and space dimension grouping, and the attention weights after grouping are respectively aggregated and calculated through the time and space dimensions.

4. The visual token removal method for accelerating video understanding model reasoning according to claim 1, characterized in that: The visual token removal operation is a progressive removal, where r% of the current visual tokens are removed each time, and the specific ratio r can be dynamically adjusted according to task requirements.

5. The visual token removal method for accelerating video understanding model reasoning according to claim 1, characterized in that: The execution conditions of the visual token removal operation include: a) Triggered when no token removal is performed in consecutive multiple layers; b) Triggered when a certain percentage of the visual token group's attention level drops continuously.

6. The visual token removal method for accelerating video understanding model reasoning according to claim 1, characterized in that: The token removal operation in the inference pre-filling phase is set with a start layer and an end layer, and the start layer is set to the Lth layer. start layer, the end layer is set to L end The setting of the starting layer ensures that key visual information is not lost prematurely, and the design of the ending layer ensures the stable output of the model.

7. The visual token removal method for accelerating video understanding model reasoning according to claim 1, characterized in that: The video frame is divided into image blocks of fixed size, and each image block serves as a visual token, including time dimension and space dimension information.

8. The visual token removal method for accelerating video understanding model reasoning according to claim 1, characterized in that: The method is compatible with the existing KV cache mechanism, token removal only occurs in the inference pre-filling phase, and the retained cache can be reused in subsequent generation processes.

9. The visual token removal method for accelerating video understanding model reasoning according to claim 1, characterized in that: The removal operation is based on the premise of maintaining the temporal and spatial structural integrity of the video sequence, and the removed tokens will not destroy the temporal continuity and spatial distribution characteristics between video frames.

10. The visual token removal method for accelerating video understanding model reasoning according to claim 1, characterized in that: The method is applicable to video understanding tasks such as video content analysis, action recognition, and video summary extraction, and can significantly reduce inference time and memory usage while maintaining output accuracy.

Images