An efficient video understanding method based on low-rank key tensor residual decomposition

Abstract

This invention discloses an efficient video understanding method based on low-rank key tensor residual decomposition, comprising: acquiring a video sequence, inputting it into a visual encoder of a multimodal large language model for block processing to obtain a block sequence, each block sequence containing multiple tokens; this efficient video understanding method based on low-rank key tensor residual decomposition calculates the key tensor residuals of each of the remaining decoding layers relative to the first decoding layer, and processes the key tensor residuals through low-rank decomposition to obtain low-rank residuals, and stores all low-rank residuals of each of the remaining decoding layers and all key tensors of the first decoding layer into the decoder's cache, which greatly reduces the cache overhead of the decoder compared to the prior art of storing all key tensors of each decoding layer; furthermore, this method compresses the tokens in each block sequence to alleviate memory constraints, and the freed-up cache space can be used to accommodate more video frame inputs, thereby improving the accuracy of video understanding.

Description

Technical Field

[0001] This invention belongs to the field of video understanding technology, specifically relating to an efficient video understanding method based on low-rank bond tensor residual decomposition. Background Technology

[0002] Large Visual Language Models (LVLMs) face significant memory consumption issues when processing long video sequences, especially when generating large amounts of token streams. LVLMs typically encode video frames, Audio Transcription (ASR) text, and Optical Character Recognition (OCR) text as separate token sequences. The intermediate state key tensors of these token sequences are stored in a cache to accelerate decoding output; therefore, the key cache is the main source of memory overhead. Existing optimization methods mainly focus on token sparsity or self-attention sparsity, but they ignore the redundancy of key tensor evolution between layers, making it difficult to scale to longer inputs.

[0003] In existing long context methods, Longformer and BigBird models reduce the complexity by reducing attention from fully connected to sparse connected through predefined sparse attention graphs. However, their connection patterns are mostly static, making it difficult to adaptively model irregular long-range dependencies (cross-shot consistency, long-term object tracking, etc.) driven by content in long videos. They often need to increase the window size or introduce more global / random connections to maintain performance, which leads to significant computational overhead. In existing methods for accelerating long context inference, one approach is to compress key-value pairs (KVs) by reducing the number of tokens that need to be cached. For example, the VisionZip method selects high-information-content dominant tokens on the visual encoder side and merges the remaining tokens based on similarity, reducing the number of visual tokens input to the LLM from the source. The DyCoke method uses cross-frame similarity to merge temporal-dimensional tokens and dynamically prunes redundant visual tokens in the KV during decoding. The PyramidKV method starts from the information convergence rules of inter-layer attention, allocates different KV retention budgets to different layers, and selects retained tokens based on attention scores. The RETAKE method further uses keyframes as pivots, retains pivot frames, and performs attention-guided pruning on the KVs of non-pivot frames. These methods demonstrate the potential of dynamic, attention / similarity-guided token-level KV compression, but they still mainly rely on discrete token selection / pruning / merging strategies, and the compression effect and robustness often depend on specific scoring rules. In contrast, another complementary approach is to perform continuous compression such as low-rank approximation of key / value states to reduce caching overhead without explicitly discarding tokens.

[0004] Furthermore, low-rank structures have been extensively studied in LLMs (Large Language Models) and LVLMs for model compression and efficient fine-tuning, such as inserting trainable low-rank adapters into LoRA models. However, these methods primarily focus on weight compression rather than runtime memory optimization. In the visual language domain, joint embeddings often reside in low-rank subspaces, but existing methods do not leverage the low-rank structures of key evolution in LVLMs for context compression. Summary of the Invention

[0005] The purpose of this invention is to address the problems mentioned in the background art by proposing an efficient video understanding method based on low-rank bond tensor residual decomposition.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] This invention proposes an efficient video understanding method based on low-rank bond tensor residual decomposition, comprising:

[0008] The video sequence is acquired and input into the visual encoder of a multimodal large language model for block processing to obtain... A sequence of blocks, each containing multiple tokens;

[0009] Each block of the sequence is positionally encoded and then decoded by a multimodal large language model to obtain the text for understanding the video sequence;

[0010] The decoder includes Each block sequence is compressed based on the attention score of the multi-head self-attention layer in the first decoding layer, resulting in a compressed block sequence.

[0011] For each compressed block sequence, the key tensor residuals of the remaining decoding layers relative to the first decoding layer are calculated, and the key tensor residuals are processed by low-rank decomposition to obtain low-rank residuals. All low-rank residuals of the remaining decoding layers and all key tensors of the first decoding layer are stored in the decoder's cache to reduce the cache overhead of the decoder.

[0012] For each compressed block sequence, the key tensors of the remaining decoding layers are reconstructed based on the key tensors of the first decoding layer and the low-rank residuals of the remaining decoding layers, and the remaining decoding layers use the reconstructed key tensors to calculate attention scores.

[0013] Preferably, the multimodal large language model is Qwen2-VL.

[0014] Preferably, the block processing procedure includes:

[0015] The video sequence was segmented into Each block is then converted by a visual encoder. vectors, i.e. A block sequence.

[0016] Preferably, each decoding layer includes a multi-head self-attention layer, a first normalization layer, a feedforward neural network, and a second normalization layer connected in sequence. The input of the multi-head self-attention layer is added to the output of the multi-head self-attention layer and used as the input of the first normalization layer. The input of the feedforward neural network is added to the output of the feedforward neural network and used as the input of the second normalization layer.

[0017] In this process, the multi-head self-attention layer performs a linear projection on each token in each block sequence after position encoding to obtain a query vector, a key vector, and a value vector. The query vector, key vector, and value vector corresponding to all tokens in the same block sequence constitute the query tensor, key tensor, and value tensor, respectively.

[0018] Preferably, each block sequence contains One token;

[0019] When compressing each block sequence, the first decoding layer uses the query vector and key vector of each token in the block sequence to perform a masked scaling dot product attention score calculation, and retains the previous... The token with the highest attention score is used as the compressed block sequence.

[0020] Preferably, the formula for calculating the key tensor residual is as follows:

[0021] ;

[0022] in, For the first The first layer of the decoding layer Key tensor residuals of a block sequence For the first The first layer of the decoding layer The key tensor of a block sequence For the first decoding layer The key tensor of a block sequence, and ;

[0023] By processing the bond tensor residuals through low-rank decomposition, low-rank residuals are obtained. Furthermore, during low-rank decomposition, the first... The largest singular values ​​are used to form a singular value vector, which can be expressed by the low-rank decomposition formula as follows:

[0024] ;

[0025] in, For the first The first layer of the decoding layer The low-rank residuals corresponding to the key tensor residuals of each block sequence. For the first The first layer of the decoding layer The product of the left singular tensor and the singular value vector obtained by low-rank decomposition of the key tensor residuals of each block sequence. For the first The first layer of the decoding layer The right singular tensor is obtained by performing low-rank decomposition on the key tensor residuals of each block sequence. This is a transpose.

[0026] Preferably, the cache includes layer caches corresponding to each decoding layer. For the first decoding layer, the layer cache stores key tensors, and for the remaining decoding layers, the layer cache stores low-rank residuals.

[0027] Preferably, the calculation formula for the key tensor reconstruction is as follows:

[0028] ;

[0029] in, For the first The first layer of the decoding layer The key tensor after reconstructing a block sequence.

[0030] Preferably, the low-rank decomposition reduces storage complexity from... Down to .

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] This efficient video understanding method based on low-rank key tensor residual decomposition calculates the key tensor residuals of each of the remaining decoding layers relative to the first decoding layer, and processes these residuals using low-rank decomposition to obtain low-rank residuals. All low-rank residuals from the remaining decoding layers and all key tensors from the first decoding layer are then stored in the decoder's cache. Compared to existing technologies that store all key tensors from each decoding layer, this significantly reduces the decoder's cache overhead. Furthermore, this method compresses tokens in each block sequence, alleviating memory usage and freeing up cache space to accommodate more video frame inputs, thereby improving the accuracy of video understanding.

[0033] This method does not require fine-tuning, distillation, or any form of retraining of the model. It can be directly applied to the inference stage of any pre-trained visual language model. At the same time, it does not modify the model architecture or weights, but only adjusts the storage method of the cache during inference. Therefore, it can be seamlessly integrated into existing inference pipelines. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the efficient video understanding method based on low-rank bond tensor residual decomposition of the present invention.

[0035] Figure 2 This is a schematic diagram comparing the accuracy of the method of the present invention with that of the existing model Retake in understanding videos under different context lengths;

[0036] Figure 3 This is a schematic diagram comparing the decoder cache occupancy rate of the existing model Retake under different context lengths in the method of the present invention;

[0037] Figure 4 This is a schematic diagram illustrating the selection of the rank size of the existing model Qwen2-VL when operating on different key tensors in low-rank decomposition as the number of decoder layers changes;

[0038] Figure 5 This is a schematic diagram showing the distribution of attention scores when the rank is 5 in this invention;

[0039] Figure 6 This is a schematic diagram showing the distribution of attention scores when the rank is 30 in this invention;

[0040] Figure 7 This is a schematic diagram showing the distribution of attention scores when the rank is 125 in this invention.

[0041] Figure 8 This is a diagram showing the change in accuracy of the method for video understanding as the rank changes, compared with the accuracy changes of existing models Qwen2-VL and Retake, as well as the changes in cache overhead, reconstruction error, and the recommended rank range.

[0042] Figure 9 This diagram illustrates the comparison of cache usage for different models under different schemes. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0045] In one embodiment, such as Figures 1-9 As shown, an efficient video understanding method based on low-rank key tensor residual decomposition is provided, including:

[0046] Step 1: Obtain the video sequence and input it into the visual encoder of the multimodal large language model for block processing to obtain... A sequence of blocks, each containing multiple tokens;

[0047] The block processing process includes:

[0048] The video sequence was segmented into Each block is then converted by a visual encoder. vectors, i.e. A block sequence, and The block sequence is represented as ... Each block sequence contains Tokens (e.g., 16).

[0049] This method addresses long videos, which often generate a large number of visual tokens, leading to a linear increase in memory overhead with the length of the prompts and a high risk of Out-of-Memory (OOM) errors. This memory bottleneck stems from traditional pre-padding mechanisms that process the entire sequence at once, failing to effectively manage peak memory demands. Especially in video understanding tasks, taking the Qwen2-VL model as an example, assuming a text prompt has only 200 tokens, a video (image resolution 480*640) generates 416 visual tokens per frame, resulting in 124,800 visual tokens over 300 frames. Key caching stores tokens, and the number of visual tokens far exceeds that of text tokens, further amplifying this challenge. To address these issues, this invention proposes a block-based processing approach.

[0050] Step 2: Each block of the sequence is positionally encoded and then decoded by a multimodal large language model to obtain the text for understanding the video sequence;

[0051] It should be noted that the multimodal large language model is Qwen2-VL, but other models can also be used, such as LLaVA-OneVision model, LLaVA-v1.6-mistral model, etc.; in this embodiment, the visual encoder is the visual encoder in the Qwen2-VL model, and the decoder is QwenLMDecoder in the Qwen2-VL model.

[0052] Step 2.1, the decoder contains Each block sequence is compressed based on the attention score of the multi-head self-attention layer in the first decoding layer, resulting in a compressed block sequence.

[0053] It should be noted that each decoding layer includes a multi-head self-attention layer, a first normalization layer, a feedforward neural network, and a second normalization layer connected in sequence. The input of the multi-head self-attention layer is added to the output of the multi-head self-attention layer and used as the input of the first normalization layer. The input of the feedforward neural network is added to the output of the feedforward neural network and used as the input of the second normalization layer.

[0054] The multi-head self-attention layer performs a linear projection on each token in each block sequence after position encoding to obtain the query vector. Key vector Sum value vector Furthermore, the query vectors, key vectors, and value vectors corresponding to all tokens in the same block sequence constitute the query tensor. Key tensors Sum tensor .

[0055] When compressing each block sequence, the first decoding layer uses the query vector and key vector of each token in the block sequence to perform a masked scaling dot product attention score calculation (the specific formula is: +mask), For query vector, For key vectors, (for the dimension of a single token), and retain the previous... The token with the highest attention score is used as the compressed block sequence (the compressed block sequence is used for subsequent forward inference in the decoder), where the compression ratio of the tokens is expressed as: .

[0056] Step 2.2: For each compressed block sequence, calculate the key tensor residuals of the remaining decoding layers relative to the first decoding layer. (Since each compressed block sequence needs to undergo linear projection operations in the multi-head self-attention layer when passing through each decoding layer, each decoding layer will have key tensors for each block sequence. Subtract the key tensors of the remaining decoding layers from the corresponding key tensors of the first decoding layer to obtain the key tensor residuals.) Process the key tensor residuals through low-rank decomposition to obtain low-rank residuals. Store all low-rank residuals of the remaining decoding layers and all key tensors of the first decoding layer into the decoder cache to reduce the cache overhead of the decoder (where the original storage method in the prior art is to store the key tensors of each decoding layer in full). In this embodiment, the remaining decoding layers are the decoding layers other than the first decoding layer.

[0057] The formula for calculating the key tensor residual is as follows:

[0058] ;

[0059] in, For the first The first layer of the decoding layer Key tensor residuals of a block sequence For the first The first layer of the decoding layer The key tensor of a block sequence For the first decoding layer The key tensor of a block sequence, and ;

[0060] Low-rank residuals are obtained by processing the bond tensor residuals through low-rank decomposition (SVD decomposition). Furthermore, during low-rank decomposition, the first few rank residuals are retained. The largest singular value (of which (referred to as rank), forming a singular value vector, the low-rank decomposition formula is as follows:

[0061] ;

[0062] in, For the first The first layer of the decoding layer The low-rank residuals corresponding to the key tensor residuals of each block sequence. For the first The first layer of the decoding layer The product of the left singular tensor and the singular value vector obtained by low-rank decomposition of the key tensor residuals of each block sequence. For the first The first layer of the decoding layer The right singular tensor is obtained by performing low-rank decomposition on the key tensor residuals of each block sequence. This is a transpose.

[0063] The cache contains layer caches corresponding to each decoding layer. For the first decoding layer, the layer cache stores the key tensor; for the remaining decoding layers, the layer cache stores the low-rank residuals, as expressed by the following formula:

[0064] ;

[0065] , ;

[0066] in, This is the layer buffer for the first decoding layer. For the first Layer buffer of the layer decoding layer;

[0067] Low-rank decomposition reduces storage complexity from Down to Compression gain ,in The compression gain of this method is approximately proportional to the number of decoding layers in the model. As the model grows to a larger scale (more decoding layers, more attention heads), the compression gain will also increase accordingly. It has good foresight in the face of the continuous growth of model size, and is especially suitable for long video inference deployment in resource-constrained scenarios.

[0068] It should also be noted that only the first few singular values ​​are retained. The largest singular value, such as the first r=60 singular vectors out of 128 singular vectors (or less than half of the singular vectors), can capture more than 90% of the total singular value energy (measured by the cumulative singular value ratio). This indicates that the interlayer evolution of the bond tensor is highly redundant, rather than a random high-rank change.

[0069] Step 2.3: For each compressed block sequence, reconstruct the key tensors of the remaining decoding layers based on the key tensor of the first decoding layer and the low-rank residuals of the remaining decoding layers. The remaining decoding layers then use the reconstructed key tensors to calculate attention scores. After forward inference through each decoding layer, the text understood from the video sequence is obtained (that is, the reconstructed key tensor is applied to the decoder to obtain a new decoder, and the new decoder is used to perform forward inference on the compressed block sequence to obtain the text understood from the video sequence).

[0070] The formula for calculating the key tensor reconstruction is as follows:

[0071] ;

[0072] in, For the first The first layer of the decoding layer The key tensor after reconstructing a block sequence.

[0073] In another embodiment, the key tensor residual reduces memory cost by truncating singular values ​​and discarding tail singular values ​​(except for the first r retained singular values). However, this truncation inevitably introduces reconstruction error. , For the first The first layer of the decoding layer To demonstrate that the reconstruction error of the key tensor in the block sequence is controllable, this implementation includes the following:

[0074] The propagation of reconstruction error in attention score calculation: For simplicity, the mask in attention score calculation is omitted, and the key tensors of each decoding layer and each block sequence are unified using... This indicates that the reconstructed key tensors are uniformly represented using... Representation, and reconstruction error using This indicates that the original attention scores for each block of the sequence (i.e., before the reconstruction of the key tensor) are calculated as follows: The attention scores for each block of the sequence after reconstructing the key tensor are calculated as follows: If the error in calculating the attention score is..., then the error in calculating the attention score is... ),because The function is derived from the logarithm (logits). The norm to the ℓ1 norm on the probability simplex is 1-Lipschitz continuous, that is, for any , have The attention error under the L1 norm is then quantized. Further using the Cauchy-Schwarz inequality, the bounds... Disturbance: ,therefore ,because That is, the largest singular value discarded. To query the L2 norm of a tensor, which is determined by the model's input, Since the scaling factor is constant, the reconstruction error is controllable. For the reconstruction error of the first The reconstruction error value corresponding to each token, and , To obtain to The maximum value in.

[0075] Unlike methods that rely on heuristic rules (such as attention score thresholds and token importance ranking), the truncation error of this method has a strict theoretical bound: the degree of shift in the attention distribution is directly constrained by the discarded singular values, so the choice of rank has a theoretical basis rather than relying on empirical parameter tuning.

[0076] To demonstrate the effectiveness of this method, the following experiment was conducted:

[0077] The experiments for this method are based on three datasets: VideoMME, MLVU, and LVBench. The VideoMME dataset is referenced in: C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang et al., “Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal LLMS in video analysis,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 24 108–24118. The MLVU dataset is referenced in: J. Zhou, Y. Shu, B. Zhao, B. Wu, Z. Liang, S. Xiao, M. Qin, X. Yang, Y. Xiong, B. Zhang et al., “Mlvu: Benchmarking multi-task long video understanding,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025. pp. 13 691–13 701. Reference for the source of the LVBench dataset: W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, X. Gu, S. Huang, B. Xu, Y. Dong et al., “Lvbench: An extreme long video understanding benchmark,” arXiv preprint arXiv:2406.08035, 2024.The VideoMME dataset contains 900 videos ranging in length from 30 seconds to 60 minutes, with an average duration of 1,024 seconds. It includes 2,700 labeled question-answer pairs covering six tasks, primarily focusing on temporal understanding and long-distance dependency modeling. The MLVU dataset contains 2,593 videos ranging in length from 3 minutes to 2 hours, evaluating model performance on nine tasks (such as topic classification and anomaly detection), aiming to examine multi-task understanding and cross-task generalization capabilities. The LVBench dataset contains 103 videos, with an average duration of 16.8 minutes and a maximum duration of 2 hours, including 1,549 complex questions, designed to evaluate long-distance temporal reasoning capabilities.

[0078] Experimental data show that different The impact of size on the understanding performance of this multimodal large language model is shown in Table 1, which represents different sizes of... Comparison of video accuracy on the LVBench dataset across different metrics (ER accuracy, EU accuracy, KIR accuracy, TG accuracy, REA accuracy, and Overall accuracy):

[0079] Table 1

[0080]

[0081] In Table 1, ER stands for Entity Recognition, EU for Event Understanding, KIR for Key Information Retrieval, TG for Temporal Grounding, and REA for Reasoning.

[0082] As shown in Table 1, the performance of the multimodal large language model in this method decreases with... The magnitude initially increases, then tends to stabilize or slightly decrease at larger values, indicating that... The choice of memory efficiency and accuracy needs to be optimized based on the specific task and model context.

[0083] The understanding accuracy of this method on the VideoMME dataset is comparable to existing models (including the GPT-4V model (e.g., from the existing literature A. Hurst, A. Lerer, AP Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o systemcard,” arXiv preprint arXiv:2410.21276, 2024.)) and the Gemini 1.5 Pro model (e.g., from the existing literature G. Team, P. Georgiev, VI Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530.) 2024.), VideoChat2-Mistral model (e.g., from existing literature K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo et al., “Mvbench:A comprehensive multi-modal video understanding benchmark,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp.22 195–22 206.), ShareGPT4Video model (e.g., from existing literature L. Chen, X. Wei, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, Z. Tang, L. Yuan et al.), and ShareGPT4Video model (e.g., from existing literature L. Chen, X. Wei, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, Z. Tang, L. Yuan et al.).The text appears to be a mix of unrelated snippets and fragments, including references to various media outlets and models. A direct translation wouldn't be meaningful without further context. Therefore, a more accurate approach would be to identify the specific models and models used in the original document. (References: "leads to super gpt-4v trustworthiness," in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 19 985–19995.); NVILA model (e.g., from existing literature J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han, "Vila: On pre-training for visual language models," in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2024, pp. 26 689–26 699.); VideoLLaMA 3 model (e.g., from existing literature B. Zhang, K. Li, Z. Cheng, Z. Hu, Y. Yuan, G. Chen, S. Leng, Y. Jiang, H. Zhang, X.).The Retake model (as shown in Table 2) is compared with existing literature such as X. Wang, Q. Si, J. Wu, S. Zhu, L. Cao, and L. Nie, “Retake: Reducing temporal and knowledge redundancy for long video understanding,” arXiv preprint arXiv:2501.13106, 2025.).

[0084] Table 2

[0085]

[0086] As shown in Table 2, this method has the highest accuracy on the VideoMME question-answering dataset, regardless of whether there are subtitles.

[0087] This method utilizes the MLVU dataset and existing models (including the Qwen-VL-Max model (e.g., from P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Geet al., “Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution”), the GPT-4 Turbo model (e.g., from A. Hurst, A. Lerer, APGoucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276,2024.), and the MovieChat model (e.g., from Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun)). Guo, Tian Ye, YantingZhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding. In CVPR, pages 18221–18232. IEEE, 2024.), Movie-LLM model (such as from existing literature Zhende Song, Chenchen Wang, JiamuSheng, Chi Zhang, Gang Yu, Jiayuan Fan, and Tao Chen. Moviellm: Enhancing longvideo understanding with ai-generated movies, 2024.arXiv:2403.01422.), LLaMA-VID model (such as from existing literature Yanwei Li, Chengyao Wang, and Jiaya Jia.Llama-vid: An image is worth 2 tokens in large language models. In ECCV (46), pages 323–340. Springer, 2024.), the Video-LLaMA-2 model (such as from the existing literature K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo et al., “Mvbench: A comprehensive multi-modal video understanding benchmark,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22195–22206.), the VideoChat2 model (such as from the existing literature Z. Cheng, S. Leng, H. Zhang, Y. Xin, X. Li, G. Chen, Y. Zhu, W. Zhang, Z. Luo, D. Zhao et al., “Videollama 2: Advancing spatial temporal modeling and audio understanding in video-llms), the Video-LLaVA model (such as from the existing literature Bin Lin, Yang Ye, Bin Zhu and ‘Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection, 2023. arXiv:2311.10122.), the LongVA model (such as from the existing literature Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu.Long Context Transfer from Language to Vision, 2024. arXiv:2406.16852.), the TimeMarker model (such as from the existing literature TimeMarker-LLM. Timemarker: A versatile video-llm for long and short video understanding with superior temporal localization ability, 2024. Accessed: 2024-11-13.), the Qwen2-VL model (such as from the existing literature Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution, 2024. arXiv:2409.12191.), the Video-XL model (such as from the existing literature Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding, 2024. arXiv:2409.14485.), InternVL-1.Model 5 (such as those from existing literature: Zhe Chen, Weiyun Wang, Hao Tian, ​​Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, HangYan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. CoRR, abs / 2404.16821,2024.) and VILA-1.5 models (such as those from existing literature: Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-context visual language models for long videos, 2024. arXiv:2408.10188.The metrics include: TR (Topic Reasoning) accuracy (assessing whether the model can understand the overall core content of the video or summarize its main topic); AR (Anomaly Recognition) accuracy (identifying abnormal or unexpected events in the video); NQA accuracy; ER (Ego Reasoning) accuracy (inferring the intention or behavior of the videographer for first-person (egocentric) videos); PQA (Plot Question-Answering) accuracy (understanding the plot development, character relationships, or causal logic of the video); AO (Action Order) accuracy (identifying the chronological order of a series of actions in the video); and AC (Action Count) accuracy. Action counting (counting the number of times a specific action (such as "jumping" or "waving") occurs in a video) and AVg accuracy (average accuracy) are compared, as shown in Table 3 (where Holistic in Table 3 represents macro / global understanding; this type of task evaluates the model's ability to grasp the overall content of the video, rather than just focusing on a local segment; it requires the model to be able to summarize the central idea of ​​the video, identify the overall atmosphere, or summarize the theme of a video that lasts for several minutes or even hours. Single represents single-point detail understanding; this type of task focuses on the precise capture of a specific moment or single action in the video; the model needs to observe a frame or short segment in the video in detail and answer questions about specific details. Multi represents multi-point detail understanding; it represents a task that requires combining details from multiple different points in time for reasoning; the model must connect multiple different details at the beginning, middle, and end of the video and perform logical integration across time).

[0088] Table 3

[0089]

[0090] As shown in Table 3, the accuracy of this method using the MLVU dataset on different metrics is higher than that of existing models.

[0091] This method utilizes the LVBenc dataset and existing models (including the GPT-4 Turbo model, the GPT-4o model (such as those from existing literature Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report, 2023. arXiv:2303.08774.), and Gemini 1.5 Pro (such as those from existing literature Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M.).Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs / 2403.05530, 2024.), MovieChat model, LLaMA-VID model, CogVLM2 model (such as from the existing literature Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding, 2024. arXiv:2408.16500.), Qwen2-VL model, InternVL-1.5 model, and Oryx-1.Table 4 compares the five models (such as those from existing literature: Zhe Chen, Weiyun Wang, Hao Tian, ​​Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, JiMa, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, BoZhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, XizhouZhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are weto gpt-4v? closing the gap to commercial multimodal models with opensourcesuites. CoRR, abs / 2404.16821, 2024.) across different metrics (ER, EU, KIR, TG, REA, Overall).

[0092] Table 4

[0093]

[0094] As shown in Table 4, the accuracy of this method on different metrics using the LVBenc dataset is higher than that of existing models.

[0095] like Figure 2 The figure shows a comparison of the accuracy of our method and the existing model (Retake) in video understanding under different context lengths (where the context length is the cumulative number of all tokens in all compressed block sequences), where K represents thousands. Figure 2 The red dashed line Qwen2-VL@256 in the image represents the accuracy of the Qwen2-VL model in understanding the video after sampling 256 frames and using the original storage method (where the original storage method is to store the key tensors of each decoding layer in their entirety), and serves as the first benchmark. Figure 2The +4.2%, +2.3%, and +2.1% values ​​indicate that this method is 4.2%, 2.3%, and 2.1% higher than the first benchmark in terms of video understanding accuracy, respectively.

[0096] like Figure 3 The figure shows a comparison of the decoder cache occupancy rate of our method and existing models (Retake) under different context lengths. Figure 3 The red dashed line Qwen2-VL@256 in the diagram represents the decoder buffer usage of the Qwen2-VL model sampling 256 frames with the original storage method, and serves as a second benchmark. Figure 3 The +78%, +67%, and +63% figures indicate that this method achieves 78%, 67%, and 63% lower decoder cache utilization compared to the second benchmark, respectively. Figure 3 The red dashed line OOM Threshold in the image indicates an error.

[0097] like Figure 4 As shown, the existing model Qwen2-VL processes different key tensors in low-rank decomposition (low-rank decomposition processes the original key tensors (i.e., the original key tensors, i.e., those without reconstructed key tensors) separately, and processes the key tensor residuals of each decoding layer relative to the first decoding layer (i.e., ... Figure 4 The decomposition in the process is based on the key tensor residuals of the first layer and is processed by the key tensor residuals of adjacent decoding layers (i.e., Figure 4 The decomposition in the code is based on the residuals of the bond tensors of adjacent layers, and the rank size is selected as the number of decoder layers changes.

[0098] like Figures 5-7 The figures show the distribution of attention scores for different rank values ​​of 5, 30, and 125.

[0099] like Figure 8 As shown, the accuracy of this method for video understanding changes with the change of rank size, compared with the accuracy changes of existing models Qwen2-VL and Retake, as well as the changes in cache usage, reconstruction error size, and the recommended rank size range (30-40).

[0100] like Figure 9 As shown, different models (including the LLaMA-VID model, Video-LLaMA-2 model, LLaVA-NeXT-Video model, and Qwen2-VL model) are compared using the traditional approach (i.e., full storage of the key tensors of each decoding layer and no block sequence compression as in this method) and the approach based on the traditional approach (i.e., block sequence compression). Figure 9 In block sequence compression, based on the traditional scheme, all low-rank residuals of the remaining decoding layers and all key tensors of the first decoding layer are stored (i.e., Figure 9 Low-rank residual storage in the method, and in this method (i.e. Figure 9 A comparison of cache usage across four schemes (combining block sequence compression with low-rank residual storage).

[0101] This efficient video understanding method based on low-rank key tensor residual decomposition calculates the key tensor residuals of each of the remaining decoding layers relative to the first decoding layer, and processes these residuals using low-rank decomposition to obtain low-rank residuals. All low-rank residuals from the remaining decoding layers and all key tensors from the first decoding layer are then stored in the decoder's cache. Compared to existing techniques that store all key tensors from each decoding layer, this significantly reduces the decoder's cache overhead. Furthermore, this method compresses tokens in each block sequence, alleviating memory usage and freeing up cache space to accommodate more video frame inputs, thereby improving the accuracy of video understanding. This method requires no model fine-tuning, distillation, or any form of retraining and can be directly applied to the inference stage of any pre-trained visual language model. It does not modify the model architecture or weights, only adjusting the cache storage method during inference, thus allowing for seamless integration into existing inference pipelines.

[0102] Existing methods (such as PALU and ReCalKV) employ static weight decomposition, and their compression strategies are independent of the input content, making their performance prone to degradation when the input distribution shifts. Another type of method (such as Eigen Attention) relies on offline calibration datasets to determine the compression strategy, increasing deployment complexity. Our method's low-rank decomposition directly applies to the key tensor generated from the actual input at each inference iteration, and the compression strategy naturally adapts to the input content, requiring no calibration data or offline preprocessing.

[0103] The low-rank residual of this method compresses the "storage redundancy of each token in the depth direction", while the compression of the number of tokens in the block sequence is a compression of the quantity. The two operate on completely different dimensions and can be used independently and superimposed.

[0104] This method can reconstruct the key tensors of the remaining decoding layers by using the key tensors of the first decoding layer in the cache and the low-rank residual time of the remaining decoding layers. This is different from the existing mainstream cache compression methods (such as VisionZip, DyCoke, ReTaKe, etc.) that reduce the cache size by directly discarding, merging or sparsely retaining tokens, thus avoiding the irreversible information loss caused by token discarding.

[0105] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. An efficient video understanding method based on low-rank bond tensor residual decomposition, characterized in that: The method includes: The video sequence is acquired and input into the visual encoder of a multimodal large language model for block processing to obtain... A sequence of blocks, each containing multiple tokens; Each block of the sequence is positionally encoded and then decoded by a multimodal large language model to obtain the text for understanding the video sequence; The decoder includes Each block sequence is compressed based on the attention score of the multi-head self-attention layer in the first decoding layer, resulting in a compressed block sequence. For each compressed block sequence, the key tensor residuals of the remaining decoding layers relative to the first decoding layer are calculated, and the key tensor residuals are processed by low-rank decomposition to obtain low-rank residuals. All low-rank residuals of the remaining decoding layers and all key tensors of the first decoding layer are stored in the decoder's cache to reduce the cache overhead of the decoder. For each compressed block sequence, the key tensors of the remaining decoding layers are reconstructed based on the key tensors of the first decoding layer and the low-rank residuals of the remaining decoding layers, and the remaining decoding layers use the reconstructed key tensors to calculate attention scores.

2. The efficient video understanding method based on low-rank bond tensor residual decomposition as described in claim 1, characterized in that: The multimodal large language model is Qwen2-VL.

3. The efficient video understanding method based on low-rank bond tensor residual decomposition as described in claim 1, characterized in that: The block processing process includes: The video sequence was segmented into Each block is then converted by a visual encoder. vectors, i.e. A block sequence.

4. The efficient video understanding method based on low-rank bond tensor residual decomposition as described in claim 1, characterized in that: Each decoding layer includes a multi-head self-attention layer, a first normalization layer, a feedforward neural network, and a second normalization layer connected in sequence. The input of the multi-head self-attention layer is added to the output of the multi-head self-attention layer and used as the input of the first normalization layer. The input of the feedforward neural network is added to the output of the feedforward neural network and used as the input of the second normalization layer. In this process, the multi-head self-attention layer performs a linear projection on each token in each block sequence after position encoding to obtain a query vector, a key vector, and a value vector. The query vector, key vector, and value vector corresponding to all tokens in the same block sequence constitute the query tensor, key tensor, and value tensor, respectively.

5. The efficient video understanding method based on low-rank bond tensor residual decomposition as described in claim 4, characterized in that: Each block sequence contains One token; When compressing each block sequence, the first decoding layer uses the query vector and key vector of each token in the block sequence to perform a masked scaling dot product attention score calculation, and retains the previous... The token with the highest attention score is used as the compressed block sequence.

6. The efficient video understanding method based on low-rank bond tensor residual decomposition as described in claim 1, characterized in that: The formula for calculating the bond tensor residual is as follows: ； in, For the first The first layer of the decoding layer Key tensor residuals of a block sequence For the first The first layer of the decoding layer The key tensor of a block sequence For the first decoding layer The key tensor of a block sequence, and ; By processing the bond tensor residuals through low-rank decomposition, low-rank residuals are obtained. Furthermore, during low-rank decomposition, the first... The largest singular values ​​are used to form a singular value vector, which can be expressed by the low-rank decomposition formula as follows: ； in, For the first The first layer of the decoding layer The low-rank residuals corresponding to the key tensor residuals of each block sequence. For the first The first layer of the decoding layer The product of the left singular tensor and the singular value vector obtained by low-rank decomposition of the key tensor residuals of each block sequence. For the first The first layer of the decoding layer The right singular tensor is obtained by performing low-rank decomposition on the key tensor residuals of each block sequence. This is a transpose.

7. The efficient video understanding method based on low-rank bond tensor residual decomposition as described in claim 1, characterized in that: The cache includes layer caches corresponding to each decoding layer. For the first decoding layer, the layer cache stores key tensors, and for the remaining decoding layers, the layer cache stores low-rank residuals.

8. The efficient video understanding method based on low-rank bond tensor residual decomposition as described in claim 1, characterized in that: The calculation formula for the key tensor reconstruction is as follows: ； in, For the first The first layer of the decoding layer The key tensor after reconstructing a block sequence.

9. The efficient video understanding method based on low-rank bond tensor residual decomposition as described in claim 1, characterized in that: The low-rank decomposition reduces storage complexity from... Down to .

Images