A method for extending long video understanding capability of a multi-modal large language model through a visual memory mechanism

Abstract

This invention discloses a method for extending the long-video understanding capability of a multimodal large language model through a visual memory mechanism. The method includes: segmenting the long video into consecutive segments and initializing a visual memory bank; iteratively encoding each segment and generating contextual memory and local memory for each segment using a dual-path compression mechanism, where the former is used to convey historical information and the latter is stored in the memory bank; retrieving relevant local memory segments from the memory bank according to textual instructions; and inputting the retrieved segments and instructions into the multimodal large language model to generate an answer. This invention effectively extends the model's long-video understanding capability without increasing the GPU memory burden, balancing temporal coherence and local details, and achieving fast and flexible memory retrieval.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and machine learning technology, and specifically relates to a method for extending the long video understanding capability of a multimodal large language model through a visual memory mechanism. Background Technology

[0002] In recent years, the rapid development of multimodal large language models (MLLMs) has revolutionized the field of visual-language understanding. By combining the powerful reasoning capabilities of large language models (LLMs) with visual encoders, these models have demonstrated exceptional performance across a wide range of tasks. However, despite significant advancements in image and short video understanding, long video understanding remains a major hurdle for existing MLLMs. Unlike images or short videos, long videos contain a massive number of visual frames, and their data volume grows linearly over time, posing significant challenges to the model's context window constraints and computational resources.

[0003] To address this issue, existing research primarily follows two paradigms: Video Retrieval Augmentation Generation (Video-RAG) and visual feature compression. Video-RAG treats the video as an external knowledge base, retrieving keyframes based on user queries. While effective for "needle-in-a-haystack" tasks, it often ignores the temporal coherence of the video, leading to a lack of understanding of the global context. On the other hand, visual feature compression methods attempt to reduce memory usage per frame by compressing visual tokens. However, these methods typically still require all compressed features to be input into an MLLM at once for the response. This means the length of the input sequence remains proportional to the video duration, leading to memory overflow (OOM) or a sharp performance drop when processing extremely long videos (e.g., over one hour).

[0004] In summary, existing methods either sacrifice temporal coherence (Video-RAG) in pursuit of efficiency, or maintain temporal coherence but are limited by the physical upper limit of the context window (visual feature compression), and neither can perfectly solve the problem of long video understanding in MLLM. Summary of the Invention

[0005] To address the problems existing in the prior art, the purpose of this invention is to provide a method for extending the long video comprehension capability of a multimodal large language model through a visual memory mechanism, which significantly extends the model's ability to understand long videos without increasing the burden on video memory.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for extending the long-video comprehension capabilities of a multimodal large language model through visual memory mechanisms includes the following steps: Step 1: Divide the input long video into N consecutive segments evenly and initialize an empty visual memory bank. Step 2: Iteratively encode each video segment and generate two types of compressed memory for each segment through a dual-path compression mechanism: contextual memory and local memory; wherein, the contextual memory is used to pass historical information in the iterative steps, and the local memory is written into the visual memory bank for storage; Step 3: After processing all video segments, retrieve at least one local memory segment related to the user's input text command from the visual memory bank. Step 4: Input the retrieved local memory fragments and text instructions into the multimodal large language model to generate the final answer.

[0007] The iterative encoding of each video segment in step 2 includes: For the first video segment in the video clip The processing is as follows: The first video clip Input a multimodal large language model, if optional text instructions exist. The data is also input along with the data; MLLM encodes the current segment and generates compressed local memory based on a preset dual-path compression mechanism. and contextual memory :

[0008] in, Indicates optional input. Represents a multimodal large language model; The generated local memory Write to visual memory bank In Chinese; and memorize using context. Update context history; For subsequent video segments within a video clip , >1, the following steps are taken: video clips Contextual history fragments, common input multimodal large language models, and optional long-term memory. Text instructions The data is also input along with the data; MLLM encodes the current segment and generates the local memory of the current segment according to the preset dual-path compression mechanism. and contextual memory :

[0009] in, Indicates from visual memory bank Long-term memory retrieved from the middle Indicates reservation A recent contextual history fragment; The generated local memory Write to visual memory bank In Chinese; and memorize using context. Update the context history.

[0010] The dual-path compression mechanism during the encoding process is as follows: The most recent contextual memory C is included in the encoding of the current segment, and the attention matrix is ​​computed. :

[0011] in, and Indicates the current video segment The query and key vectors, and and Represents text instructions The query and key vectors, This represents the self-attention mechanism in LLM. The key vector representing the context memory C, when =1 hour, =0; Based on attention matrix In the pre-filling stage, the context aggregation score is used to filter tokens to generate context memory, and in the decoding stage, the local saliency score is used to filter tokens to generate local memory. Wherein, the context aggregation score Defined as:

[0012] in, yes From the current segment j The token is added to the historical context.k Attention weight of each token; It is the first The first in the layer j A key-value vector of each token; fragment Contextual memory It consists of KVCaches of all its layers, i.e. , L Indicates the number of layers in a large language model; Local significance score The definition is as follows:

[0013] Among them, fragments Local memory , It is a visual feature.

[0014] In step 3, the memory retrieval adopts a memory retrieval mode based on question coding, as follows: Calculate the relevance score between each storage segment and the text instruction. :

[0015] Based on score Sort in descending order and select the highest score. The local memory corresponding to each fragment is used as the retrieval result; assuming this... The index of each segment is , ,... The set of local memory fragments recalled is: Recall( , )={ , ,..., }。

[0016] In step 3, the memory retrieval uses a fast memory index mode, as detailed below: Step 3.2.1: Index feature construction; For text instructions The feature of the last token in the MLLM is selected as the text index. ; For each fragment stored in the memory bank Based on local saliency score Select the highest score in each layer k The key vectors of each token form a compact visual index for this segment. ; Step 3.2.2: Selection of key cache layers; On a small amount of training data, linear regression is used to fit the relationship between the relevance scores calculated based on full attention and the independent scores of each layer, thereby learning the regression weights for each layer. Select the K layers with the highest weights to form the key cache layer set. :

[0017] in, The number of key cache layers is crucial for ultimately calculating cross-modal similarity; Indicates the first The importance of layers for correlation calculation; Step 3.2.3: Rapid correlation calculation; Only in a pre-defined set of a few key cache layers In the process of calculating the text index... Visual Index Attention score between As a fast relevance score ;

[0018] in, It is the number of features in the key vector that ultimately serves as the visual index; Step 3.2.4: Based on the rapid correlation score Sort all segments in descending order and select the highest-scoring ones. The local memory corresponding to each fragment is used as the retrieval result; assuming this... The index of each segment is , ,... The set of local memory fragments recalled is: Recall( , )={ , ,..., }。

[0019] By adopting the above scheme, this invention uses a key-value cache of visual features as a memory representation, and decouples video processing from answer generation through iterative encoding and a query-based recall mechanism. In the encoding stage, the model only needs to process the current segment and historical context, without loading all visual frames at once; in the decoding stage, only a small amount of memory related to the question is recalled for answer generation, thus effectively solving the memory overflow problem and theoretically supporting video input of any duration. Simultaneously, through a dual-path compression mechanism, contextual memory is generated using context aggregation scores in the pre-filling stage, maintaining the temporal coherence of the video and global semantic understanding capabilities; in the decoding stage, local memory is generated using local saliency scores, preserving high-resolution local details. This approach balances understanding event evolution and capturing fine-grained visual evidence while significantly reducing memory size.

[0020] Building upon this foundation, the present invention also provides a fast memory indexing mode. By adaptively selecting representative caching layers and features based on the performance upper limit, a compact memory index tensor is formed. This mode is independent of the memory encoding process, supports efficient processing of multiple different problems within the same video, and significantly reduces the computational cost of cross-modal similarity through layer selection and token filtering, achieving fast and flexible memory retrieval while maintaining high accuracy. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the method framework of the present invention; Figure 2 This is a performance comparison chart between the present invention and the SOTA method; Figure 3 This is a visual diagram illustrating the results of selecting key segments in this invention. Detailed Implementation

[0022] like Figure 1 As shown, this invention discloses a method for extending the long video comprehension capability of a multimodal large language model through a visual memory mechanism, which includes the following steps: Step 1: Video segment division and initialization.

[0023] Divide the input long video into N segments evenly. Initialize an empty visual memory. It is used to store the local memory generated subsequently.

[0024] Step 2: Perform iterative memory encoding.

[0025] For each video segment ( The process iterates from 1 to N, generating two types of compressed memory for each segment: context memory and local memory. During encoding, a dual-path compression mechanism is used to reduce information redundancy. This mechanism is based on the self-attention mechanism of MLLM and uses different importance metrics for token selection at different stages of the model.

[0026] For the first video segment in the video clip The processing is as follows: The first video clip Input a multimodal large language model (MLLM). Optional text instructions may be available. The data is also input along with the data. MLLM encodes the current segment and generates compressed local memory based on a preset dual-path compression mechanism. and contextual memory :

[0027] Among them, local memory Written into the visual memory bank For later review and contextual memory Used to propagate historical information during iterative steps. Indicates optional input. This represents a multimodal large language model.

[0028] The generated local memory Write to visual memory bank Contextual memory This information is then retained and used to pass on historical information in the next iteration step.

[0029] For subsequent video segments within a video clip ( >1), the following steps are taken: video clips Contextual historical fragments, long-term memory (Optional) Text instructions (Optional) Common Input MLLM. MLLM encodes the current segment and generates local memory for the current segment according to a preset dual-path compression mechanism. and contextual memory :

[0030] in, Indicates from visual memory bank Long-term memory retrieved from the middle Indicates reservation A recent contextual history fragment.

[0031] During the encoding process, the dual-path compression mechanism is implemented as follows: The most recent contextual memory C is included in the encoding of the current segment, and the attention matrix is ​​computed. :

[0032] in, and Indicates the current video segment The query and key vectors, and and Represents text instructions The query and key vectors, This represents the self-attention mechanism in LLM. The key vector representing the context memory C, when =1 hour, It is 0.

[0033] Based on two attention-based metrics, redundant information is strategically discarded.

[0034] In the pre-filling phase, MLLM aims to encode the current fragment based on understanding historical information and propagate information backward. Therefore, this invention proposes a Context Aggregation Score to measure whether a token effectively aggregates information from the past and propagates it to subsequent tokens:

[0035] in, yes From the current segment j The token is added to the historical context. k Attention weights for each token. It is the first The first in the layer j A key-value vector for each token. (Fragment) Contextual memory It consists of KVCaches of all its layers, i.e. , L Indicates the number of layers in the MLLM.

[0036] During the decoding phase, MLLM aims to answer textual instructions based on the most salient visual evidence. Therefore, the priority at this stage is to eliminate redundancy within each segment to retain its most discriminative information. Consequently, a local saliency score is defined. To measure the overall influence of a token within its own fragment, and to use it to obtain compressed visual features. :

[0037] Among them, fragments Local memory .

[0038] The newly generated local memory Store in visual memory bank and use context to remember Update the context history.

[0039] Step 3: Query-based memory retrieval.

[0040] After processing all video segments, according to the text instructions provided by the user... From the visual memory bank To retrieve the most relevant memory fragments, this invention provides two reading modes.

[0041] Step 3.1: Memory retrieval based on problem coding.

[0042] For memory retrieval, an effective solution is to directly utilize the cross-modal attention encoded during memory compression. Leveraging the superior visual-language alignment capabilities of MLLMs, the cross-modal attention between video segments and questions can be directly used as an indicator of memory retrieval.

[0043] Specifically, in the pre-filling stage, the instruction token is used to pair the fragment. The relevance score is calculated by summing the attention weights of the tokens. :

[0044] Since visual tokens typically receive uniform attention scores in shallow layers, in practice, attention weights from deeper layers (e.g., after layer 2) are used only to compute relevance scores.

[0045] Based on score Sort in descending order and select the highest score. The local memory corresponding to each fragment is used as the recall result. Assume this... The index of each segment is , ,... The set of local memory fragments recalled is: Recall( , )={ , ,..., } Step 3.2, Quick Memory Index Mode.

[0046] To achieve fast and flexible memory retrieval, the following properties should be present. First, it should be independent of the encoding of visual memory, thus enabling efficient handling of multiple different questions or streaming video tasks using the same video. Second, the index features of the video and instructions should be sufficiently compact, further reducing the cost of cross-modal similarity computation.

[0047] To achieve the above goals, the performance of the problem-encoding-based memory retrieval method is considered as the upper limit of the performance of the fast memory index, and the target of the fast memory index is defined as:

[0048] here, It refers to the amount of training data used for optimization. This represents a fragment obtained using the fast memory indexing method. exist The correlation score of the layer. The aim is to find a linear regression function. To minimize the difference between the target score and the target score. of L2 distance.

[0049] Specifically, given the input problem First, its features are encoded using MLLM, and cross-modal similarity is calculated:

[0050] in It is a segment stored in memory M. The compressed key vector.

[0051] This basic solution still requires computing all visual and textual features across all layers. To reduce computation while preserving accuracy as much as possible, this invention first calculates based on the fitted regression function. Selecting a visual caching layer:

[0052] in, The number of key cache layers is the final number used to calculate cross-modal similarity. Indicates the first The importance of layers in correlation calculation is emphasized; therefore, this invention selects the layer with the highest learning weight. The caching layer.

[0053] Furthermore, to further reduce the number of features involved in relevance calculation, this invention utilizes an attention-based method to obtain a more compact visual index with a higher compression ratio. For question features, this invention selects the last feature as the text index. The final efficient relevance calculation method is as follows:

[0054] here It is the number of features in the key vector that ultimately serves as the visual index.

[0055] In summary, this invention provides a fast indexing method independent of the memory encoding process, comprising the following sub-steps: Step 3.2.1, Index feature construction.

[0056] For text instructions The feature of the last token in the MLLM is selected as the text index. .

[0057] For each fragment stored in the memory bank Based on local saliency score Select the highest score in each layer k The key vectors of each token form a compact visual index for this segment. .

[0058] Step 3.2.2: Selection of key cache layers.

[0059] To further reduce computational load, this invention selects the most representative key caching layers through offline fitting. Specifically, on a small amount of training data, linear regression is used to fit the relationship between the relevance score calculated based on full attention and the independent scores of each layer, thereby learning the regression weights for each layer. Select the K layers with the highest weights to form the key cache layer set. .

[0060] Step 3.2.3: Fast correlation calculation.

[0061] Only in a pre-defined set of a few key cache layers In the process of calculating the text index... Visual Index Attention score between As a fast relevance score .

[0062] Step 3.2.4: Based on the rapid correlation score Sort all segments in descending order and select the highest-scoring ones. The local memory corresponding to each fragment is used as the recall result. Assume this... The index of each segment is , ,... The set of local memory fragments recalled is: Recall( , )={ , ,..., } Due to the critical cache layer set It involves fitting a high-precision pattern target score using linear regression on a small amount of training data. Scores at each level The relationship was determined, and regression weights were selected. The highest K layers are used to determine this. This ensures that the fast mode can approximate the retrieval performance of the high-precision mode.

[0063] Step 4: Answer generation.

[0064] MLLM will recall one or more local memory fragments in step 3. { , ,..., } With text instructions Together, they perform the final decoding to generate the answer Y to the user's question: →Y In summary, this invention employs a key-value cache of visual features as a memory representation. Through iterative encoding and a query-based recall mechanism, it decouples video processing from answer generation. During the encoding phase, only the current segment and historical context need to be processed; during the decoding phase, only a small amount of memory related to the question is recalled. It eliminates the need to load all visual frames at once, theoretically supporting video input of any duration and effectively solving the problem of video memory overflow. Simultaneously, a dual-path compression design is introduced. In the pre-filling phase, contextual memory is generated through context aggregation scores to maintain the temporal coherence of the video and global semantic understanding capabilities. In the decoding phase, local memory is generated through local saliency scores to preserve high-resolution local details. This approach balances the understanding of event evolution and the capture of fine-grained visual evidence while significantly reducing memory size. Furthermore, the equipped fast indexing method adaptively selects representative cache layers and features by fitting the performance upper limit, forming a compact memory index tensor independent of the memory encoding process. This supports efficient processing of multiple different questions within the same video and significantly reduces computational costs through layer selection and token filtering, achieving fast and flexible memory retrieval.

[0065] To verify the effectiveness of this invention, it was applied to two representative video pedestal models, LLaVA-OneVision and LLaVA-Video, and extensive experiments were conducted on a large number of highly competitive benchmarks. The experimental results not only demonstrate a significant improvement to the video pedestal models but also verify its advantages over existing efficient video understanding methods. In extensive test set validation, this invention can greatly enhance the capabilities of video pedestal models and outperform a range of state-of-the-art methods using only a single 3090 GPU.

[0066] Table 1. Performance comparison of the present invention and the state-of-the-art (SOTA) method on five VideoQA test sets.

[0067] Table 1 presents a comprehensive comparison of FlexMem (this invention) with representative VideoRAG and visual compression methods on two recent MLLMs (LLaVA-Video and LLaVA-OneVision). AKS utilizes adaptive sampling to filter keyframes to reduce redundancy, Panels expands the visual context through multi-frame stitching layout, DToMA focuses on improving the accuracy of spatiotemporal alignment, Video-RAG introduces retrieval enhancement techniques for long videos, AdaRETAKE employs adaptive resampling to optimize the selection of representative frames, and BOLT aims to improve the efficiency of long video inference. These methods were evaluated across multiple dimensions: TimeScope focuses on temporal order understanding, LVBench focuses on the overall performance of long videos, MLVU examines multi-task long video inference capabilities, Video-MME provides a comprehensive benchmark covering short, medium, and long durations, and LongVideoBench specifically stress-tests information retrieval and localization in ultra-long videos. From Table 1, it can be observed that existing methods typically require dense frame sampling and a large number of token inputs for final decoding. VideoRAG methods like AKS excel at visual evidence localization on LongVideoBench, while visual compression methods like AdaRETAKE demonstrate strong overall video understanding capabilities on Video-MME. In contrast, this invention consistently enhances the performance of both base models, achieving state-of-the-art (SOTA) results in most benchmarks, outperforming other methods built on the same MLLM. This demonstrates the effectiveness of this invention's comprehensive memory construction through iterative processing and precise reverse tracing through memory recall. For example, this invention enables LLaVA-Video to outperform its baseline by 32.2% on TimeScope and 19.7% on LVBench. Figure 2 As shown, the performance advantages of this invention are more intuitively apparent in the visual comparison, achieving optimal results in most benchmark tests. These results conclusively verify the effectiveness of this invention in improving the long-form video understanding capabilities of MLLM.

[0068] Table 2 Performance comparison between the present invention and the state-of-the-art (SOTA) method under conditions of limited computational resources

[0069] The scalability and performance gains of this invention compared to two representative methods on a single 3090 GPU—AdaRETAKE, representing visual compression, and AKS, representing VideoRAG—were evaluated. As shown in Table 2, AdaRETAKE and AKS first exhibited considerable degradation compared to the performance under unconstrained computational conditions in Table 1. For example, when the GPU memory budget was limited to 24GB, AdaRETAKE's input capacity decreased from 1024 frames to 384 frames, with its performance dropping by an average of 3.3% across all benchmarks. In contrast, this invention (FlexMem) consistently maintained superior performance under resource constraints, retaining 99.5% of its full performance. Overall, these results demonstrate the invention's ability to flexibly manage visual memory while preserving critical information.

[0070] Table 3 Performance comparison between the present invention and the SOTA video model

[0071] Table 3 further compares the performance of this invention with existing state-of-the-art Video-MLLMs on five benchmark tests. Among them, GPT-5 and GPT-4o, as top-tier closed-source general-purpose large models, demonstrated extremely high comprehensive reasoning capabilities in multiple video benchmark tests; Gemini-1.5-Pro, with its ultra-long context window, excelled in the accuracy of long video understanding; Video-XL focused on improving the modeling capabilities of ultra-long video sequences with a scale of 7 billion parameters; mPLUG-Owl3 adopted a novel modular visual design to enhance the alignment effect of multimodal information; Qwen2.5-VL inherited the powerful language foundation of the Tongyi Qianwen series, significantly improving instruction compliance and text understanding in videos; TimeMarker introduced a timestamp marking mechanism to assist in video event retrieval through precise time positioning; LongVU was specifically optimized for logical chains in long videos to solve the modeling challenges of long-term dependencies; TSPO significantly reduced computational overhead while ensuring performance through decoupled sampling in the temporal and spatial domains; LongVA... This is a long-video assistant model that focuses on achieving efficient content question answering in a large-scale visual context. ByteVideoLLM efficiently maps videos to a large language model space through dynamic visual token compression and hierarchical modeling techniques. As shown in Table 3, short-video MLLMs like Qwen2.5-VL demonstrate superior performance on Video-MME, which requires global understanding, when a uniform sampling strategy is adopted. However, this simple solution performs significantly worse than visual compression methods like TSPO on LongVideoBench, as this benchmark requires fine-grained detailed reasoning over extended video durations. Compared to other methods with comparable model sizes, this invention (FlexMem) consistently achieves competitive or superior performance. Notably, this invention improves LLaVA-Video to the level of Gemini-1.5-Pro ​​while significantly outperforming it by 54.1% on LVBench. Overall, these results strongly demonstrate the effectiveness of this invention in improving long-video understanding in MLLMs.

[0072] Table 4. Performance comparison of the indexing method of this invention with SOTA online and offline methods on the OVOBench backtracking task.

[0073] Table 4 compares the performance of integrating MemIndex (the fast memory indexing mode of this invention) with existing state-of-the-art online and offline models in streaming question answering tasks. Among them, InternVL-V2 is a powerful offline model with 8 billion parameters (8B), demonstrating robust performance across multiple reverse tracing metrics; Flash-VStream is a real-time online processing model focused on streaming video understanding at a rate of 1 frame per second; VideoLLM-online is also positioned as an online model, capturing more detailed dynamic information through a sampling rate of 2 frames per second; Dispider is also an online video understanding method, showing competitiveness in contextual memory and action sequence recognition. As shown in Table 4, we can see that although offline models like LongVU demonstrate superior overall understanding capabilities on ASI compared to online methods like Dispider, their performance declines on EPM, which requires historical memory localization. After adopting the storage and retrieval method of this invention, LLaVA-Video outperforms its standard version by an average of 3%, demonstrating the effectiveness of the method in terms of effective memory retrieval and flexible context management. To visually demonstrate the memory recall effect of this invention, Figure 3 A visual example of the key segment selection results is provided. From Figure 3 It can be seen that the recalled fragments are highly relevant to the problem, verifying the effectiveness of this invention in historical information tracing. In summary, these results demonstrate the advantages of the MemIndex model of this invention in historical information tracing.

[0074] Table 5 Ablation Experiment

[0075] Table 5 examines the effectiveness of the dual-path compression mechanism. "Context-only compression" and "Local-only compression" indicate that only the upper and lower aggregation fractions are used, respectively. and local significance score Memory compression is performed. Results show that contextual features can convey historical information for understanding long videos, while local features effectively compress memory in short videos. Notably, the performance gain of our dual-path approach becomes more pronounced with increasing video duration, validating its ability to effectively utilize the different roles of MLLM in the pre-filling and decoding stages—encoding segments with contextual memory and generating predictions with stored memory.

[0076] Table 5 validates the effectiveness of contextual and local memories during the pre-filling stage. While using contextual or local memories alone during segment encoding yields reasonable performance, their combination leads to a significant performance boost. This finding suggests that these two memory types are complementary, with contextual memory preserving temporal continuity and local memory retaining long-range dependencies. The third module examines the benefits of our memory retrieval strategy compared to indiscriminately loading all memories. Results show that the memory recall of this invention can effectively identify and prioritize a small subset of key segments from extended videos. In the final module of Table 5, performance at different block sizes is analyzed. Results indicate that regardless of video duration, MLLM always requires a smaller block size to acquire detailed visual information. Overall, these results further confirm the effectiveness of the design choices proposed in this invention.

Claims

1. A method for extending the long-video comprehension capability of a multimodal large language model through visual memory mechanisms, characterized in that, Includes the following steps: Step 1: Divide the input long video into N consecutive segments evenly and initialize an empty visual memory bank. Step 2: Iteratively encode each video segment and generate two types of compressed memory for each segment through a dual-path compression mechanism: contextual memory and local memory; wherein, the contextual memory is used to pass historical information in the iterative steps, and the local memory is written into the visual memory bank for storage; Step 3: After processing all video segments, retrieve at least one local memory segment related to the user's input text command from the visual memory bank. Step 4: Input the retrieved local memory fragments and text instructions into the multimodal large language model to generate the final answer.

2. The method for extending the long video comprehension capability of a multimodal large language model through visual memory mechanisms according to claim 1, characterized in that, The iterative encoding of each video segment in step 2 includes: For the first video segment in the video clip The processing is as follows: The first video clip Input a multimodal large language model, if optional text instructions exist. The data is also input along with the data; MLLM encodes the current segment and generates compressed local memory based on a preset dual-path compression mechanism. and contextual memory : in, Indicates optional input. Represents a multimodal large language model; The generated local memory Write to visual memory bank In Chinese; and memorize using context. Update context history; For subsequent video segments within a video clip , >1, the following steps are taken: video clips The multimodal large language model is fed with contextual history fragments as input, and optional long-term memory is also considered. Text instructions The data is also input along with the data; MLLM encodes the current segment and generates the local memory of the current segment according to the preset dual-path compression mechanism. and contextual memory : in, Indicates from visual memory bank Long-term memory retrieved from the middle. Indicates reservation A recent contextual history fragment; The generated local memory Write to visual memory bank In Chinese; and memorize using context. Update the context history.

3. The method for extending the long video comprehension capability of a multimodal large language model through visual memory mechanisms according to claim 2, characterized in that, The dual-path compression mechanism during the encoding process is as follows: The most recent contextual memory C is included in the encoding of the current segment, and the attention matrix is ​​computed. : in, and Indicates the current video segment The query and key vectors, and and Represents text instructions The query and key vectors, This represents the self-attention mechanism in LLM. The key vector representing the context memory C, when =1 hour, =0; Based on attention matrix In the pre-filling stage, the context aggregation score is used to filter tokens to generate context memory, and in the decoding stage, the local saliency score is used to filter tokens to generate local memory. Wherein, the context aggregation score Defined as: in, yes From the current segment j The token is added to the historical context. k Attention weight of each token; It is the first The first in the layer j A key-value vector of each token; fragment Contextual memory It consists of KVCaches of all its layers, i.e. , L Indicates the number of layers in a large language model; Local significance score The definition is as follows: Among them, fragments Local memory , It is a visual feature.

4. The method for extending the long video comprehension capability of a multimodal large language model through visual memory mechanisms according to claim 2, characterized in that, In step 3, the memory retrieval adopts a memory retrieval mode based on question coding, as follows: Calculate the relevance score between each storage segment and the text instruction. : Based on score Sort in descending order and select the highest score. The local memory corresponding to each fragment is used as the retrieval result; assuming this... The index of each segment is , ,... The set of local memory fragments recalled is: Recall( , )={ , ,..., }。 5. The method for extending the long video comprehension capability of a multimodal large language model through visual memory mechanism according to claim 2, characterized in that, In step 3, the memory retrieval uses a fast memory index mode, as detailed below: Step 3.2.1: Index feature construction; For text instructions The feature of the last token in the MLLM is selected as the text index. ; For each fragment stored in the memory bank Based on local saliency score Select the highest score in each layer k The key vectors of each token form a compact visual index for this segment. ; Step 3.2.2: Selection of key cache layers; On a small amount of training data, linear regression is used to fit the relationship between the relevance scores calculated based on full attention and the independent scores of each layer, thereby learning the regression weights for each layer. Select the K layers with the highest weights to form the key cache layer set. : in, The number of key cache layers is crucial for ultimately calculating cross-modal similarity; Indicates the first The importance of layers for correlation calculation; Step 3.2.3: Rapid correlation calculation; Only in a pre-defined set of a few key cache layers In the process of calculating the text index Visual Index Attention scores between As a fast relevance score ; in, It is the number of features in the key vector that ultimately serves as the visual index; Step 3.2.4: Based on the rapid correlation score Sort all segments in descending order and select the highest-scoring ones. The local memory corresponding to each fragment is used as the retrieval result; assuming this... The index of each segment is , ,... The set of local memory fragments recalled is: Recall( , )={ , ,..., }。

Images