A Multimodal Large Model Optimization Inference Method Based on Location Rebinding and Cache Reuse
By using a location rebinding cache reuse method, the problems of redundant computation and obsolescence of visual information location binding in large multimodal models are solved, achieving efficient visual information reuse and multimodal inference, and improving the model's performance and task completion capabilities.
Patent Information
- Application Number
- CN202611243383.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-17
- Publication Date
- 2026-11-17
AI Technical Summary
Existing interleaved multimodal inference methods suffer from redundant computation and obsolete visual information location bindings in large multimodal models, resulting in high computational overhead and decoding crashes.
The method of location rebinding cache reuse is adopted. By constructing the original visual evidence memory module, the visual KV cache location reallocation module, and the replay-free cache decoding module, the position coordinates of the visual cache are reallocated and bound to achieve replay-free visual cache autoregressive decoding.
It significantly reduces the computational overhead of visual revisit, maintains the continuity of text position and the consistency of visual structure, improves the performance of multimodal large models, and can complete downstream tasks such as visual question answering, image description and video understanding.
Smart Images

Figure CN123117327A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal large model inference technology, specifically involving a multimodal large model optimized inference method based on location rebinding cache reuse. Background Technology
[0002] Multimodal Large Language Models (MLLMs) integrate information from multiple modalities, including vision and language, to endow models with cross-modal understanding and reasoning capabilities, thus expanding from "language intelligence" to "general cognitive intelligence." Among them, visual-language multimodal reasoning is an important research direction of MLLMs. Its goal is to establish deep semantic connections between visual information such as images and videos and natural language descriptions, enabling comprehensive inference of scene content, object relationships, spatiotemporal evolution, and implicit knowledge.
[0003] In multimodal reasoning, large multimodal models often employ a "one-time perception" paradigm. After encoding the input image once, the intermediate reasoning steps primarily rely on the text space to complete subsequent inferences. While text-based reasoning chains have achieved great success in unimodal reasoning, they reveal fundamental limitations in multimodal scenarios: the model relies solely on the text description generated by the initial visual encoding for reasoning, leading to insufficient visual foundation and illusion problems. The reasoning steps generated by the model are often disconnected from the image content, especially evident in tasks requiring fine-grained spatial perception. Therefore, interleaved multimodal reasoning has emerged. Its core concept is to allow the model to alternately generate text-based reasoning steps and visual intermediate results during the reasoning process, enabling the two modalities to mutually promote and co-evolve. Compared to traditional single-text reasoning chains, interleaved reasoning empowers the model with the ability to "think and draw simultaneously," using not only language to think but also visual images to verify, record, and advance the reasoning process.
[0004] Interleaved multimodal inference effectively alleviates the limitations of single-modal inference by inserting visual information during the decoding process of a large multimodal model. Existing interleaved multimodal inference methods typically employ a "replay mechanism," which selects relevant visual tokens or regions based on the current inference state and re-inputs them into the model for computation. While this mechanism helps to restore the visual semantics, it also introduces significant redundant computation because the same visual information, already encoded in the multimodal pre-filling stage, still needs to be computed again on each revisit. Directly reusing historical visual key-value caches can significantly reduce computation, but it faces the problem of "position binding stigmatization." The cached visual keys are already bound to the original positional context, which can distort attention during subsequent decoding and even cause decoding crashes. Summary of the Invention
[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a multimodal large model optimization inference method based on location rebinding cache reuse.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] A multimodal large model optimization inference method based on location rebinding cache reuse includes the following steps:
[0008] S1, Visual-Text Multimodal Feature Encoding; The multimodal large model receives visual and text multimodal inputs. The input image is converted into a series of visual tokens by a visual encoder, and the text prompts are encoded into text tokens by a word segmenter.
[0009] S2. A multimodal large-scale model autoregressive decoding framework is built to achieve continuous fusion of visual and linguistic information;
[0010] S3. Construct a multimodal large model inference framework based on the location rebinding cache reuse method; the implementation of the location rebinding cache reuse method includes the original visual evidence memory module, the visual KV cache location reallocation module, and the no-replay cache decoding module;
[0011] S4. Construct the original visual evidence memory space;
[0012] S5. Location reallocation for cache reuse: For the obtained original visual evidence memory, the location coordinates of cache entries are reallocated before RoPE location encoding is reapplied to its key vector;
[0013] S6. Construct the visual cache after location reassignment. After completing the coordinate reassignment, reconstruct the selected visual cache entries into visual cache entries compatible with the current location; perform autoregressive decoding calculation of the non-replay visual cache in the multimodal large model.
[0014] S7. Integrate multimodal information to complete downstream tasks of the multimodal large model.
[0015] The present invention also includes a computer-readable storage medium storing a program that, when executed by a processor, implements the multimodal large model optimization inference method based on location rebinding cache reuse provided by the present invention.
[0016] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0017] 1. This invention proposes a location rebinding cache reuse method. This method retains the original visual key-value cache without RoPE applied during the storage phase. When selecting cache entries, it reallocates coordinates compatible with the location. Before injecting the reconstructed cache into the currently active decoder cache, it rebinds the keys. This significantly reduces the computational overhead of visual revisiting while maintaining the positional continuity and relative visual structure of the text, achieving superior performance. Through this method, a multimodal large model establishes information fusion and semantic association between visual content and linguistic semantics via a cross-modal attention mechanism. During autoregressive decoding, it achieves revisiting of the visual cache without replay and continuously integrates multimodal contextual information. Based on this unified multimodal understanding and reasoning capability, the model can generate task-related outputs, thereby completing various downstream tasks such as visual question answering, image description, multimodal reasoning, and video understanding. Attached Figure Description
[0018] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0019] Explanation of relevant terms:
[0020] Position Rebinding Cache Reuse (PRCR) is a framework that achieves "reuse-free visual revisiting" at the cache level. PRCR stores the original encoded visual key-value cache along with its spatial coordinates, reassigns the selected cache entry to position coordinates that match the current context, and then rebinds the key's position before injecting it into the current decoder cache. This method reuses historical visual information while maintaining relative consistency in text positional continuity and visual alignment.
[0021] Visual Key-Value Cache: This refers to the key (K) and value (V) representations calculated by the attention modules of each layer of the Transformer when a multimodal large language model processes visual input. These representations are cached in the decoder for repeated use in the subsequent autoregressive generation stage, without the need to re-encode the visual input, thereby reducing computational overhead and maintaining visual context consistency.
[0022] M 3CoT: Dataset, see Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M3CoT: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. In Proceedings of the Association for Computational Linguistics, pages 8199–8221, 2024.
[0023] MathVista: Dataset, see Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyang Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2024.
[0024] MMStar: Dataset, see Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056–27087, 2024.
[0025] MMMU: Data set, see Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the Computer Vision andPattern Recognition, pages 9556–9567, 2024.
[0026] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.
[0027] Examples; such as Figure 1 As shown, the multimodal large model optimization inference method based on location rebinding cache reuse includes the following steps:
[0028] S1, Visual-Text Multimodal Feature Encoding; The multimodal large model receives visual and text multimodal inputs. The input image is converted into a series of visual tokens by a visual encoder (such as ViT), and the text prompts are encoded into text tokens by a tokenizer.
[0029] The visual token obtained from the conversion is represented as follows: , Represents a visual token The original two-dimensional spatial coordinates;
[0030] The text token is represented as .
[0031] S2. Multimodal large-scale model autoregressive decoding enables continuous fusion of visual and linguistic information; in this embodiment, it specifically includes:
[0032] After inputting visual and text tokens into the multimodal large model, the multimodal large model maps visual features to the same semantic space as the language model through a projection layer, and concatenates them with text tokens in a predefined order to form a unified multimodal sequence. This sequence is then encoded by a Transformer to establish cross-modal associations, enabling the model to understand the correspondence between image content and text semantics.
[0033] During the generation phase, the multimodal large model employs an autoregressive approach to progressively predict the next text token: at each generation step, the current token accesses historical text tokens and cached visual token representations through an attention mechanism, extracting relevant evidence from visual information and existing text context, calculating the probability distribution of the next token, and outputting it; the newly generated token is added to the context and continues to participate in subsequent reasoning and generation until a terminator is generated. This entire process achieves continuous fusion of visual and linguistic information, enabling the model to perform complex tasks such as visual question answering, image-text reasoning, and multimodal dialogue.
[0034] S3. Construct a multimodal large model inference framework based on the Position Rebinding Cache Reuse (PRCR) method; the implementation of the Position Rebinding Cache Reuse method includes the original visual evidence memory module, the visual KV cache position reallocation module, and the no-replay cache decoding module;
[0035] In this embodiment, the Raw Visual Evidence Memory (RVEM) module is used to store the visual key-value cache and its original spatial coordinates before rotational position encoding (RoPE) processing. ;
[0036] The visual key-value cache position reassignment module (Position Reassignment for CacheReinsertion) is used to assign selected cache entries position coordinates compatible with the current context. ;
[0037] The Replay-Free Cache Decoding module is used to re-bind the key to the RoPE position and inject the reconstructed visual cache into the subsequent generation process.
[0038] In the standard Transformer decoder cache for multimodal large models, keys and values are stored for autoregressive reuse. When visual keys employ positional encoding methods such as RoPE, the positional information they carry limits the secure reuse of visual evidence in subsequent decoding steps. In contrast, RVEM stores the original key / value projections and their original positions before RoPE application, enabling subsequent position-compatible cache reconstruction.
[0039] S4. Construct the original visual evidence memory space; in this embodiment, specifically:
[0040] During the multimodal pre-filling stage, the original key / value projections of the visual tokens at each layer are preserved; at this point, these keys are not yet bound to specific positional encodings. This represents the set of layers in a Transformer. Represents token Entering the Hidden feature representation during layer attention module; the first The original Key / Value projection of the layer attention module is defined as:
[0041] , , ;
[0042] These raw projections are stored before the key is encoded and bound to any specific location;
[0043] Construct a raw visual evidence memory storage space on all visual tokens:
[0044] ;
[0045] This memory storage space stores reusable original key / value projections and their original locations;
[0046] In the multimodal interleaved reasoning process of large multimodal models, the model needs to improve its visual alignment ability by repeatedly revisiting visual evidence during the multi-step generation process. This revisiting of visual evidence is crucial in the triggering step. Given a selected index set Collect the relevant visual entries and denot the selected set of original visual evidence memories as follows: .
[0047] S5. Location reallocation for cache reuse; For the obtained original visual evidence memory, before reapplying RoPE location encoding to its key vector, the location coordinates of the cache entries are reallocated; In this embodiment, specifically:
[0048] This embodiment employs the Position-Consistent Reinsertion (PCR) method. While maintaining the position of the next text token, it locally expands the reinsertion interval and re-anchors it based on the normalized relative position of the selected visual item. Specifically:
[0049] set up This indicates the position of the currently selected text token before the re-insertion occurs, and the position of the next text token is... The PCR method extends the re-insertion region to the left. ,in , is a parameter that controls the length of the left-side extension;
[0050] Based on the relative positional relationship of each visual item in the original visual sequence, the selected visual cache items are remapped and anchored to the extended range, thereby preserving the original spatial structure information of the visual cache as much as possible without changing the subsequent text position index.
[0051] For the currently selected visual entry Assuming its spatial coordinates , First, calculate the coordinate range of the selected cache subset on this coordinate:
[0052] , ;
[0053] For non-degenerative cases (i.e., the selected item on the spatial coordinate axis) (The coordinate range on the screen is not zero), select the visual item. The normalized relative position is defined as:
[0054] ;
[0055] Based on the normalized relative positions, visual entries In spatial coordinate axes The redistribution position on is defined as:
[0056] , ;
[0057] Finally, visual entries The redistribution position is represented as .
[0058] Reinsertion is performed based on location consistency, and all reinserted visual cache entries are constrained to an interval. The subsequent text token retains its original position. Therefore, it does not interfere with the positional progression of the text sequence. On the other hand, since the redistribution process maps visual entries based on their normalized relative positions in the original space, position-consistent reinsertion can better preserve the spatial structural consistency of the selected visual cache subset. The redistributed position coordinates are then used for position rebinding during the cache reconstruction process, thereby achieving replay-free visual cache reuse.
[0059] S6. Construct the visual cache after location reassignment. After completing the coordinate reassignment, reconstruct the selected visual cache entries into visual cache entries compatible with the current location; perform autoregressive decoding calculation of the non-replay visual cache in the multimodal large model.
[0060] In this embodiment, the visual cache after location reallocation is constructed as follows:
[0061] After the coordinate reassignment is completed, the selected visual cache entries will be reconstructed into visual cache entries compatible with the current position;
[0062] For the Layer attention module, given a selected set of cached indices Memory from original visual evidence The corresponding original key vector Key and value vector Value are projected and represented as follows:
[0063] ;
[0064] The set of redistributed position coordinates is represented as , This represents a position matrix consisting of all redistributed positions, with each row corresponding to the two-dimensional spatial coordinates of a selected visual item;
[0065] By rebinding the key vector to positional encoding and directly reusing the corresponding value vector, a reconstructed visual cache is built.
[0066] , ;
[0067] In the current layer, value vectors do not explicitly participate in RoPE positional encoding binding, so positional rebinding is not required; however, key vectors must be re-encoded with positional codes based on the current redistributed coordinates to ensure that they are consistent with the positional system of the current decoding context.
[0068] set up This indicates that the visual cache (decodercache) already exists in the decoder before re-insertion. The reconstructed visual cache is then concatenated with this cache to form the active cache used for attention calculation.
[0069] .
[0070] In this embodiment, the autoregressive decoding calculation of the replay-free visual buffer in the multimodal large model is performed as follows:
[0071] The multimodal large model continues decoding based on the transformer decoder, and the position index of the next text token is... Under the YZ position encoding interface, this one-dimensional position is extended into a two-dimensional form. ;
[0072] For the original query representation In its first The output of the transformer attention calculation after the layer is refactored and cached is:
[0073] ;
[0074] This completes the next step of autoregressive decoding computation based on the re-insertion of the visual cache.
[0075] S7. Integrating multimodal information to complete downstream tasks of the multimodal large model; In this embodiment, the multimodal large model, through the process of steps S1 to S6, establishes information fusion and semantic association between visual content and language semantics through a cross-modal attention mechanism, achieves replay-free visual cache revisiting during autoregressive decoding, and continuously integrates multimodal contextual information. Based on unified multimodal understanding and reasoning capabilities, the model can generate task-related outputs, thereby completing various downstream tasks such as visual question answering, image description, multimodal reasoning, and video understanding.
[0076] In this embodiment, the performance of the method of the present invention is finally verified by experiments.
[0077] Validation datasets: All methods were evaluated on four benchmark datasets: M 3 CoT, MathVista, MMStar and MMMU.
[0078] For M 3 CoT, reasoning experiments are conducted in multiple directions, including natural science, social science, language science, temporal commonsense, and mathematics.
[0079] Baseline models: Experiments were conducted on four multimodal large language models (MLLMs) from two model families, including Qwen3-VL 8B / 32B-Instruct and InternVL3.5 8B / 14B, which served as baseline models in the experiments.
[0080] Following the ICoT framework, the Token-Replay method recomputes the visual tokens selected by ADS (Adaptive Dynamic Selection) and injects them into the current decoding context. The PRCR method, on the other hand, uses the exact same ADS to select evidence as Token-Replay, thus ensuring that the experimental comparison focuses on the reuse of visual evidence rather than the differences in the selection of visual evidence itself.
[0081] Table 1 below shows the overall results of the model inference experiment for the four datasets.
[0082] Table 1. Experimental results of the four datasets in the model inference experiment.
[0083]
[0084] As shown in Table 1, compared to the base model, the Token-Replay method improves inference performance by re-propagating the selected visual tokens in the current decoding context. PRCR, on the other hand, achieves performance comparable to or even better than Replay without replaying the visual tokens.
[0085] On the Qwen3-VL 8B-Instruct baseline model, PRCR performs well on dataset M. 3 The average score of CoT improved from 66.44% for the base model to 68.68%, an increase of 2.24 percentage points.
[0086] On the InternVL3.5 8B benchmark model, PRCR also exhibits stable performance gains, with the average score increasing from 64.06% of the base model to 65.40%. This result demonstrates that the cache-level reconstruction mechanism proposed in this invention can effectively function across different models, exhibiting good versatility and robustness.
[0087] In another embodiment, a computer-readable storage medium is provided storing a program that, when executed by a processor, implements the multimodal large model optimization inference method based on location rebinding cache reuse of the present invention, specifically:
[0088] Visual-text multimodal feature encoding; the multimodal large model receives visual and text multimodal inputs. The input image is converted into a series of visual tokens by a visual encoder, and the text prompts are encoded into text tokens by a tokenizer.
[0089] Multimodal large-scale model autoregressive decoding enables continuous fusion of visual and linguistic information;
[0090] A multimodal large model inference framework based on the location rebinding cache reuse method is constructed. The implementation of the location rebinding cache reuse method includes the original visual evidence memory module, the visual KV cache location reallocation module, and the no-replay cache decoding module.
[0091] Constructing a memory space for original visual evidence;
[0092] Location reallocation for cache reuse; for the obtained raw visual evidence memory, the location coordinates of cache entries are reallocated before RoPE location encoding is reapplied to their key vectors;
[0093] Construct a visual cache after location reassignment. After coordinate reassignment is completed, reconstruct the selected visual cache entries into visual cache entries compatible with the current location. Perform autoregressive decoding calculation of the replay-free visual cache in the multimodal large model.
[0094] Integrate multimodal information to complete downstream tasks of a large multimodal model.
[0095] The computer-readable storage medium may be transient or non-transient. Exemplary examples include, but are not limited to, various media capable of storing computer program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0096] For example, the processor may be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.
[0097] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0098] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A multimodal large model optimization inference method based on location rebinding cache reuse, characterized in that, Includes the following steps: S1, Visual-Text Multimodal Feature Encoding; The multimodal large model receives visual and textual multimodal inputs. The input image is converted into a series of visual tokens by a visual encoder, while the text prompts are encoded into text tokens by a tokenizer. S2. A multimodal large-scale model autoregressive decoding framework is built to achieve continuous fusion of visual and linguistic information; S3. Construct a multimodal large model inference framework based on the location rebinding cache reuse method; the implementation of the location rebinding cache reuse method includes the original visual evidence memory module, the visual KV cache location reallocation module, and the no-replay cache decoding module; S4. Construct the original visual evidence memory space; S5. Location reallocation for cache reuse: For the obtained original visual evidence memory, the location coordinates of cache entries are reallocated before RoPE location encoding is reapplied to its key vector; S6. Construct the visual cache after location reassignment. After completing the coordinate reassignment, reconstruct the selected visual cache entries into visual cache entries compatible with the current location; perform autoregressive decoding calculation of the non-replay visual cache in the multimodal large model. S7. Integrate multimodal information to complete downstream tasks of the multimodal large model.
2. The multimodal large model optimization inference method based on location rebinding cache reuse according to claim 1, characterized in that, Step S1 is as follows: The input image is converted into a series of visual tokens by a visual encoder, while the text prompts are encoded into text tokens by a tokenizer. The resulting visual token is represented as , Visual token The original two-dimensional spatial coordinates; The text token is represented as .
3. The multimodal large model optimization inference method based on location rebinding cache reuse according to claim 1, characterized in that, In step S2, the multimodal large model autoregressive decoding specifically includes: After inputting visual and text tokens into the multimodal large model, the multimodal large model maps visual features to the same semantic space as the language model through a projection layer, and concatenates them with text tokens in a predefined order to form a unified multimodal sequence. This sequence is then encoded by a Transformer to establish cross-modal associations, enabling the model to understand the correspondence between image content and text semantics. During the generation phase, the multimodal large model uses an autoregressive approach to progressively predict the next text token: at each generation step, the current token accesses historical text tokens and cached visual token representations through an attention mechanism, extracts relevant evidence from visual information and existing text context, calculates the probability distribution of the next token, and completes the output; the newly generated token is added to the context and continues to participate in subsequent reasoning and generation until a terminator is generated.
4. The multimodal large model optimization inference method based on location rebinding cache reuse according to claim 1, characterized in that, In step S3, the original visual evidence memory module is used to store the visual KV cache that has not undergone rotation position encoding (RoPE) processing and its original spatial coordinates. ; The visual key-value cache location reallocation module is used to assign selected cache entries location coordinates compatible with the current context. ; The no-replay-cache decoding module is used to re-bind the key to the RoPE position and inject the reconstructed visual cache into the subsequent generation process.
5. The multimodal large model optimization inference method based on location rebinding cache reuse according to claim 1, characterized in that, Step S4 is as follows: During the multimodal pre-filling stage, the original key / value projections of the visual tokens at each layer are preserved; at this point, these keys are not yet bound to specific positional encodings. This represents the set of layers in a Transformer. Represents token Entering the Hidden feature representation during layer attention module; No. The original Key / Value projection of the layer attention module is defined as: , , ; These raw projections are stored before the key is encoded and bound to any specific location; Construct a raw visual evidence memory storage space on all visual tokens: ; This memory storage space stores reusable original key / value projections and their original locations; In the multimodal interleaved reasoning process of large multimodal models, the model needs to improve its visual alignment ability by repeatedly revisiting visual evidence during the multi-step generation process. This revisiting of visual evidence is crucial in the triggering step. Given a selected index set Collect the relevant visual entries and denot the selected set of original visual evidence memories as follows: .
6. The multimodal large model optimization inference method based on location rebinding cache reuse according to claim 1, characterized in that, Step S5 is as follows: The Position Consistency Reinsertion (PCR) method is adopted. While keeping the position of the next text token unchanged, the reinsertion interval is locally expanded, and the selected visual item is re-anchored based on its normalized relative position. Specifically: set up This indicates the position of the currently selected text token before the re-insertion occurs, and the position of the next text token is... The PCR method extends the re-insertion region to the left. ,in , is a parameter that controls the length of the left-side extension; Based on the relative positional relationship of each visual item in the original visual sequence, the selected visual cache items are remapped and anchored to the extended range, thereby preserving the original spatial structure information of the visual cache as much as possible without changing the subsequent text position index. For the currently selected visual entry Assuming its spatial coordinates , First, calculate the coordinate range of the selected cache subset on this coordinate: , ; For non-degenerative cases That is, the selected item on the spatial coordinate axis The coordinate range on the screen is not zero; select the visual item. The normalized relative position is defined as: ; Based on the normalized relative positions, visual entries In spatial coordinate axes The redistribution position on is defined as: , ; Finally, visual entries The redistribution position is represented as .
7. The multimodal large model optimization inference method based on location rebinding cache reuse according to claim 1, characterized in that, In step S6, the visual cache after location reallocation is constructed, specifically as follows: After the coordinate reassignment is completed, the selected visual cache entries will be reconstructed into visual cache entries compatible with the current position; For the Layer attention module, given a selected set of cached indices Memory from original visual evidence The corresponding original key vector Key and value vector Value are projected and represented as follows: ; The set of redistributed position coordinates is represented as , This represents a position matrix consisting of all redistributed positions, with each row corresponding to the two-dimensional spatial coordinates of a selected visual item; By rebinding the key vector to positional encoding and directly reusing the corresponding value vector, a reconstructed visual cache is built. , ; In the current layer, the value vector does not explicitly participate in RoPE location encoding binding, so there is no need for location rebinding; The key vector must be re-encoded according to the current reallocated coordinates to ensure that it is consistent with the positional system of the current decoding context; set up This indicates that the visual cache already exists in the decoder before re-insertion. The reconstructed visual cache is then concatenated with this cache to form the active cache used for attention computation. 。 8. The multimodal large model optimization inference method based on location rebinding cache reuse according to claim 7, characterized in that, In step S6, the autoregressive decoding calculation of the replay-free visual buffer in the multimodal large model is performed, specifically as follows: The multimodal large model continues decoding based on the transformer decoder, and the position index of the next text token is... Under the YZ position encoding interface, this one-dimensional position is extended into a two-dimensional form. ; For the original query representation In its first The output of the transformer attention calculation after the layer is refactored and cached is: ; This completes the next step of autoregressive decoding computation based on the re-insertion of the visual cache.
9. The multimodal large model optimization inference method based on location rebinding cache reuse according to claim 1, characterized in that, Step S7 is as follows: Through steps S1 to S6, the multimodal large model establishes information fusion and semantic association between visual content and linguistic semantics through a cross-modal attention mechanism. During the autoregressive decoding process, it achieves revisiting the visual cache without replay and continuously integrates multimodal contextual information. Based on a unified multimodal understanding and reasoning capability, the model generates task-related outputs to complete various downstream tasks.
10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the multimodal large model optimization inference method based on location rebinding cache reuse as described in any one of claims 1-9.
