A visual large model inference kv cache management method and system for heterogeneous computing power

Abstract

This invention discloses a method and system for managing KV Caches in visual large-scale model inference for heterogeneous computing power, belonging to the field of artificial intelligence technology. The method includes: in each iteration of the visual large-scale model inference, obtaining the attention score of each KV Cache in the Transformer layer and recording the historical cumulative sum of attention scores for each KV Cache; updating the historical cumulative sum of attention scores of the KV Cache with a time-decay weighted average based on the historical cumulative sum of attention scores and the newly acquired attention score of that KV Cache in the current iteration; obtaining the sum of the GPU memory usage of all KV Caches in the current iteration, and combining the currently available GPU memory with the updated historical cumulative sum of attention scores to determine the storage location and precision status of each KV Cache in the current iteration. This invention effectively reduces GPU memory pressure and data migration overhead between the GPU and CPU, thereby improving inference efficiency without significantly affecting inference accuracy.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a KVCache management method and system for visual large-scale model inference oriented to heterogeneous computing power. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale pre-trained visual models based on the Transformer architecture have demonstrated outstanding capabilities in the field of computer vision and have been widely applied in scenarios such as image generation, video understanding, and multimodal tasks. However, large visual models face severe challenges in the inference stage, including high memory requirements and high computational resource consumption, especially in tasks such as high-resolution image processing and long video sequence understanding.

[0003] In the inference process of large-scale visual models, whether it's the iterative denoising process of image generation models (such as Stable Diffusion and DALL-E) or the processing of long-term temporal features by video understanding models (such as VideoMAE and ViViT), it is necessary to cache the intermediate Key and Value results of the feature maps previously generated by each Transformer layer, i.e., KV Cache, for subsequent Attention calculations. The size of KV Cache increases linearly with image resolution and video frame rate. In high-resolution image generation or long video understanding scenarios, its memory usage can even exceed that of the model parameters themselves, becoming a major bottleneck restricting inference efficiency.

[0004] To address the memory pressure caused by KV Cache, existing technologies have proposed several optimization schemes: (1) Model compression and quantization, which can reduce parameter storage, but cannot effectively control the size of KV Cache; (2) KV Cache quantization, which can reduce memory usage, but the global unified quantization strategy will introduce significant accuracy loss, which manifests as blurred image details or decreased video temporal consistency in visual tasks; (3) Importance-based KV Cache eviction, which evaluates importance based on indicators such as Attention score and evicts low-value cache, but the evaluation standard is singular and ignores the long-term impact of visual features on temporal sequence; (4) Heterogeneous computing and memory offloading, which offloads part of KV Cache to CPU memory, but existing strategies are mostly based on static or simple heuristic rules and lack dynamic adaptability, resulting in frequent and unnecessary data migration, which introduces significant delays in scenarios such as real-time video processing.

[0005] In summary, existing technologies have the following shortcomings when addressing the memory bottleneck of large-scale visual model KV caches: they lack an evaluation mechanism that accurately and dynamically reflects the long-term value of visual feature KV caches; their scheduling strategies are coarse-grained and fail to adapt to memory pressure; and they lack synergistic optimization between migration and compression techniques. Therefore, there is an urgent need for a technical solution that can dynamically evaluate the importance of KV caches and adaptively perform hierarchical scheduling under memory constraints, in order to improve the inference efficiency of high-resolution images and long video sequences while ensuring the quality of visual generation or the accuracy of understanding.

[0006] This invention is proposed against this background, providing a method and system for managing KVCache for visual large model inference in heterogeneous computing environments. Through a time decay weighted importance scoring mechanism and a hierarchical dynamic scheduling strategy, it realizes intelligent storage, migration and compression of KV Cache, effectively improving the performance of visual large model inference and controlling accuracy loss. Summary of the Invention

[0007] To overcome the shortcomings of existing KV Cache management methods in large-scale model inference, such as insufficient adaptability in heterogeneous computing environments and lack of dynamic, fine-grained scheduling mechanisms based on the importance of attention mechanisms, this invention provides a KV Cache management method and system for large-scale visual model inference under heterogeneous computing power. In long-sequence inference scenarios with limited GPU memory, this method dynamically evaluates the importance of each layer of KV Cache and performs hierarchical and graded storage and precision management accordingly. This effectively reduces GPU memory pressure and data migration overhead between the GPU and CPU, thereby improving inference efficiency without significantly affecting inference accuracy.

[0008] To achieve the above objectives, the technical solution of the present invention includes the following:

[0009] A method for managing the key-value cache (KV) for visual large-scale model inference based on heterogeneous computing power, the method comprising: In each iteration of the large visual model, the attention score of each KV Cache in the Transformer layer is obtained, and the historical cumulative sum of the attention scores of each KV Cache is recorded. The historical attention score accumulation of the KV Cache is updated by weighting the accumulation of the KV Cache's historical attention score with the newly acquired attention score in the current iteration, using a time decay method. Obtain the sum of the memory usage of all KV Caches in the current iteration, and combine it with the current available GPU memory and the updated historical attention score accumulation to determine the storage location and precision status of each KV Cache in the current iteration.

[0010] Furthermore, the historical attention score accumulation and sum ;in, Indicates the number of iterations in the reasoning process. For the first Layer The KV Cache in the first The historical cumulative importance score of each iteration For the first Layer The attention score newly acquired by each KV Cache in the current iteration. This is the time decay factor.

[0011] Furthermore, by combining the currently available GPU memory with the updated cumulative sum of historical attention scores, the storage location and precision state of each KV Cache in the current iteration are determined, including: Provided that the available GPU memory is not less than the sum of the memory usage of all KV Caches, all KV Caches retain their original precision and reside entirely in the GPU. When the available GPU memory is less than the sum of the memory usage of all KV Caches, the KV Cache in each Transformer layer is divided into high-importance KV Cache and low-importance KV Cache based on the updated historical cumulative attention score and the first cumulative importance score ratio threshold. The high-importance KV Cache resides in the GPU and maintains its original precision, while the low-importance KV Cache is migrated to the CPU and maintains its original precision.

[0012] Furthermore, the first cumulative importance score ratio threshold is dynamically adjusted based on the current available GPU capacity and sequence length; wherein, the sequence length is obtained based on the original input sequence pre-padded to the large visual model and the output sequence of the large visual model decoding stage.

[0013] Furthermore, after residing the high-importance KV cache in the GPU while maintaining its original precision, the method also includes: Calculate the sum of the video memory usage of the most important key-value caches; Provided that the available GPU memory is not less than the sum of the memory usage of the high-importance KV caches, all high-importance KV caches retain their original precision and reside entirely in the GPU; When the available GPU memory is less than the sum of the memory usage of the high-importance KV Caches, the high-importance KV Caches are divided into a first high-importance KV Cache, a second high-importance KV Cache, and a third high-importance KV Cache based on the updated historical cumulative attention score and the first and second cumulative importance score ratio thresholds. The first high-importance KV Cache resides in the GPU and maintains its original precision. The second high-importance KV Cache resides in the GPU after quantization and compression. The third high-importance KV Cache is migrated to the CPU and maintains its original precision. Among these, the historical cumulative attention score of the first high-importance KV Cache is greater than that of the second high-importance KV Cache, and the historical cumulative attention score of the second high-importance KV Cache is greater than that of the third high-importance KV Cache.

[0014] Furthermore, when the available GPU memory is less than the sum of the memory usage of the high-importance KV Cache, the method further includes: If the available GPU memory is less than a set threshold, the CPU and GPU will swap the resident KV Cache based on the updated historical attention score sum; wherein the set threshold is the sum of the memory usage of the first high-importance KV Cache and the second high-importance KV Cache.

[0015] A KV Cache management system for visual large-scale model inference oriented to heterogeneous computing power, the system comprising: The attention monitoring module is used to obtain the attention score of each KV Cache in the Transformer layer in each round of inference iteration of the large visual model, and record the cumulative sum of the historical attention scores of each KV Cache; The importance assessment module is used to update the historical attention score accumulation of the KV Cache by weighting it with the newly acquired attention score of the KV Cache in the current iteration, based on the historical attention score accumulation. The hierarchical scheduling module is used to obtain the sum of the memory usage of all KV Caches in the current iteration, and combine the current available GPU memory with the updated historical attention score accumulation to determine the storage location and precision status of each KV Cache in the current iteration.

[0016] A computer device, characterized in that the computer device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the KV Cache management method for visual large model inference oriented to heterogeneous computing power as described above.

[0017] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the KV Cache management method for visual large model inference oriented to heterogeneous computing power as described above.

[0018] A computer program product, characterized in that, when the computer program product is run on a computer device, it causes the computer device to execute the visual large model inference KV Cache management method for heterogeneous computing power as described above.

[0019] Compared with the prior art, the present invention has at least the following beneficial effects.

[0020] (1) This invention proposes a time-decay weighted importance scoring mechanism based on attention scores. By introducing accumulated information from historical attention scores, it comprehensively reflects the long-term contribution of the KV Cache in continuous inference, avoiding the fluctuations in importance judgment caused by relying solely on single attention scores. At the same time, it utilizes a time decay factor to enhance the influence of recent attention information, enabling the importance score to change dynamically with the inference process. This can improve the stability and accuracy of KV Cache importance assessment, reduce repeated migration between the GPU and CPU due to frequent changes in importance, and reduce data migration overhead.

[0021] (2) This invention refines the management granularity of KV Cache to the "layer" and introduces the "cumulative importance score ratio" as the scheduling threshold, thereby realizing fine-grained and dynamic scheduling of KV Cache. Compared with traditional coarse-grained eviction strategies such as global LRU, this method can more accurately retain the KV Cache that is most critical to subsequent generation.

[0022] (3) This invention constructs a multi-level, adaptive KV Cache scheduling system. This system can smoothly switch between multiple methods such as migration only and migration + quantization according to the changes in GPU memory pressure, and coordinately optimize memory usage and data migration overhead, so as to maximize the reasoning efficiency and accuracy in long text reasoning scenarios. Attached Figure Description

[0023] Figure 1 This is a system architecture diagram of the present invention.

[0024] Figure 2 This is a schematic diagram of the importance score update mechanism of the present invention.

[0025] Figure 3 This is a schematic diagram of the hierarchical dynamic scheduling process of the present invention.

[0026] Figure 4 This is a schematic diagram of the GPU-CPU collaborative storage state in this invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the objectives, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0028] The technical solutions of the present invention will be further described below with reference to the accompanying drawings. The described embodiments are some embodiments of the present invention, but not all embodiments, and will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Furthermore, some well-known technologies may not have been described in detail for those skilled in the art.

[0029] In this embodiment, as Figure 1 As shown, this invention provides a visual large-model inference KVCache management system for heterogeneous computing power, comprising: an attention monitoring module, an importance assessment module, a hierarchical scheduling module, a transfer execution module, and a quantization execution module. The attention monitoring module is connected to the large-model inference engine and is used to collect the attention score of each Transformer layer; the importance assessment module is connected to the attention monitoring module and is used to calculate the importance score of each layer's KVCache; the hierarchical scheduling module is connected to the importance assessment module and makes scheduling decisions based on the available GPU memory; the transfer execution module and the quantization execution module are respectively connected to the hierarchical scheduling module and are used to perform specific transfer and compression operations. The system runs on a heterogeneous computing device containing at least one GPU and at least one CPU.

[0030] Based on the above system, this invention provides a method for managing the KV cache of visual large-scale model inference for heterogeneous computing power, comprising the following steps: (1) Attention Score Acquisition and Accumulation: Large model inference is initiated on a computing device containing at least one GPU and at least one CPU. In each inference iteration, the attention score of each KVCache in each Transformer layer is acquired through the attention monitoring module, and the historical attention score accumulation of each KVCache is recorded. These historical data are stored in system memory for subsequent importance evaluation.

[0031] (2) Dynamic Calculation of Importance Score: After each round of inference iteration, the importance assessment module calculates the latest historical attention score of each KVCache by weighting it with the newly acquired attention score in the current iteration, based on the historical attention score accumulation of each KVCache over time decay. Figure 2 As shown, the latest historical cumulative Attention score (importance score) is calculated using the following formula: in, For the first Layer The KV Cache in the first The cumulative sum of historical attention scores from each iteration. The Attention score calculated in the current iteration. This is the time decay factor, with a value ranging from 0 to 1. By setting an appropriate value... The system can balance the impact of historical and current information on importance assessment. For long text generation scenarios, a larger value can be set. Values ​​(such as 0.9) are used to emphasize historical cumulative contributions.

[0032] (3) Hierarchical Dynamic Scheduling: The hierarchical scheduling module performs hierarchical dynamic management of the KV Cache based on the currently available GPU memory and the importance score of each KV Cache. For example... Figure 3 As shown, this step further includes four sub-stages: a) Full Resident: When the available GPU memory is sufficient to fully accommodate all layers of KV Cache, all KV Caches retain their original precision (e.g., FP16) and reside entirely in the GPU. The system is in optimal performance at this point, requiring no migration or compression operations.

[0033] b) Priority Eviction: When GPU memory is insufficient to accommodate all KV Caches, the system sorts the KV Caches in each layer from highest to lowest importance score and manages them according to the cumulative importance score ratio. Let the first cumulative importance score ratio threshold be... In each layer, the importance score is ranked higher, and the sum of its scores accounts for a certain percentage of the total score. The key-value cache is marked as high importance, retains its original precision, and resides on the GPU; the remaining key-value caches in this layer are marked as low importance, migrated to CPU memory, and retain their original precision. Threshold The sequence length can be dynamically adjusted based on the available GPU capacity and the sequence length, which is obtained from the original input sequence pre-padded to the large visual model and the output sequence from the decoding stage of the large visual model. To ensure inference accuracy, the smaller the available GPU capacity or the longer the sequence, the better. The lower the value can be set, the better. In each iteration, after calculating the Attention score for each layer, the importance scores of each layer's KVCache are recalculated, reordered, and GPU-CPU migration is performed.

[0034] c) Quantized Compact Eviction: When GPU memory is still insufficient to accommodate high-importance KV Caches after priority eviction, the system performs a finer-grained partitioning of these high-importance KV Caches. Based on the cumulative importance score ratio, high-importance KV Caches are divided into three tiers: those ranked highest in each tier, and those whose sum of scores reaches the second cumulative importance score ratio threshold. The KV Cache (the most important KV Cache, as shown in the attached diagram) retains its original precision and resides on the GPU; the score-based cache is the next most important, with a cumulative proportion of [missing information]. to The KV Cache (second most important KV Cache) between these two caches is resident on the GPU after being quantized and compressed using a 4-bit or 8-bit quantization process by the quantization execution module; the cache with the lowest score and a cumulative proportion exceeding [a certain threshold] is [not specified]. The KV Cache (the third highest importance KV Cache) continues to be stored in the CPU. In each iteration, the importance score is updated, sorted, and migrated again. From a system perspective, the first highest importance KV Cache can also be called the high importance KV Cache, the second highest importance KV Cache can also be called the medium importance KV Cache, and the low importance KV Cache in the priority eviction phase and the third highest importance KV Cache in the quantization and compaction eviction phase can be collectively referred to as the low importance KV Cache.

[0035] d) Steady-state maintenance: When the GPU capacity is full and cannot be further adjusted (i.e., the first and second most important KV caches of each layer have filled the GPU memory), the size of the original precision and quantization KV cache in the GPU remains unchanged, and the CPU continues to store the remaining third most important KV caches and low-importance KV caches of each layer. At this stage, the system continues to perform importance score updates and sorting in each iteration, but since the GPU capacity has reached its limit, the migration operation only involves replacing the resident KV caches between the CPU and the GPU (if the importance score of a low-importance KV cache in a certain layer that was originally in the CPU rises to a high importance level, it needs to be swapped with a KV cache in the GPU whose importance has decreased).

[0036] Figure 4 This demonstrates the typical distribution of KV Cache between the GPU and CPU in the four stages described above.

[0037] (4) Iterative update: After each inference iteration, the system repeats steps (2) and (3) to dynamically update the storage location and precision status of each KV Cache between the GPU and CPU. As the number of inference iterations increases, the importance score of each KV Cache changes continuously, and some layers may migrate multiple times between the GPU and CPU. The system always maintains the optimal distribution that satisfies the memory constraints, thereby adaptively coping with the memory pressure caused by the increase in sequence length.

[0038] In actual deployment, the migration execution module is responsible for performing specific GPU-CPU bidirectional migration operations, including loading the KVCache from the CPU to the GPU, releasing it from the GPU to the CPU, and updating the corresponding memory pointer information to ensure that GPU memory constraints are not exceeded. The quantization execution module is responsible for performing quantization compression operations on the specified KVCache layer, supporting multiple quantization precisions such as INT8 and INT4, reducing memory usage while maintaining inference accuracy as much as possible.

[0039] Although the present invention has been described through the above preferred embodiments, those skilled in the art will understand that various modifications in form and detail can be made therein without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A method for managing the key-value cache (KV) for visual large-scale model inference based on heterogeneous computing power, characterized in that, The method includes: In each iteration of the large visual model, the attention score of each KV Cache in the Transformer layer is obtained, and the historical cumulative sum of the attention scores of each KV Cache is recorded. The historical attention score accumulation of the KV Cache is updated by weighting the accumulation of the KV Cache's historical attention score with the newly acquired attention score in the current iteration, using a time decay method. Obtain the sum of the memory usage of all KV Caches in the current iteration, and combine it with the current available GPU memory and the updated historical attention score accumulation to determine the storage location and precision status of each KV Cache in the current iteration.

2. The method according to claim 1, characterized in that, The historical attention score accumulation and ;in, Indicates the number of iterations in the reasoning process. For the first Layer The KV Cache in the first The historical cumulative importance score of each iteration For the first Layer The attention score newly acquired by each KV Cache in the current iteration. This is the time decay factor.

3. The method according to claim 1, characterized in that, By combining the currently available GPU memory with the updated cumulative sum of historical attention scores, the storage location and precision state of each KV Cache in the current iteration are determined, including: Provided that the available GPU memory is not less than the sum of the memory usage of all KV Caches, all KV Caches retain their original precision and reside entirely in the GPU. When the available GPU memory is less than the sum of the memory usage of all KV Caches, the KV Cache in each Transformer layer is divided into high-importance KV Cache and low-importance KV Cache based on the updated historical cumulative attention score and the first cumulative importance score ratio threshold. The high-importance KV Cache resides in the GPU and maintains its original precision, while the low-importance KV Cache is migrated to the CPU and maintains its original precision.

4. The method according to claim 3, characterized in that, The first cumulative importance score ratio threshold is dynamically adjusted based on the current available GPU capacity and sequence length; wherein, the sequence length is obtained based on the original input sequence pre-filled into the large visual model and the output sequence of the large visual model decoding stage.

5. The method according to claim 3, characterized in that, After residing the high-importance key-value cache in the GPU while maintaining its original precision, the method further includes: Calculate the sum of the video memory usage of the most important key-value caches; Provided that the available GPU memory is not less than the sum of the memory usage of the high-importance KV caches, all high-importance KV caches retain their original precision and reside entirely in the GPU; When the available GPU memory is less than the sum of the memory usage of the high-importance KV Caches, the high-importance KV Caches are divided into a first high-importance KV Cache, a second high-importance KV Cache, and a third high-importance KV Cache based on the updated historical cumulative attention score and the first and second cumulative importance score ratio thresholds. The first high-importance KV Cache resides in the GPU and maintains its original precision. The second high-importance KV Cache resides in the GPU after quantization and compression. The second high-importance KV Cache is migrated to the CPU and maintains its original precision. Among these, the historical cumulative attention score of the first high-importance KV Cache is greater than that of the second high-importance KV Cache, and the historical cumulative attention score of the second high-importance KV Cache is greater than that of the third high-importance KV Cache.

6. The method according to claim 5, characterized in that, When the available GPU memory is less than the sum of the memory usage of high-importance KVCaches, the method further includes: If the available GPU memory is less than a set threshold, the CPU and GPU replace the resident KV Cache based on the updated historical attention score accumulation; wherein, the set threshold is the sum of the memory usage of the first high importance KV Cache and the second high importance KV Cache.

7. A visual large-model inference KV Cache management system for heterogeneous computing power, characterized in that, The system includes: The attention monitoring module is used to obtain the attention score of each KV Cache in the Transformer layer in each round of inference iteration of the large visual model, and record the cumulative sum of the historical attention scores of each KV Cache; The importance assessment module is used to update the historical attention score accumulation of the KV Cache by weighting it with the newly acquired attention score of the KV Cache in the current iteration, based on the historical attention score accumulation. The hierarchical scheduling module is used to obtain the sum of the memory usage of all KV Caches in the current iteration, and combine the current available GPU memory with the updated historical attention score accumulation to determine the storage location and precision status of each KV Cache in the current iteration.

8. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the visual large model inference KV Cache management method for heterogeneous computing power as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the visual large model inference KV Cache management method for heterogeneous computing power as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the computer program product is run on a computer device, the computer device performs the visual large model inference KVCache management method for heterogeneous computing power as described in any one of claims 1-6.

Images