A cache management method, apparatus, device, medium and product

Abstract

The present application relates to the technical field of artificial intelligence, in particular to a cache management method, device, equipment, medium and product, the method comprises: evaluating the importance score of different attention heads to the current inference task in real time, and dynamically allocating different levels of storage resources, including storage location and numerical accuracy, to the key-value cache of each attention head according to the importance score, so as to maximize the saving of memory under the premise of ensuring the quality of model output, especially the non-loss of key information, and then improve the inference speed.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a cache management method, apparatus, device, medium, and product. Background Technology

[0002] In the field of artificial intelligence, Transformer models and KV Cache: Transformer models (such as GPT, BERT, T5, etc.) need to cache the key and value vectors of previous time steps (or positions) during inference in autoregressive generation (such as text generation) or long sequence processing tasks (KV Cache) to avoid repeated calculations and significantly improve inference speed.

[0003] KV Cache memory bottleneck: As the context window length increases (e.g., from 1K to 100K or even longer), the memory consumption of KV Cache increases linearly, becoming a major bottleneck for deploying large Transformer models (especially large language model LLMs), limiting its application on resource-constrained devices (such as edge devices), and increasing cloud service costs.

[0004] In related technologies, to address the memory bottleneck problem of KV Cache, the main approaches include any of the following: a unified quantization method; pruning or selective caching strategies based on fixed rules; improved cache data structures; and acceleration utilizing hardware features. However, all of these methods use the same caching strategy for all attention points, which can easily lead to the loss of critical information. Summary of the Invention

[0005] The purpose of this invention is to provide a cache management method, apparatus, device, medium, and product that can evaluate the importance scores of different attention heads to the current inference task in real time, and dynamically allocate different levels of storage resources (including storage location and numerical precision) to the key-value cache of each attention head according to the importance scores. This maximizes memory saving and improves inference speed while ensuring the quality of model output, especially the preservation of key information.

[0006] Firstly, a cache management method is provided, comprising: during model inference, calculating in real time the importance score of a target attention head in the current layer, wherein the target attention head is any one of multiple attention heads in the current layer; determining the storage level corresponding to the key-value cache generated by the target attention head based on the importance score of the target attention head in the current layer, wherein the storage level includes storage location and storage precision; and storing the key-value cache according to the storage location and storage precision corresponding to the key-value cache generated by each attention head.

[0007] In a preferred embodiment, the present invention can be further configured as follows: storing key-value caches according to the storage location and storage precision corresponding to the key-value caches generated by each attention head, including: sorting the importance scores of all attention heads in the current layer to obtain a first sorting result; determining the key-value cache with the highest importance score according to the sorting result; determining whether the storage level corresponding to the key-value cache with the highest importance score is available; if available, storing the key-value cache with the highest importance score to the corresponding cache location according to the storage precision; if unavailable, adjusting the storage level corresponding to the key-value cache with the highest importance score; and storing the key-value cache according to the adjusted storage location and storage precision corresponding to the key-value cache with the highest importance score; after completing the storage of the key-value cache with the highest importance score, processing the next key-value cache in sequence until the storage of the key-value caches generated by each attention head is completed.

[0008] In a preferred embodiment, the present invention can be further configured to: adjust the storage level corresponding to the key-value cache with the highest importance score, including: if the storage level corresponding to the key-value cache with the high importance score is high-speed storage, then determine whether the medium-speed storage of the storage level corresponding to the key-value cache with the highest importance score is available; if the medium-speed storage is available, then downgrade the storage level corresponding to the key-value cache with the highest importance score to medium-speed storage; if the medium-speed storage is unavailable, then downgrade the storage level corresponding to the key-value cache with the highest importance score to low-speed storage.

[0009] In a preferred embodiment, the present invention can be further configured to: adjust the storage level corresponding to the key-value cache with the highest importance score, including: if the storage level corresponding to the key-value cache with the high importance score is medium-speed storage, then determine whether low-speed storage of the storage level corresponding to the key-value cache with the highest importance score is available; if low-speed storage is available, then downgrade the storage level corresponding to the key-value cache with the highest importance score to low-speed storage; if low-speed storage is unavailable, then upgrade the storage level corresponding to the key-value cache with the highest importance score to high-speed storage.

[0010] In a preferred embodiment, the present invention can be further configured as follows: if unavailable, the storage level corresponding to the key-value cache with the highest importance score is adjusted, including: if unavailable, determining whether the storage level corresponding to the key-value cache with the highest importance score is low-speed storage; if not low-speed storage, adjusting the storage level corresponding to the key-value cache with the highest importance score; the method further includes: if it is low-speed storage, discarding the key-value cache with the highest importance score.

[0011] In a preferred embodiment, the present invention may further be configured to include: obtaining the storage resource usage of each storage tier of the hierarchical storage resource pool; and adjusting the hierarchical storage resource pool according to the storage resource usage and importance score of each storage tier.

[0012] In a preferred embodiment, the present invention can be further configured as follows: storage resource usage includes remaining capacity; storage tiers include low-speed storage, medium-speed storage, and high-speed storage; adjusting the tiered storage resource pool based on the storage resource usage and importance score of each storage tier includes: determining whether the remaining capacity of the storage tier is less than a preset capacity threshold; if the remaining capacity of the storage tier is less than the preset capacity threshold, then filtering target key-value caches that meet the adjustment requirements in the storage tier based on their importance scores; and adjusting the tiered storage resource pool based on the target key-value caches that meet the adjustment requirements in the storage tier.

[0013] In a preferred embodiment, the present invention can be further configured as follows: adjusting the tiered storage resource pool by selecting target key-value caches that meet the adjustment requirements in the storage tier, including: adjusting the tiered storage resource pool according to the adjustment instructions after selecting target key-value caches that meet the adjustment requirements in the storage tier.

[0014] In a preferred embodiment, the present invention can be further configured as follows: if the adjustment instruction is an elimination instruction, the storage level is the low-speed storage of the hierarchical storage resource pool. For the target key-value cache, the hierarchical storage resource pool is adjusted according to the adjustment instruction, including: sorting the target key-value cache according to the elimination instruction based on at least one of importance score and last access time; and eliminating the target key-value cache sequentially based on the sorting result until the remaining capacity of the low-speed storage is not less than the first capacity.

[0015] In a preferred embodiment, the present invention can be further configured as follows: if the adjustment instruction is a migration instruction, the storage level is a medium-speed storage or high-speed storage of a tiered storage resource pool. For the target key-value cache, the tiered storage resource pool is adjusted according to the adjustment instruction, including: according to the migration instruction, the target key-value cache is migrated to the next level of storage, and the storage location of the target key-value cache in the metadata is updated.

[0016] In a preferred embodiment, the present invention may be further configured to include: during the migration process, marking the state of the migrated key-value cache as a migration state, the migration state being used to prohibit other operations; and after the migration is completed, marking the state of the migrated key-value cache as an available state.

[0017] In a preferred embodiment, the present invention can be further configured as follows: if the adjustment instruction is a precision adjustment instruction, then the storage level is a high-speed storage of a tiered storage resource pool; correspondingly, for the target key-value cache, the tiered storage resource pool is adjusted according to the adjustment instruction, including: according to the precision adjustment instruction, reducing the storage precision of the target key-value cache, replacing the original target key-value cache, and updating the precision information of the target key-value cache in the metadata.

[0018] In a preferred embodiment, the present invention can be further configured to: calculate the importance score of the target attention head in the current layer in real time, including: calculating the importance score of the target attention head in the current layer in real time based on the input sequence of model inference of the target attention head in the current layer.

[0019] In a preferred embodiment, the present invention can be further configured to: calculate the importance score of the target attention head in the current layer in real time, including: if the calculation mode is a single-selection mode, then calculate the importance score of the target attention head in the current layer in real time according to the target single-item calculation mode corresponding to the single-selection mode; if the calculation mode is a weighted comprehensive mode, then perform weighted calculation based on the values ​​calculated by each single-item calculation mode to obtain the importance score of the target attention head in the current layer.

[0020] In a preferred embodiment, the present invention can be further configured as follows: A single-item calculation mode includes: an attention output-based mode, an attention score-based mode, a historical information-based mode, and a lightweight model mode. The calculation process based on the attention output mode includes: inputting the input sequence into the corresponding attention head to obtain an output vector; determining the vector norm based on the output vector; obtaining the attention score matrix of the attention head when inferring the input sequence; calculating the entropy of each row of the attention score matrix; determining the entropy of the attention head based on the entropy of each row; and determining a first importance score based on the vector norm and / or the entropy of the attention head. The calculation process based on the attention score mode includes: inputting the input sequence into the corresponding attention head to obtain an output vector; determining the vector norm and / or the entropy of the attention head; and the ... The input sequence is converted into query vectors and key vectors; the query vectors and key vectors are split according to the number of attention heads; based on the query vectors and key vectors of the attention heads, the attention score is determined as the second importance score; the calculation process based on historical information patterns includes: obtaining the historical importance scores of the attention heads; based on the historical importance scores of the attention heads, the importance score of the attention heads is predicted as the third importance score; the calculation process of the lightweight model pattern includes: based on the input sequence of the attention heads, the state of the current layer, and the evaluation result of the previous time step, using a multilayer perceptron model, the importance score of the target attention head in the current layer is predicted as the fourth importance score.

[0021] In a preferred embodiment, the present invention may be further configured to include: when accessing the key-value cache generated by the target attention head, determining whether the storage precision of the key-value cache is the precision required for calculation; if not, performing an inverse quantization operation based on the storage location and storage precision corresponding to the target attention head.

[0022] Secondly, a cache management device is provided, comprising: an importance score determination module, used to calculate the importance score of a target attention head in the current layer in real time during model inference, wherein the target attention head is any one of multiple attention heads in the current layer; a storage level determination module, used to determine the storage level corresponding to the key-value cache generated by the target attention head based on the importance score of the target attention head in the current layer, wherein the storage level includes storage location and storage precision; and a storage module, used to store the key-value cache according to the storage location and storage precision corresponding to the key-value cache generated by each attention head.

[0023] Thirdly, an electronic device memory is provided for storing a computer program; and a processor for executing the computer program to implement the method as described in any of the first aspects.

[0024] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the method as described in any of the first aspects.

[0025] Fifthly, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the method as described in any of the first aspects.

[0026] In summary, the cache management method provided by this invention has the following beneficial technical effects: Real-time calculation of the importance score of the target attention head in the current layer, where the target attention head is any one of multiple attention heads in the current layer; determination of the storage level corresponding to the key-value cache generated by the target attention head based on its importance score, where the storage level includes storage location and storage precision; and storage of the key-value cache based on the storage location and storage precision corresponding to the key-value cache generated by each attention head. By real-time evaluation of the importance scores of different attention heads to the current inference task and dynamically allocating different levels of storage resources (including storage location and numerical precision) to the key-value cache of each attention head based on its importance score, the method maximizes memory savings and improves inference speed while ensuring the quality of model output, especially preventing the loss of key information.

[0027] In addition, the present invention also provides a cache management device, equipment, medium and product, all of which have the above-mentioned beneficial technical effects. Attached Figure Description

[0028] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of KV Cache management in related technologies;

[0030] Figure 2 This is a schematic diagram of a cache management method provided in an embodiment of the present invention;

[0031] Figure 3 This is a system architecture diagram provided in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of a specific dynamic hierarchical allocation strategy provided in an embodiment of the present invention;

[0033] Figure 5 This is a flowchart of the inference steps for cache migration and precision adjustment provided in an embodiment of the present invention;

[0034] Figure 6 This is a schematic diagram illustrating the interaction between the hierarchical storage resource pool and the cache manager provided in an embodiment of the present invention;

[0035] Figure 7 A flowchart of a method for dynamically evaluating the importance of attention heads provided in an embodiment of the present invention;

[0036] Figure 8 This is a schematic diagram of the structure of a cache management device provided in an embodiment of the present invention;

[0037] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0039] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0040] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] KVCache's memory bottleneck: As the context window length increases (e.g., from 1K to 100K or even longer), KVCache's memory consumption grows linearly, becoming a major bottleneck for deploying large Transformer models (especially large language model LLMs), limiting its application on resource-constrained devices (such as edge devices), and increasing cloud service costs.

[0042] See Figure 1 , Figure 1 This is a schematic diagram of KVCache management in related technologies, in which all attention heads are processed uniformly. Its characteristics include: all attention heads are processed in the same way, a single storage level (such as GPU memory), fixed-precision configuration (such as FP16 / INT8), and a simple eviction policy (such as FIFO / LRU).

[0043] Existing optimization techniques and their limitations: (1) Uniform quantization: uniformly reduce the precision of all elements in the entire KVCache (e.g., FP16→INT8,INT4). However, quantization of attention heads / elements with high importance may introduce large errors and affect the quality of model output; quantization of attention heads / elements with low importance may not be aggressive enough and waste memory.

[0044] (2) Cache sparsity / pruning: Discarding some key-value vectors (such as tokens with low attention scores). However, the discarding operation is irreversible and may result in the loss of key information; dynamic sparsity strategies are complex and may introduce additional computational overhead; it is difficult to guarantee that all important information is retained. Existing technologies have limitations in pruning granularity and basis.

[0045] Specifically, existing pruning techniques are mostly granular at the "token level" or "hierarchy" level, rather than at the "attention head level".

[0046] Token-level pruning: Based on the attention score of a single token, it does not distinguish the importance of the attention head to which the token belongs (for example, a token with a low attention score may belong to an attention head that is critical to the task, and key information will be lost after pruning).

[0047] Hierarchical pruning: Directly prune the KVCache of the entire Transformer layer (e.g., cache one layer every two layers), without considering the functional differences of different attention heads within the layer (e.g., some attention heads in a layer are responsible for syntactic analysis and some are responsible for semantic understanding, and will be discarded indiscriminately during pruning).

[0048] Current technologies lack modules for real-time assessment of the importance of attention heads and cannot distinguish the value of different attention heads.

[0049] For example, a certain pruning scheme may fix the pruning of "attention heads whose attention score variance is less than a threshold", but this threshold is preset offline (calibrated based on a static dataset) and cannot adapt to changes in the input sequence during inference (such as when processing "medical text", the importance of the attention head responsible for entity recognition increases, and the preset threshold may mistakenly prune this attention head).

[0050] The core difference between this invention and existing technologies is that it upgrades the decision-making basis for pruning / sparsening from "fixed rules of tokens / layers" to "real-time importance scores of attention heads", thereby achieving differentiated pruning at the attention head level (low-importance attention heads can be discarded, while high-importance attention heads are retained).

[0051] (3) Selective caching: Caching only the key-value pairs of some layers or the key-value pairs of specific attention heads. However, static policies cannot adapt to different inputs; dynamic policies require complex decision-making; and may corrupt the internal representation of the model.

[0052] (4) Recalculation: No caching or minimal caching, recalculation when needed. However, this significantly increases computation latency, which defeats the original purpose of using KVCache to accelerate inference.

[0053] (5) Sharding / Offloading: Offloading part of the KVCache to CPU memory or disk. However, CPU / disk access latency is high, which seriously affects inference speed; frequent data transfers bring additional overhead.

[0054] In summary, existing technologies generally ignore or fail to effectively utilize the significant differences in function and importance among different attention heads in the Transformer model. Using the same caching strategy (such as the same precision and the same storage location) for all attention heads leads to an unbalanced resource allocation, either sacrificing the precision of important attention heads or wasting valuable memory resources on secondary attention heads.

[0055] Based on this, the present invention aims to overcome the shortcomings of existing KVCache optimization techniques and proposes a dynamic hierarchical key-value cache management method and system based on attention head importance awareness. The core of this method lies in real-time evaluation of the importance of different attention heads to the current inference task, and dynamically allocating different levels of storage resources (including storage location and numerical precision) to the KVCache of each attention head (or elements within an attention head) according to its importance score. This maximizes memory savings and potential inference speed improvements while ensuring the quality of model output (especially preventing the loss of key information).

[0056] This invention provides a system-level optimization solution for the KVCache memory bottleneck in real-world inference scenarios for Transformer models. Its application scenarios include, but are not limited to: 1. Long text generation (such as dialogue systems and document generation), where KVCache memory usage increases linearly with sequence length; 2. Resource-constrained environments (such as edge devices and mobile deployments), where memory and computing resources are limited; 3. High-concurrency inference services (such as large model inference optimization in cloud services), where multi-tenant inference in cloud services requires efficient utilization of GPU memory to reduce costs.

[0057] This invention belongs to the field of artificial intelligence technology, specifically relating to deep learning model inference optimization technology, and particularly to efficient management and optimization technology of key-value caching in the inference process of Transformer architecture models.

[0058] The following sections will describe in detail a cache management method, apparatus, device, medium, and product provided by embodiments of the present invention.

[0059] Figure 2 This is a flowchart illustrating a cache management method provided in an embodiment of the present invention. The cache management method includes: S101, during model inference, calculating the importance score of the target attention head in the current layer in real time, wherein the target attention head is any one of multiple attention heads in the current layer.

[0060] Before inference and after model training, the "potential importance" of each attention head is fixed (due to its fixed structure, i.e., its fixed functional role). However, because the content and structure of the input sequence are variable, the specific importance score is dynamically calculated during inference in this embodiment of the invention. During inference, the importance of different attention heads changes dynamically when processing different input keys. For example, some attention heads may specialize in handling syntactic structures, some in handling semantic keywords, and some in handling long-range dependencies; their importance depends on the current input content.

[0061] In this embodiment of the invention, the importance of KVCache is innovatively evaluated dynamically at the attention head level (rather than the entire layer or the entire model), providing a key insight into leveraging the inherent characteristics of the Transformer multi-attention head mechanism.

[0062] In this embodiment of the invention, importance assessment and storage hierarchy allocation decisions are performed in real time during inference, adapting to changes in the input sequence and the evolution of the model's internal state. The input sequence refers to the sequence of inputs used for model inference (e.g., a text token sequence). This invention dynamically responds to changes in input content by calculating the importance score of each attention head in real time. For example, when the input changes from "syntactic structure" to "semantic keywords," the importance of the attention heads processing this information increases, triggering an adjustment in the storage strategy. The model's internal state refers to implicit, dynamically changing information during inference, such as changes in the distribution of attention weights, the evolution of hidden layer activation values, and changes in the content of the historical KVCache. This invention captures this state evolution through dynamic importance assessment (e.g., using historical score smoothing and lightweight prediction models), thereby making more reasonable storage decisions.

[0063] S102. Based on the importance score of the target attention head in the current layer, determine the storage level corresponding to the key-value cache generated by the target attention head. The storage level includes storage location and storage precision.

[0064] In this embodiment of the invention, hierarchical resource allocation is performed by combining two dimensions: storage location and numerical precision, to provide differentiated quality of service for attention heads of different importance.

[0065] Furthermore, tiered storage resources are predefined. Specifically, this definition includes tiered storage resource pools with multiple storage locations and numerical precisions, i.e., multiple storage tiers are defined. Each storage tier includes: storage location and storage precision. Higher storage location speed results in faster access speed, and higher storage precision results in higher data precision. For example: Storage location tier: highest speed storage (e.g., GPU HBM), medium speed storage (e.g., CPU DRAM), low speed storage (e.g., SSD / NVMe), or optional non-storage (i.e., discarding). Numerical precision tier: high precision (e.g., FP16 / BF16), medium precision (e.g., INT8), low precision (e.g., INT4 / FP8), or custom quantization (e.g., quantization based on the statistical characteristics of the attention head).

[0066] For example: High-importance attention heads: allocate storage tier to high-speed storage location + high precision → ensure fast access and low error. Medium-importance attention heads: allocate storage tier to medium-speed storage location + medium precision → maintain a balance between speed, precision, and memory. Low-importance attention heads: allocate storage tier to low-speed storage location + low precision or discard directly → maximize memory savings.

[0067] It is understandable that the above is just an example. In actual systems, the configuration can be flexibly combined according to hardware characteristics and task requirements. For example, high-speed storage + medium precision is suitable for attention heads that are of high importance but not extremely sensitive to precision; medium-speed storage + high precision is suitable for attention heads that have high precision requirements but a slightly lower access frequency.

[0068] In one feasible approach, a pre-defined correspondence between importance scores and storage levels is established. Consequently, based on the importance score of the target attention head in the current layer, the storage level corresponding to the key-value cache generated by the target attention head can be matched. The higher the importance score, the faster the storage speed and the higher the storage accuracy.

[0069] In another feasible approach, key-value caches can be allocated sequentially from the high-speed storage tiers based on importance scores and the remaining capacity of each storage tier. When storage capacity is not available, the cache is moved to the next storage tier.

[0070] The specific implementation method chosen is not limited in this embodiment of the invention, and users can customize the settings.

[0071] In this embodiment of the invention, since the importance score of each attention head is dynamically changing, a dynamic decision is made based on the importance score of each attention head to allocate the KVCache generated by that attention head at the current time step to a predefined matching storage level. By precisely allocating limited fast storage and high-precision resources to the most important attention heads, while transferring or discarding secondary attention heads to slow / low-precision storage, a significant reduction in overall memory usage is achieved, and the impact on the access speed of critical attention heads is minimized.

[0072] S103. Store the key-value cache according to the storage location and storage precision of the key-value cache generated by each attention head.

[0073] In this embodiment of the invention, if there is enough remaining storage space at the storage location corresponding to the attention head, the key value generated by the attention head can be cached with storage precision and stored at the storage location.

[0074] It is understood that in this embodiment of the invention, there are multiple attention heads whose key-value caches need to be stored at any given time. Therefore, they can be stored sequentially according to their importance scores. If the remaining storage space of the corresponding storage level is insufficient when storing the key-value cache of a certain attention head, the storage can be downgraded or upgraded. Alternatively, if the space is still insufficient after downgrading / upgrading, if the importance score of the attention head is high, the key-value cache with a low importance score that has not been accessed for a long time can be eliminated. If the importance score of the attention head is low, the key-value cache corresponding to the attention head can be eliminated.

[0075] Of course, the storage hierarchy can also periodically evict, migrate, and adjust the precision of key-value caches to ensure that the storage hierarchy has enough storage space to store new key-value caches.

[0076] As can be seen, in this embodiment of the invention, during model inference, the importance score of the target attention head in the current layer is calculated in real time. The target attention head is any one of multiple attention heads in the current layer. Based on the importance score of the target attention head in the current layer, the storage level corresponding to the key-value cache generated by the target attention head is determined. The storage level includes storage location and storage precision. The key-value cache is stored according to the storage location and storage precision corresponding to the key-value cache generated by each attention head. The importance scores of different attention heads to the current inference task are evaluated in real time, and different levels of storage resources (including storage location and numerical precision) are dynamically allocated to the key-value cache of each attention head based on the importance scores. This maximizes memory saving and improves inference speed while ensuring the quality of model output, especially the preservation of key information.

[0077] For a better understanding of the technical solution of this invention, see [link / reference]. Figure 3 , Figure 3 This is a system architecture diagram provided by an embodiment of the present invention, which includes a layer (Transformer layer), an importance assessment module, a hierarchical resource pool, a cache manager, and an allocation decision-maker.

[0078] The Transformer layer includes a multi-attention head attention mechanism; it generates key-value vectors; and outputs an attention vector `attn_output` / attention score `attn_weights`. The importance assessment module evaluates the importance based on the Transformer layer's output. It calculates the importance of attention heads in real time; based on the input `attn_output` / `attn_weights`, it uses methods such as norm / entropy / historical state to output an importance score. An allocation decision-maker receives the importance score; obtains the resource pool status; and dynamically decides the storage tier using threshold / sorting / hybrid strategies. The cache manager manages metadata, specifically including location / precision / importance, timestamps / status, and a data migration engine; eviction policies; and precision conversion. The timestamp records key time nodes for KV cache blocks, including the block's generation time (the time it was first stored in the tiered resource pool) and the time of its most recent retrieval. Its core function is to assist in the execution of eviction policies. For example, when low-speed storage is full, it combines importance scores to prioritize the eviction of cache blocks with the "lowest importance and longest unused time (based on the most recent access timestamp)," avoiding the accidental eviction of critical historical information. Status: Describes the current storage and availability status of the KV cache block, including three core statuses: Storage Status: Identifies the current storage level of the cache block (e.g., Level 0: GPU HBM, Level 1: CPU DRAM); Precision Status: Records the current numerical precision of the cache block (e.g., FP16, INT8), providing a basis for dequantization operations during retrieval; Migration / Transformation Status: Marks whether the cache block is undergoing asynchronous migration (e.g., from GPU HBM to CPU DRAM) or precision adjustment (e.g., from FP16 to INT8), avoiding data consistency issues caused by concurrent operations.

[0079] In one feasible approach, data migration includes: Step 1, migration trigger detection. Every inference time step, scan the metadata of all cache blocks to check if migration conditions are met (e.g., the attention head importance score decreases from >θ_high to <θ_high). Step 2: Enqueue Migration Tasks. Add eligible cache blocks to the "Migration Task Queue," sorting them by the "decline in Importance Score" (those with larger declines are migrated first). Step 3: Asynchronous Migration Execution. A background thread (not blocking the main inference thread) performs the migration, including: reading source cache blocks: reading low-precision cache blocks (e.g., FP16) from the source storage location (e.g., GPU HBM); precision conversion (if necessary): such as converting from FP16 to INT8 (based on attention head statistics); writing to the target location: writing the converted cache blocks to the target storage (e.g., CPU DRAM); updating metadata: updating the cache block's storage location and precision information to the metadata list. Step 4: Migration Status Monitoring. During migration, mark the cache block status as "Migration in Progress," prohibiting other operations (e.g., retrieval). After migration is complete, mark it as "Available."

[0080] In one feasible approach, the eviction policy implementation process includes: Step 1, Eviction Triggering Conditions. The eviction process is triggered when the remaining capacity of low-speed storage (such as SSD) is <10%. Step 2, Eviction Candidate Screening. From all cache blocks on the SSD, a candidate set with an "importance score < θ_low" is selected (ensuring the eviction of low-value cache blocks). Step 3, Eviction Priority Sorting. The candidate set is sorted by "importance score (ascending order) + last access time (ascending order)" (i.e., the lowest importance and longest unused blocks are prioritized for eviction). Step 4, Eviction Execution. Cache blocks are deleted sequentially according to the sorting results, releasing storage space until the remaining capacity of the SSD is ≥15%, and the corresponding metadata records are deleted simultaneously. Step 5, Eviction Rollback (Optional). If, after eviction, it is found that a certain evicted cache block needs to be reused (e.g., the KV data of that attention head is needed for subsequent inference), the KV data of that attention head is recalculated and stored (based on the current importance score and allocation level).

[0081] In one feasible approach, the precision conversion process includes: Step 1, conversion triggering conditions. When the remaining capacity of a certain storage level (e.g., CPU DRAM) is <8%, and there exists a cache block with an importance score < θ_mid and a precision of INT8. Step 2, statistics update. Recalculate the statistics (max_val, min_val) of all cache blocks for this attention point to ensure quantization precision. Step 3, precision conversion execution (taking INT8→INT4 as an example) includes: reading the INT8 cache block: reading the INT8 format key / value block from CPU DRAM. Dequantizing to FP16: based on the original INT8 statistics, first dequantize to FP16. Requantizing to INT4: based on the new statistics (max_val_new, min_val_new), perform quantization. (INT4 ranges from 0-15, mapping to the max-min range of FP16). Replace cache block: Write the INT4 cache block to CPU DRAM, delete the original INT8 cache block, and free up memory (INT4 saves 50% of the capacity compared to INT8). Step 4, Metadata Update: Update the precision information of the cache block from INT8 to INT4 and record the new statistics.

[0082] The subsequent inference steps are used to generate a KVCache request and receive the retrieved KV data.

[0083] In the Transformer model, when calculating attention in subsequent time steps of autoregressive inference, requests need to be received and key-value (KV) data retrieved. When the model generates a new token, the current layer's multi-attention head mechanism needs to combine the "query vector of the current input token" with the "KV cache stored in historical time steps (i.e., the key and value vectors of previous tokens)" to calculate the attention score, thereby generating the current layer's attention output. At this time, the cache manager receives metadata (storage location, precision) to locate and read the data. If it is stored in low precision, inverse quantization is performed first, and then the data is returned to the attention calculation module.

[0084] The resource pool is divided into three tiers: high-speed storage (Level 0), medium-speed storage (Level 1), and low-speed storage (Level 2). Specifically, Level 0 consists of GPU HBM and FP16 / BF16 memory, offering high speed and high precision; Level 1 consists of CPU DRAM and INT8 memory, offering medium speed and medium precision; and Level 2 consists of SSDs (discarded) and INT4 memory, offering low speed and low precision.

[0085] Based on the overall system architecture, in one possible implementation of this invention, S103 stores the key-value cache according to the storage location and storage precision of the key-value cache generated by each attention head, including: S1031, sorting the importance scores of all attention heads in the current layer to obtain a first sorting result.

[0086] The current layer includes at least one attention head. All attention heads in the current layer at the current time are sorted according to their importance scores to obtain the first ranking result. This refers to all attention heads in a specific layer of the Transformer. Attention heads within the same layer share the same input and output dimensions, but their functions may differ.

[0087] Multiple attention heads reside in the same layer. Here, "same layer" refers to a "single encoder / decoder layer" in the Transformer model, containing "multi-attention head attention modules + feedforward network modules." For example, in the 64 layers of GPT-3, each layer is an independent "same layer." A single layer contains h attention heads (e.g., 12), sharing the same input sequence and layer normalization parameters. The attention heads in the same layer collaboratively complete the attention computation for that layer, and their key-value caches (KVCache) need to be managed uniformly (the storage location of the attention head KVCaches in the same layer needs to consider the overall resource requirements of that layer).

[0088] In this embodiment of the invention, the h attention heads in the same layer are sorted in descending order of importance score (e.g., if h=12, the sorted values ​​are...). ).

[0089] S1032. Based on the sorting results, determine the key-value cache with the highest importance score.

[0090] All attention heads within the same layer are sorted by importance, and resources (such as high-speed storage slots and high-precision quotas) are preferentially allocated to the first N attention heads according to their order.

[0091] For a storage level determination method based on the correspondence between importance score range and storage level, in one feasible method, S1033, S1034, and S1035 can be executed sequentially.

[0092] In another feasible approach, the remaining capacity of the hierarchical resources of the model to which the current layer belongs can be obtained (e.g., 5GB remaining for GPU HBM and 20GB remaining for CPU DRAM). If all 12 attention heads correspond to GPU HBM, the number of attention heads that each layer can accommodate is calculated based on the remaining capacity: if each attention head of GPU HBM requires 0.5GB (FP16 precision), and the remaining 5GB can accommodate 10 attention heads, then GPU HBM + FP16 is allocated to the first 10 attention heads in GPU HBM. The remaining 2 attention heads (H11, H12) are allocated to CPU DRAM + INT8 (each attention head requires 0.25GB, 2 attention heads total 0.5GB, and the remaining 20GB of CPU is sufficient). It should be noted that if the remaining capacity of a certain layer is insufficient (e.g., only 2GB of GPU HBM remains, which can accommodate 4 attention heads), then the first 4 attention heads are allocated to GPU HBM + FP16, and the 5th to 12th attention heads are allocated to CPU DRAM + INT8.

[0093] S1033. Determine whether the storage level corresponding to the key-value cache with the highest importance score is available.

[0094] In this embodiment of the invention, corresponding available space size thresholds / space ratio thresholds can be set for different storage tiers; when the available space size / space ratio is greater than the corresponding threshold, it indicates that the storage tier is available. The available space size thresholds / space ratio thresholds for storage tiers can be exactly the same or not exactly the same, and can be customized by the user.

[0095] S1034. If available, the key-value cache with the highest importance score is stored in the corresponding cache location according to the storage precision.

[0096] S1035. If unavailable, adjust the storage level of the key-value cache corresponding to the highest importance score; and store the key-value cache according to the adjusted storage location and storage precision based on the level of the key-value cache corresponding to the highest importance score.

[0097] S1036. After storing the key-value cache with the highest importance score, process the next key-value cache in sequence until the storage of the key-value cache generated by each attention head is completed.

[0098] One possible implementation of this invention involves adjusting the storage level corresponding to the key-value cache with the highest importance score, including: if the storage level corresponding to the key-value cache with the high importance score is high-speed storage, then determining whether medium-speed storage of the storage level corresponding to the key-value cache with the highest importance score is available; if medium-speed storage is available, then downgrading the storage level corresponding to the key-value cache with the highest importance score to medium-speed storage; if medium-speed storage is unavailable, then downgrading the storage level corresponding to the key-value cache with the highest importance score to low-speed storage.

[0099] In this embodiment of the invention, the high-speed storage level is greater than the medium-speed storage level, which is greater than the low-speed storage level. This only indicates the order of the three levels, not that there are only three storage levels. The high-speed storage level represents the highest storage level; the low-speed storage level represents the lowest storage level; and the medium-speed storage level is the middle storage level. There is at least one medium-speed storage level, which has an upper-level storage level and a lower-level storage level.

[0100] It's important to note that, generally, key-value caches at each storage tier are managed periodically to ensure sufficient remaining storage capacity. However, in some cases, storage capacity may exceed a certain threshold, indicating that the storage tier is unavailable. In such situations, degradation can be implemented.

[0101] In one possible scenario, downgrading the storage tier corresponding to the key-value cache with the highest importance score to low-speed storage may include: if low-speed storage is available, a second downgrade is performed; if low-speed storage is unavailable, key-value caches with low importance scores and long-term inaccessibility in low-speed storage are considered for removal to free up storage space for the current key-value cache. Of course, if space in high-speed storage is subsequently cleared and there is remaining space, the downgraded key-value caches can be migrated back to high-speed storage based on their importance score / timestamp order. This embodiment of the invention does not limit this further.

[0102] One possible implementation of this invention involves adjusting the storage level corresponding to the key-value cache with the highest importance score if there is only one medium-speed storage. This includes: if the storage level corresponding to the key-value cache with the high importance score is medium-speed storage, determining whether the low-speed storage of the storage level corresponding to the key-value cache with the highest importance score is available; if the low-speed storage is available, downgrading the storage level corresponding to the key-value cache with the highest importance score to medium-speed storage; and if the low-speed storage is unavailable, upgrading the storage level corresponding to the key-value cache with the highest importance score to high-speed storage.

[0103] One possible implementation of this invention involves adjusting the storage level corresponding to the key-value cache with the highest importance score if there are only a few medium-speed storage units. This adjustment includes: if the storage level corresponding to the key-value cache with the highest importance score is medium-speed storage, determining whether the next-level storage of the storage level corresponding to the key-value cache with the highest importance score is available; if the next-level storage is available, downgrading the storage level corresponding to the key-value cache with the highest importance score to the next-level storage; if the next-level storage is unavailable, upgrading the storage level corresponding to the key-value cache with the highest importance score to the previous-level storage.

[0104] One possible implementation of this invention involves adjusting the storage level of the key-value cache corresponding to the highest importance score if the cache is unavailable. This includes: if the cache is unavailable, determining whether the storage level corresponding to the key-value cache corresponding to the highest importance score is low-speed storage; if it is not low-speed storage, adjusting the storage level of the key-value cache corresponding to the highest importance score; if it is low-speed storage, discarding the key-value cache corresponding to the highest importance score.

[0105] Based on any of the above embodiments, see Figure 4 , Figure 4 This is a schematic diagram of a specific dynamic hierarchical allocation strategy provided in an embodiment of the present invention;

[0106] Among them, the High-Speed ​​Storage 0 (Level 0) counter (vnl0) is an abbreviation for the remaining storage capacity counter of High-Speed ​​Storage 0. High-Speed ​​Storage 0 is the highest priority storage level (such as GPU HBM+FP16 / BF16). The High-Speed ​​Storage 0 counter is used to record the remaining allocable storage capacity of this level in real time (in bytes or cache blocks).

[0107] In a sorting-based allocation strategy, if the currently processed attention head has completed resource allocation, the next attention head should be taken and the process should continue until all attention heads have been processed.

[0108] Specifically, after sorting the attention heads in descending order of importance, before allocating high-speed storage 0 resources to the highest-scoring attention head, the real-time remaining capacity of high-speed storage 0 needs to be read through the high-speed storage 0 counter. If the remaining capacity is sufficient to accommodate the KV cache block of the attention head (considering the storage space corresponding to the cache block size and precision), then Level 0 is allocated; if the capacity is insufficient, a downgrade allocation is triggered (such as switching to Level 1). This process needs to be correlated with the resource pool status in real time to avoid memory overflow caused by over-allocation.

[0109] If the highest-scoring attention head H has an importance score greater than θ_high, and vnl0 indicates that Level 0 has sufficient capacity, then after successful allocation, the next attention head with the second highest importance is taken; if the highest-scoring attention head H has an importance score greater than θ_high, but vnl0 indicates that Level 0 has insufficient capacity, then after being downgraded to Level 1 (or a lower level), the next attention head is taken.

[0110] If the importance score of the current attention head is ≤ θ_mid, it is directly assigned to Level 2 (discarded or stored on SSD), and then the next attention head is retrieved.

[0111] When all levels have no remaining capacity, an eviction policy is triggered (eviction of low-importance old cache blocks). After eviction, the current attention head is reassigned. If it still cannot be reassigned, the next attention head is taken. When low-speed storage is also full, the least important and least used cache blocks are evicted first.

[0112] Level 0 storage resources (such as GPU HBM) have a fixed and limited capacity. If they are allocated solely based on importance scores greater than θ_high, multiple high-importance attention heads may request them simultaneously, causing the total demand to exceed the actual capacity of Level 0 and triggering a memory overflow. By judging the availability of L0 resources, we can achieve "prioritizing the allocation of limited optimal resources to the most important attention heads": for example, if there are 5 attention heads in the current layer with importance scores greater than θ_high, but Level 0 can only accommodate 3, then the top 3 most important attention heads will be allocated first, and the remaining 2 high-importance attention heads will be downgraded to Level 1. This ensures that resources are not over-allocated and maximizes the value of Level 0 resources.

[0113] Furthermore, one possible implementation of this invention includes: obtaining the storage resource usage of each storage level in the hierarchical storage resource pool; and adjusting the hierarchical storage resource pool according to the storage resource usage and importance score of each storage level.

[0114] In this embodiment of the invention, based on the storage resource usage and importance score, cache eviction (eviction of the least important and least used block when storage is full), migration (e.g., migrating cache blocks with reduced importance from high-speed storage to low-speed storage), or precision adjustment (e.g., reducing the precision of cache blocks with reduced importance) strategies are implemented to facilitate the adjustment of the tiered storage resource pool.

[0115] Storage resource usage includes remaining capacity, and may also include: total capacity at each tier, used capacity, and resource load. Specifically, total capacity at each tier (e.g., GPU HBM total capacity 24GB, CPU DRAM total capacity 128GB), used capacity (e.g., GPUHBM used 18GB), and remaining capacity (e.g., GPU HBM remaining 6GB). Resource load includes: read / write bandwidth utilization at each tier (e.g., GPUHBM read / write bandwidth utilization 80%) and data migration queue length (e.g., CPU DRAM → GPU HBM migration queue has 3 cache blocks).

[0116] Storage resource usage can be collected in real time through hardware monitoring interfaces and software statistics modules. GPU driver interfaces (such as NVIDIA's NVML library) are called to obtain the used / remaining capacity and bandwidth usage of the GPU HBM; operating system interfaces (such as Linux's free / df commands) are called to obtain CPU DRAM and SSD capacity information. The cache manager's built-in statistics module records the number and size of key-value cache blocks at each level in real time (e.g., each INT8 precision key-value cache block is 1MB), calculating used capacity = Σ cache block size, and remaining capacity = total capacity - used capacity.

[0117] Furthermore, the collection frequency can be set and synchronized with the model inference time step (e.g., collect data once for every Token generated) to ensure data real-time performance.

[0118] One possible implementation of this invention involves the following: storage resource usage includes remaining capacity; storage tiers include low-speed storage, medium-speed storage, and high-speed storage; and the tiered storage resource pool is adjusted based on the storage resource usage and importance score of each storage tier, including: determining whether the remaining capacity of the storage tier is less than a preset capacity threshold; if the remaining capacity of the storage tier is less than the preset capacity threshold, then selecting target key-value caches that meet the adjustment requirements from the storage tiers based on their importance scores; and adjusting the tiered storage resource pool based on the target key-value caches that meet the adjustment requirements from the storage tiers.

[0119] The preset capacity thresholds for different storage tiers can be the same or different, and this embodiment of the invention does not limit them.

[0120] In one feasible approach, adjusting the tiered storage resource pool by selecting target key-value caches that meet the adjustment requirements from the storage tier includes: selecting target key-value caches that meet the adjustment requirements from the storage tier and adjusting the tiered storage resource pool according to the adjustment instructions.

[0121] The adjustment instructions correspond to the adjustment requirements.

[0122] In one possible scenario, if the adjustment instruction is a replacement instruction, the adjustment requirements are that the storage tier is low-speed storage and the importance score is less than the minimum importance score threshold.

[0123] Specifically, if the adjustment instruction is an elimination instruction, the storage level is the low-speed storage of the tiered storage resource pool. For the target key-value cache, the tiered storage resource pool is adjusted according to the adjustment instruction, including: sorting the target key-value cache according to the elimination instruction based on at least one of importance score and last access time; and eliminating the target key-value cache in turn based on the sorting result until the remaining capacity of the low-speed storage is not less than the first capacity.

[0124] For example, for cache eviction, when the remaining capacity of low-speed storage (such as SSD) is less than the first threshold (such as 10%), and there are cache blocks (key-value cache) with an importance score less than θ_low, all cache blocks with an importance score less than θ_low in the SSD are filtered; sorted according to the "Least Recently Used (LRU)" principle; the cache block with the highest priority is evictioned until the remaining capacity of the SSD is ≥15%.

[0125] In another possible scenario, if the adjustment instruction is a migration instruction, the adjustment requirement is that the storage tier is medium-speed storage or high-speed storage, and the importance score is reduced from the first range to the second range, where the first range is the importance score range corresponding to the current storage tier, and the second range is the importance score range corresponding to the next lower tier of the current storage tier.

[0126] If the adjustment instruction is a migration instruction, the storage level is medium-speed storage or high-speed storage in the tiered storage resource pool; for the target key-value cache, the tiered storage resource pool is adjusted according to the adjustment instruction, including: according to the migration instruction, the target key-value cache is migrated to the next level of storage, and the storage location of the target key-value cache in the metadata is updated.

[0127] Specifically, for cache migration, it is triggered when the remaining capacity of the high-speed storage GPU HBM is less than the second threshold (e.g., 5%), and there is a target key-value cache whose importance score drops from >θ_high to θ_mid~θ_high; or, when the remaining capacity of the CPU DRAM is less than the third threshold (e.g., 10%), and there is a target key-value cache whose importance score drops from θ_mid~θ_high to <θ_mid.

[0128] Select cache blocks that meet the importance reduction criteria, migrate them asynchronously (run in the background, without blocking inference), and implement GPUHBM → CPU DRAM (importance reduced to θ_mid~θ_high); CPU DRAM → SSD (importance reduced to <θ_mid); update the storage location in the metadata after the migration is completed.

[0129] Furthermore, during the migration process, the status of the migrated key-value cache is marked as a migration status, which is used to prohibit other operations; after the migration is completed, the status of the migrated key-value cache is marked as an available status.

[0130] In another possible scenario, if the adjustment instruction is a precision adjustment instruction, then the adjustment requirement is that the storage level is high-speed storage, and the importance score is reduced from the first range to the second range, and the precision is the preset precision. Here, the first range is the importance score range corresponding to the current storage level, and the second range is the importance score range corresponding to the next lower level of the current storage level.

[0131] In one possible scenario, if the adjustment instruction is a precision adjustment instruction, then the storage tier is the high-speed storage of the tiered storage resource pool; correspondingly, for the target key-value cache, the tiered storage resource pool is adjusted according to the adjustment instruction, including: reducing the storage precision of the target key-value cache according to the precision adjustment instruction, replacing the original target key-value cache, and updating the precision information of the target key-value cache in the metadata.

[0132] For precision adjustment, if the remaining capacity of CPU DRAM is less than the fourth threshold (e.g., 8%), filter the cache blocks in CPU DRAM with an importance score less than θ_mid and a precision of the preset precision (e.g., INT8), perform quantization from INT8 to INT4 (based on the latest statistics of this attention head), replace the original cache block, update the precision information in the metadata, and release memory (INT4 saves 50% of the capacity compared to INT8).

[0133] In this embodiment of the invention, precision conversion, using quantization based on statistics for a single attention head or a group of similar attention heads (such as min / max, mean / std), may be more accurate than global quantization.

[0134] The smaller the quantization granularity, the more closely the statistical information matches the characteristics of the attention head, and the less the accuracy loss.

[0135] Understandably, global quantization is based on the statistics of the entire KVCache (such as the max / min of the KV vectors of all attention heads and all tokens), without considering the differences between attention heads. For example, if the KV vector range of attention head A is [0,10] and the range of attention head B is [0,1000], then the global quantization max=1000 and min=0. When the vector of attention head A (e.g., 5) is quantized to INT8, the mapping value = 5×255 / 1000≈1.275, rounded to 1, the error rate = (5-1×1000 / 255) / 5≈60%; after quantization, the mapping value of the vector of attention head B (e.g., 500) is 500×255 / 1000=127.5, rounded to 128, the error rate ≈0.2%. The quantization error of low-range attention heads is extremely large, affecting the model accuracy.

[0136] In this invention, attention head-level quantization is employed. Based on the statistical information of a single attention head or similar attention heads (e.g., max=10 for attention head A, max=1000 for attention head B), the quantization error is significantly reduced. For example, quantization of attention head A: 5×255 / 10=127.5 → rounded to 128, after dequantization = 128×10 / 255≈5.02, error rate ≈0.4%. Quantization of attention head B: 500×255 / 1000=127.5 → rounded to 128, after dequantization = 128×1000 / 255≈501.96, error rate ≈0.39%. Similar attention heads (e.g., attention heads both responsible for entity recognition) have similar KV vector distributions; sharing statistical information balances accuracy and computational cost, while avoiding the "one-size-fits-all" error of global quantization.

[0137] This invention also supports dynamic quantization (per batch / per sequence) or static quantization (based on representative data calibration).

[0138] Based on any of the above embodiments, see Figure 5 , Figure 5 This is a flowchart of the inference steps for cache migration and precision adjustment provided in an embodiment of the present invention, including: determining whether the migration / adjustment triggering condition has been met.

[0139] If the condition is triggered, the cache metadata is scanned and sorted by importance. The objects sorted by importance are all KV cache blocks stored in the current layer (i.e., cache blocks generated in the historical time step), and the sorting dimension is "importance score re-evaluated in the current time step" (score not at the initial allocation).

[0140] Determine if migration is needed; if migration is needed, perform hierarchical migration. If migration is not needed, determine if precision adjustment is needed. If so, perform precision conversion; otherwise, end the adjustment process.

[0141] If the condition is not triggered, proceed to the next token, calculate the current layer attention, generate a new token, and store the new key-value data.

[0142] Load the input sequence, initialize the key-value buffer, calculate the attention of the current layer, generate new lexical units, and store the new key-value data.

[0143] In this embodiment of the invention, the initial allocation is based on the importance score and resource status of the current time step. However, the importance and resource status will change dynamically during inference, requiring migration / adjustment to adapt to the changes. Specific reasons include: the importance score of the same attention head changes with the input token (e.g., an attention head is highly important when processing "topic sentence" but less important when processing "modifiers"). The initially allocated Level 0 resources may become "unnecessary" in subsequent time steps and need to be migrated to Level 1 to release high-speed resources; conversely, if the importance of an attention head initially allocated to Level 1 increases, it needs to be migrated to Level 0 to ensure inference accuracy. When the storage capacity of a certain level (e.g., Level 0) becomes tight due to cache writes from other attention heads, some cache blocks with decreased importance need to be migrated to lower levels to avoid the inability to allocate new high-importance attention heads; if the lower-level storage (e.g., Level 1) is full, some low-importance cache blocks need to have their precision reduced (e.g., from INT8 to INT4) or discarded to release storage space. Autoregressive inference typically involves multiple time steps (e.g., generating 100 tokens). If only the initial allocation is relied upon, low-level high-importance cache blocks will continuously cause access latency. By migrating / adjusting, the "high importance - high speed and high precision" matching relationship can be maintained in the long term, optimizing the overall inference speed.

[0144] In this embodiment of the invention, the migration judgment is based on the following criteria: combining the "reassessment of importance score" and the "storage status of each level", the specific judgment rules are as follows: if the current importance score of the cache block is > θ_high, and Level 0 has remaining capacity → trigger "migrate from the current level to Level 0"; if the current importance score of the cache block is ≤ θ_mid, and it is in Level 0 / Level 1 → trigger "migrate from the current level to Level 2 (or reduce precision, such as from FP16 to INT4)"; if the current importance score of the cache block is between θ_mid and θ_high, and Level 1 has capacity while Level 0 has no capacity → trigger "migrate from Level 0 to Level 1" (if it is currently in Level 2, then migrate to Level 1); if the storage capacity of a certain level (such as Level 1) is full, and there is a cache block with an importance score < θ_low → trigger "migrate from that level to Level 2 (discard)".

[0145] The current layer attention calculation follows the Transformer multi-attention head attention mechanism. The specific steps are as follows: Perform a linear transformation on the embedding vector of the current input token to generate a Query vector; retrieve the KV cache block stored in the historical time step (after inverse quantization) to obtain the Key vector and Value vector; calculate the dot product of the Query and Key and scale (divide by ). (where d_k is the dimension of the Key vector) to obtain the original attention score; perform Softmax normalization on the original attention score to obtain the normalized attention score; use the normalized attention score to weighted sum the Value vector to obtain the attention output (attn_output) of the current attention head; after linear transformation of the attention outputs of all attention heads, combine them with the output of the feedforward network to complete the inference of the current layer.

[0146] After the attention calculation of the current layer, the output is the core result of the current layer model inference. Finally, the probability distribution of the next token is generated through the softmax of the output layer.

[0147] Furthermore, see Figure 6 , Figure 6This is a schematic diagram illustrating the interaction between the hierarchical storage resource pool and the cache manager provided in this embodiment of the invention. Specifically, this embodiment employs real-time scheduling for several reasons: First, the remaining capacity of each storage level changes in real time with the writing, eviction, and migration of the KV cache (e.g., Level 0's capacity decreases due to new cache writing, and increases due to the migration of old cache to Level 1). Real-time scheduling can adjust the allocation strategy based on the latest capacity, avoiding failures caused by allocation based on expired resource status. Second, the importance score of the same attention head fluctuates with the input sequence (e.g., a certain attention head is highly important when processing "entity recognition" related tokens, but less important when processing "punctuation marks"). Real-time scheduling can trigger the migration of cache blocks between different levels (e.g., migrating from Level 0 to Level 1 when importance decreases), optimizing resource utilization. Third, if the model processes multiple inference tasks simultaneously (e.g., multiple user requests), the KV caches of different tasks will compete for the same hierarchical resource pool. Real-time scheduling can dynamically coordinate the resource allocation of each task based on task priority and resource status, avoiding resource contention between tasks.

[0148] Furthermore, it is understood that the technical solution of the present invention can achieve dynamic evaluation of the importance of attention head granularity; flexible precision, hierarchical storage resource pool; and state awareness, dynamic allocation and scheduling mechanism.

[0149] Dynamic evaluation of attention head importance is implemented. Based on changes in the importance score of the attention head, the data migration engine asynchronously performs the migration of KV cache blocks between different storage levels in the background (e.g., from CPU DRAM to GPU HBM when importance increases, and vice versa when importance decreases). This avoids resource waste caused by fixed storage levels and ensures that high-importance cache blocks are always stored at high speed, while low-importance cache blocks release high-speed resources.

[0150] For elastic precision, a tiered storage resource pool (location + precision) is used. For a single or group of similar attention heads, the numerical precision of the KV cache is dynamically adjusted (e.g., FP16 for high-importance attention heads and INT4 for low-importance attention heads). Precision conversion is based on the statistical information of the attention heads (e.g., min / max, mean / std) rather than global uniform quantization. While ensuring the precision of high-importance attention heads, the memory footprint of low-importance attention heads is minimized, and the precision loss is far lower than that of uniform quantization.

[0151] For state awareness, a dynamic allocation and scheduling mechanism is implemented. A resource monitor collects real-time data on remaining capacity, cache block metadata (location, precision, importance), and migration / transformation status for each storage tier, providing real-time data support for the allocation decision-maker and scheduler. This ensures that resource allocation and scheduling are based on the latest state, avoiding over-provisioning and misallocation, and improving the stability and efficiency of the entire cache management system.

[0152] Furthermore, the determination of importance scores will be elaborated below.

[0153] One possible implementation of this invention is to calculate the importance score of the target attention head in the current layer in real time, which includes: calculating the importance score of the target attention head in the current layer in real time based on the input sequence of model inference of the target attention head in the current layer.

[0154] One possible implementation of this invention is to calculate the importance score of the target attention head in the current layer in real time, including: if the calculation mode is a single-selection mode, then calculate the importance score of the target attention head in the current layer in real time according to the target single-item calculation mode corresponding to the single-selection mode; if the calculation mode is a weighted comprehensive mode, then perform weighted calculation based on the values ​​calculated by each single-item calculation mode to obtain the importance score of the target attention head in the current layer.

[0155] Among them, the single-item calculation modes include: attention-based output mode, attention-based score mode, historical information-based mode, and lightweight model mode.

[0156] Attention-based approaches calculate the norm (L1, L2), entropy, or gradient of the attention vector (attn_output) output by the attention head (requiring a lightweight approximation). Attention-score-based approaches analyze the distribution (e.g., entropy, sparsity, maximum value) of the attention scores (attn_weights) generated by the attention head. More concentrated or dispersed attention may indicate its importance. Historical information-based approaches combine the importance scores of the attention head in previous time steps for smoothing or prediction. Lightweight models train a very small auxiliary model (e.g., an MLP) to quickly predict the importance scores of each attention head based on the current layer's input, state, or the evaluation results of the previous time step. The model can be trained offline.

[0157] The calculation process based on the attention output pattern includes: inputting the input sequence into the corresponding attention head to obtain the output vector; determining the vector norm based on the output vector; obtaining the attention score matrix of the attention head when inferring the input sequence; calculating the entropy of each row of the attention score matrix; determining the entropy of the attention head based on the entropy of each row; and determining the first importance score based on the vector norm and / or the entropy of the attention head.

[0158] Specifically, the L2 norm is calculated based on the attention output pattern. This is based on the output vector of the current attention head. (d is the vector dimension, such as 512) Calculate the initial norm (S-norm) using a preset formula, which includes: Where d represents the dimension of the vector, and xi represents the component of the vector in the i-th dimension. Mapping the S-norm to [0,1], the formula is: S-norm(normalized) = (S-norm - min_S) / (max_S - min_S); where min_S is the minimum L2 norm of all attention heads in the current layer, and max_S is the maximum L2 norm of all attention heads in the current layer; S-norm(normalized) is the vector norm.

[0159] Entropy calculation based on attention scores. This is based on the attention score matrix of the current attention head. (t is the number of generated tokens, e.g., 100), each row represents the attention weight of the current token to historical tokens. First, perform Softmax normalization on each row to obtain p_{i,j} (the normalized weight of the i-th token to the j-th token). Calculate the average entropy of each row as the entropy score of that attention head, where... , Let i be the entropy of the i-th row. Normalization and inverse mapping: The smaller the entropy value (the more focused the attention), the higher the importance of the attention head. Therefore, inverse mapping is required: S-entropy value (normalized) = 1 - (S-entropy value - min_H) / (max_H - min_H), where min_H is the minimum entropy value of all attention heads in the current layer, max_H is the maximum entropy value of all attention heads in the current layer, and S-entropy value (normalized) is the entropy of the attention head.

[0160] The vector norm or the entropy of the attention head can be used as the first importance score; alternatively, the two can be weighted to obtain the first importance score.

[0161] The calculation process based on the attention score pattern includes: converting the input sequence into query vectors and key vectors; splitting the query vectors and key vectors according to the number of attention heads; and determining the attention score based on the query vectors and key vectors of the attention heads, which serves as the second importance score.

[0162] The attention score is generated by the "Query-Key interaction" of the Transformer multi-attention head attention mechanism, specifically through the following steps: a linear transformation generates a query vector (Query) and a key vector (Key); based on the input sequence vector of the current layer... (t is the number of tokens, dmodel is the model dimension, such as 768). Through 3 independent linear layers (weight matrix) Where dk = dmodel / h, h is the number of attention heads), the input sequence vector X is converted into Query(Q) and Key(K) respectively, where Q = X × W_Q; K = X × W_K; Q and K are split according to the number of attention heads h, and each attention head... (For example, if h=12 and d_model=768, then d_k=64). Calculate the dot product of Q and K for each attention head, then divide by... (To avoid softmax saturation due to excessively large scores), among which, , For autoregressive tasks (such as text generation), add a "lower triangle mask" to ensure the current token only focuses on historical tokens (to prevent future information leakage), and set the score at the mask position to -∞. For the original score... Perform Softmax normalization to obtain the final attention score attn_weights_h (each row sums to 1, representing the attention weight between tokens), where attn_weights_h = Softmax(raw_scores_h + mask).

[0163] The calculation process based on historical information patterns includes: obtaining the historical importance score of the attention head; predicting the importance score of the attention head based on the historical importance score of the attention head, which is then used as the third importance score.

[0164] The calculation process of the lightweight model mode includes: based on the input sequence of the attention head, the state of the current layer, and the evaluation result of the previous time step, using the multilayer perceptron model, predicting the importance score of the target attention head in the current layer, which is used as the fourth importance score.

[0165] Specifically, based on the local features of the current attention head, a multilayer perceptron model is used to predict the importance score of the target attention head in the current layer, which is then used as the fourth importance score.

[0166] Specifically, the local features of the current attention head include: the input sequence, the state of the current layer, and the evaluation result of the previous time step. Specifically, this includes the mean embedding vector of the input tokens in the input sequence of the attention head (e.g., the average embedding vector of the input tokens at the current time step), and the L1 norm of the attention head's query vector (representing the activation strength of the query). The state of the current layer is the "global state" of the model, including: the normalized mean of the current layer (reflecting the activation level of the current layer) and the number of tokens generated (reflecting the length of the context window). The evaluation result of the previous time step (Prev_Score) represents the importance score of the attention head at the previous inference time step (e.g., the overall score at time t-1).

[0167] In one feasible approach, the lightweight auxiliary model (multilayer perceptron model) can adopt a "2-layer MLP (Multilayer Perceptron)" structure, with offline training and online prediction. Specifically, offline training (based on labeled datasets) includes: collecting inference data for different tasks (text generation, question answering), recording the "input + state + previous time step score" of each attention head, and labeling it with the "contribution of this attention head to the final output" (e.g., the decrease in model accuracy after removing this attention head). The input sequence is concatenated (e.g., 32-dimensional) and input into the MLP (64 neurons in the first layer, ReLU activation; 1 neuron in the second layer, Sigmoid activation), minimizing the MSE loss between the prediction score and the label. Online prediction (during inference) includes: real-time acquisition of the "input + state + previous time step score" of the current attention head and standardization (based on the mean and standard deviation of the training set). The standardized features are input into the pre-trained MLP, outputting the importance score S in the [0,1] interval to assist the model.

[0168] See Figure 7 , Figure 7 This is a flowchart of a dynamic evaluation method for the importance of attention heads provided in an embodiment of the present invention. The example calculates the importance score based on the attention output norm and historical scores.

[0169] Obtain the current attention head data; calculate the L2 norm of the current attention output; determine if it is the first word;

[0170] If it is the first token, initialize the historical score: current norm value; set the importance score = current norm; output the importance score;

[0171] If it is not the first token, read the historical score (exponential moving average); calculate the smoothed score: Update historical score storage; Set importance score = smoothing score; Output importance score;

[0172] Determine if all attention heads have been evaluated. If yes, end the evaluation; otherwise, move to the next attention head.

[0173] Understandably, in autoregressive generation, the first token refers to the first generated token in the sequence. Since there is no historical KV cache at this point, importance assessment cannot rely on historical information and must be handled separately (e.g., using an initial threshold or default value).

[0174] The first input token received during the model inference process. For example, in text generation tasks, the first token in the prefix text input by the user (e.g., the first token corresponding to the input "write a short article" is "write"); in long sequence inference tasks, the first token in the input long document (e.g., the first token "contract" in a legal contract).

[0175] The first token has no historical KV cache, so the importance assessment does not need to combine historical scores (the current L2 norm is used directly as the importance score).

[0176] The smoothing score addresses the issue of excessively fluctuating importance scores for subsequent tokens after the first token. For example, if an attention focus has a score of 1.0 (high importance) for the first token, but the score for the second token drops to 0.3 (low importance) due to irrelevant input (such as punctuation), the smoothing score can mitigate this fluctuation by using historical scores (1.0), avoiding frequent adjustments to resource allocation. The larger the α value, the stronger the influence of the current time step's features on the smoothing score, and the faster the adaptation to input changes; the smaller the α value, the higher the weight of historical scores, and the more significant the smoothing effect. In practical applications, α is typically set to 0.3~0.5 (e.g., α=0.4) to achieve a balance between dynamic adaptation and stability.

[0177] The specific value of the historical score weighting coefficient α is determined by the task's "input volatility": For tasks with high input volatility (such as dialogue generation, where input topics switch frequently): α is set to 0.5, enhancing the weight of the current score and quickly adapting to input changes. For tasks with low input volatility (such as document summarization, where input topics are stable): α is set to 0.3, enhancing the weight of historical scores and maintaining score stability.

[0178] Furthermore, the entropy value of the attention weights can be calculated, and the overall score can be calculated by combining the norm and the entropy value.

[0179] The overall score is calculated using a weighted summation method, which fully integrates the contribution of attention output (norm representation) and the focus of attention (entropy representation), as shown in the following formula: .

[0180] The L2 norm of the attention head output vector (attn_output) represents the contribution of that attention head to the current inference; the larger the norm, the higher the contribution. It is then mapped to the [0,1] interval (e.g., through Min-Max normalization to eliminate numerical scale differences between different attention heads). The smaller the distribution entropy value of the attention score (attn_weights), the more focused the attention is on the key token, and the higher the importance of the attention head. Then, "1 - normalized entropy value" is calculated (converting the "negative correlation" of entropy values ​​to "positive correlation," i.e., the smaller the entropy value, the larger the term), also mapped to the [0,1] interval. Weight settings: and Weighting coefficients (satisfying) (This can be adjusted according to the characteristics of the task, such as in text generation tasks.) Prioritize the contribution of the attention head in long sequence understanding tasks. To balance contribution and focus.

[0181] One possible implementation of this invention further includes: when accessing the key-value cache generated by the target attention head, determining whether the storage precision of the key-value cache is the precision required for calculation; if not, performing an inverse quantization operation based on the storage location and storage precision corresponding to the target attention head.

[0182] When subsequent inference steps require access to this KVCache, it is retrieved based on its storage location and precision, and dequantization is performed if necessary. In one possible scenario, when retrieving the KVCache from low-precision storage (e.g., INT8 / INT4) for subsequent attention calculations, it needs to be dequantized to the precision required for model computation (e.g., FP16). Linear quantization (Min-Max quantization) can be used. ; where min and max are the original numerical ranges of the attention head (or the group), which can be saved during quantization.

[0183] Specifically, when the retrieved KVCache is stored in "low-precision" format and "subsequent inference requires high-precision computation," dequantization needs to be performed. For example, the KVCache of medium-importance attention heads is stored in CPU DRAM with INT8 precision, while low-importance attention heads are stored in SSD with INT4 precision. Furthermore, the attention computation of the Transformer model (the dot product of the query and key) requires uniform precision (usually FP16 / BF16), so low-precision KV vectors need to be converted to FP16 / BF16 before participating in the computation.

[0184] Dequantization is performed based on the "attention head statistics recorded at storage." Taking INT8→FP16 dequantization as an example, it includes: Step 1: Reading metadata. The cache manager obtains the quantization parameters of the attention head KVCache from the metadata (recorded at storage): including the maximum value (max_val) and minimum value (min_val) of the attention head KV vector (obtained based on statistics of all elements within the attention head). Step 2: Performing dequantization calculation. The formula is as follows: Where max_val / min_val: the maximum and minimum values ​​of the original FP16 data; the INT8 value range is 0-255, first mapped to -127.5-127.5, then scaled to the original FP16's max_val-min_val range. Step 3: Precision Alignment. The dequantized FP16 vector is transferred to the GPU and aligned with the current Query vector (FP16) in precision for attention calculation.

[0185] Based on any of the above embodiments, one or more embodiments for implementing the above method are described in detail, including: 1. The specific Transformer model used (such as LLaMA, GPT-NeoX).

[0186] 2. Specific algorithms for importance assessment (e.g., using the L2 norm of attn_output as the initial score, combined with exponential moving average smoothing).

[0187] 3. Define the specific levels of the hierarchical resource pool (e.g., Level 0 - GPU HBMFP16; Level 1 - CPU DRAMINT8; Level 2 - Discard).

[0188] 4. Specific rules for allocation strategy (e.g.: score > θ_high -> Level 0; θ_mid < score <= θ_high -> Level 1; score <= θ_mid -> Level 2).

[0189] 5. KVCache storage format (how to organize data of different precision and different positions).

[0190] 6. Specific implementation of the cache manager (metadata structure, migration triggering conditions, eviction algorithm).

[0191] 7. Specific implementation of quantization / dequantization (e.g., using per-attention dynamic min-max quantization to INT8).

[0192] Specifically, a dynamic hierarchical key-value cache management method based on attention head importance awareness is applied to the Transformer model inference process. This method includes: calculating the importance score of each attention head in the current layer in real time during model inference; defining a hierarchical storage resource pool containing multiple storage locations and numerical precisions; dynamically deciding which level in the hierarchical storage resource pool the key-value cache generated by each attention head should be allocated to based on its importance score, with the level specifying the storage location and numerical precision; storing the key-value cache at the decision-specified storage location with the specified numerical precision; and performing retrieval and necessary data transformation based on its storage location and numerical precision when subsequent inference steps require access to the key-value cache.

[0193] Furthermore, the method for real-time calculation of the importance score of the attention head includes at least one of the following: calculating the norm or entropy of the attention vector output by the attention head, calculating the distribution characteristics of the attention score generated by the attention head, smoothing or predicting based on the historical importance score of the attention head, and making predictions using a lightweight auxiliary model.

[0194] Furthermore, the tiered storage resource pool includes at least the following: Tier 1 (high-speed storage): high-speed graphics processor memory combined with floating-point 16 or 16-bit precision; Tier 2 (medium-speed storage): central processing unit memory combined with integer 8-bit precision; Tier 3 (low-speed storage): discarded or solid-state drive storage combined with integer 4-bit or lower precision.

[0195] Furthermore, dynamic decision-making employs either an interval division method based on importance score thresholds or a priority allocation method based on the importance ranking of attention points within the same layer.

[0196] Furthermore, it also includes cache management steps: based on the storage resource usage status and changes in attention importance scores, performing operations such as eviction, migration between different storage location levels, or adjustment of numerical precision for stored key-value caches.

[0197] Furthermore, the numerical precision conversion employs a quantization method based on statistical information from a single attention head or a group of similar attention heads.

[0198] Furthermore, it includes: an importance assessment module, configured to calculate the importance score of each attention head in real time; a hierarchical resource pool module, providing storage resources with multiple storage locations and multiple numerical precisions; an allocation decision-maker module, configured to dynamically decide the storage level of the key-value cache based on the importance score; and a cache manager module, configured to manage the storage, retrieval, metadata maintenance, and execution of eviction, migration, and precision adjustment operations of the key-value cache.

[0199] As can be seen, the above scheme can significantly save memory. Specifically, by storing or even discarding a large number of low-importance attention heads in the KVCache with low precision, or moving them to inexpensive, low-speed storage, the overall KVCache memory usage can be reduced by 30%-70% (the specific percentage depends on the model, task, resource pool definition, and the aggressiveness of the strategy), far exceeding the effect of uniform quantization (typically saving 50% but with significant precision loss) or simple discarding. It can also maintain model accuracy. By protecting the high precision and fast access of high-importance attention heads, the loss of key information is minimized, and the decline in model output quality (such as the fluency of generated text, factual accuracy, and task performance) is far less than that of uniform compression methods with equivalent memory savings. Experiments can compare metrics such as perplexity, BLEU, and accuracy. Inference can also be accelerated. Reducing GPU memory usage can decrease memory contention and potentially improve core computational efficiency. Smaller memory usage may allow for larger batch sizes or longer sequence processing. (If high-speed storage resources are sufficient) Ensure the fastest access to key attention heads. Operations offloading to CPU / Disk should be handled with caution, as their latency may offset some of the benefits. The optimization lies in the fact that only low-importance attention heads endure high-latency access, and these attention heads themselves have little impact on the output. It is more flexible, as the method can adapt to different model architectures, different hardware platforms (servers, edge devices), and different task requirements (by adjusting policy parameters to balance memory / speed / accuracy). Resource optimization is achieved by precisely allocating the most expensive resources (high-speed GPU memory, high-precision computation) to the computational parts that have the greatest impact on the output, thereby improving resource utilization.

[0200] Figure 8 A schematic diagram of a cache management device provided in an embodiment of the present invention includes: an importance score determination module 210, used to calculate the importance score of a target attention head in the current layer in real time during model inference, wherein the target attention head is any one of multiple attention heads in the current layer; a storage level determination module 220, used to determine the storage level corresponding to the key-value cache generated by the target attention head according to the importance score of the target attention head in the current layer, wherein the storage level includes storage location and storage precision; and a storage module 230, used to store the key-value cache according to the storage location and storage precision corresponding to the key-value cache generated by each attention head.

[0201] In one possible implementation of this invention, the storage module 230 is configured to: sort the importance scores of all attention heads in the current layer to obtain a first sorting result; determine the key-value cache with the highest importance score according to the sorting result; determine whether the storage level corresponding to the key-value cache with the highest importance score is available; if available, store the key-value cache with the highest importance score in the corresponding cache location according to the storage precision; if unavailable, adjust the storage level corresponding to the key-value cache with the highest importance score; and store the key-value cache according to the adjusted storage location and storage precision corresponding to the key-value cache with the highest importance score; after storing the key-value cache with the highest importance score, process the next key-value cache sequentially until the storage of the key-value cache generated by each attention head is completed.

[0202] In one possible implementation of this invention, the storage module 230 is configured to: if the storage level corresponding to the key-value cache with a high importance score is high-speed storage, determine whether the medium-speed storage corresponding to the storage level corresponding to the key-value cache with the highest importance score is available; if the medium-speed storage is available, downgrade the storage level corresponding to the key-value cache with the highest importance score to medium-speed storage; if the medium-speed storage is unavailable, downgrade the storage level corresponding to the key-value cache with the highest importance score to low-speed storage.

[0203] In one possible implementation of this invention, the storage module 230 is configured to: if the storage level corresponding to the key-value cache with a high importance score is medium-speed storage, determine whether the low-speed storage of the storage level corresponding to the key-value cache with the highest importance score is available; if the low-speed storage is available, downgrade the storage level corresponding to the key-value cache with the highest importance score to low-speed storage; if the low-speed storage is unavailable, upgrade the storage level corresponding to the key-value cache with the highest importance score to high-speed storage.

[0204] In one possible implementation of this invention, the storage module 230 is configured to: if unavailable, determine whether the storage level corresponding to the key-value cache with the highest importance score is low-speed storage; if not low-speed storage, adjust the storage level corresponding to the key-value cache with the highest importance score; if it is low-speed storage, discard the key-value cache with the highest importance score.

[0205] One possible implementation of this invention further includes: a management module, configured to: obtain the storage resource usage of each storage tier of the tiered storage resource pool; and adjust the tiered storage resource pool according to the storage resource usage and importance score of each storage tier.

[0206] One possible implementation of this invention includes storage resource usage: remaining capacity; storage tiers including low-speed storage, medium-speed storage, and high-speed storage; and a management module configured to: determine whether the remaining capacity of a storage tier is less than a preset capacity threshold; if the remaining capacity of a storage tier is less than the preset capacity threshold, then filter target key-value caches that meet the adjustment requirements in the storage tier based on their importance scores; and adjust the tiered storage resource pool based on the target key-value caches that meet the adjustment requirements in the storage tier.

[0207] One possible implementation of this invention is a management module, which is used to: filter target key-value caches that meet the adjustment requirements in the storage hierarchy, and adjust the hierarchical storage resource pool according to the adjustment instructions.

[0208] In one possible implementation of this invention, if the adjustment instruction is an elimination instruction, then the storage level is a low-speed storage of a tiered storage resource pool. The management module is used to: sort the target key-value cache according to the elimination instruction based on at least one of importance score and last access time; and based on the sorting result, sequentially eliminate the target key-value cache until the remaining capacity of the low-speed storage is not less than the first capacity.

[0209] In one possible implementation of this invention, if the adjustment instruction is a migration instruction, the storage level is a medium-speed storage or high-speed storage in a tiered storage resource pool. The management module is used to: migrate the target key-value cache to the next level of storage according to the migration instruction, and update the storage location of the target key-value cache in the metadata.

[0210] In one possible implementation of this invention, the management module is further configured to: during the migration process, mark the status of the migrated key-value cache as a migration status, the migration status being used to prohibit other operations; and after the migration is completed, mark the status of the migrated key-value cache as an available status.

[0211] In one possible implementation of this invention, if the adjustment instruction is a precision adjustment instruction, then the storage level is a high-speed storage of a tiered storage resource pool; the management module is used to: reduce the storage precision of the target key-value cache according to the precision adjustment instruction, replace the original target key-value cache, and update the precision information of the target key-value cache in the metadata.

[0212] In one possible implementation of this invention, the importance score determination module 210 is used to: calculate the importance score of the target attention head in the current layer in real time based on the input sequence of model inference of the target attention head in the current layer.

[0213] In one possible implementation of this invention, the importance score determination module 210 is used to: if the calculation mode is a single-selection mode, calculate the importance score of the target attention head in the current layer in real time according to the target single-item calculation mode corresponding to the single-selection mode; if the calculation mode is a weighted comprehensive mode, perform weighted calculation based on the values ​​calculated by each single-item calculation mode to obtain the importance score of the target attention head in the current layer.

[0214] One possible implementation of this invention includes a single-item calculation mode comprising: an attention output mode, an attention score mode, a historical information mode, and a lightweight model mode. The calculation process based on the attention output mode includes: inputting the input sequence into the corresponding attention head to obtain an output vector; determining the vector norm based on the output vector; obtaining the attention score matrix of the attention head during inference of the input sequence; calculating the entropy of each row of the attention score matrix; determining the entropy of the attention head based on the entropy of each row; and determining a first importance score based on the vector norm and / or the entropy of the attention head. The calculation process based on the attention score mode includes: inputting the input sequence into the corresponding attention head to obtain an output vector; determining the vector norm and / or the entropy of the attention head; and determining the first importance score based on the vector norm and / or the entropy of the attention head. The columns are converted into query vectors and key vectors; the query vectors and key vectors are split according to the number of attention heads; the attention score is determined based on the query vector and key vector of the attention head, which is used as the second importance score; the calculation process based on historical information patterns includes: obtaining the historical importance scores of the attention heads; predicting the importance score of the attention heads based on the historical importance scores, which is used as the third importance score; the calculation process of the lightweight model pattern includes: predicting the importance score of the target attention head in the current layer based on the input sequence of the attention head, the state of the current layer, and the evaluation result of the previous time step, using a multilayer perceptron model, which is used as the fourth importance score.

[0215] One possible implementation of this invention further includes: a dequantization module, configured to: determine whether the storage precision of the key-value cache is the required precision for calculation when accessing the key-value cache generated by the target attention head; if not, perform dequantization operation according to the storage location and storage precision corresponding to the target attention head.

[0216] Figure 8 The description of the features in the corresponding embodiments can be found in [reference needed]. Figure 2 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0217] Figure 9 A structural diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 9 As shown, the electronic device includes: a memory 60 for storing a computer program; and a processor 61 for implementing the cache management method as described in the above embodiment when executing the computer program.

[0218] The electronic devices provided in this embodiment may include, but are not limited to, smartphones, tablets, laptops, or desktop computers.

[0219] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0220] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the cache management method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc.

[0221] In some embodiments, the electronic device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0222] Those skilled in the art will understand that Figure 9 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0223] It is understood that if the cache management method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk, or optical disk, and other media capable of storing program code.

[0224] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the cache management method described above.

[0225] Based on this, embodiments of the present invention also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described cache management method.

[0226] The foregoing has provided a detailed description of a cache management method, apparatus, device, medium, and product provided by embodiments of the present invention. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0227] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0228] The foregoing has provided a detailed description of the cache management method, apparatus, device, medium, and product provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A cache management method, characterized in that, include: During model inference, the importance score of the target attention head in the current layer is calculated in real time. The target attention head is any one of multiple attention heads in the current layer. Based on the importance score of the target attention head in the current layer, the storage level corresponding to the key-value cache generated by the target attention head is determined, and the storage level includes storage location and storage precision; Store the key-value cache according to the storage location and storage precision corresponding to the key-value cache generated by each attention head; Based on the storage location and precision of the key-value cache generated by each attention head, the key-value cache is stored, including: Sort the importance scores of all attention heads in the current layer to obtain the first sorting result; Based on the first sorting result, determine the key-value cache with the highest importance score; Determine whether the storage level corresponding to the key-value cache with the highest importance score is available; If available, the key-value cache with the highest importance score will be stored in the corresponding cache location according to the storage precision. If unavailable, the storage level corresponding to the key-value cache with the highest importance score is adjusted; and the key-value cache is stored according to the adjusted storage location and storage precision based on the level corresponding to the key-value cache with the highest importance score. After storing the key-value cache with the highest importance score, process the next key-value cache in sequence until storing the key-value cache generated by each attention head is completed.

2. The cache management method according to claim 1, characterized in that, Adjust the storage level of the key-value cache corresponding to the highest importance score, including: If the storage level corresponding to the key-value cache with the high importance score is high-speed storage, then determine whether the medium-speed storage of the storage level corresponding to the key-value cache with the highest importance score is available; If medium-speed storage is available, the storage tier corresponding to the key-value cache with the highest importance score is downgraded to medium-speed storage; If medium-speed storage is unavailable, the storage tier corresponding to the key-value cache with the highest importance score is downgraded to low-speed storage.

3. The cache management method according to claim 1, characterized in that, Adjust the storage level of the key-value cache corresponding to the highest importance score, including: If the storage level corresponding to the key-value cache with the high importance score is medium-speed storage, then determine whether the low-speed storage of the storage level corresponding to the key-value cache with the highest importance score is available; If low-speed storage is available, the storage level corresponding to the key-value cache with the highest importance score is downgraded to low-speed storage; If low-speed storage is unavailable, the storage level corresponding to the key-value cache with the highest importance score is upgraded to high-speed storage.

4. The cache management method according to claim 1, characterized in that, If unavailable, the storage level corresponding to the key-value cache with the highest importance score will be adjusted, including: If unavailable, determine whether the storage level corresponding to the key-value cache with the highest importance score is low-speed storage; If it is not low-speed storage, then the storage level corresponding to the key-value cache with the highest importance score will be adjusted. The method further includes: If the storage is slow, the key-value cache with the highest importance score is discarded.

5. The cache management method according to claim 1, characterized in that, Also includes: Obtain the storage resource usage status of each storage tier in the tiered storage resource pool; Adjust the tiered storage resource pools based on the storage resource usage and importance score of each storage tier.

6. The cache management method according to claim 5, characterized in that, Storage resource usage includes: remaining capacity; storage tiers include: low-speed storage, medium-speed storage, and high-speed storage; Adjust the tiered storage resource pools based on storage resource usage and importance scores at each storage tier, including: Determine whether the remaining capacity of the storage tier is less than a preset capacity threshold; If the remaining capacity of a storage tier is less than a preset capacity threshold, then target key-value caches that meet the adjustment requirements in the storage tier are selected based on their importance scores. Based on the target key-value caches that meet the adjustment requirements in the storage tier, adjust the tiered storage resource pool.

7. The cache management method according to claim 6, characterized in that, Based on the selection of target key-value caches that meet the adjustment requirements in the storage tier, the tiered storage resource pool is adjusted, including: Based on the target key-value cache that meets the adjustment requirements in the storage tier, the tiered storage resource pool is adjusted according to the adjustment instructions.

8. The cache management method according to claim 7, characterized in that, If the adjustment instruction is a replacement instruction, then the storage tier is the low-speed storage of the tiered storage resource pool. For the target key-value cache, adjust the tiered storage resource pool according to the adjustment instructions, including: Based on the eviction instructions, sort the target key-value cache according to at least one of the importance score and the last access time; Based on the sorting results, the target key-value cache is evicted in turn until the remaining capacity of the low-speed storage is not less than the first capacity.

9. The cache management method according to claim 7, characterized in that, If the adjustment instruction is a migration instruction, then the storage tier is a medium-speed or high-speed storage within a tiered storage resource pool. For the target key-value cache, adjust the tiered storage resource pool according to the adjustment instructions, including: According to the migration instructions, the target key-value cache is migrated to the next level of storage, and the storage location of the target key-value cache in the metadata is updated.

10. The cache management method according to claim 9, characterized in that, Also includes: During the migration process, the state of the migrated key-value cache is marked as the migration state, which is used to prohibit other operations; After the migration is complete, mark the migrated key-value cache as available.

11. The cache management method according to claim 7, characterized in that, If the adjustment instruction is a precision adjustment instruction, then the storage level is a high-speed storage of a tiered storage resource pool; Accordingly, for the target key-value cache, the tiered storage resource pool is adjusted according to the adjustment instructions, including: Based on the precision adjustment instruction, reduce the storage precision of the target key-value cache, replace the original target key-value cache, and update the precision information of the target key-value cache in the metadata.

12. The cache management method according to any one of claims 1-11, characterized in that, Real-time calculation of the importance score of the target attention head in the current layer, including: The importance score of the target attention head in the current layer is calculated in real time based on the input sequence of model inference of the target attention head in the current layer.

13. The cache management method according to claim 12, characterized in that, Real-time calculation of the importance score of the target attention head in the current layer, including: If the calculation mode is a single-selection mode, then the importance score of the target attention head in the current layer is calculated in real time according to the target single-item calculation mode corresponding to the single-selection mode. If the calculation mode is a weighted comprehensive mode, then the values ​​obtained from each individual calculation mode are weighted and calculated to obtain the importance score of the target attention head in the current layer.

14. The cache management method according to claim 13, characterized in that, Single-item calculation modes include: attention-based output mode, attention-score-based mode, historical information-based mode, and lightweight model mode. The calculation process based on the attention output pattern includes: inputting the input sequence into the corresponding attention head to obtain the output vector; determining the vector norm based on the output vector; obtaining the attention score matrix of the attention head when inferring the input sequence; calculating the entropy of each row of the attention score matrix; determining the entropy of the attention head based on the entropy of each row; and determining the first importance score based on the vector norm and / or the entropy of the attention head. The calculation process based on the attention score pattern includes: converting the input sequence into query vectors and key vectors; splitting the query vectors and key vectors according to the number of attention heads; and determining the attention score based on the query vectors and key vectors of the attention heads, which serves as the second importance score. The calculation process based on historical information patterns includes: obtaining the historical importance score of the attention head; predicting the importance score of the attention head based on the historical importance score of the attention head, which is used as the third importance score; The computation process of the lightweight model mode includes: based on the input sequence of the attention head, the state of the current layer, and the evaluation result of the previous time step, using the multilayer perceptron model, predicting the importance score of the target attention head in the current layer, which is used as the fourth importance score.

15. The cache management method according to claim 1, characterized in that, Also includes: When accessing the key-value cache generated by the target attention head, determine whether the storage precision of the key-value cache is the precision required for calculation; If not, then perform inverse quantization based on the storage location and storage precision corresponding to the target attention head.

16. A cache management device, characterized in that, include: The importance score determination module is used to calculate the importance score of the target attention head in the current layer in real time during the model inference process. The target attention head is any one of multiple attention heads in the current layer. The storage level determination module is used to determine the storage level corresponding to the key-value cache generated by the target attention head based on the importance score of the target attention head in the current layer. The storage level includes storage location and storage precision. The storage module is used to store the key-value cache according to the storage location and storage precision of the key-value cache generated by each attention head; The storage module is used to sort the importance scores of all attention heads in the current layer to obtain the first sorting result; Based on the first sorting result, determine the key-value cache with the highest importance score; Determine whether the storage level corresponding to the key-value cache with the highest importance score is available; If available, the key-value cache with the highest importance score will be stored in the corresponding cache location according to the storage precision. If unavailable, the storage level corresponding to the key-value cache with the highest importance score will be adjusted. The key-value cache is stored in the adjusted storage location and with adjusted storage precision based on the level corresponding to the highest importance score. After storing the key-value cache with the highest importance score, process the next key-value cache in sequence until storing the key-value cache generated by each attention head is completed.

17. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the cache management method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the cache management method as described in any one of claims 1 to 15.

19. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the cache management method according to any one of claims 1 to 15.

Images