Cache management method and device, equipment, medium and product

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a cache management method, device, equipment, medium and product, and the method comprises the steps: evaluating the importance scores of different attention heads for a current reasoning task in real time, and dynamically allocating different levels of storage resources for the key value cache of each attention head according to the importance scores, comprising a storage position and numerical precision, so that the memory is saved to the greatest extent on the premise of ensuring the model output quality and particularly not losing key information, and the reasoning speed is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a cache management method, device, equipment, medium and product. BACKGROUND

[0002] In the field of artificial intelligence, the Transformer model and KV Cache: In the autoregressive generation (such as text generation) or long sequence processing task, the Transformer model (such as GPT, BERT, T5, etc.) needs to cache the Key and Value vectors (KV Cache) of the previous time step (or position) during inference to avoid repeated calculation and significantly improve the inference speed.

[0003] Memory bottleneck of KV Cache: With the increase of the context window length (such as from 1K to 100K or even longer), the memory consumption of KV Cache increases linearly, becoming the main bottleneck of deploying large Transformer models (especially large language models LLMs), limiting their application on resource-constrained devices (such as edge devices), and increasing the cost of cloud services.

[0004] In related technologies, in order to solve the problem of KV Cache memory bottleneck, the following methods are mainly used: unified quantization method; pruning or selective caching strategy based on fixed rules; improved cache data structure; acceleration using hardware characteristics. However, all the above methods use the same caching strategy for all attention heads, which can easily lead to the loss of key information. SUMMARY

[0005] The present application aims to provide a cache management method, device, equipment, medium and product, which can evaluate the importance score of different attention heads for the current inference task in real time, and dynamically allocate different levels of storage resources (including storage location and numerical precision) for the key-value cache of each attention head according to the importance score, thereby maximizing the saving of memory while ensuring the quality of model output, especially the non-loss of key information, and improving the inference speed.

[0006] In a first aspect, a cache management method is provided, comprising: calculating the importance score of a target attention head in a current layer in real time during model inference, the target attention head being any one of a plurality of attention heads in the current layer; determining the storage level corresponding to the key-value cache generated by the target attention head according to the importance score of the target attention head in the current layer, the storage level including storage location and storage precision; and storing the key-value cache according to the storage location and storage precision corresponding to the key-value cache generated by each attention head.

[0007] The application can be further configured in a preferred example as follows: according to the storage location and storage precision corresponding to the key-value cache generated by each attention head, the key-value cache is stored, including: sorting the importance scores of all attention heads of the current layer to obtain a first sorting result; determining the key-value cache with the highest importance score according to the sorting result; determining whether the storage level corresponding to the key-value cache with the highest importance score is available; if available, storing the key-value cache with the highest importance score to the corresponding cache location according to the storage precision; if not available, adjusting the level of the storage level corresponding to the key-value cache with the highest importance score; and storing the key-value cache according to the adjusted storage location and storage precision corresponding to the key-value cache with the highest importance score; after the storage of the key-value cache with the highest importance score is completed, the next key-value cache is processed in turn until the storage of the key-value cache generated by each attention head is completed.

[0008] The application can be further configured in a preferred example as follows: adjusting the level of the storage level corresponding to the key-value cache with the highest importance score, including: if the storage level corresponding to the key-value cache with the highest importance score is high-speed storage, determining whether medium-speed storage corresponding to the storage level of the key-value cache with the highest importance score is available; if the medium-speed storage is available, downgrading the storage level corresponding to the key-value cache with the highest importance score to medium-speed storage; if the medium-speed storage is not available, downgrading the storage level corresponding to the key-value cache with the highest importance score to low-speed storage.

[0009] The application can be further configured in a preferred example as follows: adjusting the level of the storage level corresponding to the key-value cache with the highest importance score, including: if the storage level corresponding to the key-value cache with the highest importance score is medium-speed storage, determining whether low-speed storage corresponding to the storage level of the key-value cache with the highest importance score is available; if the low-speed storage is available, downgrading the storage level corresponding to the key-value cache with the highest importance score to low-speed storage; if the low-speed storage is not available, upgrading the storage level corresponding to the key-value cache with the highest importance score to high-speed storage.

[0010] The application can be further configured in a preferred example as follows: if not available, adjusting the level of the storage level corresponding to the key-value cache with the highest importance score, including: if not available, determining whether the storage level corresponding to the key-value cache with the highest importance score is low-speed storage; if not low-speed storage, adjusting the level of the storage level corresponding to the key-value cache with the highest importance score; the method further includes: if low-speed storage, discarding the key-value cache with the highest importance score.

[0011] The application can be further configured in a preferred example as follows: further comprising: obtaining the storage resource usage of each storage level of the hierarchical storage resource pool; and adjusting the hierarchical storage resource pool according to the storage resource usage and the importance score of the storage level.

[0012] The application can be further configured in a preferred example as follows: the storage resource usage comprises: remaining capacity; and the storage level comprises: low-speed storage, medium-speed storage and high-speed storage; and adjusting the hierarchical storage resource pool according to the storage resource usage and the importance score of the storage level comprises: judging whether the remaining capacity of the storage level is less than a preset capacity threshold; and if the remaining capacity of the storage level is less than the preset capacity threshold, then according to the importance score, screening target key-value cache in the storage level that meets the adjustment requirement; and adjusting the hierarchical storage resource pool according to the target key-value cache in the storage level that meets the adjustment requirement.

[0013] The application can be further configured in a preferred example as follows: adjusting the hierarchical storage resource pool according to the target key-value cache in the storage level that meets the adjustment requirement comprises: adjusting the hierarchical storage resource pool according to the target key-value cache in the storage level that meets the adjustment requirement according to the adjustment instruction.

[0014] The application can be further configured in a preferred example as follows: if the adjustment instruction is an elimination instruction, the storage level is the low-speed storage of the hierarchical storage resource pool; and adjusting the hierarchical storage resource pool according to the target key-value cache comprises: according to the elimination instruction, sorting the target key-value cache based on at least one of the importance score and the last access time; and based on the sorting result, eliminating the target key-value cache in sequence until the remaining capacity of the low-speed storage is not less than a first capacity.

[0015] The application can be further configured in a preferred example as follows: if the adjustment instruction is a migration instruction, the storage level is the medium-speed storage or the high-speed storage of the hierarchical storage resource pool; and adjusting the hierarchical storage resource pool according to the target key-value cache comprises: according to the migration instruction, migrating the target key-value cache to a next level of storage and updating the storage location of the target key-value cache in the metadata.

[0016] The application can be further configured in a preferred example as follows: further comprising: in the migration process, marking the state of the migrated key-value cache as a migration state, the migration state being used to prohibit other operations; and after the migration is completed, marking the state of the migrated key-value cache as an available state.

[0017] The application can be further configured in a preferred example that, if the adjustment instruction is a precision adjustment instruction, the storage level is a high-speed storage of a hierarchical storage resource pool; and correspondingly, adjusting the hierarchical storage resource pool according to the adjustment instruction for the target key value cache comprises: according to the precision adjustment instruction, reducing the storage precision of the target key value cache, replacing the original target key value cache, and updating the precision information of the target key value cache in the metadata.

[0018] The application can be further configured in a preferred example that the importance score of the target attention head in the current layer is calculated in real time, comprising: calculating the importance score of the target attention head in the current layer in real time according to the input sequence of the model inference of the target attention head in the current layer.

[0019] The application can be further configured in a preferred example that the importance score of the target attention head in the current layer is calculated in real time, comprising: if the calculation mode is a single selection mode, the importance score of the target attention head in the current layer is calculated in real time according to the target single calculation mode corresponding to the single selection mode; and if the calculation mode is a weighted comprehensive mode, the importance score of the target attention head in the current layer is obtained by weighted calculation according to the values calculated by each single calculation mode.

[0020] The application can be further configured in a preferred example that the single calculation mode comprises: an attention output mode, an attention score mode, a historical information mode, and a lightweight model mode, wherein the calculation process of the attention output mode comprises: inputting the input sequence into the corresponding attention head to obtain an output vector; determining the vector norm according to the output vector; obtaining the attention score matrix of the attention head when inferring the input sequence; calculating the entropy of each row of the attention score matrix; determining the entropy of the attention head according to the entropy of each row; determining the first importance score according to the vector norm and / or the entropy of the attention head; the calculation process of the attention score mode comprises: converting the input sequence into a query vector and a key vector; splitting the query vector and the key vector according to the number of attention heads; determining the attention score as the second importance score according to the query vector and the key vector of the attention head; the calculation process of the historical information mode comprises: obtaining the historical importance score of the attention head; predicting the importance score of the attention head as the third importance score according to the historical importance score of the attention head; the calculation process of the lightweight model mode comprises: predicting the importance score of the target attention head in the current layer as the fourth importance score by using a multi-layer perception model according to the input sequence of the attention head, the state of the current layer, and the evaluation result of the previous time step.

[0021] The application can be further configured in a preferred example as follows: further comprising: when accessing the key-value cache generated by the target attention head, determining whether the storage precision of the key-value cache is the required precision for calculation; if not, performing a dequantization operation according to the storage location and storage precision corresponding to the target attention head.

[0022] In a second aspect, a cache management apparatus is provided, comprising: an importance score determination module configured to calculate, in real time, an importance score of a target attention head in a current layer during a model inference process, the target attention head being any one of a plurality of attention heads in the current layer; a storage level determination module configured to determine, according to the importance score of the target attention head in the current layer, a storage level corresponding to a key-value cache generated by the target attention head, the storage level comprising a storage location and a storage precision; and a storage module configured to store the key-value cache according to the storage location and storage precision corresponding to the key-value cache generated by each attention head.

[0023] In a third aspect, an electronic device memory is provided, configured to store a computer program; and a processor is configured to execute the computer program to implement the method of any one of the first aspect.

[0024] In a fourth aspect, a computer readable storage medium is provided, having stored thereon a computer program, the computer program being executable by a processor to implement the method of any one of the first aspect.

[0025] In a fifth aspect, a computer program product is provided, comprising a computer program or instructions, the computer program or instructions being executable by a processor to implement the method of any one of the first aspect.

[0026] In summary, the cache management method provided by the application has the following beneficial technical effects: the importance score of a target attention head in a current layer is calculated in real time, the target attention head being any one of a plurality of attention heads in the current layer; according to the importance score of the target attention head in the current layer, a storage level corresponding to a key-value cache generated by the target attention head is determined, the storage level comprising a storage location and a storage precision; and the key-value cache is stored according to the storage location and storage precision corresponding to the key-value cache generated by each attention head. The importance score of different attention heads for a current inference task is evaluated in real time, and according to the importance score, different levels of storage resources (including storage location and numerical precision) are dynamically allocated to the key-value cache of each attention head, thereby maximizing the saving of memory while ensuring the quality of model output, especially the non-loss of key information, and further improving the inference speed.

[0027] In addition, the application also provides a cache management apparatus, device, medium and product, all of which have the above beneficial technical effects. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described in the following embodiments are only some of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0029] Figure 1 is a KV Cache management schematic diagram in the related art; Figure 2 is a cache management method flowchart provided by an embodiment of the present application; Figure 3 is a system overall architecture diagram provided by an embodiment of the present application; Figure 4 is a specific dynamic hierarchical allocation strategy schematic diagram provided by an embodiment of the present application; Figure 5 is a reasoning step flowchart of cache migration and precision adjustment provided by an embodiment of the present application; Figure 6 is an interaction schematic diagram of a hierarchical storage resource pool and a cache manager provided by an embodiment of the present application; Figure 7 is a flowchart of a dynamic attention head importance evaluation method provided by an embodiment of the present application; Figure 8 is a structure schematic diagram of a cache management device provided by an embodiment of the present application; Figure 9 is a structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the protection scope of the present application.

[0031] The terms “include” and “have” and any variations of the terms “include” and “have” in the specification and the above drawings of the present application are intended to cover the non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can include steps or units not listed.

[0032] For those skilled in the art to better understand the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0033] Memory bottleneck of KVCache: As the context window length increases (e.g., from 1K to 100K or even longer), the memory consumption of KVCache increases linearly, becoming a major bottleneck for deploying large Transformer models (especially large language models LLMs), limiting their application on resource-constrained devices (such as edge devices), and increasing cloud service costs.

[0034] Referring to Figure 1 , Figure 1 is a KVCache management diagram in the related art, in which all attention heads are uniformly processed. The characteristics are: all attention heads agree to process, a single storage level (such as GPU memory), a fixed precision configuration (such as FP16 / INT8), and a simple eviction policy (such as FIFO / LRU).

[0035] Limitations of existing optimization techniques: (1) Uniform quantization: uniformly reduce the precision of all elements in the entire KVCache (e.g., FP16→INT8, INT4). However, quantizing important attention heads / elements may introduce large errors, affecting model output quality; quantizing less important attention heads / elements may not be aggressive enough, wasting memory.

[0036] (2) Cache sparsification / pruning: discard part of the KV vectors (e.g., based on low attention score tokens). However, the discard operation is irreversible and may lose critical information; dynamic sparsification strategies are complex and may introduce additional computational overhead; it is difficult to ensure that all important information is retained. The pruning granularity of existing technologies is limited.

[0037] Specifically, existing pruning techniques are mostly at the "Token level" or "layer level", rather than the "attention head level".

[0038] Token-level pruning: based on the attention score of a single Token, without distinguishing the importance of the attention head to which the Token belongs (for example, a low attention score Token may belong to an attention head that is critical to the task, and pruning it will lose critical information).

[0039] Layer-level pruning: directly prune the KVCache of the entire Transformer layer (e.g., cache every 2 layers for 1 layer), without considering the functional differences of different attention heads within the layer (e.g., some attention heads in the same layer are responsible for syntax analysis, and some are responsible for semantic understanding, and pruning will discard them indiscriminately).

[0040] Current technology lacks a module for real-time assessment of the importance of attention heads and cannot distinguish the value of different attention heads. For example, a certain pruning scheme may fix the pruning of "attention heads whose attention score variance is less than a threshold", but this threshold is preset offline (calibrated based on a static dataset) and cannot adapt to changes in the input sequence during inference (such as when processing "medical text", the importance of the attention head responsible for entity recognition increases, and the preset threshold may mistakenly prune this attention head).

[0041] The core difference between this invention and existing technologies is that it upgrades the decision-making basis for pruning / sparsening from "fixed rules of tokens / layers" to "real-time importance scores of attention heads", thereby achieving differentiated pruning at the attention head level (low-importance attention heads can be discarded, while high-importance attention heads are retained).

[0042] (3) Selective caching: Caching only the key-value pairs of some layers or the key-value pairs of specific attention heads. However, static policies cannot adapt to different inputs; dynamic policies require complex decision-making; and may destroy the internal representation of the model.

[0043] (4) Recalculation: No caching or minimal caching, recalculation when needed. However, this significantly increases computation latency, which defeats the original purpose of using KVCache to accelerate inference.

[0044] (5) Sharding / Offloading: Offloading part of the KVCache to CPU memory or disk. However, CPU / disk access latency is high, which seriously affects inference speed; frequent data transfers bring additional overhead.

[0045] In summary, existing technologies generally ignore or fail to effectively utilize the significant differences in function and importance among different attention heads in the Transformer model. Using the same caching strategy (such as the same precision and the same storage location) for all attention heads leads to an unbalanced resource allocation, either sacrificing the precision of important attention heads or wasting valuable memory resources on secondary attention heads.

[0046] Based on this, the present invention aims to overcome the shortcomings of existing KVCache optimization techniques and proposes a dynamic hierarchical key-value cache management method and system based on attention head importance awareness. The core of this method lies in real-time evaluation of the importance of different attention heads to the current inference task, and dynamically allocating different levels of storage resources (including storage location and numerical precision) to the KVCache of each attention head (or elements within an attention head) according to its importance score. This maximizes memory savings and potential inference speed improvements while ensuring the quality of model output (especially preventing the loss of key information).

[0047] The application is directed to a system-level optimization scheme for the KVCache memory bottleneck of the Transformer model in real inference scenarios. The application scenarios include but are not limited to: 1, long text generation (such as dialogue system, document generation), in which the KVCache memory occupancy increases linearly with the sequence length; 2, resource-limited environment (such as edge device, mobile terminal deployment), limited memory and computing resources; 3, high-concurrency inference service (such as large model inference optimization in cloud service), multi-tenant inference in cloud service, which needs to efficiently use GPU memory to reduce cost.

[0048] The application belongs to the technical field of artificial intelligence, and specifically relates to deep learning model inference optimization technology, in particular to efficient management and optimization technology of key-value cache in the inference process of a Transformer architecture model.

[0049] Next, a cache management method, device, equipment, medium and product provided by an embodiment of the application are introduced in detail.

[0050] Figure 2 A flowchart of a cache management method provided by an embodiment of the application, the cache management method comprising: S101, in the model inference process, calculating an importance score of a target attention head in a current layer in real time, the target attention head being any one of a plurality of attention heads in the current layer.

[0051] Among them, before inference, the "potential importance" of each attention head is fixed after model training (because its structure is fixed, i.e., its functional role is fixed), but because the content and structure of the input sequence are variable, therefore, in the embodiment of the application, the numerical value of the specific importance score is dynamically calculated in the inference process. In the inference process, the importance of different attention heads will dynamically change when processing different input tokens. For example, some attention heads may be specifically used to process syntax structure, some to process semantic keywords, and some to process long-range dependencies, and their importance depends on the current input content.

[0052] In the embodiment of the application, the importance of KVCache is dynamically evaluated at the attention head level (rather than the entire layer or the entire model), which is a key insight into the inherent characteristics of the Transformer multi-attention head mechanism.

[0053] In the embodiments of the present application, the importance evaluation and storage level allocation decision is made in real time during the inference process, which can adapt to the changes of the input sequence and the evolution of the internal state of the model. Among them, the input sequence is the input sequence of the model inference (such as the text token sequence). The present application dynamically responds to the changes of the input content by calculating the importance score of each attention head in real time. For example, when the input changes from "syntax structure" to "semantic keyword", the importance of the attention head corresponding to the processing of these information will rise, thereby triggering the adjustment of the storage strategy. The internal state of the model refers to the information implied and dynamically changing during the inference process, such as: the distribution change of attention weight; the evolution of hidden layer activation value; the content change of historical KVCache. The present application captures this state evolution through dynamic importance evaluation (such as using historical score smoothing and lightweight prediction model) to make more reasonable storage decisions.

[0054] S102, according to the importance score of the target attention head in the current layer, determine the storage level corresponding to the key-value cache generated by the target attention head, the storage level includes storage location and storage precision.

[0055] In the embodiments of the present application, hierarchical resource allocation is performed in combination with the two dimensions of storage location and numerical precision, and differentiated quality of service is provided for attention heads of different importance.

[0056] Further, the hierarchical storage resource is defined in advance, and the specific definition includes a hierarchical storage resource pool containing multiple storage locations and multiple numerical precisions, that is, multiple storage levels are defined, each storage level includes: storage location and storage precision, wherein the higher the speed of the storage location, the faster the access speed, the higher the storage precision, the higher the data precision. For example: storage location level: highest speed storage (such as GPU HBM), medium speed storage (such as CPU DRAM), low speed storage (such as SSD / NVMe) or choose not to store (i.e. discard). Numerical precision level: high precision (such as FP16 / BF16), medium precision (such as INT8), low precision (such as INT4 / FP8) or custom quantization (such as quantization based on the statistical characteristics of the attention head).

[0057] For example: high importance attention head: allocate storage level as high speed storage location + high precision → ensure fast access and low error. Medium importance attention head: allocate storage level as medium speed storage location + medium precision → balance speed, precision and memory. Low importance attention head: allocate storage level as low speed storage location + low precision or directly discard → maximize memory saving.

[0058] It can be understood that the above is only an example, and in actual systems, the hardware characteristics and task requirements can be flexibly combined, for example: high-speed storage + medium precision: suitable for attention heads with high importance but not extremely sensitive to precision; medium-speed storage + high precision: suitable for attention heads with high precision requirements but slightly lower access frequency.

[0059] In an implementable manner, a correspondence relationship between the importance score and the storage level is pre-set, and then, based on the importance score of the target attention head in the current layer, the storage level corresponding to the key-value cache generated by the target attention head can be matched out, wherein the higher the importance score is, the faster the storage speed of the storage location is, and the higher the storage precision is.

[0060] In another implementable manner, the key-value cache can be directly allocated from the high-speed storage level according to the importance score and the remaining capacity of each storage level, and when the storage capacity is not available, the key-value cache is allocated to the next storage level.

[0061] The specific implementation manner is not limited by the embodiments of the present application, and can be customized by the user.

[0062] In the embodiments of the present application, since the importance score of the attention head is dynamically changed, according to the importance score of each attention head, the KVCache generated by the attention head at the current time step is dynamically decided to be allocated to the pre-defined matching storage level. By accurately allocating limited fast storage and high-precision resources to the most important attention head, while transferring the secondary attention head to slow / low-precision storage or discarding it, the overall memory occupation is significantly reduced, and the influence on the access speed of the key attention head is minimized.

[0063] S103, store the key-value cache according to the storage location and the storage precision corresponding to the key-value cache generated by each attention head.

[0064] In the embodiments of the present application, if the remaining storage space of the storage location corresponding to the attention head is sufficient, the key-value cache generated by the attention head can be stored at the storage location with the storage precision.

[0065] It can be understood that in the embodiments of the present application, the key value cache of the current time exists multiple attention heads needs to be stored, therefore, according to the importance score of the attention head, the key value cache can be stored in turn, if the remaining storage space of the corresponding storage level is insufficient when storing the key value cache of a certain attention head, the storage can be downgraded or upgraded, or if the space is still insufficient after downgrading / upgrading, if the importance score of the attention head is high, the key value cache with low importance score and long time not accessed in the storage level can be eliminated, and if the importance score of the attention head is low, the key value cache corresponding to the attention head can be eliminated.

[0066] Of course, the elimination, migration and precision adjustment of the key value cache of the storage level can also be performed regularly, so that the storage level has sufficient storage space to store new key value caches.

[0067] It can be seen that in the embodiments of the present application, in the model inference process, the importance score of the target attention head in the current layer is calculated in real time, and the target attention head is any one of the multiple attention heads in the current layer; according to the importance score of the target attention head in the current layer, the storage level corresponding to the key value cache generated by the target attention head is determined, and the storage level includes a storage location and a storage precision; according to the storage location and the storage precision corresponding to the key value cache generated by each attention head, the key value cache is stored. The importance score of different attention heads to the current inference task is evaluated in real time, and according to the importance score, the storage resource (including the storage location and the numerical precision) of different levels is dynamically allocated to the key value cache of each attention head, so that the memory is maximized to be saved under the premise of ensuring the quality of model output, especially the lossless of key information, and the inference speed is improved.

[0068] In order to better understand the technical scheme of the present application, see Figure 3 , Figure 3 is a system overall architecture provided by the embodiments of the present application, which includes a layer (Transformer layer), an importance evaluation module, a hierarchical resource pool, a cache manager and an allocation decision maker.

[0069] The Transformer layer includes a multi-attention head attention mechanism, generates a key / value vector (Key / Value vector), and outputs an attention vector attn_output and an attention score attn_weights. The importance evaluation module evaluates the importance based on the output of the Transformer layer. The importance of the attention head is calculated in real time. Based on the input: attn_output / attn_weights, the method: norm / entropy / history state is used to output the importance score. The allocation decision maker receives the importance score, obtains the resource pool state, and uses the threshold / sorting / mixed strategy to dynamically decide the storage level. The cache manager is used for metadata management, including: location / precision / importance, timestamp / state, and data migration engine; elimination strategy; precision conversion. The timestamp records the key time nodes of the KV cache block, including the generation time (the time when the cache block is first stored in the hierarchical resource pool) and the last retrieval access time. Its core function is to assist the execution of the elimination strategy, for example, when the low-speed storage is full, combined with the importance score, the cache block with the lowest importance and the longest unused time (according to the latest access timestamp) is preferentially eliminated, avoiding the mis-elimination of key historical information. The state describes the current storage and availability state of the KV cache block, including three core states: storage state: identifies the current storage level of the cache block (such as Level0: GPUHBM, Level1: CPU DRAM); precision state: records the current numerical precision of the cache block (such as FP16, INT8), which provides a basis for the inverse quantization operation during retrieval. Migration / Conversion State: Indicates whether the cache block is in the process of asynchronous migration (such as from GPU HBM to CPU DRAM) or precision adjustment (such as from FP16 to INT8), to avoid data consistency problems caused by concurrent operations.

[0070] In an implementable manner, data migration includes: step 1, migration trigger detection. Every 1 inference time step, scan the metadata of all cache blocks to detect whether the migration condition is met (such as the importance score of the attention head from > θ_high to Step 2, Migration task enqueue. Add the cache block that meets the condition to the "migration task queue" and sort it by "importance score descending range" (the larger the descending range, the higher the priority of migration). Step 3, Asynchronous migration execution. The background thread (does not block the inference main thread) executes the migration, including: reading the source cache block: reading the low-precision cache block (such as FP16) from the source storage location (such as GPU HBM); precision conversion (if needed): such as converting from FP16 to INT8 (based on the attention head statistical information); writing to the target location: writing the converted cache block to the target storage (such as CPU DRAM); updating the metadata: updating the storage location and precision information of the cache block to the metadata list. Step 4: Migration state monitoring. During the migration process, mark the cache block state as "in migration" and prohibit other operations (such as retrieval), and mark it as "available" after the migration is completed.

[0071] In an implementable way, the elimination strategy implementation process includes: Step 1, Elimination trigger condition. When the remaining capacity of the low-speed storage (such as SSD) is <10%, trigger the elimination process. Step 2, Elimination candidate screening. From all cache blocks of the SSD, screen a candidate set of "importance score < θ_low" (to ensure that low-value cache blocks are eliminated). Step 3, Elimination priority sorting. Sort the candidate set according to "importance score (ascending order) + last access time (ascending order)" (i.e., the lowest importance and the longest unused are preferentially eliminated). Step 4, Elimination execution. Delete the cache blocks in sequence according to the sorting result, release the storage space, and delete the corresponding metadata record until the remaining capacity of the SSD is ≥15%. Step 5, Elimination rollback (optional). If it is found after elimination that a certain eliminated cache block needs to be reused (such as the KV data of the attention head is needed for subsequent inference), recalculate and store the KV data of the attention head based on the current importance score allocation level.

[0072] In an implementable way, the precision conversion implementation process includes: Step 1, Conversion trigger condition. When the remaining capacity of a certain storage level (such as CPU DRAM) is <8% and there is a cache block with "importance score < θ_mid and precision is INT8". Step 2, Statistical information update. Recalculate the statistical information (max_val, min_val) of all cache blocks of the attention head to ensure quantization precision. Step 3, Precision conversion execution (taking INT8→INT4 as an example) includes: reading the INT8 cache block: reading the Key / Value block in INT8 format from CPU DRAM. Dequantization to FP16: based on the statistical information of the original INT8, first dequantize to FP16. Requantization to INT4: based on the new statistical information (max_val_new, min_val_new), perform quantization: (INT4 range is 0-15, mapping the max-min range of FP16). Replace cache block: write INT4 cache block to CPU DRAM, delete the original INT8 cache block, release memory (INT4 saves 50% capacity compared with INT8). Step 4, metadata update. Update the precision information of the cache block from INT8 to INT4, and record the new statistical information.

[0073] Subsequent inference steps are used to generate KVCache requests; receive the retrieved KV data.

[0074] Among them, when calculating attention in the subsequent time step of the Transformer model autoregressive inference, the request needs to be received and the KV data needs to be retrieved. When the model generates a new Token, the current layer multi-attention head attention mechanism needs to combine the “Query vector of the current input Token” and the “historical time step stored KV cache (i.e. Key and Value vectors of the previous Token)” to calculate the attention score, and then generate the attention output of the current layer. At this time, the cache manager will receive the metadata (storage location, precision) to locate and read the data, and if it is low-precision storage, it will first perform inverse quantization, and then return the data to the attention calculation module.

[0075] Hierarchical resource pool, including high-speed storage Level0, medium-speed storage Level1 and low-speed storage Level2. Among them, Level0: GPU HBM, FP16 / BF16, high speed / high precision; Level1: CPU DRAM, INT8, medium speed / medium precision; Level2: SSD / discard, INT4, low speed / low precision.

[0076] Based on the overall architecture of the system, and then one possible implementation of the embodiment of the application, S103 stores the key-value cache according to the storage location and storage precision of the key-value cache generated by each attention head, including: S1031, sort the importance scores of all attention heads of the current layer to obtain a first sorting result.

[0077] The current layer includes at least one attention head, and the importance scores of all attention heads of the current layer at the current time are sorted to obtain a first sorting result. The all attention heads of a certain layer in the Transformer. The attention heads in the same layer share the same input and output dimensions, but their functions can be different.

[0078] The plurality of attention heads are located in the same layer. The same layer refers to a single encoder / decoder layer of the Transformer model, which includes a plurality of attention heads and a feedforward network. For example, in the 64 layers of GPT-3, each layer is an independent same layer. The same layer includes h attention heads (e.g., 12), which share the same input sequence and layer normalization parameters. The attention heads in the same layer cooperatively complete the attention calculation of the layer, and the key-value cache KVCache needs to be uniformly managed (e.g., the storage location of the KVCache of the attention heads in the same layer needs to consider the overall resource demand of the layer).

[0079] In the embodiment of the application, the h attention heads in the same layer are sorted in descending order of importance scores (e.g., h = 12, and after sorting, the importance scores of the 12 attention heads are 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0, 0, 0). ).

[0080] S1032, according to the sorting result, determining the key-value cache with the highest importance score.

[0081] The resources (e.g., high-speed storage slots, high-precision quotas) are allocated to the first N attention heads in the same layer according to the importance sorting.

[0082] For the storage level determination mode according to the correspondence between the importance score range and the storage level, in one implementable mode, S1033, S1034, and S1035 can be sequentially executed.

[0083] In another implementable mode, the remaining capacity of the layered resources of the model to which the current layer belongs can be obtained (e.g., GPU HBM remaining 5 GB, CPU DRAM remaining 20 GB). If the 12 attention heads all correspond to GPU HBM, the number of attention heads that can be accommodated at each level is calculated according to the remaining capacity: if each attention head requires 0.5 GB (FP16 precision) of GPU HBM, the remaining 5 GB can accommodate 10 attention heads, then the first 10 attention heads of the attention heads in the GPU HBM are allocated GPU HBM+FP16. The remaining 2 attention heads (H11, H12) are allocated to CPU DRAM+INT8 (each attention head requires 0.25 GB, and 2 attention heads require 0.5 GB, which is sufficient for the remaining 20 GB of CPU). It should be noted that if the remaining capacity of a certain level is insufficient (e.g., GPU HBM only has 2 GB remaining, which can accommodate 4 attention heads), the first 4 attention heads are allocated GPU HBM+FP16, and the 5th-12th attention heads are allocated CPU DRAM+INT8.

[0084] S1033, determining whether the storage level corresponding to the key-value cache with the highest importance score is available.

[0085] In the embodiments of the present application, the corresponding available space size threshold / space ratio threshold can be set for different storage levels; when the available space size / space ratio is greater than the corresponding threshold, it indicates that the storage level is available. The available space size threshold / space ratio threshold of the storage level can be completely the same or not completely the same, and can be set by the user.

[0086] S1034, if available, the key-value cache with the highest importance score is stored in the corresponding cache location according to the storage precision.

[0087] S1035, if not available, the storage level corresponding to the key-value cache with the highest importance score is adjusted; and the key-value cache is stored according to the storage location and storage precision after the level adjustment of the key-value cache with the highest importance score.

[0088] S1036, after the storage of the key-value cache with the highest importance score is completed, the next key-value cache is processed in turn until the storage of the key-value cache generated by each attention head is completed.

[0089] In one possible implementation of the embodiments of the present application, the storage level corresponding to the key-value cache with the highest importance score is adjusted, including: if the storage level corresponding to the key-value cache with the highest importance score is high-speed storage, it is determined whether the medium-speed storage of the storage level corresponding to the key-value cache with the highest importance score is available; if the medium-speed storage is available, the storage level corresponding to the key-value cache with the highest importance score is downgraded to the medium-speed storage; if the medium-speed storage is not available, the storage level corresponding to the key-value cache with the highest importance score is downgraded to the low-speed storage.

[0090] In the embodiments of the present application, the high-speed storage level is greater than the medium-speed storage level, and the medium-speed storage level is greater than the low-speed storage level. Only the size relationship of the three levels is represented, not only the three storage levels. The high-speed storage level represents the highest storage level; the low-speed storage level represents the lowest storage level; the medium-speed storage level is the intermediate storage level, and the number of the medium-speed storage level is at least one, which has an upper storage level and a lower storage level.

[0091] It should be noted that, in general, the key-value caches of each storage level will be managed regularly to ensure that the storage level has sufficient remaining storage capacity. In some cases, the capacity of the storage may also exceed the corresponding threshold, indicating that the storage level is not available, at which time, the downgrade processing can be performed.

[0092] In a possible case, the step of degrading the storage level corresponding to the key-value cache with the highest importance score to the low-speed storage can comprise: if the low-speed storage is available, performing secondary degradation; if the low-speed storage is unavailable, considering evicting key-value caches with low importance values and long-term inaccessibility from the low-speed storage to release storage space for the current key-value cache storage. Of course, if the space of the high-speed storage is cleaned up and there is remaining space, the key-value caches stored in the degraded storage can be migrated back to the high-level storage according to the importance scores and time stamps, and the embodiments of the present application are not limited in this regard.

[0093] In a possible implementation of the embodiments of the present application, if the number of the medium-speed storages is only one, the step of adjusting the level of the storage level corresponding to the key-value cache with the highest importance score can comprise: if the storage level corresponding to the key-value cache with the high importance score is the medium-speed storage, determining whether the low-speed storage corresponding to the storage level with the highest importance score is available; if the low-speed storage is available, degrading the storage level corresponding to the key-value cache with the highest importance score to the medium-speed storage; if the low-speed storage is unavailable, upgrading the storage level corresponding to the key-value cache with the highest importance score to the high-speed storage.

[0094] In a possible implementation of the embodiments of the present application, if the number of the medium-speed storages is only one, the step of adjusting the level of the storage level corresponding to the key-value cache with the highest importance score can comprise: if the storage level corresponding to the key-value cache with the high importance score is the medium-speed storage, determining whether the low-speed storage corresponding to the storage level with the highest importance score is available; if the low-speed storage is available, degrading the storage level corresponding to the key-value cache with the highest importance score to the medium-speed storage; if the low-speed storage is unavailable, upgrading the storage level corresponding to the key-value cache with the highest importance score to the high-speed storage.

[0095] In a possible implementation of the embodiments of the present application, if the low-speed storage is unavailable, the step of adjusting the level of the storage level corresponding to the key-value cache with the highest importance score can comprise: if the low-speed storage is unavailable, determining whether the storage level corresponding to the key-value cache with the highest importance score is the low-speed storage; if the storage level is not the low-speed storage, adjusting the level of the storage level corresponding to the key-value cache with the highest importance score; if the storage level is the low-speed storage, discarding the key-value cache with the highest importance score.

[0096] Based on any of the above embodiments, referring to Figure 4 , Figure 4 is a specific dynamic hierarchical allocation strategy diagram provided by the embodiments of the present application; Wherein, the high-speed storage 0 (Level 0) counter (vnl0) is the abbreviation of the remaining capacity counter of the storage resource of the high-speed storage 0, the high-speed storage 0 is the highest priority storage level (such as GPU HBM+FP16 / BF16), and the high-speed storage 0 counter is used to record the remaining allocable storage capacity of the level in real time (in bytes or cache block quantity).

[0097] In the allocation strategy based on sorting, if the current attention head completes resource allocation, the next attention head needs to be taken for processing until all attention heads are processed.

[0098] Specifically, after the attention heads of the current layer are sorted in descending order of importance, before the highest-score attention head is allocated with the high-speed storage 0 resource, the real-time remaining capacity of the high-speed storage 0 is read through the high-speed storage 0 counter; if the remaining capacity is sufficient to accommodate the KV cache block of the attention head (the storage space corresponding to the cache block size and precision needs to be considered), the Level 0 is allocated; if the capacity is insufficient, the downgrade allocation (such as switching to Level 1) is triggered. This process needs to be associated with the resource pool state in real time to avoid memory overflow caused by over-allocation.

[0099] Wherein, the importance score of the highest-score attention head H is greater than θ_high, and the vnl0 shows that the Level 0 capacity is sufficient, and after successful allocation, the next attention head with the second highest importance is taken; the importance score of the highest-score attention head H is greater than θ_high, but the vnl0 shows that the Level 0 capacity is insufficient, and after downgrade allocation to Level 1 (or a lower level), the next attention head is taken; The importance score of the current attention head is less than or equal to θ_mid, and after direct allocation to Level 2 (discarding or SSD storage), the next attention head is taken; When there is no remaining capacity in all levels, the elimination strategy (eliminating old cache blocks with low importance) is triggered, and after elimination, the current attention head is allocated, and if it still cannot be allocated, the next attention head is taken. When the low-speed storage is also full, the cache block with the lowest importance and the longest non-use is preferentially eliminated.

[0100] The storage resource (such as GPU HBM) of Level 0 has fixed and limited capacity, and if only the importance score greater than θ_high is allocated, when multiple high-importance attention heads apply at the same time, the total demand may exceed the actual capacity of Level 0, causing memory overflow. By judging the availability of L0 resource, the "limited optimal resource priority allocation to the most important attention head" can be realized: for example, there are 5 attention heads with importance scores greater than θ_high in the current layer, but Level 0 can only accommodate 3, so the first 3 attention heads with the highest importance are preferentially allocated, and the remaining 2 high-importance attention heads are downgraded to Level 1, which not only ensures that the resources are not over-allocated, but also maximizes the value of Level 0 resources.

[0101] Further, a possible implementation manner of the embodiment of the present application further comprises: acquiring the storage resource usage of each storage level of the hierarchical storage resource pool; and adjusting the hierarchical storage resource pool according to the storage resource usage and the importance score of the storage level.

[0102] In the embodiment of the present application, according to the storage resource usage and the importance score, the elimination (when the storage is full, the least important and the longest unused block is eliminated), the migration (for example, the cache block with reduced importance is migrated from the high-speed storage to the low-speed storage) or the precision adjustment (for example, the cache block with reduced importance is reduced in precision) strategy of the cache is implemented, so as to adjust the hierarchical storage resource pool.

[0103] The storage resource usage includes the remaining capacity, and can further include: the total capacity, the used capacity and the resource load of each level. The total capacity (for example, the total capacity of GPU HBM is 24 GB, and the total capacity of CPU DRAM is 128 GB), the used capacity (for example, the used capacity of GPU HBM is 18 GB) and the remaining capacity (for example, the remaining capacity of GPU HBM is 6 GB) of each level. The resource load: the read-write bandwidth occupancy rate (for example, the read-write bandwidth occupancy rate of GPU HBM is 80%) and the data migration queue length (for example, there are 3 cache blocks in the migration queue from CPU DRAM to GPU HBM) of each level.

[0104] The storage resource usage can be collected in real time through a hardware monitoring interface and a software statistical module. The used / remaining capacity and the bandwidth occupancy of GPU HBM are acquired by calling a GPU driver interface (for example, the NVML library of NVIDA), and the capacity information of CPU DRAM and SSD is acquired by calling an operating system interface (for example, the free / df command of Linux). Through a built-in statistical module of the cache manager, the number and size (for example, the size of each INT8 precision KV cache block is 1 MB) of the KV cache block of each level are recorded in real time, the used capacity is calculated as Σ cache block size, and the remaining capacity is calculated as total capacity-used capacity.

[0105] Further, the collection frequency can be set, and the collection frequency is synchronized with the model inference time step (for example, 1 collection is performed every time 1 Token is generated), so as to ensure the real-time performance of data.

[0106] In a possible implementation of the embodiment of the present application, the storage resource usage includes a remaining capacity; the storage hierarchy includes low-speed storage, medium-speed storage, and high-speed storage; and the adjusting the hierarchical storage resource pool according to the storage resource usage and the importance score of the storage hierarchy includes: determining whether the remaining capacity of the storage hierarchy is less than a preset capacity threshold; if the remaining capacity of the storage hierarchy is less than the preset capacity threshold, filtering, according to the importance score, a target key-value cache in the storage hierarchy that meets an adjustment requirement; and adjusting the hierarchical storage resource pool according to the target key-value cache in the storage hierarchy that meets the adjustment requirement.

[0107] The preset capacity thresholds corresponding to different storage hierarchies can be the same or different, and the embodiment of the present application does not limit this.

[0108] In a possible implementation, the adjusting the hierarchical storage resource pool according to the target key-value cache in the storage hierarchy that meets the adjustment requirement includes: adjusting the hierarchical storage resource pool according to the target key-value cache in the storage hierarchy that meets the adjustment requirement and the adjustment instruction.

[0109] The adjustment instruction corresponds to the adjustment requirement.

[0110] In a possible case, if the adjustment instruction is an elimination instruction, the adjustment requirement is that the storage hierarchy is low-speed storage, and the importance score is less than a lowest importance score threshold.

[0111] Specifically, if the adjustment instruction is an elimination instruction, the storage hierarchy is low-speed storage of the hierarchical storage resource pool, and the adjusting the hierarchical storage resource pool according to the target key-value cache includes: sorting, according to at least one of the importance score and a last access time, the target key-value cache based on the elimination instruction, and eliminating the target key-value cache in sequence based on a sorting result until the remaining capacity of the low-speed storage is not less than a first capacity.

[0112] For example, for cache elimination, when the remaining capacity of low-speed storage (such as SSD) is less than a first threshold (such as 10%), and there is a cache block (key-value cache) with an importance score less than θ_low, all cache blocks with an importance score less than θ_low in the SSD are filtered; the cache blocks are sorted according to the principle of “least recently used (LRU)”; and the cache blocks at the front of the sorting result are eliminated until the remaining capacity of the SSD is greater than or equal to 15%.

[0113] In another possible case, if the adjustment instruction is a migration instruction, the adjustment requirement is that the storage hierarchy is medium-speed storage or high-speed storage, and the importance score is reduced from a first range to a second range, where the first range is an importance score range corresponding to the current storage hierarchy, and the second range is an importance score range corresponding to a lower level of the current storage hierarchy.

[0114] If the adjustment instruction is a migration instruction, the storage level is the medium or high storage of the hierarchical storage resource pool; and for the target key-value cache, the hierarchical storage resource pool is adjusted according to the adjustment instruction, including: according to the migration instruction, migrating the target key-value cache to the next level of storage, and updating the storage location of the target key-value cache in the metadata.

[0115] Specifically, for cache migration, the high-speed storage GPU HBM remaining capacity < second threshold (such as 5%), and there is a target key-value cache with importance score from > θ_high to θ_mid ~ θ_high; or, the CPU DRAM remaining capacity < third threshold (such as 10%), and there is a target key-value cache with importance score from θ_mid ~ θ_high to < θ_mid.

[0116] Filtering the cache blocks meeting the importance drop condition, migrating asynchronously (background execution, not blocking inference), realizing GPU HBM → CPU DRAM (importance dropping to θ_mid ~ θ_high); CPU DRAM → SSD (importance dropping to < θ_mid); and updating the storage location in the metadata after migration is completed.

[0117] Further, in the migration process, the state of the migrated key-value cache is marked as a migration state, and the migration state is used to prohibit other operations; and after the migration is completed, the state of the migrated key-value cache is marked as an available state.

[0118] In another possible case, if the adjustment instruction is a precision adjustment instruction, the adjustment requirement is that the storage level is high-speed storage, and the importance score is reduced from a first range to a second range, and the precision is a preset precision, wherein the first range is the importance score range corresponding to the current storage level, and the second range is the importance score range corresponding to the lower level of the current storage level.

[0119] In one possible case, if the adjustment instruction is a precision adjustment instruction, the storage level is the high-speed storage of the hierarchical storage resource pool; and correspondingly, for the target key-value cache, the hierarchical storage resource pool is adjusted according to the adjustment instruction, including: according to the precision adjustment instruction, reducing the storage precision of the target key-value cache, replacing the original target key-value cache, and updating the precision information of the target key-value cache in the metadata.

[0120] For precision adjustment, the CPU DRAM remaining capacity < fourth threshold (such as 8%); filtering the cache blocks with importance score < θ_mid and precision being a preset precision (such as INT8) in the CPU DRAM, and performing INT8 → INT4 quantization (based on the latest statistical information of the attention head); replacing the original cache block, updating the precision information in the metadata, and releasing the memory (INT4 saves 50% capacity compared with INT8).

[0121] In the embodiments of the present application, the precision conversion uses quantization based on statistical information (such as min / max, mean / std) of a single attention head or a group of similar attention heads, which can be more accurate than global quantization.

[0122] The smaller the quantization granularity is, the more accurate the statistical information is to the characteristics of the attention head, and the smaller the precision loss is.

[0123] It can be understood that global quantization is based on statistical information (such as max / min of KV vectors of all attention heads and all tokens) of the entire KVCache, without considering the differences between attention heads. For example, the KV vector range of attention head A is [0, 10], and the range of attention head B is [0, 1000], the max of global quantization is 1000, and the min is 0. When the vector (such as 5) of attention head A is quantized to INT8, the mapping value = 5x255 / 1000≈1.275, and the integer is 1, the error rate = (5-1x1000 / 255) / 5≈60%; the vector (such as 500) of attention head B is quantized to INT8, and the mapping value = 500x255 / 1000=127.5, and the integer is 128, the error rate≈0.2%. The quantization error of the low-range attention head is extremely large, which affects the model accuracy.

[0124] In the present application, attention head-level quantization is adopted. Based on the statistical information (such as max=10 of attention head A and max=1000 of attention head B) of a single attention head or similar attention heads, the quantization error is significantly reduced. For example, attention head A quantization: 5x255 / 10=127.5→integer 128, and after dequantization =128x10 / 255≈5.02, the error rate≈0.4%. Attention head B quantization: 500x255 / 1000=127.5→integer 128, and after dequantization =128x1000 / 255≈501.96, the error rate≈0.39%. Similar attention heads (such as attention heads responsible for entity recognition) have similar KV vector distributions, and sharing statistical information can balance the accuracy and computational complexity, while avoiding the "one-size-fits-all" error of global quantization.

[0125] The embodiments of the present application also support dynamic quantization (per batch / per sequence) or static quantization (calibration based on representative data).

[0126] Based on any of the above embodiments, refer to Figure 5 , Figure 5 is a reasoning step flowchart for cache migration and precision adjustment provided by the embodiments of the present application, which includes judging whether a migration / adjustment trigger condition is reached.

[0127] If the trigger condition is triggered, scan the cache metadata; sort by importance, where the sorted objects are all KV cache blocks stored by the current layer (i.e. cache blocks generated by historical time steps), and the sorting dimension is the importance score of the current time step re-evaluation (not the score at the initial allocation).

[0128] Determine whether migration is needed; if migration is needed, perform hierarchical migration. If migration is not needed, determine whether precision adjustment is needed. If needed, perform precision conversion, otherwise end the adjustment.

[0129] If the trigger condition is not triggered, continue with the next token, calculate the current layer attention, generate a new token, and store new KV data.

[0130] Load the input sequence, initialize the KV cache, calculate the current layer attention, generate a new token, and store new KV data.

[0131] In the embodiments of the present application, the initial allocation is based on the importance score of the current time step and the resource state, but the importance and resource state change dynamically during the inference process, and need to be adapted to the changes through migration / adjustment. The specific reasons include: the importance score of the same attention head changes with the input token (e.g. the importance of a certain attention head is high when processing the "topic sentence" and low when processing the "modifier"), the Level0 resource allocated initially may become "unnecessary" at the subsequent time step and needs to be migrated to Level1 to release the high-speed resource; conversely, if the importance of the attention head allocated to Level1 initially is improved, it needs to be migrated to Level0 to ensure the inference accuracy. When the storage of a certain level (e.g. Level0) is tight due to the cache writing of other attention heads, part of the cache blocks with decreased importance need to be migrated to the lower level to avoid the new high importance attention head from being unable to be allocated; if the lower level storage (e.g. Level1) is full, part of the low importance cache blocks need to be reduced in precision (e.g. from INT8 to INT4) or discarded to release the storage space. Autoregressive inference usually includes multiple time steps (e.g. generating 100 tokens), if only relying on the initial allocation, the high importance cache blocks of the lower level will continuously cause access delay, and through migration / adjustment, the matching relationship of "high importance-high speed and high precision" can be maintained for a long time, and the overall inference speed is optimized.

[0132] In the embodiments of the present application, the migration judgment is based on the combination of the "re-evaluated importance score" and the "state of each level storage", and the specific judgment rules are as follows: if the current importance score of the cache block is > θ_high, and there is remaining capacity in Level 0, then trigger "migration from the current level to Level 0"; if the current importance score of the cache block is ≤ θ_mid, and it is in Level 0 / Level 1, then trigger "migration from the current level to Level 2 (or reduce the precision, such as from FP16 to INT4)"; if the current importance score of the cache block is between θ_mid and θ_high, and there is capacity in Level 1 but no capacity in Level 0, then trigger "migration from Level 0 to Level 1" (if currently in Level 2, then migrate to Level 1); if the capacity of a level storage (such as Level 1) is full, and there is a cache block with an importance score < θ_low, then trigger "migration from the level to Level 2 (discard)".

[0133] The current layer attention calculation follows the Transformer multi-attention head attention mechanism, and the specific steps are as follows: performing linear transformation on the embedding vector of the current input Token to generate a Query (query) vector; retrieving the KV cache block (after dequantization) stored in the historical time step to obtain a Key (key) vector and a Value (value) vector; calculating the dot product of Query and Key and scaling (dividing by , d_k is the dimension of the Key vector) to obtain the original attention score; performing Softmax normalization on the original attention score to obtain the normalized attention score; weighting and summing the Value vector with the normalized attention score to obtain the attention output (attn_output) of the current attention head; after linear transformation of the attention outputs of all attention heads, the outputs are combined with the output of the feedforward network to complete the inference of the current layer.

[0134] After the current layer attention calculation, the output is the core result of the inference of the current layer model, and the probability distribution of the next Token is finally generated through the Softmax of the output layer.

[0135] Further, referring to Figure 6 , Figure 6is a layered storage resource pool and cache manager interaction schematic diagram provided by the embodiment of the application. Among them, the embodiment of the application adopts real-time scheduling, and the specific reason is that the remaining capacity of the first each level storage changes in real time (such as the capacity of Level0 decreases due to new cache writing and the capacity increases due to old cache migration to Level1) with the writing, elimination and migration of KV cache, real-time scheduling can adjust the allocation strategy according to the latest capacity, avoid the failure caused by the allocation based on the expired resource state; second, the importance score of the same attention head fluctuates with the change of the input sequence (such as the importance of a certain attention head is high when processing "entity recognition" related Token, and the importance is low when processing "punctuation symbol"), real-time scheduling can trigger the migration of the cache block between different levels (such as migrating from Level0 to Level1 when the importance decreases), and optimize the resource utilization rate; third, if the model processes multiple inference tasks at the same time (such as multiple user requests), the KV cache of different tasks will compete for the same layered resource pool, and real-time scheduling can dynamically coordinate the resource allocation of each task according to the task priority and resource state, and avoid the resource preemption between tasks.

[0136] And it can be understood that the technical scheme of the application can realize attention head granularity importance dynamic evaluation; elastic precision, layered storage resource pool; and state-aware, dynamic allocation and scheduling mechanism.

[0137] For attention head granularity importance dynamic evaluation. Based on the change of the importance score of the attention head, the data migration engine asynchronously executes the migration of the KV cache block between different storage levels in the background (such as migrating from CPU DRAM to GPU HBM when the importance increases, and vice versa when the importance decreases) to avoid resource waste caused by fixed storage levels, and to ensure that high importance cache blocks are always in high-speed storage, and low importance cache blocks release high-speed resources.

[0138] For elastic precision, layered storage resource pool (position + precision). For a single or a group of similar attention heads, the numerical precision of the KV cache is dynamically adjusted (such as using FP16 for high importance attention heads and using INT4 for low importance attention heads), and the precision conversion is based on the statistical information (such as min / max, mean / std) of the attention head rather than global uniform quantization; while ensuring the precision of high importance attention heads, the memory occupation of low importance attention heads is maximized, and the precision loss is much lower than uniform quantization.

[0139] For state-aware, dynamic allocation and scheduling mechanism. The resource monitor collects the remaining capacity, cache block metadata (position, precision, importance), migration / transition state of each storage level in real time, and provides real-time data support for the allocation decision maker and the scheduler. Ensure that resource allocation and scheduling are based on the latest state, avoid overallocation, misallocation, and improve the stability and efficiency of the entire cache management system.

[0140] Further, the determination of the importance score is further described below.

[0141] In one possible implementation of the embodiment of the present application, the importance score of the target attention head in the current layer is calculated in real time, including: calculating the importance score of the target attention head in the current layer in real time according to the input sequence of the model inference of the target attention head in the current layer.

[0142] In one possible implementation of the embodiment of the present application, the importance score of the target attention head in the current layer is calculated in real time, including: if the calculation mode is a single selection mode, calculating the importance score of the target attention head in the current layer in real time according to the target single calculation mode corresponding to the single selection mode; if the calculation mode is a weighted comprehensive mode, performing weighted calculation according to the values calculated by each single calculation mode to obtain the importance score of the target attention head in the current layer.

[0143] The single calculation mode includes: an attention output-based mode, an attention score-based mode, a historical information-based mode, and a lightweight model mode.

[0144] The attention output-based mode is to calculate the norm (L1, L2), entropy or contribution gradient to the final layer output of the attention vector (attn_output) output by the attention head. The attention score-based mode is to analyze the distribution (such as entropy, sparsity, maximum value) of the attention score (attn_weights) generated by the attention head. More concentrated or more dispersed attention may indicate its importance. The historical information-based mode is to smooth or predict the importance score of the attention head at the previous time step in combination with the importance score. The lightweight model is to train a small auxiliary model (such as MLP) to quickly predict the importance score of each attention head according to the input, state or evaluation result of the previous time step of the current layer. The model can be trained offline.

[0145] The calculation process of the attention output-based mode includes: inputting the input sequence into the corresponding attention head to obtain an output vector; determining the vector norm according to the output vector; obtaining the attention score matrix of the attention head when the input sequence is inferred; calculating the entropy of each row of the attention score matrix; determining the entropy of the attention head according to the entropy of each row; and determining the first importance score according to the vector norm and / or the entropy of the attention head.

[0146] Specifically, the L2 norm calculation based on the attention output mode. According to the output vector of the current attention head (d is the vector dimension, such as 512) to calculate the initial norm (S_norm) by using a preset formula, and the preset formula includes: where d represents the dimension of the vector, and xi represents the component of the vector in the i-th dimension. The S-norm is mapped to [0, 1] according to the formula: S-norm (normalized) = (S-norm-min_S) / (max_S-min_S); where min_S is the minimum L2 norm of all attention heads in the current layer, and max_S is the maximum L2 norm of all attention heads in the current layer; S-norm (normalized) is the vector norm.

[0147] Entropy calculation based on attention score. According to the attention score matrix of the current attention head , (t is the number of generated tokens, such as 100), each row represents the attention weight of the current token on the historical token. First, perform Softmax normalization on each row to obtain p_{i,j} (the normalized weight of the i-th token on the j-th token). Calculate the average value of the entropy of each row as the entropy value score of the attention head, where, , is the entropy of the i-th row, ; normalization and reverse mapping, the smaller the entropy value (the more focused the attention), the higher the importance of the attention head, so reverse mapping is required: S_entropy (normalized) = 1-(S_entropy-min_H) / (max_H-min_H), where min_H is the minimum entropy value of all attention heads in the current layer, max_H is the maximum entropy value of all attention heads in the current layer, and S_entropy (normalized) is the entropy of the attention head.

[0148] The vector norm or the entropy of the attention head can be used as the first importance score; both can also be weighted to obtain the first importance score.

[0149] wherein the calculation process based on the attention score mode includes: converting the input sequence into a query vector and a key vector; splitting the query vector and the key vector according to the number of attention heads; determining the attention score according to the query vector and the key vector of the attention head as the second importance score.

[0150] The attention score is generated by the "Query-Key interaction" of the Transformer multi-attention head attention mechanism, and the specific steps are: linear transformation to generate a query vector Query and a key vector Key; according to the input sequence vector (t is the number of tokens, and dmodel is the model dimension, such as 768). Through three independent linear layers (weight matrix where dk=dmodel / h, and h is the number of attention heads), the input sequence vector X is converted into Query (Q) and Key (K), wherein Q=X×W_Q; K=X×W_K; Q and K are split according to the number of attention heads h, and the (h = 12, d_model = 768, then d_k = 64). Compute the dot product of Q and K for each attention head, and divide by (avoiding the saturation of Softmax due to too large scores), where, , For autoregressive tasks (e.g., text generation), add a "lower triangular mask" to make the current Token only focus on the historical Token (avoiding future information leakage), and set the score of the mask position to -∞. For the original score Perform Softmax normalization to get the final attention score attn_weights_h (each row sum is 1, representing the attention weight between Tokens), where attn_weights_h = Softmax(raw_scores_h + mask).

[0151] Wherein, the calculation process based on the historical information mode comprises: obtaining the historical importance score of the attention head; predicting the importance score of the attention head according to the historical importance score of the attention head as the third importance score.

[0152] Wherein, the calculation process of the lightweight model mode comprises: according to the input sequence of the attention head, the state of the current layer, and the evaluation result of the previous time step, the importance score of the target attention head in the current layer is predicted by using a multi-layer perception model as the fourth importance score.

[0153] Wherein, according to the local feature of the current attention head, the importance score of the target attention head in the current layer is predicted by using a multi-layer perception model as the fourth importance score.

[0154] Specifically, the local feature of the current attention head includes: input sequence, state of the current layer, and evaluation result of the previous time step. Wherein, the embedding vector mean of the input Token of the input sequence of the attention head (such as the average value of the embedding vector of the input Token at the current time step), and the L1 norm of the Query vector of the attention head (representing the activation strength of Query). The state of the current layer is the "global state" of the model, which includes: the layer normalization mean of the current layer (reflecting the activation level of the current layer), and the number of generated Tokens (reflecting the context window length). The evaluation result of the previous time step (Prev_Score) represents the importance score of the attention head at the previous inference time step (such as the comprehensive score at t-1 time).

[0155] In an implementable manner, the lightweight auxiliary model (a multi-layer perception model) can adopt a "2-layer MLP (multi-layer perception)" structure, offline training and online prediction. Specifically, offline training (based on a labeled data set) includes: collecting inference data of different tasks (text generation, question answering), recording the "input + state + last time step score" of each attention head, and taking the "contribution of the attention head to the final output" (such as the model accuracy drop value after removing the attention head) as the label. The input sequence is spliced (such as 32 dimensions), and the MLP (64 neurons in the first layer, ReLU activation; 1 neuron in the second layer, Sigmoid activation) is input, and the MSE loss of the predicted score and the label is minimized. Online prediction (in the inference process) includes: real-time acquisition of the "input + state + last time step score" of the current attention head, and standardization (based on the mean and standard deviation of the training set). The standardized features are input into the pre-trained MLP, and the importance score Sauxiliary model in the interval [0, 1] is output.

[0156] Referring to Figure 7 , Figure 7 A flow chart of a method for dynamically evaluating the importance of an attention head is provided for an embodiment of the application. The importance score is calculated based on the norm of the attention output and the historical score.

[0157] Obtain the current attention head data; calculate the L2 norm of the current attention output; and determine whether it is the first token. If it is the first token, initialize the historical score: the current norm value; set the importance score = the current norm; and output the importance score. If it is not the first token, read the historical score (exponential moving average); calculate the smoothed score: ; update the historical score storage; set the importance score = the smoothed score; and output the importance score. Determine whether all attention heads have been evaluated. If yes, end the evaluation, and if no, move to the next attention head.

[0158] It can be understood that in autoregressive generation, the first token refers to the first generated token of the sequence. At this time, there is no historical KVCache, so the importance evaluation cannot rely on historical information and needs to be handled separately (such as using an initial threshold or a default value).

[0159] The first input token received in the model inference process is referred to. For example, in a text generation task, the first token in the user input prefix text (such as "write an article" corresponding to the first token "write"); in a long sequence inference task, the first token in the input long document (such as the first token "agreement" of a legal contract).

[0160] The first Token has no historical KVCache, so the importance evaluation does not need to combine historical scores (directly use the current L2 norm as the importance score).

[0161] The smoothing score solves the problem of "the importance score of the subsequent Token fluctuates too much after the first Token". For example, the score of a certain attention head at the first Token is 1.0 (high importance), and the score of the second Token decreases to 0.3 (low importance) due to irrelevant input (such as punctuation). The smoothing score can alleviate this fluctuation by using the historical score (1.0), avoiding frequent adjustments of resource allocation. The larger the α, the stronger the influence of the current time step feature on the smoothing score, and the faster the adaptation to input changes. The smaller the α, the higher the weight of the historical score, and the more significant the smoothing effect. In practical applications, α is usually set to 0.3-0.5 (e.g., α=0.4), balancing between dynamic adaptation and stability.

[0162] The value of the historical score weight coefficient α is determined by the "input volatility" of the task: high input volatility tasks (such as dialogue generation, frequent input topic switching): α=0.5, enhancing the weight of the current score, quickly adapting to input changes. Low input volatility tasks (such as document summarization, stable input theme): α=0.3, enhancing the weight of the historical score, maintaining the stability of the score.

[0163] Further, the entropy value of the attention weight can be calculated, and the comprehensive score can be calculated based on the norm and entropy value.

[0164] The comprehensive score is calculated by weighted summation, fully combining the contribution degree of attention output (norm representation) and focus (entropy representation), as follows: .

[0165] The L2 norm of the attention head output vector (attn_output) represents the contribution degree of the attention head to the current reasoning, and the larger the norm, the higher the contribution. Then map it to the [0,1] interval (such as through Min-Max normalization, to eliminate the numerical scale difference between different attention heads); the smaller the distribution entropy of the attention score (attn_weights), the more focused the attention on the key Token, and the higher the importance of the attention head. Then calculate "1-normalized entropy value" (convert the "negative correlation" of entropy value to "positive correlation", i.e. the smaller the entropy value, the larger the term), also mapped to the [0,1] interval; the weight setting is: and are weight coefficients (satisfying ), which can be adjusted according to the characteristics of the task, for example, in text generation tasks , the contribution degree of the attention head is given priority; in long sequence understanding tasks , the contribution degree and focus are balanced.

[0166] In a possible implementation of the embodiment of the present application, when the key-value cache generated by the target attention head is accessed, it is determined whether the storage precision of the key-value cache is the required precision for calculation; if not, a dequantization operation is performed according to the storage location and the storage precision corresponding to the target attention head.

[0167] When the KVCache needs to be accessed in the subsequent inference step, the storage location and the precision are searched, and a dequantization operation is performed if necessary. In a possible case, when the KVCache is retrieved from low-precision storage (such as INT8 / INT4) for subsequent attention calculation, it needs to be dequantized to the required precision (such as FP16) for model calculation. Linear quantization (Min-Max quantization) can be used: ; wherein min and max are the original value range of the attention head (or the group), which can be saved during quantization.

[0168] Specifically, when the retrieved KVCache is “low-precision storage” and “subsequent inference requires high-precision calculation”, dequantization needs to be performed. For example, the KVCache of the medium importance attention head is stored in the CPU DRAM with INT8 precision, and the low importance attention head is stored in the SSD with INT4 precision. For another example, the attention calculation (dot product of Query and Key) of the Transformer model requires uniform precision (usually FP16 / BF16), and the low-precision KV vector needs to be converted to FP16 / BF16 before participating in the calculation.

[0169] Dequantization is performed based on the “attention head level statistical information recorded during storage”. Taking INT8→FP16 dequantization as an example, it includes the following steps: Step 1: Read metadata. The cache manager obtains the quantization parameters (recorded during storage) of the KVCache of the attention head from the metadata: including the maximum value (max_val) and the minimum value (min_val) of the KV vector of the attention head (based on the statistics of all elements in the attention head). Step 2: Perform dequantization calculation. The formula is as follows: ; wherein max_val / min_val: the maximum and minimum values of the original FP16 data; the INT8 value range is 0-255, which is first mapped to -127.5-127.5, and then scaled to the max_val-min_val range of the original FP16. Step 3: Precision alignment. The dequantized FP16 vector is transmitted to the GPU and aligned with the current Query vector (FP16) in precision to participate in attention calculation.

[0170] Based on any of the above embodiments, one or more embodiments implementing the above method are described in detail, including: 1. The specific Transformer model used (such as LLaMA, GPT-NeoX).

[0171] 2. Specific algorithm of importance evaluation (e.g. using L2 norm of attn_output as initial score, combined with exponential moving average smoothing).

[0172] 3. Specific levels of defined hierarchical resource pool (e.g. Level0-GPU HBM FP16; Level1-CPU DRAM INT8; Level2-Discard).

[0173] 4. Specific rules of allocation policy (e.g. score>θ_high->Level0; θ_mid<score<=θ_high->Level1; score<=θ_mid->Level2).

[0174] 5. Storage format of KVCache (how to organize data of different precision and different location).

[0175] 6. Specific implementation of cache manager (metadata structure, migration trigger condition, eviction algorithm).

[0176] 7. Specific implementation of quantization / dequantization (e.g. using dynamic min-max quantization to INT8 per attention head).

[0177] Specifically, a dynamic hierarchical key-value cache management method based on attention head importance perception is applied to the inference process of a Transformer model, which includes: in the model inference process, the importance score of each attention head in the current layer is calculated in real time; a hierarchical storage resource pool containing multiple storage locations and multiple numerical precisions is defined; according to the importance score of each attention head, it is dynamically decided which level of the hierarchical storage resource pool the key-value cache generated by the attention head should be allocated to, and the level specifies the storage location and numerical precision; the key-value cache is stored in the storage location specified by the decision with the numerical precision specified by the decision; when the key-value cache needs to be accessed in the subsequent inference step, it is retrieved and converted according to its storage location and numerical precision.

[0178] Further, the method of calculating the importance score of the attention head in real time includes at least one of the following: calculating the norm or entropy of the attention vector output by the attention head, calculating the distribution characteristics of the attention score generated by the attention head, smoothing or predicting based on the historical importance score of the attention head, and predicting using a lightweight auxiliary model.

[0179] Further, the hierarchical storage resource pool includes at least: a first level (high-speed storage): a combination of a graphics processor high-speed video memory and a floating-point 16 or brain floating-point 16 precision; a second level (medium-speed storage): a combination of a central processing unit memory and an integer 8-bit precision; a third level (low-speed storage): a combination of a solid-state disk storage and an integer 4-bit or lower precision.

[0180] Further, the dynamic decision adopts an interval division method based on an importance score threshold or a priority allocation method based on importance ranking of attention heads in the same layer.

[0181] Further, the method further comprises a cache management step: according to the storage resource usage state and the change of the attention head importance score, performing the operations of evicting, migrating between different storage location levels or adjusting the numerical precision on the stored key-value cache.

[0182] Further, the numerical precision conversion adopts a quantization method based on statistical information of a single attention head or a group of similar attention heads.

[0183] Further, the method comprises: an importance evaluation module configured to calculate the importance score of each attention head in real time; a hierarchical resource pool module providing storage resources of multiple storage locations and multiple numerical precisions; an allocation decision maker module configured to dynamically decide the storage level of the key-value cache according to the importance score; and a cache manager module configured to manage the storage, retrieval, metadata maintenance of the key-value cache and perform the operations of evicting, migrating and precision adjusting.

[0184] It can be seen that based on the above scheme, significant memory saving can be achieved, wherein by storing or even discarding the KVCache of a large number of low-importance attention heads at low precision or transferring them to cheap low-speed storage, the overall KVCache memory occupancy can be reduced by 30%-70% (the specific proportion depends on the model, task, resource pool definition and strategy aggressiveness), which is much better than uniform quantization (usually saving 50% but with large precision loss) or simple discarding. The model precision can also be maintained, wherein by protecting the high precision and fast access of high-importance attention heads, the loss of key information is minimized, and the decline in model output quality (such as the fluency, factual accuracy and task performance of generated text) is much lower than that of uniform compression methods under the same memory saving. Experiments can compare the Perplexity, BLEU, accuracy and other indicators. Inference acceleration can also be achieved, wherein reducing GPU memory occupancy can reduce memory contention and possibly improve core computing efficiency. Smaller memory occupancy can allow larger BatchSize or longer sequence processing. (If high-speed storage resources are sufficient) to ensure the fastest access of key attention heads. Operations offloaded to CPU / Disk need to be cautious, as the delay may offset some of the benefits. The optimization point is that only low-importance attention heads bear high-latency access, and these attention heads themselves have little impact on the output. It is more flexible, wherein the method can be adapted to different model architectures, different hardware platforms (servers, edge devices), and different task requirements (by adjusting strategy parameters to balance memory / speed / precision). Resource optimization is achieved, wherein the most expensive resources (high-speed memory, high-precision computing) are precisely used for the calculation part that has the greatest impact on the output, improving resource utilization.

[0185] Figure 8 A structure diagram of a cache management device provided by an embodiment of the present application comprises: an importance score determination module 210, configured to calculate, in a model inference process, an importance score of a target attention head in a current layer in real time, the target attention head being any one of a plurality of attention heads in the current layer; a storage level determination module 220, configured to determine, according to the importance score of the target attention head in the current layer, a storage level corresponding to a key-value cache generated by the target attention head, the storage level comprising a storage location and a storage precision; and a storage module 230, configured to store the key-value cache according to the storage location and the storage precision corresponding to the key-value cache generated by each attention head.

[0186] In a possible implementation of the embodiment of the present application, the storage module 230 is configured to: sort the importance scores of all the attention heads of the current layer to obtain a first sorting result; determine, according to the sorting result, a key-value cache with the highest importance score; determine whether a storage level corresponding to the key-value cache with the highest importance score is available; if the storage level is available, store the key-value cache with the highest importance score to a corresponding cache location according to the storage precision; if the storage level is not available, adjust the level of the storage level corresponding to the key-value cache with the highest importance score; and store the key-value cache according to the storage location and the storage precision of the storage level corresponding to the key-value cache with the highest importance score after the level adjustment; after the storage of the key-value cache with the highest importance score is completed, sequentially process a next key-value cache until the storage of the key-value caches generated by all the attention heads is completed.

[0187] In a possible implementation of the embodiment of the present application, the storage module 230 is configured to: if the storage level corresponding to the key-value cache with the high importance score is high-speed storage, determine whether medium-speed storage corresponding to the storage level of the key-value cache with the highest importance score is available; if the medium-speed storage is available, downgrade the storage level corresponding to the key-value cache with the highest importance score to the medium-speed storage; and if the medium-speed storage is not available, downgrade the storage level corresponding to the key-value cache with the highest importance score to low-speed storage.

[0188] In a possible implementation of the embodiment of the present application, the storage module 230 is configured to: if the storage level corresponding to the key-value cache with the high importance score is medium-speed storage, determine whether low-speed storage corresponding to the storage level of the key-value cache with the highest importance score is available; if the low-speed storage is available, downgrade the storage level corresponding to the key-value cache with the highest importance score to the low-speed storage; and if the low-speed storage is not available, upgrade the storage level corresponding to the key-value cache with the highest importance score to high-speed storage.

[0189] In a possible implementation of the embodiment of the application, the storage module 230 is configured to: if the key-value cache with the highest importance score is unavailable, determine whether the storage level corresponding to the key-value cache is low-speed storage; if the storage level corresponding to the key-value cache with the highest importance score is not low-speed storage, perform level adjustment on the storage level corresponding to the key-value cache with the highest importance score; and if the storage level corresponding to the key-value cache with the highest importance score is low-speed storage, discard the key-value cache with the highest importance score.

[0190] In a possible implementation of the embodiment of the application, the management module is further configured to: acquire the storage resource usage of each storage level of the hierarchical storage resource pool; and adjust the hierarchical storage resource pool according to the storage resource usage and the importance score of the storage level.

[0191] In a possible implementation of the embodiment of the application, the storage resource usage includes a remaining capacity, and the storage level includes low-speed storage, medium-speed storage, and high-speed storage; and the management module is configured to: determine whether the remaining capacity of the storage level is less than a preset capacity threshold; if the remaining capacity of the storage level is less than the preset capacity threshold, filter a target key-value cache that meets an adjustment requirement in the storage level according to the importance score; and adjust the hierarchical storage resource pool according to the target key-value cache that meets the adjustment requirement in the storage level.

[0192] In a possible implementation of the embodiment of the application, the management module is configured to: adjust the hierarchical storage resource pool according to the adjustment instruction, according to the target key-value cache that meets the adjustment requirement in the storage level.

[0193] In a possible implementation of the embodiment of the application, if the adjustment instruction is an elimination instruction, the storage level is low-speed storage of the hierarchical storage resource pool, and the management module is configured to: sort the target key-value cache based on at least one of the importance score and a last access time according to the elimination instruction; and eliminate the target key-value cache in sequence based on a sorting result, until the remaining capacity of the low-speed storage is not less than a first capacity.

[0194] In a possible implementation of the embodiment of the application, if the adjustment instruction is a migration instruction, the storage level is medium-speed storage or high-speed storage of the hierarchical storage resource pool, and the management module is configured to: migrate the target key-value cache to a next level of storage according to the migration instruction, and update a storage location of the target key-value cache in metadata.

[0195] In a possible implementation of the embodiment of the application, the management module is further configured to: mark a state of the migrated key-value cache as a migration state during the migration process, the migration state being used to prohibit other operations; and mark the state of the migrated key-value cache as an available state after the migration is completed.

[0196] In a possible implementation of the embodiment of the present application, if the adjustment instruction is a precision adjustment instruction, the storage level is a high-speed storage of a hierarchical storage resource pool; the management module is configured to: according to the precision adjustment instruction, reduce the storage precision of the target key-value cache, replace the original target key-value cache, and update the precision information of the target key-value cache in the metadata.

[0197] In a possible implementation of the embodiment of the present application, the importance score determination module 210 is configured to: according to the input sequence of the model inference of the target attention head in the current layer, calculate the importance score of the target attention head in the current layer in real time.

[0198] In a possible implementation of the embodiment of the present application, the importance score determination module 210 is configured to: if the calculation mode is a single selection mode, according to the target single calculation mode corresponding to the single selection mode, calculate the importance score of the target attention head in the current layer in real time; if the calculation mode is a weighted comprehensive mode, perform weighted calculation according to the values calculated by each single calculation mode to obtain the importance score of the target attention head in the current layer.

[0199] In a possible implementation of the embodiment of the present application, the single calculation mode includes: an attention output mode-based calculation mode, an attention score mode-based calculation mode, a historical information mode-based calculation mode, and a lightweight model mode, wherein the calculation process of the attention output mode-based calculation mode includes: inputting the input sequence into the corresponding attention head to obtain an output vector; determining a vector norm according to the output vector; obtaining an attention score matrix of the attention head when the input sequence is inferred; calculating the entropy of each row of the attention score matrix; determining the entropy of the attention head according to the entropy of each row; determining a first importance score according to the vector norm and / or the entropy of the attention head; the calculation process of the attention score mode-based calculation mode includes: converting the input sequence into a query vector and a key vector; splitting the query vector and the key vector according to the number of attention heads; determining the attention score as a second importance score according to the query vector and the key vector of the attention head; the calculation process of the historical information mode-based calculation mode includes: obtaining the historical importance score of the attention head; predicting the importance score of the attention head as a third importance score according to the historical importance score of the attention head; the calculation process of the lightweight model mode includes: predicting the importance score of the target attention head in the current layer as a fourth importance score by using a multi-layer perception model according to the input sequence of the attention head, the state of the current layer, and the evaluation result of the previous time step.

[0200] In a possible implementation of the embodiment of the present application, the dequantization module is configured to: when accessing the key-value cache generated by the target attention head, determine whether the storage precision of the key-value cache is the required calculation precision; if not, perform a dequantization operation according to the storage location and the storage precision corresponding to the target attention head.

[0201] Figure 8 The features of the corresponding embodiments can be seen from the above description of the embodiments Figure 2 The related description of the corresponding embodiments will not be repeated here.

[0202] Figure 9 A structural diagram of an electronic device provided by the embodiments of the present application is shown in FIG. 6. Figure 9 As shown in FIG. 6, the electronic device includes a memory 60 for storing a computer program and a processor 61 for executing the computer program to implement the steps of the cache management method of the above-described embodiments.

[0203] The electronic device provided by the embodiments of the present application can include but is not limited to a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.

[0204] The processor 61 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one of a hardware form of a Digital Signal Processing (DSP), a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA). The processor 61 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a Central Processing Unit (CPU). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 61 can be integrated with a Graphics Processing Unit (GPU). The GPU is used to render and draw the content required to be displayed on the display screen. In some embodiments, the processor 61 can also include an Artificial Intelligence (AI) processor. The AI processor is used to process computing operations related to machine learning.

[0205] The memory 60 can include one or more computer-readable storage media. The memory 60 can also include high-speed random access memory and non-volatile, computer-readable storage media such as one or more magnetic disk storage devices, flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, wherein the computer program is loaded and executed by the processor 61 and can implement the steps of the cache management method disclosed in any of the preceding embodiments. In addition, the resources stored in the memory 60 can also include an operating system 602, data 603, etc., and the storage mode can be temporary storage or permanent storage. The operating system 602 can include Windows, Unix, Linux, etc.

[0206] In some embodiments, the electronic device can further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0207] Those skilled in the art can understand that the structure shown in the above embodiments does not constitute a limitation on the electronic device, and can include more or fewer components than those shown in the figure. Figure 9

[0208] It can be understood that if the cache management method in the above embodiments is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and executes all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM, a register, a hard disk, a removable magnetic disk, a CD-ROM, a magnetic disk or an optical disk, and various media that can store program codes.

[0209] Based on this, the embodiment of the present application further provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the cache management method.

[0210] Based on this, the embodiment of the present application further provides a computer program product, which includes computer programs / instructions. The computer programs / instructions are executed by a processor to implement the steps of the cache management method.

[0211] ​The above describes in detail a cache management method, device, equipment, medium and product provided by the embodiment of the application. Each embodiment in the specification is described in a progressive manner, and each embodiment mainly describes the difference from other embodiments. The same or similar parts of each embodiment can be understood by referring to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be understood by referring to the method part.

[0212] Those skilled in the art will further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, each example has been described in general terms in the above description. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0213] The above describes in detail a cache management method, device, equipment, medium and product provided by the embodiment of the application. Each embodiment in the specification is described in a progressive manner, and each embodiment mainly describes the difference from other embodiments. The same or similar parts of each embodiment can be understood by referring to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be understood by referring to the method part.

Claims

1. A cache management method, characterized in that, include: During model inference, the importance score of the target attention head in the current layer is calculated in real time. The target attention head is any one of multiple attention heads in the current layer. Based on the importance score of the target attention head in the current layer, the storage level corresponding to the key-value cache generated by the target attention head is determined, and the storage level includes storage location and storage precision; Store the key-value cache according to the storage location and storage precision of the key-value cache generated by each attention head.

2. The cache management method according to claim 1, characterized in that, Based on the storage location and precision of the key-value cache generated by each attention head, the key-value cache is stored, including: Sort the importance scores of all attention heads in the current layer to obtain the first sorting result; Based on the sorting results, determine the key-value cache with the highest importance score; Determine whether the storage level corresponding to the key-value cache with the highest importance score is available; If available, the key-value cache with the highest importance score will be stored in the corresponding cache location according to the storage precision. If unavailable, the storage level corresponding to the key-value cache with the highest importance score is adjusted; and the key-value cache is stored according to the adjusted storage location and storage precision based on the level corresponding to the key-value cache with the highest importance score. After storing the key-value cache with the highest importance score, process the next key-value cache in sequence until storing the key-value cache generated by each attention head is completed.

3. The cache management method according to claim 2, characterized in that, Adjust the storage level of the key-value cache corresponding to the highest importance score, including: If the storage level corresponding to the key-value cache with the high importance score is high-speed storage, then determine whether the medium-speed storage of the storage level corresponding to the key-value cache with the highest importance score is available; If medium-speed storage is available, the storage tier corresponding to the key-value cache with the highest importance score is downgraded to medium-speed storage; If medium-speed storage is unavailable, the storage tier corresponding to the key-value cache with the highest importance score is downgraded to low-speed storage.

4. The cache management method according to claim 2, characterized in that, Adjust the storage level of the key-value cache corresponding to the highest importance score, including: If the storage level corresponding to the key-value cache with the high importance score is medium-speed storage, then determine whether the low-speed storage of the storage level corresponding to the key-value cache with the highest importance score is available; If low-speed storage is available, the storage level corresponding to the key-value cache with the highest importance score is downgraded to low-speed storage; If low-speed storage is unavailable, the storage level corresponding to the key-value cache with the highest importance score is upgraded to high-speed storage.

5. The cache management method according to claim 2, characterized in that, If unavailable, the storage level corresponding to the key-value cache with the highest importance score will be adjusted, including: If unavailable, determine whether the storage level corresponding to the key-value cache with the highest importance score is low-speed storage; If it is not low-speed storage, then the storage level corresponding to the key-value cache with the highest importance score will be adjusted. The method further includes: If the storage is slow, the key-value cache with the highest importance score is discarded.

6. The cache management method according to claim 1, characterized in that, Also includes: Obtain the storage resource usage status of each storage tier in the tiered storage resource pool; Adjust the tiered storage resource pools based on the storage resource usage and importance score of each storage tier.

7. The cache management method according to claim 6, characterized in that, Storage resource usage includes: remaining capacity; storage tiers include: low-speed storage, medium-speed storage, and high-speed storage; Adjust the tiered storage resource pools based on storage resource usage and importance scores at each storage tier, including: Determine whether the remaining capacity of the storage tier is less than a preset capacity threshold; If the remaining capacity of a storage tier is less than a preset capacity threshold, then target key-value caches that meet the adjustment requirements in the storage tier are selected based on their importance scores. Based on the target key-value caches that meet the adjustment requirements in the storage tier, adjust the tiered storage resource pool.

8. The cache management method according to claim 7, characterized in that, Based on the selection of target key-value caches that meet the adjustment requirements in the storage tier, the tiered storage resource pool is adjusted, including: Based on the target key-value cache that meets the adjustment requirements in the storage tier, the tiered storage resource pool is adjusted according to the adjustment instructions.

9. The cache management method according to claim 8, characterized in that, If the adjustment instruction is a replacement instruction, then the storage tier is the low-speed storage of the tiered storage resource pool. For the target key-value cache, adjust the tiered storage resource pool according to the adjustment instructions, including: Based on the eviction instructions, sort the target key-value cache according to at least one of the importance score and the last access time; Based on the sorting results, the target key-value cache is evicted in turn until the remaining capacity of the low-speed storage is not less than the first capacity.

10. The cache management method according to claim 8, characterized in that, If the adjustment instruction is a migration instruction, then the storage tier is a medium-speed or high-speed storage within a tiered storage resource pool. For the target key-value cache, adjust the tiered storage resource pool according to the adjustment instructions, including: According to the migration instructions, the target key-value cache is migrated to the next level of storage, and the storage location of the target key-value cache in the metadata is updated.

11. The cache management method according to claim 10, characterized in that, Also includes: During the migration process, the state of the migrated key-value cache is marked as the migration state, which is used to prohibit other operations; After the migration is complete, mark the migrated key-value cache as available.

12. The cache management method according to claim 8, characterized in that, If the adjustment instruction is a precision adjustment instruction, then the storage level is a high-speed storage of a tiered storage resource pool; Accordingly, for the target key-value cache, the tiered storage resource pool is adjusted according to the adjustment instructions, including: Based on the precision adjustment instruction, reduce the storage precision of the target key-value cache, replace the original target key-value cache, and update the precision information of the target key-value cache in the metadata.

13. The cache management method according to any one of claims 1-12, characterized in that, Real-time calculation of the importance score of the target attention head in the current layer, including: The importance score of the target attention head in the current layer is calculated in real time based on the input sequence of the model inference of the target attention head in the current layer.

14. The cache management method according to claim 13, characterized in that, Real-time calculation of the importance score of the target attention head in the current layer, including: If the calculation mode is a single-selection mode, then the importance score of the target attention head in the current layer is calculated in real time according to the target single-item calculation mode corresponding to the single-selection mode. If the calculation mode is a weighted comprehensive mode, then the values ​​obtained from each individual calculation mode are weighted and calculated to obtain the importance score of the target attention head in the current layer.

15. The cache management method according to claim 14, characterized in that, Single-item calculation modes include: attention-based output mode, attention-score-based mode, historical information-based mode, and lightweight model mode. The calculation process based on the attention output pattern includes: inputting the input sequence into the corresponding attention head to obtain the output vector; determining the vector norm based on the output vector; obtaining the attention score matrix of the attention head when inferring the input sequence; calculating the entropy of each row of the attention score matrix; determining the entropy of the attention head based on the entropy of each row; and determining the first importance score based on the vector norm and / or the entropy of the attention head. The calculation process based on the attention score pattern includes: converting the input sequence into query vectors and key vectors; splitting the query vectors and key vectors according to the number of attention heads; and determining the attention score based on the query vectors and key vectors of the attention heads, which serves as the second importance score. The calculation process based on historical information patterns includes: obtaining the historical importance score of the attention head; predicting the importance score of the attention head based on the historical importance score of the attention head, which is used as the third importance score; The computation process of the lightweight model mode includes: based on the input sequence of the attention head, the state of the current layer, and the evaluation result of the previous time step, using the multilayer perceptron model, predicting the importance score of the target attention head in the current layer, which is used as the fourth importance score.

16. The cache management method according to claim 1, characterized in that, Also includes: When accessing the key-value cache generated by the target attention head, determine whether the storage precision of the key-value cache is the precision required for calculation; If not, then perform inverse quantization based on the storage location and storage precision corresponding to the target attention head.

17. A cache management device, characterized in that, include: The importance score determination module is used to calculate the importance score of the target attention head in the current layer in real time during the model inference process. The target attention head is any one of multiple attention heads in the current layer. The storage level determination module is used to determine the storage level corresponding to the key-value cache generated by the target attention head based on the importance score of the target attention head in the current layer. The storage level includes storage location and storage precision. The storage module is used to store the key-value cache according to the storage location and storage precision of the key-value cache generated by each attention head.

18. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the cache management method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the cache management method as described in any one of claims 1 to 16.

20. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the cache management method according to any one of claims 1 to 16.

Images