KV cache management method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a KV cache management method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-managed key value pair KV block in response to a large model reasoning request; collecting feature attributes of the KV block, wherein the feature attributes at least comprise a recalculation cost index, an access heat index and a life cycle index; calculating a retention score of the KV block based on the feature attribute; wherein the higher the recalculation cost index is, the higher the retention score is; monitoring the resource occupation state of the current KV cache storage area; when the resource occupancy state meets the eviction condition, based on the retention score, selecting a KV block of which the retention score meets a preset elimination rule from the KV cache storage area as a target KV block; and performing a staged processing operation on the target KV block. According to the method, the recalculation cost factor is combined when the KV blocks are eliminated, the situation that the KV blocks with high recalculation cost are eliminated from the KV cache too early is avoided, and the hit rate of KV cache access is increased.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a KV cache management method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the widespread adoption of large-scale pre-trained language models (LLMs) in dialogue systems, retrieval-enhanced generation (RAG), code generation, and multimodal applications, the computational cost and latency during the inference phase have become key factors limiting online service experience and computational costs. To reduce redundant computation, the industry commonly caches intermediate states (key-value pairs) from self-attention computations during inference. When identical or similar prefixes appear, the cached key-value pairs are reused, thereby shortening the first-token latency (TTFT) and improving throughput. Such key-value caches may reside in GPU / CPU memory or are often implemented in general key-value systems (such as distributed memory / persistent key-value pairs) for cross-process and cross-node reuse.

[0003] Currently, KV cache management solutions have the following problems: 1. Simple replacement strategy has limited effectiveness: The cache cleanup strategy based solely on Least Recently Used (LRU) and / or Least Frequently Used (LFU) is insufficient to account for the differences in recalculation costs of KV blocks, resulting in entries with "high recalculation costs" being prematurely evicted.

[0004] 2. Unstable performance and uncontrollable capacity: In long context and high concurrency scenarios, the KV cache hit rate fluctuates, and the TTFT (Time to First Token) and P95 / P99 tail latency jitter is obvious; the usage of external KV (such as memory database) continues to increase, and after reaching the peak, write failures or a large number of jittery evictions occur.

[0005] 3. Synchronous cleanup of blocked main paths: The elimination / recycling process is executed synchronously with the inference thread, causing request jitter and reduced throughput.

[0006] 4. Insufficient distributed / cross-node collaboration: Each node makes decisions in isolation, lacking a global or fragmented heat map, leading to hotspot mismatch and ineffective migration.

[0007] 5. Weak multi-tenant isolation: When cache keys are not bound to tenant context or when content is deduplicated / shared across different tenants, cross-tenant false recall and information leakage may occur; if the hot summary exchange is not desensitized and encrypted, there is a risk of exposing usage patterns. Summary of the Invention

[0008] The present invention provides a KV cache management method, apparatus, electronic device and storage medium to solve at least one of the above-mentioned technical problems existing in the prior art.

[0009] This invention provides a key-value (KV) cache management method, comprising the following steps: In response to a large model inference request, retrieve the key-value (KV) blocks to be managed; Collect the characteristic attributes of the KV block, which include at least: recalculation cost index, access popularity index and life cycle index; The retention score of the KV block is calculated based on the aforementioned feature attributes; wherein, the higher the recalculation cost index, the higher the retention score. Monitor the current resource usage status of the KV cache storage area; When the resource occupancy status meets the eviction condition, based on the retention score, a KV block whose retention score meets the preset eviction rule is selected from the KV cache storage area as the target KV block; Perform hierarchical processing operations on the target KV block.

[0010] According to a KV cache management method provided by the present invention, the recalculation cost index is determined in at least one of the following ways: Determined based on the time required to recalculate the KV block; The required computational resources for recalculating the KV block are determined based on the following: The time required to recalculate the KV block and the required computing resources are determined by a weighted summation.

[0011] According to a KV cache management method provided by the present invention, the KV cache storage area includes a hot layer and a cold layer; performing hierarchical processing operations on the target KV block includes: When the retention score of a KV block located in the hot layer is less than the first score threshold, the KV block is migrated to the cold layer as a target KV block and the target KV block is compressed and stored. When a KV block located in the cold layer is accessed and its updated retention score is greater than the second score threshold, the KV block is migrated from the cold layer to the hot layer and decompressed for storage. The first score threshold is less than or equal to the second score threshold. When space needs to be released, N KV blocks with the lowest retention scores are selected from the cold layer for release, where N is greater than 0.

[0012] According to a KV cache management method provided by the present invention, the first scoring threshold is less than the second scoring threshold; The step of migrating the KV block as the target KV block to the cold layer and compressing and storing the target KV block includes: Determine whether the KV block located in the hot layer storage area has a retention score that is less than the first score threshold for multiple consecutive calculations within the preset minimum residence time; if so, then perform the migration and compression storage operation. The process of migrating the KV block from the cold layer to the hot layer and decompressing and storing it includes: Determine whether the KV block located in the cold storage area has a retention score that is greater than the second score threshold for multiple consecutive calculations within the preset minimum residence time; if so, then perform the migration and decompression storage operation.

[0013] According to a KV cache management method provided by the present invention, hierarchical processing operations are performed on the target KV block, including: Generate fingerprint data for the target key-value block; The fingerprint data is stored in an auxiliary cache area that is different from the KV cache storage area; When a KV block in the auxiliary cache is accessed, a preset admission rule is used to determine whether the accessed KV block in the auxiliary cache should be added to the KV cache storage area.

[0014] According to a KV cache management method provided by the present invention, the step of collecting the feature attributes of the KV block further includes: obtaining the identity identifier of the tenant to which the current inference request belongs; The method further includes: generating an isolated namespace or encryption key based on the tenant's identity; binding and storing the KV block with the isolated namespace or encryption key; and verifying whether the identity of the querier matches the isolated namespace or encryption key bound to the KV block when processing a cache query request.

[0015] According to a KV cache management method provided by the present invention, the retention score of the KV block is calculated based on the feature attributes, including: calculating the retention score of the KV block by weighting the feature attributes and the corresponding attribute weights; The method further includes: Obtain the performance metrics values ​​of the last batch of inference requests collected within a certain period of time; Substitute each performance index value into the objective function pre-constructed based on each performance index. The objective function is constrained by the following conditions: each weight is greater than or equal to 0, and the sum of each weight is equal to 1. The objective function is used to characterize the overall inference efficiency and / or cache hit rate of the caching system. Calculate the gradient of the objective function to solve the objective function and obtain the weights of each attribute after the gradient of the objective function changes; By performing simplex projection or nonnegative projection on the attribute weights after the gradient change of the objective function, the attribute weights when the objective function reaches its minimum value are obtained.

[0016] The present invention also provides a KV cache management device, comprising the following modules: The KV block acquisition module is used to retrieve key-value pair KV blocks to be managed in response to large model inference requests. The feature attribute acquisition module is used to collect the feature attributes of the KV block, and the feature attributes include at least: recalculation cost index, access popularity index and life cycle index. The retention score calculation module is used to calculate the retention score of the KV block based on the feature attributes; wherein, the higher the recalculation cost index, the higher the retention score; The KV cache status monitoring module is used to monitor the resource usage status of the current KV cache storage area; The target KV block selection module is used to select KV blocks whose retention scores meet the preset elimination rules from the KV cache storage area as target KV blocks when the resource occupancy status meets the eviction conditions, based on the retention score. The hierarchical processing module is used to perform hierarchical processing operations on the target KV block.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the KV cache management method as described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the KV cache management method as described above.

[0019] The KV cache management method, apparatus, electronic device, and storage medium provided by this invention calculate the retention score of KV blocks by using recalculation cost indicators, access popularity indicators, and lifecycle indicators. In calculating the retention score of KV blocks, in addition to access popularity indicators and lifecycle indicators, the recalculation cost indicator is also considered. The higher the recalculation cost indicator, the higher the retention score, and the more likely the corresponding KV block should be retained in the KV cache storage area. This avoids KV blocks with high recalculation cost indicators being prematurely evicted from the KV cache storage area, improves the access hit rate of KV blocks with high recalculation cost indicators, and thus improves the model inference speed. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts of the KV cache management method provided by the present invention.

[0022] Figure 2 This is a schematic diagram illustrating the principle of hierarchical management based on KV block retention scoring in the KV cache management method provided by this invention.

[0023] Figure 3 This is the second flowchart of the KV cache management method provided by the present invention.

[0024] Figure 4 This is the third flowchart of the KV cache management method provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the KV cache management device provided by the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The KV cache management method of this invention, as described in the embodiments, is as follows: Figure 1 As shown, the procedure includes steps S110 to S160.

[0029] Step S110: In response to the large model inference request, retrieve the key-value (KV) blocks to be managed. During the large model inference process, the Key matrix (K) and Value matrix (V) generated from previous tokens, i.e., the KV blocks, are cached in the KV cache storage area, so that the next token can be quickly inferred based on the cached KV blocks. After responding to the large model inference request, the KV blocks to be managed can be retrieved from the KV cache storage area.

[0030] Step S120: Collect the characteristic attributes of the KV block, which include at least: recalculation cost index, access popularity index and life cycle index.

[0031] Understandably, during the inference process, the key-value (KV) blocks generated by the inference executor have the following attribute information: recalculation cost metric, access frequency metric, and lifecycle metric. The access frequency metric represents the frequency of access to the KV block; the higher the access frequency, the higher the access frequency metric, indicating the KV block is more popular and should be kept in the KV cache. The lifecycle metric represents the duration of time since the KV block was last accessed; the shorter the duration of time since access, the higher the lifecycle metric, and the more likely it is to be kept in the KV cache. The recalculation cost metric represents the time cost and / or resource overhead cost of recalculating the KV block during inference. For KV blocks with a high recalculation cost metric, recalculating the KV block would be costly if it were not in the KV cache, affecting inference speed; therefore, KV blocks with a high recalculation cost metric should be kept in the KV cache.

[0032] Step S130: Calculate the retention score of the KV block based on the feature attributes; wherein, the higher the recalculation cost index, the higher the retention score.

[0033] To prevent high-cost key-value (KV) blocks from being prematurely evicted from the KV cache and to increase the hit rate of accessing these blocks, thereby improving inference speed and efficiency, this step considers not only access frequency and lifecycle metrics as scoring indicators but also introduces the recalculation cost metric to score KV blocks, ultimately resulting in a retention score. With other metrics constant, a higher recalculation cost metric leads to a higher retention score, indicating that the corresponding KV block should be retained in the KV cache. This prevents premature evicting of high-cost KV blocks from the cache, increases the hit rate of accessing these blocks, and ultimately improves model inference speed.

[0034] Step S140: Monitor the current resource occupancy status of the KV cache storage area. This resource occupancy status represents the current state of cache resources being occupied in the KV cache storage area, i.e., how much cache resources are occupied. The more cache resources are occupied, the larger the storage space occupied in the KV cache storage area and the less storage space is left.

[0035] Step S150: When the resource occupancy status meets the eviction conditions, based on the retention score, select KV blocks whose retention scores meet the preset eviction rules from the KV cache storage area as target KV blocks.

[0036] For example, eviction criteria could be that the KV cache storage area is fully occupied or the occupancy rate reaches 60%~85%, such as 80%. Another eviction criterion is that the large model system load is below a preset load threshold. The large model system load can characterize the current workload level, which can be reflected by the smoothed system resource (CPU, GPU, and memory, etc.) utilization rate. The lower the load, the less busy the current system is, allowing the KV cache storage area to be released without affecting the operation of the system. The load threshold can be set to a resource utilization rate of 40%~70%, such as 60%, meaning that when the system resource utilization rate is less than 60%, the space of the KV cache storage area can be released.

[0037] The preset elimination rule can be that the retention score is lower than the preset score threshold, and the key-value blocks below the score threshold are used as target key-value blocks, or the multiple key-value blocks with the lowest retention scores are used as target key-value blocks.

[0038] Step S160: Perform hierarchical processing operations on the target KV block. For example, the hierarchical processing operations may include hierarchical processing strategies such as deleting the target KV block from the KV cache storage area, moving the target KV block into CPU memory, or compressing storage.

[0039] The KV cache management method in this embodiment calculates the retention score of KV blocks using recalculation cost, access popularity, and lifecycle metrics. In addition to access popularity and lifecycle metrics, the recalculation cost metric is also considered when calculating the retention score of KV blocks. The higher the recalculation cost metric, the higher the retention score, and the more likely the corresponding KV block should be retained in the KV cache storage area. This avoids prematurely evicting KV blocks with high recalculation cost metrics from the KV cache storage area, improves the access hit rate of KV blocks with high recalculation cost metrics, and thus improves the model inference speed.

[0040] In some embodiments, since the recalculation cost metric represents the time cost and / or resource overhead cost of recalculating KV blocks during inference, the recalculation cost metric can be determined in at least one of the following ways: Method 1: Determine the cost based on the time required to recalculate the KV block. That is, the time required to calculate the KV block can be used as the recalculation cost indicator. The longer the time, the greater the recalculation cost indicator.

[0041] Method 2: Determine the cost based on the computing resources required to recalculate the KV block. That is, the number of computing resources required to calculate the KV block, such as the number of GPU cores, can be used as the recalculation cost indicator. The more GPU cores used, the greater the recalculation cost indicator.

[0042] Method 3: Determined based on the weighted sum of the time and computing resources required to recalculate the KV block. That is, the weighted sum of the time and computing resources required to calculate the KV block is used as the recalculation cost indicator. It can be understood that before weighted summation, the time and computing resources required to calculate the KV block can be normalized to remove dimensions, facilitating weighted summation.

[0043] In this embodiment, the recalculation cost index of the KV block can be determined by one of the three methods mentioned above. In particular, the third method is adopted, which comprehensively considers time cost and resource consumption cost, thereby more accurately determining the recalculation cost of the KV block.

[0044] In some embodiments, the KV cache storage area includes a hot layer and a cold layer, such as Figure 2 As shown, the KV cache storage area can be logically divided into a hot layer and a cold layer, with the retention score of KV blocks in the hot layer being higher than that in the cold layer. Based on this, step S160 performs a hierarchical processing operation on the target KV block, including: When the retention score of a key-value block located in the hot layer is lower than the first score threshold, this key-value block is migrated to the cold layer as a target key-value block, and its storage is compressed. If a key-value block in the hot layer has a retention score lower than the first score threshold due to prolonged inactivity, it indicates that the key-value block's popularity has decreased, meaning it is unlikely to be accessed recently. Continuing to store it in the hot layer would consume a large amount of cache space; therefore, it is migrated to the cold layer as a target key-value block, and its storage is compressed. It is understood that compressed storage includes, but is not limited to, data quantization, sparse storage, or lossless encoding compression.

[0045] For example, the KV blocks in the cold layer can be quantized, that is, the target KV block is stored in a low-bit or low-precision form, and then dequantized after a subsequent access hit. For example, the target KV block in the cold layer can be converted from the original high-precision format (e.g., FP16 / BF16) to a low-precision format (e.g., INT8 / INT4) for storage, thereby saving space in the KV cache storage area.

[0046] When a key-value block (KV block) located in the cold layer is accessed and its updated retention score is greater than the second score threshold, the KV block is migrated from the cold layer to the hot layer and decompressed for storage. The first score threshold is less than or equal to the second score threshold. If a KV block in the cold layer is accessed and its retention score is greater than the second score threshold, it indicates that the KV block's popularity has increased, meaning it is highly likely to be accessed soon. Therefore, it is migrated to the hot layer and decompressed for storage, essentially restoring it to a high-precision storage format in advance, so that the KV block can be directly accessed upon access, improving access efficiency.

[0047] When space needs to be released, select the N KV blocks with the lowest retention scores from the cold layer for release, where N is greater than 0. N can be set according to the actual situation to ensure that there is enough cache space after release.

[0048] In this embodiment, by logically setting up hot and cold layers in the KV cache storage area, KV blocks in the cold layer are compressed for storage, saving space in the entire KV cache storage area and enabling the storage of more KV blocks, thereby improving the access hit rate. Meanwhile, KV blocks in the hot layer maintain a high-precision storage format and do not require dequantization when accessed, improving access efficiency. Furthermore, a first and second scoring threshold are set. Each KV block migrates between the cold and hot layers based on its real-time retention score and the first and second scoring thresholds. For example, when the retention score of a KV block in the hot layer drops below the first scoring threshold, it is migrated to the cold layer and compressed for storage, saving cache space and enabling the caching of more KV blocks, maintaining a high access hit rate. Conversely, when the retention score of a target KV block in the cold layer rises above the second scoring threshold, it is migrated to the hot layer and decompressed for storage, allowing KV blocks with higher retention scores to enter the hot layer earlier, improving access efficiency. Thus, through the migration of KV blocks between the cold and hot layers, the entire KV cache storage area consistently maintains a high access hit rate and access efficiency.

[0049] In addition, since the KV cache storage area is logically divided into hot and cold layers, when releasing the cache, only a portion of the target KV blocks in the cold layer are released. Therefore, it is not necessary to perform KV release and cache reclamation on the inference main path or use blocking release, thus achieving asynchronous cache release. This avoids the situation of request jitter and throughput reduction caused by the synchronous execution of KV cache release and inference thread.

[0050] In this embodiment, the logically divided cold and hot layers can also be synchronized to other nodes in the distributed system, thereby solving the problem of isolated decision-making by each node in the distributed system, lack of a global or fragmented heat view, and resulting hot spot mismatch and invalid migration.

[0051] Furthermore, based on the logically defined hot and cold layers, the KV indices of KV blocks are sorted according to their retention scores using a min-heap based on the decrease-key. KV blocks closer to the top of the heap have lower retention scores and are therefore the highest priority candidates for eviction. By sorting the KV indices by retention scores using a min-heap based on the decrease-key, the sorting of KV indices is maintained in O(log n) computational complexity. When cache release is needed, the KV blocks corresponding to the top N KV indices are directly released from the KV cache storage area, thus achieving fast asynchronous cache release.

[0052] In some embodiments, the first scoring threshold is less than the second scoring threshold. Based on this, such as Figure 2 As shown, the step of migrating the KV block as the target KV block to the cold layer and decompressing and storing it includes: Determine whether the retention score of the KV block located in the hot layer storage area is less than the first score threshold in multiple consecutive calculations (e.g., three times) within the preset minimum residence time τ_min (e.g., 10 seconds); if so, perform the migration and compression storage operation.

[0053] The step of migrating the KV block from the cold layer to the hot layer and decompressing and storing it includes: determining whether the KV block located in the cold layer storage area has a retention score that is greater than the second score threshold after multiple consecutive calculations within a preset minimum residence time τ_min; if so, then performing the migration and decompression storage operation.

[0054] In this embodiment, KV blocks in the hot layer storage area will only migrate from the hot layer to the cold layer if their scores are all less than the first scoring threshold multiple times within τ_min. KV blocks in the cold layer storage area will only migrate from the cold layer to the hot layer if their scores are all greater than the second scoring threshold multiple times within τ_min. That is, a logical lag interval is formed by the difference between the first and second scoring thresholds and the minimum residence time. The average retention of KV blocks in this lag interval is between the first and second scoring thresholds. At this time, the KV blocks remain in the original layer and do not migrate, which is used to digest short-term fluctuations and avoid inter-layer migration jitter.

[0055] In some embodiments, such as Figure 3 As shown, step S160, performing hierarchical processing on the target KV block, specifically includes: Step S161: Generate fingerprint data of the target KV block. The fingerprint data can be a low-dimensional projection, a set of quantization parameters (clue vector), a hash value, or the K value of the KV block.

[0056] Step S162: Store the fingerprint data in an auxiliary cache area that is different from the KV cache storage area, such as CPU memory. That is, remove the target KV block from the KV cache storage area and move it to the auxiliary cache area. In particular, after a release operation, store the fingerprint data of the target KV block that has just been released in the auxiliary cache area instead of storing the target KV block directly, thus saving space in the auxiliary cache area.

[0057] Step S163: When a KV block in the auxiliary cache is accessed, determine whether the accessed KV block in the auxiliary cache should be added to the KV cache storage area based on a preset admission rule. Of course, when a KV block in the auxiliary cache is accessed, the KV block corresponding to the fingerprint data needs to be recalculated for inference calculation. It can be understood that when a request hits fingerprint data in the auxiliary cache, it indicates that the KV block corresponding to that fingerprint data has been frequently accessed recently. Although the KV block has been lost and needs to be recalculated, after recalculation, it can be stored in the KV cache storage area (hot layer) according to the admission rule, rather than being discarded again. For example, the admission rule could be to directly store the recalculated KV block in the hot layer, or to store the recalculated KV block in the hot layer if the recalculation cost exceeds a set cost threshold.

[0058] In this embodiment, the fingerprint data of the target KV block to be released can be stored in an auxiliary buffer. For KV blocks accessed in the auxiliary buffer, it can be determined whether they can be migrated to the KV cache storage area through preset admission rules. If the admission rules are met, the recalculated KV block corresponding to the fingerprint data is then moved into the KV cache storage area. The admission rule can be that the block has been accessed multiple times since the last release operation, i.e., there are multiple revisits in a short period of time. This can pull mistakenly eliminated popular KV blocks back into the KV cache storage area, making the access hit rate of the KV cache storage area more stable and the tail latency lower.

[0059] In some embodiments, the step of collecting the feature attributes of the KV block further includes: obtaining the identity identifier of the tenant to which the current inference request belongs.

[0060] The method further includes: generating an isolated namespace or encryption key based on the tenant's identity; binding and storing the KV block with the isolated namespace or encryption key; and when processing a cache query request, verifying whether the queryer's identity matches the isolated namespace or encryption key bound to the KV block, and only executing the corresponding query request for matching KV blocks.

[0061] In this embodiment, the key-value blocks are bound and stored with the isolation namespace or encryption key. Essentially, this means writing the "tenant" into the key space. The same key-value content will get different keys under different tenants, that is, different key-value blocks, thereby achieving strong isolation between tenants. Moreover, when processing cache query requests, the identity of the queryer is verified to see if it matches the isolation namespace or encryption key bound to the key-value block. Only for matching key-value blocks will the corresponding query request be executed, avoiding cross-tenant false recall and information leakage.

[0062] In some embodiments, calculating the retention score of the KV block based on the feature attributes includes: calculating the retention score of the KV block by weighting the feature attributes and their corresponding attribute weights.

[0063] Specifically, before calculating the retention score, the feature attributes need to be preprocessed to eliminate the dimensions of each feature attribute. For the access popularity index and the recalculation cost index, normalization can be used to eliminate the dimensions. The normalization formula is as follows: (1).

[0064] in, freq This indicates the popularity metric for website visits. cost This indicates a recalculation of cost indicators. μ x express x The mean, σ x express x variance ε This represents the numerical stability coefficient, avoiding a denominator of 0, and taking the minimum value (less than 10). -6 (This can be done by adding the denominator, which can be ignored when the denominator is not zero.)

[0065] For lifecycle metrics t Define the time decay factor recency = e -λΔt To eliminate its dimensions, where, λ Indicates the time decay coefficient. λ The value is greater than 0.

[0066] Retention score score The calculation formula is as follows: (2).

[0067] in, This indicates the weight of the access popularity metric. Indicates the weight of the time decay factor. This indicates the weight of the recalculated cost indicator. and These represent the normalized values ​​of the visit popularity index and the recalculation cost index, respectively. The values ​​of each weight are all greater than 0, and the sum is 1.

[0068] Furthermore, to avoid KV block size ( size Large and infrequently used KV blocks (large and cold) occupy the KV cache storage area for a long time. Their characteristic attributes may also include: KV block size index, which can be normalized according to the above formula (1). Considering the KV block size index, retention score... score The calculation formula is as follows: (3).

[0069] in, This represents the normalized value of the KV block size index. This represents the weights of the KV block size index. All weights are greater than 0 and their sum is 1.

[0070] Since the retention score calculation takes into account not only the lifecycle metric but also the key-value block size metric, the larger the key-value block size metric, the lower the retention score. This can prevent large and infrequently used key values ​​(large and cold) from occupying the key-value cache storage area for a long time.

[0071] For example, the statistics for each KV block in the KV cache storage area over a period of time are as follows: μ freq =10, σ freq =5;μ size =64KB, σ size =32KB; μ cost =2ms, σ cost =1ms.

[0072] Weights: w freq =0.45, w recency =0.25, w size =0.15, w cost =0.15; Time decay coefficient λ=0.01.

[0073] For a certain KV block: eA, the corresponding freq=30, size=64KB, cost=5ms, Δt=2s.

[0074] recency=e -λΔt ≈0.980, after normalization , , .

[0075] Therefore, score≈0.45×4 + 0.25×0.98 - 0.15×0 + 0.15×3 = 1.80 + 0.245 -0 + 0.45 = 2.495.

[0076] For another KV block: eB, the corresponding freq=8, size=160KB, cost=1ms, Δt=10s.

[0077] recency=e -λΔt ≈0.905, after normalization , , .

[0078] Therefore, score≈0.45×(-0.4) + 0.25×0.905 - 0.15×3 + 0.15×(-1) = -0.18+ 0.226 - 0.45 - 0.15 = -0.194.

[0079] For example, the boundary value between the cold layer and the hot layer can be set to 1. eA can be preferentially retained in the hot layer, while eB enters the cold layer as the target KV block.

[0080] In the above retention score formula, each weight and time decay coefficient can be set according to the actual situation. Preferably, each weight and time decay coefficient can be adjusted in real time according to the performance index of the KV cache storage area, so that the overall performance of the inference request of the KV cache storage area can also reach the optimal level after the retention score is released on the basis of the preset elimination rules.

[0081] Therefore, such as Figure 4 As shown, the KV cache management method in this embodiment further includes: Step S410: Obtain the performance metrics values ​​of the last batch of inference requests collected within a certain period. Performance metrics may include at least one of the following: KV cache hit rate, TTFT, P95 / P99, cleanup cycle jitter index, admission write amplification metric, and ghost cache back-to-origin bandwidth.

[0082] Step S420: Substitute each performance metric value into a pre-constructed objective function. The objective function is constrained by the following conditions: each weight is greater than or equal to 0, and the sum of all weights equals 1. The objective function is used to characterize the overall inference efficiency and / or cache hit rate of the caching system. For example, the hit rate of a KV cache. hit and characterization delay latency The objective function for constructing the TTFT and P95 / P99 indices is as follows: (4).

[0083] in latency You can choose one of the three: TTFT, P95 / P99, or you can obtain the average or weighted sum of at least two of the three.

[0084] Step S430: Calculate the gradient of the objective function The objective function is solved to obtain the attribute weights after the gradient change of the objective function. Due to the hit rate of the KV cache... hit Indices such as these are usually discrete or non-differentiable; therefore, calculating the gradient of the objective function is crucial. This can be achieved using approximate gradient estimation algorithms (such as the finite difference method), policy gradient methods, or gradient-free optimization algorithms (such as Bayesian optimization and particle swarm optimization). In this step, gradient descent is used to solve the objective function and obtain its local minimum. However, in practical applications, other optimization algorithms that seek the extrema of the objective function can be used instead.

[0085] Step S440: Perform simplex projection or nonnegative projection on the attribute weights after the gradient change of the objective function. This yields the attribute weights when the objective function reaches its minimum value. Among them, η This represents the learning rate or step size used to calculate the gradient of the objective function. For example, in the objective function of formula (4) above, the smaller the value of the objective function, the higher the hit rate of the KV cache for the last batch of inference requests and the minimum latency.

[0086] In this embodiment, the objective function can be solved by projecting subgradients or using a multi-armed gambling machine. Based on the objective function, the weights and time decay coefficients of the feature attributes are updated. Thus, the retention score is calculated based on each weight and time decay coefficient. After releasing the KV cache storage area based on the retention score meeting the preset elimination rules, the overall performance of inference requests to the KV cache storage area can also reach the optimal level.

[0087] The advantages of the KV cache management method of the present invention are illustrated below with specific experiments in the following experimental scenarios: Environment: 8×A100 80G; vLLM (7B instruction model); 16k context; long / short requests = 7 / 3; RPS boosted to 400; external Redis (maxmemory+UNLINK); NVMe Gen4.

[0088] Baseline: LRU and LFU; Experiments have shown that the KV cache management method of the present invention improves the KV cache hit rate, TTFT, P95 / P99, cleanup cycle jitter index, admission write amplification index, and ghost cache back-to-source bandwidth index. The specific experimental results are shown in Table 1 below.

[0089] Table 1. Comparison of management results between the KV cache management method of the present invention and the traditional management method.

[0090] As can be seen from Table 1 above, all performance indicators in this invention are superior to those of traditional LRU and LFU cache management methods.

[0091] An ablation experiment (with a certain function turned off) was conducted under the above experimental conditions, and the experimental results are shown in Table 2 below.

[0092] Table 2 Ablation Experiment Results

[0093] As can be seen from Table 2 above, the overall performance of the KV cache management method of the present invention is optimal when all functions are complete.

[0094] The method of the present invention was subjected to multi-tenancy and compliance testing under the above experimental conditions, and the test results are shown in Table 3 below.

[0095] As can be seen from Table 3 above, in this invention, when an isolated namespace or encryption key is generated based on the tenant's identity identifier, and the KV block is bound and stored with the isolated namespace or encryption key, there will be no problem of cross-tenant false recall or information leakage.

[0096] The KV cache management device provided by the present invention is described below. The KV cache management device described below can be referred to in correspondence with the KV cache management method described above.

[0097] The KV cache management device of this invention embodiment, such as Figure 5 As shown, it includes the following modules: KV block acquisition module 510 is used to acquire key-value pair KV blocks to be managed in response to large model inference requests.

[0098] The feature attribute acquisition module 520 is used to acquire the feature attributes of the KV block, and the feature attributes include at least: recalculation cost index, access popularity index and life cycle index.

[0099] The retention score calculation module 530 is used to calculate the retention score of the KV block based on the feature attributes; wherein, the higher the recalculation cost index, the higher the retention score.

[0100] The KV cache status monitoring module 540 is used to monitor the resource usage status of the current KV cache storage area.

[0101] The target KV block selection module 550 is used to select KV blocks whose retention scores meet the preset eviction rules from the KV cache storage area as target KV blocks when the resource occupancy status meets the eviction conditions, based on the retention score.

[0102] The hierarchical processing module 560 is used to perform hierarchical processing operations on the target KV block.

[0103] The KV cache management device in this embodiment calculates the retention score of KV blocks using recalculation cost, access popularity, and lifecycle metrics. In calculating the retention score of KV blocks, in addition to access popularity and lifecycle metrics, the recalculation cost metric is also considered. The higher the recalculation cost metric, the higher the retention score, and the more likely the corresponding KV block should be retained in the KV cache storage area. This avoids KV blocks with high recalculation cost metrics being prematurely evicted from the KV cache storage area, improves the access hit rate of KV blocks with high recalculation cost metrics, and thus improves the model inference speed.

[0104] In some embodiments, the recalculation cost metric is determined in at least one of the following ways: The time required to recalculate the KV block is determined based on this.

[0105] The required computing resources for recalculating the KV block are determined based on this.

[0106] The time required to recalculate the KV block and the required computing resources are determined by a weighted summation.

[0107] In some embodiments, the KV cache storage area includes a hot layer and a cold layer; the hierarchical processing module 560 specifically includes: The first migration module is used to migrate the KV block located in the hot layer to the cold layer as the target KV block when the retention score of the KV block is less than the first score threshold, and to compress and store the target KV block.

[0108] The second migration module is used to migrate the KV block from the cold layer to the hot layer and decompress and store it when the KV block located in the cold layer is accessed and its updated retention score is greater than the second score threshold. The first score threshold is less than or equal to the second score threshold.

[0109] The space release module is used to select N KV blocks with the lowest retention scores from the cold layer for release when space needs to be released, where N is greater than 0.

[0110] In some embodiments, the first scoring threshold is less than the second scoring threshold.

[0111] The first migration module is specifically used to determine whether the KV block located in the hot layer storage area has a retention score that is less than the first score threshold after multiple consecutive calculations within the preset minimum residence time; if so, the migration and compression storage operation is performed.

[0112] The second migration module is specifically used to determine whether the KV block located in the cold storage area has a retention score that is greater than the second score threshold after multiple consecutive calculations within the preset minimum residence time; if so, the migration and decompression storage operation is performed.

[0113] In some embodiments, the hierarchical processing module 560 specifically includes: The fingerprint compression module is used to generate fingerprint data for the target KV block.

[0114] An auxiliary caching module is used to store the fingerprint data in an auxiliary caching area that is different from the KV caching storage area.

[0115] The admission determination module is used to determine, based on a preset admission rule, whether the KV block accessed in the auxiliary cache should be added to the KV cache storage area when the KV block in the auxiliary cache is accessed.

[0116] In some embodiments, the feature attribute acquisition module 520 is further configured to obtain the identity identifier of the tenant to which the current inference request belongs.

[0117] The KV cache management device further includes: a tenant information management module, used to generate an isolated namespace or encryption key based on the tenant's identity identifier; bind and store the KV block with the isolated namespace or encryption key; and when processing a cache query request, verify whether the queryer's identity identifier matches the isolated namespace or encryption key bound to the KV block.

[0118] In some embodiments, the retention score calculation module 530 is specifically used to calculate the retention score of the KV block based on the feature attributes and the corresponding attribute weights.

[0119] The KV cache management device also includes: The performance metrics acquisition module is used to obtain the performance metric values ​​of the last batch of inference requests collected within a certain period of time.

[0120] The objective function substitution module is used to substitute each performance index value into the objective function pre-constructed based on each performance index. The constraint condition of the objective function is that each weight is greater than or equal to 0 and the sum of each weight is equal to 1. The objective function is used to characterize the overall inference efficiency and / or cache hit rate of the caching system.

[0121] The objective function solution module is used to calculate the gradient of the objective function in order to solve the objective function and obtain the weights of each attribute after the gradient of the objective function changes.

[0122] The projection module is used to perform simplex projection or nonnegative projection on the attribute weights after the gradient change of the objective function, so as to obtain the attribute weights when the objective function is minimized.

[0123] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a KV cache management method, which includes: In response to a large model inference request, retrieve the key-value (KV) blocks to be managed.

[0124] Collect the characteristic attributes of the KV block, which include at least: recalculation cost index, access popularity index and life cycle index.

[0125] The retention score of the KV block is calculated based on the aforementioned feature attributes; wherein, the higher the recalculation cost index, the higher the retention score.

[0126] Monitor the current resource usage status of the KV cache storage area.

[0127] When the resource occupancy status meets the eviction conditions, based on the retention score, a KV block whose retention score meets the preset eviction rule is selected from the KV cache storage area as the target KV block.

[0128] Perform hierarchical processing operations on the target KV block.

[0129] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the KV cache management method provided by the above methods, the method comprising: In response to a large model inference request, retrieve the key-value (KV) blocks to be managed.

[0131] Collect the characteristic attributes of the KV block, which include at least: recalculation cost index, access popularity index and life cycle index.

[0132] The retention score of the KV block is calculated based on the aforementioned feature attributes; wherein, the higher the recalculation cost index, the higher the retention score.

[0133] Monitor the current resource usage status of the KV cache storage area.

[0134] When the resource occupancy status meets the eviction conditions, based on the retention score, a KV block whose retention score meets the preset eviction rule is selected from the KV cache storage area as the target KV block.

[0135] Perform hierarchical processing operations on the target KV block.

[0136] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the KV cache management method provided by the methods described above, the method comprising: In response to a large model inference request, retrieve the key-value (KV) blocks to be managed.

[0137] Collect the characteristic attributes of the KV block, which include at least: recalculation cost index, access popularity index and life cycle index.

[0138] The retention score of the KV block is calculated based on the aforementioned feature attributes; wherein, the higher the recalculation cost index, the higher the retention score.

[0139] Monitor the current resource usage status of the KV cache storage area.

[0140] When the resource occupancy status meets the eviction conditions, based on the retention score, a KV block whose retention score meets the preset eviction rule is selected from the KV cache storage area as the target KV block.

[0141] Perform hierarchical processing operations on the target KV block.

[0142] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0144] All actions involving the acquisition of signal information or data in this invention are carried out in compliance with the relevant data protection laws and policies of the country where the device is located, and with the authorization granted by the owner of the device.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A key-value cache management method, characterized in that, include: In response to a large model inference request, retrieve the key-value (KV) blocks to be managed; Collect the characteristic attributes of the KV block, which include at least: recalculation cost index, access popularity index and life cycle index; The retention score of the KV block is calculated based on the aforementioned feature attributes; wherein, the higher the recalculation cost index, the higher the retention score. Monitor the current resource usage status of the KV cache storage area; When the resource occupancy status meets the eviction condition, based on the retention score, a KV block whose retention score meets the preset eviction rule is selected from the KV cache storage area as the target KV block; Perform hierarchical processing operations on the target KV block.

2. The KV cache management method according to claim 1, characterized in that, The recalculated cost index is determined in at least one of the following ways: Determined based on the time required to recalculate the KV block; The required computational resources for recalculating the KV block are determined based on the following: The time required to recalculate the KV block and the required computing resources are determined by a weighted summation.

3. The KV cache management method according to claim 1, characterized in that, The KV cache storage area includes a hot layer and a cold layer; hierarchical processing operations are performed on the target KV block, including: When the retention score of a KV block located in the hot layer is less than the first score threshold, the KV block is migrated to the cold layer as a target KV block and the target KV block is compressed and stored. When a KV block located in the cold layer is accessed and its updated retention score is greater than the second score threshold, the KV block is migrated from the cold layer to the hot layer and decompressed for storage. The first score threshold is less than or equal to the second score threshold. When space needs to be released, N KV blocks with the lowest retention scores are selected from the cold layer for release, where N is greater than 0.

4. The KV cache management method according to claim 3, characterized in that, The first scoring threshold is less than the second scoring threshold; The step of migrating the KV block as the target KV block to the cold layer and compressing and storing the target KV block includes: Determine whether the KV block located in the hot layer storage area has a retention score that is less than the first score threshold for multiple consecutive calculations within the preset minimum residence time; if so, then perform the migration and compression storage operation. The process of migrating the KV block from the cold layer to the hot layer and decompressing and storing it includes: Determine whether the KV block located in the cold storage area has a retention score that is greater than the second score threshold for multiple consecutive calculations within the preset minimum residence time; if so, then perform the migration and decompression storage operation.

5. The KV cache management method according to claim 1, characterized in that, Perform hierarchical processing operations on the target KV block, including: Generate fingerprint data for the target key-value block; The fingerprint data is stored in an auxiliary cache area that is different from the KV cache storage area; When a KV block in the auxiliary cache is accessed, a preset admission rule is used to determine whether the accessed KV block in the auxiliary cache should be added to the KV cache storage area.

6. The KV cache management method according to claim 1, characterized in that, The process of collecting the feature attributes of the KV block also includes: obtaining the identity identifier of the tenant to which the current inference request belongs; The method further includes: generating an isolated namespace or encryption key based on the tenant's identity; binding and storing the KV block with the isolated namespace or encryption key; and verifying whether the identity of the querier matches the isolated namespace or encryption key bound to the KV block when processing a cache query request.

7. The KV cache management method according to any one of claims 1 to 6, characterized in that, Calculating the retention score of the KV block based on the feature attributes includes: calculating the retention score of the KV block based on the feature attributes and the corresponding attribute weights. The method further includes: Obtain the performance metrics values ​​of the last batch of inference requests collected within a certain period of time; Substitute each performance index value into the objective function pre-constructed based on each performance index. The objective function is constrained by the following conditions: each weight is greater than or equal to 0, and the sum of each weight is equal to 1. The objective function is used to characterize the overall inference efficiency and / or cache hit rate of the caching system. Calculate the gradient of the objective function to solve the objective function and obtain the weights of each attribute after the gradient of the objective function changes; By performing simplex projection or nonnegative projection on the attribute weights after the gradient change of the objective function, the attribute weights when the objective function reaches its minimum value are obtained.

8. A KV cache management device, characterized in that, include: The KV block acquisition module is used to retrieve key-value pair KV blocks to be managed in response to large model inference requests. The feature attribute acquisition module is used to collect the feature attributes of the KV block, and the feature attributes include at least: recalculation cost index, access popularity index and life cycle index. The retention score calculation module is used to calculate the retention score of the KV block based on the feature attributes; wherein, the higher the recalculation cost index, the higher the retention score; The KV cache status monitoring module is used to monitor the resource usage status of the current KV cache storage area; The target KV block selection module is used to select KV blocks whose retention scores meet the preset elimination rules from the KV cache storage area as target KV blocks when the resource occupancy status meets the eviction conditions, based on the retention score. The hierarchical processing module is used to perform hierarchical processing operations on the target KV block.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the KV cache management method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the KV cache management method as described in any one of claims 1 to 7.

Images