Key-value cache offloading method, device, equipment, storage medium and program product

Abstract

The present disclosure provides a key-value cache offloading method, a key-value cache offloading device, a computer device, a computer readable storage medium and a computer program product; the method comprises: splitting a key-value cache block into a key cache and a value cache; determining a first offloading strategy of the key cache based on the sensitivity of a sample key cache to quantization error; the first offloading strategy comprises a quantization mode and / or a compression mode of the key cache; determining a second offloading strategy of the value cache based on the sensitivity of a sample value cache to quantization error; the second offloading strategy comprises a quantization mode and / or a compression mode of the value cache; using the first offloading strategy to offload the key cache from a first storage space to a second storage space; using the second offloading strategy to offload the value cache from the first storage space to a third storage space.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of computer vision technology, and in particular to a key-value cache unloading method, a key-value cache unloading device, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In large language model inference, the key-value cache (KV cache) grows linearly with the context length, becoming the primary bottleneck in long-context scenarios. In LLM inference, the KV cache stores the key / value tensors of each attention layer for reusing historical computations and accelerating generation. The longer the context and the higher the concurrency, the more GPU memory the KV cache consumes. When GPU memory is insufficient, the system offloads cold / infrequently used key-value cache blocks (KVblocks) from GPU memory to lower-cost, higher-capacity lower-level storage to free up space for active requests. However, offloading KVblocks to lower-level storage presents the following problems: if high-precision (e.g., FP16) offloading is used, there is a serious waste of storage space and transmission bandwidth; if low-precision (e.g., INT4) offloading is used, precision is severely compromised, leading to a degradation in attention quality after long context callbacks and resulting in output errors. Summary of the Invention

[0003] In view of the above, the present disclosure provides at least one key-value cache unloading method, key-value cache unloading device, computer device, computer-readable storage medium, and computer program product.

[0004] The technical solution of this disclosure embodiment is implemented as follows: On one hand, embodiments of this disclosure provide a key-value cache unloading method, including: Split the key-value cache block into a key cache and a value cache; Based on the sensitivity of the sample key cache to quantization error, a first offloading strategy for the key cache is determined; the first offloading strategy includes the quantization method and / or compression method of the key cache. Based on the sensitivity of the sample value cache to quantization error, a second offloading strategy for the value cache is determined; the second offloading strategy includes the quantization method and / or compression method of the value cache. The first unloading strategy is adopted to unload the key cache from the first storage space to the second storage space; The second unloading strategy is adopted to unload the value cache from the first storage space to the third storage space.

[0005] On the other hand, embodiments of this disclosure provide a key-value cache unloading device, comprising: The split module is configured to split the key-value cache block into a key cache and a value cache. The processing module is configured to determine a first offloading strategy for the key cache based on the sensitivity of the sample key cache to quantization error; the first offloading strategy includes the quantization method and / or compression method of the key cache. The processing module is also configured to determine a second offloading strategy for the value cache based on the sensitivity of the sample value cache to quantization error; the second offloading strategy includes the quantization method and / or compression method of the value cache. The unloading module is configured to use the first unloading strategy to unload the key cache from the first storage space to the second storage space; The unloading module is also configured to use a second unloading strategy to unload the value cache from the first storage space to the third storage space.

[0006] In another aspect, embodiments of this disclosure provide a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0007] In another aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0008] In another aspect, embodiments of this disclosure provide a computer program including computer-readable code, which, when executed in a computer device, causes a processor in the computer device to perform some or all of the steps in the above-described method.

[0009] In another aspect, embodiments of this disclosure provide a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer device, it implements some or all of the steps in the above-described method.

[0010] In this embodiment, the key-value cache block is split into a key cache and a value cache to achieve physical storage separation between the key cache and the value cache. This allows the key cache and value cache to occupy different physical address spaces, have independent metadata records, and be independently movable, quantized, compressed, and even stored in different storage layers. A first offloading strategy, determined by the sensitivity of the sample key cache to quantization errors, is used to offload the key cache, and a second offloading strategy, determined by the sensitivity of the sample value cache to quantization errors, is used to offload the value cache. By utilizing the asymmetric sensitivity of key / value pairs and employing different offloading strategies for the key cache and value cache, and even storing them in different storage layers, an optimal balance can be achieved between storage efficiency and inference accuracy.

[0011] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0013] Figure 1 A schematic diagram of the implementation process of a key-value cache unloading method provided in this embodiment of the disclosure. Figure 1 ; Figure 2 This is a schematic diagram of the composition structure of a key-value cache unloading system provided in an embodiment of this disclosure; Figure 3 A schematic diagram illustrating the process of determining the residual bit width in a key-value cache unloading method provided in this embodiment of the disclosure; Figure 4 A schematic diagram illustrating the process of determining the compression ratio prediction table in a key-value cache unloading method provided in this embodiment of the disclosure; Figure 5 A schematic diagram of the implementation flow of the callback scheme in a key-value cache unloading method provided in this embodiment of the disclosure. Figure 1 ; Figure 6 A schematic diagram illustrating the implementation process of an adaptive adjustment scheme between precision feedback and unloading strategy in a key-value cache unloading method provided in this embodiment of the disclosure; Figure 7 This is a schematic diagram illustrating the process of determining the execution location of each operation in a key-value cache unloading method provided in this embodiment of the disclosure; Figure 8 A schematic diagram illustrating the process of determining the quantization path in a key-value cache unloading method provided in this embodiment of the disclosure; Figure 9 A schematic diagram illustrating the process of determining the embedding point of a key-value cache unloading method provided in this embodiment of the disclosure; Figure 10 A schematic diagram of the implementation process of a key-value cache unloading method provided in this embodiment of the disclosure. Figure 2 ; Figure 11 A schematic diagram of the implementation flow of the callback scheme in a key-value cache unloading method provided in this embodiment of the disclosure. Figure 2 ; Figure 12 This is a schematic diagram of the composition structure of a key-value cache unloading device provided in an embodiment of the present disclosure; Figure 13 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this disclosure. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0015] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0016] If similar descriptions such as "first / second" appear in the application documents, the following explanation shall be added: the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this disclosure.

[0018] To better understand the key-value cache unloading method provided in the embodiments of this disclosure, the solutions in related technologies will be described below.

[0019] Taking Llama-3.1-70B as an example, using Grouped-Query Attention (GQA) and Half-Precision Floating Point (FP16) precision, processing 1 million token context requires approximately 305 gigabytes (GB) of KV cache, far exceeding the capacity of the highest bandwidth memory (HBM) of current GPUs. With the deployment of ultra-large models and the widespread adoption of scenarios such as long-dialogue agent scenarios and retrieval-augmented generation of long documents, the demand for KV cache capacity is growing at a rate far exceeding the expansion speed of HBM. Therefore, a precision-aware, hierarchical offloading and recovery scheduling scheme for KV cache is urgently needed.

[0020] Therefore, this disclosure provides a key-value cache unloading method applied to computer devices. The computer device refers to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device). Figure 1 As shown, the method includes the following steps 101 to 105: Step 101: Split the key-value cache block into a key cache and a value cache.

[0021] A key-value (KV) block can be split into a key cache (K_cache) and a value cache (V_cache). The key cache can also be called the key part (K-part), and the value cache can also be called the value part (V-part). Splitting a key-value block into key and value caches is not only a logical split (like the key-value tuples in HuggingFace Transformers), but also a physical storage split. That is, the K-part and V-part occupy different physical address spaces in memory / solid-state drives, have independent metadata records, and can be moved, quantized, compressed, and even stored in different storage layers independently.

[0022] In some implementations, in response to an unload request for a key-value cache block, the key-value cache block is split into a key cache and a value cache.

[0023] In some implementations, when GPU memory is insufficient, the key-value cache block that least affects inference speed is selected as the key-value cache block to be unloaded based on at least one of access frequency, usage priority, and sequence lifetime. Alternatively, the least frequently used key-value cache block can also be selected as the key-value cache block to be unloaded.

[0024] In some implementations, the key-value cache block that least affects inference speed can be selected as the key-value cache block to be unloaded, according to the least recently used strategy.

[0025] In some implementations, the key-value cache block with the fewest accesses can be prioritized for uninstallation. Alternatively, the earliest key-value cache block in the long text that is furthest from the current generation position can be prioritized for uninstallation.

[0026] Step 102: Based on the sensitivity of the sample key cache to quantization error, determine the first offloading strategy of the key cache; the first offloading strategy includes the quantization method and / or compression method of the key cache.

[0027] The sample key cache refers to multiple key caches used to test the K_cache offloading strategy. The first offloading strategy refers to the offloading strategy of the key cache. The first offloading strategy characterizes the offloading location (where the key cache is offloaded) and storage precision (what precision is used for storage). The first offloading strategy includes the key cache quantization method, meaning that the key cache is offloaded by quantizing it. The first offloading strategy includes the key cache compression method, meaning that the key cache is offloaded by compressing it. The first offloading strategy includes the key cache quantization method and / or compression method, meaning that the key cache is offloaded by quantizing and / or compressing it.

[0028] The sensitivity of key and value caches to quantization errors reflects the degree of degradation in the model's final performance (such as accuracy and perplexity). High sensitivity means that low-precision quantization of the cache will result in a significant and disproportionate drop in model performance. Low sensitivity means that even with lower precision, the performance loss is relatively small, demonstrating a higher tolerance. Key and value caches have different sensitivities to quantization errors, with the key cache being more sensitive than the value cache.

[0029] In some implementations, the sensitivity of key and value caches to quantization errors can be judged based on how well the model's final performance (such as accuracy and perplexity) responds to the quantization operation. For example, high sensitivity: when a cache is quantized, model performance experiences a significant and disproportionate decrease; this manifests as a sharp drop in model accuracy when the key cache uses low-bit quantization (e.g., 2-bit); the principle is that the key matrix has a larger norm (e.g., spectral norm and Frobenius norm), meaning it contains higher information density, and a high-norm matrix amplifies the errors introduced by quantization. Low sensitivity: when a cache is quantized, model performance remains relatively stable; this manifests as relatively small performance loss even when the value cache uses lower precision (e.g., 2-bit); the principle is that the value cache has relatively low information density and a stronger tolerance for quantization errors.

[0030] In some implementations, the sensitivity of the sample key cache to quantization error can be determined based on at least one of the following: the role of the sample key cache in the attention module, the quantization error propagation path, the tolerable accuracy, and the distribution of outliers.

[0031] In some implementations, the sample key cache can be low-bit quantized to obtain the perplexity of the sample key cache; the native perplexity corresponding to the sample key cache can be determined; the native perplexity refers to the perplexity of the model under the data precision of the key-value cache block; based on the perplexity of the sample key cache and the native perplexity, the sensitivity of the sample key cache to quantization error can be determined.

[0032] Perplexity (PPL) measures how well the output probability distribution of a language model matches the actual samples; a smaller PPL value indicates stronger predictive ability. For example, the formula for calculating the sensitivity of the sample key cache to quantization errors can be: ;in, The perplexity calculated for the sample key cache on the quantized compressed model can be determined based on the model's loss value on the test samples; The native perplexity calculated on a model without any quantization compression serves as a performance baseline; by... and By making comparisons, we can accurately measure the damage that quantization operations cause to model performance.

[0033] In some implementations, the quantization method of the sample key cache, whether further compression is needed on top of quantization, and the compression method can be determined based on the sensitivity of the sample key cache to quantization errors.

[0034] Step 103: Based on the sensitivity of the sample value cache to quantization error, determine the second offloading strategy of the value cache; the second offloading strategy includes the quantization method and / or compression method of the value cache.

[0035] The sample value cache refers to multiple value caches used to test the V_cache unloading strategy. The second unloading strategy refers to the unloading strategy for the value caches. The second unloading strategy includes: quantization method of the value cache, meaning that unloading the value cache is achieved by quantizing it; compression method of the value cache, meaning that unloading the value cache is achieved by compressing it; and quantization and / or compression method of the value cache, meaning that unloading the value cache is achieved by quantizing and / or compressing it.

[0036] In some implementations, the sensitivity of the sample value cache to quantization error can be determined based on at least one of the following: the role of the sample value cache in the attention module, the quantization error propagation path, the tolerable accuracy, and the distribution of outliers.

[0037] In some implementations, the sample value cache can be low-bit quantized to obtain the perplexity of the sample value cache; the native perplexity corresponding to the sample value cache can be determined; the native perplexity refers to the perplexity of the model under the data precision of the key-value cache block; based on the perplexity of the sample value cache and the native perplexity, the sensitivity of the sample value cache to quantization error can be determined. For example, the formula for calculating the sensitivity of the sample value cache to quantization error can be: ;in, Perplexity for caching sample values The original perplexity level.

[0038] In some implementations, the quantization method of the sample value cache, whether further compression is needed on the basis of quantization, and the compression method can be determined based on the sensitivity of the sample value cache to quantization error.

[0039] Step 104: Adopt the first unloading strategy to unload the key cache from the first storage space to the second storage space.

[0040] The first storage space refers to the storage space where the key cache is currently located. The first storage space can be GPU memory, such as GPU HBM. The second storage space refers to the storage space where the key cache will be offloaded. The second storage space can be Central Processing Unit (CPU) memory, such as CPU Dynamic Random Access Memory (DRAM). The second storage space can also be a hard disk, such as a Non-Volatile Memory Express Solid State Drive (NVMe SSD). The access speed of the first storage space is greater than that of the second storage space, and the hardware cost of the first storage space is greater than that of the second storage space, but the capacity of the first storage space is smaller than that of the second storage space.

[0041] In some implementations, the key cache can be processed using the quantization and / or compression methods in the first offloading strategy, and the processed key cache can be written from the first storage space to the second storage space to achieve the offloading of the key cache.

[0042] In some implementations, a first offloading strategy can be employed, using a hierarchical transfer method to offload the key cache from the first storage space to the second storage space. For example, the key cache is first offloaded from the first storage space to an intermediate storage space, and then from the intermediate storage space to the second storage space. Specifically, the key cache is first quantized using the quantization method in the first offloading strategy, and the quantized key cache is written from the first storage space to the intermediate storage space. Then, the quantized key cache is compressed using the compression method in the first offloading strategy, and the compressed key cache is written from the intermediate storage space to the second storage space. In this case, the first storage space can be GPU HBM, the intermediate storage space can be CPU DRAM, and the second storage space can be NVMe SSD.

[0043] Step 105: Use the second unloading strategy to unload the value cache from the first storage space to the third storage space.

[0044] The third storage space refers to the storage space where the value cache will be unloaded. The second and third storage spaces can be the same or different, depending on the data precision of the key cache and the data precision of the value cache.

[0045] In some implementations, the value cache can be processed using the quantization and / or compression methods in the second offloading strategy, and the processed value cache can be written from the first storage space to the third storage space to achieve the offloading of the value cache.

[0046] In some implementations, a second offloading strategy can be employed, using a hierarchical transfer method to offload the value cache from the first storage space to the second storage space. For example, the value cache is first offloaded from the first storage space to an intermediate storage space, and then from the intermediate storage space to a third storage space. Specifically, the value cache is first quantized using the quantization method in the second offloading strategy, and the quantized value cache is written from the first storage space to the intermediate storage space. Then, the quantized value cache is compressed using the compression method in the second offloading strategy, and the compressed value cache is written from the intermediate storage space to the third storage space. In this case, the first storage space can be a GPU HBM, the intermediate storage space can be CPU DRAM, and the second storage space can be an NVMe SSD.

[0047] In this embodiment, the key-value cache block is split into a key cache and a value cache to achieve physical storage separation between the key cache and the value cache. This allows the key cache and value cache to occupy different physical address spaces, have independent metadata records, and be independently movable, quantized, compressed, and even stored in different storage layers. A first offloading strategy, determined by the sensitivity of the sample key cache to quantization errors, is used to offload the key cache, and a second offloading strategy, determined by the sensitivity of the sample value cache to quantization errors, is used to offload the value cache. By utilizing the asymmetric sensitivity of key / value pairs and employing different offloading strategies for the key cache and value cache, and even storing them in different storage layers, an optimal balance can be achieved between storage efficiency and inference accuracy.

[0048] In some embodiments, the sensitivity of the sample key cache to quantization error in step 102 above can be determined by the following steps A1 to A3: Step A1: Determine the sensitivity of the sample key cache to quantization accuracy based on at least one of the following: the role of the sample key cache in the attention module, the quantization error propagation path, and the tolerable accuracy.

[0049] The role of the sample key cache in the attention module refers to the operations in which it participates. For example, the sample key cache participates in Q × K^T dot product matching, which determines the attention distribution. The quantization error propagation path refers to the path from the generation of quantization error to its final impact. For example, the quantization error propagation path of the sample key cache is that the quantization error is exponentially amplified by the softmax function, leading to an attention routing error. The acceptable precision of the sample key cache is no less than 8-bit floating-point (FP8) / 8-bit integer (INT8) (below 4 bits is significantly degraded).

[0050] Sensitivity to quantization precision focuses on changes in the number of bits, such as decreasing from FP16 to INT8, and then to INT4. If "a slight decrease in quantization precision leads to a significant drop in model accuracy," it indicates that the cache is highly sensitive to quantization precision; conversely, if a very low decrease in quantization precision doesn't significantly affect accuracy, it indicates that the cache is not highly sensitive to quantization precision. Sensitivity to quantization error, on the other hand, focuses on changes in numerical distribution, such as the mean squared error (MSE), signal-to-noise ratio (SNR), or the degree to which outliers are truncated during quantization. If the cache uses low-bit quantization, a 1% distortion in the cached element values ​​results in a significant drop in model accuracy, indicating high sensitivity to quantization error; conversely, if the cache uses bit-level quantization, a 1% distortion in the cached element values ​​doesn't affect model performance, indicating low sensitivity to quantization error. The propagation path of quantization precision and quantization error is as follows: a decrease in cache quantization precision (reduced number of bits) generates / amplifies quantization error (numerical distortion), which propagates through attention calculations, ultimately leading to a decrease in model performance.

[0051] In some implementations, the role of the sample key cache in the attention module, the quantization error propagation path, and the tolerable precision can be stored to obtain a quantization precision sensitivity table for the sample key cache; the quantization precision sensitivity table is used to characterize the sensitivity of the sample key cache to quantization precision.

[0052] In some implementations, the role of the sample key cache in the attention module, the quantization error propagation path, and the tolerable precision can be numerically quantized to obtain a first quantized value, a second quantized value, and a third quantized value; the first quantized value, the second quantized value, and the third quantized value are weighted and summed to obtain the sensitivity of the sample key cache to quantization precision.

[0053] For example, the role and influence of the sample key cache in the attention module can be determined based on the ratio of the spectral norm of the sample key cache to the spectral norm of the sample query cache. The role and influence are positively correlated with sensitivity, that is, the larger the value of the role and influence, the stronger the ability of the key cache to amplify input errors in the dot product operation, and the higher the sensitivity.

[0054] For example, a small amount of noise (e.g., MSE=0.01) can be added to the sample key buffer, and then the model can be run to measure the change in the KL divergence of the model output. The change in KL divergence reflects the degree of influence of the quantization error propagation path on sensitivity. The quantization error propagation path and sensitivity are positively correlated; that is, the larger the change in KL divergence, the longer the error propagation path, the wider the impact, and the higher the sensitivity.

[0055] For example, the sample key cache can be gradually reduced from high precision (e.g., from FP16 to INT4), and the critical point at which the model's perplexity begins to collapse can be observed. The magnitude of the critical point reflects the impact of tolerable precision on sensitivity. Tolerable precision and sensitivity are positively correlated; that is, the larger the critical point value, the higher the precision required to maintain performance, and the more sensitive the model is to precision.

[0056] Step A2: Based on the distribution of outliers in the sample key cache, determine the sensitivity of the sample key cache to quantization granularity.

[0057] In some implementations, outliers in the sample key cache are concentrated along the channel dimension, meaning that the numerical amplitude of some channels is significantly larger than that of other channels. Quantizing along the token dimension would place large outliers and small values ​​under the same scaling factor, resulting in a severe loss of precision for small values. Therefore, the sample key cache is sensitive to quantization granularity on a per-channel basis.

[0058] Step A3: Determine the sensitivity of the sample key cache to quantization error based on the sensitivity of the sample key cache to quantization precision and the sensitivity of the sample key cache to quantization granularity.

[0059] The sensitivity of the sample key cache to quantization error includes the sensitivity of the sample key cache to quantization accuracy and the sensitivity of the sample key cache to quantization granularity.

[0060] In this embodiment of the disclosure, the sensitivity of the sample key cache to quantization granularity is determined by the role of the sample key cache in the attention module, the quantization error propagation path, the tolerable accuracy, and the distribution of outliers in the sample key cache. This allows for an accurate determination of the sensitivity of the sample key cache to quantization errors, and thus a more accurate determination of the sample key cache unloading strategy. This enables the unloading of the key cache without affecting its accuracy.

[0061] In some embodiments, the sensitivity of the sample value cache in step 103 above to quantization error can be determined by the following steps B1 to B3: Step B1: Determine the sensitivity of the sample value cache to quantization accuracy based on at least one of the following: the role of the sample value cache in the attention module, the quantization error propagation path, and the tolerable accuracy.

[0062] The role of the sample value cache in the attention module refers to the operations in which it participates. For example, the sample value cache participates in the softmax(QK^T) × V weighted summation, which determines the output value. The quantization error propagation path of the sample value cache is such that the error is averaged to the output by the softmax normalized weights, limiting the deviation. The acceptable precision of the sample value cache can be aggressively compressed to 2-bit integers (INT2) or even 1 bit.

[0063] In some implementations, the role of the sample value cache in the attention module, the quantization error propagation path, and the tolerable precision can be stored to obtain a quantization precision sensitivity table for the sample value cache; the quantization precision sensitivity table is used to characterize the sensitivity of the sample value cache to quantization precision.

[0064] In some implementations, the role of the sample value cache in the attention module, the quantization error propagation path, and the tolerable precision can be numerically quantized to obtain a fourth quantization value, a fifth quantization value, and a sixth quantization value; the fourth quantization value, the fifth quantization value, and the sixth quantization value are weighted and summed to obtain the sensitivity of the sample value cache to quantization precision.

[0065] Step B2: Based on the distribution of outliers in the sample value cache, determine the sensitivity of the sample value cache to quantization granularity.

[0066] In some implementations, the numerical distribution of the sample value cache is relatively uniform across both the channel and token dimensions, and the attention-weighted summation is insensitive to the relative precision between channels. Therefore, the sensitivity of the sample value cache to quantization granularity is per token.

[0067] Step B3: Determine the sensitivity of the sample value cache to quantization error based on the sensitivity of the sample value cache to quantization accuracy and the sensitivity of the sample value cache to quantization granularity.

[0068] The sensitivity of the sample value cache to quantization error includes the sensitivity of the sample value cache to quantization accuracy and the sensitivity of the sample value cache to quantization granularity.

[0069] In this embodiment of the disclosure, the sensitivity of the sample value cache to quantization granularity is determined by the role of the sample value cache in the attention module, the quantization error propagation path, the tolerable accuracy, and the distribution of outliers in the sample value cache. This allows for an accurate determination of the sensitivity of the sample value cache to quantization errors, and thus a more accurate determination of the sample value cache unloading strategy. This enables the unloading of the value cache without affecting its accuracy.

[0070] It should be noted that the asymmetric sensitivity of Key and Value Cache refers to the fundamentally different sensitivities of the Key cache and Value cache to quantization errors in large language model inference. Specifically, this includes two differences: A. Differences in sensitivity to precision (bit-width), as shown in Table 1 below.

[0071] Table 1

[0072] B. Differences in sensitivity to granularity.

[0073] K_cache's outliers are concentrated along the channel dimension—meaning the numerical amplitude of some channels is significantly larger than that of other channels. If quantization is performed along the token dimension, large outliers and small values ​​will be placed on the same scale, resulting in a severe loss of precision for small values. Therefore, K_cache must be per-channel quantized (calculating the scale / zero-point independently along the channel dimension).

[0074] The numerical distribution of V_cache is relatively uniform across both the channel and token dimensions, and the weighted summation of attention is insensitive to the relative precision between channels. Therefore, V_cache can be quantized per-token (each token forms a group of scale / zero-points), achieving a more compact storage layout while maintaining precision (facilitating batch loading by the attention kernel).

[0075] In some embodiments, step 102 above can be implemented by the following steps 1021 to 1022: Step 1021: Given that the outliers in the sample key cache are concentrated in the channel dimension, the quantization method of the key cache is determined to be channel-by-channel quantization.

[0076] Since outliers in the key cache are concentrated in the channel dimension, and token-based quantization would result in a severe loss of precision for small values, the key cache is quantized per channel to preserve the precision of attention weight calculation.

[0077] The technical mechanism of preserving the precision of attention weight calculation by performing channel-by-channel quantization on the key cache consists of three steps: Step 1: Identify the outlier distribution pattern of K_cache. Statistical analysis of the K_cache distribution of several mainstream LLMs revealed that outliers are highly concentrated in a few specific channels. Within the same channel, the numerical amplitudes across tokens are similar, while the amplitudes can differ by 10 to 100 times between different channels. This characteristic stems from the fact that the W_K projection matrix of the Transformer learns stronger activation responses in certain channels during training.

[0078] Step 2: Compare the impact of the two quantization granularities on accuracy, as shown in Table 2 below.

[0079] Table 2

[0080] Step 3: Why this directly determines the accuracy of attention weights.

[0081] The core steps of attention computation: score[i,j] = (Q[i] K[j]) / √d_k ← Dot product matching, the contribution of each channel is accumulated independently; attn[i,j] = softmax_j(score[i,j]) ← Performs softmax normalization on dimension j; output[i] = Σ_j attn[i,j]× V[j] ← Weighted summation.

[0082] Dot product Q[i] K[j] is a process of summing across channels: Σ_c Q[i,c]× K[j,c]. If the quantization error of K in a certain channel is large, the entire dot product result will be contaminated.

[0083] Softmax is an exponential function used to normalize the input after amplification by exponentiation (exp). Small perturbations in the dot product are exponentially amplified after exponentiation, which may cause the top-1 token of the attention to drift from the correct position to the wrong position, i.e., the attention path is incorrect.

[0084] Per-channel quantization ensures that the contribution of each channel in the dot product remains sufficiently accurate, thus ensuring the distribution of the dot product result and the subsequent softmax distribution.

[0085] Highly aligned with the FP16 baseline (data precision of the original key-value cache block) (measured PPL difference <0.3%).

[0086] Implementation of channel-wise quantization: In the FP16 → FP8 E4M3 quantization kernel, amax and scale are calculated independently along the channel dimension: scale[c] = amax(|K[:,c]|) / FP8_max.

[0087] Quantized storage layout: FP8 tensor of [num_tokens, num_channels] + FP16 scale vector of [num_channels].

[0088] Total metadata overhead: per-channel scale vectors are approximately 0.4% of the K_cache ontology (for head_dim=128).

[0089] Step 1022: If the sample key cache satisfies the residual coding condition in the sensitivity characterization of the sample key cache to quantization error, determine that the compression method of the key cache is residual coding.

[0090] The residual coding conditions can be preset. For example, the residual coding conditions can be set to the storage space occupied by the key cache being greater than the first storage space threshold, or the residual coding conditions can be set to the data precision of the key cache being further compressed.

[0091] In some implementations, the compression method for the key cache can be determined to be residual coding based on the accuracy requirements of the sample key cache.

[0092] In this embodiment, the outliers in the sample key cache are concentrated in the channel dimension, and the quantization method of the key cache is determined to be channel-wise quantization. Channel-wise quantization can reduce the precision loss of the key cache and improve the accuracy of the key cache in attention weight calculation. When the sample key cache meets the residual encoding condition, the compression method of the key cache is determined to be residual encoding, which can further reduce the storage space of the key cache and reduce resource waste.

[0093] In some embodiments, step 103 above can be implemented by the following steps 1031 to 1032: Step 1031: Given that the sample value cache is uniformly distributed across the channel dimension and the lexical dimension, the quantization method of the value cache is determined to be lexical-by-lexical quantization.

[0094] Since the values ​​in the value cache are evenly distributed across the channel and lexical dimensions, the quantization method for the value cache is per-lexical quantization to avoid affecting the summation accuracy.

[0095] The core reason for performing word-by-word meta-quantization on the value cache is that V_cache only participates in weighted summation (rather than dot product matching) in attention, and has a triple natural error suppression mechanism: Suppression mechanism 1: Error averaging effect of softmax normalization.

[0096] Attention output formula: output[i] = Σ_j softmax(QK^T)[i,j]× V[j]; The weights w_j output by softmax satisfy Σ_j w_j = 1 (normalized); The quantization error ε_j of V_cache becomes Σ_j w_j × ε_j when summed – this is the weighted average of ε_j; According to the law of large numbers, when the sequence length N is sufficiently large, the independent quantization errors exhibit a significant difference after averaging. attenuation.

[0097] Suppression Mechanism 2: Suppression of amplified error due to sparsity of attention weights.

[0098] The attention distribution of a Transformer is typically highly sparse (top-k concentrated), with most w_j ≈ 0.

[0099] For a token with w_j ≈ 0, its V quantization error w_j × ε_j ≈ 0 has no effect on the output.

[0100] In reality, only the V quantization error of the top 1% to 5% of tokens is "seen," and these tokens are often attention sinks or recent tokens, which can be further preserved through a "high-precision window protection" strategy. The attention sink phenomenon refers to the fact that a very small number of initial tokens at the beginning of the model sequence absorb a large amount of attention weights.

[0101] Suppression mechanism 3: The granularity of per-token quantization is already fine enough.

[0102] The numerical distribution of different channels within the same token in V_cache is relatively uniform (unlike K, which has a strong channel outlier).

[0103] Per-token quantization calculates the scale individually for each token, thus fully capturing the numerical amplitude characteristics of that token.

[0104] Under the semantics of weighted summation Σ_j w_j × V[j], "different scales between different tokens" will not cause any problems - each V[j] has been correctly dequantized before being multiplied by w_j.

[0105] Step 1032: If the sample value cache satisfies the block compression condition, then the compression method of the value cache is determined to be block encoding.

[0106] Block compression conditions can be preset. For example, block compression conditions can be set to the storage space occupied by the value cache being greater than the second storage space threshold, or block compression conditions can be set to the data precision of the value cache being further compressed.

[0107] In some implementations, the compression method for the key cache can be determined to be residual coding based on the accuracy requirements of the sample key cache.

[0108] In this embodiment, the quantization method of the value cache is determined to be per-word quantization based on the uniform distribution of sample key cache values ​​across the channel and lexical dimensions. Per-word quantization allows for value cache quantization without affecting summation accuracy. When the sample value cache meets the block compression condition, block compression is determined as the compression method for the value cache, which can further reduce the storage space of the value cache and reduce resource waste.

[0109] In some embodiments, the first storage space includes the video memory of the graphics processor, and the second storage space includes a hard disk; step 104 above can be performed by unloading the key cache to the hard disk through steps C1 to C3 as follows: Step C1: Perform channel-by-channel quantization on the key cache to obtain the quantized key cache; the data precision of the key cache is the first data precision, and the data precision of the quantized key cache is the second data precision; the first data precision is higher than the second data precision.

[0110] The first data precision refers to the raw data precision of the key cache. The second data precision refers to the quantized data precision of the key cache. The first data precision can be FP16, and the second data precision can be FP8 E4M3 (per-channel).

[0111] In some implementations, the key cache is grouped along the channel dimension to obtain multiple key cache groups; quantization parameters for each key cache group are determined; and each key cache group is quantized based on its quantization parameters to obtain a quantized key cache. The quantization parameters include at least a scaling factor and a zero-point; the zero-point refers to the numerical translation offset used for asymmetric quantization.

[0112] Step C2: Perform residual encoding on the quantized key cache to obtain the encoded key cache.

[0113] In some implementations, a stepwise fitting residual method can be used to perform residual encoding on the quantized key cache to obtain an encoded key cache.

[0114] In some implementations, the strong correlation between the key vectors of adjacent words in the sequence can be used to encode the quantized key cache residuals to obtain the encoded key cache.

[0115] Step C3: Using the direct data path between the graphics processor and the hard disk, the encoded key cache is written from the graphics processor's video memory to the hard disk to unload the key cache; the first unloading strategy includes channel-by-channel quantization and residual encoding.

[0116] In some implementations, the direct data path between the graphics processor and the hard disk can be a High-efficiency Flash-based KV Cache Swapping (HiFC) path (GPU → GPU Direct Storage (GDS) → PseudoSingle-Level Cell Non-Volatile Memory Express (pSLC NVMe)).

[0117] GDS allows the GPU to directly access local NVMe SSDs or remote storage (via Remote Direct Memory Access (RDMA) networks), with data transferred directly between the storage device and GPU memory, bypassing the CPU and system memory (DRAM). GDS requires supported NVMe drivers and application programming interfaces, which are key interfaces for integration with the HiFC solution in this disclosure embodiment.

[0118] In some implementations, the HiFC path (GPU → GDS → pSLC NVMe) is used to directly write the encoded key cache from the graphics processor's video memory (CPU HBM) to the hard disk (NVMe SSD).

[0119] like Figure 2 As shown, K_Cache is first quantized per-channel, then residual (Delta) encoded, and the encoded K_Cache is written from the CPU HBM to the NVMe SSD using the HiFC path.

[0120] In this embodiment of the disclosure, by performing channel-by-channel quantization and residual encoding on the key cache, and using the direct data path between the graphics processor and the hard disk, the encoded key cache is written directly from the graphics processor's video memory to the hard disk, thereby achieving key cache offloading. This path bypasses the CPU and system memory (DRAM), improving the key cache offloading rate.

[0121] In some embodiments, step C2 above can be implemented by the following steps C21 to C23: Step C21: Determine the first term in the key cache.

[0122] The first lexical unit can be represented as K[0].

[0123] Step C22: For all other tokens in the key cache except the first token, determine the residual between the other tokens and the corresponding anchor point; select a representative token as the anchor point for each target number of tokens.

[0124] The i-th lexical unit can be represented as K[i]. The corresponding anchor point refers to the anchor point closest to other lexical units, which can be represented as k[nearest(i)].

[0125] Step C23: Based on the first lexical unit and the residuals of other lexical units and their corresponding anchor points, determine the encoded key cache.

[0126] In some implementations, the first lexical unit can be stored in its entirety, and for other lexical units besides the first lexical unit, the residuals between the other lexical units and their corresponding anchor points can be stored.

[0127] In existing schemes, the K-vector of each token is stored independently (filling the space). However, the residual (Delta) encoding of this embodiment is as follows: anchor_k[0] = K[0] / / The first token is stored completely; delta_k[i] = K[i] - anchor_k[nearest(i)] / / The residual between the subsequent token and the nearest anchor point; where i is a positive integer, and anchor_k[nearest(i)] is the nearest anchor point of the i-th token (k[i]).

[0128] Anchor selection: Select one anchor for every B tokens. B is the target number, B=32, and B is a configurable parameter.

[0129] In this embodiment of the disclosure, by storing the first word completely, and storing the residuals between the other words and their corresponding anchors, further compression of the key cache can be achieved, thereby reducing the storage space occupied by the key cache.

[0130] In some embodiments, the key-value cache block unloading method provided in this disclosure can also perform lower-precision storage through the following steps C221 to C223: Step C221: If the residuals between other words and their corresponding anchors are within the preset residual range, reduce the data precision of the residuals between other words and their corresponding anchors to the third data precision; the preset residual range is determined based on the second data precision.

[0131] The third data precision is lower than the second data precision. When the second data precision is FP8, the third data precision can be IN6 (6 bits, with per block scale). The residuals between other tokens and their corresponding anchors are within the preset residual range, indicating that the residual values ​​are concentrated within the preset residual range, and lower precision storage can be used further.

[0132] In some implementations, the preset residual range is 13% to 15% of the maximum dynamic range value that the second data type can represent. In this case, when the residuals between other tokens and their corresponding anchors are between 13% and 15% of the maximum dynamic range that the second data precision can represent, the data precision of the residuals between other tokens and their corresponding anchors is reduced to IN6.

[0133] Step C222: If the absolute value of the standard deviation of the residual between other words and the corresponding anchor point is less than the preset residual threshold, replace the data precision of the residual between other words and the corresponding anchor point with the preset value.

[0134] The preset value can be 0. In some implementations, if the absolute value of the standard deviation (σ) of the residual between other words and the corresponding anchor point is less than the preset residual threshold, then 0 (1-bit flag) is stored; if the absolute value of the standard deviation (σ) of the residual between other words and the corresponding anchor point is greater than or equal to the preset residual threshold (θ), then INT4 value + 4-bit index is stored.

[0135] The preset residual threshold (θ) is not a hard threshold, but a statistical distribution characteristic. The specific parameter can be defined in two ways: Method 1: Threshold set based on standard deviation.

[0136] Based on the measured statistics when the anchor spacing B=32, the distribution of the residual delta_k[i] = K[i]- anchor_k[nearest(i)] approximately follows a Gaussian-Laplace mixed distribution with zero mean and narrow distribution. Typical parameters are: mean μ≈ 0, standard deviation σ_delta ≈ (any value from 0.15 to 0.25) × σ_K (standard deviation of the original K).

[0137] The engineering definition of "the absolute value of the residual is less than the preset residual threshold" is: approximately 95% of the δ values ​​fall within the range of [-3σ_delta, +3σ_delta], and approximately 99% fall within the range of [-5σ_delta, +5σ_delta].

[0138] Method 2: Threshold based on quantized dynamic range.

[0139] If the dynamic range of the original K[i] in FP8 E4M3 is [ [448, +448]; the dynamic range of the measured delta residual is approximately [ [60, +60] (13% to 15% of the FP8 range); this means that most delta values ​​can be represented losslessly using data types with narrower dynamic ranges (such as FP6 or INT6).

[0140] Step C223: If the absolute value of the standard deviation of the residual between other words and the corresponding anchor point is greater than or equal to the preset residual threshold, reduce the data precision of the residual between other words and the corresponding anchor point to the fourth data precision; the third data precision is higher than the fourth data precision.

[0141] The fourth data precision is lower than the third data precision. When the third data precision is IN6, the fourth data precision can be INT4 + sparse flag bits, such as INT4 value + 4-bit index.

[0142] In some implementations, if the absolute value of the standard deviation (σ) of the residual between other tokens and the corresponding anchor point is greater than or equal to a preset residual threshold (θ), then an INT4 value + a 4-bit index is stored.

[0143] As shown in Table 3 below, the specific parameters of the graded accuracy reduction strategy adopted in this embodiment are as follows: Table 3

[0144] In some implementations, other tokens can take the form of an 8-bit anchor plus a 6-bit residual. The per-block scaling factor is used, with a block size of 32 tokens.

[0145] The storage overhead for every 32 tokens is: 1 × 128 × 8 + 31 × 128 × 6 + 32 × 16 = 2128 + 2976 + 512 = 5616 bits; compared to the traditional FP8: 32 × 128 × 8 = 32768 bits; compression ratio: 5616 / 32768 ≈ 17.1% (compression ratio of about 8.6% compared to the FP16 baseline).

[0146] In some implementations, the reduction in data precision can be obtained by solving three constraints jointly. Constraint 1: Precision loss; the end-to-end perplexity change (PPL change) of the model throughout its lifetime is less than the perplexity change threshold (0.5%). Constraint 2: Hardware alignment; bit width is within {2, 4, 6, 8}, facilitating Single Instruction, Multiple Data (SIMD) / tensor core kernel processing. Constraint 3: Compression ratio is reduced by a preset percentage (40%) relative to the second data precision.

[0147] like Figure 3 As shown, the process for determining the reduction in data precision can be as follows: Step 301, Input: K_cache statistical characteristics (dynamic range / distribution / layer sensitivity); Step 302, Set constraint 1: precision loss end-to-end PPL change < 0.5%; Step 303, Set constraint 2: hardware-friendly; bit width ∈ {2,4,6,8} aligned (facilitating SIMD / tensor core kernel processing); Step 304, Set constraint 3: compression ratio reduced by 40% compared to FP8; Step 305, Joint solution: select the minimum bit width that satisfies constraint 1 and also satisfies constraints 2 and 3. Step 306, Output: residual bit width = 6 (conservative), the aggressive version can be reduced to 4.

[0148] In some implementations, the calibration process for the reduction in accuracy is as follows: Step 1, perform an FP16 inference using a calibration set (such as a subset of WikiText / C4, 1000 × 8k tokens) and record the distribution of K_cache for each layer; Step 2, scan the candidate bit-width for each layer ∈ {4, 5, 6, 7, 8} and calculate the quantization error; Step 3, use the layer-wise error obtained from the scan to infer the perplexity impact (using a first-order Taylor approximation or a Hessian trace weighting); Step 4, select the minimum bit-width that satisfies the perplexity constraint for each layer to obtain a layer-wise mixed precision configuration table; Step 5, during deployment, apply different residual bit-widths to different layers.

[0149] In some implementations, a conservative recommendation is to use a uniform 6-bit residual across all layers (no offline calibration required, plug and play); a more refined recommendation is to use 4 bits for the shallow layer + 5 bits for the middle layer + 6 bits for the attention sink layer (obtained through the calibration process described above).

[0150] It's important to note that the "INT4 value + 4-bit index" storage strategy targets a very small number of outlier features (e.g., less than 5%). For the vast majority of normal features, only INT4 or even lower bit widths are stored. Taking a typical case where outliers account for 5% as an example: the overall expected bit width = 5% × 8 bits + 95% × 4 bits = 4.2 bits / parameter. In contrast, under the global FP8 scheme, each parameter consumes a fixed 8 bits regardless of whether the feature dimension is outlier. Therefore, the overall expected compression rate of "INT4 value + 4-bit index" is: 4.2 bits ÷ 8 bits = 52.5%. That is, compared to global FP8, storage space is saved by 47.5%. Even considering the worst-case scenario where the outlier proportion rises to 10%, the overall expected bit width is only 4.4 bits / parameter, and the compression rate still reaches 55%, demonstrating significant space savings. This result fully proves that the "INT4 value + 4-bit index" storage strategy achieves significant space savings compared to global FP8 without sacrificing data precision. The “4-bit index” can cover a cache block of size 16, and the number of bits in the index is logically matched with the group size in this embodiment of the disclosure.

[0151] In this embodiment of the disclosure, by adopting a hierarchical reduction strategy, the key cache can be compressed step by step, thereby continuously reducing the data size of the key cache and thus reducing the storage space of the key cache.

[0152] In some embodiments, the first storage space includes the video memory of the graphics processor, and the second storage space includes the video memory of the central processing unit; step 104 above can also be performed by the following steps D1 to D2 to offload the key cache to the video memory of the central processing unit: Step D1: Perform channel-by-channel quantization on the key cache to obtain the quantized key cache; the data precision of the key cache is the first data precision, and the data precision of the quantized key cache is the second data precision; the first data precision is higher than the second data precision.

[0153] Here, step D1 corresponds to the aforementioned step C1. When implementing it, the implementation method of the aforementioned step C1 can be referred to. This embodiment of the present disclosure will not repeat the details.

[0154] Step D2: Write the quantized key cache from the graphics processor's video memory to the central processing unit's video memory to unload the key cache.

[0155] In some implementations, a data transfer channel between the graphics processor and the central processing unit is used to write the quantized key cache from the graphics processor's video memory to the central processing unit's video memory.

[0156] like Figure 2 As shown, K_Cache can be quantized per-channel, and the quantized K_Cache can be written from CPUHBM to CPU DRAM.

[0157] In this embodiment of the disclosure, the key cache is quantized channel by channel, and the quantized key cache is written from the graphics processor's video memory to the central processing unit's video memory to achieve key cache offloading. This method of offloading the key cache to the central processing unit's video memory is faster than offloading it to the hard disk, and the subsequent recovery speed is also faster.

[0158] In some embodiments, the first storage space includes the video memory of the graphics processor, and the second storage space includes the hard disk; step 104 above can also be performed by unloading the key cache to the hard disk through the following steps E1 to E4: Step E1: When the memory usage of the graphics processor is greater than or equal to the first memory usage threshold, the key cache is quantized channel by channel to obtain the quantized key cache; the data precision of the key cache is the first data precision, and the data precision of the quantized key cache is the second data precision; the first data precision is higher than the second data precision.

[0159] The first occupancy threshold is used to determine when to trigger the uninstallation. The first occupancy threshold can be preset to a value; for example, it can be set to 85%.

[0160] In some implementations, if the first occupancy threshold is 85%, an offloading event is triggered when the KV Cache occupancy in the GPU HBM exceeds the high watermark W_high (default 85%).

[0161] In some implementations, the implementation of "quantizing the key cache channel by channel to obtain the quantized key cache" in step E1 is similar to the implementation of step C1 above. When implementing, the implementation of step C1 above can be referred to, and this disclosure will not elaborate further.

[0162] Step E2: Write the quantized key cache from the graphics processor's video memory to the central processing unit's video memory.

[0163] Here, step E2 corresponds to the aforementioned step D2. When implementing it, the implementation method of the aforementioned step D2 can be referred to. This embodiment of the disclosure will not be described in detail here.

[0164] Step E3: When the utilization rate of the CPU's video memory is greater than or equal to the second utilization rate threshold, perform residual encoding on the quantized key cache to obtain the encoded key cache.

[0165] The second occupancy threshold is used to determine the timing of residual coding. The second occupancy threshold can be the same as or different from the first occupancy threshold, depending on business requirements.

[0166] In some implementations, the "performing residual encoding on the quantized key cache to obtain the encoded key cache" in step E2 is similar to the implementation of step C2 above. When implementing it, the implementation of step D2 above can be referred to. This disclosure will not elaborate further on this aspect.

[0167] Step E4: Write the encoded key cache from the CPU's video memory to the hard disk to unload the key cache.

[0168] In some implementations, the data transfer channel between the central processing unit and the hard disk can be used to write the encoded key cache from the central processing unit's video memory to the hard disk, thereby unloading the key cache.

[0169] like Figure 2 As shown, K_Cache is first quantized per-channel, and the quantized K_Cache is written from CPUHBM to CPU DRAM. Then, the quantized K_Cache is residually encoded (Delta), and the encoded K_Cache is written from CPUDRAM to NVMe SSD.

[0170] In this embodiment, when the GPU memory occupancy rate is greater than or equal to a first occupancy rate threshold, the key cache is quantized channel by channel, and the quantized key cache is written from the GPU memory to the CPU memory. When the CPU memory occupancy rate is greater than or equal to a second occupancy rate threshold, the quantized key cache is further residual encoded, and the encoded key cache is written from the CPU memory to the hard disk. In this way, by adopting a layer-by-layer offloading method, the node load can be reduced layer by layer, the offloading flexibility can be improved, and the read and write requirements of large models with high throughput, low latency, and massive KVCache can be met.

[0171] In some embodiments, the first storage space includes the video memory of the graphics processor, and the third storage space includes the hard disk; step 105 above can be performed by unloading the value cache to the hard disk through steps F1 to F3 as follows: Step F1: Perform word-by-word meta-quantization on the value cache to obtain the quantized value cache; the data precision of the value cache is the first data precision, and the data precision of the quantized value cache is the fifth data precision; the first data precision is higher than the fifth data precision.

[0172] The fifth data precision refers to the data precision of the quantized value cache. For example, the fifth data precision can be INT4.

[0173] In some implementations, the value cache is grouped along the lexical dimension to obtain multiple groups of value caches; the quantization parameters of each group of value caches are determined; and each group of value caches is quantized based on the quantization parameters of each group of value caches to obtain quantized value caches.

[0174] Step F2: Perform block compression on the quantized value cache to obtain a compressed value cache.

[0175] In some implementations, the quantized value cache can be divided into key values ​​and non-key values; the key values ​​in the quantized value cache are retained, and the non-key values ​​are set to 0.

[0176] In some implementations, the quantized value cache can be coarsely quantized at the block level, and then low-bit quantization can be performed based on the residual of the coarsely quantized value cache to obtain a compressed value cache.

[0177] Step F3: Using the direct data path between the graphics processor and the hard disk, the compressed value cache is written from the graphics processor's video memory to the hard disk to unload the value cache.

[0178] In some implementations, the direct data path between the graphics processor and the hard disk can be a HiFC path (GPU→GDS→pSLC NVMe). In this way, using the HiFC path (GPU →GDS→pSLC NVMe), the encoded value cache is directly written from the graphics processor's video memory to the hard disk.

[0179] like Figure 2 As shown, V_Cache is first quantized per-token, then compressed into blocks, and the encoded V_Cache is written from the CPU HBM to the NVMe SSD using the HiFC path.

[0180] In this embodiment of the disclosure, by performing word-by-word meta-quantization and block compression on the value cache, and using the direct data path between the graphics processor and the hard disk, the encoded value cache is written directly from the graphics processor's video memory to the hard disk, thereby achieving the offloading of the value cache. This path bypasses the CPU and system memory (DRAM), improving the offloading rate of the value cache.

[0181] In some embodiments, step F2 above can be implemented by the following steps F21 to F23: Step F21: Store the compressed values ​​as multiple value data blocks according to the preset block size; the preset block size is determined based on the compression ratio prediction table.

[0182] The preset block size can be 4KB. The preset block size is set by taking into account the following factors: (1) In the INT4 data format, a 4KB block can hold 8192 INT4 values. When the typical head_dim=128, the V data of 64 tokens is well aligned with the block granularity (16 tokens to 32 tokens) of PagedAttention; (2) 4KB matches the physical page size of NVMe SSD (common 4K / 8K), and can be completed in one I / O; (3) LZ4 achieves the best balance between compression-decompression throughput and compression rate on 4KB blocks (the compression rate gain converges when it is greater than 4KB, and the header overhead is too high when it is less than 4KB).

[0183] A compression ratio prediction table is used to characterize the correspondence between the entropy of a value vector and its compression ratio. The compression ratio prediction table can be determined based on the numerical repeatability of the value vector. The numerical repeatability of the value vector can be determined based on at least one of the following: value vector type, entropy characteristics, quantized numerical distribution, and probability of repeating substrings.

[0184] In some implementations, after value quantization to INT4, each value has only 16 possibilities (0 to 15). Taking LZ4 compression as an example, the compression ratio prediction table can be shown in Table 4 below: Table 4

[0185] like Figure 4As shown, the actual calibration process of the compression ratio prediction table includes: Step 401, offline calibration stage; Step 402, run FP16 inference with the calibration set and dump each layer of V_cache; Step 403, quantize to INT4 per-token; Step 404, split into 4KB blocks; Step 405, compress each block independently with LZ4 and record the actual compression ratio; Step 406, calculate the mean, variance, and quantiles by layer / head / token position; Step 407, generate the compression ratio prediction table (layer_id→avg_ratio, p95_ratio); Step 408, use this table for storage budget planning at runtime.

[0186] In some implementations, a preset block size can be determined based on a compression ratio prediction table to achieve an optimal balance between compression-decompression throughput and compression ratio; the compressed value cache is organized into multiple value data blocks according to the preset block size, with each value data block being the size of the preset block size.

[0187] Step F2 determines the compression ratio of the value data block from the compression ratio prediction table based on the entropy characteristics of the value vectors in the value data block; the entropy characteristics of the value vectors in the value data block are used to characterize the numerical repeatability of the value vectors in the value data block.

[0188] The entropy of the value vector in the value data block represents the numerical distribution of the value vector and further represents the repetition of the value vector values.

[0189] In some implementations, as shown in Table 2, the V vector can be divided into three types: low entropy, medium entropy, and high entropy. Each type has different entropy characteristics, INT4 quantized numerical distribution, and probability of repeated substrings. Therefore, the corresponding LZ4 expected compression ratio (i.e., the compression ratio of the value data block) can be determined based on at least one of the entropy characteristics, INT4 quantized numerical distribution, and probability of repeated substrings of the value vector in the value data block.

[0190] Step F23: Compress the value data block according to the compression ratio of the value data block to obtain the compressed value cache.

[0191] In some implementations, LZ4 compression can be used to compress value data blocks according to their compression ratio, resulting in a compressed value cache. The reason for choosing LZ4 over zstd is that LZ4 decompression speed >4 GB / s (single-core), far exceeding the read bandwidth of NVMe SSDs (which can be 3 to 7 GB / s), thus decompression does not pose a bottleneck; zstd has a higher compression ratio but a slower decompression speed, which is disadvantageous in callback latency-sensitive scenarios.

[0192] In this embodiment of the disclosure, the compression ratio of the value data block is determined by the entropy feature that characterizes the numerical repeatability of the value vector in the value data block. This can achieve an optimal balance between the precision of the compressed data and the storage space occupied, thereby improving the effectiveness of compression.

[0193] In some embodiments, the first storage space includes the video memory of the graphics processor, and the third storage space includes the video memory of the central processing unit; step 105 above can unload the value cache to the video memory of the central processing unit through the following steps G1 to G2: Step G1: Perform word-by-word meta-quantization on the value cache to obtain the quantized value cache; the data precision of the value cache is the first data precision, and the data precision of the quantized value cache is the fifth data precision; the first data precision is higher than the fifth data precision.

[0194] Here, step G1 corresponds to the aforementioned step F1. When implementing it, the implementation method of the aforementioned step F1 can be referred to. This embodiment of the present disclosure will not repeat the details.

[0195] In some implementations, the value cache is grouped along the lexical dimension to obtain multiple groups of value caches; the quantization parameters of each group of value caches are determined; and each group of value caches is quantized based on the quantization parameters of each group of value caches to obtain quantized value caches.

[0196] Step G2: Write the quantized value cache from the graphics processor's video memory to the central processing unit's video memory to unload the value cache.

[0197] In some implementations, a data transfer channel between the graphics processor and the central processing unit is used to cache the quantized value from the graphics processor's video memory and write it to the central processing unit's video memory.

[0198] like Figure 2 As shown, per-token quantization can be performed on V_Cache, and the quantized K_Cache can be written from CPUHBM to CPU DRAM.

[0199] In this embodiment of the disclosure, the value cache is unloaded by performing word-by-word meta-quantization on the value cache and writing the quantized value cache from the graphics processor's video memory to the central processing unit's video memory. This unloading of the value cache to the central processing unit's video memory is faster than unloading it to the hard disk, and the subsequent recovery speed is also faster.

[0200] In some embodiments, the first storage space includes the video memory of the graphics processor, and the third storage space includes the hard disk; step 105 above can also be performed by unloading the value cache to the hard disk of the central processing unit through the following steps H1 to H4: Step H1: When the memory usage of the graphics processor is greater than or equal to the first memory usage threshold, perform word-by-word quantization on the value cache to obtain the quantized value cache; the data precision of the value cache is the first data precision, and the data precision of the quantized value cache is the fifth data precision; the first data precision is higher than the fifth data precision.

[0201] In some implementations, the implementation of "performing word-by-word meta-quantization on the value cache to obtain the quantized value cache" in step H1 is similar to the implementation of the aforementioned step F1 or step G1. When implementing, the implementation of the aforementioned step F1 can be referred to, and this disclosure embodiment will not elaborate further.

[0202] Step H2: The quantized value is cached and written from the graphics processor's video memory to the central processing unit's video memory.

[0203] Here, step H2 corresponds to the aforementioned step G2. When implementing it, the implementation method of the aforementioned step G2 can be referred to. This embodiment of the present disclosure will not repeat the details.

[0204] Step H3: When the utilization rate of the CPU's video memory is greater than or equal to the second utilization rate threshold, the quantized value cache is compressed in blocks to obtain a compressed value cache.

[0205] In some implementations, the "block compression of the quantized value cache to obtain a compressed value cache" in step H3 is similar to the implementation of step F2 above. When implementing, the implementation of step F2 above can be referred to. This disclosure will not elaborate further on this aspect.

[0206] Step H4: Write the compressed value cache from the CPU's video memory to the hard disk to unload the value cache.

[0207] In some implementations, the data transfer channel between the central processing unit and the hard disk can be used to write the compressed value cache from the central processing unit's video memory to the hard disk, thereby unloading the value cache.

[0208] like Figure 2 As shown, V_Cache is first quantized per-token, and the quantized V_Cache is written from CPUHBM to CPU DRAM. Then, the quantized V_Cache is compressed into blocks, and the compressed V_Cache is written from CPU HBM to NVMeSSD.

[0209] In this embodiment of the disclosure, when the utilization rate of the graphics processor's video memory is greater than or equal to a first utilization rate threshold, word-by-word quantization is performed on the value cache, and the quantized value cache is written from the graphics processor's video memory to the central processing unit's video memory. When the utilization rate of the central processing unit's video memory is greater than or equal to a second utilization rate threshold, block compression is further performed on the quantized value cache, and the compressed value cache is written from the central processing unit's video memory to the hard disk. In this way, by adopting a layer-by-layer offloading method, the node load can be reduced layer by layer, the offloading flexibility can be improved, and the read and write requirements of large models with high throughput, low latency, and massive KVCache can be met.

[0210] In some embodiments, the key-value cache unloading method provided in this disclosure, after step F2, further includes the following step F4: Step F4: If the size of the compressed value cache is greater than the preset cache size, the quantized value cache is written from the graphics processor's video memory to the hard disk to unload the value cache.

[0211] The preset cache size is determined based on the value cache. For example, the preset cache size can be determined based on the value cache and the tolerance coefficient. The formula for calculating the preset cache size can be expressed as: the size of the compressed value cache (such as the size of the value cache after LZ4 compression) > the tolerance coefficient α × the size of the original value cache (such as the size of the original value cache in the INT4 state) (α=0.9).

[0212] In some implementations, if the size of the compressed value cache is larger than the preset cache size, compression is abandoned and the value is directly stored in INT4 to avoid the ineffective compression overhead on high-entropy blocks.

[0213] For example, when high-entropy data results in an extremely low LZ4 compression rate (i.e., the compressed volume exceeds 90% or even expands), the Bypass mechanism is triggered, abandoning LZ4 compression and directly storing INT4 data to avoid unnecessary CPU / DPU decompression overhead.

[0214] In this embodiment of the disclosure, the compression of the value cache is judged by a preset cache size. If the size of the compressed value cache is greater than the preset cache size, the compression is abandoned and INT4 is stored directly to avoid the invalid compression overhead on high-entropy blocks.

[0215] In some embodiments, the metadata of the key-value cache block records the storage location, quantization method, and compression method of the key cache and the value cache to be unloaded, respectively; the key cache and the value cache are stored in different storage spaces.

[0216] For example, each unloaded KV block retains one metadata record in memory: BlockMeta { block_id: uint64, / / Identifier of the cache block k_location: enum { HBM, DRAM, SSD}, / / Location of k cache blocks; v_location: enum { HBM, DRAM, SSD}, / / Location of v cache blocks; k_quant: enum { FP16, FP8_PerChannel}, / / Quantization of k cache blocks; v_quant: enum { FP16, INT4_PerToken}, / / Quantization of the v cache block; k_compression: enum { None, Delta}, / / Compression of k cache blocks; v_compression: enum { None, LZ4}, / / Compression of v cache blocks; k_anchor_refs: Vec<block_id> , / / Anchor reference for residual coding; byte_offset_k: uint64, / / Offset of the storage layer of the k cache block; byte_offset_v: uint64, / / Offset of the storage layer of the v cache block; last_access_time: uint64 }

[0217] Total metadata overhead: approximately 128 bytes per block. 1 million tokens, calculated at a block size of 16 tokens, = 62,500 blocks → total metadata of approximately 8MB, negligible.

[0218] In some embodiments, such as Figure 5 As shown, the key-value cache unloading method provided in this embodiment of the disclosure further includes the following steps 501 and 503: Step 501: In response to a callback request for the key-value cache block to be called back, restore the key cache to be called back from the second storage space.

[0219] The key-value cache block to be called back refers to the key-value cache block currently being called back. The key-value cache block to be called back includes the key cache block and the value cache block to be called back.

[0220] When the inference engine needs to call back an unloaded key-value block (e.g., due to a prefix cache hit or long context backtracking), the recovery of K_cache and V_cache has different impacts on inference quality: delayed recovery of K_cache is extremely costly; attention calculation requires a precise dot product of Q × K^T, and insufficient K_cache precision directly leads to attention routing errors. Delayed recovery of V_cache is less costly: in the attention weighted calculation of softmax(QK^T)×V, the small error in V is scaled by the weights after softmax normalization, having a limited impact on the output. Therefore, this embodiment prioritizes the recovery of key cache blocks and asynchronously recovers value cache blocks.

[0221] It should be noted that restoring the key cache from the second storage space is the reverse of unloading the key cache to the second storage space.

[0222] In some implementations, if during unloading, the key cache is written from the graphics processor's video memory to the hard disk using a direct data path between the graphics processor and the hard disk, then during the callback, residual decoding can be performed on the key cache on the hard disk to obtain the decoded key cache; the decoded key cache can be dequantized to obtain the dequantized key cache (the key cache to be called back).

[0223] In some implementations, if the key cache is written from the graphics processor's video memory to the central processing unit's video memory during unloading, then during the callback, the key cache in the central processing unit's video memory is dequantized to obtain the dequantized key cache (the key cache to be called back).

[0224] In some implementations, if during unloading, the key cache is first written from the graphics processor's video memory to the central processing unit's video memory, and then written to the hard disk, then during the callback, residual decoding can be performed on the key cache on the hard disk to obtain the decoded key cache, and the decoded key cache can be transferred from the hard disk to the central processing unit's video memory; in the central processing unit's video memory, the decoded key cache is dequantized to obtain the dequantized key cache (the key cache to be called back), and the dequantized key cache can be transferred from the central processing unit's video memory to the graphics processor's video memory.

[0225] Step 502: During the process of restoring the key cache to be called back, the value cache to be called back is asynchronously restored from the third storage space.

[0226] It should be noted that restoring the value cache from the third storage space is the reverse of unloading the value cache to the second storage space.

[0227] In some implementations, if during unloading, a direct data path between the graphics processor and the hard disk is used to write the value cache from the graphics processor's video memory to the hard disk, then during the callback, the value cache in the hard disk can be decompressed in blocks to obtain the decompressed value cache; the decompressed value cache can be dequantized to obtain the dequantized value cache (the value cache to be called back).

[0228] In some implementations, if the value cache is written from the graphics processor's video memory to the central processing unit's video memory during unloading, then during the callback, the value cache in the central processing unit's video memory is dequantized to obtain the dequantized value cache (the value cache to be called back).

[0229] In some implementations, if during unloading, the value cache is first written from the graphics processor's video memory to the central processing unit's video memory, and then written to the hard disk, then during the callback, the value cache in the hard disk can be decompressed in blocks to obtain a decompressed value cache, and the decompressed value cache can be transferred from the hard disk to the central processing unit's video memory; in the central processing unit's video memory, the decompressed value cache is dequantized to obtain a dequantized value cache (the value cache to be called back), and the dequantized value cache can be transferred from the central processing unit's video memory to the graphics processor's video memory.

[0230] Step 503: Based on the key cache and value cache to be called back, determine the key-value cache block to be called back; the key-value cache block to be called back is used for context reasoning in the neural network model.

[0231] In this embodiment of the disclosure, based on the impact of restoring the key cache and value cache on inference quality, the key cache block is restored first, and the value cache block is restored asynchronously. This can improve the data accuracy of the callback key-value cache block and reduce the impact on inference quality.

[0232] In some embodiments, the second storage space includes the video memory of the central processing unit and / or the hard disk; the "recovering the value cache to be called back from the third storage space" in step 502 can be implemented through the following steps 5021 or 5022: Step 5021: If the value cache to be called back is stored in the video memory of the central processing unit, the value cache to be called back with the fifth data precision is transferred to the graphics processor to restore the value cache to be called back.

[0233] In some implementations, if the value to be called back is already cached in CPU DRAM (low latency), it is directly transferred to the GPU in INT4 state and used for computation (with acceptable error).

[0234] Step 5022: When the value cache to be called back is stored on the hard disk and the key cache to be called back is restored from the second storage space, the value portion in the key value cache block to be called back is determined as the mean vector of the value cache to be called back. When the value cache to be called back is restored, the value portion in the key value cache block to be called back is replaced with the value cache to be called back.

[0235] In some implementations, if the value to be called back is cached in SSD (high latency), the mean vector of the cached value to be called back is used as a temporary placeholder. That is, the temporary attention output result obtained by weighted summation (or matrix multiplication) of the attention distribution matrix and the mean vector of the cached value to be called back (V_cache) is used as a temporary placeholder, which is replaced after asynchronous recovery is completed.

[0236] In this embodiment, if the value to be called back is already cached in CPU DRAM, due to the low transmission latency between the CPU and GPU, the value to be called back in the INT4 state can be directly transferred to the GPU and participate in the calculation, provided the error is acceptable, thereby improving the callback rate of the value to be called back. If the value to be called back is cached in SSD, due to the high transmission latency between SSD and GPU, the mean vector of the value to be called back needs to be used as a temporary placeholder, which will be replaced after asynchronous recovery is completed. This shortens the callback latency from "waiting for all K+V to recover" to "waiting only for K to recover", reducing the GPU's waiting time, improving the callback rate of key-value blocks, and thus improving the model inference rate.

[0237] In some embodiments, such as Figure 6 As shown, the key-value cache unloading method provided in this embodiment further includes the following steps 601 to 602: Step 601: Detect the entropy change of the attention head of the neural network model.

[0238] In some implementations, when using the key-value cache block to be called back for model inference, the entropy change of the attention head of the neural network model can be detected to adjust the unloading parameters, thereby achieving an optimal balance between cache usage and unloading accuracy during the unloading process.

[0239] Step 602: If the inference accuracy after callback recovery is found to be abnormal based on the entropy change of the attention head, adjust the unloading parameters; the unloading parameters include at least one of the following: data accuracy of channel-wise quantization, data accuracy of residual coding, accuracy of word-wise meta-quantization, and accuracy of block compression.

[0240] In some implementations, a baseline for the entropy change of the attention head of the neural network model can be obtained through experimentation; the entropy change of the attention head after the callback can be compared with the baseline to determine whether there is an anomaly in the inference accuracy after the callback is restored.

[0241] For example, if the difference between the entropy change of the attention head after the callback and the entropy change benchmark is less than a difference threshold, then the inference accuracy after the callback recovery is determined to be normal. If the difference between the entropy change of the attention head after the callback and the entropy change benchmark is greater than a difference threshold, then the inference accuracy after the callback recovery is determined to be abnormal.

[0242] In this embodiment of the disclosure, the change in the entropy value of the attention head is used to determine whether there is an anomaly in the inference accuracy after the callback is restored. If there is an anomaly, the unloading parameters are adjusted, which can form an adaptive closed loop between the unloading strategy and the accuracy feedback.

[0243] In some embodiments, such as Figure 7 As shown, the key-value cache unloading method provided in this embodiment of the disclosure further includes the following step 701: Step 701: Based on at least one of the system hardware topology, the current load of each computing unit, and the bandwidth constraints of the data transmission path, allocate the execution positions of the quantization operation, encoding operation, and compression operation in the key-value cache unloading method among the general computing core of the graphics processor, the coprocessor core of the data processing unit, and the central processing unit.

[0244] In some implementations, the encoding and compression operations can be determined, depending on the hardware configuration, whether they are performed on the GPU or the Data Processing Unit (DPU).

[0245] In some implementations, the workload for each scenario can be determined by three types of hardware configuration scenarios: Scenario A: Traditional GPU server (without DPU).

[0246] Hardware configuration: Server graphics card + standard x86 server + local NVMe SSD, no DPU. Workload allocation is shown in Table 5 below: Table 5

[0247] Scenario B: DPU-accelerated data center configuration.

[0248] Hardware configuration: Super chip + intelligent network card DPU + flash drive array + Ethernet switch. The workload is divided as shown in Table 6 below: Table 6

[0249] Scenario C: Edge / Consumer-grade configuration (single machine without DPU).

[0250] Hardware configuration: Graphics card + consumer-grade motherboard + consumer-grade NVMe SSD, GDS not supported. Work breakdown is shown in Table 7 below: Table 7

[0251] like Figure 8 As shown, step 801: Runtime hardware configuration detection. Step 802: Does a DPU (Data Processing Unit) exist? If a DPU exists, proceed to step 803; if a DPU does not exist, proceed to step 804. Step 803: LZ4 / Delta compression / decompression → DPU quantization → GPU DMA → GPU → DPU → SSD. Step 804: Does GPU direct-attached storage support work? If GPU direct-attached storage is supported, proceed to step 805; if not, proceed to step 806. Step 805: Quantization + Delta + LZ4 → GPU DMA → GPU → SSD direct transfer. Step 806: Quantization + Delta → GPU LZ4→CPU DMA→GPU→CPU→SSD.

[0252] In some embodiments, the key-value cache unloading method provided in this disclosure is embedded as middleware between the key-value cache management module and the transport layer.

[0253] The key-value cache management module, also known as the key-value cache management layer (KV Cache management layer), refers to the software module in the large language model inference engine that is specifically responsible for the lifecycle management of the KV Cache, located between the "Inference Scheduler" and the "Underlying Storage / Transport".

[0254] Taking the vLLM PagedAttention architecture as an example, the responsibilities and layers of the KV Cache management layer are explained. Figure 9As shown, step 901, the inference scheduler determines which request / decode step to schedule; step 902, KV Cache management layer: 1) Block allocation / reclamation (Block Allocator); 2) Block Table (Logical → Physical); 3) Prefix sharing and reference counting (Prefix Cache + RefCnt); 4) Least Recently Used / Adaptive Replacement Cache + Eviction Policy (LRU / ARC + Eviction Policy); 5) Quantization and compression (new in this embodiment: KV Differentiated Quantization + Compression); 6) Resumption scheduling (new); step 903, Transport layer; step 904, Multi-layer physical storage GPU HBM → CPU DRAM → NVMe SSD → Remote object.

[0255] Table 8 below illustrates the key structural features of the KV Cache management layer: Table 8

[0256] The reasons for embedding the key-value cache unloading method as middleware between the key-value cache management module and the transport layer are as follows: 1. Semantic integrity: This layer has a complete view of the "complete lifecycle of a KV block" (allocation → use → eviction → unloading → callback → destruction), making it the only suitable location to implement "splitting during unloading + priority restoration during callback"; 2. Stable interface: Modern inference engines (vLLM inference engine internal plug-in abstract interface (vLLM Connector API) and KV block manager hook (KVBM hook) have provided pluggable hooks, requiring no intrusive modifications; 3. Decoupling from the underlying layer: This layer only determines the strategy of "what precision, what compression, and where to put it", while the specific DMA / RDMA / GDS transmission is still completed by the transport layer - this ensures that the embodiments disclosed can seamlessly reuse existing infrastructure such as NIXL and GDS (corresponding to middleware layer characteristics).

[0257] The following describes the application of the key-value cache unloading method provided in this embodiment in a real-world scenario.

[0258] In the multi-tier storage architecture (GPU HBM → CPU DRAM → NVMe SSD) of the LLM inference engine, when KVCache is offloaded to lower-tier storage due to insufficient capacity, the existing solution uses a unified approach for K_cache and V_cache, leading to two mutual exclusion issues: 1. If high precision (such as FP16) is used uniformly for offloading: storage space and transmission bandwidth are wasted significantly—V_cache does not need FP16 precision at all; 2. If low precision (such as INT4) is used for unloading: K_cache precision is severely damaged - attention quality degrades after long context callbacks, resulting in output errors.

[0259] The technical problems to be solved by the embodiments of this disclosure include at least the following: 1. How to leverage K / V asymmetric sensitivity in multi-level storage offloading to adopt differentiated quantization-compression-storage strategies for K_cache and V_cache, achieving the optimal balance between storage efficiency and inference accuracy; 2. How to design a priority mechanism for callback recovery so that K_cache recovers precision first and V_cache recovers with a delay, thereby minimizing callback latency; 3. How to seamlessly integrate the precision-aware strategy with the existing offloading infrastructure (vLLM Connector API, KV block management middleware, hardware driver capabilities for hostless DRAM KV cache offloading schemes for large model long context reasoning).

[0260] The technical solutions of this disclosure include: 1.1 System Overall Structure.

[0261] like Figure 2 As shown in the embodiments of this disclosure, a precision-aware KV Cache hierarchical offload-recovery scheduling system is proposed, which is embedded in the KV Cache management layer of the inference engine. The core idea is: K_cache maintains precision, V_cache compresses space; offload is divided into strategies, and callbacks are given priority.

[0262] 1.2 Differentiated Quantization-Compression Strategy for K / V Separation.

[0263] 1.2.1 K / V quantization scheme for three-layer storage.

[0264] As shown in Table 9 below, the specific K / V quantization schemes for three-layer storage include: Table 9

[0265] Basis for selecting quantification method: K-cache per-channel quantization: Research shows that K-cache quantization along the channel dimension can preserve the accuracy of attention weight calculation, while per-token quantization leads to the loss of cross-channel information. V_cache per-token quantization: V_cache participates in the weighted summation of attention (rather than dot product matching), and per-token quantization does not affect the summation accuracy; research shows that 75% of the layers can be reduced to 1 bit.

[0266] 1.2.2 K_cache Delta encoding scheme.

[0267] Traditional storage: Each token's K vector is stored independently (occupying full space).

[0268] Delta encoding: anchor_k[0] = K[0] / / The first token is stored completely; delta_k[i] = K[i] - anchor_k[nearest(i)] / / Subsequent tokens store the residual between the current token and the nearest anchor; Anchor selection: Select one anchor for every B tokens (B=32, configurable); Residual characteristics: Studies have shown that residual values ​​are concentrated around 0, which can be further stored with low precision.

[0269] Recommended preferred option: Anchor 8 bits + residual 6 bits (per-block scale, block = 32 tokens); Storage overhead for every 32 tokens: 1 × 128 × 8 + 31 × 128 × 6 + 32 × 16 = 2128 + 2976 + 512 = 5616 bits; Compared to the traditional FP8: 32 × 128 × 8 = 32768 bits; Compression ratio: 5616 / 32768 ≈ 17.1% (compression ratio is about 8.6% relative to the FP16 baseline); Compression effect: Studies have shown that K_cache can be compressed to 29% of the original (FP16 baseline); the embodiments disclosed herein perform delta on FP8, and are expected to further compress by 40 to 60% (FP8 delta value is smaller than FP16 delta value range); finally, K_cache storage on SSD is approximately 15 to 20% of the original FP16.

[0270] 1.2.3 V_cache block compression scheme.

[0271] V_cache INT4 data is organized in fixed-size blocks (Block Size = 4 KB).

[0272] The block size setting takes into account the following: (1) A 4KB block in the INT4 data format can hold 8192 INT4 values, corresponding to 64 tokens of V data when head_dim=128, which is well aligned with the block granularity of PagedAttention (16 to 32 tokens); (2) 4KB matches the physical page size of NVMe SSD (common 4K / 8K), which can be completed in one I / O; (3) LZ4 achieves the best balance between compression-decompression throughput and compression rate on 4KB blocks (the compression rate gain converges when it is greater than 4KB, and the header overhead is too high when it is less than 4KB).

[0273] Each block performs LZ4 lossless compression independently.

[0274] The compression ratio depends on the repeatability of the numerical distribution after V_cache quantization (negatively correlated with Shannon entropy). This embodiment of the disclosure measures the actual LZ4 compression ratio of each layer and each type of token location using V_cache data from a representative workload through an offline calibration process before deployment, generating a per-layer compression ratio prediction table for runtime storage budget planning. Actual measurements show that the compression ratio is typically 2X to 3X near the attention sink and in the padding region due to the concentrated numerical distribution; the compression ratio in the regular token region is 1.3X to 1.8X; and the overall average compression ratio is approximately 1.5X to 2X.

[0275] Low-entropy layer (near the attention sink token): compression ratio 2x to 3x; High-entropy layer (randomly distributed intermediate tokens): compression ratio 1.2x to 1.5x; Average expected value: 1.5x to 2x.

[0276] Reasons for choosing LZ4 over zstd: LZ4 decompression speed >4 GB / s (single core), which is much higher than the read bandwidth of NVMe SSD (usually 3 GB / s to 7 GB / s), so decompression is not a bottleneck; zstd has a higher compression ratio but a slower decompression speed, which is disadvantageous in scenarios sensitive to callback latency.

[0277] 1.3 Precision-aware offloading scheduling strategy.

[0278] 1.3.1 Uninstallation trigger conditions.

[0279] Unloading is triggered when the KV Cache in the GPU HBM exceeds the high watermark W_high (default 85%).

[0280] 1.3.2 Uninstallation execution process.

[0281] like Figure 10 As shown, the unloading execution process includes: Step 1001, HBM occupancy > W_high. Step 1002, Select the KV block to be unloaded (LRU / ARC strategy remains unchanged). Step 1003, Split the block into K-part and V-part. Step 1004, K-part: FP16→FP8 per-channel quantization. Step 1005, K-part is written to the target storage layer. Step 1006, V-part: FP16→INT4 per-token quantization. Step 1007, V-part is written to the target storage layer. Step 1008, Update metadata: Record the storage location of each part + quantization scheme + compression method. It should be noted that steps 1004 to 1005 and steps 1006 to 1007 can be executed synchronously.

[0282] Key Design: 1. Split rather than unload the whole: Split each KV block into independent K-part and V-part, allowing them to use different quantization strategies or even be stored in different layers.

[0283] 2. Inter-layer degradation: When the CPU DRAM is also full, K-part is written to the SSD after delta encoding from FP8, and V-part is written to the SSD after LZ4 block compression from INT4.

[0284] 3. Asynchronous Pipeline: Quantization and transport are performed concurrently with inference computation (reusing the asynchronous framework of vLLM Connector).

[0285] 1.3.3 Integration with GPU Direct Storage.

[0286] For SSD layer offloading, the HiFC path (GPU → GDS → pSLC NVMe) is prioritized to bypass the CPU bottleneck. Quantization is completed on the GPU side and then directly written to the SSD via DMA. GPU side: FP16 K_cache → FP8 quantized kernel → GDS DMA → pSLC NVMe; GPU side: FP16 V_cache → INT4 quantization kernel → GDS DMA → pSLC NVMe; Delta encoding and LZ4 compression can be performed on either the GPU or the DPU, depending on the hardware configuration.

[0287] 1.4 Priority recovery callback scheduling strategy.

[0288] 1.4.1 K_cache priority recovery principle.

[0289] When the inference engine needs to call back an unloaded key-value block (e.g., prefix cache hit, long context backtracking), the recovery of K_cache and V_cache has different impacts on inference quality: 1) The cost of K_cache delay recovery is extremely high: attention calculation requires an accurate dot product of Q × K^T, and insufficient K_cache precision directly leads to attention routing errors.

[0290] 2) The cost of V_cache delay recovery is low: In the attention weighted sum softmax(QK^T) × V, the small error of V is scaled by the weights after softmax normalization, and has a limited impact on the output.

[0291] 1.4.2 Two-stage callback process.

[0292] like Figure 11 As shown, Step 1101: Callback request: Block B needs to be restored. Step 1102: Stage 1: K-cache restoration prioritized. Step 1103: Read K-part from storage layer (SSD: delta decoding + FP8 dequantization) (DRAM: FP8 dequantization). Step 1104: K-cache restored to GPU HBM precision: FP8→FP16. Step 1105: Inference can begin (using restored K-cache + temporary low-precision V approximation). Step 1106: Stage 2: V-cache delayed restoration. Step 1107: Background asynchronous reading of V-part (SSD: LZ4 decompression + INT4 dequantization) (DRAM: INT4 dequantization). Step 1108: V-cache restored to GPU HBM precision: INT4→FP16. Step 1109: Complete precision inference.

[0293] Phase 1 "Temporary low-precision V approximation": 1) If V_cache is already in CPU DRAM (low latency), it is directly transferred to the GPU in INT4 state and used for calculation (the error is acceptable). 2) If V_cache is on SSD (high latency), use the mean vector of V_cache as a temporary placeholder (attention × mean_V), and replace it after asynchronous recovery is complete; 3) This design shortens the callback delay from "waiting for K+V to fully recover" to "waiting only for K to recover".

[0294] 1.4.3 Optimization of callback delay.

[0295] The performance of the callback optimization is shown in Table 10 below.

[0296] Table 10

[0297] 1.5 Metadata Management.

[0298] Each unloaded KV block retains one metadata record in memory: BlockMeta { block_id: uint64, / / Identifier of the cache block k_location: enum { HBM, DRAM, SSD}, / / Location of k cache blocks; v_location: enum { HBM, DRAM, SSD}, / / Location of v cache blocks; k_quant: enum { FP16, FP8_PerChannel}, / / Quantization of k cache blocks; v_quant: enum { FP16, INT4_PerToken}, / / Quantization of the v cache block; k_compression: enum { None, Delta}, / / Compression of k cache blocks; v_compression: enum { None, LZ4}, / / Compression of v cache blocks; k_anchor_refs: Vec<block_id> , / / Anchor reference for residual coding; byte_offset_k: uint64, / / Offset of the storage layer of the k cache block; byte_offset_v: uint64, / / Offset of the storage layer of the v cache block; last_access_time: uint64, }

[0299] Total metadata overhead: approximately 128 bytes per block. 1 million tokens, calculated at a block size of 16 tokens = 62,500 blocks → total metadata of approximately 8MB, negligible.

[0300] 4.6 Runtime accuracy feedback (optional enhancement, linkage direction A).

[0301] When the system deploys an Attention entropy sentinel in direction A, a closed-loop feedback can be formed: 1) The entropy sentinel detected an abnormally flat attention distribution → This may be due to insufficient K_cache precision after callback recovery; 2) Feedback to the unloading scheduler → Upgrade the storage strategy for the K_cache of the high-sensitivity layer (e.g., change from FP8 to FP16 unloading, or shorten the unloading time window). 3) Form an adaptive closed loop of "unloading strategy ← accuracy feedback".

[0302] 2.1 Comparison with existing technologies.

[0303] As shown in Table 11 below, this solution is compared with existing technologies in terms of multi-layer storage, key-value separation, precision-aware offloading, Delta encoding, priority recovery, and GDS direct transmission: Table 11

[0304] 2.2 Summary of core innovations.

[0305] 1) This invention is the first to apply K / V asymmetric sensitivity to multi-level memory offloading scheduling: KVTuner, AsymKV, KIVI, etc. have fully demonstrated the difference in K / V sensitivity, but they have only been used for quantization within the GPU. This invention is the first to transfer this finding to the memory scheduling domain of "where to offload, with what precision to store, and in what order to restore".

[0306] 2) K / V Split Storage and Differentiated Quantization-Compression Strategy: Traditional solutions offload KV blocks as a whole. This disclosure splits each block into independent K-parts and V-parts, allowing them to use different quantization schemes (K: FP8 per-channel; V: INT4 per-token) and different compression schemes (K: delta encoding; V: LZ4 block compression), and even store them in different storage layers.

[0307] 3) K_cache priority two-stage callback mechanism: During the callback, K_cache is restored first and V_cache is restored asynchronously with a delay. The inference engine can start calculation immediately after K_cache is ready (using temporary low-precision V approximation), shortening TTFT from "waiting for K+V" to "waiting for K only".

[0308] 4) Delta encoding extended from GPU in-memory compression to SSD storage layer: DeltaKV implements delta compression in GPU memory. This embodiment dedicates delta encoding to the SSD persistent storage of K_cache, further compressing it by 40% to 60% on top of FP8. The storage of K_cache on SSD is only 15% to 20% of the original FP16.

[0309] Seamless integration with the existing ecosystem: This disclosure embodiment is inserted as an intermediate layer between vLLMConnector and KV Manager as a "precision-aware scheduling layer", reusing existing infrastructure such as asynchronous transmission, GDS direct transmission, and compressed prefix tree (Radixtree) index, reducing implementation costs.

[0310] 3.1 Example 1: Enhanced Generation of Long Context Retrieval Augmented Generation (RAG) scenario.

[0311] Scenario: An enterprise RAG system processes 500 pages of legal documents (approximately 200k tokens), using Llama-3.1-70B inference, with a single GPU memory capacity of 80 GB.

[0312] KV Cache requirement: 200k tokens × 0.312 MB / token ≈ 62.4 GB, exceeding the available capacity (approximately 40 GB after deducting model weights).

[0313] The operation flow of this embodiment of the disclosure is as follows: 1. The first 128k tokens of the KV cache are stored in the GPU HBM (approximately 40 GB). 2. Unloading is triggered during subsequent token processing: The oldest block (LRU) is selected for unloading; K-part: FP16 → FP8 per-channel quantization → written to CPU DRAM (approximately 50% of the original); V-part: FP16 → INT4 per-token quantization → written to CPU DRAM (approximately 25% of the original); 3. Total KV Cache usage in CPU DRAM: 62.4 GB × (50%+25%) / 2 ≈ 23.4 GB (compared to 62.4 GB offloaded by traditional FP16) 4. When a user queries the content at the beginning of a referenced document, the oldest block is called back: Phase 1: K-part is restored from DRAM to GPU (FP8 → FP16), with a latency of approximately 2ms; Inference begins immediately (using direct transfers from INT4 V_cache in DRAM to participate in computation); Phase 2: V-part background resumes to FP16 (asynchronous, approximately 1ms).

[0314] As shown in Table 12 below, the scheme of this embodiment is compared with the traditional FP16 offloading scheme based on multiple indicators such as CPU DRAM usage, callback TTFT, and inference accuracy: Table 12

[0315] 3.2 Example 2: Multi-round Agent Dialogue with One Million Tokens

[0316] Scenario: The coding agent processes a 128k token code repository and accumulates 900k tokens through multiple rounds of dialogue. It uses a single AI accelerator card with 192GB HBM3e high-bandwidth video memory and a 4TB enterprise-grade NVMe solid-state drive.

[0317] The operation flow of this embodiment of the disclosure is as follows: 1. The GPU HBM can hold approximately 600k tokens in a KV cache.

[0318] 2. The excess portion (300k tokens) is directly transferred to the NVMe SSD via GDS: K-part: FP16 → FP8 per-channel → Delta encoding (anchor spacing B=32) → pSLC region; V-part: FP16 → INT4 per-token → LZ4 block compression → pSLC region.

[0319] 3. K_cache storage on SSD: 300k × 0.156 MB (FP8) × 0.4 (delta compression) ≈ 18.7GB.

[0320] 4. V_cache storage on SSD: 300k × 0.078 MB (INT4) × 0.6 (LZ4 compression) ≈ 14.0GB.

[0321] 5. Total SSD usage: 32.7 GB (65% saving compared to 93.6 GB for traditional FP16).

[0322] Callback performance (NVMe read bandwidth 6.5 GB / s) is shown in Table 13: Table 13

[0323] 3.3 Example 3: Integration with hardware and software combined system.

[0324] Scenario: Factory rack cluster, KVBM manages four layers of storage from G1 to G4.

[0325] Integration method: This disclosed embodiment serves as a "precision-aware scheduling plugin" for KVBM, inserted between the middle layer (core logic layer) and the bottom layer (NIXL transport layer) of KVBM.

[0326] KVBM determines "which block is offloaded to which layer" (the capacity / heat strategy remains unchanged).

[0327] This disclosure specifies how to quantize and compress the K-part and V-part (precision strategy).

[0328] NIXL is responsible for the actual transmission (P2P GPU, RDMA, GDS paths remain unchanged).

[0329] 4.1 Summary of expected results.

[0330] As shown in Table 14 below, the improvement of the embodiments of this disclosure is illustrated from multiple indicators such as storage space, callback TTFT, callback I / O, bandwidth, and inference accuracy: Table 14

[0331] 4.2 Complementarity with offline quantization schemes.

[0332] Table 15 below illustrates the scheme, usage stages, and functions of the embodiments of this disclosure: Table 15

[0333] 5.1 K_cache quantization alternative.

[0334] Table 16 below lists the quantization alternatives for K_cache and analyzes their advantages and disadvantages: Table 16

[0335] 5.2 Delta encoding alternatives.

[0336] Table 17 below lists the encoding alternatives for K_cache and analyzes their advantages and disadvantages: Table 17

[0337] 5.3 Alternatives to the callback strategy.

[0338] Table 18 below lists alternative callback strategies and analyzes their advantages and disadvantages: Table 18

[0339] It should be noted that the hardware models and names in the embodiments of this disclosure are merely illustrative examples. In practical applications, those skilled in the art can replace them with other types of architectures with equivalent computing power or models of the same level.

[0340] Based on the foregoing embodiments, this disclosure provides a key-value cache unloading device, which includes the included units and the modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0341] Figure 12 This is a schematic diagram of the composition structure of a key-value cache unloading device provided in an embodiment of this disclosure, as shown below. Figure 12 As shown, the data processing device 1200 includes: a splitting module 1210, a processing module 1220, and an unloading module 1230, wherein: Splitting module 1210 is configured to split the key-value cache block into a key cache and a value cache; The processing module 1220 is configured to determine a first unloading strategy for the key cache based on the sensitivity of the sample key cache to quantization error; the first unloading strategy includes the quantization method and / or compression method of the key cache to be unloaded. The processing module 1220 is also configured to determine a second unloading strategy for the value cache based on the sensitivity of the sample value cache to quantization error; the second unloading strategy includes the quantization method and / or compression method of the value cache to be unloaded. Unloading module 1230 is configured to use the first unloading strategy to unload the key cache from the first storage space to the second storage space; The unloading module 1230 is also configured to use a second unloading strategy to unload the value cache from the first storage space to the third storage space.

[0342] In some embodiments, the processing module 1220 is further configured to: determine the sensitivity of the sample key cache to quantization accuracy based on at least one of the following: the role of the sample key cache in the attention module, the quantization error propagation path, the tolerable accuracy, and the distribution of outliers; determine the sensitivity of the sample key cache to quantization granularity based on the distribution of outliers in the sample key cache; and determine the sensitivity of the sample key cache to quantization error based on the sensitivity of the sample key cache to quantization accuracy and the sensitivity of the sample key cache to quantization granularity.

[0343] In some embodiments, the processing module 1220 is further configured to perform at least one of the following: when the outliers in the sample key cache, which are characterized by their sensitivity to quantization errors, are concentrated in the channel dimension, determine that the quantization method of the key cache is channel-wise quantization; when the sample key cache, which is characterized by their sensitivity to quantization errors, satisfies the residual coding condition, determine that the compression method of the key cache is residual coding.

[0344] In some embodiments, the processing module 1220 is further configured to perform at least one of the following: when the sensitivity of the sample value cache to quantization error indicates that the values ​​of the sample value cache are uniformly distributed in the channel dimension and the word dimension, determine that the quantization method of the value cache is word-by-word quantization; when the sensitivity of the sample value cache to quantization error indicates that the sample value cache meets the block compression condition, determine that the compression method of the value cache is block encoding.

[0345] In some embodiments, the first storage space includes the video memory of the graphics processor, and the second storage space includes a hard disk; the offloading module 1230 is further configured to: perform channel-by-channel quantization on the key cache to obtain a quantized key cache; the data precision of the key cache is a first data precision, and the data precision of the quantized key cache is a second data precision; the first data precision is higher than the second data precision; perform residual encoding on the quantized key cache to obtain an encoded key cache; and use the direct data path between the graphics processor and the hard disk to write the encoded key cache from the video memory of the graphics processor to the hard disk to realize the offloading of the key cache; the first offloading strategy includes channel-by-channel quantization and residual encoding.

[0346] In some embodiments, the unloading module 1230 is further configured to: determine the first term in the key cache; for other terms in the key cache besides the first term, determine the residual between the other terms and the corresponding anchor point; select a representative term as the anchor point for each target number of terms; and determine the encoded key cache based on the first term, the residual between the other terms and the corresponding anchor point.

[0347] In some embodiments, the first storage space includes the video memory of the graphics processor, and the second storage space includes the video memory of the central processing unit; the unloading module 1230 is further configured to: perform channel-by-channel quantization on the key cache to obtain a quantized key cache; the data precision of the key cache is a first data precision, and the data precision of the quantized key cache is a second data precision; the first data precision is higher than the second data precision; and write the quantized key cache from the video memory of the graphics processor to the video memory of the central processing unit to realize the unloading of the key cache.

[0348] In some embodiments, the first storage space includes the video memory of the graphics processor, and the second storage space includes a hard disk; the unloading module 1230 is further configured to: when the occupancy rate of the graphics processor's video memory is greater than or equal to a first occupancy rate threshold, perform channel-by-channel quantization on the key cache to obtain a quantized key cache; the data precision of the key cache is a first data precision, and the data precision of the quantized key cache is a second data precision; the first data precision is higher than the second data precision; write the quantized key cache from the graphics processor's video memory to the central processing unit's video memory; when the occupancy rate of the central processing unit's video memory is greater than or equal to the second occupancy rate threshold, perform residual encoding on the quantized key cache to obtain an encoded key cache; write the encoded key cache from the central processing unit's video memory to the hard disk to realize the unloading of the key cache.

[0349] In some embodiments, the unloading module 1230 is further configured to: reduce the data precision of the residuals between other words and corresponding anchors to a third data precision when the residuals between other words and corresponding anchors are within a preset residual range; the preset residual range is determined based on a second data precision; replace the data precision of the residuals between other words and corresponding anchors with a preset value when the absolute value of the residuals between other words and corresponding anchors is less than a preset residual threshold; reduce the data precision of the residuals between other words and corresponding anchors to a fourth data precision when the absolute value of the standard deviation of the residuals between other words and corresponding anchors is greater than a preset residual threshold; the third data precision is higher than the fourth data precision.

[0350] In some embodiments, the first storage space includes the video memory of the graphics processor, and the third storage space includes the hard disk; the unloading module 1230 is further configured to: perform word-by-word meta-quantization on the value cache to obtain a quantized value cache; the data precision of the value cache is a first data precision, and the data precision of the quantized value cache is a fifth data precision; the first data precision is higher than the fifth data precision; perform block compression on the quantized value cache to obtain a compressed value cache; and use the direct data path between the graphics processor and the hard disk to write the compressed value cache from the video memory of the graphics processor to the hard disk to realize the unloading of the value cache.

[0351] In some embodiments, the unloading module 1230 is further configured to: store the compressed value cache as multiple value data blocks according to a preset block size; the preset block size is determined based on a compression ratio prediction table; determine the compression ratio of the value data block from the compression ratio prediction table based on the entropy characteristics of the value vectors in the value data block; the entropy characteristics of the value vectors in the value data block are used to characterize the numerical repeatability of the value vectors in the value data block; and compress the value data block according to the compression ratio of the value data block to obtain a compressed value cache.

[0352] In some embodiments, the first storage space includes the video memory of the graphics processor, and the third storage space includes the video memory of the central processing unit; the unloading module 1230 is further configured to: perform word-by-word meta-quantization on the value cache to obtain a quantized value cache; the data precision of the value cache is a first data precision, and the data precision of the quantized value cache is a fifth data precision; the first data precision is higher than the fifth data precision; and write the quantized value cache from the video memory of the graphics processor to the video memory of the central processing unit to realize the unloading of the value cache.

[0353] In some embodiments, the first storage space includes the video memory of the graphics processor, and the third storage space includes a hard disk; the unloading module 1230 is further configured to: when the occupancy rate of the video memory of the graphics processor is greater than or equal to a first occupancy rate threshold, perform word-by-word quantization on the value cache to obtain a quantized value cache; the data precision of the value cache is a first data precision, and the data precision of the quantized value cache is a fifth data precision; the first data precision is higher than the fifth data precision; write the quantized value cache from the video memory of the graphics processor to the video memory of the central processing unit; when the occupancy rate of the video memory of the central processing unit is greater than or equal to a second occupancy rate threshold, perform block compression on the quantized value cache to obtain a compressed value cache; write the compressed value cache from the video memory of the central processing unit to the hard disk to realize the unloading of the value cache.

[0354] In some embodiments, the unloading module 1230 is further configured to: write the quantized value cache from the graphics processor's video memory to the hard disk when the size of the compressed value cache is greater than the preset cache size, so as to unload the value cache.

[0355] In some embodiments, the apparatus further includes: a callback module configured to, in response to a callback request for a key-value cache block to be called back, restore the key cache to be called back from a second storage space; asynchronously restore the value cache to be called back from a third storage space during the process of restoring the key cache to be called back; determine the key-value cache block to be called back based on the key cache to be called back and the value cache to be called back; the key-value cache block to be called back is used by a neural network model for contextual reasoning.

[0356] In some embodiments, the callback module is further configured to perform any of the following: if the value cache to be called back is stored in the video memory of the central processing unit, transfer the value cache to be called back with a fifth data precision to the graphics processor to restore the value cache to be called back; if the value cache to be called back is stored in the hard disk and the key cache to be called back is being restored from the second storage space, determine the value portion in the key value cache block to be called back as the mean vector of the value cache to be called back; and if the value cache to be called back is restored, replace the value portion in the key value cache block to be called back with the value cache to be called back.

[0357] In some embodiments, the processing module 1220 is further configured to: detect the entropy change of the attention head of the neural network model; and adjust the unloading parameters if it is determined that there is an abnormality in the inference accuracy after callback recovery based on the entropy change of the attention head; the unloading parameters include at least one of the following: data accuracy of channel-wise quantization, data accuracy of residual coding, accuracy of word-wise meta-quantization, and accuracy of block compression.

[0358] In some embodiments, the key-value cache unloading method is embedded as middleware between the key-value cache management module and the transport layer.

[0359] In some embodiments, the metadata of the key-value cache block records the storage location, quantization method, and compression method of the key cache and the value cache, respectively; the key cache and the value cache are stored in different storage spaces.

[0360] In some embodiments, the processing module 1220 is further configured to: allocate the execution positions of quantization, encoding, and compression operations in the key-value cache unloading method among the general-purpose computing core of the graphics processor, the coprocessor core of the data processing unit, and the central processing unit, based on at least one of the system hardware topology, the current load of each computing unit, and the bandwidth constraints of the data transmission path.

[0361] In some embodiments, the processing module 1220 is further configured to: restore the key cache from the second storage space as the opposite of unloading the key cache to the second storage space; and restore the value cache from the third storage space as the opposite of unloading the value cache to the second storage space.

[0362] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0363] It should be noted that, in the embodiments of this disclosure, if the above-described key-value cache unloading method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this disclosure, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this disclosure are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0364] This disclosure provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0365] This disclosure provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0366] This disclosure provides a computer program including computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0367] This disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0368] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referenced interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0369] It should be noted that, Figure 13 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this disclosure, such as... Figure 13 As shown, the hardware entity of the computer device 1300 includes: a processor 1301, a communication interface 1302, and a memory 1303, wherein: Processor 1301 typically controls the overall operation of computer device 1300.

[0370] Communication interface 1302 enables computer devices to communicate with other terminals or servers via a network.

[0371] The memory 1303 is configured to store instructions and applications executable by the processor 1301, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 1301 and various modules in the computer device 1300. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 1301, the communication interface 1302, and the memory 1303 can be performed via bus 1304.

[0372] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above embodiments of this disclosure are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0373] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0374] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0375] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0376] In addition, each functional unit in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0377] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0378] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0379] The above description is merely an embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A key-value cache offloading method, characterized in that, The method includes: Split the key-value cache block into a key cache and a value cache; Based on the sensitivity of the sample key cache to quantization error, a first offloading strategy for the key cache is determined; the first offloading strategy includes the quantization method and / or compression method of the key cache. Based on the sensitivity of the sample value cache to quantization error, a second unloading strategy for the value cache is determined; the second unloading strategy includes the quantization method and / or compression method of the value cache. Using the first unloading strategy, the key cache is unloaded from the first storage space to the second storage space; The second unloading strategy is used to unload the value cache from the first storage space to the third storage space.

2. The method according to claim 1, characterized in that, The sensitivity of the sample key cache to quantization error is determined as follows: The sensitivity of the sample key cache to quantization accuracy is determined based on at least one of the following: the role of the sample key cache in the attention module, the quantization error propagation path, the tolerable accuracy, and the distribution of outliers. Based on the distribution of outliers in the sample key cache, the sensitivity of the sample key cache to quantization granularity is determined. The sensitivity of the sample key cache to quantization accuracy and the sensitivity of the sample key cache to quantization granularity are used to determine the sensitivity of the sample key cache to quantization error.

3. The method according to claim 1, characterized in that, The determination of a first offloading strategy for the key cache based on the sensitivity of the sample key cache to quantization error includes at least one of the following: When the sensitivity of the sample key cache to quantization error indicates that the outliers of the sample key cache are concentrated in the channel dimension, the quantization method of the key cache is determined to be channel-wise quantization. If the sensitivity of the sample key cache to quantization error indicates that the sample key cache meets the residual coding condition, then the compression method of the key cache is determined to be residual coding.

4. The method according to any one of claims 1 to 3, characterized in that, The second unloading strategy for the value cache, determined based on the sensitivity of the sample value cache to quantization error, includes at least one of the following: When the sensitivity of the sample value cache to quantization error is characterized by the uniform distribution of the sample value cache values ​​in the channel dimension and the word dimension, the quantization method of the value cache is determined to be word-by-word quantization. If the sensitivity of the sample value cache to quantization error indicates that the sample value cache meets the block compression condition, then the compression method of the value cache is determined to be block encoding.

5. The method according to any one of claims 1 to 3, characterized in that, The first storage space includes the graphics processor's video memory, and the second storage space includes a hard disk; the step of using the first offloading strategy to offload the key cache from the first storage space to the second storage space includes: The key cache is quantized channel by channel to obtain the quantized key cache; the data precision of the key cache is a first data precision, and the data precision of the quantized key cache is a second data precision; the first data precision is higher than the second data precision. The quantized key cache is residual encoded to obtain the encoded key cache; Using the direct data path between the graphics processor and the hard disk, the encoded key cache is written from the graphics processor's video memory to the hard disk to unload the key cache; the first unloading strategy includes the channel-by-channel quantization and the residual encoding.

6. The method according to claim 5, characterized in that, The step of residual encoding the quantized key cache to obtain the encoded key cache includes: Determine the first term in the key cache; For all other tokens in the key cache except the first token, determine the residual between the other tokens and the corresponding anchor points; select a representative token as the anchor point for each target number of tokens. The encoded key cache is determined based on the first lexical unit and the residuals between the other lexical units and their corresponding anchor points.

7. The method according to any one of claims 1 to 3, characterized in that, The first storage space includes the video memory of the graphics processor, and the second storage space includes the video memory of the central processing unit; the step of unloading the key cache from the first storage space to the second storage space using the first unloading strategy includes: The key cache is quantized channel by channel to obtain the quantized key cache; the data precision of the key cache is a first data precision, and the data precision of the quantized key cache is a second data precision; the first data precision is higher than the second data precision. The quantized key cache is written from the graphics processor's video memory to the central processing unit's video memory to unload the key cache.

8. The method according to any one of claims 1 to 3, characterized in that, The first storage space includes the graphics processor's video memory, and the second storage space includes a hard disk; the step of using the first offloading strategy to offload the key cache from the first storage space to the second storage space includes: When the memory occupancy rate of the graphics processor is greater than or equal to a first occupancy rate threshold, the key cache is quantized channel by channel to obtain the quantized key cache; the data precision of the key cache is a first data precision, and the data precision of the quantized key cache is a second data precision; the first data precision is higher than the second data precision. The quantized key cache is written from the graphics processor's video memory to the central processing unit's video memory; When the utilization rate of the video memory of the central processing unit is greater than or equal to the second utilization rate threshold, residual encoding is performed on the quantized key cache to obtain the encoded key cache; The encoded key cache is written from the video memory of the central processing unit to the hard disk to unload the key cache.

9. The method according to claim 6, characterized in that, The method further includes: If the residual between other lexical units and their corresponding anchor points is within a preset residual range, the data precision of the residual between the other lexical units and their corresponding anchor points is reduced to a third data precision; the preset residual range is determined based on the second data precision. If the absolute value of the residual between the other word and the corresponding anchor point is less than a preset residual threshold, the data precision of the residual between the other word and the corresponding anchor point is replaced with a preset value. If the absolute value of the standard deviation of the residual between the other lexical unit and the corresponding anchor point is greater than a preset residual threshold, the data precision of the residual between the other lexical unit and the corresponding anchor point is reduced to a fourth data precision; the third data precision is higher than the fourth data precision.

10. The method according to claim 1, characterized in that, The first storage space includes the graphics processor's video memory, and the third storage space includes a hard disk; the step of using the second offloading strategy to offload the value cache from the first storage space to the third storage space includes: The value cache is subjected to word-by-word meta-quantization to obtain the quantized value cache; the data precision of the value cache is a first data precision, and the data precision of the quantized value cache is a fifth data precision; the first data precision is higher than the fifth data precision; The quantized value cache is compressed in blocks to obtain the compressed value cache; Using the direct data path between the graphics processor and the hard disk, the compressed value cache is written from the graphics processor's video memory to the hard disk to unload the value cache.

11. The method according to claim 10, characterized in that, The step of performing block compression on the compressed value cache to obtain the compressed value cache includes: The compressed values ​​are cached and stored as multiple value data blocks according to a preset block size; the preset block size is determined based on a compression ratio prediction table. Based on the entropy characteristics of the value vectors in the value data block, the compression ratio of the value data block is determined from the compression ratio prediction table; the entropy characteristics of the value vectors in the value data block are used to characterize the numerical repeatability of the value vectors in the value data block. The value data block is compressed according to the compression ratio of the value data block to obtain the compressed value cache.

12. The method according to claim 1, characterized in that, The first storage space includes the video memory of the graphics processor, and the third storage space includes the video memory of the central processing unit; the step of using the second offloading strategy to offload the value cache from the first storage space to the third storage space includes: The value cache is subjected to word-by-word meta-quantization to obtain the quantized value cache; the data precision of the value cache is a first data precision, and the data precision of the quantized value cache is a fifth data precision; the first data precision is higher than the fifth data precision; The quantized value cache is written from the graphics processor's video memory to the central processing unit's video memory to unload the value cache.

13. The method according to claim 1, characterized in that, The first storage space includes the graphics processor's video memory, and the third storage space includes a hard disk; the step of using the second offloading strategy to offload the value cache from the first storage space to the third storage space includes: When the memory usage of the graphics processor is greater than or equal to a first memory usage threshold, word-by-word quantization is performed on the value cache to obtain the quantized value cache; the data precision of the value cache is a first data precision, and the data precision of the quantized value cache is a fifth data precision; the first data precision is higher than the fifth data precision; The quantized value is cached and written from the graphics processor's video memory to the central processing unit's video memory; If the utilization rate of the video memory of the central processing unit is greater than or equal to the second utilization rate threshold, the quantized value cache is compressed in blocks to obtain the compressed value cache. The compressed value cache is written from the CPU's video memory to the hard disk to unload the value cache.

14. The method according to claim 10, characterized in that, The method further includes: If the size of the compressed value cache is greater than the preset cache size, the quantized value cache is written from the graphics processor's video memory to the hard disk to unload the value cache.

15. The method according to any one of claims 1 to 3, or 6, or 9 to 14, characterized in that, The method further includes: In response to a callback request for a key-value cache block to be called back, the key cache to be called back is restored from the second storage space; During the process of restoring the key cache to be called back, the value cache to be called back is asynchronously restored from the third storage space; Based on the key cache to be called back and the value cache to be called back, the key-value cache block to be called back is determined; the key-value cache block to be called back is used by the neural network model for contextual reasoning.

16. The method according to claim 15, characterized in that, The second storage space includes the video memory of the central processing unit and / or the hard disk; the step of restoring the cached value to be called back from the third storage space includes any of the following: If the value to be called back is stored in the video memory of the central processing unit, the value to be called back with the fifth data precision is transferred to the graphics processor to restore the value to be called back; When the value cache to be called back is stored on the hard disk and the key cache to be called back begins to be restored from the second storage space, the value portion in the key value cache block to be called back is determined as the mean vector of the value cache to be called back. When the value cache to be called back is restored, the value portion in the key value cache block to be called back is replaced with the value cache to be called back.

17. The method according to claim 1, characterized in that, The method further includes: Detect the entropy change of the attention head in a neural network model; If the inference accuracy after callback recovery is found to be abnormal based on the entropy change of the attention head, the unloading parameters are adjusted; the unloading parameters include at least one of the following: data accuracy of channel-wise quantization, data accuracy of residual encoding, accuracy of word-wise meta-quantization, and accuracy of block compression.

18. The method according to any one of claims 1 to 3, or 6, or 9 to 14, or 16 to 17, characterized in that, The key-value cache unloading method is embedded as middleware between the key-value cache management module and the transport layer.

19. The method according to any one of claims 1 to 3, or 6, or 9 to 14, or 16 to 17, characterized in that, The metadata of the key-value cache block records the storage location, quantization method, and compression method of the key cache and the value cache, respectively; the key cache and the value cache are stored in different storage spaces.

20. The method according to any one of claims 1 to 3, or 6, or 9 to 14, or 16 to 17, characterized in that, The method further includes: Based on at least one of the system hardware topology, the current load of each computing unit, and the bandwidth constraints of the data transmission path, the execution positions of the quantization operation, encoding operation, and compression operation in the key-value cache unloading method are allocated among the general computing core of the graphics processor, the coprocessor core of the data processing unit, and the central processing unit.

21. The method according to claim 15, characterized in that, The operation of restoring the key cache from the second storage space is the reverse of the operation of unloading the key cache to the second storage space; the operation of restoring the value cache from the third storage space is the reverse of the operation of unloading the value cache to the second storage space.

22. A key-value cache unloading device, characterized in that, The device includes: The split module is configured to split the key-value cache block into a key cache and a value cache. The processing module is configured to determine a first offloading strategy for the key cache based on the sensitivity of the sample key cache to quantization error; the first offloading strategy includes the quantization method and / or compression method of the key cache. The processing module is further configured to determine a second unloading strategy for the value cache based on the sensitivity of the sample value cache to quantization error; the second unloading strategy includes the quantization method and / or compression method of the value cache. The unloading module is configured to use the first unloading strategy to unload the key cache from the first storage space to the second storage space; The unloading module is further configured to use the second unloading strategy to unload the value cache from the first storage space to the third storage space.

23. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 21.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 21.

25. A computer program product, characterized in that, The computer program product includes a non-transitory computer-readable storage medium storing a computer program, which, when read and executed by a computer device, implements the steps of the method according to any one of claims 1 to 21.

Images