Key value cache management method and device

Abstract

The invention relates to a key value cache management method and device. The invention provides a key value cache management method. The method comprises the following steps: acquiring a pair of key data K and value data V from a large language model; performing sparse processing on the key data K to generate sparse key data K; respectively storing the sparse key data K, the sparse index and the sparse scaling factor into a unified heterogeneous physical block of the key value cache based on a compilation period constant offset table; performing quantization processing on the key data K and the value data V to generate quantized key data K and quantized value data V; and storing the quantized key data K, the key data quantization scaling factor, the quantized value data V and the value data quantization scaling factor into a unified heterogeneous physical block based on a compile period constant offset table which is configured to locate a plurality of fields in the unified heterogeneous physical block.

Description

Technical Field

[0001] This disclosure relates to the field of large language model technology, and more specifically, to a method and apparatus for key-value cache management. Background Technology

[0002] As the scale of large language models continues to expand, the computational and memory overhead during inference, especially the GPU memory usage of the key-value cache (KV Cache) in the attention mechanism, has become a core bottleneck restricting its deployment and application. To alleviate this pressure, the industry generally adopts quantization techniques (e.g., converting FP16 precision to lower precisions such as INT8, INT4, or even FP4) and sparsity techniques to reduce data storage and computational load. However, existing large language models exhibit significant efficiency deficiencies and resource waste when managing the resulting heterogeneous precision (i.e., a mixture of multiple precisions) of key-value caches.

[0003] Therefore, a new key-value cache management method and device that supports heterogeneous precision is needed. Summary of the Invention

[0004] The embodiments of this application provide a key-value cache management method and apparatus, which can realize integrated management of heterogeneous precision key-value caches and solve the problems of severe video memory fragmentation, storage and bandwidth waste, and inability to support low-latency dynamic sparsity and quantization group size reconfiguration.

[0005] According to a first aspect of this application, a key-value cache management method is provided, comprising: obtaining a pair of key data K and value data V from a large language model; performing sparse processing on the key data K to generate sparsed key data K; storing the sparsed key data K, a sparse index, and a sparse scaling factor into a unified heterogeneous physical block of the key-value cache based on a compile-time constant offset table; performing quantization processing on the key data K and the value data V to generate quantized key data K and quantized value data V; and storing the quantized key data K, the key data quantization scaling factor, the quantized value data V, and the value data quantization scaling factor into the unified heterogeneous physical block based on the compile-time constant offset table, wherein the compile-time constant offset table is configured to locate multiple fields in the unified heterogeneous physical block.

[0006] According to an embodiment of the first aspect of this application, the method further includes: configuring the internal format of the unified heterogeneous physical block during the compilation of the large language model, wherein the multiple fields in the unified heterogeneous physical block include: a field for storing sparse key data K, a field for storing sparse indexes, a field for storing sparse scaling factors, a field for storing quantized key data K, a field for storing key data quantization scaling factors, a field for storing quantized value data V, and a field for storing value data quantization scaling factors.

[0007] According to an embodiment of the first aspect of this application, configuring the internal format of the unified heterogeneous physical block during the compilation of the large language model includes: obtaining the sparsity rate, quantization group size, and attention head dimension of the large language model; and configuring the internal format of the unified heterogeneous physical block based on the sparsity rate, the quantization group size, and the attention head dimension.

[0008] According to an embodiment of the first aspect of this application, the method further includes: generating the compile-time constant offset table during the compilation of the large language model.

[0009] According to an embodiment of the first aspect of this application, generating the compile-time constant offset table during the compilation of the large language model includes: generating the compile-time constant offset table based on the internal format of the unified heterogeneous physical block, wherein the compile-time constant offset table includes multiple offset values ​​corresponding to multiple fields in the unified heterogeneous physical block, and each of the multiple offset values ​​is used together with the start address of the unified heterogeneous physical block to locate the corresponding field in the unified heterogeneous physical block.

[0010] According to an embodiment of the first aspect of this application, the start address and end address of the unified heterogeneous physical block are 128-byte aligned, and the free field of the unified heterogeneous physical block is filled with padding bytes.

[0011] According to an embodiment of the first aspect of this application, the sparsed key data K, the quantized key data K, and the quantized value data V have a first precision or a second precision, the sparse index has the first precision, and the sparse scaling factor, the key data quantization scaling factor, and the data quantization scaling factor have a third precision.

[0012] According to an embodiment of the first aspect of this application, the first precision is INT8 or FP8, the second precision is INT4 or FP4, and the third precision is FP32.

[0013] According to an embodiment of the first aspect of this application, the method further includes: performing subbyte concatenation on the sparse key data K, the quantized key data K, or the quantized value data V having the second precision, and storing the concatenated sparse key data K, the quantized key data K, or the quantized value data V in the corresponding fields of the unified heterogeneous physical block in the form of the first precision.

[0014] According to an embodiment of the first aspect of this application, the method further includes: modifying the sparsity rate or the quantization group size during the operation phase of the large language model; and performing a zero-copy configuration switching operation on the unified heterogeneous physical block based on the modified sparsity rate or the quantization group size, wherein the zero-copy configuration switching operation modifies a field in the unified heterogeneous physical block for storing a sparse scaling factor, or a field for storing a key data quantization scaling factor and a field for storing a value data quantization scaling factor.

[0015] According to an embodiment of the first aspect of this application, the method further includes: reading the sparsed key data K, the sparse index, and the sparse scaling factor from the unified heterogeneous physical block based on the compile-time constant offset table; or reading the quantized key data K, the key data quantization scaling factor, the quantized value data V, and the value data quantization scaling factor from the unified heterogeneous physical block based on the compile-time constant offset table.

[0016] According to a second aspect of this application, a key-value cache management apparatus is provided, comprising: a processor, and a memory storing instructions that, when executed by the processor, cause the processor to perform the method of the first aspect of this application.

[0017] According to a third aspect of this application, a computer-readable storage medium is provided having instructions stored thereon that, when executed by a computer, cause the computer to perform the method of the first aspect of this application.

[0018] According to the key-value cache management method and apparatus of the embodiments of this application, after obtaining key data K and value data V from a large language model and performing sparse processing and quantization processing, by using a compile-time constant offset table of multiple fields in a unified heterogeneous physical block configured to locate the key-value cache, the sparse key data K, sparse index, sparse scaling factor, quantized key data K, key data quantization scaling factor, quantized value data V, and value data quantization scaling factor can be stored in the corresponding fields of the unified heterogeneous physical block, thereby realizing integrated management of heterogeneous precision key-value caches, that is, each field in the unified heterogeneous physical block can store corresponding data of different precisions. Furthermore, the sparse key data K, quantized key data K, or quantized value data V with second precision can be packaged in-situ as subbytes, thereby storing them in the corresponding fields of the unified heterogeneous physical block in the form of first precision. Furthermore, a zero-copy configuration switching operation can be performed on the unified heterogeneous physical block based on the modified sparsity rate or quantization group size. This zero-copy configuration switching operation modifies only the field used to store the sparse scaling factor, or the field used to store the key data quantization scaling factor and the field used to store the value data quantization scaling factor. Therefore, the key-value cache management method and apparatus of this application can solve the problems of severe memory fragmentation, wasted storage and bandwidth, and the inability to support low-latency dynamic sparsity rate and quantization group size reconfiguration. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0020] Figure 1 This is a flowchart of a key-value cache management method according to an embodiment of this application; Figure 2 This is a schematic diagram of the internal format of a unified heterogeneous physical block according to an embodiment of this application; Figure 3 This is a schematic diagram of the hardware structure of a key-value cache management device according to an embodiment of this application. Detailed Implementation

[0021] The features and exemplary embodiments of various aspects of this application will now be described in detail. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain this application and are not configured to limit this application. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0023] The features and exemplary embodiments of various aspects of this application will now be described in detail. Furthermore, the features, structures, or characteristics described below may be combined in any suitable manner in one or more embodiments.

[0024] Large language models have become a focus of research for scholars in recent years. They are artificial intelligence models based on deep learning technology that solve various natural language processing tasks through dialogue.

[0025] As the scale of large language models continues to expand, the computational and memory overhead during inference, especially the GPU memory usage of key-value caches in attention mechanisms, has become a core bottleneck restricting their deployment and application. To alleviate this pressure, the industry commonly employs quantization techniques (e.g., converting FP16 precision to lower precisions such as INT8, INT4, or even FP4) and sparsity techniques to reduce data storage and computational load. However, existing large language models exhibit significant efficiency deficiencies and resource waste when managing the resulting heterogeneous precision (i.e., a mixture of multiple precisions) key-value caches. For example, the inventors found that the problems mainly manifest in the following aspects after research.

[0026] First, the existing key-value cache layout of large language models is rigid and isolated, leading to low memory utilization and difficulties in dynamic reconfiguration. For example, the same physical storage layout is typically used for all network layers and all data precisions, with key-value caches of different precisions allocated in completely isolated memory regions, making it impossible to dynamically reuse the same physical storage block. When dynamic configuration adjustments are needed during the inference process of large language models (e.g., modifying the sparsity rate or quantization group size), the original cache must be released and a new cache must be reallocated. This process not only introduces millisecond-level long-tail latency but also doubles the instantaneous peak of memory usage, severely impairing system response speed and stability. Furthermore, when the model contains both high-precision and low-precision layers, this isolated allocation easily creates a large number of "intermediate gaps" within the device that cannot be occupied by high-precision data or effectively utilized by low-precision data, resulting in severe memory fragmentation.

[0027] Secondly, the existing physical storage layout does not adequately support low-bit-width data types, leading to a waste of storage space and memory bandwidth. For example, the size of cache blocks is usually fixed at 64B or 128B, but it is not optimized for data types with a bit width of less than 8 bits, such as INT4 and FP4. When storing such data, it still uses 8 bits (1 byte) as the basic unit, causing half of the bits in each storage unit to be idle, resulting in a significant waste of storage space and reducing the utilization rate of bandwidth for transmitting effective data.

[0028] Furthermore, some works (e.g., symmetric quantization patterns in large language models vLLM) attempt to optimize layout by concatenating the key data K, key data quantization scaling factor K-scale, value data V, and value data quantization scaling factor V-scale into a large tensor within a physical storage block. However, this method typically only supports INT8 symmetric quantization scenarios and fails to extend to lower precisions such as INT4 / FP4. Its physical arrangement still simply follows the order of logical tensors without optimizing the rearrangement based on the memory access locality of kernel functions, which increases cache miss rates. More importantly, this method does not employ strict alignment design at the granularity of the "physical storage block." When the attention head dimension head_size is not divisible by the sparsity rate or the quantization group size, unavoidable filler holes occur at the end of the data, still resulting in wasted GPU memory.

[0029] Therefore, embodiments of this application provide a key-value cache management method and apparatus that can achieve integrated management of heterogeneous precision key-value caches and solve the problems of severe video memory fragmentation, storage and bandwidth waste, and inability to support low-latency dynamic sparsity and quantization group size reconfiguration.

[0030] Figure 1 This is a flowchart of a key-value cache management method according to an embodiment of this application. Figure 1 As shown, the key-value cache management method according to an embodiment of this application includes the following steps S110 to S150: S110: Obtain a pair of key data K and value data V from the large language model; S120: Perform sparsification on the key data K to generate sparsed key data K, sparse index, and sparse scaling factor; S130: Based on the compile-time constant offset table, store the sparsed key data K, sparse index and sparse scaling factor into the unified heterogeneous physical block of the key-value cache respectively; S140: Quantize the key data K and value data V to generate quantized key data K, key data quantization scaling factor, quantized value data V, and value data quantization scaling factor; S150: Based on the compile-time constant offset table, the quantized key data K, the key data quantization scaling factor, the quantized value data V, and the value data quantization scaling factor are stored in a unified heterogeneous physical block.

[0031] In one embodiment, the large language model can be any large language model, such as Deepseek, Llama, Qwen, GPT, etc., and this document does not impose any restrictions on it. In one embodiment, a pair of key data K and value data V can be data generated by the large language model and stored in a key-value cache. For example, the key data K (Key) and value data V (Value) can be intermediate state data generated and temporarily stored by the attention mechanism of the large language model during the computation process.

[0032] In one embodiment, after obtaining the key data K and value data V, a sparse algorithm can be executed to perform sparse processing on the key data K, thereby generating sparsed key data K, a sparse index, and a sparse scaling factor. For example, the sparse algorithm can be any sparse algorithm, including but not limited to Top-K sparse algorithms, attention-based score-based sparse algorithms, etc., and this document does not impose any restrictions on it.

[0033] In one embodiment, the sparsed key data K can be the data obtained by processing the original key data K through a sparse algorithm, and it can have a first precision or a second precision. For example, the first precision is INT8 or FP8, and the second precision is INT4 or FP4.

[0034] In one embodiment, a sparse index is used to record the position of the sparsed key data K within the original key data K, and it may have a first precision, such as INT8 or FP8. In one embodiment, a sparse scaling factor is information used to recover the original key data K from the sparsed key data K, and it may have a third precision, such as FP32. For example, the sparse scaling factor may be shared in groups (e.g., one sparse scaling factor is shared for every 64 sparse values).

[0035] In one embodiment, after obtaining key data K and value data V, a quantization algorithm can be executed to quantize the key data K and value data V, thereby generating quantized key data K, key data quantization scaling factor, quantized value data V, and sum-value data quantization scaling factor. For example, the quantization algorithm can be a group quantization algorithm, a symmetric quantization algorithm, asymmetric quantization algorithm, etc., and this document does not limit it.

[0036] In one embodiment, the quantized key data K can be the data obtained by processing the original key data K through a quantization algorithm, and it can have a first precision or a second precision. For example, the first precision is INT8 or FP8, and the second precision is INT4 or FP4.

[0037] In one embodiment, the key data quantization scaling factor is information used to recover the quantized key data K back to the original key data K, and it may have a third precision, such as FP32. For example, the key data quantization scaling factor may be shared in groups (e.g., one key data quantization scaling factor is shared for every 64 quantized values).

[0038] In one embodiment, the quantized value data V can be the data obtained by processing the original value data V through a quantization algorithm, and it can have a first precision or a second precision. For example, the first precision is INT8 or FP8, and the second precision is INT4 or FP4.

[0039] In one embodiment, the value data quantization scaling factor is information used to recover the quantized value data V back to the original value data V, and it may have a third precision, such as FP32. For example, the value data quantization scaling factor may be shared in groups (e.g., one value data quantization scaling factor is shared for every 64 quantized values).

[0040] In one embodiment, a compile-time constant offset table is configured to locate multiple fields in a unified heterogeneous physical block. Therefore, based on the compile-time constant offset table, the sparsed key data K, sparse index, sparse scaling factor, quantized key data K, key data quantization scaling factor, quantized value data V, and sum value data quantization scaling factor can be stored in the unified heterogeneous physical block of the key-value cache, respectively.

[0041] In one embodiment, the method further includes configuring the internal format of the unified heterogeneous physical blocks during the compilation phase of the large language model. For example, the compilation phase of the large language model could be the deployment phase of the large language model.

[0042] In one embodiment, configuring the internal format of a unified heterogeneous physical block during the compile-time configuration of a large language model includes: obtaining the sparsity rate, quantization group size, and attention head dimension of the large language model; and configuring the internal format of the unified heterogeneous physical block based on the sparsity rate, quantization group size, and attention head dimension. For example, the sparsity rate, quantization group size, and attention head dimension can be associated with the large language model and can be obtained from the configuration of the large language model. Furthermore, the field lengths of multiple fields in the unified heterogeneous physical block can be configured based on the sparsity rate, quantization group size, and attention head dimension.

[0043] In one embodiment, the multiple fields in the unified heterogeneous physical block include: a field sparse_cache for storing sparsed key data K, a field sparse_index for storing sparse index, a field sparse_scale for storing sparse scaling factor, a field quant_table_k for storing quantized key data K, a field quant_scale_k for storing key data quantization scaling factor, a field quant_table_v for storing quantized value data V, and a field quant_scale_v for storing value data quantization scaling factor. Figure 2 This is a schematic diagram of the internal format of a unified heterogeneous physical block according to an embodiment of this application.

[0044] In another embodiment, when using symmetric quantization, the field quant_scale_v used to store the value data quantization scaling factor can be reused as the field quant_scale_k used to store the key data quantization scaling factor, and the field quant_scale_k used to store the key data quantization scaling factor can be omitted because the key data quantization scaling factor is the same as the value data quantization scaling factor.

[0045] In one embodiment, the precision of the fields sparse_cache, quant_table_k, and quant_table_v can be a first precision (e.g., INT8 or FP8), the precision of the field sparse_index can be a first precision (e.g., INT8 or FP8), and the precision of the fields sparse_scale, quant_scale_k, and quant_scale_v can be a third precision (e.g., FP32).

[0046] In one embodiment, the method further includes: generating a compile-time constant offset table at compile time for the large language model. In one embodiment, generating the compile-time constant offset table at compile time for the large language model includes: generating the compile-time constant offset table based on the internal format of a unified heterogeneous physical block, wherein the compile-time constant offset table includes multiple offset values ​​corresponding to multiple fields in the unified heterogeneous physical block, and each of the multiple offset values ​​is used together with the start address of the unified heterogeneous physical block to locate the corresponding field in the unified heterogeneous physical block.

[0047] For example, the offset value corresponding to each field can be calculated based on the field length of multiple fields in a unified heterogeneous physical block.

[0048] In one embodiment, the start and end addresses of the unified heterogeneous physical block are aligned to 128 bytes, and the free fields of the unified heterogeneous physical block are filled with padding bytes. For example, the padding bytes can be any meaningless garbage bytes. Using padding bytes ensures that the start and end addresses of the unified heterogeneous physical block are both multiples of 128 bytes. Thus, storage and retrieval operations on the unified heterogeneous physical block can be performed by initiating only a single memory transaction.

[0049] In one embodiment, the method further includes: concatenating subbytes of the sparse key data K with second precision, the quantized key data K, or the quantized value data V, and storing the concatenated sparse key data K, the quantized key data K, or the quantized value data V in the corresponding fields of a unified heterogeneous physical block in the form of first precision.

[0050] For example, as described above, the precision of the fields sparse_cache, quant_table_k, and quant_table_v used to store the sparsed key data K, the quantized key data K, or the quantized value data V, respectively, can be a first precision (e.g., INT8 or FP8). Therefore, when the sparsed key data K, the quantized key data K, or the quantized value data V has a second precision (e.g., INT4 or FP4), they can be concatenated into subbytes before being stored in the corresponding fields of a unified heterogeneous physical block.

[0051] For example, subbyte concatenation can combine two INT4 subbytes to obtain an INT8 byte. This allows storing the sparse key data K, the quantized key data K, or the quantized value data V within a single INT8 byte of the sparse_cache, quant_table_k, or quant_table_v fields. Therefore, it avoids storing only one INT4 subbyte within a single INT8 byte of the sparse_cache, quant_table_k, or quant_table_v fields, significantly improving storage utilization.

[0052] In one embodiment, the method further includes: during the runtime phase of the large language model, modifying the sparsity rate or quantization group size; and performing a zero-copy configuration switching operation on the unified heterogeneous physical block based on the modified sparsity rate or quantization group size, wherein the zero-copy configuration switching operation modifies the field in the unified heterogeneous physical block used to store the sparse scaling factor, or the field used to store the key data quantization scaling factor and the field used to store the value data quantization scaling factor.

[0053] For example, the runtime phase of a large language model can be the phase where the user interacts with the large language model. After modifying the sparsity rate or quantization group size, the sparse scaling factor stored in the `sparse_scale` field can be modified accordingly based on the modified sparsity rate, and the key data quantization scaling factor and value data quantization scaling factor stored in the `quant_scale_k` and `quant_scale_v` fields can be modified accordingly based on the modified quantization group size. Furthermore, it is not necessary to modify the sparsed or quantized data stored in the `sparse_cache`, `sparse_index`, `quant_table_k`, or `quant_table_v` fields, because the sparsed or quantized data is obtained based on the original data and is decoupled from the scaling factor.

[0054] Furthermore, the modified sparsity scaling factor, modified key data quantization scaling factor, or modified value data quantization scaling factor can be used to reinterpret the sparse key data K, quantized key data K, or quantized value data V accordingly (e.g., dequantization, etc.). This avoids releasing and reallocating the cache, and correspondingly increases the response speed and stability of large language models.

[0055] In one embodiment, the method further includes: reading sparsed key data K, sparse index, and sparse scaling factor from a unified heterogeneous physical block based on a compile-time constant offset table; or reading quantized key data K, key data quantization scaling factor, quantized value data V, and value data quantization scaling factor from a unified heterogeneous physical block based on a compile-time constant offset table.

[0056] For example, as described above, the compile-time constant offset table is configured to locate multiple fields within a unified heterogeneous physical block. Therefore, individual fields within a unified heterogeneous physical block can be read based on the compile-time constant offset table.

[0057] According to the key-value cache management method and apparatus of the embodiments of this application, after obtaining key data K and value data V from a large language model and performing sparse processing and quantization processing, by using a compile-time constant offset table of multiple fields in a unified heterogeneous physical block configured to locate the key-value cache, the sparse key data K, sparse index, sparse scaling factor, quantized key data K, key data quantization scaling factor, quantized value data V, and value data quantization scaling factor can be stored in the corresponding fields of the unified heterogeneous physical block, thereby realizing integrated management of heterogeneous precision key-value caches, that is, each field in the unified heterogeneous physical block can store corresponding data of different precisions. Furthermore, the sparse key data K, quantized key data K, or quantized value data V with second precision can be packaged in-situ as subbytes, thereby storing them in the corresponding fields of the unified heterogeneous physical block in the form of first precision. Furthermore, a zero-copy configuration switching operation can be performed on the unified heterogeneous physical block based on the modified sparsity rate or quantization group size. This zero-copy configuration switching operation modifies only the field used to store the sparse scaling factor, or the field used to store the key data quantization scaling factor and the field used to store the value data quantization scaling factor. Therefore, the key-value cache management method and apparatus of this application can solve the problems of severe memory fragmentation, wasted storage and bandwidth, and the inability to support low-latency dynamic sparsity rate and quantization group size reconfiguration.

[0058] The application also provides a key-value cache management device, including: a processor and a memory storing instructions, which, when executed by the processor, cause the processor to perform the above-described key-value cache management method.

[0059] Figure 3 This is a schematic diagram of the hardware structure of a key-value cache management device according to an embodiment of this application. Figure 3 The key-value cache management device shown may include a processor 31 and a memory 32 storing computer program instructions.

[0060] Specifically, the processor 31 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application. Furthermore, the processor 31 may also include an accelerator such as a graphics processing unit (GPU) or a tensor processor (TPU).

[0061] Memory 32 may include a large-capacity memory for data or instructions. Where appropriate, memory 32 may include removable or non-removable (or fixed) media. In a particular embodiment, memory 32 is a non-volatile solid-state memory. Memory 32 may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the key-value cache management method described above.

[0062] The processor 31 implements the key-value cache management method in the above embodiments by reading and executing computer program instructions stored in the memory 32.

[0063] In one example, the key-value cache management device may also include a communication interface 33 and a bus 34. Wherein, as... Figure 3 As shown, the processor 31, memory 32, and communication interface 33 are connected through bus 34 and complete communication with each other.

[0064] Bus 34 includes hardware, software, or both, that couples components of the key-value cache management device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 34 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0065] This application also provides a computer-readable storage medium storing instructions thereon, which, when executed by a computer, cause the computer to perform the above-described key-value cache management method.

[0066] Examples of computer-readable storage media include non-transitory computer-readable storage media such as portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0067] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0068] The above description is merely a specific embodiment of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A key-value cache management method, characterized in that, The method includes: Obtain a pair of key data K and value data V from a large language model; The key data K is subjected to sparse processing to generate sparse key data K, sparse index and sparse scaling factor; Based on the compile-time constant offset table, the sparsed key data K, the sparse index, and the sparse scaling factor are stored in the unified heterogeneous physical block of the key-value cache, respectively. The key data K and the value data V are quantized to generate quantized key data K, key data quantization scaling factor, quantized value data V, and value data quantization scaling factor; Based on the compile-time constant offset table, the quantized key data K, the key data quantization scaling factor, the quantized value data V, and the value data quantization scaling factor are respectively stored in the unified heterogeneous physical block. The compile-time constant offset table is configured to locate multiple fields in the unified heterogeneous physical block.

2. The method according to claim 1, characterized in that, The method further includes: Configure the internal format of the unified heterogeneous physical block during the compilation phase of the large language model. The unified heterogeneous physical block includes multiple fields: a field for storing sparse key data K, a field for storing sparse index, a field for storing sparse scaling factor, a field for storing quantized key data K, a field for storing key data quantization scaling factor, a field for storing quantized value data V, and a field for storing value data quantization scaling factor.

3. The method according to claim 2, characterized in that, The internal format of the unified heterogeneous physical block configured during the compilation of the large language model includes: Obtain the sparsity, quantization group size, and attention head dimension of the large language model; The internal format of the unified heterogeneous physical block is configured based on the sparsity, the quantization group size, and the attention head dimension.

4. The method according to claim 3, characterized in that, The method further includes: The compile-time constant offset table is generated during the compilation of the large language model.

5. The method according to claim 4, characterized in that, Generating the compile-time constant offset table during the compilation of the large language model includes: The compile-time constant offset table is generated based on the internal format of the unified heterogeneous physical block. The compile-time constant offset table includes multiple offset values ​​corresponding to multiple fields in the unified heterogeneous physical block, and each of the multiple offset values ​​is used together with the start address of the unified heterogeneous physical block to locate the corresponding field in the unified heterogeneous physical block.

6. The method according to any one of claims 1-5, characterized in that, The start and end addresses of the unified heterogeneous physical block are aligned to 128 bytes, and the free fields of the unified heterogeneous physical block are filled with padding bytes.

7. The method according to any one of claims 1-5, characterized in that, The sparsed key data K, the quantized key data K, and the quantized value data V have a first precision or a second precision, the sparse index has the first precision, and the sparse scaling factor, the key data quantization scaling factor, and the data quantization scaling factor have a third precision.

8. The method according to claim 7, characterized in that, The first precision is INT8 or FP8, the second precision is INT4 or FP4, and the third precision is FP32.

9. The method according to claim 7, characterized in that, The method further includes: Subbyte concatenation is performed on the sparsed key data K with the second precision, the quantized key data K, or the quantized value data V. The spliced ​​sparse key data K, the quantized key data K, or the quantized value data V are stored in the corresponding fields of the unified heterogeneous physical block in the form of the first precision.

10. The method according to claim 3, characterized in that, The method further includes: During the execution phase of the large language model, the sparsity rate or the quantization group size is modified; Based on the modified sparsity rate or the quantization group size, a zero-copy configuration switching operation is performed on the unified heterogeneous physical block. The zero-copy configuration switching operation modifies the field in the unified heterogeneous physical block used to store sparse scaling factors, or the field used to store key data quantization scaling factors and the field used to store value data quantization scaling factors.

11. The method according to any one of claims 1-5, characterized in that, The method further includes: Based on the compile-time constant offset table, the sparsed key data K, the sparse index, and the sparse scaling factor are read from the unified heterogeneous physical block; or Based on the compile-time constant offset table, the quantized key data K, the key data quantization scaling factor, the quantized value data V, and the value data quantization scaling factor are read from the unified heterogeneous physical block.

12. A key-value cache management device, characterized in that, The device includes: processor, and A memory storing instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, The system stores instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1-11.

Images