Ultra-long context-oriented key value cache compression method and device, equipment and medium

Abstract

The embodiment of the invention discloses a super-long context-oriented key value cache compression method and device, equipment and a medium. A specific embodiment of the method comprises the following steps: acquiring a sample high-dimensional key value vector set; performing offline training processing on the sample high-dimensional key value vector set to generate a low-dimensional key value vector codebook; obtaining each real-time key value vector generated by the large language model in the reasoning stage; according to the low-dimensional key value vector codebook, performing quantitative caching processing on each real-time key value vector to generate each piece of key value index information; writing each piece of generated key value index information into a quantization key value buffer; and in the decoding process of the large language model, updating cache information in the quantization key value buffer according to the key value index information written in the quantization key value buffer and the word segmentation unit generated by the large language model. According to the embodiment, the computing power overhead of key value quantization can be reduced, and the key value compression ratio is improved, so that the storage space utilization rate is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a key-value cache compression method, apparatus, device, and medium for ultra-long contexts. Background Art

[0002] In recent years, large language models (LLMs) have been widely used in both academia and industry. However, with the rapid growth in the size of LLMs, the computational and storage requirements for long-context scenarios have also increased significantly, leading to the introduction of key-value caching technology. Currently, key-value cache compression is commonly performed using sparse attention and low-bit quantization.

[0003] However, when using the above approach for key-value cache compression, the following technical issues often arise: While the sparse attention mechanism reduces some computational overhead, it lacks memory access optimization and has limited compression rates, making it difficult to effectively reduce key-value cache overhead for very long contexts. It also introduces additional attention filtering computations. Low-bit quantization methods are susceptible to outliers, making it difficult to achieve high-quality compression at low bit widths. Furthermore, real-time dequantization introduces significant inference computational overhead.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the Invention

[0005] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] Some embodiments of the present disclosure provide a method, apparatus, device, and medium for compressing a key-value cache to an extremely long context to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a key-value cache compression method for ultra-long contexts, the method comprising: obtaining a set of sample high-dimensional key-value vectors, wherein the sample high-dimensional key-value vectors in the sample high-dimensional key-value vector set include a sample high-dimensional key vector and a sample high-dimensional value vector, and the sample high-dimensional key vector and the sample high-dimensional value vector have the same dimension; performing offline training processing on the sample high-dimensional key-value vector set to generate a low-dimensional key-value vector codebook, wherein the low-dimensional key-value vector codebook includes each low-dimensional key vector codebook and each low-dimensional value vector codebook; obtaining each real-time key-value vector generated by a large language model during the inference phase; performing quantization cache processing on each real-time key-value vector according to the low-dimensional key-value vector codebook to generate each key-value index information, wherein the generated key-value index information corresponds to the real-time key-value vector; writing each generated key-value index information into a quantization key-value cache; and during the decoding process of the large language model, updating the cache information in the quantization key-value cache according to each key-value index information written in the quantization key-value cache and the word segmentation unit generated by the large language model.

[0008] In a second aspect, some embodiments of the present disclosure provide a key-value cache compression device for ultra-long contexts, the device comprising: a first acquisition unit, configured to acquire a sample high-dimensional key-value vector set, wherein the sample high-dimensional key-value vectors in the sample high-dimensional key-value vector set include a sample high-dimensional key vector and a sample high-dimensional value vector, and the sample high-dimensional key vector and the sample high-dimensional value vector have the same dimension; an offline training unit, configured to perform offline training processing on the sample high-dimensional key-value vector set to generate a low-dimensional key-value vector codebook, wherein the low-dimensional key-value vector codebook includes each low-dimensional key vector codebook and each low-dimensional value vector codebook; a second acquisition unit, The invention is configured to obtain each real-time key-value vector generated by the large language model during the inference phase; the quantization cache unit is configured to perform quantization cache processing on each real-time key-value vector according to the low-dimensional key-value vector codebook to generate each key-value index information, wherein the generated key-value index information corresponds to the real-time key-value vector; the writing unit is configured to write each generated key-value index information into the quantization key-value cache; the updating unit is configured to update the cache information in the quantization key-value cache according to each key-value index information written in the quantization key-value cache and the word segmentation unit generated by the large language model during the decoding process of the large language model.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation of the first aspect is implemented.

[0011] The above-described embodiments of the present disclosure have the following beneficial effects: The key-value cache compression method for very long contexts, as described in some embodiments of the present disclosure, can reduce the computational overhead of key-value quantization and improve the key-value compression ratio, thereby improving storage space utilization. Specifically, the low key-value compression ratio and high computational overhead of key-value compression are due to the following reasons: while the sparse attention mechanism reduces some computational overhead, it lacks memory access optimization and has a limited compression ratio, making it difficult to effectively reduce the key-value cache overhead under very long contexts. It also introduces additional attention filtering computations. Low-bit quantization methods are affected by outliers, making it difficult to achieve high-quality compression at low bit widths, and real-time dequantization introduces significant inference computational overhead. Based on this, the key-value cache compression method for very long contexts, as described in some embodiments of the present disclosure, first obtains a set of sample high-dimensional key-value vectors. The sample high-dimensional key-value vectors in the set include sample high-dimensional key vectors and sample high-dimensional value vectors, each having the same dimension. Then, the set of sample high-dimensional key-value vectors undergoes offline training to generate a low-dimensional key-value vector codebook. Among them, the low-dimensional key-value vector codebook includes each low-dimensional key-value vector codebook and each low-dimensional value vector codebook. Therefore, through subspace partitioning and clustering operations, the sample high-dimensional key-value vector can be compressed and mapped to a discrete sub-vector set, thereby achieving low-bit width expression. Afterwards, the real-time key-value vectors generated by the large language model in the inference stage are obtained. Secondly, according to the low-dimensional key-value vector codebook, each real-time key-value vector is quantized and cached to generate each key-value index information. Among them, the generated key-value index information corresponds to the real-time key-value vector. Therefore, the high-dimensional floating-point key-value vector can be converted into a low-bit index code in real time, thereby significantly compressing the memory usage of the key-value vector cache. In addition, the storage of floating-point numbers can be avoided, reducing the cache size. Then, the generated key-value index information is written to the quantized key-value cache. Therefore, by constructing a lightweight and indexed key-value cache structure, the memory access bandwidth pressure can be effectively reduced. Finally, during the decoding process of the large language model, the cached information in the quantized key-value buffer is updated based on the key-value index information written to the quantized key-value buffer and the word segmentation units generated by the large language model. This allows the large language model to continuously model very long contexts during inference by dynamically quantizing newly generated key-value pairs. Furthermore, by converting matrix multiplication into a subspace lookup table accumulation method and leveraging the equivalence between product quantization and inner product search, high-dimensional computations can be decomposed into low-dimensional vector operations, thereby reducing computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flowchart of some embodiments of a key-value cache compression method for ultra-long contexts according to the present disclosure;

[0014] Figure 2 It is a structural diagram of some embodiments of a key-value cache compression device for ultra-long context according to the present disclosure;

[0015] Figure 3 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0018] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] Figure 1A process 100 of some embodiments of a key-value cache compression method for an extremely long context according to the present disclosure is shown. The key-value cache compression method for an extremely long context includes the following steps:

[0023] Step 101: Obtain a sample high-dimensional key-value vector set.

[0024] In some embodiments, an execution subject (e.g., a computing device) of a key-value cache compression method for ultra-long contexts may obtain a set of sample high-dimensional key-value vectors. The sample high-dimensional key-value vectors in the set of sample high-dimensional key-value vectors include sample high-dimensional key vectors and sample high-dimensional value vectors. The sample high-dimensional key vector and the sample high-dimensional value vector have the same dimension. The sample high-dimensional key vector may be a key vector having a dimension greater than or equal to 128. The sample high-dimensional value vector may be a value vector having a dimension greater than or equal to 128.

[0025] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0026] Step 102: Perform offline training on the sample high-dimensional key-value vector set to generate a low-dimensional key-value vector codebook.

[0027] In some embodiments, the execution entity may perform offline training processing on the sample high-dimensional key-value vector set to generate a low-dimensional key-value vector codebook. The low-dimensional key-value vector codebook includes each low-dimensional key vector codebook and each low-dimensional value vector codebook. Each low-dimensional key vector codebook may be a codebook containing each low-dimensional key vector. Each low-dimensional value vector codebook may be a codebook containing each low-dimensional value vector. The low-dimensional key vector codebook may be a sub-vector set consisting of the centers of each sub-vector cluster after clustering the sub-vectors within the vector space range in a specified dimension (for example, 1 and 2 dimensions in a 128-dimensional sample high-dimensional value vector). The low-dimensional value vector codebook may be a sub-vector set consisting of the centers of each sub-vector cluster after clustering the sub-vectors within the vector space range in a specified dimension (for example, 1 and 2 dimensions in a 128-dimensional sample high-dimensional value vector).

[0028] In some optional implementations of some embodiments, the execution entity may perform offline training on the sample high-dimensional key-value vector set to generate a low-dimensional key-value vector codebook through the following steps:

[0029] In the first step, spatial clustering is performed on each sample high-dimensional key vector and each sample high-dimensional value vector included in the sample high-dimensional key-value vector set to generate each low-dimensional value vector codebook and each low-dimensional key vector codebook.

[0030] In the second step, the aforementioned low-dimensional key vector codebooks and the aforementioned low-dimensional value vector codebooks are determined as low-dimensional key-value vector codebooks.

[0031] In the process of adopting technical solutions to solve the problems mentioned in the background technology, the following problems are often accompanied: directly clustering or quantizing the entire high-dimensional vector without considering the information density between different dimensions in the key-value vector, resulting in large quantization errors and large precision loss, and directly clustering high-dimensional vectors often increases additional computing power overhead.

[0032] Faced with the above technical problems, the inventors decided to adopt the following solutions:

[0033] In some optional implementations of some embodiments, the execution entity may perform spatial clustering processing on each sample high-dimensional key vector and each sample high-dimensional value vector included in the sample high-dimensional key-value vector set to generate each low-dimensional value vector codebook through the following steps, including:

[0034] In the first step, for each sample high-dimensional key vector included in the above-mentioned sample high-dimensional key-value vector set, the above-mentioned sample high-dimensional key vector is spatially divided according to the preset division dimension to generate a low-dimensional key vector sequence. In practice, first, the above-mentioned execution subject can divide the above-mentioned sample high-dimensional key vector into sub-vectors equal to the preset division dimension as each low-dimensional key vector according to the above-mentioned preset division dimension. Then, the above-mentioned execution subject can sort the low-dimensional key vectors according to the division order or the order of dimension from small to large (that is, the dimension in the sample high-dimensional key vector) to obtain a low-dimensional key vector sequence. For example, the above-mentioned sample high-dimensional key vector is a 128-dimensional key vector, and the preset division dimension is 2. The above-mentioned execution subject can divide the above-mentioned sample high-dimensional key vector into 64 sub-vectors.

[0035] The second step is to group the generated low-dimensional key vector sequences to obtain a set of low-dimensional key vector groups, wherein each low-dimensional key vector included in the low-dimensional key vector group has the same position (i.e., order) in the corresponding low-dimensional key vector sequence. In practice, the execution entity may group the low-dimensional key vectors with the same position in each low-dimensional key vector sequence into the same group to obtain the set of low-dimensional key vector groups.

[0036] In the third step, for each low-dimensional key vector group in the above low-dimensional key vector group set, perform the following clustering steps:

[0037] The first sub-step is to determine the dimensionality information corresponding to the low-dimensional key vector group. The dimensionality information may be the number of dimensions of any low-dimensional key vector in the corresponding sample high-dimensional key vector. For example, if the low-dimensional key vector is located at dimensions 61 and 62 (i.e., rows 61 and 62 in the row vector) in the corresponding sample high-dimensional key vector, the dimensionality information corresponding to the low-dimensional key vector may be (61, 62).

[0038] The second sub-step is to cluster the low-dimensional key vectors included in the low-dimensional key vector group according to a preset number of cluster centers to determine the central low-dimensional key vectors. In practice, the execution entity may cluster the low-dimensional key vectors included in the low-dimensional key vector group using a preset clustering algorithm and a preset number of cluster centers to obtain clusters of low-dimensional key vectors. Each central low-dimensional key vector may be the center of a corresponding low-dimensional key vector cluster. As an example, the preset clustering algorithm may be a K-means algorithm. The preset number of cluster centers may be 256.

[0039] The third sub-step is to determine the determined central low-dimensional key vectors as a low-dimensional key vector codebook corresponding to the above-mentioned dimensional information.

[0040] The fourth step is to perform spatial division processing on each sample high-dimensional value vector included in the above-mentioned sample high-dimensional key-value vector set according to the above-mentioned preset division dimension to generate a low-dimensional value vector sequence. In practice, first, the above-mentioned execution subject can divide the above-mentioned sample high-dimensional value vector into sub-vectors equal to the preset division dimension as each low-dimensional value vector according to the above-mentioned preset division dimension. Then, the above-mentioned execution subject can sort the low-dimensional value vectors according to the division order or the order of dimension from small to large (that is, the dimension of the sub-vector in the sample high-dimensional value vector) to obtain a low-dimensional value vector sequence.

[0041] In the fifth step, the generated low-dimensional valued vector sequences are grouped to obtain a set of low-dimensional valued vector groups, wherein each low-dimensional valued vector included in the low-dimensional valued vector group in the set of low-dimensional valued vector groups has the same position in the corresponding low-dimensional valued vector sequence. In practice, the execution entity may group the low-dimensional key vectors with the same position in each low-dimensional valued vector sequence into the same group to obtain the set of low-dimensional valued vector groups.

[0042] Step 6: For each low-dimensional value vector group in the above low-dimensional value vector group set, perform the following clustering steps:

[0043] The first sub-step is to determine the dimension information corresponding to the low-dimensional value vector group. The dimension information may be the dimension of any low-dimensional value vector in the low-dimensional value vector group in the corresponding sample high-dimensional value vector.

[0044] In a second sub-step, clustering the low-dimensional valued vectors included in the low-dimensional valued vector group is performed according to a preset number of cluster centers to determine central low-dimensional valued vectors. In practice, the execution entity may cluster the low-dimensional valued vectors included in the low-dimensional valued vector group using the preset clustering algorithm and the preset number of cluster centers to obtain clusters of low-dimensional valued vectors. Each central low-dimensional valued vector may be the center of a corresponding low-dimensional valued vector cluster.

[0045] The third sub-step is to determine the determined central low-dimensional value vectors as a low-dimensional value vector codebook corresponding to the above-mentioned dimension information.

[0046] The first to fourth steps mentioned above serve as an inventive point of an embodiment of the present disclosure, which solves the technical problem of "directly clustering or quantizing the entire high-dimensional vector without considering the information density between different dimensions in the key-value vector, resulting in large quantization errors and large precision losses, and directly clustering high-dimensional vectors often increases additional computing power overhead". The factors that lead to large quantization errors, large precision losses and increased additional computing power are often as follows: directly clustering or quantizing the entire high-dimensional vector without considering the information density between different dimensions in the key-value vector. By decomposing the high-dimensional key vector and the high-dimensional value vector into multiple low-dimensional sub-vectors, each sub-vector can only carry part of the local features, thereby reducing the information complexity within each subspace. Then, the sub-vectors of the same dimension are divided into the same codebook, thereby improving the consistency of local subspace features. It also avoids direct processing of high-dimensional vectors, greatly reducing the computing power resources and computational complexity required for clustering.

[0047] Step 103: Obtain each real-time key-value vector generated by the large language model during the inference phase.

[0048] In some embodiments, the execution entity may obtain each real-time key-value vector generated by the large language model during the inference phase. The real-time key-value vector includes a real-time key vector and a real-time value vector. The real-time key vector may be a key vector generated in real time by the large language model during the inference phase. The real-time value vector may be a value vector generated in real time by the large language model during the inference phase.

[0049] Step 104 : performing quantization and caching processing on each real-time key-value vector according to the low-dimensional key-value vector codebook to generate each key-value index information.

[0050] In some embodiments, the execution entity may perform quantization and caching processing on each of the real-time key-value vectors according to the low-dimensional key-value vector codebook to generate key-value index information, wherein the generated key-value index information corresponds to the real-time key-value vector.

[0051] In some optional implementations of some embodiments, the execution entity may perform quantization and caching processing on each of the real-time key-value vectors according to the low-dimensional key-value vector codebook to generate key-value index information, including the following steps:

[0052] In the first step, for each of the above real-time key-value vectors, perform the following steps:

[0053] The first sub-step involves spatially partitioning the real-time key vectors included in the real-time key-value vector according to a preset partitioning dimension to obtain a set of low-dimensional key vectors. In practice, the execution entity may partition the real-time key vectors included in the real-time key-value vector according to the preset partitioning dimension into sub-vectors equal to the preset partitioning dimension as the low-dimensional key vectors to obtain the set of low-dimensional key vectors.

[0054] The second sub-step involves spatially partitioning the real-time value vectors included in the real-time key-value vector according to a preset partitioning dimension to obtain a set of low-dimensional value vectors. In practice, the execution entity may partition the real-time value vectors included in the real-time key-value vector according to the preset partitioning dimension into sub-vectors equal to the preset partitioning dimension as the low-dimensional value vectors to obtain the set of low-dimensional value vectors.

[0055] In the second step, according to each low-dimensional key vector codebook included in the above-mentioned low-dimensional key value vector codebook, each obtained low-dimensional key vector set is quantized and cached to generate each key vector index information.

[0056] In the third step, according to each low-dimensional value vector codebook included in the low-dimensional key value vector codebook, each obtained low-dimensional value vector set is quantized and cached to generate each value vector index information.

[0057] In the fourth step, the generated key vector index information and each value vector index information are determined as key-value index information.

[0058] In some optional implementations of some embodiments, the execution entity may quantize and cache each low-dimensional key vector set obtained according to each low-dimensional key vector codebook included in the low-dimensional key value vector codebook to generate each key vector index information through the following steps, including:

[0059] In the first step, for each of the above low-dimensional key vector sets, perform the following processing steps:

[0060] In the first sub-step, for each low-dimensional key vector in the above set of low-dimensional key vectors, perform the following quantization cache steps:

[0061] Sub-step 1: Determine the initial dimension information corresponding to the low-dimensional key vector. The initial dimension information may be the number of dimensions of the low-dimensional key vector in the corresponding real-time key vector. For example, if the low-dimensional key vector is located at dimensions 61 and 62 in the corresponding real-time key vector (i.e., rows 61 and 62 in the row vector), the initial dimension information corresponding to the low-dimensional key vector may be (61, 62).

[0062] In sub-step 2, based on the determined initial dimensionality information, a low-dimensional key vector codebook that satisfies a dimensionality matching condition is selected from the aforementioned low-dimensional key vector codebooks as a target low-dimensional key vector codebook. The dimensionality matching condition may be that the initial dimensionality information corresponding to the aforementioned low-dimensional key vector is the same as the dimensionality information of any central low-dimensional key vector included in the low-dimensional key vector codebook.

[0063] Sub-step three: determining a key vector index according to the target low-dimensional key vector codebook and the low-dimensional key vector.

[0064] In the second sub-step, the determined key vector indices are sorted according to the determined initial dimension information to obtain a key vector index sequence. In practice, the execution entity may sort the determined key vector indices in ascending order according to the corresponding initial dimension information to obtain the key vector index sequence.

[0065] The third sub-step is to compress the key vector index sequence to generate key vector index information. In practice, first, the execution subject can compress each key vector index in the key vector index sequence by bit-packing to generate key vector index information.

[0066] In some optional implementations of some embodiments, the execution entity may determine the key vector index according to the target low-dimensional key vector codebook and the low-dimensional key vector through the following steps:

[0067] In the first step, for each central low-dimensional key vector included in the target low-dimensional key vector codebook, a spatial distance between the central low-dimensional key vector and the low-dimensional key vector is determined. In practice, the execution entity may determine the Euclidean distance between the central low-dimensional key vector and the low-dimensional key vector as the spatial distance.

[0068] The second step is to sort the determined spatial distances to obtain a spatial distance sequence. In practice, the execution entity can sort the determined spatial distances in ascending order to obtain a spatial distance sequence.

[0069] In the third step, the central low-dimensional key vector corresponding to the first spatial distance in the spatial distance sequence is determined as the target central low-dimensional key vector.

[0070] In the fourth step, the index corresponding to the target center low-dimensional key vector is determined as the key vector index. The index corresponding to the target center low-dimensional key vector can be the index of the target center low-dimensional key vector in the corresponding low-dimensional key vector codebook. It should be noted that since each low-dimensional key vector codebook is obtained by clustering each low-dimensional key vector, and the preset number of cluster centers is 256, the index range of the target low-dimensional key vector codebook is 0 to 255.

[0071] In some optional implementations of some embodiments, the execution entity may quantize and cache each obtained low-dimensional value vector set according to each low-dimensional value vector codebook included in the low-dimensional key-value vector codebook to generate each value vector index information through the following steps, including:

[0072] In the first step, for each of the above low-dimensional value vector sets, perform the following processing steps:

[0073] In the first sub-step, for each low-dimensional value vector in the above set of low-dimensional value vectors, the following quantization caching steps are performed:

[0074] Sub-step 1: Determine the initial dimension information corresponding to the low-dimensional value vector. The initial dimension information may be the number of dimensions of the low-dimensional value vector in the corresponding real-time value vector.

[0075] In sub-step 2, based on the determined initial dimensionality information, a low-dimensional valued vector codebook that satisfies a dimensionality matching condition is selected from the various low-dimensional valued vector codebooks as a target low-dimensional valued vector codebook. The dimensionality matching condition may be that the initial dimensionality information corresponding to the low-dimensional valued vector is the same as the dimensionality information of any center low-dimensional valued vector included in the low-dimensional valued vector codebook.

[0076] Sub-step three: determining a value vector index according to the target low-dimensional value vector codebook and the low-dimensional value vector.

[0077] The second sub-step is to sort the determined value vector indices according to the determined initial dimension information to obtain a value vector index sequence. In practice, the execution entity may sort the determined value vector indices in ascending order according to the corresponding initial dimension information to obtain the value vector index sequence.

[0078] The third sub-step is to compress the value vector index sequence to generate value vector index information. In practice, first, the execution subject can compress each value vector index in the value vector index sequence by bit-packing to generate value vector index information.

[0079] In some optional implementations of some embodiments, the execution entity may determine the value vector index according to the target low-dimensional value vector codebook and the low-dimensional value vector through the following steps:

[0080] The first step is to determine, for each central low-dimensional value vector included in the target low-dimensional value vector codebook, a spatial distance between the central low-dimensional value vector and the low-dimensional value vector. In practice, the execution entity may determine the Euclidean distance between the central low-dimensional value vector and the low-dimensional value vector as the spatial distance.

[0081] The second step is to sort the determined spatial distances to obtain a spatial distance sequence. In practice, the execution entity can sort the determined spatial distances in ascending order to obtain a spatial distance sequence.

[0082] In the third step, the central low-dimensional value vector corresponding to the first spatial distance in the spatial distance sequence is determined as the target central low-dimensional value vector.

[0083] In the fourth step, the index corresponding to the target center low-dimensional value vector is determined as the value vector index. The index corresponding to the target center low-dimensional value vector may be the index of the target center low-dimensional value vector in the corresponding low-dimensional value vector codebook.

[0084] Step 105 : Write the generated key-value index information into the quantized key-value buffer.

[0085] In some embodiments, the execution entity may write the generated key-value index information into a quantized key-value cache. In practice, the execution entity may write the generated key-value index information into a quantized key-value cache. The quantized key-value cache (Quantized KV Cache) may be a cache for storing a key-value cache.

[0086] Step 106 , during the decoding process of the large language model, the cache information in the quantization key value buffer is updated according to the key value index information written in the quantization key value buffer and the word segmentation units generated by the large language model.

[0087] In some embodiments, during the decoding process of the large language model, the execution entity may update the cache information in the quantized key-value buffer according to the key-value index information written in the quantized key-value buffer and the word segmentation units generated by the large language model. The word segmentation units (i.e., tokens) may be word segmentations generated by the large language model during the decoding phase.

[0088] In the process of adopting technical solutions to solve the problems mentioned in the background technology, the following problems often arise: during the inference process of large language models, the key-value cache grows linearly with the generation length, resulting in excessive video memory usage and memory access bandwidth bottlenecks, thereby increasing the inference delay of large language models.

[0089] Faced with the above technical problems, the inventors decided to adopt the following solutions:

[0090] In some optional implementations of some embodiments, the execution entity may update the cache information in the quantization key value buffer according to the key value index information written in the quantization key value buffer and the word segmentation units generated by the large language model during the decoding process of the large language model through the following steps, including:

[0091] In the first step, for each word segmentation unit generated by the large language model in the decoding phase, the following decoding steps are performed:

[0092] The first sub-step is to read the value vector index information and key vector index information in the quantized key value buffer. In practice, the execution subject can read the value vector index information and key vector index information in the quantized key value buffer.

[0093] The second sub-step is to generate recovered key vectors based on the read key vector index information. In practice, for each key vector index information, the execution entity may first retrieve the corresponding low-dimensional key vectors from the corresponding low-dimensional key vector codebook according to the key vector indexes included in the key vector index information. Then, the execution entity may sequentially concatenate the low-dimensional key vectors according to the order of the key vector indexes in the key vector index information to generate the recovered key vector.

[0094] The third sub-step is to generate each restored value vector based on the read value vector index information. In practice, for each value vector index information, the execution entity may first retrieve the corresponding low-dimensional value vectors from the corresponding low-dimensional value vector codebook according to the value vector indexes included in the value vector index information. Then, the execution entity may sequentially concatenate the low-dimensional value vectors according to the order of the value vector indexes in the value vector index information to generate the restored value vector.

[0095] The fourth sub-step is to generate a corresponding word segmentation query vector based on the above word segmentation unit. In practice, the above execution entity can generate the query vector corresponding to the above word segmentation unit as the word segmentation query component through the attention mechanism.

[0096] The fifth sub-step is to generate the word segmentation hidden vector of the corresponding word segmentation unit based on the generated restored key vectors, restored value vectors and the above-mentioned word segmentation query vectors. In practice, the above-mentioned execution subject can perform dot product operations on each restored key vector and the word segmentation query component one by one to obtain each attention score (i.e., Attention Scores). Then, the above-mentioned execution subject can normalize the generated attention score values through Softmax to obtain each normalized weight. Finally, the above-mentioned execution subject can perform weighted summation of each normalized weight and each restored value vector to generate the word segmentation hidden vector of the corresponding word segmentation unit.

[0097] The sixth sub-step is to generate a word segmentation key-value vector based on the above-mentioned word segmentation hidden vector. The above-mentioned word segmentation key-value vector can be the key-value vector of the predicted next word segmentation unit. In practice, first, the above-mentioned execution subject can input the above-mentioned word segmentation hidden vector into the prediction layer or prediction head or output head of the large language model to generate the next word segmentation unit (i.e., the next word segmentation). Then, the above-mentioned execution subject can generate the word segmentation key-value vector corresponding to the next word segmentation unit through the attention mechanism. The above-mentioned word segmentation key-value component may include a word segmentation key vector and a word segmentation value vector.

[0098] The sixth sub-step is to generate corresponding word segmentation key-value index information based on the generated word segmentation key-value vector. In practice, "generating corresponding word segmentation key-value index information based on the generated word segmentation key-value vector" can refer to the implementation step of "quantizing and caching each low-dimensional key vector set obtained according to each low-dimensional key vector codebook included in the above-mentioned low-dimensional key-value vector codebook to generate each key vector index information" and will not be repeated here.

[0099] The seventh sub-step is to write the generated word segmentation key value index information into the quantization key value buffer to update the quantization key value buffer. In practice, the execution entity may write the generated word segmentation key value index information into the quantization key value buffer to update the quantization key value buffer.

[0100] The above steps, as an inventive point of an embodiment of the present disclosure, solve the technical problem that "in the process of large language model inference, the key-value cache grows linearly with the generation length, resulting in excessive video memory usage and memory access bandwidth bottlenecks, thereby increasing the inference delay of the large language model". The factors that lead to excessive video memory usage, memory access bandwidth bottlenecks and increased model inference delays are often as follows: in the process of large language model inference, the key-value cache grows linearly with the generation length. This application effectively reduces the video memory usage of the key-value cache, alleviates the pressure on memory access bandwidth, reduces inference delay, improves decoding throughput, and significantly enhances the reasoning continuity and system stability of large language models in ultra-long context reasoning tasks by restoring historical key-value vectors based on quantization indexes in the decoding stage and quantizing and encoding the key-value vectors of newly generated tokens in real time.

[0101] The above-described embodiments of the present disclosure have the following beneficial effects: The key-value cache compression method for very long contexts, as described in some embodiments of the present disclosure, can reduce the computational overhead of key-value quantization and improve the key-value compression ratio, thereby improving storage space utilization. Specifically, the low key-value compression ratio and high computational overhead of key-value compression are due to the following reasons: while the sparse attention mechanism reduces some computational overhead, it lacks memory access optimization and has a limited compression ratio, making it difficult to effectively reduce the key-value cache overhead under very long contexts. It also introduces additional attention filtering computations. Low-bit quantization methods are affected by outliers, making it difficult to achieve high-quality compression at low bit widths, and real-time dequantization introduces significant inference computational overhead. Based on this, the key-value cache compression method for very long contexts, as described in some embodiments of the present disclosure, first obtains a set of sample high-dimensional key-value vectors. The sample high-dimensional key-value vectors in the set include sample high-dimensional key vectors and sample high-dimensional value vectors, each having the same dimension. Then, the set of sample high-dimensional key-value vectors undergoes offline training to generate a low-dimensional key-value vector codebook. Among them, the low-dimensional key-value vector codebook includes each low-dimensional key-value vector codebook and each low-dimensional value vector codebook. Therefore, through subspace partitioning and clustering operations, the sample high-dimensional key-value vector can be compressed and mapped to a discrete sub-vector set, thereby achieving low-bit width expression. Afterwards, the real-time key-value vectors generated by the large language model in the inference stage are obtained. Secondly, according to the low-dimensional key-value vector codebook, each real-time key-value vector is quantized and cached to generate each key-value index information. Among them, the generated key-value index information corresponds to the real-time key-value vector. Therefore, the high-dimensional floating-point key-value vector can be converted into a low-bit index code in real time, thereby significantly compressing the memory usage of the key-value vector cache. In addition, the storage of floating-point numbers can be avoided, reducing the cache size. Then, the generated key-value index information is written to the quantized key-value cache. Therefore, by constructing a lightweight and indexed key-value cache structure, the memory access bandwidth pressure can be effectively reduced. Finally, during the decoding process of the large language model, the cache information in the quantized key-value buffer is updated according to the key-value index information written in the quantized key-value buffer and the word segmentation units generated by the large language model. Thus, by dynamically quantizing the newly generated key-value pairs, the large language model can continuously model the ultra-long context during the inference process. In addition, by converting matrix multiplication into a subspace table lookup accumulation method, the equivalence of product quantization and inner product retrieval can be used to decompose high-dimensional calculations into low-dimensional vector operations, thereby reducing computing power overhead. In addition, on models such as LLaMA-2-7B, the key-value cache compression method for ultra-long contexts disclosed in this disclosure can achieve a 4x compression rate and a 2.98x end-to-end acceleration at a context length of 32K, with a perplexity error of less than 0.5%. The F1 score fluctuation of the long text question answering task is less than 2%. System optimization has increased the attention kernel computing speed by 1.88 times and the memory bandwidth utilization rate to more than 90%.The quantization buffer strategy controls the additional delay within 2%, which is significantly better than the OOM risk and high dequantization overhead of traditional quantization schemes (such as KIVI and KVQuant).

[0102] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a key-value cache compression device for ultra-long contexts. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the key-value cache compression device for ultra-long contexts can be specifically applied to various electronic devices.

[0103] like Figure 2 As shown, some embodiments of the key-value cache compression device 200 for ultra-long contexts include: a first acquisition unit 201, an offline training unit 202, a second acquisition unit 203, a quantization cache unit 204, a writing unit 205, and an updating unit 206. The first acquisition unit 201 is configured to acquire a set of sample high-dimensional key-value vectors, wherein the sample high-dimensional key-value vectors in the sample high-dimensional key-value vector set include sample high-dimensional key vectors and sample high-dimensional value vectors, and the sample high-dimensional key vectors and the sample high-dimensional value vectors have the same dimension; the offline training unit 202 is configured to perform offline training processing on the sample high-dimensional key-value vector set to generate a low-dimensional key-value vector codebook, wherein the low-dimensional key-value vector codebook includes each low-dimensional key vector codebook and each low-dimensional value vector codebook; the second acquisition unit 203 is configured to acquire the codebook generated by the large language model in the inference stage. Each real-time key-value vector; the quantization cache unit 204 is configured to perform quantization cache processing on each real-time key-value vector according to the low-dimensional key-value vector codebook to generate each key-value index information, wherein the generated key-value index information corresponds to the real-time key-value vector; the writing unit 205 is configured to write the generated each key-value index information into the quantization key-value cache; the updating unit 206 is configured to update the cache information in the quantization key-value cache according to the each key-value index information written in the quantization key-value cache and the word segmentation unit generated by the large language model during the decoding process of the large language model.

[0104] It is understandable that the various units recorded in the key-value cache compression device 200 for ultra-long contexts are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the key-value cache compression device 200 for ultra-long contexts and the units included therein, and will not be described in detail here.

[0105] Reference below Figure 3 , which shows a structural diagram of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0106] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 or a program loaded from a storage device 308 into a random access memory 303. Various programs and data required for the operation of the electronic device 300 are also stored in the random access memory 303. The processing device 301, the read-only memory 302, and the random access memory 303 are connected to each other via a bus 304. An input / output interface 305 is also connected to the bus 304.

[0107] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0108] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 309, or installed from the storage device 308, or installed from the read-only memory 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0109] It should be noted that the computer-readable medium described in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0110] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0111] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When executed by the electronic device, the electronic device: obtains a set of sample high-dimensional key-value vectors, wherein the sample high-dimensional key-value vectors in the set of sample high-dimensional key-value vectors include sample high-dimensional key vectors and sample high-dimensional value vectors, and the sample high-dimensional key vectors and the sample high-dimensional value vectors have the same dimension; performs offline training processing on the set of sample high-dimensional key-value vectors to generate a low-dimensional key-value vector codebook, wherein the low-dimensional key-value vector codebook includes each low-dimensional key vector codebook and each low-dimensional value vector codebook; obtains each real-time key-value vector generated by the large language model during the inference phase; performs quantization and caching processing on each real-time key-value vector based on the low-dimensional key-value vector codebook to generate each key-value index information, wherein the generated key-value index information corresponds to the real-time key-value vector; writes each generated key-value index information into a quantization key-value buffer; and during the decoding process of the large language model, updates the cache information in the quantization key-value buffer based on each key-value index information written in the quantization key-value buffer and the word segmentation unit generated by the large language model.

[0112] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] The units described in some embodiments of the present disclosure may be implemented in software or in hardware. The described units may also be provided in a processor. For example, they may be described as follows: a processor comprising a first acquisition unit, an offline training unit, a second acquisition unit, a quantization cache unit, a write unit, and an update unit. The names of these units do not, in some cases, constitute a limitation on the units themselves. For example, the first acquisition unit may also be described as a "unit for acquiring a set of high-dimensional key-value vectors of samples."

[0115] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0116] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A key-value cache compression method for extremely long contexts, comprising: Obtaining a sample high-dimensional key-value vector set, wherein the sample high-dimensional key-value vectors in the sample high-dimensional key-value vector set include a sample high-dimensional key vector and a sample high-dimensional value vector, and the sample high-dimensional key vector and the sample high-dimensional value vector have the same dimension; Performing offline training on the sample high-dimensional key-value vector set to generate a low-dimensional key-value vector codebook, wherein the low-dimensional key-value vector codebook includes each low-dimensional key vector codebook and each low-dimensional value vector codebook; Obtain the real-time key-value vectors generated by the large language model during the inference phase; According to the low-dimensional key-value vector codebook, the real-time key-value vectors are quantized and cached to generate key-value index information, wherein the generated key-value index information corresponds to the real-time key-value vector; Writing each generated key value index information into the quantized key value buffer; During the decoding process of the large language model, the cache information in the quantization key value buffer is updated according to the key value index information written in the quantization key value buffer and the word segmentation units generated by the large language model.

2. The method according to claim 1, wherein The offline training process of the sample high-dimensional key-value vector set to generate a low-dimensional key-value vector codebook includes: Performing spatial clustering processing on each sample high-dimensional key vector and each sample high-dimensional value vector included in the sample high-dimensional key-value vector set to generate each low-dimensional value vector codebook and each low-dimensional key vector codebook; The respective low-dimensional key vector codebooks and the respective low-dimensional value vector codebooks are determined as low-dimensional key-value vector codebooks.

3. The method according to claim 2, wherein: The step of performing quantization and caching processing on each of the real-time key-value vectors according to the low-dimensional key-value vector codebook to generate each key-value index information includes: For each of the real-time key-value vectors, the following steps are performed: Performing spatial division on the real-time key vectors included in the real-time key-value vector according to a preset division dimension to obtain a low-dimensional key vector set; Performing spatial division on the real-time value vectors included in the real-time key-value vector according to a preset division dimension to obtain a low-dimensional value vector set; According to the low-dimensional key vector codebooks included in the low-dimensional key-value vector codebook, the obtained low-dimensional key vector sets are quantized and cached to generate respective key vector index information; according to the low-dimensional value vector codebooks included in the low-dimensional key-value vector codebook, the obtained low-dimensional value vector sets are quantized and cached to generate respective value vector index information; the generated key vector index information and the respective value vector index information are determined as key value index information.

4. The method according to claim 3, wherein: The step of quantizing and caching each low-dimensional key vector set obtained according to each low-dimensional key vector codebook included in the low-dimensional key value vector codebook to generate each key vector index information includes: For each low-dimensional key vector set in the respective low-dimensional key vector sets, the following processing steps are performed: For each low-dimensional key vector in the set of low-dimensional key vectors, the following quantization caching steps are performed: Determining initial dimension information corresponding to the low-dimensional key vector; According to the determined initial dimension information, selecting a low-dimensional key vector codebook that meets a dimension matching condition from the various low-dimensional key vector codebooks as a target low-dimensional key vector codebook; Determining a key vector index according to the target low-dimensional key vector codebook and the low-dimensional key vector; Sort the determined key vector indexes according to the determined initial dimension information to obtain a key vector index sequence; The key vector index sequence is compressed to generate key vector index information.

5. The method according to claim 3, wherein: The step of quantizing and caching each low-dimensional value vector set obtained according to each low-dimensional value vector codebook included in the low-dimensional key value vector codebook to generate each value vector index information includes: For each low-dimensional value vector set in the respective low-dimensional value vector sets, the following processing steps are performed: For each low-dimensional value vector in the set of low-dimensional value vectors, the following quantization caching steps are performed: Determining initial dimension information corresponding to the low-dimensional value vector; Selecting, according to the determined initial dimension information, a low-dimensional valued vector codebook that satisfies a dimension matching condition from the respective low-dimensional valued vector codebooks as a target low-dimensional valued vector codebook; Determining a value vector index according to the target low-dimensional value vector codebook and the low-dimensional value vector; According to the determined initial dimension information, the determined value vector indexes are sorted to obtain a value vector index sequence; The value vector index sequence is compressed to generate value vector index information.

6. The method according to claim 4, wherein: The determining the key vector index according to the target low-dimensional key vector codebook and the low-dimensional key vector includes: For each central low-dimensional key vector included in the target low-dimensional key vector codebook, determining a spatial distance between the central low-dimensional key vector and the low-dimensional key vector; Sort the determined spatial distances to obtain a spatial distance sequence; Determine the central low-dimensional key vector corresponding to the first spatial distance in the spatial distance sequence as the target central low-dimensional key vector; The index corresponding to the target center low-dimensional key vector is determined as the key vector index.

7. The method according to claim 5, wherein: The determining a value vector index according to the target low-dimensional value vector codebook and the low-dimensional value vector includes: For each central low-dimensional value vector included in the target low-dimensional value vector codebook, determining a spatial distance between the central low-dimensional value vector and the low-dimensional value vector; Sort the determined spatial distances to obtain a spatial distance sequence; Determine the central low-dimensional value vector corresponding to the first spatial distance in the spatial distance sequence as the target central low-dimensional value vector; The index corresponding to the target center low-dimensional value vector is determined as the value vector index.

8. A key-value cache compression device for ultra-long context, comprising: A first acquisition unit is configured to acquire a sample high-dimensional key-value vector set, wherein the sample high-dimensional key-value vectors in the sample high-dimensional key-value vector set include a sample high-dimensional key vector and a sample high-dimensional value vector, and the sample high-dimensional key vector and the sample high-dimensional value vector have the same dimension; An offline training unit is configured to perform offline training processing on the sample high-dimensional key-value vector set to generate a low-dimensional key-value vector codebook, wherein the low-dimensional key-value vector codebook includes each low-dimensional key vector codebook and each low-dimensional value vector codebook; A second acquisition unit is configured to acquire each real-time key-value vector generated by the large language model during the inference phase; a quantization cache unit configured to perform quantization cache processing on each of the real-time key-value vectors according to the low-dimensional key-value vector codebook to generate each key-value index information, wherein the generated key-value index information corresponds to the real-time key-value vector; a writing unit configured to write each generated key-value index information into the quantized key-value buffer; The updating unit is configured to update the cache information in the quantization key value buffer according to the key value index information written in the quantization key value buffer and the word segmentation unit generated by the large language model during the decoding process of the large language model.

9. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images