Kv cache method, apparatus, device, medium, and program product

Abstract

This application provides a key-value (KV) caching method, apparatus, device, medium, and program product, relating to the field of artificial intelligence technology, to alleviate the pressure of video memory usage, thereby supporting long-sequence inference with longer context, higher throughput, and more stable latency under limited video memory. The specific technical solution is as follows: based on the lexical input sequence of the target file, at least one KV block is obtained; based on the importance information of the lexical corresponding to each KV block, the first importance quantization value of each KV block is calculated; the target level threshold range of the first importance quantization value of each KV block is determined; each KV block is stored in the storage level corresponding to the target level threshold range of the first importance quantization value of each KV block. This application is applied to the KV caching management scenario in the inference process of large language models.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a KV caching method, apparatus, device, medium, and program product. Background Technology

[0002] As the scale of Large Language Models (LLMs) continues to grow, the issue of GPU memory usage during model inference becomes increasingly prominent. In techniques for optimizing key-value pairs (KV), all key-value pairs are typically cached in the high-bandwidth memory (HBM) of a high-speed but expensive graphics processing unit (GPU). When the cache is full, a simple strategy is used to passively evict some KV pairs to main memory.

[0003] However, in the methods described above, the process of generating tokens using key-value vectors during the inference phase requires loading the key-value vectors, which may cause a significant delay, leading to "stuttering" in token generation, and even interrupting the generation of long sequences due to insufficient GPU memory. This limits the concurrency capabilities of the inference phase, making the generation of long sequences prone to interruption. Summary of the Invention

[0004] This application provides a KV caching method, apparatus, device, medium, and program product to alleviate the pressure of video memory usage, so as to support long-sequence inference with longer context, higher throughput, and more stable latency under limited video memory.

[0005] In a first aspect, embodiments of this application provide a key-value (KV) caching method, which includes: obtaining at least one key-value (KV) block based on a lexical input sequence of a target file, wherein a KV block includes a preset number of KV vectors corresponding to lexical units; calculating a first importance quantization value for each KV block based on the importance information of the lexical units corresponding to each KV block, wherein the importance information includes attention information, time decay information, access frequency information, semantic importance information, and position information; determining the target level threshold range where the first importance quantization value of each KV block is located; and storing each KV block to a storage level corresponding to the target level threshold range where the first importance quantization value of each KV block is located, wherein different storage levels correspond to different level threshold ranges and different storage levels correspond to different quantization precipitates.

[0006] The technical solution provided in this application brings at least the following beneficial effects: The KV caching device can cache the KV vectors of words in the target file in the form of KV blocks. Furthermore, the KV caching device can calculate the importance quantization value of each KV block, and store each KV block to the corresponding storage level based on the importance quantization value of the KV block. Different storage levels correspond to different quantization precisions. That is, the KV caching device can adaptively store KV blocks according to their importance and select quantization precision for them. This can reduce or avoid the situation where important KVs are over-quantized and lose precision, and unimportant KVs occupy high-precision resources. In this way, the pressure of video memory occupation can be alleviated, so as to support long sequence inference with longer context, higher throughput, and more stable latency under limited video memory.

[0007] One possible implementation involves calculating the first importance quantization value of each KV block based on the importance information of the lexical units corresponding to each KV block, including: obtaining the importance information of each lexical unit corresponding to the first KV block, wherein the first KV block is one of at least one KV blocks; performing a weighted summation of the importance information of each lexical unit based on a preset importance weight to obtain a second importance quantization value of each lexical unit; and determining the second importance quantization value with the largest value among the at least one second importance quantization values ​​as the first importance quantization value of the first KV block.

[0008] Another possible implementation, wherein storing each KV block to the storage level corresponding to the target level threshold range of the first importance quantization parameter of each KV block, includes: storing the second KV block to the first storage level when the first importance quantization value of the second KV block is within the first level threshold range; storing the second KV block to the second storage level when the first importance quantization value of the second KV block is within the second level threshold range; and storing the second KV block to the third storage level when the first importance quantization value of the second KV block is within the third level threshold range; wherein the lower limit of the first level threshold range is equal to the upper limit of the second level threshold range, and the lower limit of the second level threshold range is equal to the upper limit of the third level threshold range; the second KV block is one of at least one KV block; the KV block access latency of the first storage level is less than the KV block access latency of the second storage level, and the KV block access latency of the second storage level is less than the KV block access latency of the third storage level.

[0009] Another possible implementation is that, after calculating the first importance quantization value of each KV block based on the importance information of the lexical corresponding to each KV block, the KV caching method further includes: assigning quantization precision to each KV block based on the first importance quantization value and a preset mapping table, wherein the preset mapping table includes at least one importance quantization value and at least one quantization precision that maps one-to-one with the at least one importance quantization value.

[0010] Another possible implementation is that, after storing each KV block to the storage level corresponding to the target level threshold range of the first importance quantization parameter of each KV block, the KV caching method further includes: predicting the third KV block used to generate the second KV block during the process of generating the first lexical based on the target file, wherein the second KV block is at least one lexical after the first KV block, and the third KV block is one or more KV blocks among at least one KV block; and storing the third KV block to the first storage level.

[0011] Another possible implementation, wherein storing the third KV block to the first storage level, includes: generating a prefetch instruction for the third KV block when the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold and the third KV block is stored in the second or third storage level, the prefetch instruction being used to instruct the adjustment of the storage level of the KV block; and storing the third KV block to the first storage level based on the prefetch instruction.

[0012] Another possible implementation involves generating a prefetch instruction for the third KV block when the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold and the third KV block is stored in the second or third storage level. This includes: calculating a prefetch benefit value for the third KV block when the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold and the third KV block is stored in the second or third storage level, where the prefetch benefit value represents the benefit brought about by adjusting the storage level of the third KV block; and generating a prefetch instruction for the third KV block when the product of the third importance quantization value and the prefetch benefit value is greater than the pre-replacement value, where the pre-replacement value represents the cost required to adjust the storage level of the third KV block.

[0013] Secondly, embodiments of this application provide a KV caching device, including: an acquisition module, a calculation module, a determination module, and a storage module; the acquisition module is used to obtain at least one KV block based on the lexical input sequence of a target file, and a KV block includes KV vectors corresponding to a preset number of lexical units; the calculation module is used to calculate a first importance quantization value for each KV block based on the importance information of the lexical units corresponding to each KV block obtained by the acquisition module, the importance information including attention information, time decay information, access frequency information, semantic importance information, and position information; the determination module is used to determine the target level threshold range where the first importance quantization value of each KV block obtained by the calculation module is located; the storage module is used to store each KV block to a storage level corresponding to the target level threshold range where the first importance quantization value of each KV block is located, the level threshold range corresponding to different storage levels is different, and the quantization precision corresponding to different storage levels is different.

[0014] In one possible implementation, the acquisition module is specifically used to acquire the importance information of each word element corresponding to the first KV block, wherein the first KV block is one of at least one KV blocks; the calculation module is specifically used to perform a weighted summation of the importance information of each word element acquired by the acquisition module based on a preset importance weight to obtain a second importance quantization value for each word element; and the determination module is used to determine the second importance quantization value with the largest value among the at least one second importance quantization values ​​calculated by the calculation module as the first importance quantization value of the first KV block.

[0015] Another possible implementation is that the aforementioned storage module is specifically used to store the second KV block to the first storage level when the first importance quantization value of the second KV block is within the first level threshold range; or, when the first importance quantization value of the second KV block is within the second level threshold range, store the second KV block to the second storage level; or, when the first importance quantization value of the second KV block is within the third level threshold range, store the second KV block to the third storage level; wherein the lower limit of the first level threshold range is equal to the upper limit of the second level threshold range, and the lower limit of the second level threshold range is equal to the upper limit of the third level threshold range; the second KV block is one of at least one KV block; the KV block access latency of the first storage level is less than the KV block access latency of the second storage level, and the KV block access latency of the second storage level is less than the KV block access latency of the third storage level.

[0016] Another possible implementation, the KV caching device provided in this application embodiment further includes: a processing module; the processing module is used to calculate the first importance quantization value of each KV block based on the importance information of the lexical corresponding to each KV block, and then assign quantization precision to each KV block based on the first importance quantization value and a preset mapping table, the preset mapping table including at least one importance quantization value and at least one quantization precision that is mapped one-to-one with the at least one importance quantization value.

[0017] Another possible implementation, the KV caching device provided in this application embodiment further includes: a prediction module; the prediction module is used to predict the third KV block used to generate the second word during the process of generating the first word based on the target file after storing each KV block to the storage level corresponding to the target level threshold range of the first importance quantization parameter of each KV block, wherein the second word is at least one word after the first word, and the third KV block is one or more KV blocks among at least one KV block; the storage module is further used to store the third KV block predicted by the prediction module to the first storage level.

[0018] Another possible implementation, the KV caching device provided in this application embodiment further includes: a generation module; the generation module is used to generate a prefetch instruction for the third KV block when the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold and the third KV block is stored in the second storage level or the third storage level, the prefetch instruction being used to instruct the adjustment of the storage level of the KV block; the storage module is specifically used to store the third KV block in the first storage level based on the prefetch instruction generated by the generation module.

[0019] In another possible implementation, the aforementioned calculation module is specifically used to calculate the prefetch benefit value of the third KV block when the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold and the third KV block is stored in the second or third storage level. This prefetch benefit value is used to characterize the benefit brought about by adjusting the storage level of the third KV block. The aforementioned generation module is specifically used to generate a prefetch instruction for the third KV block when the product of the third importance quantization value and the prefetch benefit value calculated by the calculation module is greater than the pre-replacement value. This pre-replacement value is used to characterize the cost required to adjust the storage level of the third KV block.

[0020] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the method of the first aspect described above.

[0021] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a computer, implement the method of the first aspect described above.

[0022] Fifthly, this application provides a computer program product stored in a storage medium, which, when executed by a computer, implements the method described in the first aspect.

[0023] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0024] The beneficial effects of the second to sixth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description

[0025] Figure 1 A schematic diagram of a network architecture for an application of a KV caching method provided in this application embodiment;

[0026] Figure 2A flowchart illustrating a KV caching method provided in an embodiment of this application;

[0027] Figure 3 A flowchart illustrating another KV caching method provided in this application embodiment;

[0028] Figure 4 A flowchart illustrating yet another KV caching method provided in this application embodiment;

[0029] Figure 5 A flowchart illustrating yet another KV caching method provided in this application embodiment;

[0030] Figure 6 A flowchart illustrating yet another KV caching method provided in this application embodiment;

[0031] Figure 7 A flowchart illustrating yet another KV caching method provided in this application embodiment;

[0032] Figure 8 This is a schematic diagram of the structure of a KV caching system provided in an embodiment of this application;

[0033] Figure 9 This is a schematic diagram of the structure of a KV buffer device provided in an embodiment of this application;

[0034] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The KV caching method, apparatus, device, medium, and program products provided in this application will now be described in detail with reference to the accompanying drawings.

[0036] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0037] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0038] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0039] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0040] This application provides a key-value (KV) caching method, apparatus, device, medium, and program product that can be applied to KV caching management scenarios in the reasoning process of large language models. It is particularly suitable for memory optimization and inference acceleration in long sequence generation scenarios.

[0041] As the scale of large language models continues to grow, the issue of GPU memory consumption during model inference becomes increasingly prominent. In the autoregressive generation process, each newly generated token needs to have its corresponding key-value pair (key and value vector) cached for subsequent attention calculations. For models with a large number of parameters, the increase in sequence length will cause a large amount of GPU memory consumption in the KV cache, severely limiting the number of concurrent requests and the maximum sequence length.

[0042] In related technologies, all key-value (KV) caches can be placed in high-speed but expensive GPU HBM. Then, when the cache is full, a simple strategy can be used to passively evict some KV values ​​to main memory. However, if they are needed later, reloading will cause a long delay, resulting in "stuttering" of token generation, and may even interrupt the generation of long sequences due to insufficient video memory.

[0043] However, most quantization schemes in related technologies are statically configured, that is, the same fixed precision is used for all key-value blocks. This ignores the different impacts of key-value pairs with different attention heads and different position tokens on the generation quality. This may result in important key-value pairs being over-quantized and losing precision, while unimportant key-value pairs occupy high-precision resources.

[0044] Thus, high memory usage and static quantization precision lead to limited concurrency capabilities and technical problems such as the easy interruption of long sequence generation.

[0045] To address the aforementioned technical problems, embodiments of this application provide a KV caching method, apparatus, device, medium, and program product. The KV caching apparatus can cache the KV vectors of terms in a target file in the form of KV blocks. Furthermore, the KV caching apparatus can calculate the importance quantization value of each KV block, and store each KV block to the corresponding storage level based on the importance quantization value of the KV block. Different storage levels correspond to different quantization precisions, that is, the KV caching apparatus can adaptively store KV blocks according to their importance and select quantization precision for them. This can reduce or avoid the situation where important KVs are over-quantized and lose precision, and unimportant KVs occupy high-precision resources. In this way, the pressure on video memory can be alleviated, so as to support long sequence inference with longer context, higher throughput, and more stable latency under limited video memory.

[0046] The following description, in conjunction with the accompanying drawings, details the KV caching method, apparatus, device, medium, and program products provided in the embodiments of this application.

[0047] Figure 1 This illustration shows a network architecture for a KV caching method provided in an embodiment of this application. For example... Figure 1 As shown, the network architecture includes a KV cache device 101 and a terminal device 102. The KV cache device 101 and the terminal device 102 are interconnected.

[0048] In some embodiments, the KV caching device 101 may be a server, a computer, or a processor or processing unit within a server or computer. The server may be a single server or a server cluster consisting of multiple servers. It should be noted that the specific device form of the KV caching device 101 is not limited in the embodiments of this application. Figure 1 The example shown is a single server using the KV cache device 101.

[0049] In some embodiments, the terminal device may be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., and the embodiments of this application do not specifically limit it. Figure 1 The example shown is a mobile phone, with terminal device 102 as an example.

[0050] In some embodiments, the terminal device 102 can upload a target file, and the KV caching device 101 can receive the target file uploaded by the terminal device 102 and obtain at least one KV block based on the lexical input sequence of the target file. Each KV block includes KV vectors corresponding to a preset number of lexical units. Then, the KV caching device 101 can calculate the first importance quantization value of each KV block based on importance information such as attention information, time decay information, access frequency information, semantic importance information, and positional information of the lexical units corresponding to each KV block, to determine the target level threshold range where the first importance quantization value of each KV block is located. Each KV block is then stored in the storage level corresponding to the target level threshold range where the first importance quantization parameter of each KV block is located. Different storage levels correspond to different level threshold ranges, and different storage levels correspond to different quantization precisions.

[0051] It should be noted that the network architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As network architectures evolve, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0052] See Figure 2 This is a flowchart illustrating a KV caching method provided in an embodiment of this application. Figure 2 As shown, the KV caching method provided in this application embodiment can be implemented by the above-mentioned KV caching device, specifically including the following steps 201 to 204.

[0053] Step 201: The KV caching device obtains at least one KV block based on the word input sequence of the target file.

[0054] In some embodiments, a key-value block may include a preset number of key-value vectors corresponding to tokens.

[0055] For example, the KV caching device can receive a sequence of paper text (100K Tokens) as input, divide the input sequence into multiple micro-batches (e.g., 256 Tokens per batch) to obtain at least one KV block as described above.

[0056] Step 202: The KV caching device calculates the first importance quantization value of each KV block based on the importance information of the tokens corresponding to each KV block.

[0057] In some embodiments, the aforementioned importance information may include attention information, time decay information, access frequency information, semantic importance information, and location information.

[0058] In some embodiments, the KV caching device may calculate the aforementioned first importance quantification value through an importance evaluation module.

[0059] In some embodiments, the aforementioned first importance quantification value can be used to assess the importance of KV blocks, thereby allowing for assessment based on the predicted likelihood of the KV block being accessed in future generation steps and its potential impact on generation quality. A high score indicates that the KV block is crucial for subsequent generation and should be retained in high-precision hot storage, while a low score allows for degradation or compression.

[0060] In some embodiments, combined with Figure 2 ,like Figure 3 As shown, step 202 can be implemented through steps 202a to 202c.

[0061] Step 202a: The KV caching device obtains the importance information of each word corresponding to the first KV block.

[0062] In some embodiments, the first KV block mentioned above may be one of at least one KV block.

[0063] In some embodiments, the importance assessment module can extract importance information for each Token Key Value (KV).

[0064] In some embodiments, the importance information of each lexical unit may include attention information. Time decay information Access frequency information Semantic importance information and location information .

[0065] In some embodiments, the above attention information Attention contribution, or KV vector, is a key dimension for evaluating quality influence. Specifically, it can be evaluated by analyzing the weight of this KV vector in historical attention calculations.

[0066] In some embodiments, the above attention information It can be the first Each token at time step Attention score (normalized to) This reflects the degree of attention the token receives from subsequent generation.

[0067] In some embodiments, the normalized attention score can be calculated using the following formula (1).

[0068] Formula (1)

[0069] in, It can be used to represent the raw attention score, i.e. , specifically It can be used to represent the index of historical tokens that are being followed. , It can be used to represent the attention head index;

[0070] It can be used to represent the minimum attention score in the current batch, i.e. ;

[0071] It can be used to represent the maximum attention score in the current batch, i.e. ;

[0072] It can be used to represent numerical stability. .

[0073] In some embodiments, the above time decay information This refers to time-order decay. The lifetime can be the time step since the KV vector was created. The importance of older KV vectors should decay over time unless they have high access popularity or attention contributions.

[0074] In some embodiments, the above time decay information It can be the first Each token at time step The time decay factor reflects the time freshness of the token.

[0075] In some embodiments, the above time decay information It can be calculated using the following formula (2).

[0076] Formula (2)

[0077] in, It can be used to represent the maximum sequence length;

[0078] It can be used to represent the time decay coefficient, which can be 2 to 5. The larger the value of the time decay coefficient, the faster the decay, and the more likely the token is to receive attention in the near future.

[0079] In some embodiments, the above access frequency information The access popularity is determined by counting the number of times a key-value vector has been accessed within the most recent time window; frequently accessed key-value vectors receive higher scores.

[0080] In some embodiments, the above access frequency information It can be the first Each token at time step Access frequency statistics (normalized to) This reflects the extent to which the token is used.

[0081] In some embodiments, the normalized access frequency information can be calculated using the following formula (3).

[0082] Formula (3)

[0083] in, It can be used to represent the original access count, such as , specifically For the indicator function, when Token Visited at time step The value is 1 when the access is active and 0 when the access is not active.

[0084] It can be used to represent weighted access counts. , specifically ; It can be used to represent the maximum sequence length. It can be used to represent the frequency attenuation coefficient, and can be taken as 1 to 2;

[0085] It can be used to represent the minimum access count of the current batch, i.e. ;

[0086] It can be used to represent the maximum access count of the current batch, i.e. , ;

[0087] It can be used to represent numerical stability .

[0088] In some embodiments, the above semantic importance information This refers to semantic importance. Different categories of tokens have different semantic importance.

[0089] In some embodiments, the above semantic importance information It can be the first Each token at time step Semantic importance (normalized to) ), capturing the semantic / structural importance of the token.

[0090] In some embodiments, the semantic importance after normalization can be calculated using the following formula (4).

[0091] Formula (4)

[0092] in, It can be identified based on a special token. , For indicator functions. When the category For named entities (person names, place names, organization names, etc.), Numeric / time expressions ; punctuation marks (period, question mark, etc.), Special markers ([CLS], [SEP], <bos> , <eos>wait), Common vocabulary ;

[0093] It can be used to represent the minimum semantic importance of the current batch, i.e. ;

[0094] It can be used to represent the maximum semantic importance of the current batch, i.e. , ;

[0095] It can be used to represent numerical stability .

[0096] In some embodiments, the above location information Locality can be captured by introducing relative distance.

[0097] In some embodiments, the above location information It can be the first Each token at time step Positional scores (normalized to) (This captures both relative distance and absolute position effects.)

[0098] In some embodiments, the normalized position score can be calculated using the following formula (5).

[0099] Formula (5)

[0100] in, It can be used to represent raw positional fractions, such as , specifically This can be used to represent the absolute importance of a token's position in a sequence;

[0101] It can be used to represent the distance between the token and the current generation location;

[0102] It can be used to represent the current generation position. With Token Position The relative distance;

[0103] It can be used to represent the maximum sequence length;

[0104] It can be used to represent the distance attenuation coefficient, and can be taken as 2 to 4;

[0105] It can be used to represent the minimum positional score in the current batch. ;

[0106] It can be used to represent the maximum positional score in the current batch. , ;

[0107] It can be used to represent numerical stability .

[0108] Step 202b: The KV caching device performs a weighted summation of the importance information of each word based on a preset importance weight to obtain the second importance quantification value of each word.

[0109] In some embodiments, the aforementioned second importance quantification value can be used to assess the importance of a token in order to predict the likelihood that the KV block corresponding to the token will be accessed in future generation steps and its potential impact on generation quality.

[0110] Understandably, for the second most important quantification value, a high score means that the token is crucial to subsequent generation and should be kept in high-precision hot storage, while a low score allows for degradation or compression.

[0111] In some embodiments, the aforementioned second importance quantification value can be obtained through... The calculation yielded the following result. , It can be used to represent weighting coefficients. , .

[0112] Step 202c: The KV buffer device determines the second importance quantization value with the largest value among at least one second importance quantization value as the first importance quantization value of the first KV block.

[0113] In some embodiments, the KV buffer device can be of a fixed size (e.g. (Each token) is divided into blocks to calculate the KV block importance. , This is the block ID.

[0114] It should be noted that the token KV vectors within the same KV block are physically stored contiguously.

[0115] In this way, the KV cache device can use information from multiple dimensions to calculate the importance quantization value of each word in the KV block, thereby obtaining the importance quantization value of the KV block. Based on the importance quantization value of the KV block, each KV block can be stored in the corresponding storage level, reducing or avoiding the situation where important KV is over-quantized and loses precision, and unimportant KV occupies high-precision resources, thus alleviating the pressure on video memory usage.

[0116] Step 203: The KV cache device determines the target level threshold range where the first importance quantization value of each KV block is located.

[0117] Step 204: The KV caching device stores each KV block to the storage level corresponding to the target level threshold range of the first importance quantization parameter of each KV block.

[0118] In some embodiments, different storage tiers may correspond to different tier threshold ranges.

[0119] In some embodiments, combined with Figure 2 ,like Figure 4 As shown, step 204 can be implemented through steps 204a to 204c.

[0120] Step 204a: If the first importance quantization value of the second KV block is within the first level threshold range, the KV cache device stores the second KV block to the first storage level.

[0121] Step 204b: If the first importance quantization value of the second KV block is within the second level threshold range, the KV cache device stores the second KV block to the second storage level.

[0122] Step 204c: If the first importance quantization value of the second KV block is within the third level threshold range, the KV cache device stores the second KV block to the third storage level.

[0123] In some embodiments, the lower limit of the first-level threshold range may be equal to the upper limit of the second-level threshold range.

[0124] In some embodiments, the lower limit of the second-level threshold range may be equal to the upper limit of the third-level threshold range.

[0125] In some embodiments, the second KV block mentioned above may be one of at least one KV block.

[0126] In some embodiments, the KV block access latency of the first storage level may be less than the KV block access latency of the second storage level.

[0127] In some embodiments, the KV block access latency of the second storage level may be less than that of the KV block access latency of the third storage level.

[0128] In some embodiments, the first storage level can be hot storage (GPU HBM / video memory, etc.), which can be used to store key-value caches of highly important tokens with low access latency.

[0129] In some embodiments, the second storage tier described above can be warm storage (CPU DRAM / memory, etc.), that is, it can be used to store KV caches of medium importance tokens, accessible via PCIe.

[0130] In some embodiments, the third storage tier can be cold storage (SSD / HBF, etc.), which can be used to store low-importance tokens as a key-value cache with high access latency.

[0131] In some embodiments, the KV caching device can implement multi-level caching of KV blocks using Table 1 below.

[0132] Table 1 Hierarchical Storage Parameter Table

[0133]

[0134] In some embodiments, the KV caching device can set a hierarchy threshold. and To quantify the importance value The token key-value pair is marked as hot data and stored in the L1 layer; The token key-value pair is marked as warm data and stored in the L2 layer; the importance value is quantified. The token key-value pair is marked as cold data and stored in layer L3.

[0135] In this way, the KV cache device can store each KV block to the corresponding storage level based on the importance quantization value of the KV block, and can adaptively store KV blocks according to their importance. This can reduce or avoid the situation where important KV is over-quantized and loses precision, while unimportant KV occupies high-precision resources. This can alleviate the pressure on video memory usage and support long-sequence inference with longer context, higher throughput, and more stable latency under limited video memory.

[0136] In some embodiments, different storage tiers may correspond to different quantization precisions.

[0137] In the KV caching method provided in this application, the KV caching device can cache the KV vectors of terms in the target file in the form of KV blocks. Furthermore, the KV caching device can calculate the importance quantization value of each KV block, and store each KV block to the corresponding storage level based on the importance quantization value of the KV block. Different storage levels correspond to different quantization precisions. That is, the KV caching device can adaptively store KV blocks according to their importance and select quantization precision for them. This can reduce or avoid the situation where important KVs are over-quantized and lose precision, and unimportant KVs occupy high-precision resources. In this way, the pressure of video memory occupation can be alleviated, so as to support long sequence inference with longer context, higher throughput, and more stable latency under limited video memory.

[0138] In some embodiments, the KV caching device can manage multi-level storage pools, has a unified address mapping table, and maintains the physical location (storage level + intra-block offset), current quantization precision, importance quantization value, and scheduling records of each Token KV. It is also responsible for scheduling KV blocks.

[0139] In some embodiments, the above scheduling may include, but is not limited to, at least one of the following:

[0140] Correction trigger: Asynchronously migrate key-value blocks whose importance quantization values ​​do not meet the preset precision-importance mapping strategy from the current storage layer to the storage layer that meets the mapping strategy, and call the adaptive quantization module to perform precision conversion.

[0141] Prefetch Trigger: Based on the prefetch instruction from the prediction prefetch module, if the target block to be accessed is located in low-level storage or is of low precision, an asynchronous prefetch operation is initiated to migrate the KV block between different storage levels. At the same time, during the migration process, an adaptive quantizer is called to perform precision conversion.

[0142] Capacity Trigger: Real-time monitoring of capacity utilization at each level. When the capacity of L1 / L2 layer reaches a certain threshold, such as 90%, the storage layer of the current KV block is reduced to L2 / L3 layer, and the adaptive quantization module is triggered to perform precision conversion.

[0143] In some implementations, since the key-value pairs of each micro-batch are converted and distributed to different storage levels immediately after generation, the GPU memory can retain only the intermediate computational state of the current micro-batch, and the peak memory usage is much smaller than that of the full sequence key-value cache.

[0144] In some embodiments, since the micro-batch size is not aligned with the block size, the KV cache device can maintain a small block assembly buffer (in CPU memory) to temporarily store tokens that are not yet full, so that they can be written to the corresponding storage level after a KV block is assembled.

[0145] In some embodiments, after step 202 above, the KV caching method provided in this application embodiment may further include step 301 below.

[0146] Step 301: The KV cache device assigns quantization precision to each KV block based on the first importance quantization value and a preset mapping table.

[0147] In some embodiments, the preset mapping table may include at least one importance quantization value and at least one quantization precision that maps one-to-one with the at least one importance quantization value.

[0148] It is understandable that the KV cache device can receive the importance quantization value calculated by the importance assessment module, and automatically determine and perform precision conversion such as quantization and dequantization of KV blocks according to the preset precision-importance mapping strategy (i.e. the preset mapping table mentioned above).

[0149] In some embodiments, the above precision-importance mapping strategy may include:

[0150] (1) Importance quantification value The token key-value pair is marked as hot data, and high precision, such as FP16, is used for storage.

[0151] (2) Importance quantification value The token key-value pair is marked as warm data, and the storage precision adopts quantization precision, such as INT8;

[0152] (3) Importance quantification value The token key-value pair is marked as cold data, and the storage precision uses deep compression precision, such as INT4.

[0153] In some embodiments, the KV caching device can perform corresponding precision conversions such as quantization and dequantization based on the scheduling requirements of the hierarchical storage management module and in accordance with the multi-level caching strategy.

[0154] In some embodiments, the KV cache device can construct a hierarchical storage system by probing the hardware environment, identifying and quantifying the performance and capacity of different storage media. For example, the hierarchical storage system can be as shown in Table 2 below.

[0155] Table 2 Hierarchical Storage Parameter Table

[0156]

[0157] It should be noted that the KV cache device can independently quantize parameters for each storage level and monitor the mean square error before and after quantization in real time. When the mean square error exceeds a certain threshold, it will automatically fall back to a higher precision to achieve quantization error control.

[0158] In this way, the KV cache device can adaptively select the quantization precision for KV blocks according to their importance, so as to perform precision transformations such as quantization and dequantization when migrating between different storage levels, thereby maximizing the efficiency of video memory compression while ensuring the accuracy of critical information.

[0159] In some embodiments, the KV caching device can maintain the storage location, current quantization precision, importance quantization value, and scheduling record of each Token KV through a unified address mapping table. Specifically, the field definitions of this unified address mapping table can be as shown in Table 3 below.

[0160] Table 3. Field Definitions of the Uniform Address Mapping Table

[0161]

[0162] In some embodiments, combined with Figure 2 ,like Figure 5 As shown, after step 204 above, the KV caching method provided in this application embodiment may further include the following steps 401 and 402.

[0163] Step 401: During the process of generating the first word based on the target file, the KV caching device predicts the third KV block used to generate the second word.

[0164] In some embodiments, the second lexical unit can be at least one lexical unit following the first lexical unit.

[0165] In some embodiments, the aforementioned third KV block may be one or more KV blocks from at least one KV block.

[0166] Step 402: The KV cache device stores the third KV block to the first storage level.

[0167] Understandably, a KV caching device can predict the KV blocks required for future generation steps and asynchronously prepare the data during the computation steps, thereby completely hiding storage access latency.

[0168] Specifically, the KV caching device can accurately predict the probability of KV blocks being accessed when generating the (t+1)th to (t+K)th tokens, and asynchronously prepare the most likely accessed KV cache entries from low-speed storage to high-speed video memory when generating the t-th token. Furthermore, the KV caching device can also record the actual access status of KV blocks during the inference step, whether the prefetch hits or not, and the latency caused by data not being ready.

[0169] In some embodiments, combined with Figure 2 ,like Figure 6 As shown, step 402 can be implemented through steps 402a and 402b.

[0170] Step 402a: When the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold and the third KV block is stored in the second or third storage level, the KV cache device generates a prefetch instruction for the third KV block.

[0171] In some embodiments, the above prefetch instructions can be used to instruct the adjustment of the storage level of the KV block.

[0172] In some embodiments, for the first Each token, the KV cache device can maintain its historical importance quantification value sequence. .

[0173] In some embodiments, the KV cache device can employ a lightweight time series model (such as exponential smoothing or a small-scale LSTM) to predict future data. Importance quantification of steps , .

[0174] In some embodiments, for key-value blocks stored in the L2 / L3 layer, the key-value caching device can calculate block-level predicted importance. .

[0175] In some embodiments, the KV caching device may set a prefetch threshold. and lead time (Based on the number of decoding steps), when stored in the KV block at layer L2 / L3, At that time, the corresponding prefetch instruction is generated and transmitted to the hierarchical storage management module.

[0176] In some embodiments, the aforementioned lead time ;in, It can be used to represent the access latency of the target storage level; It can be used to represent the time consumed by inverse quantization and error compensation; It can be used to represent the average decoding time per step; It can be used to represent a safety margin (e.g., 2 to 5 steps).

[0177] For example, if L3 layer access is delayed , , , Then lead time step.

[0178] In some embodiments, combined with Figure 2 ,like Figure 7 As shown, step 402a can be implemented through steps 402a1 and 402a2.

[0179] Step 402a1: If the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold and the third KV block is stored in the second or third storage level, the KV cache device calculates the prefetch revenue value of the third KV block.

[0180] Understandably, in order to avoid excessive prefetching consuming bandwidth and computing resources, KV cache devices can introduce cost-benefit analysis to generate prefetch instructions only when the prefetch decision conditions are met.

[0181] In some embodiments, the aforementioned prefetch benefit value can be used to characterize the benefit resulting from adjusting the storage level of the third KV block.

[0182] In some embodiments, the aforementioned pre-extraction revenue .in, It can be used to indicate the access latency of the current level of data. It can be used to represent the latency of target layer access.

[0183] Step 402a2: If the product of the third importance quantification value and the prefetch benefit value is greater than the pre-replacement value, the KV cache device generates a prefetch instruction for the third KV block.

[0184] In some embodiments, the aforementioned pre-replacement value can be used to characterize the cost required to adjust the storage level of the third KV block.

[0185] In some embodiments, the aforementioned pre-replacement value .in, It can be used to represent the storage space occupied by a prefetched data block. It can be used to represent the currently available data transmission bandwidth. It can be used to represent the time consumed by inverse quantization and error compensation. It can be used to represent the weighted sum of the importance of the data that was squeezed out (usually selecting the data with the lowest prediction score). It can be used to represent a reference time that unifies various items (e.g., 1ms). , These are weights used to adjust different types of costs.

[0186] In some embodiments, the above-mentioned prefetching decision conditions can be .

[0187] in, It can be used to represent a risk preference coefficient, used to control the degree of aggressiveness in pre-selection. The smaller the value, the more data is pre-collected. A value of 1 indicates that the benefit must exceed the cost.

[0188] In this way, the KV cache device can introduce cost-benefit analysis to generate prefetch instructions only when the prefetch decision conditions are met, thereby avoiding excessive prefetching that consumes bandwidth and computing resources.

[0189] Step 402b: The KV cache device stores the third KV block to the first storage level based on the prefetch instruction.

[0190] In some embodiments, the KV caching device can send the aforementioned prefetch instructions into an execution pipeline that runs in parallel with the computation flow, so as to prefetch the required KV blocks in advance when calculating the attention of the current token.

[0191] In some embodiments, the KV caching device can record the actual access status of the KV block in this inference step, whether the prefetch hit was successful, and the delay caused by data not being ready, through the prediction prefetch module.

[0192] In this way, the KV caching device can predict future changes in the importance of tokens based on historical importance sequences and trigger prefetching operations, effectively hiding low-speed storage access latency through cost-aware prefetching decisions.

[0193] In some embodiments, during the process of generating a new token in the KV cache device, its KV cache can be calculated, and a higher initial importance quantification value can be assigned to the current step through the hierarchical storage management module.

[0194] It should be noted that since the new token mentioned above belongs to a new block (a new block is created if the block size is full), it can be temporarily stored in the L1 layer with FP16 precision and the unified address mapping table is updated.

[0195] In some embodiments, the KV cache device may periodically trigger a background maintenance phase to perform at least one of the following operations:

[0196] Online learning and updates: Use the deviation between prediction and reality to adjust the access prediction model parameters and the weights in cost-benefit calculation of the prediction prefetch module, thereby improving prediction accuracy and optimizing resource investment efficiency;

[0197] Cache consistency check: The tiered storage management module checks whether the address mapping table is consistent with the actual data at each level, cleans up invalid cache, and schedules correction triggers and capacity triggers.

[0198] Cold data compression and backup: Consider recompressing cold data blocks that have not been accessed for a long time, or asynchronously backing them up to cloud storage.

[0199] Figure 8 This is a schematic diagram of a key-value (KV) caching system provided in an embodiment of this application. This application provides a KV caching method, the core idea of ​​which is: to evaluate the importance of tokens from multiple dimensions, to hierarchically store KV data in different levels of cache such as GPU memory, CPU memory, and SSD, and to adaptively select the quantization precision according to importance. Simultaneously, access pattern prediction is used to achieve prefetching and computation overlap, enabling long-sequence inference with longer context, higher throughput, and more stable latency under limited GPU memory. Figure 8 As shown, the KV caching system mainly includes an importance assessment module, a hierarchical storage management module, an adaptive quantization module, and a predictive prefetching module.

[0200] 1. Importance Assessment Module

[0201] It can be used to calculate a dynamic importance quantification value for each smallest key-value management unit. The score integrates factors such as attention contribution, temporal decay, access frequency, semantic importance, and location information.

[0202] Specifically, the main function of the aforementioned importance assessment module is to assess the importance of tokens and predict the likelihood of the KV block being accessed in future generation steps and its potential impact on generation quality.

[0203] A high score means that the token is crucial for subsequent generation and should be stored in high-precision hot storage, while a low score allows for degradation or compression.

[0204] The importance assessment module can extract and calculate the following features for each token key-value pair as a scoring basis:

[0205] Attention information Attention contribution, or KV vector, is a key dimension for evaluating quality impact. Specifically, it can be evaluated by analyzing the weight of this KV vector in historical attention calculations.

[0206] Time decay information This refers to time-order decay. The lifetime can be the time step since the KV vector was created. The importance of older KV vectors should decay over time unless they have high access frequency or attention contributions.

[0207] Access frequency information In other words, access popularity is determined by counting the number of times a key-value vector has been accessed within a recent time window; frequently accessed key-value vectors receive higher scores.

[0208] Semantic importance information This refers to semantic importance. Different categories of tokens have different semantic importance.

[0209] Location information Locality can be captured by introducing relative distance.

[0210] Importance quantification value calculation formula: ;Specifically:

[0211] Attention information It can be the first Each token at time step Attention score (normalized to) The attention score reflects the degree to which the token receives attention in subsequent generation. The normalized attention score can be calculated using the following formula (1).

[0212] Formula (1)

[0213] in, It can be used to represent the raw attention score, i.e. , specifically It can be used to represent the index of historical tokens that are being followed. , It can be used to represent the attention head index;

[0214] It can be used to represent the minimum attention score in the current batch, i.e. ;

[0215] It can be used to represent the maximum attention score in the current batch, i.e. ;

[0216] It can be used to represent numerical stability. .

[0217] Time decay information It can be the first Each token at time step The time decay factor reflects the token's freshness over time. Time decay information. It can be calculated using the following formula (2).

[0218] Formula (2)

[0219] in, It can be used to represent the maximum sequence length;

[0220] It can be used to represent the time decay coefficient, which can be 2 to 5. The larger the value of the time decay coefficient, the faster the decay, and the more likely the token is to receive attention in the near future.

[0221] Access frequency information It can be the first Each token at time step Access frequency statistics (normalized to) The normalized access frequency information can be calculated using the following formula (3).

[0222] Formula (3)

[0223] in, It can be used to represent the original access count, such as , specifically For the indicator function, when Token Visited at time step The value is 1 when the access is active and 0 when the access is not active.

[0224] It can be used to represent weighted access counts. , specifically ; It can be used to represent the maximum sequence length. It can be used to represent the frequency attenuation coefficient, and can be taken as 1 to 2;

[0225] It can be used to represent the minimum access count of the current batch, i.e. ;

[0226] It can be used to represent the maximum access count of the current batch, i.e. , ;

[0227] It can be used to represent numerical stability .

[0228] Semantic importance information It can be the first Each token at time step Semantic importance (normalized to) ), capturing the semantic / structural importance of the token. The normalized semantic importance can be calculated using the following formula (4).

[0229] Formula (4)

[0230] in, It can be identified based on a special token. , For indicator functions. When the category For named entities (person names, place names, organization names, etc.), Numeric / time expressions ; punctuation marks (period, question mark, etc.), Special markers ([CLS], [SEP], <bos> , <eos>wait), Common vocabulary ;

[0231] It can be used to represent the minimum semantic importance of the current batch, i.e. ;

[0232] It can be used to represent the maximum semantic importance of the current batch, i.e. , ;

[0233] It can be used to represent numerical stability .

[0234] Location information It can be the first Each token at time step Positional scores (normalized to) This captures both relative distance and absolute position effects. The normalized position score can be calculated using the following formula (5).

[0235] Formula (5)

[0236] in, It can be used to represent raw positional fractions, such as , specifically This can be used to represent the absolute importance of a token's position in a sequence;

[0237] It can be used to represent the distance between the token and the current generation location;

[0238] It can be used to represent the current generation position. With Token Position The relative distance;

[0239] It can be used to represent the maximum sequence length;

[0240] It can be used to represent the distance attenuation coefficient, and can be taken as 2 to 4;

[0241] It can be used to represent the minimum positional score in the current batch. ;

[0242] It can be used to represent the maximum positional score in the current batch. , ;

[0243] It can be used to represent numerical stability .

[0244] , Weighting coefficients , .

[0245] Importance assessment modules can be of a fixed size (e.g.) Divide the tokens into blocks and calculate the key-value block importance. , This is the block ID. Token key-value vectors within the same block are physically stored contiguously.

[0246] 2. Tiered Storage Management Module

[0247] (1) Manage multi-level storage such as GPU memory, host memory, and SSD. The KV cache in each storage layer is managed in blocks of fixed size (e.g., 64 tokens).

[0248] KV blocks are cached in different storage levels based on their importance quantification values:

[0249] Hot storage (GPU HBM / video memory, etc.): KV cache for storing highly important tokens, with low access latency.

[0250] Warm storage (CPU DRAM / memory, etc.): A key-value cache that stores medium-importance tokens and is accessed via PCIe.

[0251] Cold storage (SSD / HBF, etc.): Key-value cache for storing low-importance tokens, with higher access latency.

[0252] A multi-level caching strategy is adopted, and the specific storage parameters for each level are shown in Table 1 below:

[0253] Table 1 Hierarchical Storage Parameter Table

[0254]

[0255] Specific tiered storage strategy: setting thresholds and Quantify the importance value The token key-value pair is marked as hot data and stored in the L1 layer; The data is labeled as temperature data and stored in the L2 layer; The data is marked as cold data and stored in layer L3.

[0256] (2) KV Block Migration Scheduling: Manages multi-level storage pools, possesses a unified address mapping table, and maintains the physical location (storage level + intra-block offset), current quantization precision, importance quantization value, and scheduling records for each Token KV. It is also responsible for KV block scheduling. Its core scheduling strategies include, but are not limited to, at least one of the following:

[0257] ① Correction trigger: Asynchronously migrate the KV blocks whose importance quantization values ​​do not meet the preset precision-importance mapping strategy from the current storage layer to the storage layer that meets the mapping strategy, and call the adaptive quantization module to perform precision conversion.

[0258] ② Prefetch Trigger: Based on the prefetch instruction from the prediction prefetch module, if the target block to be accessed is located in low-level storage or is of low precision, an asynchronous prefetch operation is initiated to migrate the KV block between different storage levels. At the same time, during the migration process, an adaptive quantizer is called to perform precision conversion.

[0259] ③ Capacity Trigger: Real-time monitoring of capacity utilization at each level. When the capacity of L1 / L2 layer reaches a certain threshold, such as 90%, the storage layer of the current KV block is reduced to L2 / L3 layer, and the adaptive quantization module is triggered to perform precision conversion.

[0260] 3. Adaptive quantization module:

[0261] (1) Assign quantization precision to each KV block according to the importance quantization value, and support multi-level quantization, such as FP16, INT8, INT4, etc. Receive the importance quantization value, and automatically determine and perform precision conversion such as quantization and dequantization of KV blocks according to the preset precision-importance mapping strategy.

[0262] (2) Respond to the scheduling requirements of the hierarchical storage management module and perform corresponding precision conversions such as quantization and dequantization based on the multi-level caching strategy.

[0263] Precision-Importance Mapping Strategy:

[0264] Importance quantification value The token key-value pair is marked as hot data, and high precision, such as FP16, is used for storage.

[0265] Importance quantification value The data is labeled as temperature data and stored with quantization precision, such as INT8.

[0266] Importance quantification value The data is marked as cold data and stored using deep compression precision, such as INT4.

[0267] Quantization error control: Each layer has independent quantization parameters, and the mean square error before and after quantization is monitored in real time. When the mean square error exceeds a certain threshold, it automatically reverts to a higher precision.

[0268] 4. Prediction and prefetching module:

[0269] While the model is being computed, based on the importance prediction trend, the most likely key-value blocks to be accessed in the next decoding step, as well as key-value blocks that will not be accessed in the near future, are evaluated, and prefetch instructions are generated.

[0270] In this architecture: the key-value vector of each token is stored independently, and importance quantification values ​​are calculated per token. This is done with a fixed size (e.g., ...). Each token is divided into key-value blocks, serving as the smallest unit for storage management, quantization, migration, and prefetching. All tokens within a block are stored contiguously, sharing the same storage level and quantization precision. A unified address mapping table is block-indexed (not token-indexed), with each record corresponding to one block.

[0271] The predictive prefetch module can accurately predict the key-value cache items required for future generation steps and asynchronously prepare the data during the computation steps, thereby completely hiding storage access latency.

[0272] Specifically, when generating the t-th token, the system accurately predicts the likelihood of accessing the KV cache entries during the generation of the (t+1)-t+K-th tokens, and asynchronously prepares the most likely accessed KV cache entries from low-speed storage to high-speed video memory. Furthermore, it records the actual access status of KV blocks during the inference process, prefetch hits, and delays caused by data incompleteness.

[0273] For the Each token maintains a sequence of its historical importance quantification values. Lightweight time series models (such as exponential smoothing or small-scale LSTM) are used to predict the future. Importance quantification of steps , For KV blocks stored in L2 / L3 layers, calculate block-level predicted importance. .

[0274] Prefetch strategy: Set prefetch threshold and lead time (Based on decoding steps). A corresponding prefetch instruction is generated and transmitted to the hierarchical storage management module when the following conditions are met: for KV blocks stored in L2 / L3 layers, .

[0275] Among them, lead time ;in, It can be used to represent the access latency of the target storage level; It can be used to represent the time consumed by inverse quantization and error compensation; It can be used to represent the average decoding time per step; It can be used to represent a safety margin (e.g., 2 to 5 steps).

[0276] For example, if L3 layer access is delayed , , , Then lead time step.

[0277] To avoid excessive prefetching consuming bandwidth and computing resources, a cost-benefit analysis is introduced, and prefetch instructions are only generated when the prefetch decision conditions are met.

[0278] Pre-collected income .in, It can be used to indicate the access latency of the current level of data. It can be used to represent the latency of target layer access.

[0279] Pre-substitution value .in, It can be used to represent the storage space occupied by a prefetched data block. It can be used to represent the currently available data transmission bandwidth. It can be used to represent the time consumed by inverse quantization and error compensation. It can be used to represent the weighted sum of the importance of the data that was squeezed out (usually selecting the data with the lowest prediction score). It can be used to represent a reference time that unifies various items (e.g., 1ms). , These are weights used to adjust different types of costs.

[0280] Prefetching decision conditions can be .

[0281] in, It can be used to represent a risk preference coefficient, used to control the degree of aggressiveness in pre-selection. The smaller the value, the more data is pre-collected. A value of 1 indicates that the benefit must exceed the cost.

[0282] Prefetching decision conditions , This is a risk preference coefficient used to control the aggressiveness of pre-selection. The smaller the value, the more data is pre-collected. A value of 1 indicates that the benefit must exceed the cost.

[0283] The implementation process of the KV caching method provided in this application includes the following S1 to S3:

[0284] S1. Warm-up Phase: Before starting formal inference, the system is initialized and prepared.

[0285] (1) Detect the hardware environment, identify and quantify the performance and capacity of different storage media, and build a hierarchical storage system.

[0286] Layer L1 serves as hot storage for high-importance tokens (KV); Layer L2 serves as warm storage for medium-importance tokens (KV); and Layer L3 serves as cold storage for low-importance tokens (KV).

[0287] (2) Initialize the importance assessment module, hierarchical storage management module, adaptive quantization module and prediction prefetching module.

[0288] (3) Receive the input sequence, divide the input sequence into multiple micro-batches (e.g., 256 Tokens per batch), perform forward calculation of the model in sequence, generate the initial KV cache, and temporarily store the L1 layer.

[0289] (4) The adaptive quantization module automatically determines and executes the quantization of the KV block based on the importance quantization value and the preset precision-importance mapping strategy.

[0290] (5) The hierarchical storage management module allocates the KV cache to different storage levels according to the importance quantization value, and performs correction triggering and capacity triggering scheduling. The unified address mapping table maintains the storage location, current quantization precision, importance quantization value and scheduling record of each Token KV.

[0291] Because the key-value pairs (KV) of each micro-batch are immediately converted and distributed to different storage levels after generation, the GPU memory only retains the intermediate computational state of the current micro-batch, resulting in a peak memory usage far less than the full-sequence KV cache. Since the micro-batch size is not aligned with the block size, the system maintains a small block assembly buffer (in CPU memory) to temporarily store tokens that are not yet full, writing them to the target level only after a complete KV block has been assembled.

[0292] S2, Inference Execution Loop Phase: This phase executes repeatedly for each generated token. Its core idea is to parallelize the computation of the current step with the data preparation for the next step.

[0293] (1) Parallel evaluation and prediction: The importance evaluation module calculates the importance quantification value of the currently active KV block. The prediction prefetching module analyzes the current generation status and historical patterns, predicts the probability of accessing the KV block in the future, and performs cost-benefit analysis. It generates a prefetching instruction only when the prefetching decision conditions are met.

[0294] (2) Prefetching and Overlapping Computation: The hierarchical storage management module receives prefetching instructions and performs prefetching-triggered scheduling. The adaptive quantization module responds to the scheduling requirements of the hierarchical storage management module and performs corresponding precision conversions based on the multi-level caching strategy to ensure the uniformity of the precision required for computation. The prefetching-triggered scheduling instructions are sent to an execution pipeline that runs parallel to the computation flow, enabling the prefetching of the required key-value blocks when calculating the attention of the current token. The prediction prefetching module records the actual access status of key-value blocks in this inference step, whether the prefetching is successful, and the delay caused by data incompleteness.

[0295] (3) Generation and Update: Generate a new Token and calculate its KV cache. The hierarchical storage management module assigns a higher initial importance quantification value to the current step. This Token belongs to a new block (or a new block if the block size is full). It is temporarily stored in the L1 layer with FP16 precision and the unified address mapping table is updated.

[0296] S3, Backend Maintenance (Periodic Triggering) Phase:

[0297] (1) Online learning and updating: Use the deviation between prediction and reality to adjust the access prediction model parameters and the weights in cost-benefit calculation of the prediction prefetch module, improve prediction accuracy and optimize resource investment efficiency.

[0298] (2) Cache consistency check: The hierarchical storage management module checks whether the address mapping table is consistent with the actual data of each level, cleans up invalid cache, and schedules correction trigger and capacity trigger.

[0299] (3) Cold data compression and backup: Consider recompressing cold data blocks that have not been accessed for a long time, or asynchronously backing them up to cloud storage.

[0300] Thus, the KV caching method provided in this application embodiment can maximize the utilization of GPU memory for the efficient deployment and application of large-scale language models. It addresses the technical problems of high GPU memory consumption and static quantization accuracy in KV caching management schemes in large language model long-sequence inference scenarios through a hardware-software collaborative, data-value-aware joint optimization mechanism, which limits concurrency capabilities and makes long-sequence generation prone to interruption.

[0301] Specifically:

[0302] 1. An adaptive quantization precision allocation method based on importance perception is proposed, which calculates the multi-dimensional token importance quantization value.

[0303] 2. Coordinated scheduling of quantization and storage levels, adaptive quantization precision selection based on importance, and precision transformation such as quantization and dequantization when migrating between different storage levels, maximizing memory compression efficiency while ensuring the accuracy of key information.

[0304] 3. Active migration and prefetching based on importance prediction: A lightweight time series model is used to predict the importance changes of future tokens based on historical importance sequences and trigger prefetching operations. Cost-aware prefetching decisions effectively hide low-speed storage access latency.

[0305] It should be noted that the descriptions of each step S1 to S3 in this embodiment can be found in the descriptions in the above embodiments, and will not be repeated here.

[0306] To more clearly illustrate the purpose, technical solution, and advantages of this patent, the KV caching method of this application will be described below through specific embodiments.

[0307] Application scenarios and tasks:

[0308] This embodiment uses long document summarization of a large-scale language model as its application scenario. The open-source model LLaMA-65B is used as the base model, taking a scientific paper with 100,000 tokens as input and requiring the model to generate a summary of approximately 500 tokens. The hardware environment consists of a single NVIDIA A100 80GB GPU, equipped with 512GB of CPU memory and a 2TB NVMe SSD. The proposed KV cache hierarchical storage and adaptive quantization joint optimization method is employed.

[0309] Specific implementation steps:

[0310] 1. Preheating stage:

[0311] (1) Probe the hardware environment, identify and quantify the performance and capacity of different storage media, and construct a hierarchical storage system. See Table 2 for details:

[0312] Table 2 Implementation Example Hierarchical Storage Parameter Table

[0313]

[0314] (2) Initialize the importance assessment module, hierarchical storage management module, adaptive quantization module and prediction prefetching module.

[0315] Weighting coefficients in the formula for calculating the importance quantification value of the importance assessment module , , , ;

[0316] The hierarchical storage strategy of the hierarchical storage management module is shown in Table 2. , The KV cache block size is 64 tokens / block, and the capacity control scheduling threshold is set to 90%.

[0317] The adaptive quantization module supports FP16 to INT8 to INT4 conversion. , ;

[0318] The access prediction model for the predictive prefetch module uses exponential smoothing, with a prefetch threshold. and lead time (L3 level) (L2 layer). Parameters for cost-benefit analysis. , , , .

[0319] (3) Receive the input sequence of the paper (100K Tokens), divide the input sequence into multiple micro-batches (e.g., 256 Tokens per batch), perform forward calculation of the model in sequence, generate the initial KV cache, and temporarily store the L1 layer.

[0320] (4) The adaptive quantization module automatically determines and executes the quantization of the KV block based on the importance quantization value and the preset precision-importance mapping strategy.

[0321] (5) The hierarchical storage management module allocates the KV cache to different storage tiers according to the importance quantization value, and performs correction triggering and capacity triggering scheduling. The storage location, current quantization precision, importance quantization value and scheduling record of each Token KV are maintained through a unified address mapping table. The field definitions of the unified address mapping table are shown in Table 3.

[0322] Table 3. Field Definitions of the Uniform Address Mapping Table

[0323]

[0324] 2. Inference execution loop phase:

[0325] (1) Parallel evaluation and prediction:

[0326] The importance assessment module calculates the importance quantification value for the currently active key-value blocks;

[0327] The predictive prefetching module analyzes the current generation state and historical patterns to predict the future. The possibility of accessing the KV block is determined step by step, and a cost-benefit analysis is performed to generate the corresponding prefetch instructions.

[0328] (2) Prefetch scheduling and overlap computation:

[0329] The tiered storage management module receives prefetch instructions and performs prefetch trigger scheduling;

[0330] The adaptive quantization module responds to the scheduling requirements of the hierarchical storage management module and performs corresponding precision conversions based on the multi-level caching strategy to ensure the consistency of the precision required for computation.

[0331] The hierarchical storage management module adds prefetching tasks to an internal queue, and the asynchronous thread executes them immediately: when calculating the attention of the current token, the required key-value blocks are prefetched in advance;

[0332] The prediction prefetching module records the actual access status of the KV block in this inference step, whether the prefetch hit was successful, and the delay caused by data not being ready.

[0333] (3) Generation and Update: Generate a new Token and calculate its KV cache. The hierarchical storage management module assigns a higher initial importance quantification value to the current step. This Token belongs to a new block (if there are already 64, a new block is created). It is temporarily stored in the L1 layer with FP16 precision, and the unified address mapping table is updated.

[0334] 3. Backend maintenance phase (triggered every 100 steps):

[0335] (1) Online learning and updating: Based on the deviation between the prediction and the actual results over 100 steps, adjust the access prediction model parameters and the weights in the cost-benefit calculation of the prediction prefetch module. For example, adaptively adjust the prefetch threshold according to the prefetch hit rate. .

[0336] (2) Cache consistency check: The hierarchical storage management module checks the actual storage data of each level according to the address mapping table, cleans up invalid cache, and schedules correction trigger and capacity trigger.

[0337] (3) Cold data compression and backup: Scan the L3 layer storage data and find that a block has not been accessed for more than 10 minutes. Compress its precision from INT4 to INT2 to free up SSD space.

[0338] In this way, the KV caching device can support the generation of ultra-long sequences while maximizing memory utilization through three major mechanisms: importance-aware precision allocation, storage hierarchy and quantization coordinated scheduling, and prediction-based proactive migration prefetching. At the same time, it can maintain near-full-precision generation quality and controllable inference latency, providing a high-capacity and high-quality solution for the efficient service deployment of large-scale language models.

[0339] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there is no conflict, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0340] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0341] This application embodiment can divide the KV cache device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0342] In some embodiments, this application also provides a key-value (KV) caching device. The KV caching device may include one or more functional modules for implementing the KV caching method of the above method embodiments.

[0343] For example, Figure 9 This is a schematic diagram of a KV buffer device provided in an embodiment of this application. Figure 9 As shown, the KV cache device 900 includes: an acquisition module 901, a calculation module 902, a determination module 903, and a storage module 904;

[0344] The aforementioned acquisition module 901 is used to obtain at least one KV block based on the word input sequence of the target file, wherein a KV block includes a preset number of KV vectors corresponding to words.

[0345] The aforementioned calculation module 902 is used to calculate the first importance quantization value of each KV block based on the importance information of the lexical corresponding to each KV block obtained by the acquisition module 901. The importance information includes attention information, time decay information, access frequency information, semantic importance information, and position information.

[0346] The aforementioned determining module 903 is used to determine the target level threshold range of the first importance quantization value of each KV block obtained by the calculation module 902;

[0347] The aforementioned storage module 904 is used to store each KV block to the storage level corresponding to the target level threshold range of the first importance quantization parameter of each KV block. Different storage levels correspond to different level threshold ranges, and different storage levels correspond to different quantization accuracies.

[0348] The KV caching device provided in this application can cache the KV vectors of terms in the target file in the form of KV blocks. Furthermore, the KV caching device can calculate the importance quantization value of each KV block and store each KV block to the corresponding storage level based on the importance quantization value. Different storage levels correspond to different quantization precisions. That is, the KV caching device can adaptively store KV blocks according to their importance and select the quantization precision for them. This can reduce or avoid the situation where important KVs are over-quantized and lose precision, and unimportant KVs occupy high-precision resources. In this way, the pressure on video memory can be alleviated, so as to support long sequence inference with longer context, higher throughput, and more stable latency under limited video memory.

[0349] In some embodiments, the acquisition module 901 is specifically used to acquire the importance information of each word corresponding to the first KV block, wherein the first KV block is one of at least one KV blocks; the calculation module 902 is specifically used to perform a weighted summation of the importance information of each word acquired by the acquisition module 901 based on a preset importance weight to obtain a second importance quantization value for each word; and the determination module 903 is used to determine the second importance quantization value with the largest value among the at least one second importance quantization values ​​calculated by the calculation module 902 as the first importance quantization value of the first KV block.

[0350] In other embodiments, the storage module 904 is specifically configured to: store the second KV block in a first storage level when the first importance quantization value of the second KV block is within a first-level threshold range; or store the second KV block in a second storage level when the first importance quantization value of the second KV block is within a second-level threshold range; or store the second KV block in a third storage level when the first importance quantization value of the second KV block is within a third-level threshold range; wherein the lower limit of the first-level threshold range is equal to the upper limit of the second-level threshold range, and the lower limit of the second-level threshold range is equal to the upper limit of the third-level threshold range; the second KV block is one of at least one KV block; the KV block access latency of the first storage level is less than the KV block access latency of the second storage level, and the KV block access latency of the second storage level is less than the KV block access latency of the third storage level.

[0351] In some other embodiments, the KV caching device 900 provided in this application further includes: a processing module; the processing module is configured to, after calculating a first importance quantization value for each KV block based on the importance information of the lexical corresponding to each KV block, assign quantization precision to each KV block based on the first importance quantization value and a preset mapping table, the preset mapping table including at least one importance quantization value and at least one quantization precision that is mapped one-to-one with the at least one importance quantization value.

[0352] In some other embodiments, the KV caching device 900 provided in this application further includes: a prediction module; the prediction module is configured to, after storing each KV block to the storage level corresponding to the target level threshold range of the first importance quantization parameter of each KV block, predict the third KV block used to generate the second word in the process of generating the first word based on the target file, wherein the second word is at least one word after the first word, and the third KV block is one or more KV blocks among at least one KV block; the storage module 904 is further configured to store the third KV block predicted by the prediction module to the first storage level.

[0353] In some other embodiments, the KV caching device 900 provided in this application further includes: a generation module; the generation module is used to generate a prefetch instruction for a third KV block when the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold and the third KV block is stored in a second storage level or a third storage level, the prefetch instruction being used to instruct the adjustment of the storage level of the KV block; the aforementioned storage module 904 is specifically used to store the third KV block in a first storage level based on the prefetch instruction generated by the generation module.

[0354] In some other embodiments, the calculation module 902 is specifically used to calculate the prefetch benefit value of the third KV block when the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold and the third KV block is stored in the second or third storage level. The prefetch benefit value is used to characterize the benefit brought about by adjusting the storage level of the third KV block. The generation module is specifically used to generate a prefetch instruction for the third KV block when the product of the third importance quantization value and the prefetch benefit value calculated by the calculation module 902 is greater than the pre-replacement value. The pre-replacement value is used to characterize the cost required to adjust the storage level of the third KV block.

[0355] It should be noted that the KV caching device can implement all the processes implemented in the above method embodiments and achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0356] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 10 As shown, the electronic device 90 includes: a processor 92, a communication interface 93, and a bus 94. Optionally, the electronic device 90 may also include a memory 91.

[0357] Processor 92 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0358] Communication interface 93 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0359] The memory 91 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0360] As one possible implementation, the memory 91 can exist independently of the processor 92. The memory 91 can be connected to the processor 92 via a bus 94 and is used to store instructions or program code. When the processor 92 calls and executes the instructions or program code stored in the memory 91, it can implement the KV caching method provided in the embodiments of this application.

[0361] In another possible implementation, memory 91 can also be integrated with processor 92.

[0362] Bus 94 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 94 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0363] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0364] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described KV caching method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0365] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0366] This application also provides a readable storage medium storing a program or instructions that, when executed by a computer, implement the KV caching method provided in the above embodiments. It is understood that all or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware; the readable storage medium can be any of the foregoing embodiments or memory; the readable storage medium can also be an external storage device of the service invocation device, such as a pluggable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, flash card, etc., equipped on the service invocation device. Further, the readable storage medium can include both internal storage units of the service invocation device and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the service invocation device. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0367] This application also provides a computer program product, which is stored in a storage medium and implements the KV caching method provided in the above embodiments when the computer program product is executed by a computer.

[0368] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0369] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0370] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.< / eos> < / bos> < / eos> < / bos>

Claims

1. A key-value caching method, characterized in that, include: Based on the word input sequence of the target file, at least one KV block is obtained. A KV block includes a preset number of KV vectors corresponding to words. Based on the importance information of the lexical units corresponding to each KV block, a first importance quantization value is calculated for each KV block. The importance information includes attention information, time decay information, access frequency information, semantic importance information, and positional information. Determine the target level threshold range for the first importance quantization value of each KV block; Each KV block is stored in the storage level corresponding to the target level threshold range of the first importance quantization parameter of each KV block. Different storage levels correspond to different level threshold ranges and different storage levels correspond to different quantization accuracies.

2. The KV caching method according to claim 1, characterized in that, The calculation of the first importance quantization value for each KV block based on the importance information of the lexical units corresponding to each KV block includes: Obtain the importance information of each word element corresponding to the first KV block, wherein the first KV block is one of the at least one KV blocks; The importance information of each word element is weighted and summed based on a preset importance weight to obtain a second importance quantification value for each word element; The second importance quantization value with the largest value among the at least one second importance quantization values ​​is determined as the first importance quantization value of the first KV block.

3. The KV caching method according to claim 1, characterized in that, The step of storing each KV block to the storage level corresponding to the target level threshold range of the first importance quantization parameter of each KV block includes: If the first importance quantization value of the second KV block is within the first level threshold range, the second KV block is stored in the first storage level. If the first importance quantization value of the second KV block is within the second level threshold range, the second KV block is stored in the second storage level. If the first importance quantization value of the second KV block is within the third-level threshold range, the second KV block is stored in the third storage level. Wherein, the lower limit of the first level threshold range is equal to the upper limit of the second level threshold range, and the lower limit of the second level threshold range is equal to the upper limit of the third level threshold range; The second KV block is one of the at least one KV blocks; the KV block access latency of the first storage level is less than the KV block access latency of the second storage level, and the KV block access latency of the second storage level is less than the KV block access latency of the third storage level.

4. The KV caching method according to claim 1, characterized in that, After calculating the first importance quantization value of each KV block based on the importance information of the lexical units corresponding to each KV block, the KV caching method further includes: Based on the first importance quantization value and the preset mapping table, a quantization precision is assigned to each KV block. The preset mapping table includes at least one importance quantization value and at least one quantization precision that maps one-to-one with the at least one importance quantization value.

5. The KV caching method according to claim 3, characterized in that, After storing each KV block to the storage level corresponding to the target level threshold range of the first importance quantization parameter of each KV block, the KV caching method further includes: In the process of generating the first word element based on the target file, the third KV block used to generate the second word element is predicted, wherein the second word element is at least one word element after the first word element, and the third KV block is one or more KV blocks among the at least one KV blocks; The third KV block is stored in the first storage level.

6. The KV caching method according to claim 5, characterized in that, The step of storing the third KV block to the first storage level includes: When the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold, and the third KV block is stored in the second storage level or the third storage level, a prefetch instruction for the third KV block is generated, and the prefetch instruction is used to indicate the adjustment of the storage level of the KV block; Based on the prefetch instruction, the third KV block is stored in the first storage level.

7. The KV caching method according to claim 6, characterized in that, The step of generating a prefetch instruction for the third KV block when the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold, and the third KV block is stored in the second storage level or the third storage level, includes: If the third importance quantization value of the third KV block is greater than or equal to the KV block prefetch threshold, and the third KV block is stored in the second storage level or the third storage level, calculate the prefetch benefit value of the third KV block. The prefetch benefit value is used to characterize the benefit brought about by adjusting the storage level of the third KV block. If the product of the third importance quantification value and the prefetch benefit value is greater than the pre-replacement value, a prefetch instruction for the third KV block is generated, whereby the pre-replacement value is used to characterize the cost required to adjust the storage level of the third KV block.

8. A KV buffer device, characterized in that, include: The module consists of an acquisition module, a calculation module, a determination module, and a storage module. The acquisition module is used to obtain at least one KV block based on the word input sequence of the target file. A KV block includes a preset number of KV vectors corresponding to words. The calculation module is used to calculate the first importance quantization value of each KV block based on the importance information of the lexical corresponding to each KV block obtained by the acquisition module. The importance information includes attention information, time decay information, access frequency information, semantic importance information and position information. The determining module is used to determine the target level threshold range of the first importance quantization value of each KV block obtained by the calculation module; The storage module is used to store each KV block to the storage level corresponding to the target level threshold range of the first importance quantization parameter of each KV block. Different storage levels correspond to different level threshold ranges and different storage levels correspond to different quantization accuracies.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions implementing the KV caching method as described in any one of claims 1-7 when executed by the processor.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a computer, implement the KV caching method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product is stored in a storage medium, and when executed by a computer, the computer program product implements the KV caching method as described in any one of claims 1-7.

Images