Reusable key value cache generation method and device, equipment, storage medium and product

Abstract

The invention discloses a reusable key value cache generation method and device, equipment, a storage medium and a product, and relates to the technical field of computers.The reusable key value cache generation method comprises the steps that an obtained text block is input into a preset large model, and a reusable key value cache corresponding to the text block is output based on a preset fixed prefix key value, the preset large model is obtained by optimizing calculation logic of an attention layer of the original large model based on an attention distribution mechanism. According to the method, the reusability of the key value cache is realized through the preset large model based on the preset fixed prefix key value, the problem of unstable model output caused by position coding and attention distribution changes is effectively solved, and the reuse efficiency of the key value cache is remarkably improved. Meanwhile, the computational logic of the attention layer of the model is optimized, so that the performance and stability of the model during multiplexing of the key value cache of the text block are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and product for generating a reusable key-value cache. Background Art

[0002] Currently, similar technical applications face significant challenges and defects when processing tokens input by large models. Especially in the retrieval-augmented generation (RAG) scenario, the computing time and cost increase dramatically due to the need to process a large number of imported retrieval text blocks. Whenever the same text is called multiple times on a large model for processing, the problem of repeated calculations is particularly prominent, which not only wastes precious computing resources, but also limits the improvement of the overall efficiency of the system. In addition, a key difficulty is that the KV (key-value) cache cannot be directly reused. In large models, the attention mechanism determines that the same text will generate different KV caches in different contexts. Therefore, when the text blocks are rearranged or new contextual information is inserted, the position encoding and attention distribution in the original KV cache will no longer be accurate, resulting in a decline in model performance or even failure. Summary of the Invention

[0003] The main purpose of this application is to provide a reusable key-value cache generation method, device, equipment, storage medium and product, aiming to solve the technical problem of how to improve the reusability of KV (key-value) cache and improve the performance and stability of the model when reusing the key-value cache of text blocks.

[0004] To achieve the above objectives, the present application proposes a reusable key-value cache generation method, which includes:

[0005] The obtained text block is input into the preset large model, and based on the preset fixed prefix key value, the reusable key value cache corresponding to the text block is output, wherein the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model based on the attention distribution mechanism.

[0006] In one embodiment, the step of inputting the acquired text block into a preset macro model and outputting a reusable key-value cache corresponding to the text block based on a preset fixed prefix key value includes:

[0007] Setting a fixed prefix key value at the beginning of the preset large model;

[0008] The obtained text block is input into the preset large model, the preset fixed prefix key value and the text block are processed, and a reusable key value cache sequence corresponding to the text block is generated and output.

[0009] In one embodiment, after the step of processing the preset fixed prefix key value and the text block to generate and output a reusable key value cache sequence corresponding to the text block, the step further includes:

[0010] When a reuse instruction of a reused text block is received, the preset fixed prefix key value and the reused text block are spliced ​​together using a preset large model to obtain a spliced ​​key value cache;

[0011] Position encoding is performed on the spliced ​​key-value cache to obtain a position-encoded key-value cache, and the position-encoded key-value cache is output as a reusable key-value cache corresponding to the reusable text block.

[0012] In one embodiment, before the step of inputting the acquired text block into a preset macro model and outputting a reusable key-value cache corresponding to the text block based on a preset fixed prefix key value, the following steps are included:

[0013] The preset rotation position encoding mechanism is used to adjust the key-value cache generation logic of the large model attention layer, delete the position information of the key-value cache generated in the large model attention layer, and obtain the preset large model.

[0014] In one embodiment, the step of adjusting the key-value cache generation logic of the large model attention layer using a preset rotation position encoding mechanism includes:

[0015] Generate query vector, key vector and value vector through linear projection;

[0016] Applying the rotational position encoding to the query vector to obtain a position-encoded query vector sequence;

[0017] caching the key vector and the value vector in a key-value cache, and retaining the key vector and the value vector that are not associated with a position in the key-value cache;

[0018] Applying the rotational position encoding to the key vectors in the key-value cache that are not associated with the position to obtain a position-encoded key sequence;

[0019] Calculating a dot product attention score using the position-encoded query vector sequence and the position-encoded key sequence;

[0020] A preset activation function is applied to the dot product attention score and weighted summed with the value vector in the key-value cache.

[0021] In one embodiment, after the step of inputting the acquired text block into a preset macro model and outputting a reusable key-value cache corresponding to the text block based on a preset fixed prefix key value, the following steps are included:

[0022] Retrieving the key-value cache blocks with the highest relevance to the user question, and loading the key-value cache blocks into the reusable key-value cache in descending order according to the relevance scores;

[0023] The updated reusable key-value cache is used to execute the reasoning process of the preset large model.

[0024] In addition, to achieve the above-mentioned purpose, the present application also proposes a reusable key-value cache generation device, which includes:

[0025] A generation module is used to input the acquired text block into a preset large model, and output a reusable key value cache corresponding to the text block based on a preset fixed prefix key value, wherein the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model based on the attention distribution mechanism.

[0026] In addition, to achieve the above-mentioned purpose, the present application also proposes a reusable key-value cache generation device, which includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor, and the computer program is configured to implement the steps of the reusable key-value cache generation method as described above.

[0027] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the reusable key-value cache generation method described above are implemented.

[0028] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the reusable key-value cache generation method as described above.

[0029] The present application proposes a method, device, equipment, storage medium and product for generating a reusable key-value cache. The method includes: inputting the acquired text block into a preset large model, and outputting a reusable key-value cache corresponding to the text block based on a preset fixed prefix key value, wherein the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model based on the attention distribution mechanism. This method realizes the reusability of the key-value cache based on the preset fixed prefix key value through the preset large model, effectively solves the problem of unstable model output due to changes in position encoding and attention distribution, and significantly improves the reuse efficiency of the key-value cache. At the same time, the calculation logic of the attention layer of the model is optimized to improve the performance and stability of the model when reusing the key-value cache of the text block. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 A flowchart of the first embodiment of the method for generating a reusable key-value cache according to the present invention is provided;

[0033] Figure 2 A schematic diagram of a sliding window provided in Example 1 of the present application;

[0034] Figure 3 This is a schematic diagram of a short-input, long-output scenario test provided in Example 1 of the present application;

[0035] Figure 4 A schematic diagram of generating a reusable key-value cache provided in Example 1 of the present application;

[0036] Figure 5 A flowchart of the second embodiment of the method for generating a reusable key-value cache according to the present invention is provided;

[0037] Figure 6 This is a logic flow chart of the large model attention layer calculation during the reusable key-value cache generation phase provided in Example 2 of this application;

[0038] Figure 7 A schematic diagram of a retrieval reusable key-value cache provided by the reusable key-value cache generation method of this application;

[0039] Figure 8 The first experimental results of the reusable key-value cache generation method provided by this application;

[0040] Figure 9 The second experimental results provided for the reusable key-value cache generation method of this application;

[0041] Figure 10 The third experimental result provided for the reusable key-value cache generation method of this application;

[0042] Figure 11 A comparative diagram of the inference time consumption in a GPU environment provided by the reusable key-value cache generation method of this application;

[0043] Figure 12A schematic diagram comparing the CPU inference time consumption provided for the reusable key-value cache generation method of this application;

[0044] Figure 13 This is a schematic diagram of the module structure of a reusable key-value cache generation device according to an embodiment of the present application;

[0045] Figure 14 This is a schematic diagram of the device structure of the hardware operating environment involved in the reusable key-value cache generation method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The main solution of the embodiment of the present application is: input the acquired text block into a preset large model, and output a reusable key value cache corresponding to the text block based on a preset fixed prefix key value, wherein the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model based on the attention distribution mechanism.

[0050] In this embodiment, for ease of description, the following description is made with the large model reasoning system as the execution subject.

[0051] When processing tokens in large models, the KV (key-value) cache cannot be directly reused. In large models, the attention mechanism determines that the same text in different contexts will generate different KV caches. Therefore, when text blocks are rearranged or new contextual information is inserted, the position encoding and attention distribution in the original KV cache will no longer be accurate, resulting in degraded model performance or even failure.

[0052] The present application provides a solution that uses a preset large model to infer fixed prefix key values ​​for text blocks and text blocks input by users, and generates a reusable key value cache corresponding to the text blocks. This can solve the problem of non-reusability of text block key value caches caused by changes in position encoding and attention distribution, and ensure that the large model can work properly. And using the generated key value cache for subsequent reuse not only reduces the computing cost, but also reduces the memory usage during the model inference process. In addition, the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model, which not only effectively improves the performance and stability of the model, but also does not require any additional training of the model. It has a high degree of versatility and low transformation cost.

[0053] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of performing the above functions. The following uses a personal computer as an example to illustrate this embodiment and the following embodiments.

[0054] Based on this, the embodiment of the present application provides a reusable key-value cache generation method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the reusable key-value cache generation method of the present application.

[0055] In this embodiment, the reusable key-value cache generation method includes step S10:

[0056] Step S10: input the acquired text block into a preset large model, and output a reusable key value cache corresponding to the text block based on a preset fixed prefix key value, wherein the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model based on the attention distribution mechanism.

[0057] It should be noted that reusable key-value cache means that when processing sequence data, the stored key (Key) and value (Value) vectors can be reused in different contexts without having to be recalculated each time.

[0058] It is understandable that due to the changes in position encoding information and attention distribution, the key-value cache corresponding to the text block cannot be directly reused. Therefore, step S10 is performed to de-position the key-value cache and optimize the attention mechanism, which can effectively avoid the cache invalidation problem caused by context changes, thereby significantly improving the reuse efficiency of the key-value cache and the model reasoning performance.

[0059] Optionally, the reusable key-value cache generation principle provided in this embodiment is: due to the current phenomenon of the attention layer of the large model, except for the first two layers (layers of the neural network), the attention scores of the first few tokens after the output of the softmax (activation function) of all the subsequent layers of the model are relatively high. During the model inference process, once the frontmost token is discarded, the perplexity of the model will increase significantly, resulting in unstable output. Therefore, in order to ensure the stability and consistency of the model throughout the generation process, this embodiment proposes a method in which the output of the large model can always remain stable as long as the first few tokens are kept fixed.

[0060] Based on the above principles, this embodiment performs sliding window transformation on the large model and constructs a short input and long output scenario for experimentation.

[0061] Specifically, first, the preset large model is transformed into a sliding window, and an experimental scenario with short input and long output is designed, such as Figure 2 As shown, Figure 2 is a schematic diagram of a sliding window. Figure 2 It can be seen that Indicates that the fixed prefix key value of the preset large model input is fixed at the beginning position; Indicates that the generated part of the large model is used as a sliding window; Indicates the discarded token. During the output process, once the entire KV cache length reaches the length limit set by the attention window, it will be discarded; The first token of the part.

[0062] The principle behind short input and long output is that the model needs to generate a large number of tokens to complete a specific task, and the output length far exceeds the preset attention window length. During this process, the currently generated token does not depend on previously discarded tokens. By fixing a prefix key value at the beginning of the input sequence (i.e., the user input), the model can always clearly understand the goal and semantic intent of the current task. The output uses a sliding window mechanism, allowing the model to gradually advance task execution based on the local context within the current window. Although early token information is lost during the sliding window process, as long as the initial fixed prefix key value is retained and combined with the context within the current sliding window, the model can still stably generate subsequent content. In theory, this mechanism supports output generation of unlimited length. Experiments have shown that as long as the fixed prefix key value remains unchanged, the model's output remains highly stable regardless of the sliding window position, consistently generating subsequent text coherently and fluently based on the current window content.

[0063] Suppose the text input by the user is to let the preset large model introduce tourist attractions in the 50 states of the United States one by one, selecting 3 attractions in each state, and the model cannot stop until the introduction is completed.

[0064] like Figure 3 As shown, Figure 3 This diagram illustrates a test scenario with short input and long output. First, the default large model sets the sliding window size to 1536. As the default model continues to output, when the length of the key-value cache sequence reaches the attention window size setting, the first token in the default large model sliding window is discarded and the most recently generated token is added. Finally, the default model generates 15,492 tokens at once for this task. The stable output of the sliding window indicates that while the key-value cache remains fixed at the beginning, the default large model allows for changes in the attention score distribution and position information of subsequent tokens.

[0065] Based on the above experimental phenomena, step S10 may include steps A11 to A12:

[0066] Step A11, setting a fixed prefix key value at the beginning of the preset large model;

[0067] Step A12: input the acquired text block into the preset large model, process the preset fixed prefix key value and the text block, and generate and output a reusable key value cache sequence corresponding to the text block.

[0068] It should be noted that fixed prefix keys refer to a portion of the key-value cache that is always retained and used to calculate attention scores when generating long text or performing multi-step reasoning tasks. This key-value cache usually corresponds to the beginning of the input sequence (i.e., the prefix), ensuring that the model can continuously reference this initial input information throughout the generation process, thereby maintaining consistent and coherent output.

[0069] like Figure 4 As shown, Figure 4 A schematic diagram for generating a reusable key-value cache. Specifically, before the user inputs the text block to be processed into the preset large model, a fixed prefix key value is set at the beginning of the preset large model. The purpose of this prefix key value is to ensure that the text blocks to be processed subsequently can be processed in the same context environment, thereby improving the reusability and consistency of the generated key-value cache. For example, this prefix key value can be a task instruction, a guiding prompt, or other necessary contextual information. The fixed prefix key value is loaded into the model to ensure that it remains unchanged throughout the generation process to maintain the consistency and coherence of the task.

[0070] Then, when the pre-set large model receives the text blocks entered by the user, it combines these text blocks with the pre-set fixed prefix key value for processing. This means that the large model will understand and process each text block based on the context provided by the prefix.

[0071] After being processed by the model, a corresponding key-value cache is generated for each text block. This key-value cache contains a data structure or encoding that can represent the information of the text block, aiming to facilitate rapid retrieval and use in the future, while ensuring that it can be reused in similar contexts to reduce the cost of repeated calculations.

[0072] Finally, these generated key-value cache sequences are output and can be used for subsequent reuse.

[0073] For example, when the system detects that a new text block is a "reused text block" (ie, a text block that has been processed before) and there is a corresponding reusable key-value cache, the reuse process is triggered.

[0074] The previously set fixed prefix key value is retrieved from the preset large model, and the prefix is ​​spliced ​​with the current reused text block to form a complete context representation.

[0075] Since the preset large models usually rely on position information to understand the order and structure of the input sequence, it is necessary to apply position encoding to the spliced ​​key-value cache.

[0076] The position-encoded key-value cache accurately reflects the position and semantic information of the current text block in the overall context. Finally, it is output as the final reusable key-value cache of the reused text block for subsequent decoding, reasoning, or cache updates.

[0077] In addition, the preset large model is obtained by optimizing the computational logic of the attention layer of the original large model based on the attention mechanism.

[0078] Through the above steps, by introducing a fixed prefix key value, it is ensured that even for text blocks with the same content, a consistent and effective key value cache can be generated in different contexts, thereby improving processing efficiency and flexibility.

[0079] Through the above-mentioned embodiment method, the obtained text block is input into the preset large model, and based on the preset fixed prefix key value, the reusable key value cache corresponding to the text block is output, wherein the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model based on the attention distribution mechanism. This method realizes the reusability of the key value cache based on the preset fixed prefix key value through the preset large model, effectively solves the problem of unstable model output due to changes in position encoding and attention distribution, and significantly improves the reuse efficiency of the key value cache. At the same time, the calculation logic of the attention layer of the model is optimized to improve the performance and stability of the model when reusing the key value cache of the text block.

[0080] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction and will not be described in detail later. Figure 5 Before step S10, the reusable key-value cache generation method further includes step S01:

[0081] Step S01, using a preset rotation position encoding mechanism, adjusts the key value cache generation logic of the large model attention layer, deletes the position information of the key value cache generated in the large model attention layer, and obtains the preset large model.

[0082] It should be noted that the rotational position encoding mechanism refers to a position encoding method used to process sequence data.

[0083] Because the key-value cache generated by the attention layer of a traditional large model contains the position information of the current input and output sequence, changes in the position when reusing the key-value cache will cause the original input and output sequence position information to become invalid. Therefore, in this embodiment, before generating a reusable key-value cache using a preset large model, it is necessary to modify the calculation logic of the attention layer of the large model to remove the position information of the key-value cache. During subsequent reuse, new position information is injected based on the current key-value cache sequence position.

[0084] In a feasible embodiment, step S01 may include steps S011 to S016:

[0085] Step S011, generating a query vector, a key vector, and a value vector through linear projection;

[0086] Step S012, applying the rotation position encoding to the query vector to obtain a position-encoded query vector sequence;

[0087] Step S013: caching the key vector and the value vector into a key-value cache, and retaining the key vector and the value vector that are not associated with the position in the key-value cache;

[0088] Step S014, applying the rotation position encoding to the key vectors in the key-value cache that are not associated with the position, to obtain a position-encoded key sequence;

[0089] Step S015, calculating a dot product attention score using the position-encoded query vector sequence and the position-encoded key sequence;

[0090] Step S016: Apply a preset activation function to the dot product attention score and perform a weighted sum on the dot product attention score with the value vector in the key-value cache.

[0091] Please refer to Figure 6 , Figure 6 This is a logical flow chart for the calculation of the large model's attention layer during the reusable key-value cache generation phase. Specifically, first, the large model generates a query vector (Q), a key vector (K), and a value (V) vector using linear projection of the text sequence input by the user. The linear projection method refers to first converting the text sequence input by the user into an embedding vector. Then, in order to retain the position information of the token in the sequence, the large model adds a position encoding based on the embedding vector. Subsequently, based on the embedding vector and the corresponding position encoding, the input sequence is linearly transformed through three different weight matrices to generate a query vector, a key vector, and a value vector, respectively.

[0092] After generating the query vector (Q), key vector (K), and value vector (V), Rotational Position Encoding (RoPE) is applied to the query vector (Q) alone, unlike traditional methods that encode position information for both Q and K during generation. Specifically, this embodiment fixes the position encoding of Q in advance, allowing the position encoding of K to be dynamically adjusted.

[0093] Assume that the tth Token is currently being processed. Apply the rotation matrix R of position t to the query vector Q corresponding to the Token. t . Among them, the rotation matrix R t It can be calculated from the absolute position t and implemented using the complex rotation form of RoPE:

[0094]

[0095] Among them, Q t Represents the query vector of the tth Token; R t represents the rotation matrix associated with position t.

[0096] At the same time, the currently generated original key vector and value vector are saved in the key-value cache (key-value cache), and the key-value cache only retains the original vectors that are independent of the position, so that the position encoding can be dynamically adjusted for subsequent reuse. Among them, the expression for saving the currently generated original key vector and value vector in the key-value cache is:

[0097] KV Cache←KV Cache∪{K t ,V t}where, key-value cache represents key-value cache; K t represents the key vector at the tth time step; V t A vector of values ​​representing the t-th time step.

[0098] Then, the rotation position encoding is dynamically applied to the key sequence in the key-value cache that stores the currently generated original key vector and value vector. Specifically, each key K in the key-value cache is first traversed. i (i=1,2,...,t). Then, for each K i Apply the rotation matrix corresponding to the actual position i, where each K i The position code of i strictly corresponds to its actual position (the actual position code only depends on its own position). Among them, for each K in the key-value cache i The expression for the rotation change is:

[0099]

[0100] Where K i Represents the i-th key vector in the key-value cache; R i represents the rotation matrix associated with position i.

[0101] Next, after applying Rotational Position Encoding (RoPE) to Q and K respectively, the encoded Q and K vectors both contain the position information of their respective positions. The attention score is calculated using the encoded Q and K. The basic formula is as follows:

[0102]

[0103] in, is the query matrix; is the bond matrix; d k is the dimension of the key vector.

[0104] Due to the use of Rotation Position Encoding (RoPE), when calculating the dot product, the relative positions between tokens are actually taken into account. Specifically, for any two tokens i and j, their relative position relationship is calculated by their respective rotation matrices R i and R j Encoded to Q i , K i and Q j , K j In calculation , the result not only reflects the semantic relevance of the two tokens, but also implies the relative position information between them.

[0105] Therefore, by using the encoded Q and K to calculate the point-integrated attention score, the model can not only capture the semantic associations between tokens in the sequence, but also effectively understand the relative position information between them.

[0106] Finally, the Softmax (activation) function is applied to the calculated attention scores to obtain the attention weight matrix. The calculation formula is:

[0107] Attention Weights=Softmax(Attention Scores)

[0108] Where Softmax is a function that transforms a vector of K real numbers into another vector of K real numbers; Attention Scores is the attention score; Attention Weights represents the attention weight matrix.

[0109] The purpose of this step is to convert the original attention score into a probability distribution so that the importance of each token relative to the query is normalized.

[0110] The attention weight matrix after Softmax processing is weighted and summed with the value vector in the key-value cache to generate a new representation for each query.

[0111] For example, suppose the attention weight matrix after Softmax processing is Attention Weights, and its dimension is (n,n).

[0112] The value vectors stored in the key-value cache are organized into a matrix with dimension (t, d_v), where t is the time step or sequence length and d_v is the dimension of the value vector.

[0113] For each query vector, all value vectors are weighted and summed by using their corresponding attention weight distribution as weight. Specifically, for the attention weight distribution Attention Weights, the weighted sum is calculated as follows:

[0114] Output=Attention Weights·V 1:t

[0115] Where Output is the new representation of the query, V is the value vector, and Attention Weights is the attention weight matrix.

[0116] By applying Softmax to the attention score and weighted summing it with the value vector in the key-value cache, the model can dynamically focus on the most important pieces of information when processing sequence data, whether local details or global context, thereby enhancing the model's ability to understand and generate complex sequences.

[0117] Through the above-mentioned embodiment method, a preset rotation position encoding mechanism is adopted to adjust the key-value cache generation logic of the attention layer of the large model, which not only removes the model's dependence on absolute position information, but also effectively improves the performance and stability of the model. There is no need for any additional training of the model, and it has high versatility and low modification cost.

[0118] In this embodiment, a practical example of introducing RAG (Retrieval-Augmented Generation) into a reusable key-value cache is also proposed to effectively illustrate the scenario of the method of this application. This embodiment implements a new RAG architecture that introduces a reusable key-value cache to accelerate the model reasoning process, such as Figure 7 As shown, Figure 7 A diagram of a reusable key-value cache for retrieval.

[0119] Specifically, the processed text is divided into blocks, and a corresponding reusable key-value cache is generated in advance for each text block. Based on the user question, the most relevant key-value caches are retrieved and loaded into the key-value cache in descending order of similarity for model inference.

[0120] For example, using the first chapter of the first part of the novel "A" as experimental material, we segmented the text into chunks based on sentence boundaries. Each chunk had a maximum length of 1,000 characters, with a 200-character overlap to ensure contextual coherence. Ultimately, the resulting chunks were split into 32 chunks. For each chunk, we generated a corresponding reusable key-value cache and its representative vector for subsequent retrieval and reasoning.

[0121] The experiment used the Qwen2.5-1.5B-Instruct model, running on a device equipped with an RTX 2060 GPU (6GB of video memory) and 16GB of RAM. The embedding vectors for the text blocks were generated using the jina-embeddings-v3 model to ensure semantically relevant retrieval capabilities.

[0122] In the "Key-Value Cache Reusability Experiment", this embodiment retrieves the four most relevant KV Blocks from all text blocks based on user questions, and loads them into the key-value cache in descending order of similarity for model reasoning. The first experimental results are as follows: Figure 8 The second experimental results are shown in Figure 9As shown in the third experiment results are Figure 10 Even if key-value caches from different text blocks are arbitrarily combined, the pre-set model can still perform normal reasoning, accurately understand the context, and generate reasonable responses. This demonstrates that reusable key-value caches are highly available and stable without relying on the continuity of the original context, effectively breaking the limitation of traditional key-value caches that strictly match the context.

[0123] Due to the output diversity of large models during the generation process, the same question may produce answers of varying lengths in different invocations, thus affecting the stability of inference time. To eliminate the impact of this variation on experimental results, this example explicitly requires the model to directly provide the answer in its input and sets the sampling strategy for the model output to top-k=1, which selects only the word with the highest probability as the output each time. This ensures consistent output when the model is invoked multiple times for the same question.

[0124] For example, when the user input is: "Does the Dumbledore gold watch have numbers? Answer the question directly, don't explain it", no matter 2 to 14 text blocks are introduced, the result returned by the RAG system is always: "No, the Dumbledore gold watch does not have numbers.".

[0125] like Figure 11 As shown, Figure 11 This is a comparison diagram of the inference time in the GPU environment. Figure 11 As can be seen in the figure, there is a significant difference in the total time from receiving input to final output between the traditional RAG approach and the RAG model that introduces a reusable key-value cache. By introducing a reusable key-value cache mechanism, the model exhibits shorter inference latency when processing the same task, effectively improving overall inference efficiency.

[0126] Experimental results show that the introduction of a reusable key-value cache mechanism has significant advantages in both inference efficiency and resource usage.

[0127] First, in the traditional RAG method, large models directly process retrieved text blocks as input. As the number of introduced text blocks increases, the model's inference time increases linearly. In contrast, when the KV cache of the text blocks is pre-generated and reused, the model inference time is significantly reduced and is almost unaffected by the number of introduced text blocks. In particular, when more than 10 text blocks are introduced, the model inference time of the reused key-value cache remains stable at approximately 2.2 seconds, demonstrating extremely high efficiency and stability.

[0128] Secondly, without reusing the key-value cache, when more than 11 text blocks are introduced, GPU memory consumption approaches the 6GB limit, and the system begins to use shared memory to supplement it, resulting in reduced inference efficiency. When 14 text blocks are introduced, inference time has reached 17 seconds. However, after adopting the key-value cache reuse mechanism, the inference process basically maintains a stable level of video memory usage (less than 4GB), and inference time remains unchanged, without any obvious performance bottlenecks.

[0129] This shows that reusing key-value caches is crucial for improving inference efficiency when computing resources are limited. By avoiding repeated query-key-value calculations on long texts, not only does it effectively reduce computing costs, but it also significantly reduces real-time memory consumption during inference, thereby improving overall inference speed and stability.

[0130] The following is the experimental data of the total time taken from receiving input to output of the traditional RAG and multiplexed key-value cache models in the CPU environment.

[0131] Please refer to Figure 12 , Figure 12 This is a comparison diagram of CPU inference time. Figure 12 As can be seen, in the CPU inference environment, just like in the GPU inference environment, the time taken by traditional RAG increases linearly with the amount of text introduced, while the reused key-value cache barely increases inference time. Because CPU inference speed is much slower than GPU inference speed, the acceleration effect of the reused key-value cache is even more significant than that of the GPU. When 14 texts were introduced, the inference time without the reused key-value cache reached over an hour. However, the inference time with the reused key-value cache remained within 45 seconds, a speed increase of nearly 90 times, from the original completely unacceptable speed to a usable level. This makes it possible for even low-spec personal computers to use the CPU to infer long text input tasks.

[0132] This application also provides a reusable key-value cache generation device, please refer to Figure 13 , the reusable key-value cache generating device includes:

[0133] The generation module 10 is used to input the acquired text block into a preset large model, and output a reusable key value cache corresponding to the text block based on a preset fixed prefix key value, wherein the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model based on the attention distribution mechanism.

[0134] The reusable key-value cache generation device provided by this application adopts the reusable key-value cache generation method in the above-mentioned embodiment, which can solve the technical problem of how to improve the reusability of KV (key-value) cache and enhance the performance and stability of the model when reusing the key-value cache of text blocks. Compared with the prior art, the beneficial effects of the reusable key-value cache generation device provided by this application are the same as the beneficial effects of the reusable key-value cache generation method provided by the above-mentioned embodiment, and the other technical features of the reusable key-value cache generation device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0135] The present application provides a reusable key-value cache generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the reusable key-value cache generation method in the above-mentioned embodiment one.

[0136] Reference below Figure 14 , which shows a schematic diagram of the structure of a reusable key-value cache generation device suitable for implementing embodiments of the present application. The reusable key-value cache generation device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., as well as fixed terminals such as digital TVs and desktop computers. Figure 14 The reusable key-value cache generation device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0137] like Figure 14As shown, the reusable key-value cache generation device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the reusable key-value cache generation device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the reusable key-value cache generation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a reusable key-value cache generation device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or have alternatively.

[0138] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0139] The reusable key-value cache generation device provided by this application adopts the reusable key-value cache generation method in the above embodiment, which can solve the technical problem of how to improve the reusability of KV (key-value) cache and enhance the performance and stability of the model when reusing the key-value cache of text blocks. Compared with the prior art, the beneficial effects of the reusable key-value cache generation device provided by this application are the same as the beneficial effects of the reusable key-value cache generation method provided by the above embodiment, and the other technical features of the reusable key-value cache generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0140] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0141] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0142] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the reusable key-value cache generation method in the above-mentioned embodiment.

[0143] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0144] The computer-readable storage medium may be included in the reusable key-value cache generation device; or may exist independently without being assembled into the reusable key-value cache generation device.

[0145] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the reusable key-value cache generation device, the reusable key-value cache generation device: inputs the acquired text block into a preset large model, and based on a preset fixed prefix key value, outputs the reusable key-value cache corresponding to the text block, wherein the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model based on the attention distribution mechanism.

[0146] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0148] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0149] The readable storage medium provided by this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., a computer program) for executing the above-mentioned reusable key-value cache generation method, and can solve the technical problem of how to improve the reusability of KV (key-value) cache and improve the performance and stability of the model when reusing the key-value cache of text blocks. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as the beneficial effects of the reusable key-value cache generation method provided by the above-mentioned embodiment, and will not be repeated here.

[0150] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned reusable key-value cache generation method when executed by a processor.

[0151] The computer program product provided in this application can solve the technical problem of how to improve the reusability of KV (key-value) caches and enhance the performance and stability of models when reusing key-value caches of text blocks. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the reusable key-value cache generation method provided in the above-mentioned embodiment, and will not be repeated here.

[0152] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating a reusable key-value cache, characterized in that: The reusable key-value cache generation method includes: The obtained text block is input into the preset large model, and based on the preset fixed prefix key value, the reusable key value cache corresponding to the text block is output, wherein the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model based on the attention distribution mechanism.

2. The reusable key-value cache generation method according to claim 1, characterized in that: The step of inputting the acquired text block into a preset large model and outputting a reusable key value cache corresponding to the text block based on a preset fixed prefix key value includes: Setting a fixed prefix key value at the beginning of the preset large model; The obtained text block is input into the preset large model, the preset fixed prefix key value and the text block are processed, and a reusable key value cache sequence corresponding to the text block is generated and output.

3. The reusable key-value cache generation method according to claim 2, characterized in that: After the step of processing the preset fixed prefix key value and the text block to generate and output a reusable key value cache sequence corresponding to the text block, the method further includes: When a reuse instruction of a reused text block is received, the preset fixed prefix key value and the reused text block are spliced ​​together using a preset large model to obtain a spliced ​​key value cache; Position encoding is performed on the spliced ​​key-value cache to obtain a position-encoded key-value cache, and the position-encoded key-value cache is output as a reusable key-value cache corresponding to the reusable text block.

4. The reusable key-value cache generation method according to any one of claims 1 to 3, characterized in that: Before the step of inputting the acquired text block into a preset large model and outputting a reusable key value cache corresponding to the text block based on a preset fixed prefix key value, the following steps are included: The preset rotation position encoding mechanism is used to adjust the key-value cache generation logic of the large model attention layer, delete the position information of the key-value cache generated in the large model attention layer, and obtain the preset large model.

5. The reusable key-value cache generation method according to claim 4, characterized in that: The step of adjusting the key-value cache generation logic of the large model attention layer using the preset rotation position encoding mechanism includes: Generate query vector, key vector and value vector through linear projection; Applying the rotational position encoding to the query vector to obtain a position-encoded query vector sequence; caching the key vector and the value vector in a key-value cache, and retaining the key vector and the value vector that are not associated with a position in the key-value cache; Applying the rotational position encoding to the key vectors in the key-value cache that are not associated with the position to obtain a position-encoded key sequence; Calculating a dot product attention score using the position-encoded query vector sequence and the position-encoded key sequence; A preset activation function is applied to the dot product attention score and weighted summed with the value vector in the key-value cache.

6. The reusable key-value cache generation method according to claim 1, characterized in that: After the step of inputting the acquired text block into the preset large model and outputting the reusable key value cache corresponding to the text block based on the preset fixed prefix key value, the following steps are included: Retrieving the key-value cache blocks with the highest relevance to the user question, and loading the key-value cache blocks into the reusable key-value cache in descending order according to the relevance scores; The updated reusable key-value cache is used to execute the reasoning process of the preset large model.

7. A reusable key-value cache generation device, characterized in that: The reusable key-value cache generating device includes: A generation module is used to input the acquired text block into a preset large model, and output a reusable key value cache corresponding to the text block based on a preset fixed prefix key value, wherein the preset large model is obtained by optimizing the calculation logic of the attention layer of the original large model based on the attention distribution mechanism.

8. A reusable key-value cache generation device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the reusable key-value cache generation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the reusable key-value cache generation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the reusable key-value cache generation method according to any one of claims 1 to 6 are implemented.

Images