A large language model key-value cache reuse method and system based on fuzzy semantic matching

Abstract

The application relates to the technical field of large language model reasoning, and discloses a large language model key-value cache reuse method and system based on fuzzy semantic matching, which comprises the following steps: generating a key-value cache according to word units in a historical reasoning request of a user, gathering multiple word units into a word unit block, generating an embedding vector of the word unit block, and establishing a vector database; calculating the cosine similarity of the embedding vector of a word unit block of a new reasoning request and a historical embedding vector in the vector database; if there is a historical word unit block with a cosine similarity exceeding a set threshold, corresponding key-value cache is obtained through a hash index and reused; calculating an attention score, and distinguishing multiple word units in the current word unit block into key word units and non-key word units based on the attention score; recalculating the key-value cache of the key word units; and composing a hybrid key-value cache by the recalculated key-value cache of the key word units and the reused key-value cache of the non-key word units. The method can expand the key-value cache reuse technology from accurate matching to fuzzy semantic matching without reducing the model accuracy, effectively reduce unnecessary calculation overhead, and further improve the reasoning efficiency.

Description

Technical Field

[0001] This invention relates to the field of large language model inference technology, specifically to a method and system for reusing key-value caches of large language models based on fuzzy semantic matching. Background Technology

[0002] In the field of large language modeling, mainstream models often employ a decoder-only Transformer architecture for lexical prediction. The inference process is divided into two stages: pre-filling and decoding. In the pre-filling stage, the model projects all input lexical units into query, key, and value vectors, generating the first lexical unit through an attention mechanism. In the decoding stage, an autoregressive approach is used to iteratively generate the next lexical unit based on existing units until a preset condition is met. To reduce computational redundancy caused by autoregression, the inference system saves the intermediate data generated in the pre-filling stage—a key-value cache—and uses it directly during lexical unit generation to accelerate the decoding stage and improve overall inference efficiency.

[0003] Since causal decoders only focus on historical information, inference requests with the same prefix tokens have the same key-value vector for their corresponding segments. Therefore, key-value caches can be shared to avoid redundant calculations and accelerate inference; this is known as key-value cache reuse technology. However, most existing technologies rely on prefix hashing for precise token matching, making it difficult for requests with similar semantics to reuse key-value caches, thus limiting inference acceleration opportunities and system performance improvement potential. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method and system for reusing key-value caches in large language models based on fuzzy semantic matching. This invention, through fuzzy semantic matching, enables broader and more efficient key-value cache reuse, thereby reducing the inference computation overhead of large language models and accelerating their response speed.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a method for reusing key-value caches in large language models based on fuzzy semantic matching, comprising:

[0007] Based on the lexical elements in the user's historical reasoning requests, a key-value cache is generated, and multiple lexical elements are aggregated into lexical blocks. An embedding vector for the lexical blocks is generated through a word embedding model. A vector database is built with lexical blocks as the index and embedding vectors as the content.

[0008] Calculate the cosine similarity between the embedding vector of the new inference request's word block and the historical embedding vector in the vector database. If there is a historical word block with a cosine similarity exceeding a set threshold, retrieve the corresponding key-value cache through the hash index and reuse it.

[0009] In the first Transformer attention calculation layer, the attention score is calculated using a standard process. Based on the attention score, multiple words in the current word block are distinguished into key words and non-key words. In subsequent Transformer attention calculation layers, the key-value cache is recalculated for key words. The recalculated key-value cache for key words and the reused key-value cache for non-key words constitute a hybrid key-value cache.

[0010] The hybrid key-value cache will be treated as a key-value cache generated by the user request in the pre-filling stage, and will be continuously used for the prediction of new terms in the decoding stage until the inference process ends.

[0011] In one embodiment, if the cosine similarity between the embedding vector of the new inference request's lexical block and the historical embedding vector in the vector database is less than a set threshold, then the key-value cache of the current lexical block is generated according to the standard process, and the current lexical block and its corresponding embedding vector are updated to the vector database.

[0012] In a second aspect, the present invention provides a computer system including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any embodiment of the first aspect.

[0013] The system and method in this invention correspond to each other; the specific technical solutions applicable to the method are also applicable to the system.

[0014] Compared with the prior art, the beneficial technical effects of the present invention are:

[0015] The method of this invention can extend the key-value caching reuse technology from exact matching to fuzzy semantic matching with almost no reduction in model accuracy, significantly expanding the reusable space of the inference system, effectively reducing unnecessary computational overhead, and thus improving inference efficiency. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method in an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of the overall system architecture in an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram illustrating the process of selecting keyword elements in the first attention calculation layer in an embodiment of the present invention;

[0019] Figure 4 This is a schematic diagram illustrating the vector recalculation of keyword elements and the approximate key-value caching reuse of non-keyword elements in an embodiment of the present invention. Detailed Implementation

[0020] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] like Figure 1 As shown, this invention provides a method for reusing key-value caches in large language models based on fuzzy semantic matching, comprising the following steps:

[0022] S1. Generate key-value cache based on the lexical elements in the user's historical reasoning requests, and aggregate multiple lexical elements into lexical blocks. Generate the embedding vector of the lexical blocks through the word embedding model, and build a vector database with lexical blocks as indexes and embedding vectors as content.

[0023] S2, calculate the cosine similarity between the embedding vector of the new inference request word block and the historical embedding vector in the vector database. If there is a historical word block with a cosine similarity exceeding a set threshold, obtain the corresponding key-value cache through the hash index and reuse it.

[0024] S3, in the first Transformer attention calculation layer, the attention score is calculated using the standard process. Based on the attention score, multiple words in the current word block are distinguished into key words and non-key words. In subsequent Transformer attention calculation layers, the key-value cache is recalculated for key words. The recalculated key-value cache for key words and the reused key-value cache for non-key words constitute a hybrid key-value cache.

[0025] S4, the hybrid key-value cache will be regarded as the key-value cache generated by the user request in the pre-filling stage, and will be continuously used for the prediction of new terms in the decoding stage until the inference process ends.

[0026] The basic idea of ​​this invention is to use a word embedding model to convert input words into embedding vectors and establish a "word-embedding vector" vector database. During inference, vector database retrieval is used to replace traditional prefix hash matching, achieving key-value cache reuse under fuzzy semantics. Finally, partial vector recalculation is used to compensate for the accuracy loss caused by approximate reuse, thereby effectively improving the reuse range of the key-value cache and accelerating inference speed while maintaining almost no decrease in model accuracy.

[0027] Specifically, such as Figure 2 As shown, the present invention comprises the following three key parts:

[0028] 1. Establish a word vector database

[0029] Large language model inference systems generate key-value caches based on input lexical units and aggregate multiple lexical units into blocks. The hash value of each lexical block is used as an index to the corresponding key-value cache, linking and storing these "lexical-key-value cache" pairs. This invention, however, uses a word embedding model to additionally generate embedding vectors for the input lexical blocks. A lexical vector database is then built, using the lexical blocks as indices and the embedding vectors as content. The word embedding model maps lexical blocks to a high-dimensional semantic space, and the cosine similarity between different embedding vectors reflects the semantic similarity of corresponding lexical blocks. Therefore, the lexical vector database can overcome the limitations of exact matching in hash indexes, achieving fuzzy matching of input lexical blocks.

[0030] 2. Approximate key-value cache retrieval and reuse

[0031] During inference, when a new inference request is received, the system aggregates input words into blocks and uses a vector database to retrieve historical word blocks that are semantically similar to the current input word block. Specifically, it calculates the cosine similarity between the embedding vector of the current word block and the embedding vectors in the vector database, and filters out similar word blocks based on a pre-set similarity threshold. Then, it obtains the key-value cache corresponding to the similar word blocks through a hash index and reuses this cache during the decoding stage to reduce redundant calculations and improve inference efficiency. If the current input word block has no matching similar word block in the database, it proceeds with normal inference, calculates the corresponding key-value cache, and updates the word vector database with the new word block and its embedding vector to enrich the database content and provide a more comprehensive cache reuse foundation for subsequent inference.

[0032] 3. Partial vector recalculation

[0033] To effectively compensate for the potential accuracy deficiencies caused by reusing approximate key-value caches, this invention proposes a targeted partial vector recalculation strategy. Specifically, in the first Transformer attention calculation layer, the use of approximate key-value caches is abandoned, and instead, accurate attention scores are calculated via standard procedures. Based on the obtained attention scores, high-scoring key-value words representing a certain percentage (e.g., the top 10%) of the total number of words are selected. In subsequent Transformer attention calculation layers, for these key-value words, the approximate key-value cache is no longer used; instead, the key-value cache is recalculated, thereby ensuring the accuracy of the key-value attention calculation. Simultaneously, for the remaining words that are not selected key-value words, if a semantically similar approximate key-value cache is available, it is reused, thereby reducing unnecessary computational overhead and further optimizing inference efficiency.

[0034] Example 1:

[0035] Suppose there are two reasoning requests and Each request has 8 lexical units. Two requests may have similar semantics but different lexical units (e.g., "There is a bird on the branch" and "A bird is perched on the treetop"). For simplicity, assume a lexical block contains 4 lexical units. Having word blocks , Having word blocks , assuming and Semantic similarity, and and The semantics are quite different. The system responds to the request first. And cache the corresponding historical information before processing the request. .

[0036] Existing methods for handling requests At that time, it will be based on Generate key-value caches separately and use respectively and To index. Then process the request. At that time, the system will be based on and Determine if reusable historical information is cached. However, since hash indexes can only perform exact matches, requests... lexical blocks Since it cannot match the existing key-value cache, the system can only execute the normal inference process and cannot use key-value cache reuse technology to accelerate the inference process.

[0037] In processing this embodiment, the present invention performs the following three main steps:

[0038] The first step is to establish a word vector database:

[0039] This invention processes requests At that time, additional word embedding models will be used to add word blocks. Generate embedding vectors and will "", Stored in the vector database. Then, during request processing. At the same time, the system also uses word embedding models to create word blocks. Generate embedding vectors And retrieve data from the vector database related to... The word blocks corresponding to similar embedding vectors. In this example, Will match ,and No semantically similar words were found.

[0040] The second step is approximate key-value cache retrieval and reuse:

[0041] After retrieving semantically similar terms from the vector database, the system will use a hash index to... Key-value cache retrieved This key-value cache This will be directly reused in the request. In attention calculations, it replaces what should be calculated. And word blocks Then, the result is calculated according to the normal procedure. And its embedding vector It will be added to the lexical vector database to enrich historical information.

[0042] The third step is to recalculate some vectors:

[0043] During inference, the system uses a standard procedure to calculate precise attention scores during the first-level attention operation, and selects some high-attention-score lexical units as key lexical units. In this example, the system will select one key lexical unit from eight lexical units, i.e. Figure 3 The black portion in the middle. (For example...) Figure 4 As shown, in subsequent attention layers, the key-value vectors of keyword elements will be recalculated, even if an approximate key-value cache exists. Non-keyword elements, however, will be evaluated based on whether an approximate key-value cache exists, determining whether standard attention computation is performed.

[0044] The specific implementation process of this invention is described above. Experimental results show that, using the open-source long text dataset Longbench / Musique as the workload and selecting LLaMA2-7B as the base model, this invention can reduce the response time of the inference system by 44% while maintaining 96% accuracy, demonstrating significant performance advantages.

[0045] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0046] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0047] In one embodiment, the present invention provides a computer system, which may be a server. The computer system includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data used in the methods described above. The network interface communicates with external terminals via a network connection. The computer program is executed by the processor to implement the methods described above.

[0048] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0049] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0050] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for key-value caching and reuse of large language models based on fuzzy semantic matching, characterized in that, include: Generate key-value cache based on the lexical elements in the user's historical reasoning requests, aggregate multiple lexical elements into lexical blocks, generate embedding vectors for lexical blocks through a word embedding model, and build a vector database with lexical blocks as indexes and embedding vectors as content; Calculate the cosine similarity between the embedding vector of the new inference request's word block and the historical embedding vector in the vector database. If there is a historical word block with a cosine similarity exceeding a set threshold, retrieve the corresponding key-value cache through the hash index and reuse it. In the first Transformer attention calculation layer, the attention score is calculated using a standard process. Based on the attention score, multiple words in the current word block are distinguished into key words and non-key words. Specifically, the words with the highest attention scores in the current word block are selected as key words. In subsequent Transformer attention calculation layers, the key-value cache is recalculated for key words. The recalculated key-value cache for key words and the reused key-value cache for non-key words constitute a hybrid key-value cache. The hybrid key-value cache will be regarded as a key-value cache generated by the user request in the pre-filling stage, and will be continuously used for the prediction of new terms in the decoding stage until the inference process ends. If the cosine similarity between the embedding vector of the new inference request's lexical block and the historical embedding vector in the vector database is less than a set threshold, then the key-value cache of the current lexical block is generated according to the standard process, and the current lexical block and its corresponding embedding vector are updated to the vector database.

2. A computer system comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 1.

Images