Dialogue system interaction method and device based on KV cache reuse

Abstract

The present application provides a dialogue system interaction method and apparatus based on KV cache reuse, including: upon receiving a target cue word, determining a reference cue word, then determining a set of edit operations corresponding to the minimum number of edits required to convert the reference cue word into the target cue word, determining a unified attention map for the reference cue word at each decoder layer, and for each decoder layer, determining the target word-units for the target cue word that require recalculation of the KV cache at that decoder layer based on the set of edit operations and the unified attention map for the reference cue word at that decoder layer. By determining the target word-units using the set of edit operations and the unified attention map, the target word-units for the target cue word that require recalculation of the KV cache at each decoder layer can be accurately identified. For other word-units of the target cue word at each decoder layer, the KV cache of the reference cue word can be reused, thereby maximizing the KV cache reuse rate and the dialogue system's response speed.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a dialogue system interaction method and device based on KV cache reuse. Background Art

[0002] In multi-turn dialogue systems based on large models, the latest query is often input along with previous dialogue context. As dialogue history accumulates, the length of the input prompt increases, leading to increased TTFT (Time To First Token) latency, which reduces the dialogue system's responsiveness. Currently, the KV cache for tokens sharing the same prefix can be reused to partially reduce TTFT latency.

[0003] However, this method of reducing latency by sharing the KV cache of word units with the same prefix can only be used when the prefix of the prompt word is exactly the same as the prefix of the word unit in the KV cache, and only the KV cache of the prefix part can be reused, resulting in limited reuse efficiency of the KV cache, making it difficult to minimize TTFT latency and further improve the response speed of the dialogue system. Summary of the Invention

[0004] The purpose of the present application is to solve at least one of the above-mentioned technical defects, especially the technical defect that the method of reducing delay by sharing the KV cache of word units with the same prefix in the prior art can only be used when the prefix of the prompt word is exactly the same as the prefix of the word unit in the KV cache, and only the KV cache of the prefix part can be reused, resulting in limited reuse efficiency of the KV cache, making it difficult to minimize the TTFT delay and further improve the response speed of the dialogue system.

[0005] In a first aspect, the present application provides a dialogue system interaction method based on KV cache reuse, the method comprising:

[0006] When receiving the target prompt word input by the user, determining the reference prompt word;

[0007] Determine, based on dynamic programming, a set of edit operations corresponding to the minimum number of edits required to convert the reference cue word into the target cue word, and determine a unified attention map for the reference cue word at each decoder layer;

[0008] For each decoder layer, determining a target word-unit of the KV cache to be recalculated for the target word at the decoder layer based on the set of edit operations and the unified attention map of the reference word at the decoder layer;

[0009] When the target word corresponding to the target prompt word in each decoder layer is determined, a KV cache of the target word corresponding to the target prompt word in each decoder layer is calculated;

[0010] The KV cache corresponding to the reference prompt word is reused to obtain the KV cache of other word-grams corresponding to the target prompt word in each decoder layer, and the output content of the target prompt word is determined according to the KV cache of the target word-gram and other word-grams corresponding to the target prompt word in each decoder layer, where the other word-grams refer to word-grams other than the target word-gram.

[0011] In one embodiment, determining the target word-element of the KV cache to be recalculated for the target word at the decoder layer based on the edit operation set and the unified attention map of the reference word at the decoder layer includes:

[0012] Determining, according to the editing operation set, a word-gram to be deleted and a word-gram to be inserted corresponding to the reference prompt word;

[0013] Determine each word to be inserted as the target word corresponding to the target prompt word at the decoder layer;

[0014] For each word to be deleted, the attention score of the subsequent word of the word to be deleted in the reference prompt word is evaluated according to the unified attention map of the reference prompt word at the decoder layer. If there is a word with an attention score higher than a first preset threshold among the subsequent word, the word with an attention score higher than the first preset threshold among the subsequent word is determined as the target word corresponding to the target prompt word at the decoder layer.

[0015] In one embodiment, calculating the KV cache of the target word corresponding to the target prompt word in each decoder layer includes:

[0016] In the current decoder layer, obtaining a first weight set and a second weight set corresponding to the current decoder layer, and obtaining an output vector of a previous decoder layer of the current decoder layer;

[0017] Determining a key vector and a value vector of a target word unit of the target prompt word in the current decoder layer according to the first weight set and the second weight set corresponding to the current decoder layer and the output vector of the previous decoder layer of the current decoder layer, and storing the key vector and the value vector in a preset KV cache pool to obtain a KV cache of the target word unit corresponding to the target prompt word in the current decoder layer;

[0018] An attention score is calculated based on the key vector and the value vector, and the output vector of the current decoder layer is determined based on the attention score. The next decoder layer is entered until all decoder layers are processed in hierarchical order to obtain the KV cache of the target word corresponding to the target prompt word in each decoder layer.

[0019] In one embodiment, reusing the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-units corresponding to the target prompt word in each decoder layer, and determining the output content of the target prompt word based on the KV cache of the target word-units and other word-units corresponding to the target prompt word in each decoder layer, includes:

[0020] For each decoder layer, determine other word-units corresponding to the target cue word in the decoder layer, and search and obtain the KV cache corresponding to the other word-units of the target cue word in the decoder layer in the KV cache corresponding to the reference cue word, so as to obtain the KV cache of the other word-units corresponding to the target cue word in the decoder layer;

[0021] When determining the KV cache of other word units corresponding to the target prompt word in each decoder layer, the output content of the target prompt word is determined based on the KV cache corresponding to other word units of the target prompt word in each decoder layer and the KV cache corresponding to the target word unit of the target prompt word in each decoder layer.

[0022] In one embodiment, determining a unified attention map of the reference cue word at each decoder layer includes:

[0023] Determine the preset attention map cache pool;

[0024] For each decoder layer, obtaining the multi-head attention map of the reference prompt word at the decoder layer from the attention map buffer pool;

[0025] The multi-head attention maps of the reference prompt word at the decoder layer are summed to obtain a unified attention map of the reference prompt word at each decoder layer.

[0026] In one embodiment, determining the set of editing operations corresponding to the minimum number of editing times required to convert the reference prompt word into the target prompt word based on dynamic programming includes:

[0027] Determining a state transition equation based on the objective of determining the minimum number of edits required to convert the reference prompt word into the target prompt word;

[0028] Constructing a two-dimensional transfer table based on the state transfer equation, and performing state backtracking according to the two-dimensional transfer table;

[0029] During the state backtracking process, the editing operations are recorded to obtain the editing operation set.

[0030] In one embodiment, the method further comprises:

[0031] When determining the target word-gram corresponding to the target prompt word in each decoder layer, counting the number of word-grams corresponding to the target word-gram in each decoder layer;

[0032] The decoder layer whose number of word units does not exceed the second preset threshold is marked as the target decoder layer, and the calculation of the attention score of the previous layer of each target decoder layer is skipped in the process of calculating the KV cache of the target word units corresponding to the target prompt word in each decoder layer.

[0033] In a second aspect, the present application provides a dialogue system interaction device based on KV cache reuse, the device comprising:

[0034] a prompt word determination module, configured to determine a reference prompt word upon receiving a target prompt word input by a user;

[0035] an editing operation determination module, configured to determine, based on dynamic programming, a set of editing operations corresponding to a minimum number of edits required to convert the reference cue word into the target cue word, and determine a unified attention map for the reference cue word at each decoder layer;

[0036] a word-unit determination module, configured to determine, for each decoder layer, a target word-unit of the KV cache to be recalculated for the target word at the decoder layer based on the set of edit operations and the unified attention map of the reference word at the decoder layer;

[0037] a cache calculation module, configured to calculate a KV cache of the target word corresponding to the target word in each decoder layer when the target word corresponding to the target word in each decoder layer is determined;

[0038] A cache reuse module is configured to reuse the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-grams corresponding to the target prompt word in each decoder layer, and determine the output content of the target prompt word based on the KV cache of the target word-gram and other word-grams corresponding to the target prompt word in each decoder layer, where the other word-grams refer to word-grams other than the target word-gram.

[0039] In a third aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the dialogue system interaction method based on KV cache reuse as described in any of the above embodiments.

[0040] In a fourth aspect, the present application provides a computer device, comprising: one or more processors, and a memory;

[0041] The memory stores computer-readable instructions, and when the one or more processors execute the computer-readable instructions, they perform the steps of the dialogue system interaction method based on KV cache reuse as described in any of the above embodiments.

[0042] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0043] The present application provides a dialogue system interaction method and apparatus based on KV cache reuse. The method comprises: upon receiving a target prompt word input by a user into the dialogue system, determining a corresponding reference prompt word; determining a set of edit operations corresponding to the minimum number of edits required to convert the reference prompt word into the target prompt word based on dynamic programming; and determining a unified attention map for the reference prompt word at each decoder layer. For each decoder layer, determining, based on the edit operation set and the unified attention map for the reference prompt word at that decoder layer, a target word-meta for the target prompt word requiring recalculation of the KV cache at that decoder layer is determined. By determining each edit operation corresponding to the minimum number of edits using the edit operation set and then combining the unified attention map with the target word-meta, the target word-meta can be accurately identified for the target prompt word requiring recalculation of the KV cache at each decoder layer. For other word-meta corresponding to the target prompt word at each decoder layer, the KV cache of the reference prompt word can be reused to maximize the KV cache reuse rate, thereby further improving the response speed of the dialogue system and reducing TTFT latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A flowchart of a dialog system interaction method based on KV cache reuse provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a process for determining a target word element in the KV cache to be recalculated at the decoder layer for a target prompt word according to an embodiment of the present application;

[0047] Figure 3 This is an example diagram of an application of the KV cache reuse-based dialogue system interaction method provided in an embodiment of the present application;

[0048] Figure 4 An example diagram of the calculation acceleration optimization provided in the embodiment of the present application;

[0049] Figure 5 An application flow chart of the interaction of the dialogue system based on KV cache reuse provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of a dialogue system interaction device based on KV cache reuse provided in an embodiment of the present application;

[0051] Figure 7 This is a diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] In one embodiment, this application provides a method for interacting with a dialog system based on KV cache reuse. The following embodiments illustrate this method as applied to a server. It is understood that the method for interacting with a dialog system based on KV cache reuse can be implemented on a single server or a server cluster consisting of multiple servers, and this application does not impose specific limitations on this.

[0054] like Figure 1 As shown, the present application provides a dialogue system interaction method based on KV cache reuse, the method comprising:

[0055] S101: When a target prompt word input by a user is received, a reference prompt word is determined.

[0056] In this step, when the target prompt word is received from the user into the dialogue system, the server determines a corresponding reference prompt word based on the target prompt word. Specifically, the reference prompt word is the prompt word that is most similar to the target prompt word in all previous dialogue rounds in the dialogue system. Therefore, the server can compare the target prompt word with the prompt words in all previous dialogue rounds to determine the prompt word with the highest similarity and use it as the reference prompt word. It is important to note that the KV cache for each word in the reference prompt word can be directly obtained from the KV cache pool.

[0057] In another embodiment, if the number of prompt words in all previous rounds of conversation is too large, comparing the target prompt word with the prompt words in all previous rounds of conversation for similarity will consume a lot of time and resources. Generally, the prompt word closest to the target prompt word appears in the prompt words of the previous N rounds of conversation, where N is a positive integer greater than zero and N is an empirical value. Therefore, to improve efficiency, the target prompt word can be compared with the prompt words in the previous N rounds of conversation for similarity to improve the efficiency of determining the reference prompt word.

[0058] Among them, the dialogue system is an artificial intelligence application that can be used to understand the text or voice information input by the user, and then generate corresponding responses based on a pre-trained model to simulate the dialogue process between humans.

[0059] S102: Determine a set of editing operations corresponding to the minimum number of editing times required to convert a reference prompt word into a target prompt word based on dynamic programming, and determine a unified attention map of the reference prompt word at each decoder layer.

[0060] The unified attention map refers to a representation of the attention maps of each attention head in the corresponding decoder layer.

[0061] In this step, dynamic programming is used to find the minimum number of edits required to transform a reference cue word into a target cue word. After the solution is found, a set of edit operations is determined based on the solution process. Next, a unified attention map for the reference cue word at each decoder layer is determined based on a preset attention map buffer. It should be understood that the attention map buffer is used to store attention maps generated during the dialogue system's inference process.

[0062] Specifically, the decoder layer in this application refers to the decoder layer in the corresponding large model in the dialogue system. The edit operation set includes two types of operations: insertion and deletion. When determining the edit operation set, if a replacement operation is recorded, the replacement operation is decomposed into a combination of deletion and insertion operations.

[0063] S103: For each decoder layer, determine the target word element of the KV cache to be recalculated for the target word in the decoder layer according to the set of editing operations and the unified attention map of the reference word in the decoder layer.

[0064] KV caching refers to key-value caching. In an autoregressive model, the calculation of the attention representation for each token involves three vectors: query, key, and value. KV caching caches the key and value vectors of previously calculated tokens. During the next calculation, these historical key and value vectors are directly retrieved from the cache. Only the query vector for the current token needs to be recalculated. The cached key and value vectors and the new query vector are then used to calculate the attention representation for the current token. This avoids repeated calculation of the key and value vectors and the associated attention representations for previous tokens, thereby improving model inference speed.

[0065] In this step, for each decoder layer in the dialogue system, the word-gram that needs to be edited in the reference prompt word can be determined based on the editing operation set. Then, the unified attention map of the reference prompt word at this decoder layer is combined to analyze the impact of the word-gram to be edited on the attention score of the subsequent word-gram. In this way, the word-gram that is more affected can be determined as the target word-gram that needs to recalculate the KV cache, thereby accurately calculating the word-gram that needs to recalculate the KV cache in each decoder layer, so as to maximize the reuse rate of the KV cache while ensuring the inference accuracy of the dialogue system, thereby reducing the TTFT delay.

[0066] S104: When the target word-unit corresponding to the target prompt word in each decoder layer is determined, a KV cache of the target word-unit corresponding to the target prompt word in each decoder layer is calculated.

[0067] In this step, when it is determined that the target word corresponding to the target prompt word in each decoder layer needs to be recalculated, the Key and Value vectors of the target word corresponding to the target prompt word in each decoder layer are calculated and stored in the preset KV cache pool. When the calculation of the Key and Value vectors of the target word corresponding to the target prompt word in each decoder layer is completed, the KV cache of the target word corresponding to the target prompt word in each decoder layer is obtained.

[0068] S105: Reuse the KV cache corresponding to the reference prompt word to obtain the KV cache of other word units corresponding to the target prompt word in each decoder layer, and determine the output content of the target prompt word based on the KV cache of the target word unit and other word units corresponding to the target prompt word in each decoder layer.

[0069] Among them, other words refer to words other than the target word.

[0070] In this step, when the KV cache of the target word corresponding to the target prompt word in each decoder layer is determined, the word elements of the target prompt word other than the target word element in each decoder layer can be determined as other word elements, thereby determining the other word elements corresponding to the target prompt word in each decoder layer. This part of the word elements does not need to recalculate the KV cache, and the KV cache matching the corresponding other word elements in the reference prompt word can be directly reused. At this time, the KV cache of the other word elements corresponding to the target prompt word in each decoder layer can be obtained. Finally, the KV cache of the target word element corresponding to the target prompt word in each decoder layer and the KV cache of the other word elements corresponding to the target prompt word in each decoder layer are combined to determine the output content of the target prompt word in the dialogue system.

[0071] For example, suppose the target prompt word is "Is the dog on the mat" and the reference prompt word is "Is the cat on the mat". At this time, if the target prompt word corresponds to dog, the (the second the), and mat in decoder layer A, then the other word units corresponding to the target prompt word in decoder layer A are Is, the (the first the), and on.

[0072] The present application provides a dialogue system interaction method and apparatus based on KV cache reuse. The method comprises: upon receiving a target prompt word input by a user into the dialogue system, determining a corresponding reference prompt word; determining a set of edit operations corresponding to the minimum number of edits required to convert the reference prompt word into the target prompt word based on dynamic programming; and determining a unified attention map for the reference prompt word at each decoder layer. For each decoder layer, determining, based on the edit operation set and the unified attention map for the reference prompt word at that decoder layer, a target word-meta for the target prompt word requiring recalculation of the KV cache at that decoder layer is determined. By determining each edit operation corresponding to the minimum number of edits using the edit operation set and then combining the unified attention map with the target word-meta, the target word-meta can be accurately identified for the target prompt word requiring recalculation of the KV cache at each decoder layer. For other word-meta corresponding to the target prompt word at each decoder layer, the KV cache of the reference prompt word can be reused to maximize the KV cache reuse rate, thereby further improving the response speed of the dialogue system and reducing TTFT latency.

[0073] like Figure 2 As shown, in one embodiment, determining the target word element of the KV cache to be recalculated for the target word at the decoder layer based on the edit operation set and the unified attention map of the reference word at the decoder layer includes:

[0074] S201: Determine the word-grams to be deleted and the word-grams to be inserted corresponding to the reference prompt word according to the editing operation set.

[0075] The edit operation set records the operation type and the word to be edited.

[0076] In this step, based on the editing operation set, the word elements that need to be deleted when the reference prompt word is converted into the target prompt word are determined, and the word elements that need to be inserted when the reference prompt word is converted into the target prompt word are determined. It can be understood that the word elements to be inserted corresponding to the reference prompt word can be found in the target prompt word.

[0077] S202: Determine each word to be inserted as the target word corresponding to the target prompt word at the decoder layer.

[0078] Specifically, since the KV cache of the word to be inserted does not exist in the KV cache corresponding to the reference prompt word, each word to be inserted can be used as the target prompt word. The word in the KV cache needs to be recalculated in each decoder layer, that is, each word to be inserted is determined as the target word corresponding to the target prompt word at the decoder layer.

[0079] S203: For each word to be deleted, the attention score of the subsequent word in the reference prompt word is evaluated according to the unified attention map of the reference prompt word at the decoder layer. If there is a word with an attention score higher than a first preset threshold among the subsequent word, the word with an attention score higher than the first preset threshold among the subsequent word is determined as the target word corresponding to the target prompt word at the decoder layer.

[0080] The attention score is used to measure the richness of the semantic information of the corresponding to-be-deleted word contained in the subsequent word.

[0081] In this step, for each word to be deleted, the attention scores of the subsequent words of the word to be deleted are evaluated according to the unified attention map of the reference prompt word at the decoder layer. This can quantify the impact of deleting the word to be deleted on its subsequent words. For example, if there is a word in the subsequent words whose attention score exceeds the first preset threshold, it means that the word carries more semantic information of the word to be deleted. If the subsequent word is directly reused simply because it is consistent with the reference prompt word, it will affect the accuracy of the dialogue system's reasoning. Therefore, the word in the subsequent words whose attention score is higher than the first preset threshold will also be determined as the target word corresponding to the target prompt word at the decoder layer.

[0082] In one example, if Figure 3 As shown, Figure 3 This is an example diagram of an application of the dialogue system interaction method based on KV cache reuse provided in an embodiment of the present application.

[0083] exist Figure 3In the example, the target cue is "Is the dog on the mat," and the KV cache pool already contains the KV cache for the reference cue, "Is the cat on the mat." In the prior art, KV cache sharing is limited to independent segments with exact prefix matches or high frequency occurrences. For example, prefix tokens like "Is the" can directly reuse the KV cache, but partially similar segments like "on the mat" cannot. In contrast, the method provided in this application performs attention score analysis on the subsequent tokens (on, the, and mat) following the deleted token "cat." Therefore, in certain decoder layers, tokens like "on," "the," and "mat" can reuse the KV cache corresponding to the reference cue. For example, for the target cue, "Is the dog on the mat," tokens that share the same tokens as the reference cue include "Is the" and "on the mat." In this case, only the partial KV cache for the newly added tokens "dog" and "on the mat" can be recalculated. This allows for maximum KV cache recognition and reuse, avoiding duplicate calculations.

[0084] In this embodiment, for each decoder layer, the target word is determined by combining the editing operation set and the unified attention map corresponding to the reference prompt word. This can identify as many word-units as possible in the KV cache that can reuse the reference prompt word, thereby improving the reuse rate of the KV cache and the response speed of the dialogue system.

[0085] In one embodiment, calculating the KV cache of the target word corresponding to the target prompt word in each decoder layer includes:

[0086] S1: In the current decoder layer, obtain the first weight set and the second weight set corresponding to the current decoder layer, and obtain the output vector of the previous decoder layer of the current decoder layer.

[0087] The first weight set includes the key weights for each attention head in the corresponding decoder layer, and the second weight set includes the value weights for each attention head in the corresponding decoder layer.

[0088] S2: Determine the key vector and value vector of the target word corresponding to the target prompt word in the current decoder layer according to the first weight set and the second weight set corresponding to the current decoder layer and the output vector of the previous decoder layer of the current decoder layer, and store the key vector and value vector in a preset KV cache pool to obtain the KV cache of the target word corresponding to the target prompt word in the current decoder layer.

[0089] Among them, the key vector refers to the Key vector, and the value vector refers to the Value vector.

[0090] In this step, the output vector of the previous decoder layer can be used as the input vector of the current decoder layer. The vector corresponding to the target word in the input vector is weighted and summed with each weight in the first weight set to obtain the key vector of the target word corresponding to the target prompt word at the current decoder layer. The vector corresponding to the target word in the input vector is weighted and summed with each weight in the second weight set to obtain the value vector of the target word corresponding to the target word at the current decoder layer. The key vector and value vector are then stored in a preset KV cache pool to obtain the KV cache of the target word corresponding to the target prompt word in the current decoder layer.

[0091] S3: Calculate the attention score based on the key vector and the value vector, and determine the output vector of the current decoder layer based on the attention score, and enter the next decoder layer until all decoder layers are processed in hierarchical order, and obtain the KV cache of the target word corresponding to the target prompt word in each decoder layer.

[0092] It can be understood that the output vector of the current decoder layer can be used as the input vector of the next decoder layer. Specifically, the output content of the target word corresponding to the target prompt word in the current decoder layer can be determined based on the attention score, and then the output vector of the previous decoder layer of the current decoder layer can be updated according to the output content to obtain the output vector of the current decoder layer.

[0093] In this embodiment, when the target word units corresponding to the target prompt word at each decoder layer are determined, KV cache calculations can be performed only on these target word units, thereby saving computing resources and avoiding repeated calculations as much as possible.

[0094] In one embodiment, reusing the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-units corresponding to the target prompt word in each decoder layer, and determining the output content of the target prompt word based on the KV cache of the target word-units and other word-units corresponding to the target prompt word in each decoder layer, includes:

[0095] S1: For each decoder layer, determine the other word-units corresponding to the target prompt word in the decoder layer, and search and obtain the KV cache corresponding to the other word-units of the target prompt word in the decoder layer in the KV cache corresponding to the reference prompt word, so as to obtain the KV cache of the other word-units corresponding to the target prompt word in the decoder layer.

[0096] S2: When determining the KV cache of other word units corresponding to the target prompt word in each decoder layer, determine the output content of the target prompt word based on the KV cache corresponding to other word units of the target prompt word in each decoder layer and the KV cache corresponding to the target word unit of the target prompt word in each decoder layer.

[0097] In this embodiment, when determining the KV cache of the target word corresponding to the target prompt word in each decoder layer, the KV cache of other words corresponding to the target prompt word in each decoder layer is determined by reusing the KV cache of the reference prompt word. Finally, the output content of the target prompt word in the dialogue system is determined based on the KV caches corresponding to other words of the target prompt word in each decoder layer and the KV caches corresponding to the target words of the target prompt word in each decoder layer, thereby improving the reuse rate of the KV cache in the dialogue system and improving the response speed of the dialogue system.

[0098] In one embodiment, determining a unified attention map for a reference cue word at each decoder layer includes:

[0099] S1: Determine the preset attention map cache pool.

[0100] S2: For each decoder layer, obtain the multi-head attention map of the reference prompt word at this decoder layer from the attention map cache pool.

[0101] Among them, the multi-head attention map includes the attention map of each attention head.

[0102] S3: Sum the multi-head attention maps of the reference prompt word at the decoder layer to obtain a unified attention map of the reference prompt word at each decoder layer.

[0103] In this embodiment, the multi-head attention maps of each decoder layer are summed to obtain a unified attention map of the reference prompt word at each decoder layer. Since the unified attention map can comprehensively reflect the attention information of each attention head in the multi-head attention map, this can simplify the subsequent analysis of the impact of the deleted word on the subsequent word based on the attention score, thereby improving the reasoning efficiency of the dialogue system.

[0104] In one embodiment, determining a set of editing operations corresponding to the minimum number of editing times required to convert a reference prompt word into a target prompt word based on dynamic programming includes:

[0105] S1: Determine the state transition equation based on the goal of determining the minimum number of edits required to transform the reference cue word into the target cue word.

[0106] In one example, the state transition equation can be expressed as follows:

[0107]

[0108] Where, Indicates that the substring [0:i] Convert to substring [0:j] The minimum number of edits required, [0:i] represents the substring formed by the first i characters in the reference prompt word, [0:j] represents the substring formed by the first j characters in the target prompt word, deletion represents the deletion operation, insertion represents the insertion operation, and subtitution represents the replacement operation. [i]= [j] hour, [i] The value of [j] is 0. [i] [j] hour, [i] The value of [j] is 1.

[0109] Specifically, the update principles for delete, insert, and replace operations are as follows:

[0110] 1. Deletion operation: If you delete the i-th character of the reference prompt and then convert the remaining i-1 characters to the first j characters of the target prompt, then the number of edits at this time is equal to the number of edits to convert the first i-1 characters of the reference prompt to the first j characters of the target prompt (i.e., d(i-1, j)) plus 1, because one character was deleted.

[0111] 2. Insertion operation: If a character is inserted based on the first i characters of the reference prompt so that it matches the first j characters of the target prompt, then the number of edits at this time is equal to the number of edits required to convert the first i characters of the reference prompt to the first j - 1 characters of the target prompt (i.e., d(i, j-1)) plus 1, because one character was inserted.

[0112] 3. Replacement: If the i-th character of the reference cue is replaced with the j-th character of the target cue, and then the first i-1 characters of the reference cue are converted to the first j-1 characters of the target cue, then the number of edits is equal to the number of edits required to convert the first i-1 characters of the reference cue to the first j-1 characters of the target cue (i.e., d(i-1, j-1)). However, if the i-th character of the reference cue and the j-th character of the target cue are the same, no replacement is performed, and the number of edits is equal to d(i-1, j-1). If they are different, the number of edits is increased by 1, indicating that a replacement operation has been performed.

[0113] S2: Construct a two-dimensional transfer table based on the state transfer equation, and perform state backtracking according to the two-dimensional transfer table.

[0114] In this step, a blank two-dimensional table can be constructed first, and the data calculated based on the state transfer equation can be filled into the two-dimensional table to achieve a two-dimensional transfer table. Specifically, the value in the two-dimensional transfer table can be expressed as the minimum number of edits required to convert the substring formed by the first i characters of the reference prompt word into the substring formed by the first j characters of the target prompt word.

[0115] S3: Record the editing operations during the state backtracking process to obtain the editing operation set.

[0116] In this step, after the two-dimensional transfer table is determined, backtracking is performed from the lower right corner d(n, m) of the two-dimensional transfer table to d(0, 0), and specific editing operations and editing words are recorded during the backtracking process.

[0117] In one example, the state of the two-dimensional transition table can be traced back according to the following rules:

[0118] (1) If d(i, j) = d(i-1, j) + 1, it means that a deletion operation was performed in the previous step, that is, the i-th character of the reference prompt word was deleted, and the process is backtracked to d(i-1, j).

[0119] (2) If d(i, j) = d(i, j-1)+1, it means that an insertion operation was performed in the previous step, that is, a character was inserted to match the j-th character of the target prompt word, and then backtracking to d(i, j-1).

[0120] (3) If d(i, j) = d(i- 1, j - 1)+ ( [i] [j]), it means that the previous step took a replacement operation (when [i] [j]), or no action is taken (when [i] [j]), then backtrack to d(i- 1 , j - 1).

[0121] In this embodiment, dynamic programming can be used to determine the set of editing operations corresponding to the minimum number of edits required to convert a reference prompt word into a target prompt word, and then determine the editing method that can minimize the recalculation of the KV cache. Then, based on this method, the editing operations are recorded to form an editing operation set to avoid unnecessary repeated calculations, thereby improving the response speed of the dialogue system and reducing the TTFT delay.

[0122] In one embodiment, the dialogue system interaction method based on KV cache reuse further includes:

[0123] S1: When determining the target word-unit corresponding to the target prompt word in each decoder layer, count the number of word-units of the target word-unit corresponding to the target prompt word in each decoder layer.

[0124] S2: Mark the decoder layer whose word number does not exceed the second preset threshold as the target decoder layer, and skip the calculation of the attention score of the previous layer of each target decoder layer in the process of calculating the KV cache of the target word corresponding to the target prompt word in each decoder layer.

[0125] In this embodiment, the number of target word units corresponding to the target prompt word at each decoder layer is counted. For each decoder layer, if the number of word units does not exceed the second preset threshold, it means that skipping the attention score calculation of the previous layer of the decoder layer has little impact on the decoder layer. Therefore, in order to further improve the response speed of the dialogue system, the attention score calculation of the previous layer of the marked target decoder layer can be skipped, and the attention scores of the first two layers of the target decoder layer can be directly used. This can further improve the response speed of the dialogue system.

[0126] In one example, if Figure 4 As shown, Figure 4 This is an example diagram of the calculation acceleration optimization provided by the embodiment of this application. Figure 4 In the figure, the dotted box represents the target decoder layer. After optimization, the layer above the target decoder layer represented by the dotted box will skip the attention score and output calculation. Figure 4 It can be seen that after optimization, some computing time can be saved compared to before optimization, so as to further improve the response speed of the dialogue system.

[0127] In one example, if Figure 5 As shown, Figure 5 This is an application flow chart of the interaction of the dialogue system based on KV cache reuse provided in the embodiment of this application. Figure 5 In the example, the previous conversation prompt is the reference prompt, and the new query prompt is the target prompt. The new KV cache refers to the KV cache of the target tokens corresponding to the calculated target prompt at each decoder layer. The KV cache and attention map corresponding to the target prompt determined during the calculation process are stored in the KV cache pool and attention map cache pool, respectively.

[0128] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0129] The following describes a dialogue system interaction device based on KV cache reuse provided in an embodiment of the present application. The dialogue system interaction device based on KV cache reuse described below and the dialogue system interaction method based on KV cache reuse described above can be referenced to each other.

[0130] like Figure 6 As shown, the present application provides a dialogue system interaction device 300 based on KV cache reuse, the device comprising:

[0131] The prompt word determination module 301 is used to determine a reference prompt word when receiving a target prompt word input by a user;

[0132] An edit operation determination module 302 is configured to determine, based on dynamic programming, a set of edit operations corresponding to the minimum number of edits required to convert a reference cue word into a target cue word, and determine a unified attention map for the reference cue word at each decoder layer;

[0133] A word-unit determination module 303 is configured to determine, for each decoder layer, a target word-unit of the KV cache to be recalculated for the target word at the decoder layer based on the set of edit operations and the unified attention map of the reference word at the decoder layer;

[0134] A cache calculation module 304 is configured to calculate a KV cache of the target word corresponding to the target word in each decoder layer when the target word corresponding to the target word in each decoder layer is determined;

[0135] The cache reuse module 305 is used to reuse the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-units corresponding to the target prompt word in each decoder layer, and determine the output content of the target prompt word based on the KV cache of the target word-unit and other word-units corresponding to the target prompt word in each decoder layer, where other word-units refer to word-units other than the target word-unit.

[0136] In one embodiment, the word unit determination module includes:

[0137] An editing word-unit determination submodule is used to determine the word-unit to be deleted and the word-unit to be inserted corresponding to the reference prompt word according to the editing operation set;

[0138] The first word unit determination submodule is used to determine each word unit to be inserted as the target word unit corresponding to the target prompt word at the decoder layer;

[0139] The second word-unit determination submodule is used to evaluate, for each word-unit to be deleted, the attention score of the subsequent word-units of the word-unit to be deleted in the reference prompt word according to the unified attention map of the reference prompt word at the decoder layer; if there is a word-unit with an attention score higher than a first preset threshold among the subsequent word-units, the word-unit with an attention score higher than the first preset threshold among the subsequent word-units is determined as the target word-unit corresponding to the target prompt word at the decoder layer.

[0140] In one embodiment, the cache calculation module includes:

[0141] The weight acquisition submodule is used to obtain, in the current decoder layer, the first weight set and the second weight set corresponding to the current decoder layer, and to obtain the output vector of the previous decoder layer of the current decoder layer;

[0142] a cache calculation submodule, configured to determine a key vector and a value vector corresponding to a target word element of a target prompt word at a current decoder layer based on a first weight set and a second weight set corresponding to a current decoder layer and an output vector of a previous decoder layer of the current decoder layer, and to store the key vector and the value vector in a preset KV cache pool to obtain a KV cache of the target word element corresponding to the target prompt word at the current decoder layer;

[0143] The iterative processing submodule is used to calculate the attention score based on the key vector and the value vector, and determine the output vector of the current decoder layer based on the attention score, and enter the next decoder layer until all decoder layers are processed in hierarchical order, and the KV cache of the target word corresponding to the target prompt word in each decoder layer is obtained.

[0144] In one embodiment, the cache reuse module includes:

[0145] A cache determination submodule is configured to determine, for each decoder layer, other word-units corresponding to the target cue word in the decoder layer, and search and obtain the KV cache corresponding to the other word-units of the target cue word in the decoder layer in the KV cache corresponding to the reference cue word, so as to obtain the KV cache of the other word-units corresponding to the target cue word in the decoder layer;

[0146] The cache reuse submodule is used to determine the output content of the target prompt word based on the KV cache corresponding to other word units of the target prompt word in each decoder layer and the KV cache corresponding to the target word unit of the target prompt word in each decoder layer when determining the KV cache of other word units corresponding to the target prompt word in each decoder layer.

[0147] In one embodiment, the editing operation determination module includes:

[0148] The buffer pool determination submodule is used to determine the preset attention map buffer pool;

[0149] The graph acquisition submodule is used to obtain the multi-head attention graph of the reference prompt word at each decoder layer from the attention graph cache pool;

[0150] The graph processing submodule is used to sum the multi-head attention graphs of the reference prompt word at the decoder layer to obtain a unified attention graph of the reference prompt word at each decoder layer.

[0151] In one embodiment, the editing operation determination module includes:

[0152] an equation determination submodule for determining a state transition equation based on the purpose of determining a minimum number of edits required to convert a reference prompt word into a target prompt word;

[0153] A state backtracking submodule is used to construct a two-dimensional transfer table based on the state transfer equation and perform state backtracking according to the two-dimensional transfer table;

[0154] The operation record submodule is used to record the editing operation during the state backtracking process to obtain the editing operation set.

[0155] In one embodiment, the KV cache reuse-based dialogue system interaction device further includes:

[0156] a quantity counting module, configured to count the number of target word units corresponding to the target prompt word in each decoder layer when determining the target word units corresponding to the target prompt word in each decoder layer;

[0157] A calculation skipping module is used to mark the decoder layer whose word number does not exceed the second preset threshold as the target decoder layer, and skip the calculation of the attention score of the previous layer of each target decoder layer in the process of calculating the KV cache of the target word corresponding to the target prompt word in each decoder layer.

[0158] The division of the various modules in the above-mentioned dialogue system interaction device based on KV cache reuse is only for illustration. In other embodiments, the dialogue system interaction device based on KV cache reuse can be divided into different modules as needed to complete all or part of the functions of the above-mentioned dialogue system interaction device based on KV cache reuse. The various modules in the above-mentioned dialogue system interaction device based on KV cache reuse can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0159] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the dialogue system interaction method based on KV cache reuse as described in any of the above embodiments.

[0160] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the dialogue system interaction method based on KV cache reuse as described in any of the above embodiments.

[0161] Schematically, as Figure 7 As shown, Figure 7 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 400 can be provided as a server. Figure 7 Computer device 400 includes a processing component 402, which further includes one or more processors, and a memory resource represented by memory 401 for storing instructions executable by processing component 402, such as an application. The application stored in memory 401 may include one or more modules, each corresponding to a set of instructions. In addition, processing component 402 is configured to execute the instructions to perform the dialogue system interaction method based on KV cache reuse according to any of the above embodiments.

[0162] The computer device 400 may further include a power supply component 403 configured to perform power management of the computer device 400, a wired or wireless network interface 404 configured to connect the computer device 400 to a network, and an input / output (I / O) interface 405. The computer device 400 may operate based on an operating system stored in the memory 401, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or the like.

[0163] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0164] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment comprising a series of elements not only include those elements, but also include other elements not clearly listed, or also include elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment comprising the elements. Herein, the singular forms "one", "an" and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" etc. specify the existence of stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the existence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0165] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referenced to each other.

[0166] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dialogue system interaction method based on KV cache reuse, characterized in that: The method comprises: When receiving a target prompt word input by the user, determining a reference prompt word; the reference prompt word is the prompt word that is most similar to the target prompt word in all previous rounds of dialogue in the dialogue system; Determine, based on dynamic programming, a set of edit operations corresponding to the minimum number of edits required to convert the reference cue word into the target cue word, and determine a unified attention map for the reference cue word at each decoder layer; For each decoder layer, determining a word to be deleted and a word to be inserted corresponding to the reference prompt word according to the set of editing operations; Determine each word to be inserted as the target word corresponding to the target prompt word at the decoder layer; For each to-be-deleted word-gram, the attention scores of subsequent words of the to-be-deleted word-gram in the reference cue word are evaluated according to the unified attention map of the reference cue word at the decoder layer. If any of the subsequent words-grams has an attention score higher than a first preset threshold, the word-gram with an attention score higher than the first preset threshold is determined as the target word-gram corresponding to the target cue word at the decoder layer. When the target word corresponding to the target prompt word in each decoder layer is determined, a KV cache of the target word corresponding to the target prompt word in each decoder layer is calculated; The KV cache corresponding to the reference prompt word is reused to obtain the KV cache of other word-grams corresponding to the target prompt word in each decoder layer, and the output content of the target prompt word is determined according to the KV cache of the target word-gram and other word-grams corresponding to the target prompt word in each decoder layer, where the other word-grams refer to word-grams other than the target word-gram.

2. The method for interactive dialogue system based on KV cache reuse according to claim 1, characterized in that: The calculating of the KV cache of the target word corresponding to the target prompt word in each decoder layer includes: In the current decoder layer, obtaining a first weight set and a second weight set corresponding to the current decoder layer, and obtaining an output vector of a previous decoder layer of the current decoder layer; Determining a key vector and a value vector of a target word corresponding to the target prompt word in the current decoder layer according to the first weight set and the second weight set corresponding to the current decoder layer and the output vector of the previous decoder layer of the current decoder layer, and storing the key vector and the value vector in a preset KV cache pool to obtain a KV cache of the target word corresponding to the target prompt word in the current decoder layer; An attention score is calculated based on the key vector and the value vector, and the output vector of the current decoder layer is determined based on the attention score. The next decoder layer is entered until all decoder layers are processed in hierarchical order to obtain the KV cache of the target word corresponding to the target prompt word in each decoder layer.

3. The method for interactive dialogue system based on KV cache reuse according to claim 1, characterized in that: The reusing the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-units corresponding to the target prompt word in each decoder layer, and determining the output content of the target prompt word according to the KV cache of the target word-units and other word-units corresponding to the target prompt word in each decoder layer, includes: For each decoder layer, determine other word-units corresponding to the target cue word in the decoder layer, and search and obtain the KV cache corresponding to the other word-units of the target cue word in the decoder layer in the KV cache corresponding to the reference cue word, so as to obtain the KV cache of the other word-units corresponding to the target cue word in the decoder layer; When determining the KV cache of other word units corresponding to the target prompt word in each decoder layer, the output content of the target prompt word is determined based on the KV cache corresponding to other word units of the target prompt word in each decoder layer and the KV cache corresponding to the target word unit of the target prompt word in each decoder layer.

4. The method for interactive dialogue system based on KV cache reuse according to claim 1, characterized in that: Determining a unified attention map of the reference prompt word at each decoder layer includes: Determine the preset attention map cache pool; For each decoder layer, obtaining the multi-head attention map of the reference prompt word at the decoder layer from the attention map buffer pool; The multi-head attention maps of the reference prompt word at the decoder layer are summed to obtain a unified attention map of the reference prompt word at each decoder layer.

5. The method for interactive dialogue system based on KV cache reuse according to claim 1, characterized in that: The dynamic programming-based determination of a set of editing operations corresponding to the minimum number of editing times required to convert the reference prompt word into the target prompt word includes: Determining a state transition equation based on the objective of determining the minimum number of edits required to convert the reference prompt word into the target prompt word; Constructing a two-dimensional transfer table based on the state transfer equation, and performing state backtracking according to the two-dimensional transfer table; During the state backtracking process, the editing operations are recorded to obtain the editing operation set.

6. The method for interactive dialogue system based on KV cache reuse according to any one of claims 1 to 5, characterized in that: The method further comprises: When determining the target word-gram corresponding to the target prompt word in each decoder layer, counting the number of word-grams corresponding to the target word-gram in each decoder layer; The decoder layer whose number of word units does not exceed the second preset threshold is marked as the target decoder layer, and the calculation of the attention score of the previous layer of each target decoder layer is skipped in the process of calculating the KV cache of the target word units corresponding to the target prompt word in each decoder layer.

7. A dialogue system interaction device based on KV cache reuse, characterized in that: The device comprises: a prompt word determination module, configured to determine a reference prompt word upon receiving a target prompt word input by the user; the reference prompt word being the prompt word that is most similar to the target prompt word in all previous rounds of dialogue in the dialogue system; an editing operation determination module, configured to determine, based on dynamic programming, a set of editing operations corresponding to a minimum number of edits required to convert the reference cue word into the target cue word, and determine a unified attention map for the reference cue word at each decoder layer; a word-gram determination module configured to determine, for each decoder layer, word-grams to be deleted and word-grams to be inserted corresponding to the reference prompt word according to the set of edit operations; determine each word-gram to be inserted as a target word-gram corresponding to the target prompt word at that decoder layer; and, for each word-gram to be deleted, evaluate the attention score of subsequent word-grams of the word-gram to be deleted in the reference prompt word according to the unified attention map of the reference prompt word at that decoder layer; if any word-gram in the subsequent word-grams has an attention score higher than a first preset threshold, determine the word-gram in the subsequent word-grams with an attention score higher than the first preset threshold as the target word-gram corresponding to the target prompt word at that decoder layer; a cache calculation module, configured to calculate a KV cache of the target word corresponding to the target word in each decoder layer when the target word corresponding to the target word in each decoder layer is determined; A cache reuse module is configured to reuse the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-grams corresponding to the target prompt word in each decoder layer, and determine the output content of the target prompt word based on the KV cache of the target word-gram and other word-grams corresponding to the target prompt word in each decoder layer, where the other word-grams refer to word-grams other than the target word-gram.

8. A storage medium, characterized in that: The storage medium stores computer-readable instructions, which, when executed by one or more processors, enable the one or more processors to perform the steps of the dialogue system interaction method based on KV cache reuse as described in any one of claims 1 to 6.

9. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, execute the steps of the dialogue system interaction method based on KV cache reuse as described in any one of claims 1 to 6.

Images