Conversation system interaction method and device based on KV cache multiplexing

Abstract

The invention provides a conversation system interaction method and device based on KV cache multiplexing, and the method comprises the steps: determining a reference cue word when a target cue word is received, and then determining an editing operation set corresponding to the minimum number of editing times needed for converting the reference cue word into the target cue word, and for each decoder layer, determining a unified attention map of the reference cue word in each decoder layer, and for each decoder layer, determining a target lexical element of the target cue word needing to recalculate the KV cache in the decoder layer according to the editing operation set and the unified attention map of the reference cue word in the decoder layer. According to the method, the target lexical elements are determined by editing the operation set and the unified attention graph, so that the target lexical elements, needing to be recalculated in the KV cache, of the target cue word in each decoder layer can be accurately recognized, and the KV cache of the reference cue word can be reused for other lexical elements, in each decoder layer, of the target cue word; the reuse rate of the KV cache and the response speed of the dialogue system are improved as much as possible.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a dialogue system interaction method and device based on KV cache reuse. Background Art

[0002] In a multi-turn dialogue system based on a large model, the latest query is usually input together with the previous dialogue context information. As the dialogue history accumulates, the length of the input prompt word continues to increase, and the TTFT (Time To First Token) delay also increases, which leads to a decrease in the response speed of the dialogue system. Currently, a part of the KV cache can be reused by sharing the KV cache of the word units with the same prefix. This method reduces the TTFT delay to a certain extent.

[0003] However, this method of reducing latency by sharing the KV cache of word units with the same prefix can only be used when the prefix of the prompt word is exactly the same as the prefix of the word unit in the KV cache, and only the KV cache of the prefix part can be reused, resulting in limited reuse efficiency of the KV cache, making it difficult to minimize TTFT latency and further improve the response speed of the dialogue system. Summary of the invention

[0004] The purpose of the present application is to solve at least one of the above-mentioned technical defects, especially the technical defect that the method of reducing delay by sharing the KV cache of word units with the same prefix in the prior art can only be used when the prefix of the prompt word is exactly the same as the prefix of the word unit in the KV cache, and only the KV cache of the prefix part can be reused, resulting in limited reuse efficiency of the KV cache, making it difficult to minimize the TTFT delay and further improve the response speed of the dialogue system.

[0005] In a first aspect, the present application provides a dialogue system interaction method based on KV cache reuse, the method comprising:

[0006] When receiving the target prompt word input by the user, determining the reference prompt word;

[0007] Determine a set of edit operations corresponding to the minimum number of edits required to convert the reference cue word into the target cue word based on dynamic programming, and determine a unified attention map of the reference cue word at each decoder layer;

[0008] For each decoder layer, determining a target word-element of the KV cache to be recalculated for the target word in the decoder layer according to the set of edit operations and a unified attention map of the reference word in the decoder layer;

[0009] When the target word element corresponding to the target prompt word in each decoder layer is determined, a KV cache of the target word element corresponding to the target prompt word in each decoder layer is calculated;

[0010] The KV cache corresponding to the reference prompt word is reused to obtain the KV cache of other word-grams corresponding to the target prompt word in each decoder layer, and the output content of the target prompt word is determined according to the KV cache of the target word-gram and other word-grams corresponding to the target prompt word in each decoder layer, where the other word-grams refer to word-grams other than the target word-gram.

[0011] In one embodiment, determining the target word element of the KV cache to be recalculated for the target word at the decoder layer according to the edit operation set and the unified attention map of the reference word at the decoder layer includes:

[0012] Determine the word-gram to be deleted and the word-gram to be inserted corresponding to the reference prompt word according to the editing operation set;

[0013] Determine each word to be inserted as the target word corresponding to the target prompt word at the decoder layer;

[0014] For each word-gram to be deleted, the attention score of the subsequent word-grams of the word-gram to be deleted in the reference prompt word is evaluated according to the unified attention map of the reference prompt word at the decoder layer. If there are word-grams with attention scores higher than a first preset threshold among the subsequent word-grams, the word-grams with attention scores higher than the first preset threshold among the subsequent word-grams are determined as the target word-grams corresponding to the target prompt word at the decoder layer.

[0015] In one embodiment, the step of calculating the KV cache of the target word element corresponding to the target prompt word in each decoder layer includes:

[0016] In the current decoder layer, obtaining a first weight set and a second weight set corresponding to the current decoder layer, and obtaining an output vector of a previous decoder layer of the current decoder layer;

[0017] Determine a key vector and a value vector of a target word unit of the target prompt word in the current decoder layer according to a first weight set and a second weight set corresponding to the current decoder layer and an output vector of a previous decoder layer of the current decoder layer, and store the key vector and the value vector in a preset KV cache pool to obtain a KV cache of the target word unit corresponding to the target prompt word in the current decoder layer;

[0018] An attention score is calculated according to the key vector and the value vector, and an output vector of the current decoder layer is determined according to the attention score to enter the next decoder layer until all decoder layers are processed in hierarchical order to obtain a KV cache of the target word element corresponding to the target prompt word in each decoder layer.

[0019] In one embodiment, the reusing of the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-units corresponding to the target prompt word in each decoder layer, and determining the output content of the target prompt word according to the KV cache of the target word-units and other word-units corresponding to the target prompt word in each decoder layer, includes:

[0020] For each decoder layer, determine other word-units corresponding to the target prompt word in the decoder layer, and search and obtain the KV cache corresponding to other word-units of the target prompt word in the decoder layer in the KV cache corresponding to the reference prompt word, so as to obtain the KV cache of other word-units corresponding to the target prompt word in the decoder layer;

[0021] When determining the KV cache of other word units corresponding to the target prompt word in each decoder layer, the output content of the target prompt word is determined based on the KV cache corresponding to other word units of the target prompt word in each decoder layer and the KV cache corresponding to the target word unit of the target prompt word in each decoder layer.

[0022] In one embodiment, determining a unified attention map of the reference cue word at each decoder layer includes:

[0023] Determine the preset attention map cache pool;

[0024] For each decoder layer, obtaining a multi-head attention map of the reference prompt word at the decoder layer from the attention map buffer pool;

[0025] The multi-head attention maps of the reference prompt word at the decoder layer are summed to obtain a unified attention map of the reference prompt word at each decoder layer.

[0026] In one embodiment, the determining, based on dynamic programming, a set of editing operations corresponding to the minimum number of editing times required to convert the reference prompt word into the target prompt word comprises:

[0027] Determining a state transition equation for the purpose of determining the minimum number of edits required to convert the reference prompt word into the target prompt word;

[0028] Constructing a two-dimensional transfer table based on the state transfer equation, and performing state backtracking according to the two-dimensional transfer table;

[0029] The editing operations are recorded during the state backtracking process to obtain an editing operation set.

[0030] In one embodiment, the method further comprises:

[0031] When determining the target word-meta corresponding to the target prompt word in each decoder layer, counting the number of word-meta of the target word-meta corresponding to the target prompt word in each decoder layer;

[0032] The decoder layer whose word number does not exceed the second preset threshold is marked as the target decoder layer, and in the process of calculating the KV cache of the target word corresponding to the target prompt word in each decoder layer, the calculation of the attention score of the previous layer of each target decoder layer is skipped.

[0033] In a second aspect, the present application provides a dialog system interaction device based on KV cache reuse, the device comprising:

[0034] A prompt word determination module, used to determine a reference prompt word when receiving a target prompt word input by a user;

[0035] An editing operation determination module, configured to determine, based on dynamic programming, a set of editing operations corresponding to a minimum number of editing times required to convert the reference prompt word into the target prompt word, and determine a unified attention map of the reference prompt word at each decoder layer;

[0036] A word unit determination module is used to determine, for each decoder layer, a target word unit of the KV cache to be recalculated for the target word in the decoder layer according to the set of editing operations and the unified attention map of the reference word in the decoder layer;

[0037] A cache calculation module, used for calculating the KV cache of the target word corresponding to the target prompt word in each decoder layer when the target word corresponding to the target prompt word in each decoder layer is determined;

[0038] A cache reuse module is used to reuse the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-units corresponding to the target prompt word in each decoder layer, and determine the output content of the target prompt word according to the KV cache of the target word-unit and other word-units corresponding to the target prompt word in each decoder layer, wherein the other word-unit refers to the word-unit other than the target word-unit.

[0039] In a third aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the dialogue system interaction method based on KV cache reuse as described in any of the above embodiments.

[0040] In a fourth aspect, the present application provides a computer device, comprising: one or more processors, and a memory;

[0041] The memory stores computer-readable instructions, and when the one or more processors execute the computer-readable instructions, the steps of the dialogue system interaction method based on KV cache reuse as described in any of the above embodiments are performed.

[0042] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0043] The present application provides a method and device for a dialogue system interaction based on KV cache reuse, the method comprising: when receiving a target prompt word input by a user into the dialogue system, determining a corresponding reference prompt word, determining a set of edit operations corresponding to the minimum number of edits required to convert the reference prompt word into the target prompt word based on dynamic programming, and determining a unified attention map of the reference prompt word at each decoder layer, for each decoder layer, determining the target word element of the KV cache that needs to be recalculated for the target prompt word in the decoder layer according to the set of edit operations and the unified attention map of the reference prompt word at the decoder layer. The edit operations corresponding to the minimum number of edits are determined by the edit operation set, and then the target word element is determined in combination with the unified attention map, so that the target word element of the KV cache that needs to be recalculated for the target prompt word in each decoder layer can be accurately identified, and the KV cache of the reference prompt word can be reused for other word elements corresponding to the target prompt word in each decoder layer, so as to maximize the reuse rate of the KV cache, thereby further improving the response speed of the dialogue system and reducing the TTFT delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A flowchart of a dialog system interaction method based on KV cache reuse provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a process for determining a target word element of the KV cache to be recalculated at the decoder layer for a target prompt word provided in an embodiment of the present application;

[0047] Figure 3 An example diagram of an application of a dialog system interaction method based on KV cache reuse provided in an embodiment of the present application;

[0048] Figure 4 An example diagram of the calculation acceleration optimization provided by the embodiment of the present application;

[0049] Figure 5 An application flow chart of the interaction of the dialogue system based on KV cache reuse provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of a dialogue system interaction device based on KV cache reuse provided in an embodiment of the present application;

[0051] Figure 7 An internal structure diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] In one embodiment, the present application provides a method for interacting with a dialog system based on KV cache reuse, and the following embodiment is described by applying the method to a server. It can be understood that the method for interacting with a dialog system based on KV cache reuse can be performed by a single server or a server cluster composed of multiple servers, and the present application does not make any specific restrictions on this.

[0054] like Figure 1 As shown, the present application provides a dialogue system interaction method based on KV cache reuse, the method comprising:

[0055] S101: When receiving a target prompt word input by a user, determining a reference prompt word.

[0056] In this step, when receiving the target prompt word input by the user into the dialogue system, the server determines the corresponding reference prompt word according to the target prompt word. Specifically, the reference prompt word refers to the prompt word that is most similar to the target prompt word in all previous rounds of dialogue in the dialogue system. Therefore, the server can compare the target prompt word with the prompt words in all previous rounds of dialogue to determine the prompt word with the highest similarity and use it as the reference prompt word. It should be noted that the KV cache of each word unit in the reference prompt word can be directly obtained from the KV cache pool.

[0057] In another embodiment, if the number of prompt words in all previous rounds of dialogue is too large, then comparing the target prompt word with the prompt words in all previous rounds of dialogue for similarity will consume more time and resources. In general, the prompt word closest to the target prompt word appears in the prompt words of the first N rounds of dialogue, where N is a positive integer greater than zero and N is an empirical value. Therefore, in order to improve efficiency, the target prompt word can be compared with the prompt words of the first N rounds of dialogue for similarity to improve the efficiency of determining the reference prompt word.

[0058] Among them, the dialogue system is an artificial intelligence application that can be used to understand the text or voice information input by the user, and then generate corresponding replies based on a pre-trained model to simulate the dialogue process between humans.

[0059] S102: Determine a set of editing operations corresponding to the minimum number of editing times required to convert a reference prompt word into a target prompt word based on dynamic programming, and determine a unified attention map of the reference prompt word at each decoder layer.

[0060] Among them, the unified attention map refers to the representation map that integrates the attention maps of each attention head in the corresponding decoder layer.

[0061] In this step, dynamic programming is used to solve the minimum number of edits required to convert the reference prompt word into the target prompt word, and then the edit operation set is determined according to the solving process after the solution. Then, the unified attention map of the reference prompt word at each decoder layer is determined based on the preset attention map buffer pool. It can be understood that the attention map buffer pool is used to store the attention map generated during the reasoning process of the dialogue system.

[0062] Specifically, the decoder layer in this application refers to the decoder layer in the corresponding large model in the dialogue system. The edit operation set includes two types of operations: insertion and deletion. When determining the edit operation set, if a replacement operation is recorded, the replacement operation is decomposed into a combination operation of deletion and insertion.

[0063] S103: For each decoder layer, according to the set of editing operations and the unified attention map of the reference prompt word at the decoder layer, determine the target word element of the KV cache to be recalculated for the target prompt word at the decoder layer.

[0064] Among them, KV cache refers to Key-Value cache. In the autoregressive model, the three vectors of Query, Key, and Value are involved when calculating the attention representation of each word. KV cache is to cache the Key and Value vectors of previously calculated words, that is, KV cache. When calculating next time, these historical Key and Value vectors are directly obtained from the cache. At this time, only the Query vector of the current word needs to be recalculated, and then the cached Key and Value vectors and the new Query vector are used to calculate the attention representation of the current word. This avoids repeated calculation of the Key and Value vectors of historical words and related attention representations, thereby improving the model reasoning speed.

[0065] In this step, for each decoder layer in the dialogue system, the word-gram that needs to be edited in the reference prompt word can be determined according to the editing operation set, and then the unified attention map of the reference prompt word in the decoder layer is combined to analyze the impact of the word-gram to be edited on the attention score of the subsequent word-gram. In this way, the word-gram that is more affected can be determined as the target word-gram that needs to recalculate the KV cache, so as to accurately calculate the word-gram that needs to recalculate the KV cache in each decoder layer, so as to maximize the reuse rate of the KV cache while ensuring the reasoning accuracy of the dialogue system, thereby reducing the TTFT delay.

[0066] S104: When the target word-unit corresponding to the target cue word in each decoder layer is determined, a KV cache of the target word-unit corresponding to the target cue word in each decoder layer is calculated.

[0067] In this step, when it is determined that the target word corresponding to the target prompt word in each decoder layer needs to be recalculated, the Key and Value vectors of the target word corresponding to the target prompt word in each decoder layer are calculated and stored in a preset KV cache pool. When the calculation of the Key and Value vectors of the target word corresponding to the target prompt word in each decoder layer is completed, the KV cache of the target word corresponding to the target prompt word in each decoder layer is obtained.

[0068] S105: Reuse the KV cache corresponding to the reference prompt word to obtain the KV cache of other word units corresponding to the target prompt word in each decoder layer, and determine the output content of the target prompt word according to the KV cache of the target word unit and other word units corresponding to the target prompt word in each decoder layer.

[0069] Among them, other words refer to words other than the target word.

[0070] In this step, when the KV cache of the target word corresponding to the target prompt word in each decoder layer is determined, the word-grams of the target prompt word in each decoder layer except the target word-gram can be determined as other word-grams, thereby determining the other word-grams corresponding to the target prompt word in each decoder layer. This part of the word-grams does not need to recalculate the KV cache, and the KV cache matching the corresponding other word-grams in the reference prompt word can be directly reused. At this time, the KV cache of the other word-grams corresponding to the target prompt word in each decoder layer can be obtained. Finally, the KV cache of the target word-gram corresponding to the target prompt word in each decoder layer and the KV cache of the other word-grams corresponding to the target prompt word in each decoder layer are combined to determine the output content of the target prompt word in the dialogue system.

[0071] For example, suppose the target prompt word is "Is the dog on the mat" and the reference prompt word is "Is the cat on the mat". At this time, if the target prompt word in decoder layer A corresponds to dog, the (the second the), and mat, then the other word units corresponding to the target prompt word in decoder layer A are Is, the (the first the), and on.

[0072] The present application provides a method and device for a dialogue system interaction based on KV cache reuse, the method comprising: when receiving a target prompt word input by a user into the dialogue system, determining a corresponding reference prompt word, determining a set of edit operations corresponding to the minimum number of edits required to convert the reference prompt word into the target prompt word based on dynamic programming, and determining a unified attention map of the reference prompt word at each decoder layer, for each decoder layer, determining the target word element of the KV cache that needs to be recalculated for the target prompt word in the decoder layer according to the set of edit operations and the unified attention map of the reference prompt word at the decoder layer. The edit operations corresponding to the minimum number of edits are determined by the edit operation set, and then the target word element is determined in combination with the unified attention map, so that the target word element of the KV cache that needs to be recalculated for the target prompt word in each decoder layer can be accurately identified, and the KV cache of the reference prompt word can be reused for other word elements corresponding to the target prompt word in each decoder layer, so as to maximize the reuse rate of the KV cache, thereby further improving the response speed of the dialogue system and reducing the TTFT delay.

[0073] like Figure 2 As shown, in one embodiment, according to the edit operation set and the unified attention map of the reference prompt word at the decoder layer, determining the target word element of the KV cache to be recalculated at the decoder layer for the target prompt word includes:

[0074] S201: Determine the word-grams to be deleted and the word-grams to be inserted corresponding to the reference prompt word according to the editing operation set.

[0075] The editing operation set records the operation type and the word to be edited.

[0076] In this step, based on the editing operation set, the word elements that need to be deleted when the reference prompt word is converted into the target prompt word are determined, and the word elements that need to be inserted when the reference prompt word is converted into the target prompt word are determined. It can be understood that the word elements to be inserted corresponding to the reference prompt word can be queried in the target prompt word.

[0077] S202: Determine each word-element to be inserted as a target word-element corresponding to the target prompt word at the decoder layer.

[0078] Specifically, since the KV cache of the word to be inserted does not exist in the KV cache corresponding to the reference prompt word, each word to be inserted can be used as the target prompt word and the word in the KV cache needs to be recalculated in each decoder layer, that is, each word to be inserted is determined as the target word corresponding to the target prompt word at the decoder layer.

[0079] S203: For each word-gram to be deleted, the attention score of the subsequent word-grams of the word-gram to be deleted in the reference prompt word is evaluated according to the unified attention map of the reference prompt word at the decoder layer. If there are words with an attention score higher than a first preset threshold value in the subsequent word-grams, the words with an attention score higher than the first preset threshold value in the subsequent word-grams are determined as the target word-grams corresponding to the target prompt word at the decoder layer.

[0080] The attention score is used to measure the richness of the semantic information of the corresponding to-be-deleted word contained in the subsequent word.

[0081] In this step, for each word-gram to be deleted, the attention scores of the subsequent words of the word-gram to be deleted are evaluated according to the unified attention map of the reference prompt word at the decoder layer. In this way, the impact of deleting the word-gram to be deleted on its subsequent words can be quantified. For example, if there is a word-gram whose attention score exceeds the first preset threshold among the subsequent words, it means that the word-gram carries more semantic information of the word-gram to be deleted. If the subsequent word-gram is directly reused simply because it is consistent with the reference prompt word, it will affect the accuracy of the dialogue system's reasoning. Therefore, the word-gram with an attention score higher than the first preset threshold among the subsequent words will also be determined as the target word-gram corresponding to the target prompt word at the decoder layer.

[0082] In one example, if Figure 3 As shown, Figure 3 This is an example diagram of an application of the dialogue system interaction method based on KV cache reuse provided in an embodiment of the present application.

[0083] exist Figure 3In the example, the target cue word is "Is the dog on the mat", and the KV cache pool already contains the KV cache of the previous reference cue word "Is the cat on the mat". In the prior art, the sharing of KV cache is limited to independent segments whose prefixes of the cue words are completely matched or appear frequently. For example, prefix word units such as "Is the" can directly reuse the KV cache, but partially similar segments such as "on the mat" cannot be reused. In contrast, the method provided in the present application will perform attention score analysis on the subsequent word units (on, the, mat) of the deleted word unit cat, so in some decoder layers, the word units on, the, and mat can reuse the KV cache corresponding to the reference cue word. For example, for the target cue word "Is the dog on the mat", the word units that are the same as the reference cue word include "Is the" and "on the mat". In this case, only part of the KV cache of the newly added word units "dog" and "on the mat" can be recalculated, based on which the KV cache can be identified and reused as much as possible to avoid repeated calculations.

[0084] In this embodiment, for each decoder layer, the target word is determined by combining the editing operation set and the unified attention map corresponding to the reference prompt word. This can identify as many word units as possible in the KV cache that can reuse the reference prompt word, thereby improving the reuse rate of the KV cache and improving the response speed of the dialogue system.

[0085] In one embodiment, calculating the KV cache of the target word unit corresponding to the target prompt word in each decoder layer includes:

[0086] S1: In a current decoder layer, obtain a first weight set and a second weight set corresponding to the current decoder layer, and obtain an output vector of a previous decoder layer of the current decoder layer.

[0087] The first weight set includes the key weights of each attention head in the corresponding decoder layer, and the second weight set includes the value weights of each attention head in the corresponding decoder layer.

[0088] S2: Determine the key vector and value vector of the target word corresponding to the target prompt word in the current decoder layer according to the first weight set and the second weight set corresponding to the current decoder layer and the output vector of the previous decoder layer of the current decoder layer, and store the key vector and value vector in a preset KV cache pool to obtain the KV cache of the target word corresponding to the target prompt word in the current decoder layer.

[0089] Among them, the key vector refers to the Key vector, and the value vector refers to the Value vector.

[0090] In this step, the output vector of the previous decoder layer of the current decoder layer can be used as the input vector of the current decoder layer, and the vectors corresponding to the target word in the input vector are weighted summed with each weight in the first weight set to obtain the key vector of the target word corresponding to the target prompt word in the current decoder layer, and the vectors corresponding to the target word in the input vector are weighted summed with each weight in the second weight set to obtain the value vector of the target word corresponding to the target prompt word in the current decoder layer. Then, the key vector and the value vector are stored in a preset KV cache pool to obtain the KV cache of the target word corresponding to the target prompt word in the current decoder layer.

[0091] S3: Calculate the attention score based on the key vector and the value vector, and determine the output vector of the current decoder layer based on the attention score, and enter the next decoder layer until all decoder layers are processed in hierarchical order to obtain the KV cache of the target word corresponding to the target prompt word in each decoder layer.

[0092] It can be understood that the output vector of the current decoder layer can be used as the input vector of the next decoder layer. Specifically, the output content of the target word corresponding to the target prompt word in the current decoder layer can be determined based on the attention score, and then the output vector of the previous decoder layer of the current decoder layer is updated according to the output content to obtain the output vector of the current decoder layer.

[0093] In this embodiment, when the target word-units corresponding to the target prompt word at each decoder layer are determined, KV cache calculations may be performed only on these target word-units to save computing resources and avoid repeated calculations as much as possible.

[0094] In one embodiment, reusing the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-units corresponding to the target prompt word in each decoder layer, and determining the output content of the target prompt word according to the KV cache of the target word-unit and other word-units corresponding to the target prompt word in each decoder layer, including:

[0095] S1: For each decoder layer, determine the other word-units corresponding to the target prompt word in the decoder layer, and search and obtain the KV cache corresponding to the other word-units of the target prompt word in the decoder layer in the KV cache corresponding to the reference prompt word, so as to obtain the KV cache of other word-units corresponding to the target prompt word in the decoder layer.

[0096] S2: When determining the KV cache of other word units corresponding to the target prompt word in each decoder layer, determine the output content of the target prompt word based on the KV cache corresponding to other word units of the target prompt word in each decoder layer and the KV cache corresponding to the target word unit of the target prompt word in each decoder layer.

[0097] In this embodiment, when determining the KV cache of the target word corresponding to the target prompt word in each decoder layer, the KV cache of other word-units corresponding to the target prompt word in each decoder layer is determined by reusing the KV cache of the reference prompt word, and finally, the output content of the target prompt word in the dialogue system is determined according to the KV caches corresponding to other word-units of the target prompt word in each decoder layer and the KV caches corresponding to the target word-units of the target prompt word in each decoder layer, thereby improving the reuse rate of the KV cache in the dialogue system to improve the response speed of the dialogue system.

[0098] In one embodiment, determining a unified attention map of a reference cue word at each decoder layer includes:

[0099] S1: Determine the preset attention map cache pool.

[0100] S2: For each decoder layer, obtain the multi-head attention map of the reference prompt word at this decoder layer from the attention map cache pool.

[0101] Among them, the multi-head attention map includes the attention map of each attention head.

[0102] S3: Sum the multi-head attention maps of the reference cue word at the decoder layer to obtain a unified attention map of the reference cue word at each decoder layer.

[0103] In this embodiment, the multi-head attention maps of each decoder layer are summed to obtain a unified attention map of the reference prompt word at each decoder layer. Since the unified attention map can comprehensively reflect the attention information of each attention head in the multi-head attention map, this can simplify the subsequent analysis of the impact of the subsequent word-units to be deleted based on the attention score, thereby improving the reasoning efficiency of the dialogue system.

[0104] In one embodiment, determining a set of editing operations corresponding to the minimum number of editing times required to convert a reference prompt word into a target prompt word based on dynamic programming includes:

[0105] S1: Determine the state transition equation based on the goal of determining the minimum number of edits required to transform the reference cue word into the target cue word.

[0106] In one example, the state transition equation may be expressed as follows:

[0107]

[0108] In the formula, Indicates that the substring [0:i] Convert to substring [0:j] The minimum number of edits required, [0:i] represents the substring formed by the first i characters in the reference prompt word, [0:j] represents the substring formed by the first j characters in the target prompt word, deletion represents a deletion operation, insertion represents an insertion operation, and subtitution represents a replacement operation. [i]= [j], [i] The value of [j] is 0. [i] [j], [i] The value of [j] is 1.

[0109] Specifically, the update principles for delete, insert, and replace operations are as follows:

[0110] 1. Deletion operation: If you delete the i-th character of the reference prompt word and then convert the remaining first i-1 characters into the first j characters of the target prompt word, then the number of edits at this time is equal to the number of edits to convert the first i-1 characters of the reference prompt word into the first j characters of the target prompt word (that is, d(i-1, j)) plus 1, because one character has been deleted.

[0111] 2. Insertion operation: If a character is inserted based on the first i characters of the reference prompt word so that it can match the first j characters of the target prompt word, then the number of edits at this time is equal to the number of edits to convert the first i characters of the reference prompt word to the first j - 1 characters of the target prompt word (that is, d(i, j-1)) plus 1, because a character is inserted.

[0112] 3. Replacement operation: If the i-th character of the reference prompt word is replaced with the j-th character of the target prompt word, and then the first i-1 characters of the reference prompt word are converted to the first j-1 characters of the target prompt word, then the number of edits is equal to the number of edits for converting the first i-1 characters of the reference prompt word to the first j-1 characters of the target prompt word (i.e., d(i-1, j-1)). However, if the i-th character of the reference prompt word is the same as the j-th character of the target prompt word, then no replacement operation is required, and the number of edits is equal to d(i-1, j-1); if they are different, 1 needs to be added, indicating that a replacement operation has been performed.

[0113] S2: Construct a two-dimensional transfer table based on the state transfer equation, and perform state backtracking according to the two-dimensional transfer table.

[0114] In this step, a blank two-dimensional table can be constructed first, and the data calculated according to the state transfer equation can be filled into the two-dimensional table to achieve a two-dimensional transfer table. Specifically, the values ​​in the two-dimensional transfer table can be expressed as the minimum number of edits required to convert the substring formed by the first i characters of the reference prompt word into the substring formed by the first j characters of the target prompt word.

[0115] S3: Record the editing operations during the state backtracking process to obtain the editing operation set.

[0116] In this step, after the two-dimensional transfer table is determined, backtracking is performed from the lower right corner d(n, m) of the two-dimensional transfer table to d(0, 0), and specific editing operations and editing words are recorded during the backtracking process.

[0117] In one example, the state of the two-dimensional transfer table can be traced back according to the following rules:

[0118] (1) If d(i, j) = d(i-1, j)+1, it means that a deletion operation was performed in the previous step, that is, the i-th character of the reference prompt word was deleted, and the process is backtracked to d(i-1, j).

[0119] (2) If d(i, j) = d(i, j-1)+1, it means that an insertion operation was performed in the previous step, that is, a character was inserted to match the j-th character of the target prompt word. At this time, backtrack to d(i, j-1).

[0120] (3) If d(i, j) = d(i- 1, j - 1)+ ( [i] [j]), it means that the previous step took a replacement operation (when [i] [j]), or no action is taken (when [i] [j]), then backtrack to d(i- 1, j - 1).

[0121] In this embodiment, dynamic programming can be used to determine the set of editing operations corresponding to the minimum number of edits required to convert a reference prompt word into a target prompt word, and then determine the editing method that can minimize the recalculation of the KV cache. Then, the editing operations are recorded in this way to form an editing operation set to avoid unnecessary repeated calculations, thereby improving the response speed of the dialogue system and reducing TTFT delay.

[0122] In one embodiment, the dialog system interaction method based on KV cache reuse further includes:

[0123] S1: When determining the target word-meta corresponding to the target prompt word in each decoder layer, count the number of word-meta of the target word-meta corresponding to the target prompt word in each decoder layer.

[0124] S2: Mark the decoder layer whose word number does not exceed the second preset threshold as the target decoder layer, and skip the calculation of the attention score of the previous layer of each target decoder layer in the process of calculating the KV cache of the target word corresponding to the target prompt word in each decoder layer.

[0125] In this embodiment, the number of word units of the target word units corresponding to the target prompt word at each decoder layer is counted. For each decoder layer, if the number of word units does not exceed the second preset threshold, it means that skipping the attention score calculation of the previous layer of the decoder layer has little effect on the decoder layer. Therefore, in order to further improve the response speed of the dialogue system, the attention score calculation of the previous layer of the marked target decoder layer can be skipped, and the attention scores of the first two layers of the target decoder layer can be directly used. This can further improve the response speed of the dialogue system.

[0126] In one example, if Figure 4 As shown, Figure 4 This is an example diagram of the calculation acceleration optimization provided by the embodiment of the present application. Figure 4 In the figure, the dotted box represents the target decoder layer. After optimization, the layer above the target decoder layer represented by the dotted box will skip the attention score and output calculation. Figure 4 It can be seen that after optimization, part of the computing time can be saved compared to before optimization, so as to further improve the response speed of the dialogue system.

[0127] In one example, if Figure 5 As shown, Figure 5 This is an application flow chart of the interaction of the dialogue system based on KV cache reuse provided in the embodiment of the present application. Figure 5 In , the previous conversation prompt word represents the reference prompt word, and the new query prompt word represents the target prompt word. The new KV cache refers to the KV cache of the target word corresponding to the calculated target prompt word at each decoder layer, and the KV cache and attention map corresponding to the target prompt word determined during the calculation process will be stored in the KV cache pool and the attention map cache pool respectively.

[0128] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0129] The following describes a dialogue system interaction device based on KV cache reuse provided in an embodiment of the present application. The dialogue system interaction device based on KV cache reuse described below and the dialogue system interaction method based on KV cache reuse described above can be referenced to each other.

[0130] like Figure 6 As shown, the present application provides a dialogue system interaction device 300 based on KV cache reuse, the device comprising:

[0131] The prompt word determination module 301 is used to determine a reference prompt word when receiving a target prompt word input by a user;

[0132] An editing operation determination module 302 is used to determine, based on dynamic programming, a set of editing operations corresponding to the minimum number of editing times required to convert a reference prompt word into a target prompt word, and determine a unified attention map of the reference prompt word at each decoder layer;

[0133] A word unit determination module 303 is used to determine, for each decoder layer, a target word unit of the KV cache to be recalculated for the target word in the decoder layer according to the edit operation set and the unified attention map of the reference word in the decoder layer;

[0134] A cache calculation module 304, configured to calculate a KV cache of the target word element corresponding to the target word element in each decoder layer when the target word element corresponding to the target word element in each decoder layer is determined;

[0135] The cache reuse module 305 is used to reuse the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-grams corresponding to the target prompt word in each decoder layer, and determine the output content of the target prompt word according to the KV cache of the target word-gram and other word-grams corresponding to the target prompt word in each decoder layer, where other word-grams refer to word-grams other than the target word-gram.

[0136] In one embodiment, the word unit determination module includes:

[0137] An editing word-unit determination submodule is used to determine the word-unit to be deleted and the word-unit to be inserted corresponding to the reference prompt word according to the editing operation set;

[0138] The first word unit determination submodule is used to determine each word unit to be inserted as a target word unit corresponding to the target prompt word at the decoder layer;

[0139] The second word-unit determination submodule is used to evaluate, for each word-unit to be deleted, the attention score of the subsequent word-units of the word-unit to be deleted in the reference prompt word according to the unified attention map of the reference prompt word at the decoder layer; if there is a word-unit with an attention score higher than a first preset threshold value among the subsequent word-units, the word-unit with an attention score higher than the first preset threshold value among the subsequent word-units is determined as the target word-unit corresponding to the target prompt word at the decoder layer.

[0140] In one embodiment, the cache computing module includes:

[0141] A weight acquisition submodule, used to acquire, in a current decoder layer, a first weight set and a second weight set corresponding to the current decoder layer, and to acquire an output vector of a previous decoder layer of the current decoder layer;

[0142] A cache calculation submodule, for determining a key vector and a value vector corresponding to a target word unit of a target prompt word in a current decoder layer according to a first weight set and a second weight set corresponding to a current decoder layer and an output vector of a previous decoder layer of the current decoder layer, and storing the key vector and the value vector in a preset KV cache pool to obtain a KV cache of the target word unit corresponding to the target prompt word in the current decoder layer;

[0143] The iterative processing submodule is used to calculate the attention score based on the key vector and the value vector, and determine the output vector of the current decoder layer based on the attention score, and enter the next decoder layer until all decoder layers are processed in hierarchical order to obtain the KV cache of the target word corresponding to the target prompt word in each decoder layer.

[0144] In one embodiment, the cache reuse module includes:

[0145] A cache determination submodule is used to determine, for each decoder layer, other word units corresponding to the target prompt word in the decoder layer, and search and obtain the KV cache corresponding to other word units of the target prompt word in the decoder layer in the KV cache corresponding to the reference prompt word, so as to obtain the KV cache of other word units corresponding to the target prompt word in the decoder layer;

[0146] The cache reuse submodule is used to determine the output content of the target prompt word based on the KV cache corresponding to other word units of the target prompt word in each decoder layer and the KV cache corresponding to the target word unit of the target prompt word in each decoder layer when determining the KV cache of other word units corresponding to the target prompt word in each decoder layer.

[0147] In one embodiment, the editing operation determination module includes:

[0148] A cache pool determination submodule is used to determine a preset attention map cache pool;

[0149] A graph acquisition submodule is used to obtain the multi-head attention graph of the reference prompt word at each decoder layer from the attention graph cache pool;

[0150] The graph processing submodule is used to sum the multi-head attention graphs of the reference prompt word at the decoder layer to obtain a unified attention graph of the reference prompt word at each decoder layer.

[0151] In one embodiment, the editing operation determination module includes:

[0152] An equation determination submodule, for determining a state transition equation for the purpose of determining a minimum number of edits required to transform a reference cue word into a target cue word;

[0153] A state backtracking submodule is used to construct a two-dimensional transfer table based on the state transfer equation and perform state backtracking according to the two-dimensional transfer table;

[0154] The operation record submodule is used to record the editing operation during the state backtracking process to obtain the editing operation set.

[0155] In one embodiment, the dialog system interaction device based on KV cache reuse further includes:

[0156] A quantity counting module, used for counting the number of word units of the target word units corresponding to the target prompt word in each decoder layer when determining the target word units corresponding to the target prompt word in each decoder layer;

[0157] A calculation skip module is used to mark the decoder layer whose word number does not exceed the second preset threshold as the target decoder layer, and skip the calculation of the attention score of the previous layer of each target decoder layer in the process of calculating the KV cache of the target word corresponding to the target prompt word in each decoder layer.

[0158] The division of each module in the above-mentioned dialogue system interaction device based on KV cache reuse is only for illustration. In other embodiments, the dialogue system interaction device based on KV cache reuse can be divided into different modules as needed to complete all or part of the functions of the above-mentioned dialogue system interaction device based on KV cache reuse. Each module in the above-mentioned dialogue system interaction device based on KV cache reuse can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0159] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the dialogue system interaction method based on KV cache reuse as described in any of the above embodiments.

[0160] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the dialogue system interaction method based on KV cache reuse as described in any of the above embodiments.

[0161] Indicatively, Figure 7 As shown, Figure 7 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 400 may be provided as a server. Figure 7 The computer device 400 includes a processing component 402, which further includes one or more processors, and a memory resource represented by a memory 401, for storing instructions executable by the processing component 402, such as an application. The application stored in the memory 401 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 402 is configured to execute instructions to perform the dialogue system interaction method based on KV cache reuse of any of the above embodiments.

[0162] The computer device 400 may further include a power supply component 403 configured to perform power management of the computer device 400, a wired or wireless network interface 404 configured to connect the computer device 400 to a network, and an input / output (I / O) interface 405. The computer device 400 may operate based on an operating system stored in the memory 401, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0163] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish an entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only include those elements, but also include other elements that are not clearly listed, or also include elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, the singular "one", "one" and "described / the" may also include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "include / comprise" or "have" etc. specify the existence of stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the existence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0165] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0166] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dialogue system interaction method based on KV cache reuse, characterized in that: The method comprises: When receiving the target prompt word input by the user, determining the reference prompt word; Determine a set of edit operations corresponding to the minimum number of edits required to convert the reference cue word into the target cue word based on dynamic programming, and determine a unified attention map of the reference cue word at each decoder layer; For each decoder layer, determining a target word-element of the KV cache to be recalculated for the target word in the decoder layer according to the set of edit operations and a unified attention map of the reference word in the decoder layer; When the target word element corresponding to the target prompt word in each decoder layer is determined, a KV cache of the target word element corresponding to the target prompt word in each decoder layer is calculated; The KV cache corresponding to the reference prompt word is reused to obtain the KV cache of other word-grams corresponding to the target prompt word in each decoder layer, and the output content of the target prompt word is determined according to the KV cache of the target word-gram and other word-grams corresponding to the target prompt word in each decoder layer, where the other word-grams refer to word-grams other than the target word-gram.

2. The method for interactive dialogue system based on KV cache reuse according to claim 1, characterized in that: Determining the target word element of the KV cache to be recalculated for the target word at the decoder layer according to the edit operation set and the unified attention map of the reference word at the decoder layer includes: Determine the word-gram to be deleted and the word-gram to be inserted corresponding to the reference prompt word according to the editing operation set; Determine each word to be inserted as the target word corresponding to the target prompt word at the decoder layer; For each word-gram to be deleted, the attention score of the subsequent word-grams of the word-gram to be deleted in the reference prompt word is evaluated according to the unified attention map of the reference prompt word at the decoder layer. If there are word-grams with attention scores higher than a first preset threshold among the subsequent word-grams, the word-grams with attention scores higher than the first preset threshold among the subsequent word-grams are determined as the target word-grams corresponding to the target prompt word at the decoder layer.

3. The method for interactive dialogue system based on KV cache reuse according to claim 1, characterized in that: The calculating of the KV cache of the target word element corresponding to the target prompt word in each decoder layer includes: In the current decoder layer, obtaining a first weight set and a second weight set corresponding to the current decoder layer, and obtaining an output vector of a previous decoder layer of the current decoder layer; Determine a key vector and a value vector of a target word element corresponding to the target prompt word in the current decoder layer according to a first weight set and a second weight set corresponding to the current decoder layer and an output vector of a previous decoder layer of the current decoder layer, and store the key vector and the value vector in a preset KV cache pool to obtain a KV cache of the target word element corresponding to the target prompt word in the current decoder layer; An attention score is calculated according to the key vector and the value vector, and an output vector of the current decoder layer is determined according to the attention score to enter the next decoder layer until all decoder layers are processed in hierarchical order to obtain a KV cache of the target word element corresponding to the target prompt word in each decoder layer.

4. The method for interactive dialogue system based on KV cache reuse according to claim 1, characterized in that: The reusing the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-units corresponding to the target prompt word in each decoder layer, and determining the output content of the target prompt word according to the KV cache of the target word-units and other word-units corresponding to the target prompt word in each decoder layer, includes: For each decoder layer, determine other word-units corresponding to the target prompt word in the decoder layer, and search and obtain the KV cache corresponding to other word-units of the target prompt word in the decoder layer in the KV cache corresponding to the reference prompt word, so as to obtain the KV cache of other word-units corresponding to the target prompt word in the decoder layer; When determining the KV cache of other word units corresponding to the target prompt word in each decoder layer, the output content of the target prompt word is determined based on the KV cache corresponding to other word units of the target prompt word in each decoder layer and the KV cache corresponding to the target word unit of the target prompt word in each decoder layer.

5. The method for interactive dialogue system based on KV cache reuse according to claim 1, characterized in that: Determining a unified attention map of the reference cue word at each decoder layer includes: Determine the preset attention map cache pool; For each decoder layer, obtaining a multi-head attention map of the reference prompt word at the decoder layer from the attention map buffer pool; The multi-head attention maps of the reference prompt word at the decoder layer are summed to obtain a unified attention map of the reference prompt word at each decoder layer.

6. The method for interactive dialogue system based on KV cache reuse according to claim 1, characterized in that: The set of editing operations corresponding to the minimum number of editing times required to convert the reference prompt word into the target prompt word determined based on dynamic programming includes: Determining a state transition equation for the purpose of determining the minimum number of edits required to convert the reference prompt word into the target prompt word; Constructing a two-dimensional transfer table based on the state transfer equation, and performing state backtracking according to the two-dimensional transfer table; The editing operations are recorded during the state backtracking process to obtain an editing operation set.

7. The method for interactive dialogue system based on KV cache reuse according to any one of claims 1 to 6, characterized in that: The method further comprises: When determining the target word-meta corresponding to the target prompt word in each decoder layer, counting the number of word-meta of the target word-meta corresponding to the target prompt word in each decoder layer; The decoder layer whose word number does not exceed the second preset threshold is marked as the target decoder layer, and in the process of calculating the KV cache of the target word corresponding to the target prompt word in each decoder layer, the calculation of the attention score of the previous layer of each target decoder layer is skipped.

8. A dialogue system interaction device based on KV cache reuse, characterized in that: The device comprises: A prompt word determination module, used to determine a reference prompt word when receiving a target prompt word input by a user; An editing operation determination module, configured to determine, based on dynamic programming, a set of editing operations corresponding to a minimum number of editing times required to convert the reference prompt word into the target prompt word, and determine a unified attention map of the reference prompt word at each decoder layer; A word unit determination module is used to determine, for each decoder layer, a target word unit of the KV cache to be recalculated for the target word in the decoder layer according to the set of editing operations and the unified attention map of the reference word in the decoder layer; A cache calculation module, used for calculating the KV cache of the target word corresponding to the target prompt word in each decoder layer when the target word corresponding to the target prompt word in each decoder layer is determined; A cache reuse module is used to reuse the KV cache corresponding to the reference prompt word to obtain the KV cache of other word-units corresponding to the target prompt word in each decoder layer, and determine the output content of the target prompt word according to the KV cache of the target word-unit and other word-units corresponding to the target prompt word in each decoder layer, wherein the other word-unit refers to the word-unit other than the target word-unit.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the dialogue system interaction method based on KV cache reuse as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the dialogue system interaction method based on KV cache reuse as described in any one of claims 1 to 7 are performed.

Images