Question and answer reasoning method and device based on key-value cache compression, equipment and medium

Abstract

The application discloses a question and answer reasoning method and device based on key-value cache compression, equipment and medium, relates to the technical field of natural language processing, including: dividing the prompt word in the current question and answer task into a word sequence and generating an initial key-value cache of the word sequence; dividing the word sequence into corresponding context words and end words of each layer based on a preset end window size of each attention layer of a target large language model; screening key words from the context words according to the importance score between the key matrix of the context words and the query matrix average of the end words; removing the key-value pairs of the words in the initial key-value cache except the key words to obtain a compressed key-value cache; and generating a reasoning result corresponding to the compressed key-value cache by using the target large language model. The application reduces the high computing power consumption of the large language model caused by the growth of the key-value cache data and breaks through the dependence of the existing key-value cache compression method on the complete attention weight matrix.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a question-answering reasoning method, apparatus, device, and medium based on key-value caching compression. Background Technology

[0002] Large Language Models (LLMs) are crucial in the field of natural language processing and are widely used in various question-answering inference tasks such as chatbots, content generation, code completion, and document analysis. When performing question-answering inference tasks, the model generates text through an autoregressive approach. To accelerate this process, it computes and stores intermediate state representations of preceding tokens—a key-value cache—to avoid redundant computation.

[0003] However, existing key-value caching methods have significant drawbacks. On the one hand, with the widespread application of large language models, their computational demands are extremely high, and the key-value cache data continues to grow, severely impacting the inference engine's ability to meet service level objectives (SLAs) with limited computing resources, such as the initial processing time of prompt words (Time To First Token, TTFT) and the sequential generation time of new lexical units (Inter-Token Latency, ITL). On the other hand, existing key-value cache compression methods typically rely on a complete attention weight matrix, which is difficult to deploy in typical service engines equipped with highly optimized attention implementations (such as FlashAttention).

[0004] In summary, how to reduce the high computational cost of large language models due to the growth of key-value cache data and overcome the dependence of existing key-value cache compression methods on the complete attention weight matrix are problems to be solved in this field. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a question-answering reasoning method, apparatus, device, and medium based on key-value cache compression, reducing the high computational power consumption caused by the growth of key-value cache data in large language models and overcoming the dependence of existing key-value cache compression methods on the complete attention weight matrix. The specific solution is as follows:

[0006] Firstly, this application discloses a question-answering reasoning method based on key-value caching compression, including:

[0007] The target large language model is used to segment the prompt words in the current question answering task into a word sequence, and an initial key-value cache of the word sequence is generated; wherein, the target large language model includes a neural network architecture with an attention mechanism;

[0008] Based on the preset end window size of each attention layer of the target large language model, the word sequence is divided into context words and end words corresponding to each layer;

[0009] Keyword units for each attention layer are selected from the context units based on the importance score between the key matrix of the context units and the mean of the query matrix of the terminal units; wherein, the importance score is used to characterize the contribution of the context units to the generation of the terminal units;

[0010] Remove key-value pairs of words other than the keyword word from the initial key-value cache to obtain the compressed key-value cache;

[0011] The target large language model is used to generate inference results corresponding to the compressed key-value cache.

[0012] Optionally, the initial key-value cache for generating the lexical sequence includes:

[0013] Each word in the word sequence is sequentially identified as the current word;

[0014] The key matrix and value matrix of the current word are obtained using the hidden state and projection matrix of the current word; wherein the hidden state is determined based on the length of the word sequence and the dimension of the hidden layer.

[0015] Add the key matrix and value matrix of the current word to the current key-value cache to obtain the new current key-value cache;

[0016] When the key matrix and value matrix of all words in the word sequence are added to the current key-value cache, the new current key-value cache is determined as the initial key-value cache of the word sequence.

[0017] Optionally, the step of filtering keyword units for each attention layer from the context units based on the importance score between the key matrix of the context units and the mean of the query matrix of the terminal units includes:

[0018] The mean of the query matrix of the terminal word is taken along the sequence dimension to obtain the mean of the query matrix of the terminal word, and the dot product of the key matrix of the context word and the mean of the query matrix of the terminal word is taken to obtain the importance score.

[0019] Keyword units for each attention layer are selected from the contextual units based on the importance score.

[0020] Optionally, the step of filtering keyword units for each attention layer from the context lexical units based on the importance score includes:

[0021] The importance scores of the current attention layer are normalized to obtain a probability distribution sequence arranged in descending order;

[0022] The target probability distribution is determined from the probability distribution sequence based on a cumulative score threshold mechanism;

[0023] Select the keyword tokens of the current attention layer that correspond to the target probability distribution from the context tokens.

[0024] Optionally, determining the target probability distribution from the probability distribution sequence based on the cumulative score threshold mechanism includes:

[0025] The current probability distribution is determined from the probability distribution sequence, and the cumulative distribution sum from the first probability distribution in the probability distribution sequence to the current probability distribution is determined;

[0026] Determine whether the cumulative distribution sum is not less than a preset threshold;

[0027] If the cumulative sum of the distributions is not less than a preset threshold, then the probability distributions from the first end to the current probability distribution are all determined as the target probability distribution.

[0028] If the cumulative sum of the distributions is less than a preset threshold, the next probability distribution is determined from the probability distribution sequence in order from the first end to the last end, and the next probability distribution is updated to the current probability distribution. Then, the process jumps back to the step of determining the cumulative sum of the probability distributions from the first probability distribution to the current probability distribution in the probability distribution sequence.

[0029] Optionally, the formula for obtaining the importance score is:

[0030] ;

[0031] in, The key matrix, For querying the matrix, For terminal words, For contextual words, T represents the matrix transpose along the sequence dimension.

[0032] Optionally, the step of dividing the lexical sequence into context lexical units and terminal lexical units corresponding to each layer based on the preset end window size of each attention layer of the target large language model includes:

[0033] The preset end window size is determined by multiplying the preset window division ratio of the current attention layer in the target large language model with the length of the word sequence;

[0034] The lexical units in the lexical sequence that correspond to the preset end window size are classified as the end lexical units of the current attention layer, and the lexical units in the lexical sequence other than the end lexical units are classified as the context lexical units of the current attention layer.

[0035] Secondly, this application discloses a question-answering inference device based on key-value compression caching, comprising:

[0036] An initial key-value pair generation module is used to segment the prompt words in the current question-answering task into word sequence using a target large language model, and generate an initial key-value cache of the word sequence; wherein, the target large language model includes a neural network architecture with an attention mechanism;

[0037] The lexical segmentation module is used to divide the lexical sequence into context lexical units and terminal lexical units corresponding to each layer based on the preset end window size of each attention layer of the target large language model;

[0038] A lexical filtering module is used to filter key lexical units for each attention layer from the context lexical units based on the importance score between the key matrix of the context lexical units and the mean of the query matrix of the terminal lexical units; wherein, the importance score is used to characterize the contribution of the context lexical units to the generation of the terminal lexical units;

[0039] The key-value pair compression module is used to remove key-value pairs of words other than the keyword words in the initial key-value cache to obtain a compressed key-value cache.

[0040] The reasoning result generation module is used to generate reasoning results corresponding to the compressed key-value cache using the target large language model.

[0041] Thirdly, this application discloses an electronic device, including:

[0042] Memory, used to store computer programs;

[0043] A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed question-answering reasoning method based on key-value cache compression.

[0044] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed question-answering reasoning method based on key-value cache compression.

[0045] The beneficial effects of this application are as follows: This application utilizes a target large language model to segment the prompt words in the current question-answering task into a lexical sequence and generates an initial key-value cache of the lexical sequence; wherein, the target large language model includes a neural network architecture with an attention mechanism; based on the preset end window size of each attention layer of the target large language model, the lexical sequence is divided into context lexical units and end lexical units corresponding to each layer; based on the importance score between the key matrix of the context lexical units and the mean of the query matrix of the end lexical units, key lexical units of each attention layer are selected from the context lexical units; wherein, the importance score is used to characterize the contribution of the context lexical units to the generation of the end lexical units; key-value pairs of lexical units other than the key lexical units in the initial key-value cache are removed to obtain a compressed key-value cache; and the target large language model is used to generate inference results corresponding to the compressed key-value cache. Therefore, this application removes key-value pairs of lexical units other than key words from the initial key-value cache, compresses the key-value cache, reduces computational overhead, and lowers the first word generation time and inter-word latency. This improves throughput without affecting the robustness of the service system and utilizes computational resources more efficiently, thus reducing the high computational cost of large language models due to the growth of key-value cache data. Furthermore, the lexical importance scoring mechanism used in this application for key-value compression does not require access to the complete attention matrix. It determines the importance score between the key matrix of context lexical units and the mean of the query matrix of terminal lexical units, i.e., the contribution of context lexical units to the generation of terminal lexical units. Then, based on the importance score, key words for each attention layer are selected from the context lexical units. This eliminates the need to explicitly generate the attention matrix, effectively reducing complexity. It achieves approximate global attention distribution through mean query, retains key semantic information, and breaks through the dependence of existing key-value cache compression methods on the complete attention weight matrix. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 This is a flowchart of a question-answering reasoning method based on key-value compression caching disclosed in this application;

[0048] Figure 2 This is a specific lexical scoring diagram disclosed in this application;

[0049] Figure 3 This is a schematic diagram of a question-answering reasoning device based on key-value compression caching disclosed in this application;

[0050] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0052] Large language models are crucial in the field of natural language processing and are widely used in various question-answering reasoning tasks such as chatbots, content generation, code completion, and document analysis. When performing question-answering reasoning tasks, the model generates text through an autoregressive approach. To accelerate this process, it calculates and saves the intermediate state representations of the preceding tokens, i.e., a key-value cache (KVCache), to avoid redundant calculations.

[0053] However, existing key-value caching has significant drawbacks. On the one hand, with the widespread application of large language models, their computational consumption is extremely high, and the key-value cache data continues to grow, severely impacting the inference engine's ability to meet service level objectives under limited computing resources, such as the initial processing time of prompt words and the sequential generation time of new lexical units. On the other hand, existing key-value cache compression methods typically rely on a complete attention weight matrix, which is difficult to deploy in typical service engines equipped with highly optimized attention implementations.

[0054] To address this, this application provides a question-answering reasoning scheme based on key-value compression caching, which reduces the high computational cost of large language models due to the growth of key-value cache data and breaks through the dependence of existing key-value caching compression methods on the complete attention weight matrix.

[0055] See Figure 1 As shown in the figure, this application discloses a question-answering reasoning method based on key-value caching compression, including:

[0056] Step S11: Use the target large language model to segment the prompt words in the current question answering task into word sequence and generate an initial key-value cache of the word sequence; wherein, the target large language model includes a neural network architecture with an attention mechanism.

[0057] In the pre-filling stage, the prompt words in the current question-answering task are segmented into a sequence of tokens using the target large language model. A token is the smallest unit into which text is divided in natural language processing; it can be a word, subword, or character. For example, if the prompt word in the current question-answering task is "What to eat tonight?", the target large language model's tokenizer will segment this prompt word, resulting in a specific sequence of tokens: ["tonight", "tonight", "eat", "what", "things"]. Different tokenizers may segment tokens in different ways; for instance, "things" might be segmented into "east" and "west". The target large language model includes a neural network architecture with an attention mechanism; that is, the target large language model is a Transformer model.

[0058] In this embodiment, generating the initial key-value cache of the lexical sequence includes: determining each lexical in the lexical sequence as the current lexical; obtaining the key matrix and value matrix of the current lexical using the hidden state and projection matrix of the current lexical; wherein the hidden state is determined based on the length of the lexical sequence and the hidden layer dimension; adding the key matrix and value matrix of the current lexical to the current key-value cache to obtain a new current key-value cache; when the key matrices and value matrices of all lexicals in the lexical sequence are added to the current key-value cache, the new current key-value cache is determined as the initial key-value cache of the lexical sequence.

[0059] Hidden state The hidden layer dimension is determined based on the length of the lexical sequence and the hidden layer dimension, where n represents the length of the lexical sequence and d represents the hidden layer dimension. The hidden state and projection matrix of each lexical in the sequence can project each lexical into three different representations, thus obtaining the query cache (Q), key cache (K), and value cache (V) of the lexical sequence, where Q = HW. q K=HW k V=HW u ,and These are the projection matrices corresponding to the query matrix, the key matrix, and the value matrix, respectively; they are learnable parameter matrices. It is the dimension of the key space, meaning that during generation step t, the model only needs to calculate the key-value projection for the new token, i.e., k. t =h t W k u t =h t W u The key matrix k of word t t Sum matrix u t Add it to the current key-value cache to get the new current key-value cache, i.e., K=[K, k t ], V=[V, ut Then, in step t+1, a key matrix k is generated for word t+1. t+1 Sum matrix u t+1 Then the key matrix k of word t+1 t+1 Sum matrix u t+1 Add it to the current key-value cache to get the new current key-value cache, i.e., K=[K, k t k t+1 ], V=[V, u t u t+1 In this way, we obtain the key matrix and value matrix of all words in the word sequence. When the key matrix and value matrix of all words in the word sequence are added to the current key-value cache, the new current key-value cache is determined as the initial key-value cache of the word sequence.

[0060] Step S12: Based on the preset end window size of each attention layer of the target large language model, divide the word sequence into context words and end words corresponding to each layer.

[0061] In this embodiment, dividing the lexical sequence into context lexical units and terminal lexical units corresponding to each layer based on the preset terminal window size of each attention layer of the target large language model includes: determining the preset terminal window size by multiplying the preset window division ratio of the current attention layer in the target large language model by the length of the lexical sequence; classifying the lexical units in the lexical sequence that correspond to the preset terminal window size as terminal lexical units of the current attention layer, and classifying the lexical units in the lexical sequence other than the terminal lexical units as context lexical units of the current attention layer.

[0062] The lexical sequence includes lexical units arranged in the order of their generation. The generation time of lexical units near the end of the sequence is updated, while the generation time of lexical units near the beginning of the sequence is earlier. Traditional techniques retain the key-value cache of all lexical units or only retain the most recently generated lexical units. However, this embodiment rationally selects the more critical lexical units and saves the key-value cache of these lexical units, that is, it selects them according to the importance of different lexical units.

[0063] For example Figure 2 The diagram shown illustrates a specific word segment scoring method. Input... (KeyMatrix) (Query Matrix) (i.e., preset window division ratio), output importance score s (score vector), thereby dividing the token sequence into context tokens and recent tokens. It is understandable that the preset recent window size of different attention layers may be the same or different, so the context tokens and recent tokens corresponding to each layer may be the same or different.

[0064] Preset end window size of the current attention layer in the target large language model The preset window division ratio r of the current attention layer is the product of the length n of the word sequence. For example, if the word sequence is ["today", "evening", "eat", "what", "thing"], the length of the word sequence is 5, and the preset window division ratio is 40%, then the preset end window size is 2. Therefore, the two words at the end of the word sequence are the end words, that is, "what" and "thing". The remaining words in the word sequence are the context words of the current attention layer, that is, "today", "evening", and "eat" are the context words of the current attention layer.

[0065] Step S13: Select key words for each attention layer from the context words based on the importance score between the key matrix of the context words and the mean of the query matrix of the terminal words; wherein, the importance score is used to characterize the contribution of the context words to the generation of the terminal words.

[0066] In this embodiment, the step of selecting key words for each attention layer from the context words based on the importance score between the key matrix of the context words and the mean of the query matrix of the terminal words includes: taking the mean of the query matrix of the terminal words along the sequence dimension to obtain the mean of the query matrix of the terminal words, and performing a dot product between the key matrix of the context words and the mean of the query matrix of the terminal words to obtain an importance score; and selecting key words for each attention layer from the context words based on the importance score.

[0067] Keyword units are selected from context units based on their contribution to the generated terminal units. The contribution of context units to the generated terminal units is characterized by an importance score between the key matrix of context units and the mean of the query matrix of terminal units. For example... Figure 2 As shown, the focus is on the interaction between context tokens and terminal tokens, specifically the contribution of the context token to recently generated tokens. This interaction can be effectively captured through the attention weights between the context token and recent tokens. Figure 2 Highlighted in red. The key matrix of context words is... Query matrix for terminal words The mean of the query matrix for the terminal words is obtained by taking the mean along the sequence dimension. The importance score is then obtained by taking the dot product of the key matrix of the context words and the mean of the query matrix for the terminal words. and will The value is assigned to s, and the final output is the importance score, which is used to filter the keyword units for each attention layer from the context units based on the importance score.

[0068] In this embodiment, the formula for obtaining the importance score is:

[0069] ;

[0070] in, The key matrix, For querying the matrix, For terminal words, For contextual words, T represents the matrix transpose along the sequence dimension.

[0071] The formula for obtaining the importance score can maintain the effectiveness of attention-based scoring, and by taking the mean of the query matrix of the terminal word and then multiplying it with the key matrix of the context word, the complexity can be reduced from O(wnd) to O(nd), which is compatible with optimization engines such as FlashAttention.

[0072] In this embodiment, the step of selecting keyword elements for each attention layer from the context words based on the importance score includes: normalizing the importance score of the current attention layer to obtain a probability distribution sequence arranged in descending order; determining a target probability distribution from the probability distribution sequence based on a cumulative score threshold mechanism; and selecting keyword elements for the current attention layer that correspond to the target probability distribution from the context words.

[0073] Furthermore, after determining the importance score, it is necessary to select key words for each attention layer from the context words based on the importance score, and make decisions using an adaptive probability distribution. The distribution of attention patterns may vary greatly between different layers and / or requests. In some cases, a few words carry most of the context importance (e.g., key entities or statements that set the context). Taking the current attention layer as an example, firstly, the importance score of the current attention layer is normalized using the softmax function to obtain a descending probability distribution sequence. That is, the softmax operation normalizes the importance score of each context word as a probability distribution on the word, and then sorts it in descending order to obtain a descending probability distribution sequence. Next, the target probability distribution is determined from the probability distribution sequence based on a cumulative score threshold mechanism, which formalizes this adaptive allocation. Finally, key words for the current attention layer that correspond to the target probability distribution are selected from the context words.

[0074] In this embodiment, determining the target probability distribution from the probability distribution sequence based on the cumulative score threshold mechanism includes: determining the current probability distribution from the probability distribution sequence, and determining the cumulative sum of the probability distributions from the first probability distribution to the current probability distribution in the probability distribution sequence; determining whether the cumulative sum of the probability distributions is not less than a preset threshold; if the cumulative sum of the probability distributions is not less than the preset threshold, then determining all probability distributions from the first probability distribution to the current probability distribution as the target probability distribution; if the cumulative sum of the probability distributions is less than the preset threshold, then determining the next probability distribution from the probability distribution sequence based on the order from the first to the last probability distribution, updating the next probability distribution to the current probability distribution, and jumping back to the step of determining the cumulative sum of the probability distributions from the first probability distribution to the current probability distribution in the probability distribution sequence.

[0075] Taking the current attention layer as an example, the input importance score s and the preset threshold are... Next, the target probability distribution is determined from the probability distribution sequence based on the cumulative score threshold mechanism. Taking the descending probability distribution sequence [0.48, 0.24, 0.16, 0.12] and the preset threshold of 0.7 as an example, the first determined current probability distribution is 0.48. Since it is the first probability distribution, the cumulative sum of the probability distributions from the first probability distribution to the current probability distribution in the probability distribution sequence is still 0.48. The cumulative sum of 0.48 is less than the preset threshold of 0.7, so the next probability distribution of 0.24 is determined from the probability distribution sequence based on the order from the first to the last, and the next probability distribution is updated to the current probability distribution. Then, the cumulative sum of the probability distributions from the first probability distribution to the current probability distribution is calculated as 0.48 + 0.24 = 0.72. Since the cumulative sum is not less than the preset threshold, the probability distributions from the first probability distribution to the current probability distribution are determined as the target probability distributions, that is, 0.48 and 0.24 are both determined as target probability distributions.

[0076] The following example, using the importance score of the current attention layer as s=[0.8,1.2,0.5,0.3], illustrates the process of determining keyword meta-elements. First, the initial index sequence of each context meta-element A, B, C, and D in the meta-element sequence is recorded as [0, 1, 2, 3]. Secondly, the importance scores s are normalized to obtain [0.28, 0.44, 0.15, 0.13], that is, the normalized importance score s of context word A is 0.28, the normalized importance score s of context word B is 0.44, the normalized importance score s of context word C is 0.15, and the normalized importance score s of context word D is 0.13. Then, they are sorted in descending order to obtain the probability distribution sequence [0.44, 0.28, 0.15, 0.13]. In order to accurately find the corresponding word after the target probability distribution is determined, the initial index sequence is adjusted according to the descending probability distribution sequence to obtain the target index sequence [1, 0, 2, 3]. Next, the preset threshold is set to 0.8, and the initially determined current probability distribution is 0.44. Since this is the first probability distribution, the cumulative sum of the probability distributions from the first probability distribution to the current probability distribution is still 0.44. This cumulative sum of 0.48 is less than the preset threshold of 0.7. Therefore, based on the order from the first to the last probability distribution, the next probability distribution (0.28) is determined from the probability distribution sequence, and this next probability distribution is updated to the new current probability distribution. Then, the cumulative sum of the probability distributions from the first probability distribution to the current probability distribution is calculated: 0.44 + 0.28 = 0. If the cumulative sum is less than the preset threshold, a new probability distribution of 0.15 is determined from the probability distribution sequence based on the order from the first to the last. This next probability distribution is then updated to the new current probability distribution. The cumulative sum from the first probability distribution to the current probability distribution is calculated as 0.44 + 0.28 + 0.15 = 0.87. Since this cumulative sum is not less than the preset threshold, all probability distributions from the first to the current probability distribution are determined as target probability distributions, i.e., 0.44, 0.28, and 0.15 are all determined as target probability distributions. After determining the target probability distributions, keyword elements are determined based on the target index sequence. That is, the context words corresponding to the target probability distributions 0.44, 0.28, and 0.15 are context words B, A, and C, respectively, meaning that context words B, A, and C are keyword elements.

[0077] This preset threshold allows for fine-tuning the trade-off between computational efficiency and context preservation in the key-value cache. When the preset threshold is set to 1, no tokens are removed, while slightly reducing its value can prune redundant context. This trade-off is particularly effective due to the "attention-focusing" phenomenon, where the generated text often heavily relies on a small subset of tokens, allowing for significant pruning with minimal performance impact.

[0078] Step S14: Remove the key-value pairs of words other than the keyword words from the initial key-value cache to obtain the compressed key-value cache.

[0079] It is understandable that keyword elements contribute significantly to the generation of terminal words, while non-keyword elements contribute very little. Therefore, key-value pairs of words other than keyword elements are removed from the initial key-value cache to obtain the compressed key-value cache.

[0080] Determine the index corresponding to the keyword elements at each level. And based on index Hidden state pruning and non-keyword key-value pair pruning are performed on each layer to obtain a compressed key-value cache that retains only the key-value pairs of keyword elements and the hidden state corresponding to the keyword elements.

[0081] This embodiment reduces computational overhead by eviction of non-keyword key-value pairs during the Transformer model pre-filling stage. It identifies and removes redundant context tokens iteratively, while retaining tokens that are crucial to maintaining model quality. It also ensures compatibility with modern service engines that feature virtual pagination blocks, tensor parallelism, and prefix caching, without relying on the full attention weight matrix.

[0082] Step S15: Use the target large language model to generate inference results corresponding to the compressed key-value cache.

[0083] In the process of generating inference results corresponding to the compressed key-value cache using the target large language model, the time for the target large language model to read the compressed key-value cache is also effectively reduced because the amount of data in the compressed key-value cache is greatly reduced compared to the initial key-value cache, thereby improving the efficiency of the target large language model in generating inference results.

[0084] The beneficial effects of this application are as follows: This application utilizes a target large language model to segment the prompt words in the current question-answering task into a lexical sequence and generates an initial key-value cache of the lexical sequence; wherein, the target large language model includes a neural network architecture with an attention mechanism; based on the preset end window size of each attention layer of the target large language model, the lexical sequence is divided into context lexical units and end lexical units corresponding to each layer; based on the importance score between the key matrix of the context lexical units and the mean of the query matrix of the end lexical units, key lexical units of each attention layer are selected from the context lexical units; wherein, the importance score is used to characterize the contribution of the context lexical units to the generation of the end lexical units; key-value pairs of lexical units other than the key lexical units in the initial key-value cache are removed to obtain a compressed key-value cache; and the target large language model is used to generate inference results corresponding to the compressed key-value cache. Therefore, this application removes key-value pairs of lexical units other than key words from the initial key-value cache, compresses the key-value cache, reduces computational overhead, and lowers the first word generation time and inter-word latency. This improves throughput without affecting the robustness of the service system and utilizes computational resources more efficiently, thus reducing the high computational cost of large language models due to the growth of key-value cache data. Furthermore, the lexical importance scoring mechanism used in this application for key-value compression does not require access to the complete attention matrix. It determines the importance score between the key matrix of context lexical units and the mean of the query matrix of terminal lexical units, i.e., the contribution of context lexical units to the generation of terminal lexical units. Then, based on the importance score, key words for each attention layer are selected from the context lexical units. This eliminates the need to explicitly generate the attention matrix, effectively reducing complexity. It achieves approximate global attention distribution through mean query, retains key semantic information, and breaks through the dependence of existing key-value cache compression methods on the complete attention weight matrix.

[0085] Furthermore, the equivalence between the importance score acquisition formula in this embodiment and the traditional average attention score acquisition method is established by the following lemma:

[0086] For matrix and Where w is any positive integer representing the number of queries, and n is any positive integer representing the number of keys:

[0087] ;

[0088] Let Q = [q1, ..., q w ] T ,in Let K represent the i-th query vector, where K = [k1, ..., k]. n ] T ,in Represents the j-th key vector. Matrix product. for:

[0089] ;

[0090] First take the average of Q and then multiply by :

[0091] ;

[0092] The last equation is derived from the definition of matrix multiplication and the averaging operation along axis=0.

[0093] See Figure 3 As shown in the figure, this application discloses a question-answering inference device based on key-value compression caching, including:

[0094] The initial key-value pair generation module 11 is used to segment the prompt words in the current question-answering task into word sequence using the target large language model, and generate an initial key-value cache of the word sequence; wherein, the target large language model includes a neural network architecture with an attention mechanism;

[0095] Lexical segmentation module 12 is used to divide the lexical sequence into context lexical units and terminal lexical units corresponding to each layer based on the preset end window size of each attention layer of the target large language model;

[0096] The lexical filtering module 13 is used to filter out key words for each attention layer from the context words based on the importance score between the key matrix of the context words and the mean of the query matrix of the terminal words; wherein, the importance score is used to characterize the contribution of the context words to the generation of the terminal words;

[0097] The key-value pair compression module 14 is used to remove key-value pairs of words other than the keyword words in the initial key-value cache to obtain a compressed key-value cache.

[0098] The reasoning result generation module 15 is used to generate reasoning results corresponding to the compressed key-value cache using the target large language model.

[0099] The beneficial effects of this application are as follows: This application utilizes a target large language model to segment the prompt words in the current question-answering task into a lexical sequence and generates an initial key-value cache of the lexical sequence; wherein, the target large language model includes a neural network architecture with an attention mechanism; based on the preset end window size of each attention layer of the target large language model, the lexical sequence is divided into context lexical units and end lexical units corresponding to each layer; based on the importance score between the key matrix of the context lexical units and the mean of the query matrix of the end lexical units, key lexical units of each attention layer are selected from the context lexical units; wherein, the importance score is used to characterize the contribution of the context lexical units to the generation of the end lexical units; key-value pairs of lexical units other than the key lexical units in the initial key-value cache are removed to obtain a compressed key-value cache; and the target large language model is used to generate inference results corresponding to the compressed key-value cache. Therefore, this application removes key-value pairs of lexical units other than key words from the initial key-value cache, compresses the key-value cache, reduces computational overhead, and lowers the first word generation time and inter-word latency. This improves throughput without affecting the robustness of the service system and utilizes computational resources more efficiently, thus reducing the high computational cost of large language models due to the growth of key-value cache data. Furthermore, the lexical importance scoring mechanism used in this application for key-value compression does not require access to the complete attention matrix. It determines the importance score between the key matrix of context lexical units and the mean of the query matrix of terminal lexical units, i.e., the contribution of context lexical units to the generation of terminal lexical units. Then, based on the importance score, key words for each attention layer are selected from the context lexical units. This eliminates the need to explicitly generate the attention matrix, effectively reducing complexity. It achieves approximate global attention distribution through mean query, retains key semantic information, and breaks through the dependence of existing key-value cache compression methods on the complete attention weight matrix.

[0100] Furthermore, embodiments of this application also provide an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0101] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the key-value cache compression-based question-answering reasoning method disclosed in any of the foregoing embodiments.

[0102] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0103] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0104] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0105] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the key-value cache compression-based question-answering reasoning method disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0106] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned question-answering reasoning method based on key-value cache compression. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0107] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0108] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly in hardware, software modules executed by a processor, or a combination of both. The software module may be located in random access memory (RAM), memory, read-only memory (ROM), electrically programmable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), register, hard disk, removable disk, CD-ROM (Compact Disc Read-Only Memory), or any other form of storage medium known in the art.

[0109] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] The foregoing has provided a detailed description of a question-answering reasoning method, apparatus, device, and medium based on key-value caching compression provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A question-answering reasoning method based on key-value caching compression, characterized in that, include: The target large language model is used to segment the prompt words in the current question answering task into a word sequence, and an initial key-value cache of the word sequence is generated; wherein, the target large language model includes a neural network architecture with an attention mechanism; Based on the preset end window size of each attention layer of the target large language model, the word sequence is divided into context words and end words corresponding to each layer; Keyword units for each attention layer are selected from the context units based on the importance score between the key matrix of the context units and the mean of the query matrix of the terminal units; wherein, the importance score is used to characterize the contribution of the context units to the generation of the terminal units; Remove key-value pairs of words other than the keyword word from the initial key-value cache to obtain the compressed key-value cache; The target large language model is used to generate inference results corresponding to the compressed key-value cache.

2. The question-answering reasoning method based on key-value caching compression according to claim 1, characterized in that, The initial key-value cache for generating the lexical sequence includes: Each word in the word sequence is sequentially identified as the current word; The key matrix and value matrix of the current word are obtained using the hidden state and projection matrix of the current word; wherein the hidden state is determined based on the length of the word sequence and the dimension of the hidden layer. Add the key matrix and value matrix of the current word to the current key-value cache to obtain the new current key-value cache; When the key matrix and value matrix of all words in the word sequence are added to the current key-value cache, the new current key-value cache is determined as the initial key-value cache of the word sequence.

3. The question-answering reasoning method based on key-value caching compression according to claim 2, characterized in that, The step of selecting keyword elements for each attention layer from the context words based on the importance score between the key matrix of the context words and the mean of the query matrix of the terminal words includes: The mean of the query matrix of the terminal word is taken along the sequence dimension to obtain the mean of the query matrix of the terminal word, and the dot product of the key matrix of the context word and the mean of the query matrix of the terminal word is taken to obtain the importance score. Keyword units for each attention layer are selected from the contextual units based on the importance score.

4. The question-answering reasoning method based on key-value caching compression according to claim 3, characterized in that, The step of filtering keyword units for each attention layer from the contextual units based on the importance score includes: The importance scores of the current attention layer are normalized to obtain a probability distribution sequence arranged in descending order; The target probability distribution is determined from the probability distribution sequence based on a cumulative score threshold mechanism; Select the keyword tokens of the current attention layer that correspond to the target probability distribution from the context tokens.

5. The question-answering reasoning method based on key-value caching compression according to claim 4, characterized in that, The determination of the target probability distribution from the probability distribution sequence based on the cumulative score threshold mechanism includes: The current probability distribution is determined from the probability distribution sequence, and the cumulative distribution sum from the first probability distribution in the probability distribution sequence to the current probability distribution is determined; Determine whether the cumulative distribution sum is not less than a preset threshold; If the cumulative sum of the distributions is not less than a preset threshold, then the probability distributions from the first end to the current probability distribution are all determined as the target probability distribution. If the cumulative sum of the distributions is less than a preset threshold, the next probability distribution is determined from the probability distribution sequence in order from the first end to the last end, and the next probability distribution is updated to the current probability distribution. Then, the process jumps back to the step of determining the cumulative sum of the probability distributions from the first probability distribution to the current probability distribution in the probability distribution sequence.

6. The question-answering reasoning method based on key-value caching compression according to claim 3, characterized in that, The formula for obtaining the importance score is: ； in, The key matrix, For querying the matrix, For terminal words, For contextual words, T represents the matrix transpose along the sequence dimension.

7. The question-answering reasoning method based on key-value caching compression according to any one of claims 1 to 6, characterized in that, The method of dividing the lexical sequence into context lexical units and terminal lexical units corresponding to each layer based on the preset end window size of each attention layer of the target large language model includes: The preset end window size is determined by multiplying the preset window division ratio of the current attention layer in the target large language model with the length of the word sequence; The lexical units in the lexical sequence that correspond to the preset end window size are classified as the end lexical units of the current attention layer, and the lexical units in the lexical sequence other than the end lexical units are classified as the context lexical units of the current attention layer.

8. A question-answering reasoning device based on key-value compression caching, characterized in that, include: An initial key-value pair generation module is used to segment the prompt words in the current question-answering task into word sequence using a target large language model, and generate an initial key-value cache of the word sequence; wherein, the target large language model includes a neural network architecture with an attention mechanism; The lexical segmentation module is used to divide the lexical sequence into context lexical units and terminal lexical units corresponding to each layer based on the preset end window size of each attention layer of the target large language model; A lexical filtering module is used to filter key lexical units for each attention layer from the context lexical units based on the importance score between the key matrix of the context lexical units and the mean of the query matrix of the terminal lexical units; wherein, the importance score is used to characterize the contribution of the context lexical units to the generation of the terminal lexical units; The key-value pair compression module is used to remove key-value pairs of words other than the keyword words in the initial key-value cache to obtain a compressed key-value cache. The reasoning result generation module is used to generate reasoning results corresponding to the compressed key-value cache using the target large language model.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the question-answering reasoning method based on key-value cache compression as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the steps of the question-answering reasoning method based on key-value cache compression as described in any one of claims 1 to 7.

Images