KV Cache data multiplexing method and device and electronic equipment

Abstract

The embodiment of the invention relates to the technical field of artificial intelligence, in particular to a KV Cache data multiplexing method and device and electronic equipment, and the method can improve the efficiency of model prefill. The method comprises the steps of generating a prompt corresponding to the round of questions and answers in response to questions input by a user in the round of questions and answers; the prompt comprises a system prompt part, a user prompt part and a user input part which are arranged according to a fixed sequence; the position ID of each part in the prompt is fixed; the user prompting part comprises a plurality of segmented blocks, and the position ID of each of the plurality of blocks in the prompt is fixed; a user input section including a question; inputting a prompt to the large language model, and querying whether a current block in the plurality of blocks hits the KV Cache data in the memory or not in a Prefill stage; and under the condition of hit, reading KV Cache data corresponding to the current block from the memory.

Description

Technical Field

[0001] This application relates to the field of AI technology, and in particular to a KV Cache data reuse method, apparatus and electronic device. Background Technology

[0002] KV Cache, or key-value cache, is a technique for optimizing inference by caching the key (K) and value (V) data in the attention mechanism. In large language models based on the Transformer architecture, the attention mechanism is one of the most computationally expensive parts, involving a large number of matrix multiplications and softmax operations. KVCache significantly improves inference efficiency by caching the K and V of previously used tokens, avoiding recalculation of these values ​​on each inference iteration.

[0003] Currently, the inference task in the prefill phase consumes a significant amount of time and / or computing resources. In other words, the inference efficiency of the prefill phase needs to be further improved. Summary of the Invention

[0004] This application provides a KV Cache data reuse method, apparatus, and electronic device, which can classify the content in the prompt into multiple parts, and one or more of these parts can be further divided into multiple blocks, thereby realizing KV Cache data reuse on a block-by-block basis, improving the inference efficiency of the Prefill stage, and reducing prefill latency.

[0005] In a first aspect, embodiments of this application provide a KV Cache data reuse method, which is applied to an electronic device. The method may include: generating a first prompt corresponding to the current round of question-and-answer in response to a first question (Query) input by a user in the current round of question-and-answer; the first prompt includes multiple parts; the multiple parts include a system prompt part, a user prompt part, and a user input part arranged in a fixed order; wherein the position identifier ID of each part in the first prompt is fixed; at least one part includes multiple segments, and the position ID of each segment in the first prompt is fixed; the system prompt part includes prompts common to multiple rounds of question-and-answer; the user prompt part includes at least the user's corresponding historical question-and-answer records, and / or tool information to be invoked in the current round of question-and-answer; the user input part includes the first question; inputting the first prompt into a large language model; during the prefill stage, querying whether the current segment among the multiple segments hits KV Cache data in memory; if a hit occurs, reading the KV Cache data corresponding to the current segment from memory.

[0006] In the above method, the prompt is divided into multiple parts, and at least one part is divided into multiple blocks. The position ID of each part and each block in the prompt remains fixed. When the content in the preceding text changes, it will not affect the position ID of the part or block in the following text. In this way, the problem of the position ID of the part or block in the following text changing dynamically due to the dynamic changes in the preceding text can be solved. For a given prompt length, under the premise of ensuring the accuracy of prefill inference, the KV Cache data corresponding to each block in the prompt can be fully or partially reused to replace computation with storage, thereby improving the efficiency of prefill inference and reducing the prefill latency.

[0007] In one possible implementation, the user prompt section includes a dynamic user prompt section and / or a static user prompt section; wherein the position ID of the dynamic user prompt section and / or the static user prompt section in the first prompt remains fixed; the dynamic user prompt section includes the user's corresponding historical question and answer records, and / or the tool information to be called to execute this round of tasks; the static user prompt section includes the user's profile information.

[0008] In one possible implementation, the fixed order is that the system prompts, static user prompts, dynamic user prompts, and user input are arranged from front to back.

[0009] The term "fixed order" does not mean the only order. A fixed order refers to the fact that the relative order of the parts in a given embodiment is fixed and does not change dynamically. Different embodiments may use different orders.

[0010] In one possible implementation, each of the multiple blocks has an independent block code ID, position ID, and maximum length; wherein, the block ID is used to uniquely identify a block; and the maximum length of a block is fixed.

[0011] In one possible implementation, the method further includes: if the data length of the content within a block in the first prompt is less than the maximum length, padding with a specified character; and the data length of the content within the padded block reaches the maximum length.

[0012] In one possible implementation, the KV Cache data corresponding to the current block includes first KV Cache data and second KV Cache data; the first KV Cache data is data obtained by calculating the attention of the current block relative to the previous block; the second KV Cache data is data obtained by calculating the attention of the current block relative to the preceding text; if the content in the system prompt section changes, the system prompt section includes multiple segments; reading the KV Cache data corresponding to the current block from memory includes: determining, based on the position ID of the current block, that the current block is not the first block in the system prompt section, and reading the first KV Cache data corresponding to the current block from memory; or, if the content in the system prompt section remains unchanged, reading the KV Cache data corresponding to the current block from memory includes: determining, based on the position ID of the current block, that the current block is a block in the user prompt section, and reading the first KV Cache data corresponding to the current block from memory; the first KV Cache data is data obtained by calculating the attention of the current block relative to the previous block.

[0013] In one possible implementation, the method further includes: determining that the current block is not the first block in the system prompt section, or determining that the current block is a block in the user prompt section, and performing a recalculation of the current block.

[0014] In one possible implementation, before performing the recomputation corresponding to the current block, the method further includes: determining at least one high-biased token from multiple element tokens corresponding to the current block; the high-biased token is a token whose bias between the KV Cache data recomputed in the current round of question answering and the second KV Cache data calculated in the previous round of question answering meets a predetermined condition; performing the recomputation corresponding to the current block includes: performing the recomputation corresponding to at least one high-biased token in the current block.

[0015] In this implementation, meeting a predetermined condition can be achieved by the bias value exceeding a predetermined threshold. Alternatively, meeting the predetermined condition can be achieved by sorting the bias values ​​from largest to smallest, with the top Q being the most significant. This is equivalent to selecting the Q tokens with the largest biases from a pool of tokens. Q is a positive integer.

[0016] In one possible implementation, determining at least one high-bias token from multiple tokens corresponding to the current block includes: performing attention calculation for the current block in a specified network layer of the large language model to obtain a first set of key-value (KV) data, which includes KV data corresponding to multiple tokens; reading the KV cache data corresponding to the current block calculated in the previous round of question answering from memory as a second set of KV data; calculating the bias between the KV data corresponding to the same token in the first and second sets of KV data; and selecting tokens whose bias meets predetermined conditions from the multiple tokens corresponding to the current block as high-bias tokens.

[0017] In this implementation, the calculation of attention for the current block can be performed by calculating the attention of multiple tokens in the current block relative to the tokens mentioned above. Alternatively, it can be performed by performing all calculations for multiple tokens in the current block, including the attention calculations of multiple tokens in the current block relative to the tokens mentioned above, and the attention calculations of the current block relative to the tokens within the block.

[0018] In one possible implementation, after recompiling the recalculation of at least one high-biased token in the current block, the method further includes: recompiling the attention corresponding to at least one high-biased token in the subsequent network layers of the specified network layer in the large language model; writing the KV data corresponding to at least one high-biased token recomputed by the specified network layer and the subsequent network layers in the current question answering round into memory, replacing the KV data stored in memory in the previous question answering round.

[0019] In one possible implementation, the system hint section includes multiple segments. When the content of the system hint section remains fixed, the KV Cache data corresponding to the current segment is read from memory, including: determining that the current segment is a segment within the system hint section based on its position ID, and then reading all the KV Cache data corresponding to the current segment from memory. Alternatively, when the content of the system hint section changes, the KV Cache data corresponding to the current segment is read from memory, including: determining that the current segment is the first segment in the system hint section based on its position ID, and then reading all the KV Cache data corresponding to the current segment from memory.

[0020] Secondly, embodiments of this application propose a KV Cache data reuse device, which includes: a generation module, used to generate a first prompt corresponding to the current round of question and answer in response to a first question Query input by a user in the current round of question and answer; the first prompt includes multiple parts; the multiple parts include a system prompt part, a user prompt part, and a user input part arranged in a fixed order; wherein, the position identifier ID of each part in the first prompt is fixed; at least one part in the multiple parts includes multiple blocks after segmentation, and the position ID of each block in the first prompt is fixed; the system prompt part includes a prompt common to multiple rounds of question and answer; the user prompt part includes at least the user's corresponding historical question and answer records, and / or, tool information to be called in the current round of question and answer; the user input part includes the first question; a query module, used to input the first prompt into a large language model, and in the prefill stage, query whether the current block among the multiple blocks hits a block in the memory; the memory is used to store KV Cache data corresponding to at least one block; and a reading module, used to read the KV Cache data corresponding to the current block from the memory if a hit occurs.

[0021] In one possible implementation, the user prompt section includes a dynamic user prompt section and / or a static user prompt section; wherein the position ID of the dynamic user prompt section and / or the static user prompt section in the first prompt remains fixed; the dynamic user prompt section includes the user's corresponding historical question and answer records, and / or the tool information to be called to execute this round of tasks; the static user prompt section includes the user's profile information.

[0022] In one possible implementation, the fixed order is that the system prompts, static user prompts, dynamic user prompts, and user input are arranged from front to back.

[0023] In one possible implementation, each of the multiple blocks has an independent block code ID, position ID, and maximum length; wherein, the block ID is used to uniquely identify a block; and the maximum length of a block is fixed.

[0024] In one possible implementation, the device further includes a padding module for padding a block of content in the first prompt with specified characters if the data length of the content is less than the maximum length; the data length of the content in the padded block reaches the maximum length.

[0025] In one possible implementation, the KV Cache data corresponding to the current block includes first KV Cache data and second KV Cache data; the first KV Cache data is obtained by calculating the attention of the current block relative to the previous block; the second KV Cache data is obtained by calculating the attention of the current block relative to the preceding text; when the content in the system prompt section changes, the system prompt section includes multiple segments; when reading the KV Cache data corresponding to the current block from the memory, the reading module is specifically used to: determine, based on the position ID of the current block, that the current block is not the first block in the system prompt section, and read the first KV Cache data corresponding to the current block from the memory;

[0026] Alternatively, if the content in the system prompt section remains fixed, when reading the KV Cache data corresponding to the current block from memory, the reading module is specifically used to: determine that the current block is a block in the user prompt section based on the position ID of the current block, and read the first KV Cache data corresponding to the current block from memory; the first KV Cache data is data obtained by calculating the attention of the current block relative to the current block.

[0027] In one possible implementation, the device further includes a recalculation module for performing a recalculation of the current block if it is determined that the current block is not the first block in the system prompt section, or if it is determined that the current block is a block in the user prompt section.

[0028] In one possible implementation, before performing the recomputation corresponding to the current block, the recomputation module is further configured to: determine at least one high-bias token from the multiple element tokens corresponding to the current block; the high-bias token is a token whose bias between the KV Cache data recomputed in the current round of question answering and the KV Cache data calculated in the previous round of question answering meets a predetermined condition; when performing the recomputation corresponding to the current block, the recomputation module is specifically configured to: perform the recomputation corresponding to at least one high-bias token in the current block.

[0029] In one possible implementation, determining at least one high-bias token from multiple tokens corresponding to the current block includes: performing attention calculation for the current block in a specified network layer of the large language model to obtain a first set of key-value (KV) data; the first set of KV data includes KV data corresponding to multiple tokens; reading the KV cache data corresponding to the current block calculated in the previous round of question answering from memory as a second set of KV data; the second set of KV data includes KV data corresponding to multiple tokens; calculating the bias between the KV data corresponding to the same token in the first and second sets of KV cache data; and selecting tokens whose bias meets predetermined conditions from the multiple tokens corresponding to the current block as high-bias tokens.

[0030] In one possible implementation, after recompiling at least one high-bias token in the current block, the recompiling module is further configured to: recompile the attention corresponding to at least one high-bias token in the subsequent network layers of the specified network layer in the large language model; and write the KV data corresponding to at least one high-bias token recomputed by the specified network layer and the subsequent network layers in the current question answering round into the memory, replacing the KV data stored in the memory in the previous question answering round.

[0031] In one possible implementation, the system hint section includes multiple segments. When the content of the system hint section remains fixed, when reading the KV Cache data corresponding to the current block from memory, the reading module is specifically used to: determine, based on the position ID of the current block, that the current block is a block within the system hint section, and read all the KV Cache data corresponding to the current block from memory. Alternatively, when the content of the system hint section changes, when reading the KV Cache data corresponding to the current block from memory, the reading module is specifically used to: determine, based on the position ID of the current block, that the current block is the first block in the system hint section, and read all the KV Cache data corresponding to the current block from memory.

[0032] Thirdly, embodiments of this application also provide an electronic device, which includes a processor for executing computer programs or instructions in a memory to implement the methods described in any of the first aspects above.

[0033] Fourthly, embodiments of this application also propose a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method as described in any of the first aspects above.

[0034] Fifthly, embodiments of this application also propose a computer program product containing instructions that, when run by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of the first aspects above.

[0035] In a sixth aspect, embodiments of this application also provide a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of the first aspects above.

[0036] In a seventh aspect, embodiments of this application also provide a chip system, including: a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the method as described in any one of the first aspects above. Attached Figure Description

[0037] Figure 1 A schematic diagram illustrating the reasoning process in the prefill and decode stages of a large language model;

[0038] Figure 2 Delay the prefill for large language models under different prompt lengths;

[0039] Figure 3 This is a schematic diagram illustrating the system-hinted KV Cache reuse during the prefill stage in related technologies;

[0040] Figure 4 This is a schematic diagram of the structure of the electronic device (terminal device) provided in the embodiments of this application;

[0041] Figure 5 This is a schematic diagram of the structure of the electronic device (server) provided in the embodiments of this application;

[0042] Figure 6 A schematic diagram illustrating an application scenario of the KV Cache data reuse method provided in this application embodiment;

[0043] Figure 7 A system architecture example diagram of the KV Cache data reuse method provided in the embodiments of this application;

[0044] Figure 8An example diagram of a prompt template is provided in some embodiments of this application;

[0045] Figure 9 A schematic diagram showing that each part or block has a fixed position ID in some embodiments provided in this application;

[0046] Figure 10 Another example diagram of a prompt template is provided in some other embodiments of this application;

[0047] Figure 11 Another example diagram of a prompt template provided in some of the embodiments of this application;

[0048] Figure 12 Example diagrams of multi-turn question-and-answer scenarios in some embodiments of the KV Cache data reuse method provided in this application;

[0049] Figure 13 Flowcharts of the KV Cache data reuse method provided in some embodiments of this application;

[0050] Figure 14 A schematic diagram of high-bias KV recalculation in the KV Cache data reuse method provided in some embodiments of this application;

[0051] Figure 15 Another schematic diagram of high-bias KV recalculation in the KV Cache data reuse method provided in some embodiments of this application;

[0052] Figure 16 Schematic diagrams of computing devices provided in some embodiments of this application;

[0053] Figure 17 Schematic diagrams of computing device clusters provided in some embodiments of this application;

[0054] Figure 18 This application provides a schematic diagram illustrating how one or more computing devices in a computing device cluster can be connected via a network in some embodiments. Detailed Implementation

[0055] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more.

[0057] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0058] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options.

[0059] Recently, Large Language Models (LLMs) have emerged as a type of artificial intelligence model designed to understand and generate human language. They can handle various natural language tasks, such as text classification, question answering, and dialogue. Generative Pre-trained Transformer (GPT) models, represented by generative models, have a distinctive reasoning process. Given a sequence of input tokens of length S, the model outputs a sequence of output tokens of length N. For example, if a user inputs the question "What did you eat for lunch today?", the LLM might output the textual answer, "As an AI model, I have no physical existence and no ability to eat, so I did not eat anything."

[0060] As shown, taking the reasoning process of a GPT-type large language model as an example, a task round can include one context prefilling process and multiple decoding processes. In the following description, the context prefilling process will be referred to as the Prefill stage, and the decoding process will be referred to as the Decode stage.

[0061] In the Prefill stage, the input data dimension is [B, S]. B is the dimension of the group batch, which is the batch in machine learning and can be understood as the number of tasks that the large language model can process in parallel at one time, that is, the number of requests or questions that the model can respond to simultaneously at one time. For example, if the model allows answering B questions raised by different users simultaneously in one round of tasks, then the group batch is B. S represents the length of the input text sequence, which can be recorded as the number of text elements (tokens). S represents S tokens. The data processing and transmission dimension of the middle layer of the model is [B, S, H], where H is the hidden layer dimension. The output data dimension is [B, 1], that is, B tokens for one batch. If there is only one batch, there is only one token. The Prefill stage only performs inference once.

[0062] For example, as Figure 1 shown, in the Prefill stage, in response to the question "What did you have for lunch today?" raised by the user, the first token in the predicted answer will be obtained through the prefill process. The first token may be the word "As" or may be the character "作".

[0063] In the Decode stage, the input data dimension is [B, 1]. The data processing and transmission dimension of the middle layer of the model is [B, 1, H]. The output data dimension is [B, 1]. The Decode stage will perform inference N times until it stops executing when encountering an end condition. In one inference during the N inferences, the GPT-like model only outputs one token. The output token will be concatenated with the previous S input tokens to obtain S + 1 tokens, which will be used as the input for the next inference. And so on, in the N inferences, (S + 1, S + 2,..., S + N - 1) tokens will be input respectively until a termination symbol or the maximum output length is encountered.

[0064] For example, as Figure 1 shown, when the first token output in the Prefill stage is "As", in one inference process, it is concatenated with the previous multiple tokens "What did you have for lunch today?" and through decoding, the next token output is "a", and so on. After performing the N inference processes, the final output answer "As an AI model, I have no physical existence and no ability to eat, so I didn't eat anything" is obtained.

[0065] This processing method will introduce a problem: as the number of iterations increases, the computational amount of each inference will increase accordingly; and since each iteration includes the input part of the previous iteration, there is a large amount of redundant calculation. Therefore, the KV cache technology can be used to keep the computational amount of each inference after the first round of iteration basically constant, so as to save computing power and reduce the inference latency.

[0066] For example, in a task round, during the Prefill phase, the key-value data generated by the intermediate Transformer layer is cached in memory during inference for use in the Decode phase. The generated key-value data has dimensions [B, 2, S, H], where 2 represents the two types of data: key-value (K) and value-value (V). Figure 1 The KV Cache shown represents memory, such as a cache.

[0067] During the Decode phase, in each inference process, the key-value data stored in the previous round can be retrieved from the key-value cache. The dimensions of the key-value data stored in the previous round are [B, 2, S+n, H], where n = 0, 1, ..., N-1. In the first inference of the decode phase, n = 0. Furthermore, the newly generated key-value data for this inference is cached in the key-value cache for updates. The dimensions of the newly generated key-value data for this inference are [B, 2, 1, H], where 1 represents the key-value data corresponding to one newly generated token. This newly generated key-value data is concatenated with the key-value data of the previous 's' tokens and cached in the key-value cache. In the next inference, the concatenated key-value data corresponding to the 's+n' tokens can be read.

[0068] The method proposed in the embodiments of this application, such as Figure 1 The dashed box markers in the text mainly refer to improvements to the KV data caching mechanism in the Prefill stage.

[0069] During the prefill inference phase, the inference latency is positively correlated with the length of the prompt. The computational complexity of attention is quadratically related to the length of the prompt, while the computational complexity of the fully connected feed-forward neural network (FFN) is linearly related to the length of the prompt. The longer the prompt length, the longer the prefill computational latency.

[0070] The relationship between the length of the input prompt and the prefill latency in a typical large language model based on the transformer architecture is as follows: Figure 2 As shown, Figure 2 The prefill delay (ms) of the XX model under different prompt lengths is shown. Figure 2 The horizontal axis represents the length of the input prompt sequence, and the number line represents the inference delay of one round of the task. Figure 2It can be seen that during the prefill phase, the inference latency of a single task round is positively correlated with the length of the input prompt. When the input is too long, the prefill latency is insufficient to meet the end-to-end requirements of the business.

[0071] The aforementioned caching mechanism in the prefill stage caches the key-value data generated by the intermediate Transformer in a single task into memory for retrieval in the Decode stage. This can help improve the inference efficiency of the Decode stage, but the caching mechanism cannot reduce the inference latency of the prefill stage.

[0072] In practical applications, such as intelligent question answering based on large language models, to respond to different questions posed by the same user in different rounds of question answering, or to respond to different questions posed by different users in different rounds of question answering, the large language model needs to perform multiple rounds of tasks to generate corresponding answers for different questions. In this process, there may be duplicate content in the prompts from different rounds of question answering. During the prefill stage, if the duplicate content in the prompts from different rounds can be reused using a key-value cache, the inference efficiency of the prefill stage can be improved, and the prefill latency can be reduced.

[0073] Taking a question-and-answer scenario as an example, user interaction with a large language model may involve multiple rounds of question-and-answer sessions. This means the user asks multiple questions in succession, and the large language model responds to these multiple requests in multiple rounds, answering one question per round. Generally, the questions asked by the user in different rounds are usually different. For example, in the first round, user A asks, "What did you eat for lunch today?" The large language model answers, "As an AI model, I don't have a physical body and the ability to eat, so I didn't eat anything. However, I can help you plan your lunch or provide some lunch suggestions." In the second round, the user continues, "I want to eat braised pork belly, how do you make it?" The large language model answers, "Braised pork belly is a very popular traditional Chinese dish, delicious and visually appealing. Here's a classic recipe: Ingredients: Pork belly… Serve the braised pork belly and garnish with a little chopped green onion."

[0074] It's important to note that the actual input data fed into the large language model isn't just the user's questions. Instead, it's based on the user's questions, with a prompt generated through a prompting process before being input into the large language model. During the interaction with the large model, due to factors such as insufficient user experience and model differences, inappropriate prompt commands may occur. Therefore, the prompting process is needed to further refine the user's questions, enabling them to control the model's output and generate accurate and high-quality text responses.

[0075] In related technologies, user-submitted questions are sometimes referred to as prompts. In this embodiment of the application, to avoid confusion, user-submitted questions are referred to as queries, and the inputs obtained based on user questions are referred to as prompts.

[0076] For example, in the exemplary two-round question-and-answer session above, the user's first question, "What did you eat for lunch today?", could generate a prompt based on this question:

[0077] "System: Your name is Xiao X, and you are a smart assistant developed by Huawei. Please select the appropriate tool based on the information in the [Historical Conversations] and [Candidate Tools], match the corresponding parameters, and output the tool invocation command. If multiple candidate tools are matched, ask the user for more information to select the unique tool. Consider the content in the user's [Specific Context], which can help understand the user's true intention when expressing a specific command."

[0078] [Candidate Tools]: [Tool] Name: NA. Description: Tools that cannot be matched above are classified into this category. Parameter Definition: No parameters.

[0079] [Tool] Name: QuestionAnswering. Description: Answers user questions. All knowledge-related questions should be handled by this API, which is particularly adept at handling questions related to Huawei devices. The API's question processing method is: obtaining general answers through a search network and internal knowledge base, then refining and enhancing these answers using a large model to provide users with richer, more timely, and more valuable answers. Parameter Definition: searchQuery: The text used for retrieval, a required parameter.

[0080] [Historical Dialogue]: <s>User: What did you eat for lunch today? <eot> <s>Little X:

[0081] In the second round of Q&A, the user asked, "I want to eat braised pork belly, how do you make braised pork belly?" Based on this question, the generated prompt might be:

[0082] System: Your name is Xiao X, and you are a smart assistant developed by Huawei. Based on the information in the [Historical Conversations] and [Candidate Tools], please select the appropriate tool, match the corresponding parameters, and output the tool invocation command. If multiple candidate tools are matched, ask the user for more information to select the unique tool. Consider the content in the user's [Specific Context], which can help understand the user's true intention when expressing a specific command.

[0083] [Candidate Tools]:

[0084] [Tool] Name: NA. Description: Tools that cannot be matched above are categorized into this class. Parameter Definition: No parameters.

[0085] [Tool] Name: Navigate. Description: Activates navigation, guiding the user to their desired destination. Parameter Definition: desLocation: Destination, required parameter.

[0086] [Historical Dialogue]: <unused0>User: What did you eat for lunch today? <unused1> <unused0>Little X: <unused2>Tool usage: QuestionAnswering(searchQuerv = "What did you eat for lunch today?"]) <unused3> <unused1> <unused0>Xiao X: As an AI model, I don't have a physical body and I don't have the ability to eat, so I don't eat anything. However, I can help you plan your lunch or offer some lunch suggestions. <unused1> <unusedo>User: I want to eat braised pork. How do you make braised pork? <unused1> <unused0>Little X:

[0087] As can be seen from the above two exemplary rounds of question and answer, the questions raised by users in different rounds of tasks may be different, but the prompts generated in the two rounds of tasks may contain duplicate content.

[0088] For example, a templated prompt includes a system prompt and a user prompt; generally, the system prompt is fixed. To accelerate inference during the prefill phase of a large language model (hereinafter referred to as the large model), one approach in related technologies is to reuse the key-value data in the system prompt section.

[0089] like Figure 3 As shown, the complete prompt obtained based on user input consists of two parts: a system prompt and a user prompt. The system prompt is the same for different user requests and different rounds of requests from the same user. During the first request, the system prompt is inferred and calculated, and the corresponding KVCache is cached. In subsequent inferences, the system prompt does not need to be inferred again; the corresponding key / value pairs are directly loaded from the KVCache for calculation, thus significantly reducing prefill latency.

[0090] However, the problem with this method is:

[0091] (1) Content that is the same as other parts of the user prompt or prompt in previous rounds cannot be reused.

[0092] In practical applications, for different rounds of prompts, besides the repeated system prompt, other parts of the prompt may contain duplicate content. For example, in the example above, the second round of prompts, compared to the first round, contains duplicates such as "User: What did you eat for lunch today?" and "[Candidate Tool]: [Tool] Name: NA. Description: All tools that cannot be matched above are classified into this category. Parameter Definition: No parameters." However, this part of the content does not belong to the system prompt, and therefore, in related technologies, such as... Figure 3 The proposed solution cannot reuse this repetitive content; that is, it cannot reuse the user prompt in the prompt.

[0093] Figure 3 The proposed solution fails to adequately reduce the time and / or computing resources required for the prefill phase, meaning that inference during the prefill phase still consumes a significant amount of time and / or computing power, leaving room for further optimization.

[0094] (2) It is only applicable to the fixed prompt template of prompt = System prompt + user prompt, and the system prompt must be exactly the same.

[0095] In practical applications, the content obtained from the prompt project is diverse, and may include other types of content that are neither system prompts nor user prompts, such as character descriptions and task descriptions. This content may also be repetitive, as proposed in related technologies. Figure 3 The method shown cannot be reused for other content such as character design or task description.

[0096] Furthermore, the system prompts are not always exactly the same across different rounds of the task; there may be variations in details. However, the aforementioned technical points are consistent with the above-mentioned... Figure 3 The proposed solution requires the system prompts to remain completely unchanged, which is not applicable to scenarios where the system prompts may change.

[0097] Regarding technical issue (1), apart from the system prompt, other identical content in the prompt cannot be used. Figure 3 The reason for reusing the KV Cache method in the related technologies shown is as follows:

[0098] 1) Different location information (Position Identity).

[0099] Each part of the Prompt has corresponding location information, which is the spatial number of this part of the content in the entire Prompt. The location information can also be called Position ID or Location ID.

[0100] 2) The above text is different.

[0101] In the prompt, except for the system prompt that does not contain the preceding text, all other parts of the content contain the preceding text, which is the prompt content preceding this part of the content.

[0102] In a large language model based on the transformer architecture, the calculation of key-value (KV) data is related to both the position ID and the context. This can be understood as involving a ternary function, which uses the feature vector corresponding to the current token, the position ID of the current token, and the feature vector corresponding to the context of the current token as variables. Therefore, when the position ID and context change, the calculated KV data also changes. This means that KV data calculated and stored in the cache for the same content in previous rounds (hereinafter referred to as KVCache data) cannot be directly reused. Directly reusing KVCache data would lead to precision issues.

[0103] The reason why the system prompt can be reused is that it is at the beginning of the entire prompt and has no preceding context. In other words, the system prompt has the same position ID and preceding context in different rounds of input.

[0104] In view of this, in order to solve at least one of the above problems, such as the inability to reuse KV Cache data due to changes in position ID and the preceding text, this application proposes a KV Cache data reuse method, apparatus and electronic device. For a given prompt length, under the premise of ensuring the accuracy of prefill inference, fragments of dynamically changing content in the prompt can be reused to replace computation with storage, thereby improving the efficiency of prefill inference and reducing prefill latency.

[0105] Specifically, in this application embodiment, a templated prompt template with a fixed format is proposed based on the prompt project. Based on this template design, the problem of position ID changes can be solved. In addition, a high-bias recalculation mechanism is proposed to address the problems caused by the above changes.

[0106] For example, Figure 4 This is a schematic diagram of the structure of an electronic device provided in some embodiments of this application. The electronic device may be a smartphone that supports running large language models, such as... Figure 4 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0107] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0108] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0109] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0110] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0111] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0112] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0113] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0114] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0115] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0116] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0117] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0118] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0119] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0120] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0121] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-CDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS). In other words, the electronic device 100 has positioning and wireless communication capabilities.

[0122] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0123] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0124] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0125] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0126] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0127] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0128] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0129] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0130] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0131] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0132] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0133] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0134] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0135] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0136] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0137] The electronic device proposed in this application can also be other terminal devices besides smartphones, such as tablet computers (Tablet PCs), laptops, desktop computers, wearable devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs). This application does not impose any limitations on the specific type of electronic device.

[0138] For example, the electronic device proposed in the embodiments of this application can also be a server. For example, Figure 5 This is a schematic diagram of the server structure in one embodiment of this application. Figure 5 As shown, server 200 may include: one or more processors 210, communication interface 220, memory 230, and communication bus 240 connecting different components (including memory 230, communication interface 220 and processor 210).

[0139] Communication bus 240 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, or a local bus using any of the various bus architectures. For example, communication bus 240 may include, but is not limited to, an industry standard architecture (ISA) bus, a micro channel architecture (MCA) bus, an enhanced ISA bus, a video electronics standards association (VESA) local bus, and a peripheral component interconnection (PCI) bus.

[0140] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0141] Memory 230 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Memory 230 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the KV Cache data multiplexing method provided in the embodiments of this application.

[0142] A program / utility having a set (at least one) of program modules can be stored in memory 230. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this application.

[0143] The processor 210 executes various functional applications and data processing by running programs stored in the memory 230, such as implementing the KV Cache data reuse method provided in the embodiments of this application.

[0144] It should be understood that Figure 3 The processor 210 in the server 200 shown can be a system-on-a-chip (SoC). The processor 210 may include a central processing unit (CPU) and may further include other types of processors, such as a graphics processing unit (GPU).

[0145] like Figure 6 As shown, the KV Cache data reuse method proposed in this application can be applied to application scenarios such as natural language processing, content creation, writing assistance, dialogue systems and virtual assistants, knowledge base construction and question answering systems, speech recognition and speech synthesis, as well as many other application scenarios such as data analysis and intelligent decision-making, personalized recommendation systems, education and online learning, medical and health management, social media and public opinion analysis, and legal and compliance management. Specifically, it can be applied to the prefill inference stage of large models built on the transformer architecture with KV Cache mechanism enabled.

[0146] For example, the KV Cache data reuse method provided in this application embodiment can be based on Figure 7 The system architecture implementation is shown.

[0147] like Figure 7 As shown, this system architecture may include a server 301 and one or more terminal devices 302 that communicate remotely with the server 301. For example, the terminal device 302 may be a smartphone, desktop computer, tablet computer, etc. A memory 303 is also deployed on the server 301 side to store KV cache data. In this system architecture example, the server 301 may be a standalone server or a server cluster, such as a Highly Available (HA) cluster, a Load Balance (LB) cluster, or a High Performance Computing (HPC) cluster. The memory 303 may be Dynamic Random Access Memory (DRAM) or a cache, where DRAM may include one or more of the following types of memory: global memory, high bandwidth memory (HBM), shared memory, or constant memory.

[0148] The large language model can be deployed on server 301. Terminal device 302 can receive user questions and upload them to server 301. In response to the user questions, server 301 generates a corresponding prompt according to the prompt template proposed in this embodiment, inputs it into the large language model, generates an answer through the large language model, and sends the answer to terminal device 302. Terminal device 302 displays the answer or outputs the answer in other ways, such as voice broadcasting. During the operation of the large language model, in the prefill phase, for different questions raised by the same user in different rounds, or for different questions raised by different users in different rounds, KV Cache data from previous rounds can be reused.

[0149] In the KV Cache data reuse method proposed in this application embodiment, the prompt is processed according to the templated prompt rules to obtain a complete prompt as the input of the large language model. This templated prompt can ensure that the same content in the prompt has the same position ID, that is, the same spatial position.

[0150] Specifically, the entire content of the prompt to be input into the large language model is divided into multiple parts according to type. Each of at least one of these parts is further divided into multiple prompt blocks (hereinafter referred to as blocks), which can also be called units or fragments. Each of the multiple parts can at least contain a type name promptName, a start position startPos, and a maximum length maxLen field. The start position and maximum length of each part are fixed, meaning that the data length occupied by each part is fixed.

[0151] Each of the multiple parts may further include multiple blocks (or cells), each block having independent basic attribute fields such as block name cellName, start position startPos, and maximum length maxLen.

[0152] To prevent confusion, the field name for the starting position of each part can be startPos0, and the field name for the maximum length of each part can be maxLen0; the field name for the starting position of each block can be startPos1, and the field name for the maximum length can be maxLen1.

[0153] For example, such as Figure 8 As shown, in some embodiments, based on the type of content, the prompt can be divided into several parts, such as system prompt, user prompt, and user input. The general prompts within the prompt can be categorized as follows: System prompts, which are common prompts used for multiple rounds of question-and-answer requests initiated by the same user or by different users; the questions entered by the user in the current round of question-and-answer, which can be categorized as user input; and the tools that may be invoked when predicting answers to the user's questions in the current round, as well as historical question-and-answer records or dialogues generated by the same user in previous rounds, which can be categorized as user prompts.

[0154] For example, regarding the prompt example given above:

[0155] System: Your name is Xiao X, and you are a smart assistant developed by Huawei. Based on the information in the [Historical Conversations] and [Candidate Tools], please select the appropriate tool, match the corresponding parameters, and output the tool invocation command. If multiple candidate tools are matched, ask the user for more information to select the unique tool. Consider the content in the user's [Specific Context], which can help understand the user's true intention when expressing a specific command.

[0156] [Candidate Tools]:

[0157] [Tool] Name: NA. Description: Tools that cannot be matched above are categorized into this class. Parameter Definition: No parameters.

[0158] [Tool] Name: Navigate. Description: Activates navigation, guiding the user to their desired destination. Parameter Definition: desLocation: Destination, required parameter.

[0159] [Historical Dialogue]: <unused0>User: What did you eat for lunch today? <unused1> <unused0>Little X: <unused2>Tool usage: QuestionAnswering(searchQuerv = "What did you eat for lunch today?"]) <unused3> <unused1> <unused0>Xiao X: As an AI model, I don't have a physical body and I don't have the ability to eat, so I don't eat anything. However, I can help you plan your lunch or offer some lunch suggestions. <unused1> <unusedo>User: I want to eat braised pork. How do you make braised pork? <unused1> <unused0>Little X: ";

[0160] If we follow the traditional format of system prompt + user prompt, then "System: Your name is Xiao X... This content can help understand the user's true intention when expressing a specific command" is the system prompt, and the rest is the user prompt.

[0161] According to the templated prompt proposed in the embodiments of this application, the system prompt is as follows: "System: Your name is Xiao X... This content can help understand the user's true intention when expressing a specific instruction"; "[Candidate Tool]: [Tool] Name: NA. Description: All tools that cannot be matched above are classified into this category. Parameter definition: No parameters. [Tool] Name: Navigate... [History Dialogue]: <unused0>User: What did you have for lunch today? ...However, I can help you plan your lunch or offer some lunch suggestions" is the user prompt section; <unused1> <unusedo>User: I want to eat braised pork. How do you make braised pork? <unused1> <unused0>Xiao X: "Regarding the user input section, it can be seen that the user input section includes the question 'I want to eat braised pork, how do you make braised pork?' The user input section can be based on the user's question (Query) with added prompts, such as adding prompts..." <unused1> <unused0>"wait.

[0162] One of the prompt rules used in this application embodiment is to sort multiple parts according to the principle of fixed first and variable last. Fixed parts (such as system prompt) are placed at the beginning of the entire prompt, and variable parts are placed at the end. Generally speaking, the questions entered by the user in different rounds are likely to change, so the user input is arranged at the end.

[0163] For example, Figure 8 As shown, the system prompts, user prompts, and user input sections are arranged in a sequential order. The user prompts may contain fixed or dynamically changing content.

[0164] This design principle of prioritizing fixed content and then introducing variable content can minimize changes in the preceding text and reduce the amount of recalculation required due to these changes. This point will be explained in detail later.

[0165] from Figure 8 As can be seen, the system prompt and user prompt sections are further divided into multiple blocks, while the user input section does not require segmentation. The system prompt section includes X blocks, namely Block 11, Block 12...Block X; the user prompt section includes Y blocks, namely Block 21, Block 22, Block 23...Block Y.

[0166] It should be noted that the user input portion may or may not be segmented. In other embodiments, the user input portion can be segmented, meaning it can include multiple segmented blocks. For example, if the user's question in the first round is "What's the weather like in Beijing?", the user input portion obtained from the question can be segmented into blocks, such as "Beijing" and "What's the weather like?". If the question in the second round is "What's the weather like in Shanghai?", then the "What's the weather like?" block can be matched with the "What's the weather like?" block from the previous round, and the KV Cache data corresponding to this block calculated in the previous round can be reused.

[0167] Among them, a block (cell) is the basic unit for KV Cache data matching and reuse during inference.

[0168] A block can contain an optional or required prompt message. The prompt message within a block can be fixed or dynamically changing. For example, the content of the first block in a system prompt is usually fixed. A block in the user prompt section can contain tool information. This tool can be a required tool or a tool selected from a list of candidate tools based on the current task. For example, in the prompt example above, after segmentation, "[Tool] Name: Navigate. Description: Activates navigation function, guiding the user on how to reach the desired destination. Parameter definition: desLocation: Destination, required parameter" can be a block.

[0169] It should be noted that in the templated prompt proposed in this application embodiment, the position ID of each part is fixed. Specifically, the starting position ID remains unchanged, and the total data length occupied by this part remains unchanged. Therefore, the position ID of this part remains unchanged in the entire prompt.

[0170] In addition, the position ID of each block in the prompt is also fixed. Specifically, the starting position ID of each block remains unchanged, and the data length occupied by each block remains unchanged. Therefore, the position ID of a block in the entire prompt remains unchanged.

[0171] In practical applications, the length of the prompt information contained in a block changes dynamically. For example, the length changes dynamically depending on the tool or the history of the conversation. The value of the maximum length field, maxlen, is actually the actual data length occupied by each block. If the data length occupied by the prompt information in a block is less than maxlen, a specified character can be used to fill it in. For example, the specified character can be a space. That is, for prompt blocks whose actual length does not reach the maxlen length, space information can be used to fill in the space information so that the actual data length occupied by each block is a fixed length, which is equal to maxlen.

[0172] Based on the above design, each part and block in the prompt can have a fixed position ID. When the prompt information in the part or block above changes dynamically, the position ID of each block or part in the subsequent prompt will not be affected.

[0173] For example, such as Figure 9 As shown in the figure, a block with a starting position ID of Addr and a length of L can be a block in the user prompt section, for example, it could be... Figure 8 The block shown is either Block 23 or Block Y. When this block represents Block 23, the subsequent prompt can represent the next Block 2. Whether the tool called within Block 23 is the navigation tool or the music tool, it will not affect the position ID of Block 24. When this block represents Block Y, the subsequent content can be the user input section. Regardless of how the content within this block changes dynamically, it will not affect the position ID of the next section.

[0174] It should be noted that, Figure 8 The prompt template shown is only one example.

[0175] In other embodiments, such as Figure 9 As shown, the prompt can also be divided into several parts, such as system prompt, user static prompt, user dynamic prompt, and user input. The user static prompt and user dynamic prompt can be understood as further splitting the user prompt into two parts.

[0176] The system prompts are divided into several parts: The static user prompts are divided into N blocks (Block 201, Block 202, ..., Block N); the dynamic user prompts are divided into T blocks (Block 301, Block 302, ..., Block T); and the system prompts are divided into M blocks (Block 101, Block 102, ..., Block M). M, N, and T are all integers, where M ≥ 1, N ≥ 1, and T ≥ 1. In some embodiments, N ≥ 2 and T ≥ 2.

[0177] The static user prompt section contains information that remains constant for the same user but may change for different users. In other words, the prompts in the prompt that represent individual user characteristics or preferences can be categorized as static user prompts. For example, static user prompts may include user profile information, such as age, gender, hobbies, and historical behavior. Setting static user information in the prompt allows the model to generate answers that better meet the user's individual needs, thus better matching personalized requirements.

[0178] User dynamic information is used to contain dynamic prompts indicating changes in the same user across different rounds of Q&A or changes in different users across different rounds of Q&A. For example, the user dynamic prompt section includes the user's corresponding historical Q&A records and / or information on the tools required to perform the current round of tasks. In each round of Q&A, the tools that may need to be called to respond to user questions are different, and the historical Q&A records (historical dialogues) generated by the user in previous Q&A sessions are generally also dynamically changing.

[0179] like Figure 10 As shown, the static user prompts will not change for the same user in the short term, while the dynamic user prompts will change in most cases. According to the principle of fixed content first and changed content later, the static user prompts are listed before the dynamic user prompts.

[0180] In other embodiments, methods such as Figure 11 The prompt template example shown. Figure 11 The prompt template example shown is compared to Figure 10 As shown, an additional "Others" section has been added to accommodate other types of prompts that may appear in the prompt. In practical applications, during prompt engineering, to improve the accuracy and controllability of the model-generated answers and make the generated answers more valuable, it may be necessary to add other types of prompts besides the system prompts, static user prompts, dynamic user prompts, and user inputs mentioned above. To improve the template's versatility, the prompt template proposed in this application embodiment may also include an "Others" section, which can categorize other types of prompts that may appear in the prompt content. For example... Figure 11 As shown, one possible sorting method is to place the Others section after the user's dynamic prompts and before the user's input.

[0181] Thus, the above design can solve the problem of dynamic changes in Position ID and avoid the recalculation of key-value data caused by changes in position ID.

[0182] As mentioned above, another problem that needs to be addressed in the prefill phase regarding KV Cache data reuse is the inability to reuse KV Cache data from previous rounds due to changes in the prompt. To address this issue, this application proposes a high-bias recalculation mechanism, which will be specifically described in the following embodiments.

[0183] like Figure 12 , Figure 12 This example illustrates the original prompts (to be segmented) generated for each of the three rounds of question-and-answer (dialogue) initiated by the same user, based on the questions posed in different rounds. In the original prompts, underlined sections indicate text content consistent with prompts generated in previous rounds. The entire original prompt includes system prompts, candidate tools, and historical dialogue information.

[0184] From the original prompts in round 1 (i.e., the row with round number 1) to round 3 (i.e., the row with round number 3), it can be seen that the questions "Are the maple leaves on Qixia Mountain red yet?" entered by the user in different rounds are stored in the history of the conversation. The most recently asked question in the current round is placed at the end of the history of the conversation, and the questions asked in previous rounds are placed at the beginning of the history of the conversation. In other words, in the history of the original prompt to be segmented, the questions and answers are sorted according to the order in which the questions were asked, and the most recently asked question in the current round is placed at the end of the history of the conversation.

[0185] Specifically, the model's response to the user's question in the current round (LLM Response) is added to the prompt information for the next round. For example, the LLM response from the first round... <unused2>Tool usage: QuestionAnswering(searchQuery="Are the maple leaves red at Qixia Mountain?") <unused3>It will be added to the prompt in round 2.

[0186] The LLM output response serves as the dialogue history for the next round. For example, the response from the first round, "Xiao X: The maple leaves at Qixia Mountain are already red. Qixia Mountain is one of the four major maple viewing spots in China, attracting many tourists every year. The best viewing period is from early to mid-December, when the maple leaves on Qixia Mountain are a deep, vibrant red, and stunningly beautiful," can be added to the dialogue history for the second round.

[0187] As can be seen, there is a large amount of identical content among the original prompts generated in different rounds of dialogue (identical content is indicated by underlines):

[0188] Between the second round and the first round:

[0189] The system prompts are the same; the first tool in the candidate tools is the same; the first question in the history dialogues is the same.

[0190] The LLM Response in the first round is identical to the information used to invoke the Question Answering (QA) tool in the historical dialogue fragment of the second round.

[0191] Between the third round and previous rounds (the first and second rounds):

[0192] Similar to the first round: the system prompt is the same; the first candidate tool is the same; the LLM response in the first round is the same as the QA content in the historical dialogue fragments of the third round.

[0193] Similar to the second round: the system prompt is the same; the first and second candidate tools are the same; the LLM response in the second round is the same as the Navigate content in the historical dialogue fragments of the third round.

[0194] To facilitate understanding of the KV Cache data reuse method proposed in the embodiments of this application, the following is based on... Figure 12 The multi-turn dialogue scenario shown has Figure 4 The electronic device 100 with the structure shown is or Figure 5 Taking the server 200 with the structure shown as an example, combined with Figure 6 The application scenarios shown and Figure 7 The system architecture shown is illustrated with a specific embodiment to exemplify the KV Cache data reuse method provided in this embodiment of the application.

[0195] like Figure 13 As shown, in this embodiment, the KV Cache data reuse method may include the following process:

[0196] S1001: Electronic device receives user input.

[0197] In this embodiment, the electronic device may be Figure 5 The server 200 is shown. In other embodiments, the electronic device may be... Figure 4 The electronic device 100 shown or other electronic devices.

[0198] For example, in the first round of question and answer, the user enters the question (Query) "Are the maple leaves on Qixia Mountain red yet?" The electronic device receives the question and, in response to the user's input, performs the following subsequent steps.

[0199] S1002: The electronic device, based on the preset templated prompt rules, segments the original prompt to be segmented through the prompt project to obtain multiple segments.

[0200] The original prompt to be segmented is segmented, including dividing it into multiple parts, and then each of at least one of the multiple parts is segmented into multiple blocks.

[0201] Keyword searches can be used to identify different parts or different blocks within the same part. Keywords can be specified text with natural language meaning or identifiers without natural language meaning.

[0202] Different parts of a prompt often begin or end with identifying keywords. For example, system prompts typically begin with the keyword "system," historical conversations usually begin with "[Historical Conversations]," and candidate tools usually begin with "[Candidate Tools]." These keywords can be used to identify the type of prompt. Alternatively, different parts or blocks may have corresponding identifiers, which can also be used to identify them. For example, searching... <eot> <s>Identifiers such as these can be used to identify different parts or different blocks.

[0203] like Figure 13 As shown, in the third round of question and answer, the original prompt to be segmented, generated based on the user's question, is as follows:

[0204] System: Your name is Xiao X, and you are a smart assistant developed by Huawei. Based on the information in the [Historical Conversations] and [Candidate Tools], please select the appropriate tool, match the corresponding parameters, and output the tool invocation command. If multiple candidate tools are matched, ask the user for more information to select the unique tool. Consider the content in the user's [Specific Context], which can help understand the user's true intention when expressing a specific command.

[0205] [Candidate Tools]:

[0206] [Tool] Name: NA. Description: Tools that cannot be matched above are categorized into this class. Parameter Definition: No parameters.

[0207] [Tool] Name: Navigate. Description: Activates navigation, guiding the user to their desired destination. Parameter Definition: desLocation: Destination, required parameter.

[0208] [Historical Dialogue]: <eot> <s>User: Are the maple leaves red at Qixia Mountain yet? <eot> <s>Little X: <unused2>Tool usage: QuestionAnswering(searchQuery="Are the maple leaves red at Qixia Mountain?"1) <unused3> <eot> <s>Little X: The maple leaves on Qixia Mountain have turned red. Qixia Mountain is one of China's four major maple viewing spots, attracting numerous tourists every year. The best viewing period is from early to mid-December, when the maple leaves on Qixia Mountain turn a deep, vibrant red, creating a stunningly beautiful scene. <eot> <s>User: Navigate to this <eot> <s>Little X: <unused2>Tool call: Navigate(desLocation="Qixia Mountain"]) <unused3> <eot> <s>Xiao X: Okay. <eot> <s>User: It's so congested, I'll just take the subway. <eot> <s>Little X:

[0209] Using this prompt as an example, let's explain how to segment it:

[0210] First, divide it into several parts:

[0211] Specifically, for the user input section, the original prompt is scanned from back to front. When the keyword "user:" is encountered, the content following "user:" is identified as the most recent question raised by the user in this round, and this question is assigned to the user input section. For example, in the 3rd round of the prompt, "User: It's too congested, I'd better take the subway" is assigned to the user input section. <eot> <s>Xiao X: "Divided into user input section."

[0212] Continuing to scan backwards, when the keyword "[Historical Conversations]" is encountered, the portion between "[Historical Conversations]" and the last occurrence of the keyword "user" in the prompt is identified as historical conversations and included as part of the user's dynamic prompts. For example, in the third round of the prompt, "[Historical Conversations]:" <eot> <s>User: Are the maple leaves red at Qixia Mountain yet? <eot> <s>Little X:

[0213] <unused2>Tool usage: QuestionAnswering(searchQuery="Are the maple leaves red at Qixia Mountain?"1) <unused3> <eot> <s>Little X: The maple leaves on Qixia Mountain have turned red. Qixia Mountain is one of China's four major maple viewing spots, attracting numerous tourists every year. The best viewing period is from early to mid-December, when the maple leaves on Qixia Mountain turn a deep, vibrant red, creating a stunningly beautiful scene. <eot> <s>User: Navigate to this <eot> <s>Little X: <unused2>Tool call: Navigate(desLocation="Qixia Mountain"]) <unused3> <eot> <s>Xiao X: Okay. (This is divided into user prompts.) Figure 8 The template shown can be used to create a user-generated prompt section (or a similar template). Figure 10 (The template shown).

[0214] Similarly, candidate tools represent the tool information that needs to be called from the candidate tool list in this round of question-and-answer task. By searching for the keyword "[Candidate Tools]", the prompt information of the candidate tool content can be identified and categorized into the user dynamic prompt section. For example, in the 3rd round prompt, "[Candidate Tools]: [Tool] Name: NA. Description: All tools that cannot be matched above are classified into this category. Parameter definition: No parameters. [Tool] Name: Navigate. Description: Activates the navigation function to guide the user on how to reach the desired destination. Parameter definition: desLocation: Destination, required parameter." should be categorized into the user prompt section or the user dynamic prompt section.

[0215] For system prompts, you can scan from front to back and identify the part between the keyword "system" and "[candidate tools]" as the system prompt part.

[0216] It should be noted that the keyword search-based segmentation method (dividing into parts or chunks) is only one example. Other segmentation methods can also be used, or other methods can be combined with keyword search. For example, a natural language processing model with semantic recognition or semantic classification capabilities can be used to segment the original prompt into multiple parts, and / or to segment a part into multiple chunks. Alternatively, in another embodiment, recursive character text splitting or character splitting can be used to achieve the segmentation of parts or chunks.

[0217] In this way, multiple parts can be divided.

[0218] After obtaining multiple parts, the parts that need to be segmented are further segmented into multiple blocks. For example, using a predetermined segmentation strategy, the historical dialogue part of the user dynamic prompt section in the above example is segmented, resulting in:

[0219] First block: "[Historical Dialogue]:" <eot> <s>User: "Are the maple leaves red at Qixia Mountain yet?"

[0220] The second block: <eot> <s>Little X: <unused2>Tool usage: QuestionAnswering(searchQuery="Are the maple leaves red at Qixia Mountain?"1) <unused3> <eot> <s>Little X: The maple leaves on Qixia Mountain have turned red. Qixia Mountain is one of China's four major maple viewing spots, attracting numerous tourists every year. The best viewing period is from early to mid-December, when the maple leaves on Qixia Mountain are a deep, vibrant red, breathtakingly beautiful.

[0221] The third block: <eot> <s>User: Navigate to this location;

[0222] The fourth block: <eot> <s>Little X: <unused2>Tool call: Navigate(desLocation="Qixia Mountain"]) <unused3> <eot> <s>Xiao X: Okay.

[0223] For example, specifically, in this embodiment, the user prompt section can be segmented using keyword search. For the historical dialogue in the user prompt section, the portion between two adjacent different keywords in the historical dialogue can be identified as a block by searching for keywords representing user questions, such as "user," and keywords representing model answers, such as "Xiao X."

[0224] For example, in the above example, the history dialogue section in the user dynamic prompt section reads: "[History Dialogue]: <eot> <s>User: Are the maple leaves red at Qixia Mountain yet? <eot> <s>Little X: <unused2>Tool usage: QuestionAnswering(searchQuery="Are the maple leaves red at Qixia Mountain?"1) <unused3> <eot> <s>Little X: The maple leaves on Qixia Mountain have turned red. Qixia Mountain is one of China's four major maple viewing spots, attracting numerous tourists every year. The best viewing period is from early to mid-December, when the maple leaves on Qixia Mountain turn a deep, vibrant red, creating a stunningly beautiful scene. <eot> <s>User: Navigate to this <eot> <s>Little X: <unused2>Tool call: Navigate(desLocation="Qixia Mountain"]) <unused3> <eot> <s>Little X: Okay. (Segmentation follows)

[0225] The scanning proceeds sequentially from beginning to end. Upon detecting the first "user" and the first "X" keyword, the content between these two keywords is segmented into the first block. The content between the first "X" and the second "user" is segmented into the second block. The content between the second "user" and the next "X" is segmented into the third block, and so on. The resulting segmentation is as follows:

[0226] First block: "[Historical Dialogue]:" <eot> <s>User: "Are the maple leaves red at Qixia Mountain yet?"

[0227] The second block: <eot> <s>Little X: <unused2>Tool usage: QuestionAnswering(searchQuery="Are the maple leaves red at Qixia Mountain?"1) <unused3> <eot> <s>Little X: The maple leaves on Qixia Mountain have turned red. Qixia Mountain is one of China's four major maple viewing spots, attracting numerous tourists every year. The best viewing period is from early to mid-December, when the maple leaves on Qixia Mountain are a deep, vibrant red, breathtakingly beautiful.

[0228] The third block: <eot> <s>User: Navigate to this location;

[0229] The fourth block: <eot> <s>Little X: <unused2>Tool call: Navigate(desLocation="Qixia Mountain"]) <unused3> <eot> <s>Xiao X: Okay.

[0230] If the user prompt section includes a static user prompt section, the content of the static user prompt section remains unchanged for the same user and can be segmented according to the maximum length limit of the static user prompt section and the maximum length limit of each block.

[0231] The system prompt section can be divided into multiple blocks or not. In practical applications, system prompts can be fixed or variable. For system prompts that may change, segmentation is necessary. For system prompts that remain unchanged, segmentation is optional.

[0232] In this embodiment, the system prompts are segmented because the basic unit for KV Cache reuse is a block. Segmenting the system prompts into blocks eliminates the need to set up a separate set of reuse logic for the system prompts, and is compatible with the block-based reuse mechanism proposed in this application.

[0233] For the segmentation of the system prompt section, adaptive segmentation can also be performed based on the maximum length limit of the system prompt section and the maximum length limit of each block.

[0234] Thus, in S1002, the original prompt is divided according to the templated prompt rules to obtain multiple prompt blocks, such as dividing it into Z blocks from Block 1 to Block Z, where Block 1 is the system prompt.

[0235] S1003 queries whether each Prompt block in the multiple split blocks matches the prompt block in the KVCache Store.

[0236] The rule for block-based key-value cache logical matching is as follows: compare each prompt block in the multiple segments with the prompt blocks in the key-value cache store, and determine whether each prompt block matches a block in the key-value cache store. A match means that the key-value data corresponding to the prompt block is already stored in the key-value cache store. The key-value cache store is the key-value data cache storage area.

[0237] Specifically, such as Figure 13 As shown, the KV Cache store records the block ID of each block. A hit can be when the block ID of one of the multiple blocks split into blocks is the same as the block ID of a block in the KV Cache store.

[0238] For example, the block ID can be a hash value obtained by hashing the block name (cellname) described above; for example, it can be a 65-bit hash value. A block ID is used to uniquely identify a block, and different blocks have different block IDs.

[0239] S1004: For the first block 1, if a hit occurs, the KV Cache data corresponding to Block 1 is directly moved from the KV Cache Store.

[0240] In this embodiment, the first block, Block 1, represents the first block in the system prompt.

[0241] S1005: For the first block 1, if a miss occurs, calculate the KV data corresponding to block 1.

[0242] The first block, block 1, represents the first block in the system prompt section. If it is not hit, it means that this block has not been calculated before. This round is the first round of multi-round question and answer, and the first round of calculation needs to be performed. Through the calculation, the key-value data of block 1 can be obtained directly.

[0243] S1006: For any one of the blocks from the second block 2 to the Zth block, if a hit occurs, then move the corresponding part of the KV Cache data (the first KV Cache data) from the KV CacheStore and perform the high-bias KV recalculation corresponding to this block.

[0244] The KV Cache data corresponding to a Block can be KV data calculated by the attention of multiple tokens within a block relative to the attention of the multiple tokens within the block themselves in the attention calculation mechanism.

[0245] If the hit occurs, it means that the same block content has been calculated in previous rounds, but due to changes in the context of this round's task, some of the KV data calculated in previous rounds cannot be directly transferred.

[0246] The computation for a block can include an attention calculation relative to itself and an attention calculation relative to the preceding context. Key-value data calculated from the attention of the block relative to itself in previous rounds is unaffected by the preceding context and can be directly transferred in the current round. However, key-value data calculated from the attention of the block in previous rounds cannot be directly transferred in the current round due to changes in the preceding context; therefore, the attention of each block relative to the preceding context needs to be recalculated. Thus, high-bias key-value recalculation can be the recalculation of the attention of the block relative to the preceding context, resulting in the second key-value cache data.

[0247] S1007: For blocks 2 to Z, if a match is not found, calculate the KV data for each block from 2 to Z to obtain the corresponding KV data.

[0248] A miss means that the content in this block has not been calculated in previous rounds, so it needs to be calculated for the first time. The calculation in S1007 includes the attention calculation of the block relative to itself and the attention calculation of the block equivalent to the above.

[0249] S1008: After obtaining the KV data of each block, the KV data is concatenated and cached in the KVCache store.

[0250] The key-value data corresponding to the hit and miss blocks are concatenated to obtain the complete key-value cache. During concatenation, the blocks can be concatenated in the order of their position IDs in the prompt.

[0251] It should be noted that the block ID is used to uniquely identify a block, while the position ID is used to indicate the relative position of a block in the prompt.

[0252] S1009: For parts that are not segmented into prompt blocks (such as user input), perform calculations, generate the final prefill result, and return it.

[0253] S1009 is equivalent to performing the prefill process for other calculations.

[0254] As can be seen from the above process, for blocks that are the same as the prompts in previous rounds of question answering, it is only necessary to load the corresponding KV Cache information from the KVCache store, thereby avoiding all repeated calculations and improving the inference speed of prefill.

[0255] It should be noted that in this embodiment, the explanation is based on the example of Block 1 being the first block in the system prompt section. In other embodiments, when the system prompt remains unchanged, the system prompt may not be segmented; instead, the system prompt section can directly reuse the KV Cache data. Figure 13 If the first block shown is not the first block in the system prompt, but is the first block in the user prompt section, the steps for the first blocks Block 1 to Block Z are the same. That is, if Block 1 is the first block in the user prompt section, steps such as S1006 and S1007 can be executed.

[0256] The following is a detailed explanation of the high-bias KV recalculation in S1006.

[0257] like Figure 14 As shown, KV Cache 1 to KV Cache 3 (abbreviated as KV1 to KV3) are generated by corresponding calculations from block1 to block3.

[0258] The templated prompt proposed in this application can guarantee that blocks 1 to 3 have the same position ID when their content is the same as the blocks in other prompts, but it cannot guarantee that they have the same preceding text. For example, the preceding text of block 2-1 can be block 1-0 or block 1-1.

[0259] like Figure 14 As shown on the right side, the KV1 calculated by block1 relative to block1, and the KV2 calculated by block2 relative to block2, are valid KV data calculated based on self-attention. However, the KV-KV2 data calculated by block2 relative to the preceding text, such as the attention between block2 and block1, cannot be directly reused in this round of the task because the block in block1 above may have changed from block1-0 to block1-1. In other words, the KV1-KV2 data in the previous round was calculated using block2 relative to block1-0, but in this round of question answering, the preceding text has become block1-1, and the attention information between block2 and block1-1 is missing. Figure 14 In this context, "cross attention" represents the attention given to block2 relative to block1.

[0260] In some embodiments, one possible solution is to perform full computation on cross attention, which is equivalent to a computational complexity of O(L1*L2), where L1 and L2 are the lengths of block1 and block2, respectively, which can be understood as the number of tokens.

[0261] It's important to note that while recalculation is necessary, the block-based reuse mechanism allows for the reuse of a portion of the KV Cache data calculated relative to the attention of each block, achieving at least partial data reuse and reducing computational load. In contrast, related technologies require recalculation of all remaining prompt content except for system hints, making partial data reuse impossible.

[0262] In other embodiments, this application also proposes a high-bias recalculation mechanism, which does not require recalculating attention for all tokens in a block, but instead selects to recalculate a portion of tokens with significant differences at specific positions in cross attention. This is equivalent to selecting Q tokens with high bias from L1 tokens for recalculation, reducing the algorithm complexity to O(Q*L1), where Q is the number of tokens recalculated, Q < L1, and L1 is the length of block1.

[0263] like Figure 15 As shown, Figure 15 This diagram illustrates high-bias key-value recomputation. In the network layer specified by the large language model (e.g., the first network layer), all key / value pairs are recomputed to obtain the first set of key-value cache data. This data is then compared with the key / value data stored in the key-value cache store (the second set of key-value cache data). The bias of the two key / value pairs is calculated, and the Q tokens with the largest biases are recorded—that is, the K tokens whose biases exceed a predetermined threshold, where K is a hyperparameter. The recomputed key / value pairs replace the previously stored reused key / value pairs in the first network layer. In subsequent network layers, the key / value pairs at the top Q corresponding positions are all recomputed, replacing the previously stored reused key / value pairs.

[0264] Based on the above exemplary description, it can be seen that the method proposed in this application divides the prompt into blocks according to a given prompt template, performs matching based on the prompt blocks, and recomputes specific high-biased KVs to achieve KV cache reuse, thereby improving the prefill inference speed.

[0265] Key-value (KV) cache reuse schemes in related technologies cannot reuse content outside of system prompts because they cannot address the issues of different Position IDs and different preceding contexts. Different Position IDs mean the content's spatial location within the entire prompt is different; different preceding contexts mean the content preceded by that prompt is different. KV cache calculation is dependent on both position ID and context, which means that directly reusing the KV cache when prompt content is the same can lead to precision issues. The system prompt can be reused because it is at the beginning of the entire prompt and has no preceding context; that is, in different input rounds, the system prompt has the same position ID and preceding context.

[0266] To address the aforementioned issues, this application proposes a templated prompt and a high-bias recomputation mechanism. The complete, original prompt is segmented according to predefined prompt template rules. Each segmented prompt block is matched against a stored KV cache store. Based on the result, a decision is made whether to directly prefetch the KV cache or recompile the prompt block. If the prompt block is compared with a block ID in the KV cache store, and a match is found, it is prefetched directly from the KV cache store; otherwise, the prompt block is recomputed. Furthermore, if the matched prompt is not a system prompt, after prefetching, the high-bias KV cache needs to be recomputed. Finally, all prompt blocks are concatenated with their corresponding KV caches (caching prefetching and recomputation) to obtain the complete KV cache. Further calculations are then performed on the user input portion of the prompt to obtain the final prefill result.

[0267] In multi-turn question answering, the prompt blocks may be identical, but their position IDs may differ, meaning the spatial location of this content within the entire prompt varies, resulting in different KV caches for that prompt. In the method proposed in this application, the prompt is divided according to type, and the position ID of each part remains fixed across multiple rounds of question answering. For prompt blocks whose actual length does not reach the maxLen length, spaces or other specified information can be directly padded to ensure a fixed data length for the block. Each block has a fixed position ID that does not change with variations in the block content. During inference, matching and reuse are performed based on blocks or cells as the basic unit. Thus, in multi-turn question answering, as long as the prompt block content is identical, the spatial location within the two blocks is guaranteed to be the same, avoiding the impact of spatial location on the KV cache.

[0268] In multi-turn question-and-answer sessions, when block content is the same, the preceding context may differ. This difference in preceding context results in different cross-attention values ​​during the calculation of the prompt compared to previous prompts, which affects the KV cache content of the next block. To address this, this application proposes recalculating all Key / Value pairs in the first network layer and comparing them with stored reused Key / Value pairs. The bias between the two Key / Value pairs is calculated, and the top S tokens with the largest biases are recorded. These S tokens are then recalculated. The recalculated Key / Value pairs replace the stored reused Key / Value pairs in the first network layer. In subsequent network layers, the Key / Value pairs corresponding to the top S tokens are all recalculated, replacing the stored reused Key / Value pairs. In this way, important cross-attention content can be kept consistent with minimal computational cost, ensuring that the KV cache content of subsequent blocks is as consistent as possible.

[0269] Table 1

[0270]

[0271] As shown in Table 1 above, Table 1 provides a delay prediction for a large model under different input lengths.

[0272] Wherein, seqlen represents the length of the input sequence, and 1K represents 1024 English characters or 1000 tokens. Non-Cache Prefill latency (ms) represents the latency in the prefill stage (first latency) when the KV Cache data reuse method proposed in this application is not used. KV load latency (ms) represents the latency in the prefill stage when the method proposed in this application is used (second latency). Cache gain represents the gain of the second latency relative to the first latency. KVSize (GB) represents the data length of KV data that needs to be stored in one round of question answering, or the size of the storage space occupied by the KV data; GQA indicates that the large model uses the Grouped-Query Attention (GQA) mechanism. GQA=4 means that 4 heads share the same set of KV Cache data, and GQA=1 means that 1 head uses a separate set of KV Cache data. Int8 represents an 8-bit signed integer.

[0273] As shown in Table 1, latency gains of 46% to 82% can be achieved under different configurations. Furthermore, this method does not affect the inference accuracy of the prefill; the solution in this application significantly improves the inference speed or efficiency of the prefill while maintaining prefill accuracy.

[0274] This application also provides a KV Cache data reuse device, comprising: a generation module, configured to generate a first prompt corresponding to the current round of question-and-answer in response to a first question (Query) entered by a user in the current round of question-and-answer; the first prompt includes multiple parts; the multiple parts include a system prompt part, a user prompt part, and a user input part arranged in a fixed order. Each of the multiple parts has a fixed position identifier (ID) in the first prompt; at least one of the multiple parts includes multiple segments, and the position ID of each segment in the first prompt is fixed; the system prompt part includes prompts common to multiple rounds of question-and-answer; the user prompt part includes at least the user's corresponding historical question-and-answer records, and / or tool information to be invoked in the current round of question-and-answer; and the user input part includes the first question.

[0275] The query module is used to input the first prompt into the large language model. During the prefill phase, it queries whether the current block among multiple blocks hits a block in the memory. The memory is used to store KVCache data corresponding to at least one block.

[0276] The read module is used to read the KV Cache data corresponding to the current block from memory when a hit occurs.

[0277] The generation, query, and reading modules can all be implemented in software or hardware. For example, the implementation of the reading module will be described below. Similarly, the implementation of the query and reading modules can be referenced from that of the generation module.

[0278] As an example of a software functional unit, a module can include code running on a compute instance. A compute instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the compute instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0279] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0280] As an example of a hardware functional unit, a module can include at least one computing device, such as a server. Alternatively, a module can also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0281] The multiple computing devices included in the generation module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the generation module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the generation module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0282] It should be noted that, in other embodiments, the generation module can be used to execute a KV Cache data reuse method (e.g., ...). Figures 8 to 15 In any step of the embodiment corresponding to any figure in the above exemplary description or any of the methods in the above exemplary description, the query module can be used to execute the KV Cache data reuse method (e.g., as shown in the figure). Figures 8 to 15 In any step of the embodiment corresponding to any figure in the above exemplary description, or any embodiment of the method, the reading module can be used to execute the KV Cache data reuse method (e.g., as shown in the figure). Figures 8 to 15 Any step in the method of any embodiment corresponding to any figure or any embodiment in the above exemplary description, the steps implemented by the generation module, query module and reading module can be specified as needed, and the generation module, query module and reading module respectively implement different steps in the KV Cache data reuse method to realize all the functions of the KV Cache data reuse device.

[0283] As an example of a software functional unit, a KV Cache data reuse device may include code running on a compute instance. This compute instance can be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Furthermore, the aforementioned computing device may be one or more. For example, the KV Cache data reuse device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed within the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed within the same Availability Zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0284] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0285] As an example of a hardware functional unit, a KV Cache data multiplexing device may include at least one computing device, such as a server. Alternatively, the KV Cache data multiplexing device may also be a device implemented using an ASIC or a PLD. The aforementioned PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0286] The KV Cache data multiplexing device includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the multiple computing devices can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0287] This application also provides a computing device 400. For example... Figure 16 As shown, the computing device 400 includes a bus 402, a processor 404, a memory 406, and a communication interface 408. The processor 404, the memory 406, and the communication interface 408 communicate with each other via the bus 402. The computing device 400 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 400.

[0288] Bus 402 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus 402 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 402 may include a path for transmitting information between various components of the computing device 400 (e.g., memory 406, processor 404, communication interface 408).

[0289] Processor 404 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0290] Memory 406 may include volatile memory, such as random access memory (RAM). Processor 404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0291] The memory 406 stores executable program code, and the processor 404 executes this executable program code to implement the functions of the aforementioned generation module, query module, and read module, thereby realizing the KV Cache data reuse method described above. That is, the memory 406 stores code for executing the above-described... Figures 8 to 15 Instructions for the KV Cache data reuse method of any embodiment corresponding to any of the figures in the above exemplary description.

[0292] The communication interface 408 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 400 and other devices or communication networks.

[0293] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0294] like Figure 17 As shown, the computing device cluster includes at least one computing device 400. The memory 406 of one or more computing devices 400 in the computing device cluster may store the same memory for performing tasks such as... Figure 18 The instructions for the KV Cache data reuse method are shown.

[0295] In some possible implementations, the memory 406 of one or more computing devices 400 in the computing device cluster may also store partial instructions for executing the KV Cache data multiplexing method described above. In other words, a combination of one or more computing devices 400 can jointly execute instructions for executing the KV Cache data multiplexing method described above.

[0296] It should be noted that the memory 406 in different computing devices 400 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the KV Cache data multiplexing device. That is, the instructions stored in the memory 406 of different computing devices 400 can implement the functions of one or more modules among the generation module, query module, and read module.

[0297] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 18 One possible implementation is shown. For example... Figure 18 As shown, the two computing devices 400A and 400B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 406 in computing device 400A stores instructions for executing the functions of the generation module. Simultaneously, the memory 406 in computing device 400B stores instructions for executing the functions of the query module and the read module.

[0298] Figure 18 The connection method between the computing device clusters shown can be implemented by delegating the functions of the query module and the read module to computing device 400B, considering that the KV Cache data reuse method provided in this application requires a large amount of data storage and computation. For example, the query module can input the first prompt into the large language model, and during the prefill stage, query whether the current block among multiple blocks hits a block in memory. The read module can read the KV Cache data corresponding to the current block from memory if a hit occurs.

[0299] It should be understood that Figure 18 The functions of the computing device 400A shown can also be performed by multiple computing devices 400. Similarly, the functions of the computing device 400B can also be performed by multiple computing devices 400.

[0300] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 17 and Figure 18 The connection method of the computing device cluster is different in that the memory 406 in one or more computing devices 400 in the computing device cluster can store the same instructions for executing the KV Cache data reuse method in some embodiments of the above embodiments.

[0301] In some possible implementations, the memory 406 of one or more computing devices 400 in the computing device cluster may also store partial instructions for executing the KV Cache data multiplexing method in a portion of the above embodiments. In other words, a combination of one or more computing devices 400 can jointly execute instructions for executing the KV Cache data multiplexing method in a portion of the above embodiments.

[0302] It should be noted that the memory 406 in different computing devices 400 within the computing device cluster can store different instructions to execute some functions of the KV Cache data multiplexing device. That is, the instructions stored in the memory 406 of different computing devices 400 can implement the functions of the KV Cache data multiplexing device.

[0303] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to perform the above-described KVCache data multiplexing method.

[0304] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-described KV Cache data multiplexing method, or instruct the computing device to execute the KV Cache data multiplexing method.

[0305] This application also provides an electronic device, the electronic device comprising: a processor, the processor being configured to execute a computer program or instructions in a memory to implement the method as described in any of the above embodiments.

[0306] It should be noted that a processor can be a chip with computing capabilities, and is not limited to a central processing unit (CPU). For example, a processor can be a chip that includes one or more transistors, resistors, capacitors, and other circuit elements to perform a certain function; or it can be an integrated circuit in various packages that can implement the above methods.

[0307] This application also provides a chip system, including: a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the KV Cache data multiplexing method as described in any of the above embodiments.

[0308] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.< / s> < / eot> < / unused3> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / unused3> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / unused3> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / unused3> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / unused3> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / unused3> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / unused3> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / unused3> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / unused3> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / unused3> < / s> < / eot> < / s> < / eot> < / s> < / eot> < / unused1> < / unused1> < / unusedo> < / unused1> < / unused1> < / unusedo> < / unused1> < / unused1> < / unused3> < / unused1> < / unused1> < / unusedo> < / unused1> < / unused1> < / unused3> < / unused1> < / s> < / eot> < / s>

Claims

1. A method for reusing KV cache data, characterized in that, The method includes: In response to the first question (Query) entered by the user in this round of question and answer, a first prompt corresponding to this round of question and answer is generated; the first prompt includes multiple parts; the multiple parts include a system prompt part, a user prompt part, and a user input part arranged in a fixed order; Wherein, the position ID of each of the plurality of parts in the first prompt remains fixed; at least one of the plurality of parts includes multiple segments, and the position ID of each of the multiple segments in the first prompt remains fixed; the system prompt part includes prompts applicable to multiple rounds of question and answer; the user prompt part includes at least the user's historical question and answer records, and / or tool information to be invoked in the current round of question and answer; the user input part includes the first question; The first prompt is input into the large language model. During the prefill stage, it is queried whether the current block among the multiple blocks hits a block in the memory. The memory is used to store KV Cache data corresponding to at least one block. If a hit occurs, the KV Cache data corresponding to the current block is read from the memory.

2. The method as described in claim 1, characterized in that, The user prompt section includes a dynamic user prompt section and / or a static user prompt section; The position ID of the user dynamic prompt and / or the user static prompt in the first prompt remains fixed. The user dynamic prompt section includes the user's corresponding historical question and answer records, and / or the tool information to be called to execute this round of tasks; The static user prompt section includes the user's profile information.

3. The method as described in claim 2, characterized in that, The fixed order is that the system prompts, the static user prompts, the dynamic user prompts, and the user input are arranged from front to back.

4. The method according to any one of claims 1-3, characterized in that, Each of the plurality of blocks has an independent block code ID, position ID, and maximum length; wherein, the block ID is used to uniquely identify a block; the maximum length of a block is fixed.

5. The method as described in claim 4, characterized in that, The method further includes: If the data length of the content within a block in the first prompt is less than the maximum length, a specified character is used to fill it; after filling, the data length occupied by the content within the block reaches the maximum length.

6. The method according to any one of claims 1-5, characterized in that, The KV Cache data corresponding to the current block includes first KV Cache data and second KV Cache data; the first KV Cache data is obtained by calculating the attention of the current block relative to the previous block; the second KV Cache data is obtained by calculating the attention of the current block relative to the preceding text. If the content of the system prompt section changes, the system prompt section includes multiple segments. Reading the KV Cache data corresponding to the current block from the memory includes: Based on the location ID of the current block, it is determined that the current block is not the first block in the system prompt section, and the first KV Cache data corresponding to the current block is read from the memory; or, If the content in the system prompt section remains unchanged, Reading the KV Cache data corresponding to the current block from the memory includes: Based on the location ID of the current block, it is determined that the current block is a block in the user prompt section, and the first KV Cache data corresponding to the current block is read from the memory; the first KV Cache data is data obtained by calculating the attention of the current block relative to the current block.

7. The method as described in claim 6, characterized in that, The method further includes: If the current block is determined not to be the first block in the system prompt section, or if the current block is determined to be a block in the user prompt section, then the recalculation corresponding to the current block is performed.

8. The method as described in claim 7, characterized in that, Before performing the recalculation corresponding to the current block, the method further includes: From the multiple element tokens corresponding to the current block, at least one high-bias token is determined; the high-bias token is the token whose bias between the KV Cache data recalculated in this round of question answering and the KV Cache data calculated in the previous round of question answering meets a predetermined condition. Performing the recalculation corresponding to the current block includes: Perform recompute on at least one high-biased token in the current block.

9. The method as described in claim 8, characterized in that, From the multiple tokens corresponding to the current block, at least one high-bias token is determined, including: In a specified network layer of the large language model, attention calculation is performed for the current block to obtain the first set of key-value data; the first set of key-value data includes key-value data corresponding to multiple tokens. The KV Cache data corresponding to the current block calculated in the previous round of question and answer is read from the memory and used as the second set of KV data; the second set of KV data includes KV data corresponding to multiple tokens; Calculate the bias between the KV data corresponding to the same token in the first group of KV Cache data and the second group of KV Cache data; From the multiple tokens corresponding to the current block, select the tokens whose bias meets the predetermined conditions and use them as high-bias tokens.

10. The method as described in claim 8 or 9, characterized in that, After performing recomputation on at least one high-biased token in the current block, the method further includes: In the subsequent network layers of the specified network layer in the large language model, the attention corresponding to the at least one high-biased token is recomputed; Write the key-value data corresponding to at least one high-biased token recalculated by the specified network layer and the subsequent network layer in this round of question and answer into the memory, replacing the key-value data stored in the memory in the previous round of question and answer.

11. The method according to any one of claims 1-10, characterized in that, The system prompt section includes multiple segments after being split; If the content in the system prompt section remains unchanged, Reading the KV Cache data corresponding to the current block from the memory includes: Based on the location ID of the current block, determine that the current block is a block in the system prompt section, and read all the KV Cache data corresponding to the current block from the memory; or, If the content in the system prompt section changes... Reading the KV Cache data corresponding to the current block from the memory includes: Based on the location ID of the current block, determine that the current block is the first block in the system prompt section, and read all the KV Cache data corresponding to the current block from the memory.

12. A KV Cache data multiplexing device, characterized in that, The device includes: A generation module is used to generate a first prompt corresponding to the current round of question and answer in response to the first question Query entered by the user in the current round of question and answer; the first prompt includes multiple parts; the multiple parts include a system prompt part, a user prompt part, and a user input part arranged in a fixed order; Wherein, the position ID of each of the plurality of parts in the first prompt remains fixed; at least one of the plurality of parts includes multiple segments, and the position ID of each of the multiple segments in the first prompt remains fixed; the system prompt part includes prompts applicable to multiple rounds of question and answer; the user prompt part includes at least the user's historical question and answer records, and / or tool information to be invoked in the current round of question and answer; the user input part includes the first question; The query module is used to input the first prompt into the large language model, and during the prefill stage, to query whether the current block among the multiple blocks hits a block in the memory; the memory is used to store KV Cache data corresponding to at least one block; The read module is used to read the KVCache data corresponding to the current block from the memory when a hit occurs.

13. The apparatus as claimed in claim 12, characterized in that, The user prompt section includes a dynamic user prompt section and / or a static user prompt section; The position ID of the user dynamic prompt and / or the user static prompt in the first prompt remains fixed. The user dynamic prompt section includes the user's corresponding historical question and answer records, and / or the tool information to be called to execute this round of tasks; The static user prompt section includes the user's profile information.

14. The apparatus as claimed in claim 13, characterized in that, The fixed order is that the system prompts, the static user prompts, the dynamic user prompts, and the user input are arranged from front to back.

15. The apparatus as claimed in any one of claims 12-14, characterized in that, Each of the plurality of blocks has an independent block code ID, position ID, and maximum length; wherein, the block ID is used to uniquely identify a block; the maximum length of a block is fixed.

16. The apparatus as claimed in claim 15, characterized in that, The device further includes: The filling module is used to fill in specified characters when the data length of the content in a block of the first prompt is less than the maximum length; after filling, the data length occupied by the content in the block reaches the maximum length.

17. The apparatus as claimed in any one of claims 12-16, characterized in that, The KV Cache data corresponding to the current block includes first KV Cache data and second KV Cache data; the first KV Cache data is obtained by calculating the attention of the current block relative to the previous block; the second KV Cache data is obtained by calculating the attention of the current block relative to the preceding text. If the content of the system prompt section changes, the system prompt section includes multiple segments. When reading the KV Cache data corresponding to the current block from the memory, the reading module is specifically used for: Based on the location ID of the current block, it is determined that the current block is not the first block in the system prompt section, and the first KV Cache data corresponding to the current block is read from the memory; or, If the content in the system prompt section remains unchanged, When reading the KV Cache data corresponding to the current block from the memory, the reading module is specifically used for: Based on the location ID of the current block, determine that the current block is a block in the user prompt section, and read the first KV Cache data corresponding to the current block from the memory; The first KV Cache data is obtained by calculating the attention of the current block relative to the current block.

18. The apparatus as claimed in claim 17, characterized in that, The device further includes: The recalculation module is used to perform recalculation of the current block if it is determined that the current block is not the first block in the system prompt section, or if it is determined that the current block is a block in the user prompt section.

19. The apparatus as claimed in claim 18, characterized in that, Before performing the recalculation corresponding to the current block, the recalculation module is further configured to: From the multiple element tokens corresponding to the current block, at least one high-bias token is determined; the high-bias token is the token whose bias between the KV Cache data recalculated in this round of question answering and the KV Cache data calculated in the previous round of question answering meets a predetermined condition. When performing the recalculation corresponding to the current block, the recalculation module is specifically used for: Perform recompute on at least one high-biased token in the current block.

20. The apparatus as claimed in claim 19, characterized in that, From the multiple tokens corresponding to the current block, at least one high-bias token is determined, including: In a specified network layer of the large language model, attention calculation is performed for the current block to obtain the first set of key-value data; the first set of key-value data includes key-value data corresponding to multiple tokens. The KV Cache data corresponding to the current block calculated in the previous round of question and answer is read from the memory and used as the second set of KV data; the second set of KV data includes KV data corresponding to multiple tokens; Calculate the bias between the KV data corresponding to the same token in the first group of KV Cache data and the second group of KV Cache data; From the multiple tokens corresponding to the current block, select the tokens whose bias meets the predetermined conditions and use them as high-bias tokens.

21. The apparatus as claimed in claim 19 or 20, characterized in that, After performing recalculation for at least one high-biased token in the current block, the recalculation module is further configured to: In the subsequent network layers of the specified network layer in the large language model, the attention corresponding to the at least one high-biased token is recomputed; Write the key-value data corresponding to at least one high-biased token recalculated by the specified network layer and the subsequent network layer in this round of question and answer into the memory, replacing the key-value data stored in the memory in the previous round of question and answer.

22. The apparatus as claimed in any one of claims 12-21, characterized in that, The system prompt section includes multiple segments after being split; If the content in the system prompt section remains unchanged, When reading the KV Cache data corresponding to the current block from the memory, the reading module is specifically used for: Based on the location ID of the current block, determine that the current block is a block in the system prompt section, and read all the KV Cache data corresponding to the current block from the memory; or, If the content in the system prompt section changes... When reading the KV Cache data corresponding to the current block from the memory, the reading module is specifically used for: Based on the location ID of the current block, determine that the current block is the first block in the system prompt section, and read all the KV Cache data corresponding to the current block from the memory.

23. An electronic device, characterized in that, The electronic device includes: A processor for executing a computer program or instructions in memory to implement the method as described in any one of claims 1-11.

24. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-11.

25. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1-11.

26. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-11.

Images