A pre-filling method and device applied to model inference and electronic equipment

Abstract

The application provides a pre-filling method and device applied to model reasoning and electronic equipment, and is applied to the technical field of large language model reasoning. The method comprises the following steps: obtaining a reasoning context sequence of a current round reasoning request, wherein the reasoning context sequence comprises a prompt word, hidden reasoning content and generated output content; comparing the length of the hidden reasoning content with a first preset threshold value, and determining a retention strategy according to the comparison result; determining target key-value caches to be retained from key-value caches of a historical conversation according to the retention strategy; writing the target key-value caches into a storage system according to the retention range corresponding to the target key-value caches; and when a preset trigger condition is met, performing asynchronous pre-filling on a text sequence corresponding to a missing key-value cache, and writing the completed key-value cache into the storage system. Through threshold value judgment to dynamically adjust the cache retention range and combined with asynchronous pre-filling, the application can effectively reduce the first word generation delay of the next round of conversation.

Description

Technical Field

[0001] This invention relates to the field of large language model reasoning technology, and in particular to a pre-filling method, apparatus and electronic device for model reasoning. Background Technology

[0002] Currently, in multi-turn dialogue reasoning scenarios, before outputting the final answer, the model first generates a text sequence containing intermediate reasoning processes and places these intermediate reasoning processes in a predetermined starting label (e.g., "..."). <think>") and closing tags (e.g., "< / think> The system outputs the final answer between the start and end labels, and then outputs the final answer after the end label is closed. To save context window space, the system usually discards the intermediate reasoning process between the start and end labels and only retains the final answer as historical context.

[0003] However, this processing method alters the sequence of prompts for the next round compared to the complete sequence generated by the cache. Existing key-value caches cannot be reused, and the system must recalculate the key values ​​for all tokens of the previous round's prompts and the final answer during the pre-filling phase of the next dialogue. This significantly increases the delay in generating the first token of the next dialogue. The text of call parameters between tool call tag pairs and function call tag pairs also faces the same discarding process in subsequent rounds, leading to a similar key-value cache invalidation problem.

[0004] Therefore, in multi-turn dialogue scenarios, how to reduce the delay in generating the first lexical unit in the next round of dialogue, in order to address the problem of key-value cache invalidation caused by discarding intermediate inference content or calling parameter text, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention provides a pre-filling method, apparatus, and electronic device for model inference that overcomes or at least partially solves the above problems. The technical solution is as follows:

[0006] A pre-filling method for model inference, comprising:

[0007] Obtain the reasoning context sequence of the reasoning request in the current round, wherein the reasoning context sequence includes prompt words, hidden reasoning content and generated output content;

[0008] The length of the hidden inference content is compared with a first preset threshold, and the retention strategy of the hidden inference content in historical sessions is determined based on the comparison result.

[0009] Based on the retention policy, the target key-value cache to be retained is determined from the key-value cache of historical sessions;

[0010] Write the target key-value cache into the storage system according to the retention range corresponding to the target key-value cache;

[0011] When a preset trigger condition is met, asynchronous pre-filling is performed on the text sequence corresponding to the missing key-value cache in the storage system, and the pre-filled complete key-value cache is written into the storage system.

[0012] A pre-filling device for model inference includes: an inference context sequence acquisition unit, a retention policy determination unit, a target key-value cache determination unit, a target key-value cache writing unit, and a pre-filling unit;

[0013] The reasoning context sequence acquisition unit is used to acquire the reasoning context sequence of the reasoning request in the current round, wherein the reasoning context sequence includes prompt words, hidden reasoning content and generated output content;

[0014] The retention strategy determination unit is used to compare the length of the hidden inference content with a first preset threshold, and determine the retention strategy of the hidden inference content in the historical session based on the comparison result.

[0015] The target key-value cache determination unit is used to determine the target key-value cache to be retained from the key-value cache of historical sessions according to the retention policy;

[0016] The target key-value cache writing unit is used to write the target key-value cache into the storage system according to the retention range corresponding to the target key-value cache;

[0017] The pre-filling unit is used to perform asynchronous pre-filling on the text sequence corresponding to the missing key-value cache in the storage system when a preset trigger condition is met, and write the pre-filled completed key-value cache into the storage system.

[0018] An electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the pre-filling method applied to model inference.

[0019] By employing the above technical solution, this invention provides a pre-filling method, apparatus, and electronic device for model inference. By comparing the length of the hidden inference content with a first preset threshold, determining a retention strategy based on the comparison result, and writing the corresponding target key-value cache into the storage system according to the retention strategy, the hidden inference content is only discarded when a trigger condition is met. This overcomes the problem of frequent key-value cache failures caused by unconditionally discarding intermediate inference content in existing technologies. Furthermore, it maximizes the retention range of the key-value cache without losing effective contextual information, providing a data foundation for reducing the latency of the first word generation in subsequent rounds. Simultaneously, when the preset trigger condition is met, asynchronous pre-filling is performed on the text sequence corresponding to the missing key-value cache in the storage system, and the completed key-value cache is written into the storage system. This allows the historical key-value cache required for the next round of dialogue to be calculated in advance during the idle period of the current round, overcoming the latency problem caused by the forced synchronous re-execution of pre-filling in the next round of dialogue due to missing key-value cache. This achieves the technical effect of reducing the latency of the first word generation in the next round of dialogue and improving the response speed of the inference system.

[0020] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0021] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0022] Figure 1 A flowchart illustrating one embodiment of the pre-filling method for model inference provided by this invention is shown.

[0023] Figure 2 The diagram shows a specific implementation of step S140 in the pre-filling method for model reasoning provided in this embodiment of the invention.

[0024] Figure 3 This diagram illustrates the structure of a pre-filling device for model inference provided in an embodiment of the present invention.

[0025] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0026] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0027] The Large Language Model (LLM) inference process typically includes a prefill phase and a decoding phase. In the prefill phase, the model receives prompts from the user and uses a multi-head attention mechanism to perform parallel computation on each token within the prompts, generating the first output token and the key-value caches corresponding to each attention layer. In the subsequent decoding phase, the model predicts the next token word by word based on the generated tokens. When generating a new token, only the query vector of the current token needs to be calculated, and attention calculations are completed using the previously stored key-value caches. This eliminates the need to repeatedly perform key-value projection operations on historical tokens, significantly reducing the computational load of the word-by-word generation phase. This two-stage processing flow makes the Time to First Token (TTFT) a key factor affecting the user's perceived response latency, and also makes the reuse efficiency of the Key-Value Cache (KV Cache) a core factor affecting the throughput and computational cost of the inference system.

[0028] In large language model inference systems, there is a strict binding constraint between the key-value cache and the inference context sequence that generates it. Specifically, effective reuse of the key-value cache requires that the prompt word sequence of the current inference request be completely identical to the historical sequence corresponding to the cache at the lexical level. When a lexical change occurs at any position in the sequence, including insertion, deletion, or replacement, the key-value caches corresponding to all subsequent lexical terms become invalid due to positional encoding offsets and changes in attention distribution. Therefore, the reuse hit rate of the key-value cache is highly dependent on the precise matching degree of the inference context sequence.

[0029] In the Chain-of-Thought (CoT) reasoning model, before outputting the final answer, the model first generates a text sequence containing intermediate reasoning processes, typically placing these intermediate reasoning processes between specific label pairs, such as " <think> "and"< / think>"Between the label pairs, the final answer is output after the label is closed. In multi-turn dialogue scenarios, due to the limitation of the context window length, inference systems typically adopt a processing method that only retains the final answer as historical context and discards the intermediate reasoning process in order to effectively utilize the limited context space. The above processing method results in the prompt word sequence of the next round of dialogue consisting of the prompt words of the previous round and the final answer, rather than consisting of the prompt words of the previous round, the intermediate reasoning process, and the final answer together."

[0030] Because the prompt sequence for the next round of dialogue is inconsistent with the complete sequence used to generate the cache, the existing key-value cache cannot be directly reused. During the pre-filling phase of the next round of dialogue, the system must recalculate the key values ​​for all tokens of the prompts from the previous round and the final answer to build a new key-value cache. This recalculation process significantly increases the time required to generate the first token in the N+1th round of dialogue. Furthermore, in addition to thought chain reasoning, the text of call parameters between tool call tag pairs and function call tag pairs also faces discarding in subsequent rounds of dialogue, leading to a similar key-value cache invalidation problem.

[0031] Therefore, in multi-turn dialogue scenarios, how to reduce the delay in generating the first lexical unit in the next round of dialogue, in order to address the problem of key-value cache invalidation caused by discarding intermediate inference content or calling parameter text, is a technical problem that urgently needs to be solved in this field.

[0032] Based on this, this invention provides a pre-filling method for model inference. The method involves obtaining the inference context sequence of the current round of inference requests, which includes prompt words, hidden inference content, and generated output content. The length of the hidden inference content is compared with a first preset threshold, and a retention strategy is determined based on the comparison result. According to the retention strategy, a target key-value cache to be retained is determined from the key-value cache of historical sessions. The target key-value cache is written to the storage system according to its retention range. When a preset trigger condition is met, asynchronous pre-filling is performed on the text sequence corresponding to the missing key-value cache, and the completed key-value cache is written to the storage system. This invention, by dynamically adjusting the cache retention range through threshold judgment and combining it with asynchronous pre-filling, can effectively reduce the delay in generating the first lexical unit of the next round of dialogue.

[0033] like Figure 1 The diagram shows a flowchart of one embodiment of the pre-filling method for model inference provided by this invention. The method may include:

[0034] S100. Obtain the reasoning context sequence of the reasoning request in the current round, wherein the reasoning context sequence includes prompt words, hidden reasoning content and generated output content.

[0035] Among them, inference request refers to a single question-and-answer or task execution request initiated by the user to the large language model.

[0036] The inference context sequence refers to the complete lexical sequence corresponding to the inference request in the current round. It can include user-provided prompts, hidden inference content generated by the model, and generated output content generated by the model, serving as the complete context input for the pre-filling stage of the large language model.

[0037] The prompt refers to the text content that the user directly enters in the current inference request, which serves as the conditional input for the model to perform the inference task.

[0038] Hidden inference content refers to the intermediate text sequence generated by the model during the inference task that is not directly displayed to the user.

[0039] The generated output content refers to the final output text sequence that the model generates and displays to the user in the current round of inference requests.

[0040] Specifically, in each inference request, the inference engine receives the prompt text input by the user and converts it into a lexical sequence. During the inference process, the model generates hidden inference content, which is placed between predetermined start and end labels. After the hidden inference content is generated, the model continues to generate the final output text to be displayed to the user. The inference engine concatenates the prompt text, hidden inference content, and generated output content at the lexical level to form the inference context sequence for the current round, and assigns positional codes to each lexical in the sequence as input data for the pre-filling stage. For example, if the user inputs the prompt "Please analyze the 2024 financial report of a certain company," the model generates "..." <think> First, we need to extract the revenue data... (a total of 1200 tokens)< / think> The output is "The company's revenue will grow by 15% in 2024". The reasoning context sequence obtained by the reasoning engine includes the above prompt words, 1200 hidden reasoning words between tags, and 10 generated output words after the tags.

[0041] S110. Compare the length of the hidden inference content with a first preset threshold, and determine the retention strategy of the hidden inference content in the historical session based on the comparison result.

[0042] The first preset threshold (threshold_hidden_token_length) is a pre-defined length threshold used for comparison with the length of the hidden inference content (output_hidden_token_length), determining whether the hidden inference content should be retained in historical sessions. The first preset threshold can be pre-set based on the hardware configuration parameters of the inference service, the context window length of the deployed large language model, and the business scenario's requirements for inference latency. When the inference service starts, the system automatically calculates and loads the first preset threshold into the system configuration file based on the above factors; when receiving an inference request carrying the first preset threshold, the value in the request parameters is used.

[0043] Among them, historical conversations refer to the records of multiple rounds of dialogue that have occurred before the current round and their corresponding reasoning state information.

[0044] The retention strategy refers to the decision rule for retaining or discarding hidden inference content in historical sessions, which is determined based on the comparison between the length of the hidden inference content and a first preset threshold.

[0045] Specifically, embodiments of the present invention can determine the length category to which the hidden inference content belongs based on the comparison result between the length of the hidden inference content and a first preset threshold. A corresponding cache retention mode is then matched according to the length category, wherein different length categories correspond to different cache retention ranges.

[0046] S120. Based on the retention policy, determine the target key-value cache to be retained from the key-value cache of historical sessions.

[0047] Here, the key-value cache of historical sessions refers to the key-value matrix and value matrix generated during the pre-filling phase of historical sessions and used for attention calculation during the decoding phase.

[0048] The target key-value cache refers to the subset of key-value caches that need to be written to the storage system, determined from the key-value caches of historical sessions based on the retention policy.

[0049] Specifically, embodiments of the present invention can determine the text content that needs to be retained in historical sessions based on a retention strategy. Key-value caches corresponding to the position range of the text content to be retained are extracted from the key-value cache of the historical sessions and used as the target key-value cache.

[0050] S130. Write the target key-value cache to the storage system according to the retention range corresponding to the target key-value cache.

[0051] The retention range refers to the lexical range corresponding to the target key-value cache, which may include: the key-value cache corresponding to the prompt word, or the key-value cache corresponding to the prompt word, the hidden inference content and the generated output content as a whole.

[0052] The storage system refers to the storage medium used to persistently store key-value caches for reuse in subsequent inference requests.

[0053] Specifically, in this embodiment of the invention, the key-value cache to be written and the key-value cache to be released can be determined based on the retention range corresponding to the target key-value cache. The key-value cache to be written is written to the storage system, and the key-value cache to be released is released from memory.

[0054] S140. When the preset triggering condition is met, perform asynchronous pre-filling on the text sequence corresponding to the missing key-value cache in the storage system, and write the pre-filled complete key-value cache into the storage system.

[0055] The preset trigger conditions refer to the set of conditions that trigger the execution of asynchronous pre-filling operations.

[0056] The missing key-value cache refers to the word sequence corresponding to the missing key-value cache in the storage system, that is, the output text sequence corresponding to the generated output content that was discarded in the previous round. When asynchronous prefilling is performed, the output text sequence and the text sequence corresponding to the prompt words together constitute the complete input text sequence in the prefilling stage. The completion key-value cache is a subset of the key-value cache corresponding to the position interval of the output text sequence in the complete input text sequence.

[0057] Among them, the completion key-value cache refers to the key-value cache generated by asynchronously pre-filling the text sequence corresponding to the missing key-value cache in the storage system, which is used to concatenate with the stored key-value cache to form a complete historical key-value cache.

[0058] Specifically, in this embodiment of the invention, when a preset triggering condition is met, the text sequence to be supplemented corresponding to the missing key-value cache in the storage system can be determined. The text sequence to be supplemented is input into the model to perform completion calculation, resulting in a completed key-value cache. The completed key-value cache is written to the storage system to fill the key-value cache gap in the storage system.

[0059] This invention provides a pre-filling method for model inference. By comparing the length of hidden inference content with a first preset threshold, a retention strategy is determined based on the comparison result. The corresponding target key-value cache is then written into the storage system according to the retention strategy. This ensures that hidden inference content is only discarded when a trigger condition is met, overcoming the frequent key-value cache failures caused by the unconditional discarding of intermediate inference content in existing technologies. This maximizes the retention range of the key-value cache without losing effective contextual information, providing a data foundation for reducing the latency of the first word generation in subsequent rounds. Simultaneously, when the preset trigger condition is met, asynchronous pre-filling is performed on the text sequence corresponding to the missing key-value cache in the storage system, and the completed key-value cache is written into the storage system. This allows the historical key-value cache required for the next round of dialogue to be calculated in advance during the idle period of the current round, overcoming the latency problem caused by the forced synchronous re-execution of pre-filling in the next round of dialogue due to missing key-value cache. This achieves the technical effect of reducing the latency of the first word generation in the next round of dialogue and improving the response speed of the inference system.

[0060] Optionally, in the above Figure 1 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, the preset triggering condition may include: the length of the generated output content in the current round is greater than or equal to a second preset threshold, and the load of the pre-filling stage in the inference service is lower than a preset load threshold.

[0061] The second preset threshold (threshold_output_token_length) refers to a preset length threshold value used to compare with the length of the output content generated in the current round (output_token_length), and is used to determine whether the length condition for triggering asynchronous pre-filling is met.

[0062] Among them, reasoning service refers to a computing system that deploys large language models and provides reasoning capabilities, including a pre-filling stage and a decoding stage.

[0063] The prefilling phase refers to the computation phase in the inference service where the input lexical sequence is computed in parallel as a key-value cache and the first output lexical is generated. When the prefilling phase is deployed using a prefill-decode-disaggregation (PD) architecture, the inference service scheduling system obtains the load status information of the prefilling service nodes through an independent service discovery mechanism.

[0064] The preset load threshold refers to a baseline value used to compare with the actual load of the pre-filling phase in the inference service. It determines whether the pre-filling phase is idle enough to perform asynchronous pre-filling. The preset load threshold can be pre-determined based on the maximum concurrent processing capacity of the pre-filling phase in the inference service, the number of currently deployed pre-filling service nodes, and the desired reserved processing redundancy. When the actual load is lower than the preset load threshold, it indicates that there are computing resources available for performing asynchronous pre-filling operations during the pre-filling phase.

[0065] Specifically, after the target key-value cache is written to the storage system, the inference engine extracts the lexical sequence for generating the output content from the inference context sequence of the current round, calculates the length of the lexical sequence, and reads the second preset threshold from the system configuration. The length of the generated output content is compared with the second preset threshold. If the length of the generated output content is less than the second preset threshold, the length condition is deemed unmet, the asynchronous pre-filling triggering process is terminated, and no pre-filling operation is performed. If the length of the generated output content is greater than or equal to the second preset threshold, the length condition is deemed met, and load determination continues: the inference service scheduling system obtains the number of tasks currently being processed, the waiting queue depth, and the computing resource utilization rate through the monitoring interface, and determines the current load value of the pre-filling stage based on these indicators. The current load value is compared with the system's preset load threshold. If the current load value is less than the preset load threshold, the load condition is deemed met. When both the length condition and the load condition are met, the preset triggering condition is determined to be met, and the asynchronous pre-filling operation is triggered. If either condition is not met, the preset triggering condition is determined not to be met, the asynchronous pre-filling operation is not triggered, and the process terminates.

[0066] In addition, when the pre-filling phase is deployed using a PD separation architecture, the inference service scheduling system obtains the load status information of the pre-filled service nodes through an independent service discovery mechanism, compares the current load value of each pre-filled service node with the preset load threshold, and determines that the load condition is met when the current load value of all pre-filled service nodes is less than the preset load threshold.

[0067] To facilitate understanding, an example is provided below: Assume the second preset threshold is set to 200 words, and the length of the output content generated in the current round is 350 words. Since 350 ≥ 200, the length condition is met. The inference service scheduling system obtains the current load value of the pre-filling stage. This value is determined by the ratio of the current number of tasks being processed (2) to the maximum number of concurrent tasks (10). The load value is 20%. If the preset load threshold is 50%, 20% < 50%, and the load condition is met. Since both the length condition and the load condition are met, the preset triggering condition is satisfied, and the inference engine triggers the asynchronous pre-filling operation, starting to perform pre-filling calculations on the output text sequences corresponding to the missing key-value cache in the storage system. If the current load value is 60%, since 60% ≥ 50%, the load condition is not met, and the asynchronous pre-filling operation is not triggered.

[0068] This invention limits asynchronous prefilling to a predetermined trigger condition, which only triggers when the length of the generated output content reaches a second predetermined threshold and there is available computing power during the prefilling stage. This avoids the waste of computing resources caused by invalid prefilling due to the output content being too short, and also prevents the preemption of current round inference resources during the high load of the prefilling stage, thus affecting service stability. In this way, without affecting the normal inference service of the current round, the idle computing power is used to complete the completion calculation of the key-value cache required for the next round in advance, thereby achieving the technical effect of reducing the generation delay of the first word in the next round of dialogue.

[0069] Optionally, in the above Figure 1 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S110 may specifically include:

[0070] If the length of the hidden inference content is greater than or equal to the first preset threshold, the retention strategy is determined to discard the hidden inference content and retain the prompt words and generated output content in the historical session.

[0071] Specifically, in this embodiment of the invention, the first preset threshold can be read from the request parameters of the current inference request by the inference engine, or, if the request parameters do not carry the first preset threshold, the first preset threshold can be read from the system configuration file. The inference engine locates the start and end tag positions of the hidden inference content from the inference context sequence of the current round, extracts the word subsequence between the two tags, and counts the number of words in the word subsequence as the length value of the hidden inference content. The inference engine compares the length value with the first preset threshold; when the comparison result is that the length value is greater than or equal to the first preset threshold, the inference engine determines that the hidden inference content of the current round belongs to the long sequence category and should be removed from the historical session. The inference engine divides the inference context sequence of the current round into a prompt word interval, a hidden inference content interval, and a generated output content interval, and determines that the retained content of the historical session is all words in the prompt word interval and the generated output content interval, while words in the hidden inference content interval are not retained. The inference engine records the boundary position information of the above-mentioned retained content in the session context for subsequent key-value cache selection operations. When the first preset threshold is preset to zero, the inference engine does not need to perform length comparison and directly determines the retention strategy as discarding hidden inference content.

[0072] As examples, assuming a first preset threshold of 1000 words, the inference context sequence in the current round contains " <think> "and"< / think> The reasoning process between tags involves 1200 lexical units, with a length of 1200 ≥ 1000. The reasoning engine determines the retention strategy to discard the hidden reasoning content of the aforementioned 1200 lexical units. The prompt word in the current round is "Analyze a company's financial report," and the generated output content is "The company's net profit increased by 10% year-on-year." The historical session only retains the aforementioned prompt word and the generated output content, totaling approximately 20 lexical units. When the next round of requests arrives, the historical session read by the system does not include the intermediate reasoning process. If the first preset threshold is set to zero, the system directly discards the hidden reasoning content without performing a length comparison.

[0073] When the length of the hidden inference content is less than the first preset threshold, the retention strategy is determined to retain the hidden inference content, and the prompt words, hidden inference content and generated output content are retained in the historical session.

[0074] Specifically, in this embodiment of the invention, the first preset threshold can be read from the request parameters of the current inference request by the inference engine, or, if the request parameters do not carry the first preset threshold, the first preset threshold can be read from the system configuration file. The inference engine locates the start and end tag positions of the hidden inference content from the inference context sequence of the current round, extracts the word subsequence between the two tags, and counts the number of words in the word subsequence as the length value of the hidden inference content. The inference engine compares the length value with the first preset threshold; when the comparison result is that the length value is less than the first preset threshold, the inference engine determines that the hidden inference content of the current round belongs to the short sequence category and should be completely retained in the historical session. The inference engine divides the inference context sequence of the current round into a prompt word interval, a hidden inference content interval, and a generated output content interval, and determines that the retained content of the historical session is all the words in the above three intervals. The inference engine records the boundary position information of all the above retained content in the session context for subsequent key-value cache selection operations. In addition, the inference engine marks the inference context sequence of this round as a fully preserved state and writes this state identifier into the session context for subsequent rounds to read when making cache reuse judgments.

[0075] As examples, assuming a first preset threshold of 1000 words, the inference context sequence in the current round contains " <think> "and"< / think> The tags contain a brief reasoning process of 400 units, with a length of 400 < 1000. The reasoning engine determines the retention strategy to retain the hidden reasoning content of these 400 units. The current round's prompt is "Recommend three nearby coffee shops," and the generated output is "Coffee A, Coffee B, Coffee C." The historical session retains approximately 410 units, including the prompt, hidden reasoning content, and generated output. When the next round's request arrives, the system reads the historical session containing the complete reasoning trajectory.

[0076] This invention compares the length of the hidden inference content with a first preset threshold and discards or retains the hidden inference content based on the comparison result. This allows short sequences of hidden inference content to be fully retained to maintain the reusability of the key-value cache, while long sequences of hidden inference content are discarded to save context window space. At the same time, at least the key-value cache corresponding to the prompt word is retained as the basis for subsequent asynchronous completion. This solves the problem of the key-value cache becoming completely invalid due to unconditionally discarding the hidden inference content, and thus maximizes the retention range of the key-value cache under the condition of limited context length.

[0077] Optionally, in the above Figure 1 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S120 may specifically include:

[0078] If the retention policy is to discard hidden inference content, the key-value cache corresponding to the prompt word will be determined as the target key-value cache.

[0079] Specifically, in this embodiment of the invention, the retention policy identifier can be read from the session context by the inference engine to determine that the retention policy for the current round is to discard hidden inference content. The inference engine obtains complete cached data from the key-value cache of the historical session and locates the start and end positions corresponding to the prompt words based on the position range information of each text content in the inference context sequence of the current round. Using the start and end positions of the prompt words as the index range, the inference engine extracts the key-value matrix and value matrix within this position range from the key-value cache of the historical session as a subset of the key-value cache corresponding to the prompt words. The inference engine marks this subset of key-value cache as the target key-value cache and writes the marking result into the session context for subsequent cache write operations. The inference engine does not select the key-value caches corresponding to the hidden inference content range and the generated output content range as the target key-value caches.

[0080] As examples, assume a first preset threshold of 1000, a hidden inference content length of 1200, and a retention policy of discarding hidden inference content. The total length of the inference context sequence in the current round is 1250 lexical units, where the cue word range is positions 1 to 10, the hidden inference content range is positions 11 to 1210, and the generated output content range is positions 1211 to 1250. The inference engine extracts the key-value matrix and value matrix corresponding to positions 1 to 10 from the key-value cache of historical sessions and marks this subset as the target key-value cache. The key-value cache corresponding to positions 11 to 1250 is not included in the target range.

[0081] When the retention strategy is to retain the hidden inference content, the key-value caches corresponding to the prompt words, hidden inference content, and generated output content are collectively determined as the target key-value cache.

[0082] Specifically, in this embodiment of the invention, the retention policy identifier can be read from the session context by the inference engine to determine that the retention policy for the current round is to retain the hidden inference content. The inference engine obtains complete cache data from the key-value cache of the historical session, and locates the start and end positions of the prompt words, the start and end positions of the hidden inference content, and the start and end positions of the generated output content based on the position range information of each text content in the inference context sequence of the current round. Taking the start position of the prompt words as the starting point and the end position of the generated output content as the ending point, the inference engine extracts the key-value matrix and value matrix within the above entire position range as the key-value cache corresponding to the prompt words, hidden inference content, and generated output content as a whole. The inference engine marks this overall key-value cache as the target key-value cache and writes the marking result into the session context for subsequent cache writing operations.

[0083] As examples, assume a first preset threshold of 1000, a hidden inference content length of 400, and a retention strategy of retaining the hidden inference content. The total length of the inference context sequence in the current round is 450 lexical units, where the cue word range is positions 1 to 10, the hidden inference content range is positions 11 to 410, and the generated output content range is positions 411 to 450. The inference engine extracts all key-value matrices and value matrices corresponding to positions 1 to 450 from the key-value cache of the historical session, marks the entire set as the target key-value cache, and does not perform any segmentation or discarding.

[0084] This invention directly determines the target key-value cache from the key-value cache of historical sessions based on a retention policy. The retention policy determines the retention range of the key-value cache. Under the discard policy, only the key-value cache corresponding to the prompt word is retained as the basis for subsequent completion. Under the retention policy, the entire key-value cache is retained directly, so that the target key-value cache written in the storage system is completely matched with the retention policy. This avoids the waste of storage space caused by writing too much invalid cache under the discard policy, and avoids the confusion of subsequent completion logic caused by writing incomplete cache under the retention policy. Thus, it provides a cache data foundation consistent with the retention policy for the implementation of asynchronous prefilling and reduces the computational complexity of completing the key-value cache.

[0085] Optionally, in the above Figure 1 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S130 may specifically include:

[0086] If the target key-value cache is the key-value cache corresponding to the prompt word, write the key-value cache corresponding to the prompt word to the storage system and discard the key-value caches corresponding to the hidden inference content and the generated output content.

[0087] Specifically, in this embodiment of the invention, the inference engine can read the marking results of the target key-value cache from the session context to determine that the target key-value cache to be written to the storage system in the current round is the key-value cache corresponding to the prompt word. The inference engine obtains the start and end positions of the prompt word in the inference context sequence, and extracts the corresponding key-value matrix and value matrix from the key-value cache of the historical session based on the above position range as the data to be written. The inference engine organizes the data to be written with the sequence position index as the key through the storage system interface, writes it to the specified storage area in the distributed file system or KV database, and records the metadata of the sequence position range corresponding to the written key-value cache. After completing the write operation, the inference engine locates the key-value cache corresponding to the hidden inference content range and the generated output content range in memory, calls the memory release interface, and removes the key-value matrix and value matrix of the above range from memory to release video memory or memory resources. After completing the write and release operations, the inference engine updates the session state to "cache persisted" and returns the write completion status to the scheduling system.

[0088] As examples, assume the total length of the inference context sequence in the current round is 1250 lexical units, the hint word range is positions 1 to 10, the hidden inference content range is positions 11 to 1210, the generated output content range is positions 1211 to 1250, and the target key-value cache is the key-value cache corresponding to the hint words. The inference engine extracts the key-value matrix and value matrix corresponding to positions 1 to 10, writes the above data to the distributed file system through the storage system interface, and records the sequence position range corresponding to the cache as 1 to 10 in the metadata after writing. Subsequently, the inference engine locates the key-value cache corresponding to positions 11 to 1250 in memory, calls the memory release interface to remove all the above caches from the video memory, and releases the storage resources occupied by the cache corresponding to approximately 1240 lexical units.

[0089] If the target key-value cache is a key-value cache corresponding to the prompt words, hidden inference content, and generated output content as a whole, the entire key-value cache is written to the storage system.

[0090] Specifically, in this embodiment of the invention, the inference engine can read the tagging results of the target key-value cache from the session context to determine that the target key-value cache to be written to the storage system in the current round is the key-value cache corresponding to the prompt word, hidden inference content, and generated output content as a whole. The inference engine obtains the entire position range from the start position of the prompt word to the end position of the generated output content in the inference context sequence, and extracts the key-value matrix and value matrix corresponding to all positions from the key-value cache of the historical session based on the above position range as the data to be written. The inference engine organizes the data to be written with the sequence position index as the key through the storage system interface, writes it to the designated storage area in the distributed file system or KV database, and records the metadata of the complete sequence position range corresponding to the written key-value cache. After completing the write operation, the inference engine does not perform any cache discard operation, keeps all key-value caches of the current round in memory for continued use in the decoding stage, records the write status in the session context, and returns the write completion status to the scheduling system.

[0091] As examples, assume the total length of the inference context sequence in the current round is 450 words, the hint word range is positions 1 to 10, the hidden inference content range is positions 11 to 410, the generated output content range is positions 411 to 450, and the target key-value cache is the overall corresponding key-value cache. The inference engine extracts all key-value matrices and value matrices corresponding to positions 1 to 450, and writes the above overall data to the KV database through the storage system interface. After writing, the metadata records that the sequence position range corresponding to this cache is 1 to 450. After writing, the inference engine does not perform any discarding operation; all 450 key-value cache positions are retained in memory to continue serving the subsequent decoding calculations in the current round.

[0092] This invention's embodiments utilize a target key-value cache that, when the retention policy is discarded, contains only the key-value cache corresponding to the prompt word; when the retention policy is retained, it contains the key-value cache corresponding to the prompt word, hidden inference content, and the generated output content as a whole. During writing to the storage system, partial cache writing and full cache writing are performed according to the aforementioned retention range, ensuring that the key-value cache in the storage system is completely consistent with the retention range determined by the retention policy. This avoids storage redundancy caused by writing the key-value cache corresponding to the hidden inference content and generated output content that have already been determined to be discarded into the storage system. Simultaneously, it ensures that under the discard policy, only the key-value cache corresponding to the prompt word is saved as the base data for subsequent asynchronous completion, while under the retention policy, the entire key-value cache is saved to directly serve the next round of reuse. This reduces the storage system's write overhead and provides a clear cache gap range for asynchronous pre-filling.

[0093] Optional, based on Figure 1 The method shown is as follows: Figure 2The diagram shows a specific implementation of step S140 in the pre-filling method for model inference provided in this embodiment of the invention. Step S140 may specifically include:

[0094] S200: Obtain the output text sequence corresponding to the output content that was discarded in the previous round.

[0095] Specifically, in embodiments of the present invention, when the current round is not the first round of dialogue, the inference engine can read cached records of historical sessions from the session context and identify marker information indicating that key-value caches have been partially discarded in the previous round. The position range of the discarded text content in the inference context sequence of the previous round is extracted from the aforementioned marker information. Based on the aforementioned position range, the inference engine extracts the corresponding lexical subsequence from the inference context sequence record of the previous round as the output text sequence. The inference engine stores the extracted output text sequence in a temporary cache and records the start and end positions of the text sequence in the inference context sequence of the previous round for use during completion calculations.

[0096] As examples, suppose the prompt in the previous round of reasoning context was "Please analyze a company's financial report," and the hidden reasoning content was "..." <think> First, extract the revenue data... (a total of 1200 tokens)< / think> The generated output content is "The company's net profit increased by 10% year-on-year (10 tokens in total)". The conversation context record shows that in the previous round, because the length of the hidden inference content (1200) exceeded the first preset threshold (1000), only the key-value cache corresponding to the prompt word was written to the storage system, and the key-value cache corresponding to the 10 tokens of the generated output content was discarded. The inference engine extracts the tag information of the discarded content from the cache record of the conversation context, and extracts the 10 tokens of "The company's net profit increased by 10% year-on-year" from the inference context sequence of the previous round as the output text sequence based on the position range in the tag information.

[0097] S210. Use the output text sequence as input for the pre-filling stage, perform key-value calculation, and obtain the completion key-value cache.

[0098] Specifically, in this embodiment of the invention, the output text sequence can be read from a temporary cache via an inference engine. During key-value calculation, the inference engine concatenates the output text sequence with the text sequence corresponding to the key-value cache of the previously stored prompt word. The concatenated complete text sequence is used as input for the pre-filling stage, enabling attention calculation of the key-value of each word in the output text sequence with the key-value cache corresponding to the prompt word as a prefix. The inference engine loads the weight parameters of the large language model and sequentially inputs each word in the concatenated complete text sequence into the model's embedding layer, converting them into word vectors. The inference engine performs forward propagation calculation on each word vector through the model's multi-head attention layer, generating a key matrix and a value matrix corresponding to each attention layer. From the aforementioned key matrix and value matrix, the inference engine extracts the key-value cache within the corresponding position interval of the output text sequence as a completion key-value cache, and stores the completion key-value cache in memory for later writing.

[0099] As examples, embodiments of the present invention can use an inference engine to concatenate the extracted sequence of 10 lexical units ("the company's net profit increased by 10% year-on-year") with the text sequence corresponding to the key-value cache of the previously stored prompt word "Please analyze the financial report of a certain company". The concatenated complete text sequence is then used as input for the pre-filling stage. After loading the model weights, each lexical unit in the complete text sequence is sequentially converted into a lexical unit vector. Through multi-layer multi-head attention calculation, all key-value caches corresponding to the complete sequence are obtained. The completion key-value cache corresponding to the 10 lexical units of the output text sequence is extracted from this cache. This completion key-value cache is temporarily stored in memory, separately from the key-value cache of the prompt word.

[0100] It is understandable that asynchronous prefilling is executed in parallel with the decoding stage of the current round, and the calculation results of asynchronous prefilling do not affect the generated output content already output in the current round. When a PD separation architecture is adopted, asynchronous prefilling is executed by an independent prefilling service node, and the decoding service node of the current round is not affected by the load of asynchronous prefilling.

[0101] S220. Write the completion key-value cache to the storage system so that the completion key-value cache and the key-value cache corresponding to the stored prompt words are concatenated to form a complete historical key-value cache.

[0102] Specifically, in this embodiment of the invention, the inference engine can read the completion key-value cache from memory and simultaneously read the stored key-value cache corresponding to the prompt words from the storage system. The inference engine determines the positional relationship between the completion key-value cache and the stored key-value cache corresponding to the prompt words, based on the positional range of the output text sequence in the previous round's inference context sequence and the positional range of the prompt words in the previous round's inference context sequence. Since the prompt words are the starting part of the inference context sequence, the complete sequence after concatenating the completion key-value cache is completely consistent with the previous round's inference context sequence at the lexical level. The inference engine writes the completion key-value cache to the storage system through the storage system interface and updates the position index metadata of the key-value cache in the storage system according to the above positional relationship, so that the completion key-value cache and the stored key-value cache corresponding to the prompt words together constitute the key-value cache corresponding to the complete lexical sequence. After writing, the inference engine updates the integrity flag of the key-value cache in the session context, marking the complete historical key-value cache as available for reuse in the next round of inference requests.

[0103] As examples, embodiments of the present invention can read the stored key-value cache corresponding to the prompt phrase "Please analyze a company's financial report" from the storage system via the inference engine, and read the completed key-value cache from memory to determine that the sequence position corresponding to the prompt phrase comes first and the position corresponding to the output text sequence comes later. The completed key-value cache is written to the storage system, and the position index metadata is updated so that all key-value caches corresponding to the prompt phrase and the output text sequence are arranged consecutively in the storage system according to the sequence order, forming a complete historical key-value cache. The key-value cache integrity flag in the session context is updated from "partially available" to "fully available," and when the next round of inference request arrives, the complete historical key-value cache can be directly read from the storage system and reused.

[0104] This invention asynchronously prefills the output text sequence corresponding to the discarded generated output content from the previous round, uses the output text sequence as input for the prefilling stage to perform key-value calculation to obtain a complete key-value cache, and concatenates the complete key-value cache with the key-value cache corresponding to the stored prompt words to form a complete historical key-value cache. This allows the complete historical key-value cache to be read directly from the storage system and reused when the next round of inference request arrives, without the need to synchronously perform key-value calculation in the prefilling stage in the next round. This removes the prefilling calculation from the critical path of the request-response in the next round, achieving the technical effect of reducing the delay in the generation of the first word in the next round of dialogue.

[0105] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, the method may further include:

[0106] When the next round of inference requests arrives, the historical key-value cache is read from the storage system and used as a reuse cache for the next round of prefilling phase.

[0107] Specifically, in this embodiment of the invention, when the next round of inference request arrives, the inference engine receives the request and parses the session identifier carried in the request. The inference engine initiates a query request to the storage system based on the session identifier to obtain the key-value cache metadata corresponding to the session. The inference engine reads the location index information and integrity flag of the complete historical key-value cache from the metadata to verify whether the session has a complete historical key-value cache. When the integrity flag is in an available state, the inference engine reads the key-value cache corresponding to the prompt word and the completion key-value cache from the storage system through the storage system interface, according to the location index order. The inference engine loads the read key-value cache into memory according to the sequence position order, establishes a mapping relationship between sequence position and cache data in memory, marks the loaded complete key-value cache as a reuse cache, and records the sequence position range covered by the reuse cache for use in the pre-filling stage.

[0108] Based on the reuse cache, pre-filling is performed on the inference context sequence of the next round to obtain the first output word of the next round.

[0109] Specifically, in this embodiment of the invention, the current inference context sequence for the next round can be obtained through the inference engine, and the newly added lexical parts not covered by the reused cache in the current inference context sequence can be extracted. The inference engine uses the lexical sequence of the newly added lexical parts as input for the supplementary calculation stage, performs key-value calculation, and obtains the key-value cache of the newly added parts. The inference engine concatenates the key-value cache of the newly added parts with the reused cache read from the storage system in sequence order to form a complete key-value cache set. The inference engine performs attention calculation on the query vector corresponding to each lexical position in the current round's inference context sequence and the key-value matrix in the complete key-value cache set to obtain the probability distribution of the first lexical in the current round. The inference engine samples or performs greedy decoding based on the probability distribution, and selects the lexical with the highest probability as the first output lexical in the current round.

[0110] This invention, through asynchronous pre-filling, fills in and stores the missing key-value cache in the current round. This allows the historical key-value cache to be directly read from the storage system as a reuse cache when the next round's inference request arrives. Based on the reuse cache, pre-filling is performed on the inference context sequence of the next round. This removes the key-value calculations that overlap with the historical sequence in the pre-filling stage from the request-response path of the next round. Only the newly added lexical units need to be calculated, thus achieving the technical effect of reducing the generation delay of the first lexical unit in the next round.

[0111] Optionally, the hidden inference content includes at least one of the following types of text sequences: inference text between thought chain tag pairs, call parameter text between tool call tag pairs, and call parameter text between function call tag pairs.

[0112] In this context, the reasoning text between thought chain tag pairs refers to the sequence of intermediate reasoning steps generated by the model during the execution of reasoning tasks to achieve the step-by-step derivation of complex problems. This sequence of intermediate reasoning steps is enclosed in predetermined start and end tags and is not directly displayed to the user.

[0113] Understandably, when the model performs an inference task and needs to generate intermediate inference steps, it first writes a predefined thought chain start label (e.g., "...") into the output sequence. <think> Then, word-by-word generation of a text sequence representing the reasoning process continues until all intermediate reasoning steps are completed, at which point a predetermined thought chain end tag (e.g., "") is written.< / think> When the inference engine receives the model output, it scans the label positions in the output sequence, identifies the text interval between the start label and the end label, and extracts the word subsequence within this interval as the thought chain inference text. The inference engine marks the above inference text as hidden inference content, does not display it to the user, and records the start label position, end label position, and word length of the text in the session context.

[0114] The tool invocation parameter text between the tool invocation label pairs refers to the text sequence generated by the model during the inference task to describe the tool invocation method and input parameters for invoking external tools. This invocation parameter text sequence is enclosed in predetermined start and end labels and is not directly displayed to the user.

[0115] Understandably, when a model performs an inference task and needs to invoke external tools to obtain additional information or perform specific operations, it first writes a predefined tool invocation start label (e.g., "...") into the output sequence.<tool_call> Then, it generates a structured text sequence (e.g., JSON format or function signature format) describing the name of the tool to be invoked and the invocation parameters word by word, until all invocation parameter descriptions are completed and a predetermined tool invocation end tag (e.g., "...") is written.< / tool_call> When the inference engine receives the model output, it scans the label positions in the output sequence, identifies the text interval between the start and end labels, and extracts the word subsequence within this interval as tool call parameter text. The inference engine marks the above call parameter text as hidden inference content, does not display it to the user, and records the start label position, end label position, and word length of the text in the session context. Among them, the call parameter text between function call label pairs refers to the text sequence generated by the model during the execution of the inference task to describe the function call method and input parameters for calling the preset function. The call parameter text sequence is wrapped by predetermined start and end labels and is not directly displayed to the user.

[0116] Understandably, when a model performs an inference task and needs to call a pre-defined function to complete a specific computation or data operation, it first writes a pre-defined function call start label (e.g., "...") into the output sequence.<function_call> Then, it generates a structured text sequence (e.g., JSON format or key-value pair format) describing the name of the function to be called and its input parameters word by word, until all parameter descriptions are completed and a predefined function call end tag (e.g., "") is written.< / function_call> When the inference engine receives the model output, it scans the label positions in the output sequence, identifies the text interval between the start and end labels, and extracts the word subsequence within this interval as the function call parameter text. The inference engine marks the above call parameter text as hidden inference content, does not display it to the user, and records the start label position, end label position, and word length of the text in the session context.

[0117] By limiting the hidden inference content to text sequences between thought chain tag pairs, tool call tag pairs, and function call tag pairs, this invention can uniformly apply to the aforementioned types of intermediate generated content that is not displayed to the user. Thus, without designing different processing logic for each type of tag, it can achieve unified length determination, retention strategy selection, and cache writing operations for various types of hidden inference content, expanding its applicability in multiple inference scenarios.

[0118] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0119] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0120] Corresponding to the above method embodiments, this invention also provides a pre-filling device for model inference, the structure of which is as follows: Figure 3 As shown, it may include: an inference context sequence acquisition unit 10, a retention policy determination unit 20, a target key value cache determination unit 30, a target key value cache writing unit 40, and a pre-filling unit 50.

[0121] The reasoning context sequence acquisition unit 10 is used to acquire the reasoning context sequence of the reasoning request in the current round, wherein the reasoning context sequence includes prompt words, hidden reasoning content and generated output content.

[0122] The retention strategy determination unit 20 is used to compare the length of the hidden inference content with a first preset threshold and determine the retention strategy of the hidden inference content in the historical session based on the comparison result.

[0123] The target key-value cache determination unit 30 is used to determine the target key-value cache to be retained from the key-value cache of historical sessions according to the retention policy.

[0124] The target key-value cache writing unit 40 is used to write the target key-value cache into the storage system according to the retention range corresponding to the target key-value cache.

[0125] The pre-filling unit 50 is used to perform asynchronous pre-filling on the text sequence corresponding to the missing key-value cache in the storage system when a preset trigger condition is met, and write the pre-filled completed key-value cache into the storage system.

[0126] Optionally, the preset triggering conditions include: the length of the generated output content in the current round is greater than or equal to the second preset threshold, and the load of the pre-filling stage in the inference service is lower than the preset load threshold.

[0127] Optionally, the retention strategy determination unit 20 can be used to determine the retention strategy as discarding the hidden inference content and retaining the prompt words and generated output content in the historical session when the length of the hidden inference content is greater than or equal to a first preset threshold; and to determine the retention strategy as retaining the hidden inference content and retaining the prompt words, hidden inference content and generated output content in the historical session when the length of the hidden inference content is less than the first preset threshold.

[0128] Optionally, the target key-value cache determination unit 30 can be used to determine the key-value cache corresponding to the prompt word as the target key-value cache when the retention strategy is to discard the hidden inference content; and to determine the key-value caches corresponding to the prompt word, the hidden inference content, and the generated output content as the target key-value cache when the retention strategy is to retain the hidden inference content.

[0129] Optionally, the target key-value cache writing unit 40 can be used to write the key-value cache corresponding to the prompt word to the storage system and discard the key-value caches corresponding to the hidden inference content and the generated output content when the target key-value cache is the key-value cache corresponding to the prompt word, the hidden inference content and the generated output content; and to write the entire key-value cache to the storage system when the target key-value cache is the key-value cache corresponding to the prompt word, the hidden inference content and the generated output content.

[0130] Optionally, the pre-filling unit 50 can be used to obtain the output text sequence corresponding to the generated output content that has been discarded in the previous round; use the output text sequence as input for the pre-filling stage, perform key value calculation, and obtain the completion key value cache; write the completion key value cache into the storage system so that the completion key value cache and the key value cache corresponding to the stored prompt words are concatenated to form a complete historical key value cache.

[0131] Optionally, the pre-filling device applied to model inference may further include: a first output word acquisition unit, used to read historical key-value cache from the storage system when the next round of inference request arrives, as a reuse cache for the next round of pre-filling stage; and based on the reuse cache, perform pre-filling on the inference context sequence of the next round to obtain the first output word of the next round.

[0132] Hidden inference content includes at least one of the following types of text sequences: inference text between thought chain tag pairs, call parameter text between tool call tag pairs, and call parameter text between function call tag pairs.

[0133] This invention provides a pre-filling device for model inference. It compares the length of hidden inference content with a first preset threshold, determines a retention strategy based on the comparison result, and writes the corresponding target key-value cache into a storage system according to the retention strategy. This ensures that hidden inference content is only discarded when a trigger condition is met, overcoming the frequent key-value cache failure problem caused by the unconditional discarding of intermediate inference content in existing technologies. This maximizes the retention range of the key-value cache without losing effective contextual information, providing a data foundation for reducing the latency of the first word generation in subsequent rounds. Simultaneously, when the preset trigger condition is met, asynchronous pre-filling is performed on the text sequence corresponding to the missing key-value cache in the storage system, and the completed key-value cache is written into the storage system. This allows the historical key-value cache required for the next round of dialogue to be calculated in advance during the idle period of the current round, overcoming the latency problem caused by the forced synchronous re-execution of pre-filling in the next round of dialogue due to missing key-value cache. This achieves the technical effect of reducing the latency of the first word generation in the next round of dialogue and improving the response speed of the inference system.

[0134] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0135] The pre-filling device applied to model inference includes a processor and a memory. The inference context sequence acquisition unit 10, retention policy determination unit 20, target key value cache determination unit 30, target key value cache writing unit 40 and pre-filling unit 50 are all stored in the memory as program units. The processor executes the above program units stored in the memory to realize the corresponding functions.

[0136] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured. By adjusting kernel parameters, the cache retention range can be dynamically adjusted based on thresholds, and combined with asynchronous pre-filling, the latency of generating the first lexical unit in the next round of dialogue can be reduced.

[0137] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the pre-filling method applied to model inference.

[0138] This invention provides a processor for running a program, wherein the program executes the pre-filling method applied to model inference.

[0139] like Figure 4 As shown, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned pre-filling method applied to model inference. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0140] The present invention also provides a computer program product that, when executed on an electronic device, is adapted to execute a program that initializes pre-filled method steps applied to model reasoning.

[0141] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0143] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0144] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0145] In the description of this invention, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0146] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0147] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the present invention.

Claims

1. A pre-filling method applied to model inference, characterized in that, include: Obtain the reasoning context sequence of the reasoning request in the current round, wherein the reasoning context sequence includes prompt words, hidden reasoning content and generated output content; The length of the hidden inference content is compared with a first preset threshold, and the retention strategy of the hidden inference content in historical sessions is determined based on the comparison result. Based on the retention policy, the target key-value cache to be retained is determined from the key-value cache of historical sessions; Write the target key-value cache into the storage system according to the retention range corresponding to the target key-value cache; When a preset trigger condition is met, asynchronous pre-filling is performed on the text sequence corresponding to the missing key-value cache in the storage system, and the pre-filled complete key-value cache is written into the storage system.

2. The method according to claim 1, characterized in that, The preset triggering conditions include: the length of the generated output content in the current round is greater than or equal to the second preset threshold, and the load of the pre-filling stage in the inference service is lower than the preset load threshold.

3. The method according to claim 1, characterized in that, The step of comparing the length of the hidden inference content with a first preset threshold and determining the retention strategy of the hidden inference content in historical sessions based on the comparison result includes: If the length of the hidden inference content is greater than or equal to the first preset threshold, the retention strategy is determined to be to discard the hidden inference content and retain the prompt word and the generated output content in the historical session; When the length of the hidden inference content is less than the first preset threshold, the retention strategy is determined to be to retain the hidden inference content, and the prompt word, the hidden inference content, and the generated output content are retained in the historical session.

4. The method according to claim 3, characterized in that, The step of determining the target key-value cache to be retained from the key-value cache of historical sessions according to the retention policy includes: If the retention policy is to discard the hidden inference content, the key-value cache corresponding to the prompt word will be determined as the target key-value cache; When the retention strategy is to retain the hidden inference content, the key-value caches corresponding to the prompt words, the hidden inference content, and the generated output content are collectively determined as the target key-value cache.

5. The method according to claim 4, characterized in that, The step of writing the target key-value cache into the storage system according to the retention range corresponding to the target key-value cache includes: If the target key-value cache is the key-value cache corresponding to the prompt word, the key-value cache corresponding to the prompt word is written to the storage system, and the key-value caches corresponding to the hidden inference content and the generated output content are discarded. If the target key-value cache is a key-value cache corresponding to the prompt word, the hidden inference content, and the generated output content as a whole, the entire key-value cache is written to the storage system.

6. The method according to claim 1, characterized in that, The step of asynchronously pre-filling the text sequence corresponding to the missing key-value cache in the storage system, and writing the pre-filled completed key-value cache into the storage system, includes: Get the output text sequence corresponding to the output content that was discarded in the previous round; The output text sequence is used as input to the pre-filling stage to perform key-value calculation and obtain a complete key-value cache; The completion key-value cache is written to the storage system so that the completion key-value cache and the key-value cache corresponding to the stored prompt word are concatenated to form a complete historical key-value cache.

7. The method according to claim 6, characterized in that, The method further includes: When the next round of inference request arrives, the historical key-value cache is read from the storage system and used as a reuse cache for the next round of prefilling phase; Based on the reused cache, pre-filling is performed on the reasoning context sequence for the next round to obtain the first output word of the next round.

8. The method according to any one of claims 1 to 7, characterized in that, The hidden reasoning content includes at least one of the following types of text sequences: reasoning text between thought chain tag pairs, call parameter text between tool call tag pairs, and call parameter text between function call tag pairs.

9. A pre-filling device for model reasoning, characterized in that, include: The system includes an inference context sequence acquisition unit, a retention policy determination unit, a target key-value cache determination unit, a target key-value cache writing unit, and a pre-filling unit. The reasoning context sequence acquisition unit is used to acquire the reasoning context sequence of the reasoning request in the current round, wherein the reasoning context sequence includes prompt words, hidden reasoning content and generated output content; The retention strategy determination unit is used to compare the length of the hidden inference content with a first preset threshold, and determine the retention strategy of the hidden inference content in the historical session based on the comparison result. The target key-value cache determination unit is used to determine the target key-value cache to be retained from the key-value cache of historical sessions according to the retention policy; The target key-value cache writing unit is used to write the target key-value cache into the storage system according to the retention range corresponding to the target key-value cache; The pre-filling unit is used to perform asynchronous pre-filling on the text sequence corresponding to the missing key-value cache in the storage system when a preset trigger condition is met, and write the pre-filled completed key-value cache into the storage system.

10. An electronic device, characterized in that, The electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the pre-filling method applied to model reasoning as described in any one of claims 1 to 8.

Images