Inference method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of artificial intelligence, and discloses an inference method and system, electronic equipment and a storage medium. The reasoning method comprises the steps that a user request is received, and the user request carries input information; according to the input information, performing matching in a cache to obtain a matched token sequence and KV parameters thereof; reasoning according to the matched token sequence, the KV parameters of the matched token sequence and the input information; and storing the token sequence obtained by reasoning and the KV parameters thereof to the cache, and responding to the user request according to a reasoning result. The reasoning efficiency is at least improved, the response to the user request is accelerated, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a reasoning method and system, an electronic device, and a storage medium. Background Art

[0002] An artificial intelligence (AI) model is a program that is trained on a set of data to recognize certain patterns or make certain decisions without further human intervention. AI models apply different algorithms to relevant data inputs to achieve the programmed task or output. For example, the Transformer architecture has become the mainstream choice for AI models due to its outstanding performance in natural language processing, image recognition, and other fields.

[0003] However, as the scale of models continues to expand, the existing inference architecture faces many challenges in handling large-scale concurrent requests. In particular, as the scale of the model increases, its inference efficiency is also affected, which seriously reduces the user experience. Summary of the invention

[0004] The embodiments of the present application provide a reasoning method and system, an electronic device, and a storage medium, which are at least beneficial to improving reasoning efficiency, accelerating response to user requests, and improving user experience.

[0005] According to some embodiments of the present application, a first aspect of the embodiments of the present application provides an inference method, including: receiving a user request, wherein the user request carries input information; matching in a cache based on the input information to obtain a matching token sequence and its KV parameters; performing inference based on the matching token sequence and its KV parameters and the input information; storing the inferred token sequence and its KV parameters in the cache, and responding to the user request based on the result of the inference.

[0006] In some embodiments, the cache adopts a radix tree structure, each edge in the radix tree structure corresponds to a token sequence and its KV parameters, and storing the inferred token sequence and its KV parameters in the cache includes: generating a new branch for the edge corresponding to the matching token sequence and its KV parameters based on the inferred token sequence and its KV parameters.

[0007] In some embodiments, the token sequence and its KV parameters obtained by the reasoning are used to generate a new branch for the edge corresponding to the matching token sequence and its KV parameters, including: detecting whether the token sequence obtained by the reasoning has a common prefix with the token sequence in the cache outside the matching token sequence; if the common prefix exists, updating the token sequence and its KV parameters corresponding to the common prefix to the edge corresponding to the matching token sequence in the radix tree structure, and adding the token sequences and their KV parameters corresponding to different parts as branches of the updated edge of the radix tree structure; if the common prefix does not exist, adding a new edge to the edge corresponding to the matching token and its KV parameters based on the token sequence and its KV parameters obtained by the reasoning.

[0008] In some embodiments, the cache includes a first cache and a second cache, both of which adopt the radix tree structure; storing the inferred token sequence and its KV parameters in the cache includes: during the inference process, continuously storing the currently obtained token and its KV parameters based on the first cache; after the inference corresponding to the user request is completed, synchronizing the update of the first cache to the second cache.

[0009] In some embodiments, the method also includes: when the cache currently needs to delete data, determining at least one token to be deleted; if there is a corresponding node that is a leaf node and the corresponding node is deleted and a node with only one branch is generated after the deletion of the token to be deleted, then deleting the corresponding leaf node, and starting from the edge of the node with only one branch, merging it with the edge of the parent node step by step upward until the node with only one branch disappears.

[0010] In some embodiments, the determining of at least one token to be deleted includes: determining a token corresponding to a leaf node that is least recently used as the at least one token to be deleted.

[0011] In some embodiments, the reasoning based on the matched token sequence and its KV parameters and the input information includes: calling the prefill node in the prefill cluster, and performing calculations based on the matched token sequence and the KV parameters corresponding to the matched token sequence according to the input information; calling the decode node in the decode cluster, and performing calculations based on the matched token sequence, the KV parameters corresponding to the matched token sequence, and the token identifier and the corresponding KV parameters calculated by prompt; wherein the scheduling strategy of the prefill node in the prefill cluster includes: when the current resource idleness is greater than a first preset value, adopting a polling scheduling mechanism; and / or, when the current resource idleness is less than a second preset value, adopting a minimum queue scheduling mechanism; the scheduling strategy of the decode node in the decode cluster includes a load balancing strategy based on NCCL Buffer.

[0012] According to some embodiments of the present application, the second aspect of the embodiments of the present application further provides an inference system, including: a scheduling module, a prefill cluster, a decode cluster and a storage module; the storage module is connected to the scheduling module, the prefill cluster and the decode cluster respectively, so that the scheduling module, the prefill cluster and the decode cluster can perform data query and data storage, and the scheduling module is also connected to the prefill cluster and the decode cluster respectively; the scheduling module is used to receive user requests, wherein the user requests carry input information; according to the input information, matching is performed in the storage module to obtain a matching token sequence and its KV parameters; and the matching token sequence and its KV parameters, the The input information is forwarded to the prefill node in the prefill cluster for prefill calculation, so as to notify the decode node in the decode cluster of the result calculated by the prefill node; the prefill cluster includes a plurality of prefill nodes, each of which is used to perform calculation based on the matching token sequence and its KV parameters issued by the scheduling module and the input information, and store the calculation result in the storage module; the decode cluster includes a plurality of decode nodes, each of which is used to read the result calculated by the prefill node from the storage module based on the notification of the scheduling module to perform decode calculation, and store the calculation result in the storage module.

[0013] According to some embodiments of the present application, a third aspect of the embodiments of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the reasoning method as described in any one of the first aspects.

[0014] According to some embodiments of the present application, a fourth aspect of the embodiments of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the reasoning method as described in any one of the first aspects.

[0015] The technical solution provided in the embodiment of the present application has at least the following advantages:

[0016] After reasoning for a user request, the token sequence and its KV parameters generated by the reasoning of the user request are not directly deleted. Instead, the token sequence and its KV parameters generated by the reasoning are stored in the cache, so that subsequent user requests received can be matched in the cache and the matching token sequence and its KV parameters can be reused, reducing the tokens and their KV parameters that need to be calculated for subsequent requests, reducing the amount of calculation, speeding up the reasoning process, improving the response efficiency of user requests, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0018] Figure 1 is a flow chart of the reasoning method provided in the embodiments of the present application;

[0019] Figure 2 is a schematic diagram of node splitting comparison involved in the reasoning method provided in the embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the structure of the reasoning system provided in the embodiment of the present application;

[0021] Figure 4 is an application diagram of the reasoning system and reasoning method provided in the embodiments of the present application;

[0022] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] As can be seen from the background technology, the reasoning process of existing artificial intelligence models urgently needs to improve efficiency.

[0024] After analysis, it was found that the efficiency of existing artificial intelligence models is at least limited by the fact that each user request must be calculated token by token based on the input information. Specifically, in the application process of existing artificial intelligence models, after each response to a request, the data generated by the response process is cleared in order to respond to the next user request.

[0025] However, after further analysis, we found that not every user request is a completely new request. There may be duplicate content or even completely identical requests. Based on these similarities, there are some requests that actually have some reusable information in their response process, and this reusable information provides optimization space for improving the reasoning efficiency of the artificial intelligence model.

[0026] Based on this, the embodiments of the present application provide an inference method and system, an electronic device, and a storage medium, which store the token sequence and its KV parameters that may be reused in the subsequent user request response process during the user request response process, so that after receiving the user request subsequently, the reused token sequence and its KV parameters are found and reused through matching, so that subsequent user requests no longer need to start from the first token and calculate all KV parameters token by token. Therefore, the calculation amount of subsequent user requests is reduced, the inference process is accelerated, the inference efficiency is improved, and the user experience is enhanced.

[0027] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented.

[0028] The division of the following embodiments is for the convenience of description and shall not constitute any limitation on the specific implementation of the present application. The embodiments may be combined with each other and referenced to each other without contradiction.

[0029] The first aspect of the embodiment of the present application provides an inference method, which is applied to any device or device cluster that is deployed with a corresponding inference model. Among them, the inference method provided by the embodiment of the present application can reduce the calculations required for subsequent user requests by storing and reusing the data generated by previous user requests, thereby improving the inference efficiency and enhancing the user experience. In order to facilitate those skilled in the art to better understand the inference method provided in the embodiment of the present application and the implementation of its effect, the following will be combined with Figure 1 to Figure 2 To explain, Figure 1 A flowchart of the reasoning method provided in the embodiment of the present application, Figure 2 A comparative diagram of node splitting involved in one implementation of the inference method.

[0030] In some embodiments, Figure 1 As shown, the reasoning method includes the following steps:

[0031] Step 101: receiving a user request, wherein the user request carries input information.

[0032] Step 102: Match the input information in the cache to obtain a matching token sequence and its KV parameters.

[0033] Step 103, perform reasoning based on the matched token sequence and its KV parameters and input information.

[0034] Step 104: store the token sequence and its KV parameters obtained by inference into a cache, and respond to the user request according to the inference result.

[0035] In this way, after reasoning for a user request, the token sequence and its KV parameters generated by the reasoning of the user request are not directly deleted. Instead, the token sequence and its KV parameters generated by the reasoning are stored in the cache, so that subsequent user requests received can be matched in the cache and the matching token sequence and its KV parameters can be reused, reducing the tokens and their KV parameters that need to be calculated for subsequent requests, reducing the amount of calculation, speeding up the reasoning process, improving the response efficiency of user requests, and improving the user experience.

[0036] For ease of understanding, the following Figure 1 Each step in the process shown is explained.

[0037] In step 101, a user request is received, wherein the user request carries input information. Among them, the embodiment of the present application does not limit the type of user request, which can be any request that can carry corresponding input information. For example, the user request can be a Hypertext Transfer Protocol (HTTP) request. For example, the user request can also be a Hypertext Transfer Protocol Secure (HTTPS) request to enhance the security of the request, etc., which are not listed here one by one. And the embodiment of the present application does not limit the input information, which can be any information that can trigger the artificial intelligence model to perform reasoning. For example, in a dialogue task, the input information can be "What is the weather like today"; for example, in image processing characters, the input information can be a picture to be processed, etc., which are not listed here one by one.

[0038] In step 102, a match is performed in the cache according to the input information to obtain a matching token sequence and its KV parameters. Among them, the cache stores the token sequence and its KV (Key-Value) parameters corresponding to the user request to which the response has been completed (also called "KV Cache"). At this time, for the currently received user request, the input information it carries can also be converted into a corresponding token sequence. Therefore, the token sequence can be matched in the cache to obtain a token sequence matched to the token sequence stored in the cache and obtain its corresponding KV parameters. Of course, in some cases, it is also possible to directly match the input information carried by the user request with the completed response stored in the cache with the input information carried by the currently received user request, and further obtain the token sequence and its KV parameters corresponding to the matched input information, etc., which will not be repeated here.

[0039] It should be noted that the "match" in the embodiments of the present application is not a match of the complete input information or the complete token sequence, but a partial match. Taking the matching of input information as an example, assuming that the input information of the current user request is "ABCDE", if there is an input information "ABCHUIRHIOGVNERFD" carried by the user request that has been responded to, then it can be considered that the current user request matches "ABC", then the matched token sequence and its KV parameters, that is, the token sequence and its KV parameters corresponding to "ABC", will not be repeated here and in the future. Among them, the input sequence listed above only uses letters to represent the input information as an example, which is mainly for ease of understanding and has no other meaning.

[0040] It should also be noted that the token sequence matching involved in the embodiments of the present application mainly refers to the matching object involving the token sequence, rather than necessarily directly matching the token sequence itself. For example, when providing a corresponding identifier such as an ID number for a token, the identifier sequence corresponding to each token in the token sequence, i.e., tokenids, etc., can be directly used for matching, which will not be described one by one here.

[0041] In step 103, reasoning is performed based on the matched token sequence and its KV parameters and input information. The embodiment of the present application does not limit the specific model used for reasoning. It is understandable that it can be any model with reasoning function, which has been described in the relevant technology and will not be listed here one by one.

[0042] It should be noted that, in the embodiments of the present application, it mainly performs reasoning on the model architecture using prefill-decode, wherein prefill and decode are the two stages of model reasoning. In the prefill stage, the user input is processed and the corresponding KV parameters are calculated; in the decode stage, the KV parameters calculated in the prefill stage are used to sequentially calculate the tokens, only one token is calculated at a time, and a token sequence will be generated through continuous iterative calculation.

[0043] Furthermore, considering that the existing artificial intelligence model reasoning adopts the prefill-decode architecture, its processing of user requests depends on the serial processing method, that is, in each reasoning request, the model needs to complete the pre-loading of data in the prefill stage before entering the result generation of the decode stage. This serial processing flow has obvious inefficiency in the utilization of computing resources, especially when facing large-scale concurrent requests, it is easy to cause resource bottlenecks and response delays. Therefore, in some embodiments, the prefill stage and the decode stage are separated, and on the basis of separating prefill and decode, a separate cluster is also used for deployment. In this way, compared with the existing model, all reasoning stages are usually completed in a single cluster, resulting in complex cluster load management and resource scheduling, and it is difficult to flexibly respond to different requirements for computing resources at different stages. For example, in the prefill stage, the model needs to load a large amount of parameters and data, and the demand for memory and bandwidth is high; while in the decode stage, the complexity of the calculation increases, and the demand for GPU computing power is more significant. This embodiment separates the prefill and decode stages and processes them respectively by different prefill clusters and decode clusters, which greatly improves the parallelization level of the inference process. This not only improves the resource utilization efficiency, but also enhances the flexibility of the system, enabling it to better handle large-scale inference requests.

[0044] In some embodiments, a scheduling mechanism can be further introduced so that the prefill and decode in the response process of different user requests can be fully and appropriately completed in parallel on the prefill cluster and the decode cluster, further improving the efficiency of reasoning and enhancing the user experience. At this time, reasoning is performed based on the matching token sequence and its KV parameters and input information, which can be achieved in the following way: calling the prefill node in the prefill cluster, and starting the operation from the matching token sequence and the KV parameters corresponding to the matching token sequence according to the input information; calling the decode node in the decode cluster, and performing the operation based on the matching token sequence, the KV parameters corresponding to the matching token sequence, and the token identifier calculated by the prompt and the corresponding KV parameters.

[0045] In this way, through the direct access mode, the decoder accesses the cache to obtain the corresponding data, effectively reducing the data transmission time of prefill and decode. Through the monitoring requests of the prefill cluster and the decode cluster, feedback can be provided based on the utilization rate of the cluster nodes to achieve dynamic scheduling to complete the scheduling of prefill and decode, effectively improving the system flexibility and improving the stability of the system.

[0046] It should be noted that the embodiments of the present application do not limit the calling (or "scheduling") method of the prefill node in the prefill cluster and the decode node in the decode cluster, which can be adjusted according to specific application scenarios, device performance, concurrency characteristics of requests, etc. For example, in some embodiments, the scheduling strategy of the prefill node in the prefill cluster includes: when the current resource idleness is greater than the first preset value, a polling scheduling mechanism is adopted to balance the resources occupied by each prefill node when the resources are sufficient; and / or, when the current resource idleness is less than the second preset value, a minimum queue scheduling mechanism is adopted to fully utilize the resources in the case of insufficient resources to complete the prefill calculation with better efficiency. Among them, the embodiments of the present application do not set the first preset value and the second preset value, which can be set according to different application scenarios, node performance, etc. For example, in some embodiments, the scheduling strategy of the decode node in the decode cluster includes a load balancing strategy based on NCCL (NVIDIA Collective Communications Library) Buffer, which is conducive to making full use of the decode node resources, completing the decode calculation more efficiently, and responding to user requests. For example, in some embodiments, cluster monitoring can be set to perform dynamic scheduling based on the cluster operation status and the current cluster processing throughput returned by the monitoring, thereby improving the flexibility of scheduling through the monitoring and early warning mechanism. Of course, the above are only examples, and other scheduling strategies can also be used in some embodiments, which will not be described here one by one.

[0047] In some embodiments, for the decode cluster, a GPU machine with relatively high computing power (such as A100, H100, etc.) can be used to accelerate the calculation. First, after the prefill calculation is completed, the prefill calculation result is sent to the decode node through scheduling for asynchronous reasoning, thereby realizing the asynchronous implementation of prefill and decode, effectively reducing the synchronous serial blocking of prefill and decode, effectively reducing the time consumption, and continuous asynchronous execution can reduce the bubbles between prefill and decode.

[0048] That is to say, the existing artificial intelligence model faces significant efficiency bottlenecks when processing large-scale concurrent requests due to the use of serial processing methods during reasoning, namely: the serial reasoning process leads to low resource utilization; a single cluster processes all reasoning stages, resulting in inflexible resource scheduling. In the embodiment of the present application, a new and efficient distributed artificial intelligence model architecture is used to separate the prefill and decode stages, which are processed by different prefill clusters and decode clusters respectively, thereby transforming the serial reasoning process into a more efficient parallel reasoning mode, greatly improving the parallelization degree of the reasoning process, which not only improves the resource utilization efficiency, but also enhances the flexibility of the system, enabling it to better handle large-scale reasoning requests, further improving the efficiency of reasoning.

[0049] Step 104, store the token sequence and its KV parameters obtained by reasoning into the cache, and respond to the user request according to the result of reasoning. Among them, the embodiment of the present application does not limit the data structure used by the cache, which can be any storage space capable of storing token sequences and their KV parameters, such as a MySQL database, etc., which will not be listed here one by one. In this way, the calculated KV parameters are not stored in the prefill cluster to prevent affecting the prefill calculation of other requests, and the intermediate results of the calculation are stored in the cache outside the cluster to prepare for the decode calculation.

[0050] However, considering that with the increasing size of the model and the amount of data, the large transformer model faces the problems of high video memory occupancy and large computational delay during the calculation process. These models are composed of multiple layers of encoders and decoders, each layer of which contains a self-attention mechanism and a feedforward neural network, resulting in its calculation process showing obvious serial characteristics. When processing long sequence data, the model needs to manage a large number of weight matrices and intermediate results, which poses a severe challenge to cache management. Frequent data transmission and cache access not only increase the video memory pressure, but also bring significant data overhead and delay. Therefore, in some embodiments, the cache can adopt a radix tree structure. That is to say, when the KV parameters are stored, the radix tree is used for storage, and the KV parameters are not discarded after the generation request is completed, but the prompt (i.e., the calculated token sequence) and the KV cache of the generation result are retained in the radix tree, so that efficient prefix search, insertion and expulsion can be achieved based on the radix tree structure.

[0051] For example, in some embodiments, it is assumed that each edge in the radix tree structure corresponds to a token sequence and its KV parameters. At this time, the inferred token sequence and its KV parameters are stored in the cache, which can be achieved in the following way: according to the inferred token sequence and its KV parameters, a new branch is generated for the edge corresponding to the matching token sequence and its KV parameters. In other words, based on the splitting characteristics of the radix tree when it is updated, when a new token sequence is stored, the token will be split as the storage occurs, forming a more refined token division. Therefore, the subsequent matching in the cache can be more refined, easier to match successfully, and more tokens and their KV parameters can be reused, so that the reasoning difficulty is lower and the reasoning efficiency is higher. Taking dialogue reasoning as an example, a dialogue with complex content can be decomposed into multiple concise single-round dialogues, making it easier to match successfully, more content to match, more content that can be reused, less reasoning difficulty, and higher reasoning efficiency.

[0052] For ease of understanding, the following will be combined Figure 2 As shown, the application of the splitting characteristics of the radix tree structure is explained.

[0053] like Figure 2 As shown in the figure, suppose that based on the user's request, multiple rounds of dialogues "You are a helpfulassistant", "Hello", "Hi", "Solve this problem..." and "Sure!..." are generated. At this time, the token sequence will be as shown in the structure on the left. The content corresponding to the first edge under the root node includes multiple rounds of dialogues "You are a helpfulassistant", "Hello", and "Hi". Suppose that based on the above dialogue, further dialogues "You are ahelpful assistant", "What can you do", and "I can..." are generated. In the case of not using a radix tree, a new branch parallel to it will be added to the structure on the left, "You are a helpfulassistant", "What can youdo", and "I can...", and a new node will be added upward as the root node, connecting Figure 2 The branch structure and the newly added branch structure shown on the left. After applying the radix tree structure, the prefix "You are a helpful assistant" shared by the two conversations will make Figure 2 In the structure shown on the left, the first edge under the root node on the left splits out a new subordinate node downward, so the stored content will be updated as follows Figure 2As shown on the right, the first edge under the root node corresponds to "You are a helpful assistant", and the next level corresponds to "Hello", "Hi", and "What can you do", "I can..." That is, the node is split and the content corresponding to a single edge is reduced, making it easier to match more content and more likely to match successfully.

[0054] That is to say, at this time, after applying the radix tree, while retaining the characteristics of the prefix tree, the node can store variable-length sequences, that is, the large node in the cache can be dynamically split into small nodes, thereby realizing the multi-round dialogue scenario that is very common in artificial intelligence models. Splitting a multi-round dialogue into several single-round dialogue storage improves the reusability of KV parameters and effectively improves the hit rate (i.e. matching rate), which is especially suitable for resource-intensive tasks such as ultra-large parameter models and long texts.

[0055] Specifically, in some embodiments, combined with the splitting characteristics of the radix tree, new branches are generated for the edges corresponding to the matching token sequence and its KV parameters based on the inferred token sequence and its KV parameters, which can be achieved in the following way: detect whether the inferred token sequence has a common prefix with the token sequence in the cache outside the matching token sequence; if there is a common prefix, add the inferred token sequence to the edge where the common prefix is ​​located; if there is no common prefix, add a new edge.

[0056] More specifically, if there is a common prefix, the token sequence and its KV parameters corresponding to the common prefix are updated to the edge corresponding to the matching token sequence in the radix tree structure, and the token sequences and their KV parameters corresponding to the different parts are added as branches of the updated edge of the radix tree structure; if there is no common prefix, a new edge is added to the edge corresponding to the matching token and its KV parameters based on the inferred token sequence and its KV parameters.

[0057] In some embodiments, considering the possibility of insufficient cache, some content needs to be deleted, such as Figure 2 As shown, the process of the reasoning method also includes the following: in the case where the cache currently needs to delete data, determine at least one to-be-deleted token; if there is a corresponding node that is a leaf node and the corresponding node is deleted and generates a to-be-deleted token of a node with only one branch, delete the corresponding leaf node, and start from the edge of the node with only one branch, merge the edge with the parent node step by step upward until the node with only one branch disappears. In other words, when deleting, merge the edge of the node with only one branch, and further reduce the amount of cached data without affecting subsequent matching.

[0058] It is also understandable that various strategies can be used to determine the token to be deleted, such as the least recently used strategy (Least Recently Used, LRU), but considering that the intermediate node of the radix tree corresponding to the token to be deleted is actually ineffective in reducing the cache content, in some embodiments, determining at least one token to be deleted can be achieved in the following way: determining the token corresponding to the least recently used leaf node as the at least one token to be deleted. In other words, deleting from the leaf node ensures that after the token to be deleted is deleted, an effective data deletion operation will be generated in the cache, ensuring the normal reduction of cache data.

[0059] It should be noted that the above-mentioned least recently used strategy is only a strategy for determining the token to be deleted. In some embodiments, an ARC scenario-based replacement strategy can be further provided on the basis of the least recently used strategy, that is, the token to be deleted is determined in a manner that uses the least recently used strategy as the main strategy and the least frequently used (Least Frequently Used, LFU) strategy and the most recently used (Most Recently Used, MRU) strategy as secondary fluctuations. In this way, for different application characteristics of different scenarios such as the fact that there are few repeated text summarization tasks in long text summarization tasks, the low reusability of token sequences and their KV parameters, and the strong reusability of token sequences and their KV parameters in multi-round conversations, the corresponding appropriate tokens to be deleted will be adaptively generated, making the deletion more in line with current needs.

[0060] In some embodiments, a distributed structure can also be set up to implement caching, that is, the cache includes a first cache and a second cache. Accordingly, storing the token sequence obtained by reasoning and its KV parameters in the cache can be achieved in the following way: during the reasoning process, the currently obtained token and its KV parameters are continuously stored based on the first cache; after the reasoning corresponding to the user request is completed, the update of the first cache is synchronized to the second cache.

[0061] In this way, the distributed cache service can effectively manage KV parameters, cache intermediate data, and reduce the amount of data calculation for prefill.

[0062] It should be noted that, in some embodiments, in order to achieve better results, the first cache and the second cache may both adopt the radix tree structure as described above. Of course, in some embodiments, the first cache and the second cache may also adopt other data structures, which will not be described one by one here.

[0063] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0064] On the other hand, the embodiment of the present application also provides a reasoning system, such as Figure 3 As shown, it includes: a scheduling module, a prefill cluster, a decode cluster and a storage module. Among them, the storage module is connected to the scheduling module, the prefill cluster and the decode cluster respectively, so that the scheduling module, the prefill cluster and the decode cluster can perform data query and data storage. The scheduling module is also connected to the prefill cluster and the decode cluster respectively.

[0065] Specifically, the coordination between the models is as follows:

[0066] The scheduling module is used to receive user requests, where the user requests carry input information; match the input information in the storage module to obtain a matching token sequence and its KV parameters; and forward the matching token sequence and its KV parameters and input information to the prefill node in the prefill cluster for prefill calculation, so as to notify the decode node in the decode cluster of the result calculated by the prefill node;

[0067] The prefill cluster includes several prefill nodes. Each prefill node is used to perform calculations based on the matching token sequence and its KV parameters and input information issued by the scheduling module, and store the calculation results in the storage module;

[0068] The decode cluster includes several decode nodes. Each decode node is used to read the results calculated by the prefill node from the storage module based on the notification of the scheduling module to perform decode calculations, and store the calculation results in the storage module.

[0069] In some embodiments, the prefill cluster and decode cluster are composed of multiple inference instances. When the cluster is operating, multiple inference instances are running, and all prefill stages and decode stages are performed in the prefill cluster and decode cluster respectively.

[0070] In some embodiments, the storage involved in the inference process, including the cache of KV parameters, the intermediate data in the inference process, and the communication and storage between processes, are all managed by the storage module;

[0071] In some embodiments, the storage module internally sets storage corresponding to two different stages under the prefill-decode separation architecture, where the KV cache generated by the prefill node is implemented by KV storage, and the inference structure generated by the decode node is high-speed communication, which can be directly managed by the L1 KV Cache GPU VRAM module.

[0072] In some embodiments, the scheduling module uses a parallel processing architecture to handle process scheduling, data transmission, and dynamic resource allocation for multiple inference instances within the entire inference cluster (including the prefill cluster and the decode cluster), and it integrates a fault detection and recovery mechanism to achieve stable and reliable dynamic scheduling.

[0073] In some embodiments, the scheduling module supports processing different inputs of multiple reasoning instances in parallel at the same time, and performs monitoring and alarms to adjust the scheduling strategy.

[0074] In some embodiments, when multiple inference instances are running simultaneously, the scheduling module pre-allocates resources according to different models and running parameters of each instance.

[0075] In some embodiments, when the reasoning system starts to provide reasoning services, the scheduling module will be responsible for loading the initial data and initializing the system.

[0076] In some embodiments, when the inference system starts to provide inference services, the scheduling module initializes the data and prefill cluster configuration, and deploys them nearby according to similar models and similar inputs, which can effectively reduce the overhead of KV parameter queries.

[0077] In some embodiments, when the prefill cluster completes the prefill calculation, the scheduling module communicates the data to the decode cluster according to the strategy, which uses cache optimization technology, communication technology and prefix tree radix-tree technology to reduce repetitive overhead.

[0078] In some embodiments, when the decode cluster completes the decode calculation, the scheduling module will communicate with the storage module to store the corresponding data.

[0079] In some embodiments, when the inference service ends, the scheduling module returns resource utilization and overhead to optimize the self-learning and self-supervising scheduling algorithm based on the current results, or the scheduling can be configured by the user.

[0080] In some embodiments, for the prefill cluster, it can adopt a distributed model parameter loading method and initialization strategy to reduce repeated prefill calculations, plan resources in a node manner, and accelerate calculations.

[0081] In some embodiments, when the scheduling module communicates data to the prefill cluster and the prefill node, the prefill cluster uses a sharing mechanism to reduce data storage overhead for highly repetitive resources.

[0082] In some embodiments, when multiple inference instances within a prefill cluster use the same model, the prefill cluster will use the same prefill node to provide services.

[0083] In some embodiments, the utilization of resources between prefill nodes in the prefill cluster is in an occupying manner. Resources and tasks are allocated to nodes in a dynamic insertion manner.

[0084] In some embodiments, after the prefill cluster completes the task, it passes the result to the storage module, and the specific process is managed by the scheduling module.

[0085] In some embodiments, the prefill cluster and the decode cluster support asynchronous work, and the communication between the two is achieved through the storage module, rather than direct node interaction.

[0086] In some embodiments, the prefill cluster works asynchronously with the decode cluster by sending requests. After the prefill cluster completes the calculation, the scheduling module synchronizes the corresponding data to the L1 KV Cache GPU VRAM.

[0087] In some embodiments, the decode cluster uses a distributed startup mechanism to asynchronously perform decode calculations, and cooperates with an early stopping mechanism to reduce synchronization overhead.

[0088] In some embodiments, after the inference system is started, the decode cluster will pre-allocate resources according to the initialization configuration of the scheduling module, and adjust them under the dynamic monitoring of the scheduling module, including occupying or releasing resources.

[0089] In some embodiments, when the decode cluster receives a service request sent by the first prefill cluster through the storage module, it assigns a task to the decode node to perform decode calculations.

[0090] In some embodiments, when a decode node completes a decode calculation, it performs a decode inference on a single token.

[0091] In some embodiments, when the decode node infers, it will use the cache information of the previous round of decoding, which is transmitted by the storage module.

[0092] In some embodiments, when the decode inference is completed, the inference result will be sent to the scheduling module, which will perform early stopping on the decode cluster according to user configuration and service request.

[0093] In some embodiments, the inference results output by the decode cluster will be transmitted to the L1 KV Cache GPU VRAM, and data access operations will be performed in the cache.

[0094] It is not difficult to find that this embodiment is a system embodiment corresponding to the method embodiment, and this embodiment can be implemented in conjunction with the method embodiment. The relevant technical details mentioned in the method embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method embodiment.

[0095] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0096] In order to facilitate those skilled in the art to better understand the above-mentioned reasoning method and reasoning system, the following will be combined with Figure 4 The reasoning system shown and the data interaction between the structures in the reasoning system are explained schematically.

[0097] like Figure 4 As shown, the reasoning system Figure 3Based on the above, the scheduling module is further divided into KV scheduling unit, prefill scheduling unit (prefill Controller) and decode scheduling unit (decode Controller), and the storage module is divided into the first cache (deployed on the CPU) and the second cache (deployed on the GPU). When applied, the request is first sent to the first cache on the CPU for token ids (i.e., a sequence of multiple token identifiers in a token sequence) matching, and the prefix of the matched token ids and the KV post-reasoning scheduler (KV Pool manager, i.e., KV scheduler) are pushed to the Prefill Controller (which manages the parent process of all prefill nodes (i.e., all P workers)) through zmq, and then the Prefill Controller will dynamically schedule and select the appropriate P worker, and pass the matched token ids to the selected P worker for calculation. In this process, the Prefill Controller first determines the scheduling strategy based on the current amount of resources. If the resources are sufficient and the load is evenly distributed, polling is used as the scheduling strategy. If the resources are tight, the minimum queue is used as the scheduling strategy. After selecting the appropriate P worker, the relevant data is passed to the corresponding P worker through multiprocessing.Queue(). After the Pworker completes the Prefill Forward calculation (i.e. the aforementioned prefill calculation), it will generate a series of KV parameters and their corresponding token ids (also known as Output ids), and store the generated data in the second cache. At the same time, the corresponding identifier is passed to the decode Controller for the decode Controller to dynamically schedule the D Worker through the load balancing strategy based on the NCCL Buffer. After the D Worker dynamically schedules the D Worker through the load balancing strategy based on the NCCL Buffer, the D Worker reads the corresponding data from the second cache based on the received identifier to perform the decode Forward calculation (i.e. the aforementioned decode calculation). After the D worker completes the Forward calculation until the Decode is completed, it updates the generated token ids and KV parameters to the second cache. The second cache synchronizes its updates to the first cache in real-time or periodic push.

[0098] It can be seen that the streaming method and reasoning system provided by the embodiment of the present application greatly improve the parallelization degree of the reasoning process by separating the prefill and decode stages, which are processed by different prefill clusters and decode clusters respectively. This design not only improves the efficiency of resource utilization, but also enhances the flexibility of the system, enabling it to better handle large-scale reasoning requests. In the case of introducing the prefill-decode separation technology as the basic system, a KV cache storage solution adapted to the prefill-decode separation technology is also proposed, that is, the radix-tree structure of Radixattention is used to effectively reduce memory access overhead and synchronization overhead. In the prefill stage, a dedicated cluster can more effectively manage the loading of model parameters and the processing of initial data, reducing the occupancy of computing power in the decode stage. For the decode stage, independent clusters can focus more on calculating complex generation tasks and make full use of the computing power advantages of the GPU. This architectural separation and parallel processing strategy not only reduces the resource bottleneck in the reasoning process, but also significantly reduces the response delay, thereby improving the performance of the overall system and user experience.

[0099] Another aspect of the present application embodiment further provides an electronic device, such as Figure 5 As shown, it includes: at least one processor 501; and a memory 502 that is communicatively connected to the at least one processor 501; wherein the memory 502 stores instructions that can be executed by the at least one processor 501, and the instructions are executed by the at least one processor 501 so that the at least one processor 501 can execute the reasoning method described in any of the above method embodiments.

[0100] The memory 502 and the processor 501 are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 501 and the memory 502 together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor 501 is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor 501.

[0101] The processor 501 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 502 can be used to store data used by the processor 501 when performing operations.

[0102] Another aspect of the present application embodiment further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the above method embodiment is implemented.

[0103] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0104] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A reasoning method, characterized in that: include: receiving a user request, wherein the user request carries input information; According to the input information, matching is performed in the cache to obtain a matching token sequence and its KV parameters; Perform reasoning based on the matched token sequence and its KV parameters and the input information; The token sequence and its KV parameters obtained by inference are stored in the cache, and the user request is responded to according to the inference result.

2. The inference method according to claim 1, characterized in that: The cache adopts a radix tree structure, and each edge in the radix tree structure corresponds to a token sequence and its KV parameter. The storing the inferred token sequence and its KV parameters in the cache includes: According to the token sequence and its KV parameter obtained by the reasoning, a new branch is generated for the edge corresponding to the matching token sequence and its KV parameter.

3. The inference method according to claim 2, characterized in that: The generating a new branch for the edge corresponding to the matched token sequence and its KV parameter according to the token sequence and its KV parameter obtained by the reasoning includes: Detecting whether the inferred token sequence has a common prefix with the token sequence in the cache, outside the matched token sequence; If the common prefix exists, the token sequence and its KV parameter corresponding to the common prefix are updated as the edge corresponding to the matching token sequence in the radix tree structure, and the token sequence and its KV parameter corresponding to the different parts are added as branches of the updated edge of the radix tree structure; If the common prefix does not exist, according to the token sequence and its KV parameter obtained by the inference, a new edge is added for the edge corresponding to the matching token and its KV parameter.

4. The inference method according to claim 2 or 3, characterized in that: The cache includes a first cache and a second cache, and both the first cache and the second cache adopt the radix tree structure; The storing the inferred token sequence and its KV parameters in the cache includes: During the inference process, the currently obtained token and its KV parameters are continuously stored based on the first cache; After the inference corresponding to the user request is completed, the update of the first cache is synchronized to the second cache.

5. The inference method according to claim 2 or 3, characterized in that: The method further comprises: In the case where data deletion is currently required in the cache, determining at least one token to be deleted; If there is a corresponding node that is a leaf node and after the corresponding node is deleted, the to-be-deleted token of a node with only one branch is generated, then the corresponding leaf node is deleted, and starting from the edge of the node with only one branch, it is merged with the edge of the parent node step by step upward until the node with only one branch disappears.

6. The inference method according to claim 5, characterized in that: The determining at least one token to be deleted includes: A token corresponding to the least recently used leaf node is determined as the at least one token to be deleted.

7. The inference method according to any one of claims 1 to 3, characterized in that: The reasoning according to the matched token sequence and its KV parameter and the input information includes: Call the prefill node in the prefill cluster, and start the operation from the matched token sequence and the KV parameter corresponding to the matched token sequence according to the input information; Call the decode node in the decode cluster to perform calculations based on the matched token sequence, the KV parameter corresponding to the matched token sequence, and the token identifier and the corresponding KV parameter calculated by prompt; Among them, the scheduling strategy of the prefill node in the prefill cluster includes: when the current resource idleness is greater than a first preset value, a polling scheduling mechanism is adopted; and / or, when the current resource idleness is less than a second preset value, a minimum queue scheduling mechanism is adopted; the scheduling strategy of the decode node in the decode cluster includes a load balancing strategy based on NCCL Buffer.

8. An inference system, characterized in that include: Scheduling module, prefill cluster, decode cluster and storage module; The storage module is connected to the scheduling module, the prefill cluster and the decode cluster respectively, so that the scheduling module, the prefill cluster and the decode cluster can perform data query and data storage, and the scheduling module is also connected to the prefill cluster and the decode cluster respectively; The scheduling module is used to receive a user request, wherein the user request carries input information; perform matching in the storage module according to the input information to obtain a matching token sequence and its KV parameter; and forward the matching token sequence and its KV parameter and the input information to a prefill node in the prefill cluster for prefill calculation, so as to notify the decode node in the decode cluster of the result calculated by the prefill node; The prefill cluster includes a plurality of prefill nodes, each of which is used to perform calculations based on the matching token sequence and its KV parameters issued by the scheduling module and the input information, and store the calculation results in the storage module; The decode cluster includes a plurality of decode nodes, each of which is used to read the result calculated by the prefill node from the storage module based on the notification of the scheduling module to perform decode calculation, and store the calculation result in the storage module.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the reasoning method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the inference method according to any one of claims 1 to 7 is implemented.

Images