Large language model inference method and system

Abstract

A large language model inference method and system are disclosed. The system includes a first data center and a second data center, the first data center deploying a pre-padding inference unit and a first decoding inference unit, and the second data center deploying a second decoding inference unit. The pre-padding inference unit performs pre-padding processing on an input sequence to obtain key-value cache data, and provides the key-value cache data to the first decoding inference unit and the second decoding inference unit via different communication paths, respectively. The first decoding inference unit performs decoding in advance to generate a first output sequence, and the processing and output process thereof at least partially overlaps with the process of transmitting the key-value cache data to the second decoding inference unit. The second decoding inference unit generates incremental key-value cache data according to the key-value cache data and the first output sequence information, and then generates a second output sequence. In this way, the transmission waiting and communication overhead across data centers can be reduced, and the resource utilization, scalability and system robustness can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to a method and system for reasoning using large language models. Background Technology

[0002] With the development of Large Language Models (LLMs), intelligent applications such as programming assistants and general chatbots are becoming increasingly popular. Due to the large parameter scale and high computational complexity of LLMs, their inference process consumes significant computing and storage resources. To meet the demands of high concurrency access and large-scale computing, cloud computing platforms typically rely on large data centers to provide online inference services.

[0003] LLM inference typically includes a prefill phase and a decode phase, which differ significantly in terms of computational intensity, memory access characteristics, and hardware requirements. To improve hardware utilization efficiency, related technologies usually adopt a deployment architecture that separates prefilling and decoding, configuring heterogeneous hardware resource pools suitable for the prefilling and decoding phases respectively. Key-value cache (KVCache) data is transferred between the two resource pools via a high-speed network within the data center.

[0004] However, the above deployment method requires pre-determining the size and ratio of the two types of resource pools. When the model structure, input / output length, or request load changes, the ratio between pre-filling load and decoding load will also change accordingly, easily leading to insufficient hardware resources in one type while idle hardware resources in the other. Adjusting the size or ratio of the resource pools usually involves large-scale hardware reconfiguration, which is costly and lacks flexibility.

[0005] To improve the reuse rate of existing heterogeneous hardware resources, related technologies have also attempted to combine hardware clusters located in different data centers for large language model inference. However, compared to intra-data center networks, cross-data center networks typically have lower bandwidth and higher transmission latency, which significantly increases the transmission time of key-value caches between pre-filled resources and decoding resources, thereby limiting the overall performance of cross-data center inference systems. Summary of the Invention

[0006] In view of this, this disclosure proposes an LLM inference method and system. Decoding is performed first on a RelayDecode (RLD) inference unit co-located with the Prefill (P) inference unit to generate a pre-output token sequence. Simultaneously, a key-value cache is transmitted across data centers to the MainDecode (MD) inference unit. Thus, the local decoding process and the transmission of the key-value cache to the remote end can be overlapped, effectively hiding the latency introduced by cross-data center Ethernet transmission. After the key-value cache transmission is complete, the subsequent decoding process is switched to the MD unit, which continues to generate the subsequent output token sequence.

[0007] According to a first aspect of this disclosure, an LLM inference method is provided, applied to an LLM inference system including a first data center and a second data center, the first data center deploying a pre-filled inference unit and a first decoding inference unit, and the second data center deploying a second decoding inference unit. The method includes: performing a pre-filling process on an input sequence for LLM inference by the pre-filled inference unit to obtain key-value cache data of the input sequence; providing the key-value cache data to the first decoding inference unit via a first communication path, and providing the key-value cache data to the second decoding inference unit via a second communication path; performing a first decoding process by the first decoding inference unit based on the key-value cache data received via the first communication path to generate a first output sequence and outputting first inference content based on the first output sequence, wherein the first decoding process and the output of the first inference content at least partially overlap with the transmission process of providing the key-value cache data to the second decoding inference unit via the second communication path; and transmitting first output sequence information to the second decoding inference unit such that the second decoding inference unit generates a second output sequence based on the key-value cache data and the first output sequence and outputs second inference content based on the second output sequence.

[0008] Optionally, providing the key-value cache data to the first decoding inference unit via a first communication path and to the second decoding inference unit via a second communication path includes: determining whether the input sequence satisfies the relay decoding condition; and in response to the input sequence satisfying the relay decoding condition, providing the key-value cache data to the first decoding inference unit via the first communication path and to the second decoding inference unit via the second communication path.

[0009] Optionally, the method further includes: in response to the input sequence not satisfying the relay decoding condition, providing the key-value cache data to the second decoding inference unit only via the second communication path, so that the second decoding inference unit performs decoding processing based on the key-value cache data and outputs the corresponding inference content.

[0010] Optionally, transmitting first output sequence information to the second decoding inference unit so that the second decoding inference unit generates a second output sequence based on the key-value cache data and the first output sequence and outputs second inference content based on the second output sequence includes: transmitting first output sequence information for characterizing the first output sequence to the second decoding inference unit so that after obtaining the key-value cache data via the second communication path, the second decoding inference unit obtains incremental key-value cache data corresponding to the first output sequence based on the first output sequence information, and performs a second decoding process based on the key-value cache data and the incremental key-value cache data to generate a second output sequence and subsequently output the second inference content obtained based on the second output sequence.

[0011] Optionally, in response to a predetermined stopping condition being met, the first decoding inference unit stops executing the first decoding process and provides the second decoding inference unit with first output sequence information characterizing all output tokens generated by the first decoding inference unit before stopping the execution of the first decoding process.

[0012] Optionally, during the process of the second decoding inference unit generating incremental key-value cache data based on the first output sequence information, the first decoding inference unit continues to perform the first decoding process to expand the first output sequence and continues to provide the second decoding inference unit with the lexical information corresponding to the newly generated output lexical in the first output sequence; in response to the sequence length of the currently generated first output sequence and the sequence length corresponding to the incremental key-value cache data not exceeding a predetermined threshold, the first decoding inference unit stops performing the first decoding process, and the second decoding inference unit performs the second decoding process.

[0013] Optionally, the method further includes: determining the current resource usage state of the first decoding inference unit; and based on the current resource usage state, performing at least one of the following: adjusting the judgment threshold used to determine whether the input sequence meets the relay decoding condition; adjusting the maximum duration of the first decoding inference unit performing the first decoding process for a single input sequence or the upper limit of the number of output tokens generated; and selecting a handover mode between a one-time handover mode and a catch-up handover mode to hand over the decoding task from the first decoding inference unit to the second decoding inference unit, wherein the current resource usage state includes the number of concurrent requests of the first decoding inference unit, the memory usage, etc. The first decoding inference unit transfers the decoding task and transfers the first output sequence information to the second decoding inference unit in one go, wherein, in the one-time handover mode, the first decoding inference unit transfers the decoding task and transfers the first output sequence information to the second decoding inference unit in one go; in the catch-up handover mode, the first decoding inference unit continues to execute the first decoding process while the second decoding inference unit generates the incremental key-value cache data corresponding to the first output sequence, and transfers the decoding task to the second decoding inference unit in response to the number of output tokens corresponding to the incremental key-value cache data catching up with the number of output tokens currently generated by the first decoding inference unit.

[0014] According to a second aspect of this disclosure, an LLM inference system is provided, comprising a first data center and a second data center. The first data center is equipped with a pre-filled inference unit and a first decoding inference unit, wherein the pre-filled inference unit is deployed on a pre-filled inference server, and the first decoding inference unit is deployed on the first decoding inference server. The second data center is equipped with a second decoding inference unit, and the second decoding inference unit is deployed on the second decoding inference server. The pre-filled inference unit communicates with the first decoding inference unit via a first communication path, and the pre-filled inference unit communicates with the second decoding inference unit via a second communication path. The pre-filled inference unit is configured to: perform pre-filling processing on an input sequence for LLM inference to obtain key-value cache data corresponding to the input sequence; and provide the key-value cache data to the first decoding inference unit via the first communication path, and provide the key-value cache data to the second decoding inference unit via the second communication path. The unit is configured to: perform a first decoding process based on the key-value cache data received via the first communication path to generate a first output sequence, and output a first inference content obtained based on the first output sequence, wherein the first decoding process and the output process of the first inference content at least partially overlap with the transmission process of providing the key-value cache data to the second decoding inference unit via the second communication path; and provide the second decoding inference unit with first output sequence information for characterizing the first output sequence; and the second decoding inference unit is configured to: generate incremental key-value cache data corresponding to the first output sequence based on the key-value cache data received via the second communication path and the first output sequence information; and perform a second decoding process based on the key-value cache data and the incremental key-value cache data to subsequently generate a second output sequence, and output a second inference content obtained based on the second output sequence.

[0015] Optionally, the pre-filled inference server includes a first type of hardware accelerator for performing the pre-filled process, the first decoding inference server includes a second type of hardware accelerator for performing the first decoding process, and the computing power of the first type of hardware accelerator is higher than that of the second type of hardware accelerator, and at least one of the storage capacity and storage bandwidth of the second type of hardware accelerator is higher than that of the first type of hardware accelerator.

[0016] Optionally, the number of the third type of hardware accelerators used to perform the second decoding process in one of the second data centers is greater than the number of the second type of hardware accelerators used to perform the first decoding process in one of the first data centers.

[0017] Optionally, the LLM inference system includes multiple first data centers, each of the multiple first data centers being connected to the second data center via its own second communication path, and sharing multiple second decoding inference units deployed in the second data center.

[0018] Therefore, the LLM inference method and system disclosed herein, through a three-level separation architecture of P-RLD-MD, enables the RLD inference unit to decode the key-value cache data during cross-datacenter transmission, and the MD inference unit to restore the incremental key-value cache locally based on the output lexical identifier, thereby reducing transmission waiting time and communication overhead. Combined with the small heterogeneous RLD resource requirements and the MD-side prefix cache, deployment costs can also be reduced, and the system's resource utilization, scalability, and robustness can be improved. Attached Figure Description

[0019] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0020] Figure 1 An example of a heterogeneous pre-filled-decoder separation architecture within a data center is shown.

[0021] Figure 2 An example of a heterogeneous pre-filled-decode separation architecture across data centers is shown.

[0022] Figure 3 A schematic flowchart of a large language model reasoning method according to some embodiments of the present disclosure is shown.

[0023] Figure 4 A three-tier cross-datacenter pre-fill-decode separation architecture and its operation process are illustrated according to some embodiments of the present disclosure.

[0024] Figure 5 The relevant cross-datacenter pre-filling-decoding separation scheme is illustrated, as well as two decoding task handover modes that can be adopted in this disclosure.

[0025] Figure 6 A schematic diagram of the composition of an LLM inference system according to some embodiments of the present disclosure is shown. Detailed Implementation

[0026] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0027] As used in this specification and the appended claims, the singular expressions “a,” “the,” “the,” and “the” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. The term “comprising” and its variations, as used herein, indicate an open-ended inclusion, i.e., “including but not limited to.” Unless specifically stated otherwise, the term “or” means “and / or.” The term “according to” means “at least in part according to.” The terms “an example embodiment” and “an embodiment” mean “at least one example embodiment.” The term “another embodiment” means “at least one additional embodiment.” The terms “first,” “second,” etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0028] During LLM inference, user-inputted text and commands, combined with information such as historical dialogues, search results, or tool-returned content, are organized into an input sequence for LLM inference. This input sequence is then lexicalized to form a lexical sequence containing multiple tokens. Each token can be represented by a corresponding token ID and further converted into a vector representation that can be processed by a large language model.

[0029] The inference process for an input lexical sequence includes a pre-filling stage and a decoding stage. In the pre-filling stage, the LLM computes multiple lexical units in the input sequence and generates key-value cache data corresponding to the input sequence. This key-value cache data stores the key vectors and value vectors obtained during attention computation for reuse in the subsequent decoding stage. In the decoding stage, the LLM generates output lexical units progressively in an autoregressive manner based on the key-value cache data. Each time a new output lexical unit is generated, the LLM also generates new key-value cache data corresponding to that output lexical unit and appends it to the existing key-value cache data. Thus, the LLM can continue to generate subsequent output lexical units based on the input sequence and previously generated output lexical units.

[0030] The pre-filling and decoding stages have different computational characteristics. The pre-filling stage requires large-scale parallel computation on multiple tokens in the input sequence, placing high demands on hardware computing power. The decoding stage, on the other hand, requires repeated reading of model parameters and key-value cache data, placing high demands on storage capacity, storage bandwidth, and data access efficiency. Therefore, pre-filling and decoding are best performed by different types of hardware accelerators. Based on these differences, a heterogeneous pre-filling-decoding separation architecture has been proposed, employing a first type of hardware accelerator for pre-filling and a second type for decoding. By using different hardware for their more suitable inference stages, overall inference efficiency can be improved.

[0031] However, in a heterogeneous pre-filled-decode (PD or PD) separation architecture within a data center (DC), it is usually necessary to build a large-scale pre-filled resource pool and a decoding resource pool simultaneously. Figure 1 This illustrates an example of a heterogeneous pre-filling / decoding separation architecture within a data center. The pre-filling resource pool handles the overall pre-filling load of the system, and the decoding resource pool handles the overall decoding load. Therefore, both types of resource pools need to be configured with a corresponding number of hardware devices according to the expected maximum or typical business load. Furthermore, an appropriate configuration ratio needs to be maintained between pre-filling and decoding resources. This ratio is related to factors such as model structure, input sequence length, output sequence length, number of concurrent requests, and cache hit rates. Figure 1 As shown, the DC is configured with the same number of P inference units (i.e., "pre-filled inference units", which can also be referred to as "P nodes") and D inference units (i.e., "decoding inference units", which can also be referred to as "D nodes"). The P nodes and D nodes are interconnected through a high-speed network within the DC. For example, they can be interconnected through RDMA (Remote Direct Memory Access), meaning that all nodes are in the RDMA domain.

[0032] When the service model or business load changes, the ratio between pre-filled load and decoding load may also change, leading to an imbalance in the original hardware configuration. For example, pre-filled resources may have reached their processing capacity limit, while some decoding resources remain idle; conversely, there may be insufficient decoding resources, resulting in some pre-filled resources not being fully utilized. Since traditional decoding resource pools need to handle the entire decoding load, an imbalance in resource allocation can lead to a large-scale idle of hardware resources. For example... Figure 1As shown, half of the D nodes are in an "underutilized" state. Therefore, building a complete heterogeneous pre-filled resource pool and decoding resource pool within the same data center not only requires high initial construction costs, but also makes it difficult to flexibly adapt to continuous changes in models and business loads.

[0033] To avoid rebuilding a large-scale heterogeneous resource pool as much as possible, the proposed solutions attempt to reuse hardware clusters already deployed in different data centers. Figure 2 An example of a heterogeneous pre-filled-decoder separation architecture across data centers is shown. Figure 2 As shown, pre-filling processing can be performed using the existing P cluster in data center A, and decoding processing can be performed using the existing D cluster in data center B. Thus, without reconfiguring large-scale hardware resources capable of handling the full decoding load within data center A, pre-filling and decoding processing can be performed separately using different types of hardware.

[0034] However, in this cross-datacenter architecture, the key-value cache data generated by the pre-filling process needs to be transmitted across the cross-DC network. High-bandwidth, low-latency communication networks are typically used within the DC, such as... Figure 2 As shown, the P nodes within data center A are in the same RDMA domain, and the D nodes within data center B are also in the same RDMA domain. Different data centers are typically interconnected via wide area Ethernet, such as TCP (Transmission Control Protocol) connections. Compared to intra-data center networks, cross-data center networks have lower bandwidth and higher transmission latency. Therefore, when the amount of key-value cache data to be transmitted is large, the D nodes in data center B need to experience a long waiting time before they can begin decoding, which may cause Service Level Objectives (SLOs) such as TTFT (Time To First Token) and E2EL (End-to-End Latency) to fail to meet the targets.

[0035] In view of this, this disclosure proposes an LLM inference method. This method generates a pre-output token sequence by performing decoding on a RelayDecode (RLD) inference unit co-located with the Prefill (P) inference unit; simultaneously, the key-value cache is transferred across data centers to the MainDecode (MD) inference unit. Thus, the local decoding process and the process of transferring the key-value cache to the remote end can be overlapped, thereby masking at least part of the latency introduced by the cross-data center Ethernet transmission. After the key-value cache transfer is complete, the subsequent decoding process can be switched to the MD, which continues to generate the subsequent output token sequence.

[0036] Figure 3 A schematic flowchart of a large language model inference method according to some embodiments of the present disclosure is shown. This large language model inference method can be applied to an LLM inference system including a first data center and a second data center. The first data center is deployed with pre-filled inference units and a first decoding inference unit, and the second data center is deployed with a second decoding inference unit. The pre-filled inference unit may correspond to the aforementioned P inference unit, and may also be referred to as a "P node"; the first decoding inference unit may correspond to the aforementioned RLD inference unit, and may also be referred to as an "RLD node"; and the second decoding inference unit may correspond to the aforementioned MD inference unit, and may also be referred to as an "MD node". Accordingly, the first data center may be referred to as a "P / RLD data center", and the second data center may be referred to as an "MD data center". The above-described inference architecture including P nodes, RLD nodes, and MD nodes can be referred to as a P-RLD-MD three-level architecture.

[0037] Figure 4 A three-tier cross-datacenter pre-fill-decode separation architecture and its operation process are illustrated according to some embodiments of this disclosure. In one embodiment, the P / RLD data center may be a main data center that primarily undertakes pre-fill processing and is configured with relay decoding resources, while the MD data center may be another data center geographically separated from the P / RLD data center. For example, the MD data center may be a large data center deployed far from densely populated areas. For ease of understanding, the following will be combined with... Figure 4 right Figure 3 The steps of the reasoning method shown are explained.

[0038] In this disclosure, an "inference unit" is a logical entity defined according to its function in the LLM inference process, and can also be referred to as a "node" in the software sense. For example, P nodes, RLD nodes, and MD nodes represent logical nodes that perform pre-filling processing, relay decoding processing, and main decoding processing, respectively. Here, "node" is mainly used to distinguish different functional roles and does not necessarily correspond to a separate physical server or a separate hardware accelerator.

[0039] In one embodiment, an inference unit can be implemented through an inference instance. The "inference instance" refers to the specific running entity of the corresponding inference function in the actual operating environment. For example, a pre-fill instance can be configured to receive an input sequence and perform pre-filling processing, while a decoding instance can be configured to perform decoding processing based on key-value cached data. In engineering implementations, an inference instance can typically be represented as a running copy of a service and can be hosted by a process, container, virtual machine, or other independently deployable and schedulable software entity.

[0040] Additionally, it should be noted that the processing performed by both the RLD inference unit and the MD inference unit is essentially LLM decoding, and both can generate output terms in an autoregressive manner based on key-value cache data. "Relay decoding" and "main decoding" primarily distinguish the deployment locations of the two decoding inference units and their functional roles in the corresponding inference processes, and do not indicate that they perform different types of model computations. Specifically, the RLD inference unit can be used to perform decoding processing in advance during the transfer of key-value cache data across data centers, while the MD inference unit can be used to undertake the main decoding processing for the corresponding inference request, and can continue the subsequent decoding processing from the output sequence already generated by the RLD inference unit.

[0041] The first data center can employ a high-bandwidth, low-latency communication network. Therefore, P nodes and RLD nodes can communicate via the first communication path, for example via... Figure 4 The RDMA communication path shown is used for communication. The bandwidth of the RDMA communication path can be as high as, for example, 8. 400Gbps. The P node and MD node are located in different data centers, but they can communicate across data centers via a second communication path, for example, through... Figure 4 The diagram illustrates communication via a wide-area Ethernet connection based on TCP. The communication bandwidth of wide-area Ethernet is significantly smaller than that within a data center (DC), even a dedicated line only has a bandwidth of approximately 2... 10Gbps. In this embodiment, since the first communication path is an intra-data center communication path, while the second communication path is an inter-data center communication path, the transmission latency of the first communication path is lower than that of the second communication path. For example, in some application scenarios, the transmission latency of the first communication path is typically several milliseconds or tens of milliseconds, while the transmission latency of the second communication path is typically several seconds or even tens of seconds.

[0042] In the case of performing inference operations, for example, it can be done by Figure 4 The HTTP server shown retrieves the user's current input text or command, as well as relevant information such as historical conversations, search results, or tool-returned content. This information can be combined and lexicalized to form an input lexical sequence for LLM inference, also known as an input sequence. This input sequence is first provided to the P node for pre-filling, and then the corresponding decoding node performs decoding based on the key-value cache data generated by the pre-filling process.

[0043] In step S310, the pre-filled inference unit can perform pre-filling processing on the input sequence used for LLM inference to obtain key-value cache data corresponding to the input sequence.

[0044] After obtaining the key-value cache data, key-value cache data transmission to the RLD node and MD node can be initiated in parallel, or the two transmission processes can at least partially overlap. Therefore, in step S320, the key-value cache data is provided to the RLD node via a first communication path and to the MD node via a second communication path.

[0045] Because the transmission delay of the first communication path is lower than that of the second communication path, the RLD node can obtain the key-value cache data required for decoding processing before the MD node and start the decoding operation first. Therefore, in step S330, the RLD node performs the first decoding process based on the key-value cache data received via the first communication path to generate a first output sequence and output the first inference content obtained based on the first output sequence.

[0046] Specifically, before the transmission of key-value cache data to the MD node via the second communication path is completed, the RLD node can already generate a first output sequence based on the key-value cache data received via the first communication path and provide the corresponding first inference content to the user or downstream processing module. Therefore, the process of the RLD node performing the first decoding process and outputting the first inference content at least partially overlaps with the process of transmitting key-value cache data to the MD node via the second communication path. Thus, the generation of the first output sequence and the output of the corresponding first inference content can cover at least part of the waiting time generated by the transmission of key-value cache data via the second communication path, thereby reducing the impact of cross-data center transmission on first-word latency and end-to-end latency.

[0047] Subsequently, in step S340, the MD node is provided with first output sequence information so that the MD node can generate a second output sequence based on the key-value cache data and the first output sequence information, and output the second inference content based on the second output sequence.

[0048] Specifically, the first output sequence information is used to characterize the first output sequence generated by the RLD node. For example, it may include the token identifiers corresponding to each token in the first output sequence, i.e., the token IDs, and the generation order of these tokens; or it may include a token ID sequence composed of multiple token IDs; it may be the tokens themselves; or other information that can be used to reconstruct the first output sequence (i.e., all tokens generated by the RLD node and the order in which they were generated), such as token IDs and generation timestamps, vocabulary indexes, or even the character sequences corresponding to each token. For instance, the first output sequence information may include the token IDs corresponding to all tokens in the first output sequence generated by the RLD node up to the time of decoding task handover, and their generation order.

[0049] After the MD node obtains the key-value cache data corresponding to the input sequence via the second communication path, it can perform corresponding incremental calculations on the tokens in the first output sequence based on the first output sequence information to generate incremental key-value cache data corresponding to the first output sequence. Unlike the standard autoregressive decoding process, which predicts the next token step by step based on the current token, in the above incremental calculation process, the tokens in the first output sequence have already been determined by the RLD node. The MD node can process multiple known token IDs in batches or in segments, calculating and filling the incremental key-value cache data corresponding to these tokens without regenerating them. This incremental calculation process can be called extended-decode processing.

[0050] Therefore, the MD node can obtain the key-value cache state corresponding to the decoding state after the RLD node generates the first output sequence. Subsequently, the MD node can perform a second decoding process based on the key-value cache data corresponding to the input sequence and the incremental key-value cache data corresponding to the first output sequence to continue generating the second output sequence and output the second inference content obtained based on the second output sequence.

[0051] By transmitting the first output sequence information and having the MD node generate the corresponding incremental key-value cache data locally, it is unnecessary to transmit the incremental key-value cache data generated by the RLD node during the first decoding process across data centers to the MD node. The first output sequence information can be, for example, a token ID sequence composed of multiple token IDs, and its data volume is usually significantly smaller than the corresponding incremental key-value cache data volume, thus significantly reducing the cross-data center communication overhead during the decoding task handover process.

[0052] In one embodiment, the token ID sequence residing in the CPU (Central Processing Unit) memory can be sent directly via TCP-based network communication, thus eliminating the step of copying it from accelerator memory to CPU memory before sending. Since the token ID sequence is relatively small, its transmission time is negligible or only accounts for a small proportion of the cross-datacenter transmission time of key-value cache data.

[0053] exist Figure 4In the illustrated embodiment, the first output sequence corresponds to the sequence of eager output tokens generated first by the RLD node, and the second output sequence corresponds to the sequence of tail output tokens generated subsequently by the MD node. Both the first and second output sequences are token sequences consisting of one or more output tokens. The second output sequence is generated after the first output sequence and constitutes the subsequent sequence of the first output sequence.

[0054] After the MD node takes over the decoding process, it can continuously expand the second output sequence until the decoding termination condition is met. The decoding termination condition may include, for example, the generation of the sequence end terminator, reaching a predetermined maximum output length, or meeting other stopping conditions. Therefore, the second output sequence may include output terms generated by the MD node up to the end of the decoding process for the current inference request.

[0055] After the MD node takes over the decoding process, it can continuously expand the second output sequence until the decoding termination condition is met. The decoding termination condition may include, for example, generating the end word of the sequence, reaching a predetermined maximum output length, or meeting other stopping conditions. Therefore, the second output sequence may include output words generated by the MD node until the decoding process of the current inference request ends. The first and second output sequences, in the order of word generation, together constitute the complete output sequence corresponding to the current input sequence, and the decoding output of the current inference request is completed when the second output sequence is fully generated.

[0056] The first inference content is derived from the first output sequence, and the second inference content is derived from the second output sequence. Both can be provided sequentially to the user or downstream processing modules, together forming a complete inference result for the current input sequence. For example, in a text generation scenario, the output tokens in the first and second output sequences can be arranged in the generation order to form a complete output token sequence, which is then converted into natural language text through detokenization. The first part of the text converted from the first output sequence can be displayed to the user as the first inference content, and the second part of the text converted from the second output sequence can be displayed to the user as the second inference content, thus together forming a complete text response for the current input sequence.

[0057] For example, in agent application scenarios, the first and second output sequences can be used to generate program code, tool call instructions, application programming interface (API) call parameters, or other structured operation information. The corresponding output tokens can be de-lexicalized and parsed to form programs or instructions executable by the agent system. In this case, the first and second inference content do not need to be directly displayed to the user, but can serve as intermediate results for the agent to perform subsequent operations. For example, the agent can call external tools based on the generated program or tool call instructions, and use the tool execution results for subsequent inference or to generate the final content returned to the user. In this scenario, the completion of the second output sequence indicates the completion of the current LLM decoding output, but does not necessarily mean that the entire agent task, including tool execution and subsequent LLM calls, has been completed.

[0058] In summary, in the above embodiments, the RLD node can quickly obtain key-value cache data using the low-latency communication path within the data center, and generate output tokens while the key-value cache data is still being transmitted across data centers to the MD node. This allows the first decoding process and the cross-data center transmission process to at least partially overlap, thereby reducing the impact of cross-data center transmission on the first token latency and end-to-end latency. After receiving the key-value cache data, the MD node can generate corresponding incremental key-value cache data locally based on the first output sequence information with a smaller data volume, and then continue to perform subsequent decoding processing, thus eliminating the need for cross-data center transmission of the incremental key-value cache data generated by the RLD node.

[0059] In some embodiments, steps S320 to S340 described above can be performed for each input sequence to utilize the advance decoding of the RLD node to cover at least part of the cross-data center transmission latency of the key-value cache data. In this case, a relatively large number of relay decoding resources need to be deployed in the first data center to meet the requirement of relay decoding for each input sequence. To control the resource consumption of the RLD node, the RLD node can be limited to generating only a predetermined number of output tokens for each input sequence, or performing decoding processing only within a predetermined time and subsequently handing over the decoding processing to the MD node.

[0060] In other embodiments, considering that the actual key-value cache data to be transmitted for each input sequence is different, it can be determined first whether the input sequence meets the relay decoding conditions, and steps S320 to S340 are executed only when the relay decoding conditions are met. The relay decoding conditions are used to determine whether to enable the RLD node to perform relay decoding, so that the decoding processing and inference content generation of the RLD node at least partially overlaps with the process of transmitting key-value cache data to the MD node.

[0061] The relay decoding conditions can be used to reflect the cross-data center transmission volume of key-value cache data corresponding to the current input sequence, or the expected transmission delay. The relay decoding conditions can be determined to be met when the amount of data to be transmitted exceeds a predetermined volume threshold, the expected transmission delay exceeds a predetermined delay threshold, or there is a risk of SLO (Site Default Logging) failure. Specifically, the determination can be made based on one or more of the following: prefix cache hit rate of the input sequence, input sequence length, number of cache misses, amount of key-value cache data to be transmitted, available bandwidth of the second communication path, or expected transmission delay.

[0062] The inventors discovered that the cross-datacenter transmission latency varies depending on the inference request. When the relevant historical key-value cache data is already cached on the decoding side, only the key-value cache data corresponding to the missed parts needs to be transmitted. This characteristic is particularly pronounced in agent workloads characterized by long contexts, high prefix cache hit rates, and short outputs. For requests with high prefix cache hit rates, the amount of data to be transmitted is small, and transmission can be completed quickly. For requests with low prefix cache hit rates, long input sequences, or large amounts of data to be transmitted, the transmission latency may increase significantly. In other words, the severe latency caused by cross-datacenter transmission is concentrated in some requests, rather than evenly distributed across all requests.

[0063] Based on the above understanding, only a small number of relay decoding resources can be configured in the first data center, and these resources can be used for requests with longer transmission latency across data centers. This reduces the impact of long transmission wait times on inference output and also minimizes the consumption of limited RLD resources by requests with shorter transmission latency.

[0064] Therefore, in one embodiment, the succession decoding condition can be determined based on the prefix cache hit rate of the input sequence. Here, "prefix" refers to a sequence of consecutive terms in the input sequence, starting from the beginning. Different inference requests may include the same or partially the same input prefix. For example, different requests may include the same system prompts, historical dialogues, fixed task descriptions, or shared document content. For input prefixes that have already been processed, the inference system can save their corresponding key-value cache data (e.g., in the cache pools corresponding to the P node and MD node). If subsequent requests include the same input prefix, the already saved key-value cache data can be reused directly without recalculating or retransmitting that portion of the key-value cache data. The cache used to save and reuse this historical key-value cache data can be called a prefix cache.

[0065] Prefix cache hit rate represents the proportion of the current input sequence that can be reused using the prefix cache. For example, the prefix cache hit rate can be determined as the ratio of the number of tokens in the longest prefix that matches the current input sequence to the total number of tokens in the current input sequence. A higher prefix cache hit rate indicates that more tokens in the current input sequence can reuse existing key-value cache data, and less key-value cache data needs to be regenerated or transmitted across data centers; a lower prefix cache hit rate indicates that the input sequence portion with cache misses is longer, and the amount of key-value cache data that needs to be transmitted is usually larger.

[0066] like Figure 4 As shown, a prefix buffer pool connected to the P node, a decoding Radix buffer pool (i.e., the first decoding-side buffer pool) connected to the RLD node, and a decoding Radix buffer pool (i.e., the second decoding-side buffer pool) connected to the MD node can be set up separately. These buffer pools can be used to store key-value cache data corresponding to one or more historical input sequences or historical output sequences. The P node can reuse the key-value cache data calculated in the pre-filling stage according to the prefix buffer pool it is connected to, while the RLD node and MD node can reuse the historical key-value cache data required for decoding processing according to their respective connected decoding Radix buffer pools.

[0067] like Figure 4 As shown, the P node can connect to the prefix cache pool, while the RLD and MD nodes can connect to the decoder-side cache pools respectively. The decoder-side cache pool can employ a radix cache and organize the key-value cache data according to the prefix relationships of the word sequences. By performing longest prefix matching on a new input sequence, the reusable key-value cache data and cache misses can be determined.

[0068] The cache pools connected to the P node, RLD node, and MD node can be independent of each other, and their cache contents can be the same or different. In one embodiment, the coverage of the term prefixes already cached on the MD node side can correspond to the prefix cache on the P node side. For example, for a request with a particularly long input sequence and a particularly short output sequence, the RLD node may complete the decoding process before the key-value cache data is completely transmitted to the MD node. Even if the decoding process of the current request has been completed, the remaining key-value cache data can still be transmitted to the MD node and written to its cache, so that the MD node can reuse as much existing key-value cache data locally as possible when processing subsequent requests with the same or similar input prefixes. In contrast, since the P node and RLD node are connected via a high-speed, low-latency communication path such as RDMA, the cache of the RLD node does not need to be consistent with the cache of the P node.

[0069] It should be understood that in embodiments where prefix caching is enabled, the P-node does not need to re-perform pre-padding for all tokens in the input sequence. For the portion of the input sequence where the prefix cache hits, the P-node can directly reuse existing key-value cache data and perform pre-padding only for the cache-missed portion to generate new key-value cache data corresponding to the cache-missed portion. Therefore, the key-value cache data obtained or generated by the P-node as described in this disclosure may only include the key-value cache data corresponding to the prefix cache-missed portion of the input sequence, and does not necessarily include the complete key-value cache data corresponding to all tokens in the input sequence.

[0070] Similarly, if the RLD node or MD node has already cached key-value cache data corresponding to a portion of the prefix of the input sequence, the key-value cache data provided to the corresponding node via the first or second communication path may only include the key-value cache data corresponding to the cache miss portion of that node. The corresponding node can combine the locally cached hit portion with the received miss portion to form the key-value cache state required for performing decoding processing.

[0071] Since this disclosure is primarily used to reduce the latency caused by key-value cache data being transmitted to the MD node via the second communication path, in one embodiment, the prefix cache hit rate of the current input sequence can be determined based on the decoder-side radix cache to which the MD node is connected. Specifically, longest prefix matching can be performed on the current input sequence in the decoder-side radix cache of the MD node, the prefix cache hit rate can be determined based on the length of the matching prefix, and the amount of key-value cache data that needs to be transmitted via the second communication path can be estimated accordingly.

[0072] When the prefix cache hit rate is lower than the predetermined hit rate threshold, it indicates that a large portion of the current input sequence requires the transmission of key-value cache data across data centers, and the corresponding transmission process may result in a long waiting time. At this time, it can be determined that the input sequence meets the relay decoding condition, and the P-RLD-MD three-level processing path is enabled, i.e., steps S320~S340 above are executed.

[0073] If the prefix cache hit rate is not lower than a predetermined hit rate threshold, it can be determined that the input sequence does not meet the relay decoding conditions, and the key-value cache data corresponding to its cache miss portion is only provided to the MD node via the second communication path. The MD node can combine the received key-value cache data with the existing key-value cache data in its decoding-side cardinality cache, and perform decoding processing based on the combined key-value cache data to generate and output the inference content corresponding to the current input sequence.

[0074] Therefore, RLD nodes can be selectively activated based on the cross-data center transmission requirements corresponding to different input sequences. By concentrating limited RLD resources on requests with low hit rates, large data transmission volumes, or high SLO default risks, the impact of long-tail cross-data center transmission latency can be reduced while decreasing the configuration scale of relay decoding resources.

[0075] After determining that the RLD node is enabled for the current input sequence, the decoding task needs to be transferred from the RLD node to the MD node at an appropriate time. Figure 5 The relevant cross-datacenter pre-filling-decoding separation scheme is illustrated, as well as two decoding task handover modes that can be adopted in this disclosure.

[0076] Figure 5 Each “RLD” and “MD” box in the diagram represents a single autoregressive decoding step performed by the corresponding inference unit to generate an output token. The “Extend-Decode” box represents extended decoding, where the MD node performs a one-time incremental cache computation operation on a set of known prior output tokens. The number of tokens shown in the diagram is merely an example illustrating the relationship between different processing stages and does not constitute a limitation on the actual length of the output sequence.

[0077] like Figure 5 As shown in section (a), in the relevant cross-DC PD-separated inference scheme, the P node first performs pre-padding on the input sequence. The key-value cache data generated by the pre-padding is then transmitted across the DC to the MD node (e.g., via Ethernet). The MD node only begins generating output tokens after receiving the key-value cache data required for decoding. For example, assuming the current inference request generates a total of 1K output tokens, the MD node needs to wait for the key-value cache data transmission to complete before generating all 1K output tokens sequentially. Therefore, the cross-datacenter transmission time of the key-value cache data is directly added to TTFT and E2EL and cannot be masked by the output token generation process.

[0078] In contrast, in this disclosure, the decoding process can be performed first by the RLD node, which is co-located with the P node, while the key-value cache data is being transmitted to the MD node via the second communication path. Thus, the process of the RLD node generating the preceding output tokens at least partially overlaps with the cross-datacenter transmission of the key-value cache data, thereby masking at least part of the cross-datacenter transmission delay. After the corresponding handover conditions are met, the MD node then continues to perform the subsequent decoding process. Figure 5 Parts (b) and (c) show two different decoding task handover modes, respectively.

[0079] like Figure 5As shown in section (b), in some embodiments, a one-shot handoff mode may be employed. In this mode, the RLD node hands over the decoding task and transfers the first output sequence information to the MD node "all at once". Specifically, the RLD node performs a first decoding process based on key-value cache data obtained via a first communication path and generates a first output sequence including N antecedent output tokens. In response to the satisfaction of a predetermined stopping condition, for example, when the MD node has completed receiving the key-value cache data, the RLD node can stop performing the first decoding process and provide the MD node with the first output sequence information, such as the token ID sequence corresponding to the aforementioned N antecedent output tokens.

[0080] After the RLD node stops the first decoding process, it can release the computing and memory resources occupied by the current inference request. The MD node then performs extended decoding based on the received token ID to generate incremental key-value cache data corresponding to the aforementioned N preceding output tokens. After obtaining the corresponding incremental key-value cache data, the MD node performs the second decoding process based on the key-value cache data corresponding to the input sequence and the incremental key-value cache data corresponding to the first output sequence, continuing to generate the second output sequence after the first output sequence. For example, when the total output length is 1K tokens, the MD node can continue to generate the remaining 1K-N subsequent output tokens.

[0081] Therefore, the time it takes for the RLD node to generate N preceding output tokens at least partially covers the cross-datacenter transmission time of the key-value cache data, thereby reducing the TTFT and E2EL of the current inference request. Simultaneously, since the RLD node stops decoding after meeting predetermined stopping conditions, its continuous occupation of computing and memory resources is relatively short. However, if the extended decoding process takes a long time, the output process may experience brief pauses or "output freezes," leading to periodic fluctuations in OTPS (Output Tokens Per Second).

[0082] like Figure 5As shown in section (c), in some embodiments, a catch-up handoff mode may be employed. Unlike the one-time handoff mode, in this mode, the RLD node continues to perform the first decoding process while the MD node generates the incremental key-value cache data corresponding to the first output sequence. In response to the number of output tokens corresponding to the incremental key-value cache data catching up with the number of output tokens currently generated by the first decoding inference unit (e.g., the number of output tokens corresponding to the incremental key-value cache data is the same as, or differs by no more than, a predetermined threshold from, the decoding task is then handed over to the MD node. That is, after the MD node begins performing extended decoding processing, the RLD node does not immediately stop the first decoding process, but continues to generate new output tokens, causing the first output sequence to continue expanding until the incremental key-value cache data generated by the MD node performing extended decoding processing can catch up with the token generation progress of the first output sequence.

[0083] Specifically, the RLD node can first generate N antecedent output tokens and provide the MD node with the first output sequence information corresponding to the N antecedent output tokens. After obtaining the key-value cache data corresponding to the input sequence and the first output sequence information, the MD node begins to perform extended decoding processing on the N antecedent output tokens to generate corresponding incremental key-value cache data.

[0084] During the extended decoding process performed by the MD node, the RLD node continues to generate new output tokens in an autoregressive manner. For example, the RLD node can further generate M output tokens, expanding the first output sequence from the initial N output tokens to N+M output tokens. These M output tokens remain part of the first output sequence, rather than being a separate output sequence independent of it.

[0085] For newly generated output tokens in the first output sequence, the RLD node can provide the corresponding token information, such as the corresponding token ID, to the MD node one by one or in batches. The MD node can continue to perform extended decoding processing based on the subsequently received token IDs, and append the key-value cache data corresponding to the corresponding output token to the already generated incremental key-value cache data.

[0086] Therefore, in the catch-up handover mode, the RLD node continuously extends the first output sequence forward, while the MD node continuously generates corresponding incremental key-value cache data based on the already determined first output sequence, so as to gradually narrow the processing progress gap between the two.

[0087] To determine whether the MD node has caught up with or is close to the RLD node's generation progress, we can determine the sequence length of the first output sequence currently generated, and the sequence length corresponding to the incremental key-value cache data already generated by the MD node. The former represents the number of output tokens generated by the RLD node up to the current moment, and the latter represents the number of output tokens for which the MD node has completed incremental key-value cache calculation up to the current moment. The difference between the two can be used to characterize the processing progress gap between the MD node and the RLD node.

[0088] If the difference between the sequence length of the currently generated first output sequence and the sequence length corresponding to the incremental key-value cache data does not exceed a predetermined threshold, a decoding task handover can be triggered. The predetermined threshold can be zero or other small integer values.

[0089] When the predetermined threshold is zero, the incremental key-value cache data generated by the MD node already corresponds to all output tokens in the current first output sequence, thus the MD node has caught up with the generation progress of the RLD node. At this point, the first decoding process performed by the RLD node can be stopped, and the MD node can directly continue to perform the second decoding process based on the current key-value cache state.

[0090] When the predetermined threshold is greater than zero, there may still be a small number of output tokens between the MD node and the RLD node that have not yet completed incremental calculation. In this case, the RLD node can be stopped from generating new output tokens, and the MD node can complete the incremental key-value cache calculation corresponding to the small number of output tokens before starting the second decoding process.

[0091] For example, if the first output sequence generated by the RLD node eventually includes N+M output tokens, the MD node can continue to generate the remaining 1K-NM output tokens after completing the corresponding incremental key-value cache calculation, so as to form the second output sequence and complete the decoding output of the current inference request.

[0092] In catch-up handover mode, the RLD node continues to generate output tokens while the MD node performs extended decoding, thus utilizing the continuous output of the RLD node to cover at least part of the extended decoding processing time. Especially when the MD node catches up with the RLD node before performing the handover, it can reduce output pauses caused by decoding state recovery, making the output process before and after the handover smoother.

[0093] Correspondingly, the catch-up handover mode usually causes the RLD node to generate an additional set of output tokens compared to the one-time handover mode, and extends the active time of the RLD node for the current inference request, which may increase the computational load and memory usage of the RLD node.

[0094] Therefore, the two handover modes can be applied to different operating states. The one-time handover mode allows the RLD node to stop decoding and release resources earlier, which is suitable for scenarios where RLD resources are scarce or where priority needs to be given to controlling the relay decoding load; the catch-up handover mode can reduce the impact of extended decoding processing on the output process, which is suitable for scenarios that are more sensitive to output interruptions or OTPS fluctuations.

[0095] Building upon the aforementioned decoding task handover model, the multi-stage processing flow between P nodes, RLD nodes, and MD nodes can be further orchestrated. Since RLD nodes are primarily used to perform decoding processing during the cross-datacenter transmission of key-value cache data, their resource configuration can be smaller than that of the MD nodes, which handle the main decoding tasks. Therefore, it is necessary to control the concurrency of RLD nodes and the time a single inference request occupies RLD resources to reduce the probability of RLD nodes becoming a system performance bottleneck due to insufficient GPU memory or computational overload.

[0096] In some embodiments, the system may primarily employ the P-MD path and the P-RLD-MD path. In the P-MD path, the RLD node is not enabled; the P node transmits the key-value cache data required for decoding to the MD node, and the MD node performs the decoding processing for the current inference request. In the P-RLD-MD path, while the key-value cache data is being transmitted to the MD node, the RLD node performs the first decoding process, and then, according to the aforementioned one-time handover mode or catch-up handover mode, the subsequent decoding tasks are handed over to the MD node.

[0097] The system can select between the P-MD path and the P-RLD-MD path based on the cross-data center transmission requirements corresponding to the current input sequence and the real-time resource status of the RLD node. For example, when the RLD node load is low, the RLD node can be enabled for a large number of input sequences, or relay decoding can be performed for each input sequence; when the RLD node load is high, the RLD node can be enabled only for input sequences that meet the relay decoding conditions, thereby prioritizing the allocation of limited RLD resources to requests with large cross-data center transmission volumes or long expected transmission delays.

[0098] The resource status of the RLD node may include one or more of the following: current concurrent request count, memory utilization, computing resource utilization, number of pending requests, or estimated waiting time. The relay decoding conditions can also be dynamically adjusted based on the resource status of the RLD node.

[0099] For example, when the prefix cache hit rate is lower than a predetermined hit rate threshold and the relay decoding condition is met, if the RLD node load is high, the predetermined hit rate threshold can be lowered so that only requests with a lower prefix cache hit rate and a larger amount of key-value cache data to be transmitted enter the P-RLD-MD path; if the RLD node load is low, the predetermined hit rate threshold can be raised so that more requests can utilize the RLD node's advance decoding to mask cross-data center transmission latency.

[0100] The system can also select the decoding task handover mode based on the resource status of the RLD node. For example, when the RLD node is under high load or needs to release RLD resources as soon as possible, a one-time handover mode can be selected, which allows the RLD node to stop the first decoding process after meeting a predetermined stopping condition. When the RLD node is under low load and the current inference request is sensitive to output pauses, a catch-up handover mode can be selected, which allows the RLD node to continue generating output tokens while the MD node is performing extended decoding processing, and completes the decoding task handover after the MD node catches up with or approaches the generation progress of the RLD node.

[0101] To reduce the continuous consumption of RLD node's computing and memory resources by inference requests with abnormally long output sequences, an upper limit can be set for the relay decoding process of a single inference request. For example, the RLD node can be limited to generating a predetermined number of output tokens for a single input sequence, or the duration of the first decoding process performed by the RLD node for a single input sequence can be limited to a predetermined time threshold. After reaching the corresponding upper limit for the number of output tokens or the time limit, the RLD node can stop continuing to perform the first decoding process and trigger the handover of the decoding task to the MD node.

[0102] The upper limit on the number of output tokens or the upper limit on the time can also be dynamically adjusted according to the real-time load of the RLD node. For example, when the load on the RLD node is high, the number of output tokens allowed to be generated can be reduced or the time allowed to perform the first decoding process can be shortened; when the load on the RLD node is low, the upper limit on the number or the upper limit on the time can be appropriately increased.

[0103] In some cases, the current inference request may actually manifest as a P-RLD path during execution. For example, the system may initially select the P-RLD-MD path for the current input sequence, but because the output sequence of the current inference request is short, the RLD node may have already generated the sequence end token or met other decoding termination conditions before the key-value cache data is completely transmitted to the MD node, thus completing the entire decoding process for the current inference request. In this case, there is no need for the MD node to continue generating subsequent output tokens.

[0104] Therefore, the P-RLD path can be a special execution result that naturally forms when the P-RLD-MD path completes decoding ahead of schedule, rather than being a regular processing path independently selected by the system. The system typically still schedules between the P-MD path and the P-RLD-MD path, ensuring that requests requiring relay decoding eventually have the conditions for the MD node to take over the decoding task.

[0105] Even if the RLD node has completed all decoding processing for the current inference request, the remaining key-value cache data corresponding to the input sequence can still be transmitted to the MD node and written to the MD node's decoding-side cache pool. This maintains a consistent correspondence between the coverage of the cached prefixes on the MD node side and the prefix cache on the P node side, allowing the MD node to reuse the corresponding key-value cache data locally when processing subsequent requests with the same or similar input prefixes.

[0106] Similarly, regardless of whether the current inference request uses the P-MD path or the P-RLD-MD path, the key-value cache data corresponding to the input sequence can be transferred and retained on the MD node side. Therefore, the decoding cache pool on the MD node side can continuously accumulate key-value cache data corresponding to the prefix cache on the P node side, thereby improving the prefix cache hit rate of subsequent requests on the MD node side and reducing the amount of data transferred across data centers.

[0107] Through the aforementioned multi-stage pipeline orchestration, relay decoding can be dynamically selected based on the cross-datacenter transmission requirements of different input sequences and the real-time resource status of RLD nodes. Furthermore, relay decoding conditions, single-request resource consumption limits, and decoding task handover modes can be adjusted. This allows for control over RLD node concurrency and resource consumption while prioritizing limited relay decoding resources for requests with higher long-tail transmission latency risks, and enables MD nodes to continuously accumulate key-value cache data that can be reused for subsequent requests.

[0108] Figure 6 A schematic diagram of the composition of an LLM inference system according to some embodiments of the present disclosure is shown. For example... Figure 6 As shown, the LLM inference system 600 may include a first data center 610 and a second data center 620. The first data center 610 is deployed with multiple P nodes and at least one RLD node, while the second data center 620 is deployed with multiple MD nodes. The P nodes and RLD nodes may be located in the same RDMA domain of the first data center 610 and interconnected via high-bandwidth, low-latency communication networks such as RDMA; the first data center 610 and the second data center 620 may be connected via cross-data center communication networks such as wide-area Ethernet.

[0109] Each P node can correspond to a pre-filled inference unit deployed on a pre-filled inference server, each RLD node can correspond to a first decoding inference unit deployed on a first decoding inference server, and each MD node can correspond to a second decoding inference unit deployed on a second decoding inference server. Each pre-filled inference server, the first decoding inference server, and the second decoding inference server can include one or more hardware accelerators.

[0110] For ease of explanation, the hardware accelerator configured in the pre-filled inference server will be referred to as the first type of hardware accelerator, the hardware accelerator configured in the first decoding inference server will be referred to as the second type of hardware accelerator, and the hardware accelerator configured in the second decoding inference server will be referred to as the third type of hardware accelerator. These hardware accelerators can be implemented as GPUs or other dedicated chips.

[0111] In some embodiments, the first type of hardware accelerator and the second type of hardware accelerator can be the same type or even the same model of hardware accelerator. In this case, the first data center is deployed in a fine-grained homogeneous manner, meaning that the servers do not actually have different identities and can be flexibly deployed as needed, either as P instances or RLD instances. In other embodiments, considering that pre-filling processing requires parallel computation on multiple tokens in the input sequence, while decoding processing requires repeated reading of model parameters and key-value cache data, the first type of hardware accelerator and the second type of hardware accelerator can be different types and / or models of hardware accelerator. In this case, the first type of hardware accelerator can have high computing power and be suitable for performing pre-filling processing (e.g., high-performance domestically produced chips); the second type of hardware accelerator can have high storage capacity, storage bandwidth, or storage access efficiency and be suitable for performing decoding processing. This achieves a fine-grained heterogeneous deployment within the first data center.

[0112] Similarly, the third type of hardware accelerator can be the same as or different from the first and / or second type of hardware accelerator. For example, the second decoding inference server can be configured with the same third type of hardware accelerator as the second type of hardware accelerator, suitable for decoding processing, or it can be configured with other hardware accelerators with higher storage capacity and storage bandwidth. For example, the RLD node mainly undertakes short-term relay decoding processing, while the MD node undertakes the main decoding load; therefore, the storage capacity or computing scale of the hardware accelerator used by the RLD node can be lower than that of the hardware accelerator used by the MD node.

[0113] In traditional heterogeneous pre-filling-decoding separation architectures within data centers, large-scale pre-filling resources capable of handling the entire pre-filling load and large-scale decoding resources capable of handling the entire decoding load are typically deployed simultaneously. Therefore, a significant number of both types of hardware accelerators need to be configured within the same high-speed communication domain. When business characteristics such as input length, output length, concurrent request volume, or cache hit rate change, the fixed configuration ratio between pre-filling and decoding resources may no longer match the actual load, leading to some hardware resources being idle.

[0114] In contrast, in the LLM inference system 600 disclosed herein, the RLD nodes in the first data center 610 are primarily used to generate a portion of the output tokens during the cross-data center transmission of key-value cache data, without bearing the entire decoding load. Most or the remaining decoding processing can be performed by the MD nodes in the second data center 620. Therefore, in fine-grained heterogeneous deployment within a data center, the first data center 610 does not need to deploy a large-scale decoding resource equivalent to the number of P nodes; instead, only a small number of RLD nodes can be configured to form a fine-grained heterogeneous deployment.

[0115] For example, such as Figure 6 As shown, a relatively large number of P-nodes and a relatively small number of RLD nodes can be deployed within the first data center 610. The number of first-type hardware accelerators used to perform pre-filling processing can be greater than the number of second-type hardware accelerators used to perform relay decoding processing. In some fine-grained heterogeneous deployment embodiments, the number of first-type hardware accelerators can be at least five times, or even ten times, the number of second-type hardware accelerators. For example, ten, twenty, or more P-nodes can be deployed within the first data center 610, with only one, two, or a small number of RLD nodes deployed.

[0116] The aforementioned deployment ratio can be determined based on factors such as the proportion of requests served by the RLD node, the number of relay decoding tokens per request, the concurrent processing capacity of the RLD node, and the allowed resource consumption limit. For example, if RLD nodes are only enabled for requests with low prefix cache hit rates, large amounts of key-value cache data to be transmitted, or long expected cross-data center transmission latency, only a small portion of the requests actually enter the P-RLD-MD processing path. Therefore, a small number of RLD nodes can provide relay decoding services for these requests.

[0117] Furthermore, by limiting the number of output tokens generated by the RLD node for a single request or the duration of relay decoding processing, the computational and memory resources consumed by a single request on the RLD node can be further limited. Therefore, the RLD resource size in the first data center 610 can be significantly smaller than the decoding resource size that bears the entire decoding load in a traditional PD-separated architecture within a data center.

[0118] This fine-grained heterogeneous deployment reduces the number of Type II hardware accelerators required within Data Center 610, thereby lowering the cost of deploying additional hardware to build a heterogeneous inference cluster. Simultaneously, since RLD nodes only handle a limited pre-decoding load, their number and resource size can be adjusted based on actual relay decoding requests, reducing the likelihood of resource idleness due to changes in the pre-filling load / decoding load ratio. Even if Data Center 610 contains an underutilized RLD node as shown in the figure, the cost is far less than... Figure 1 The image shows a traditional heterogeneous data center.

[0119] In some embodiments, the RLD nodes in the first data center 610 can also be deployed elastically. For example, the number of running RLD inference instances can be increased or decreased based on current service traffic, cross-data center transmission latency, the number of low prefix cache hit rate requests, or the current concurrency of the RLD nodes. Alternatively, some RLD inference instances can be deactivated when RLD load is low, and new RLD inference instances can be started when RLD load increases.

[0120] It should be understood that, Figure 6 The number of P nodes, RLD nodes, and MD nodes shown is for illustrative purposes only. Multiple P nodes can share a single RLD node or a group of RLD nodes, and a single RLD node can concurrently process relay decoding requests from multiple P nodes. Accordingly, a correspondence between multiple P nodes and multiple RLD nodes can be established through a scheduler, a shared request queue, or a load balancing component, without requiring a one-to-one correspondence between P nodes and RLD nodes.

[0121] Furthermore, it should be understood that nodes, inference servers, and hardware accelerators represent entities at different levels. Specifically, P-nodes, RLD nodes, and MD nodes are logical nodes categorized according to their inference functions; an inference server is a hardware computing platform that hosts the corresponding logical nodes; and a hardware accelerator is a hardware device located within the inference server that actually performs model computations. In one embodiment, a logical node can correspond to a single inference instance, which is deployed on a corresponding inference server and invokes one or more hardware accelerators within that inference server to perform pre-filling or decoding processing. An inference server can host only one inference instance, or it can host multiple inference instances if computing and storage resources permit. Therefore, Figure 6 The P, RLD, and MD boxes shown primarily represent the corresponding logical nodes or inference instances, and their number does not necessarily correspond one-to-one with the number of inference servers or hardware accelerators.

[0122] The second data center 620 can deploy a large-scale MD resource pool. The MD resource pool may include multiple MD nodes and is used to receive key-value cache data and first output sequence information from the first data center 610. The scheduling component can allocate inference requests from the first data center 610 to the corresponding MD nodes based on the memory usage, computing load, cache hit status, or number of pending requests of each MD node.

[0123] Because the P-nodes and RLD nodes are decoupled from the MD nodes in terms of function and deployment location, the topology between the first data center 610 and the second data center 620 can be flexibly configured. Figure 6 In the illustrated embodiment, a first data center 610 can be connected to a second data center 620, with multiple MD nodes in the second data center 620 jointly handling the main decoding load from the first data center 610. The LLM inference system may include multiple first data centers, such as... Figure 6 As shown in the section on "Traffic from Other Data Centers" (Traffic from other Data Centers can also represent inference requests from other pre-populated clusters), multiple first data centers can share the decoding capabilities of the MD nodes in the same second data center by connecting via a second communication path. Each first data center can deploy corresponding P nodes and a small number of RLD nodes, while the second data center can deploy a centralized MD resource pool. Thus, multiple first data centers with different geographical locations or serving different user areas can share the decoding resources in the second data center.

[0124] For example, multiple first data centers can perform pre-filling processing at locations close to the source of the business request, and, when necessary, have local RLD nodes perform relay decoding processing first. Each first data center can transmit the key-value cache data corresponding to the cache-missed portion of the input sequence to the second data center via corresponding cross-data center communication paths. After receiving the corresponding key-value cache data, the MD nodes in the second data center can then proceed to complete the decoding processing of each inference request.

[0125] In one embodiment, the number of the third type of hardware accelerators in a second data center used to perform the second decoding process can be greater than the number of the second type of hardware accelerators in a first data center used to perform the first decoding process. For example, the former can be at least five times, or even at least 10 times, 15 times, or 20 times, the latter.

[0126] By enabling multiple primary data centers to share a large MD resource pool, MD resources can be uniformly scheduled based on the overall decoding load from different primary data centers. For example, when the decoding request volume in one primary data center temporarily decreases while the decoding request volume in another primary data center increases, MD resources in the second data center can be dynamically allocated to the primary data center with higher request volume, thereby reducing the idle decoding resources caused by fluctuations in business traffic in different regions.

[0127] Through this deployment method, the LLM inference system can centrally configure pre-filled resources with strong computing power in the first data center, configure only a small amount of RLD resources for pre-decoding, and utilize the large-scale MD resources in the second data center to handle the main decoding load. This reduces the deployment scale of heterogeneous hardware within a single data center and allows for separate expansion and unified scheduling of pre-filled resources, relay decoding resources, and main decoding resources based on the service traffic and resource status of different data centers.

[0128] The LLM inference method and system according to embodiments of this disclosure have been described above with reference to the accompanying drawings. This disclosure constructs a three-tiered architecture comprising P nodes, RLD nodes, and MD nodes. The P nodes provide key-value cache data to the RLD nodes via a low-latency communication path within the data center, and provide corresponding key-value cache data to the MD nodes via a cross-data center communication path. The RLD nodes can generate output tokens in advance during cross-data center transmission, thereby masking at least part of the WAN transmission latency with decoding processing, reducing the impact of cross-data center transmission on TTFT and E2EL. During decoding task handover, the RLD nodes do not need to transmit their generated incremental key-value cache data to the MD nodes; instead, they can provide only a small output token identifier sequence. The MD nodes then generate corresponding incremental key-value cache data locally based on the received key-value cache data and the output token identifier sequence, and continue performing subsequent decoding processing. This reduces cross-data center communication volume and lowers the communication and data copying overhead caused by decoding task handover.

[0129] Furthermore, since RLD nodes only handle a limited pre-decoding load, the first data center can deploy a large number of pre-filled resources while configuring only a small number of heterogeneous hardware accelerators suitable for decoding processing, thereby achieving cross-data center latency masking with lower additional deployment costs. By setting up a prefix-matching-based cache on the MD node side, only key-value cache data corresponding to cache misses can be generated and transmitted. The enabling of relay decoding and the corresponding handover mode can be selected based on the prefix cache hit rate, the amount of data to be transmitted, network status, and RLD resource status. For scenarios with long inputs, low cache hit rates, short outputs, or network fluctuations, RLD nodes can also serve as temporary decoding resources during cross-data center transmissions to reduce the impact of abnormal requests and network status changes on the overall system service level objectives, thereby improving the utilization of heterogeneous inference resources and the scalability and robustness of the cross-data center inference system.

[0130] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information all comply with relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. User authorization or consent has been obtained before acquiring or collecting user personal information.

[0131] Furthermore, the method according to the present invention can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of the present invention.

[0132] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the various steps of the method described above according to the present invention.

[0133] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A large language model (LLM) inference method, applied to an LLM inference system including a first data center and a second data center, wherein the first data center is deployed with a pre-filled inference unit and a first decoding inference unit, and the second data center is deployed with a second decoding inference unit, the method comprising: The pre-filled inference unit performs pre-filling processing on the input sequence used for LLM inference to obtain key-value cache data of the input sequence; The key-value cache data is provided to the first decoding inference unit via a first communication path, and to the second decoding inference unit via a second communication path; The first decoding inference unit performs a first decoding process based on the key-value cache data received via the first communication path to generate a first output sequence and output first inference content based on the first output sequence, wherein the first decoding process and the output process of the first inference content at least partially overlap with the transmission process of providing the key-value cache data to the second decoding inference unit via the second communication path; as well as The first output sequence information is transmitted to the second decoding inference unit so that the second decoding inference unit generates a second output sequence based on the key-value cache data and the first output sequence and outputs the second inference content based on the second output sequence.

2. The method of claim 1, wherein, Providing the key-value cache data to the first decoding inference unit via a first communication path, and providing the key-value cache data to the second decoding inference unit via a second communication path, includes: Determine whether the input sequence satisfies the relay decoding condition; and In response to the input sequence satisfying the relay decoding condition, the key-value cache data is provided to the first decoding inference unit via the first communication path, and the key-value cache data is provided to the second decoding inference unit via the second communication path.

3. The method according to claim 2, further comprising: In response to the input sequence not satisfying the relay decoding condition, the key-value cache data is provided to the second decoding inference unit only via the second communication path, so that the second decoding inference unit performs decoding processing based on the key-value cache data and outputs the corresponding inference content.

4. The method according to any one of claims 1 to 3, wherein, Transmitting first output sequence information to the second decoding inference unit, so that the second decoding inference unit generates a second output sequence based on the key-value cache data and the first output sequence, and outputs second inference content based on the second output sequence, includes: The first output sequence information, which characterizes the first output sequence, is transmitted to the second decoding inference unit so that after the second decoding inference unit obtains the key-value cache data via the second communication path, it obtains the incremental key-value cache data corresponding to the first output sequence based on the first output sequence information, and performs a second decoding process based on the key-value cache data and the incremental key-value cache data to generate a second output sequence and then outputs the second inference content obtained based on the second output sequence.

5. The method of claim 4, wherein, In response to the fulfillment of a predetermined stopping condition, the first decoding inference unit stops executing the first decoding process and provides the second decoding inference unit with first output sequence information characterizing all output tokens generated by the first decoding inference unit before stopping the execution of the first decoding process.

6. The method according to claim 4, wherein, During the second decoding inference unit's generation of incremental key-value cache data based on the first output sequence information, the first decoding inference unit continues to execute the first decoding process to expand the first output sequence and continue to provide the second decoding inference unit with lexical information corresponding to newly generated output lexicals in the first output sequence; In response to the fact that the sequence length of the currently generated first output sequence does not exceed a predetermined threshold compared to the sequence length corresponding to the incremental key-value cache data, the first decoding process executed by the first decoding inference unit is stopped, and the second decoding process is executed by the second decoding inference unit.

7. The method according to claim 2 or 3, further comprising: Determine the current resource usage status of the first decoding inference unit; as well as Based on the current resource usage status, perform at least one of the following: Adjust the threshold used to determine whether the input sequence satisfies the relay decoding condition; Adjust the maximum duration of the first decoding inference unit performing the first decoding process for a single input sequence or the upper limit of the number of output tokens generated. as well as Choose between a one-time handover mode and a catch-up handover mode to transfer the decoding task from the first decoding inference unit to the second decoding inference unit. The current resource usage status includes at least one of the following: the number of concurrent requests to the first decoding inference unit, the video memory usage rate, the computing resource utilization rate, the number of pending requests, or the estimated waiting time. In the one-time handover mode, the first decoding inference unit hands over the decoding task and transfers the first output sequence information to the second decoding inference unit in one go. In the catch-up handover mode, the first decoding inference unit continues to perform the first decoding process while the second decoding inference unit generates the incremental key-value cache data corresponding to the first output sequence, and in response to the number of output tokens corresponding to the incremental key-value cache data catching up with the number of output tokens currently generated by the first decoding inference unit, the first decoding inference unit hands over the decoding task to the second decoding inference unit.

8. A large language model LLM inference system, comprising a first data center and a second data center. The first data center is equipped with a pre-filled inference unit and a first decoding inference unit, wherein, The pre-filled inference unit is deployed on the pre-filled inference server, and the first decoding inference unit is deployed on the first decoding inference server; The second data center is equipped with a second decoding inference unit, and the second decoding inference unit is deployed on a second decoding inference server; The pre-filled inference unit communicates with the first decoding inference unit via a first communication path, and the pre-filled inference unit communicates with the second decoding inference unit via a second communication path, wherein: The pre-filled inference unit is configured as follows: Pre-padded data is performed on the input sequence used for LLM inference to obtain key-value cache data corresponding to the input sequence; and The key-value cache data is provided to the first decoding inference unit via the first communication path, and the key-value cache data is provided to the second decoding inference unit via the second communication path; The first decoding inference unit is configured as follows: A first decoding process is performed based on the key-value cache data received via the first communication path to generate a first output sequence, and a first inference content obtained based on the first output sequence is output, wherein the first decoding process and the output process of the first inference content at least partially overlap with the transmission process of providing the key-value cache data to the second decoding inference unit via the second communication path; and The second decoding inference unit is provided with first output sequence information for characterizing the first output sequence; and The second decoding inference unit is configured as follows: Based on the key-value cache data received via the second communication path and the first output sequence information, incremental key-value cache data corresponding to the first output sequence is generated; and A second decoding process is performed based on the key-value cache data and the incremental key-value cache data to generate a second output sequence and output the second inference content obtained based on the second output sequence.

9. The LLM inference system according to claim 8, wherein, The pre-filled inference server includes a first type of hardware accelerator for performing the pre-filled process, and the first decoding inference server includes a second type of hardware accelerator for performing the first decoding process. The computing power of the first type of hardware accelerator is higher than that of the second type of hardware accelerator, and at least one of the storage capacity and storage bandwidth of the second type of hardware accelerator is higher than that of the first type of hardware accelerator.

10. The LLM inference system according to claim 8, wherein, The number of the third type of hardware accelerators used to perform the second decoding process in one of the second data centers is greater than the number of the second type of hardware accelerators used to perform the first decoding process in one of the first data centers.

11. The LLM inference system according to any one of claims 8 to 10, wherein, The LLM inference system includes multiple first data centers, each of which is connected to a second data center via its own second communication path and shares multiple second decoding inference units deployed in the second data center.

Images