Large language model reasoning method and system

Abstract

The invention provides an LLM reasoning method and system which are suitable for a computing cluster with a plurality of computing devices. The method comprises the steps that a topological graph is generated according to the communication connection relation of a plurality of computing devices, Hamiltonian decomposition is carried out on the topological graph to obtain a plurality of directed loops which are orthogonal to one another, each computing device corresponds to one vertex in the topological graph, and the directed loops are connected with the computing devices; the communication connection meeting a predetermined condition between every two computing devices corresponds to an edge in the topological graph; distributing a query sequence segment and an original KV cache block for each computing device; and in the subsequent iteration, each computing device simultaneously transmits and receives the KV cache sub-blocks in a plurality of directed loops. According to the invention, the Hamiltonian decomposition is utilized to convert the single ring communication among the plurality of computing devices into a plurality of parallel ring communication, the communication efficiency and the network resource utilization rate among the computing devices are improved, and the load balance under the causal mask can be realized through the symmetric distribution of the sequence segments.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to a method and system for reasoning using large language models. Background Technology

[0002] In the field of Natural Language Processing (NLP), Large Language Models (LLMs) based on the Transformer architecture have been widely applied to tasks such as text generation, question answering, and dialogue. With the continuous growth of model size and context length, long-context inference has become a pressing problem. Especially in the prefill stage of inference, attention computation requires processing large-scale sequences, leading to a sharp increase in computational and storage overhead. To reduce this overhead, existing techniques typically employ sequence parallelism (SP) schemes, with Ring Attention and DeepSpeed ​​Ulysses being typical implementations.

[0003] However, the above-mentioned solutions still have limitations. For example, DeepSpeed ​​Ulysses requires the parallelism to be divisible by the number of key-value (KV) heads, and its limitations are even more pronounced under Multi-Query Attention (MQA) or Grouped Query Attention (GQA), making it difficult to flexibly adapt to different model structures. Ring Attention fails to fully utilize bandwidth resources during inter-device communication, resulting in limited overall performance improvement. These issues make it difficult for existing technologies to balance versatility and efficiency. Summary of the Invention

[0004] To this end, this disclosure proposes an LLM inference method and system that utilizes Hamiltonian decomposition to transform a single ring communication between multiple computing devices into multiple parallel ring communications, thereby improving the communication efficiency between computing devices and the utilization of network resources.

[0005] According to a first aspect of the present disclosure, a Large Language Model (LLM) inference method is proposed, applicable to a computing cluster with multiple computing devices. The method includes: generating a topology graph based on the communication connection relationships of the multiple computing devices, and performing Hamiltonian decomposition on the topology graph to obtain multiple directed loops that are orthogonal to each other, wherein each computing device corresponds to a vertex in the topology graph, and a communication connection between every two computing devices that satisfies predetermined conditions corresponds to an edge in the topology graph; allocating a query sequence segment and an original KV cache block to each computing device; and in subsequent iterations, each computing device simultaneously transmits and receives KV cache sub-blocks in multiple directed loops.

[0006] Optionally, the multiple orthogonal directed loops obtained are m directed loops, where m is an integer not less than 2. Each computing device divides the original KV cache block it is allocated into m KV cache sub-blocks, and in the first iteration: each computing device passes its m KV cache sub-blocks to the next computing device connected after each computing device in the m directed loops.

[0007] Optionally, in subsequent iterations, each computing device simultaneously transmits and receives KV cache subblocks on multiple directed loops, including: in each iteration after the first iteration, each computing device simultaneously receives the KV cache subblocks transmitted by the previous computing device in this iteration on each directed loop, and transmits the KV cache subblocks received in the previous iteration to the next computing device.

[0008] Optionally, generating a topology graph based on the communication connection relationships of the plurality of computing devices, and performing Hamiltonian decomposition on the topology graph to obtain multiple directed loops that are orthogonal to each other includes: generating a complete graph K in response to the fact that there are n computing devices and the communication connections between the plurality of computing devices are fully interconnected communication connections. n As the topological graph, where n is an integer not less than 8 and divisible by 4; and for the complete graph K n Perform Hamiltonian decomposition to obtain n-1 Hamiltonian cycles as the plurality of directed loops.

[0009] Optionally, generating a topology graph based on the communication connection relationships of the plurality of computing devices, and performing Hamiltonian decomposition on the topology graph to obtain a plurality of mutually orthogonal directed loops includes: in response to the computing cluster comprising p computing nodes and each computing node comprising q computing devices with fully interconnected communication connections, and the communication bandwidth between computing nodes being within a predetermined threshold range, dividing the plurality of computing devices into p subsets of computing devices to generate p complete graphs K. q and adjacent complete graph K qThe topology graph has q connecting lines, where p is an integer not less than 2, q is an integer not less than 8 and divisible by 4, and the plurality of computing devices are pq computing devices; for each complete graph K q Perform Hamiltonian decomposition to obtain q Hamiltonian paths and assign each of the q Hamiltonian paths a number from 0 to q-1; and decompose the graphs from the p complete graphs K... q Hamiltonian paths with the same number are connected end-to-end to form a Hamiltonian ring, so as to obtain q Hamiltonian rings as the plurality of directed loops.

[0010] Optionally, generating a topology graph based on the communication connection relationships of the multiple computing devices, and performing Hamiltonian decomposition on the topology graph to obtain multiple orthogonal directed loops, includes: generating a new complete graph K in response to the addition of a new computing node as the (p+1)th computing node to the computing cluster. q ; Regarding this new complete graph K q Perform Hamiltonian decomposition to obtain q new Hamiltonian paths; and disconnect q connections between any two computation nodes in the q Hamiltonian rings, and connect the new q Hamiltonian paths from the disconnection points into the q Hamiltonian rings to form new q Hamiltonian rings that incorporate the q new vertices corresponding to the p+1th computation node.

[0011] Optionally, generating a topology graph based on the communication connection relationship of the plurality of computing devices, and performing Hamiltonian decomposition on the topology graph to obtain a plurality of mutually orthogonal directed loops includes at least one of the following: when the plurality of computing devices are interconnected using the same communication method and / or directly interconnected, and the bidirectional bandwidth between two computing devices is greater than a preset first threshold, establishing an edge between corresponding two vertices; and when the plurality of computing devices are interconnected using two or more communication methods, including indirect connections via exchange nodes, determining the number of edges established via exchange nodes, such that the difference between the equivalent bidirectional bandwidth of each edge obtained therefrom and the bidirectional bandwidth established via other communication methods is not greater than a second threshold.

[0012] Optionally, the plurality of computing devices are n computing devices, and the plurality of directed loops are m directed loops, where n is an integer not less than 8 and divisible by 4, and m is an integer not less than 2. Allocating a query sequence segment and an original KV cache block to each computing device includes: dividing the KV cache into n KV cache blocks along the sequence length direction, and allocating one of the n KV cache blocks to each computing device as its original KV cache block; and each computing device sequentially dividing its allocated original KV cache block into m sequentially numbered KV cache sub-blocks, which are used for the initial iteration of the transmission on the m directed loops.

[0013] Optionally, the plurality of computing devices are n computing devices, and the plurality of directed loops are m directed loops, where n is an integer not less than 3 and m is an integer not less than 2. Allocating a query sequence segment and an original KV cache block to each computing device includes: dividing the KV cache into 2n KV cache blocks along the sequence length direction; allocating the i-th KV cache block and the i-th-last KV cache block to the i-th computing device as the original KV cache block for that computing device, where i = 1, ..., n; and the i-th computing device further divides the i-th KV cache block and the i-th-last KV cache block into m KV cache sub-blocks, and pairs the m KV cache sub-blocks of the i-th KV cache block and the m KV cache sub-blocks of the i-th KV cache block for the first iteration of propagation on the m directed loops, wherein the query sequence segment allocated to each computing device is aligned with the allocated KV cache block.

[0014] According to a second aspect of the present disclosure, a terminal device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in the first aspect.

[0015] According to a third aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method as described in the first aspect.

[0016] Therefore, the solution disclosed herein can construct multiple orthogonal directed loops through Hamiltonian decomposition based on different network connection conditions between computing devices. These loops are used for the parallel transfer of KV cache subblocks in sequence parallelism, thereby improving communication efficiency and network resource utilization while achieving both communication and computational parallelism and low computational and storage requirements. Furthermore, the solution disclosed herein can also achieve load balancing under causal masking through the symmetrical distribution of KV cache blocks and query sequence segments. Attached Figure Description

[0017] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0018] Figure 1 The diagram shows a fully connected topology with eight computing devices in different hardware connection methods.

[0019] Figure 2 The logical topology between eight computing devices under a fully connected, all-to-one condition is shown, along with portions involving unidirectional ring communication links.

[0020] Figure 3 The iterative process involved in performing one Ring All-Gather operation by eight computing devices is shown.

[0021] Figure 4 A schematic flowchart of an LLM inference method according to an embodiment of the present disclosure is shown.

[0022] Figure 5 The diagram shows the complete topological graph of 8 vertices and the 7 orthogonal directed loops obtained by its Hamiltonian decomposition.

[0023] Figure 6 An example of multi-ring full collection communication according to one embodiment of the present disclosure is shown.

[0024] Figure 7 This illustrates an example of the physical connections, communication topology modeling, and acquisition of multiple directed loops in a 2-node computing cluster.

[0025] Figure 8 This example illustrates the construction of a Hamiltonian path when adding a new node to two existing compute nodes. Detailed Implementation

[0026] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0027] As used in the specification and appended claims of this application, the singular expressions “a,” “the,” “the,” and “the” are intended to also include expressions such as “one or more,” unless the context explicitly indicates otherwise. The term “comprising” and its variations, as used herein, indicate an open-ended inclusion, i.e., “including but not limited to.” Unless specifically stated otherwise, the term “or” means “and / or.” The term “according to” means “at least in part according to.” The terms “an example embodiment” and “an embodiment” mean “at least one example embodiment.” The term “another embodiment” means “at least one additional embodiment.” The terms “first,” “second,” etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0028] With the rapid development of LLM, ultra-long context processing has gradually become a focus of research and application. Existing systems can already support context windows with millions of tokens, which is of great value for processing lengthy legal documents, analyzing large-scale codebases, or conducting long-duration multimodal dialogues. However, the mainstream Transformer-based architecture has a fundamental bottleneck in scalability: the computational complexity of the self-attention mechanism increases quadratically with the sequence length, resulting in a requirement of O(n log n) for the prefill stage of inference. 2 The computational complexity and memory consumption of O(n) (where n is the sequence length) are significant. When the sequence length reaches millions, a single accelerator struggles to handle the computational and storage demands, limiting the feasibility of inference for ultra-long sequences.

[0029] To alleviate the aforementioned problems, sequence parallelism has become a mainstream approach. The basic idea is to distribute the long sequence dimension across multiple computing devices to amortize computational and storage overhead. Typical methods include DeepSpeed ​​Ulysses and RingAttention. The Ulysses method partitions the input along the sequence dimension and uses All-to-All communication to reallocate the Key-Value Cache (KV Cache), allowing each device to independently compute the complete attention at its local KV header, thus reducing the load on a single card. However, this method has three limitations: First, the parallelism must be divisible by the number of KV headers, making its applicability particularly limited in MQA or GQA scenarios; second, computation and communication cannot completely overlap, as the All-to-All allocation of the KV Cache must be completed before computation, while the All-to-All output of the attention must be performed after local computation, resulting in communication and computation alternating serially; third, uneven KV header partitioning can lead to unbalanced computational loads between devices, further reducing overall performance.

[0030] The Ring Attention method divides query (Q) and key-value (KV) cache blocks across devices and can iteratively pass KV cache blocks using Ring All-Gather communication to achieve complete attention computation for local queries. This method reduces storage requirements to some extent, but two main problems remain: First, Ring All-Gather communication has limited efficiency, failing to fully utilize the point-to-point interconnect bandwidth provided by modern accelerators; when the sequence length is not large enough, communication latency may become a performance bottleneck. Second, the attention computation load is unevenly distributed along the sequence length, especially in causal masking scenarios, and existing optimization strategies cannot effectively address the load balancing problem.

[0031] Therefore, it is necessary to propose a novel sequence parallelism scheme to address the limitations of existing technologies. This scheme should achieve high parallelism while maximizing the overlap between computation and communication to improve overall efficiency. Furthermore, the scheme should also have the ability to optimize load balancing between devices to fully utilize the bandwidth resources of modern accelerators, thereby achieving both versatility and high performance in ultra-long context inference, enabling large language models to handle larger-scale sequences while reducing computational and storage overhead.

[0032] In view of this, this disclosure proposes an LLM inference method and system that improves the network resource utilization between computing devices under sequential parallel operation through an improved multi-loop parallel data transfer mechanism. This scheme can significantly improve communication efficiency and also has advantages such as overlapping communication and computation and low additional storage requirements, thereby achieving a more efficient inference process.

[0033] To facilitate understanding of the principles of this disclosure, the relevant concepts of sequence parallelism will be introduced first.

[0034] Figure 1 The figure illustrates a fully-connected topology with eight computing devices using different hardware connection methods. As shown, the eight computing devices are represented by their logical numbers rank0 to rank7. Here, a fully-connected topology means that each computing device in the system can communicate with all other devices. This logical full interconnectivity can be achieved through different physical connection methods, such as… Figure 1 The left side shows a direct full mesh connection, or as shown in the image. Figure 1 The right side shows a switch-based interconnect. This type of topology enables all-to-all logical communication, allowing for large-scale data exchange between devices.

[0035] In a full-mesh direct-connect system, each computing device connects directly to all other computing devices used for inference in the system via a dedicated bidirectional communication link. This minimizes communication latency between any two devices and fully utilizes bandwidth without relying on intermediate devices. Figure 1 As shown on the left, each rank has direct communication links to the other seven ranks. In contrast, in switch-based connections, each computing device connects to only one or more high-performance switches for inter-device communication. Figure 1 As shown on the right, each rank is connected to switches 1 through 5, and bidirectional communication links between each computing device are indirectly realized through the switches.

[0036] Furthermore, Figure 1The diagram also uses red lines to illustrate the communication link used to pass the KV Cache in the vanilla (basic or standard) Ring Attention model. Figure 1 In the full-mesh connection shown on the left, only the unidirectional ring communication link indicated by the red arrow is needed to achieve the KV cache passing involved in Ring Attention. And... Figure 1 As shown on the right, in the switch-based connection, depending on the specific switch connection method, the transmission of KV Cache requires each rank to interact with multiple switches.

[0037] Regardless of the actual physical connection method used. Figure 1 The different hardware interconnect structures shown on the left and right sides can all be logically abstracted into a unified topology. The unidirectional ring communication link for transmitting KV cache in Ring Attention can also be described equivalently based on the topology. Figure 2 The logical topology between eight computing devices under a fully connected, all-to-one condition is shown, along with portions involving unidirectional ring communication links.

[0038] like Figure 2 As shown on the left, in the logical topology diagram, each vertex corresponds to a computing device and its logical number (i.e., rank). Here, Figure 2 Vertices 0-7 in Figure 1 The rank 0 to rank 7 correspond one-to-one. For visual differentiation, in addition to the numbers, eight different colors are used to identify different vertices in the diagram. Figure 3 as well as Figures 5-8 This color-coding method is also reused in the diagram. Each edge represents a connection between two computing devices. Due to the use of all-to-all connections, each vertex is connected to every other vertex. To simplify the illustration, each black edge in the diagram represents bidirectional communication between two points. For example, the black edge between vertex 0 and vertex 1 corresponds to two directed edges from 0 to 1 and from 1 to 0.

[0039] When performing vanilla ring attention, only one unidirectional ring communication link passing through each rank is used, for example... Figure 2The loop shown on the right. In the actual execution of this ring communication, each computing device is first allocated an original KV cache chunk. In one embodiment, assuming the loop contains n vertices (i.e., corresponding to n computing devices), all KV tokens (KV representations) in the KV cache can be divided along a sequence length S (i.e., the KV cache contains S pairs of KV tokens; correspondingly, the length of the query sequence is also S, i.e., it contains S query tokens), resulting in n equally divided KV cache chunks (in this paper, a KV cache chunk and subsequent KV cache sub-chunks obtained by dividing the KV cache chunks can be considered as a segment of KV tokens), and each chunk is allocated to a corresponding computing device as its original KV cache chunk. Specifically, its KV cache allocation strategy can be:

[0040]

[0041] Where i = 0, 1, ..., n-1, t[i] represents the KV cache allocated to the corresponding ranki.

[0042] In one embodiment, the allocation of the query sequence can be aligned with the allocation of the KV cache. That is, a sequence segment Q of the query sequence can be allocated to each ranki. i (Similarly, a sequence segment can be viewed as a query token), where the query token within the sequence segment is aligned to t[i]. It should be understood that in other embodiments, the query sequence may also be allocated in other ways.

[0043] After allocating the query sequence segments and the original KV cache blocks, the iterative process for KV cache block passing and computation can begin, thereby utilizing the Ring All-Gather communication principle to achieve full collection of KV caches on each rank. It should be understood that since each rank i has already acquired its assigned KV token, t[i], before the iteration begins, all other KV tokens can be acquired after n-1 iterations. However, to achieve attention calculation between all queries and all KV caches, n iterations are required. That is, the nth iteration only involves attention calculation between each rank and the KV tokens acquired in the previous iteration, and no further KV token passing is needed (because at this point, the node preceding each rank i in the loop holds the initially allocated t[i] for that rank). During the iteration process, the sequence segment Q... i It does not participate in the transfer between ranks, that is, it remains fixed on its respective ranki.

[0044] In iteration 0 (iter 0, i.e., the 1st iteration), each rank i passes its held t[i] to the next rank on the circular path, and can use the local Q. i Attention is calculated for t[i].

[0045] In iteration 1 (iter 1, i.e., the second iteration), rank i passes the KV cache block received in iteration 0 to the next rank on the ring path, and uses the local Q... i Attention calculations are performed on the KV cache blocks received in iteration 0.

[0046] Repeat the attention calculation and transfer operations as described above until iteration n-2 (itern-2, i.e., the n-1th iteration). ranki receives the last KV cache block in t[0]~t[n-1] and uses the local Q... i Attention is calculated between the received KV cache blocks and the previous iteration. In iteration n-1 (itern-1, i.e., the nth iteration), rank i completes the calculation of the last KV cache block in t[0]~t[n-1] and the sequence segment Q. i Attention calculation. Thus, Q is completed on rank i. i Attention calculation is performed for all t[0] to t[n-1]. An attention calculation is completed by concatenating the attention calculation results for all ranks.

[0047] Figure 3 The diagram illustrates the iterative process involved in performing one Ring All-Gather operation using eight computing devices. For ease of understanding, the sequence length S can be assumed to be 560,000 (i.e., 560k), meaning that eight computing devices are needed to perform attention calculations between 560k query tokens and 560k KV token pairs.

[0048] First, based on equation (1), the original KV cache blocks and query sequence segments can be allocated to each device (at this time, it can be assumed that the allocation of the query sequence and the KV cache are aligned). Then, rank 0 is allocated Q0 = [q0, q...]. 70k ), t[0]=KV[0,70k); Q1 assigned to rank 1=[q 70k ,q 140k ), t[1]=KV[70k,140k); and so on, until rank 7 is assigned Q7=[q 490k ,q 560k ), t[1]=KV[490k,560k).

[0049] After the allocation is completed, proceed as follows: Figure 3The iterative calculation is shown. In iter 0, rank 0 passes t[0] to the next device rank 1, receives t[7] passed by the previous device rank 7, and can perform attention calculation between Q0 and t[0]; rank 1 passes t[1] to the next device rank 2, receives t[0] passed by the previous device rank 0, and can perform attention calculation between Q1 and t[1]; ..., rank 7 passes t[7] to the next device rank 0, receives t[6] passed by the previous device rank 6, and can perform attention calculation between Q7 and t[7].

[0050] In iter 1, rank 0 passes the t[7] received in the previous iteration to the next device rank 1, receives the t[6] passed by the previous device rank 7, and can perform Q. θ Attention calculation between Q1 and t[7]; rank 1 passes t[0] received in the previous iteration to the next device rank 2, and at the same time receives t[7] passed by the previous device rank 0, and can perform attention calculation between Q1 and t[0]; ..., rank 7 passes t[6] to the next device rank 0, and at the same time receives t[5] passed by the previous device rank 6, and can perform attention calculation between Q7 and t[6].

[0051] This process continues until iterative iteration 6, where rank 0 passes the received t[2] from the previous iteration to the next device rank 1, receives t[1] from the previous device rank 7, and can perform attention calculation between Q0 and t[2]; rank 1 passes the received t[3] from the previous iteration to the next device rank 2, receives t[2] from the previous device rank 0, and can perform attention calculation between Q1 and t[3]; ..., rank 7 passes t[1] to the next device rank 0, receives t[0] from the previous device rank 6, and can perform attention calculation between Q7 and t[1]. Thus, after 7 (=n-1) iterations, each rank has received all the KV tokens.

[0052] Therefore, in iter 7, there is no need to pass t[i] anymore (because as shown in the figure, if it is passed, then each rank i will get its originally assigned t[i]), and only the final round of attention calculation needs to be completed, that is: rank0 performs attention calculation between Q0 and t[1] received in the previous iteration to complete the attention calculation between Q0 and all 560k KV token pairs; similarly, rank 1 to rank 7 also complete the attention calculation between their respective sequence segments and the KV cache blocks received in the previous iteration to complete the attention calculation between Q1 to Q7 and all 560k KV token pairs. By concatenating these attention calculation results, the attention calculation between 560k query tokens and 560k KV token pairs can be completed.

[0053] While vanilla Ring Attention reduces storage requirements to some extent and allows for overlap between computation and communication, the Ring All-Gather communication used has limited efficiency and fails to fully utilize the point-to-point interconnect bandwidth provided by modern accelerators. To address this, this disclosure proposes an optimized scheme for multi-path parallel ring communication using Hamiltonian decomposition. This scheme can be implemented as the LLM inference method and system described below. By transforming a single ring communication between multiple computing devices into multiple parallel ring communications, it improves communication efficiency and network resource utilization between computing devices while retaining the advantages of Ring Attention, such as overlap between communication and computation and low additional storage requirements.

[0054] Figure 4 A schematic flowchart of an LLM inference method according to an embodiment of this disclosure is shown. This LLM inference method is applicable to computing clusters with multiple computing devices. In the context of large model inference, a computing cluster refers to an overall system comprising multiple computing devices that work collaboratively to complete large-scale computational tasks. In sequential parallelism, computing devices are the basic units that perform operations such as matrix operations, attention calculations, and cache management during model inference or training. A computing device typically corresponds to a graphics processing unit (GPU), but can also be a tensor processor (TPU), neural network processor (NPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or high-performance central processing unit (CPU), etc. In a distributed execution framework, each computing device typically corresponds to a logical number, i.e., the aforementioned rank, which is used to identify the role and order of different computing devices in the computing cluster.

[0055] In step S410, a topology graph is generated based on the communication connections of multiple computing devices within the computing cluster, and the topology graph is decomposed using Hamiltonian to obtain multiple directed loops that are orthogonal to each other. In this topology graph, each computing device corresponds to a vertex, and every communication connection between two computing devices that satisfies predetermined conditions corresponds to an edge in the topology graph.

[0056] To describe the communication relationships between computing devices, a directed graph G = (V, E) can be used for modeling. Each vertex in the vertex set V corresponds to a computing device and its rank, and each edge in the edge set E represents the data transmission relationship between the devices. In a computing cluster, if the communication connection between two computing devices meets predetermined conditions—for example, in the case of a direct connection, the communication bandwidth is greater than or equal to a preset threshold; in the case of an indirect connection, bandwidth meeting the threshold can be allocated through a path—then it can be considered that these two computing devices can perform bidirectional communication. In the topology graph, this corresponds to a connection between two vertices. To clarify the direction of data flow, this connection is represented in the topology graph by two directed edges: one directed edge represents the data transmission capability from device A to device B, and the other directed edge represents the data transmission capability from device B to device A, reflecting the bidirectional communication capability between the devices.

[0057] After generating the topology graph, Hamiltonian decomposition can be performed to obtain multiple orthogonal directed cycles. Hamiltonian decomposition can involve dividing the edges of the topology graph into several non-intersecting Hamiltonian cycles—that is, multiple orthogonal directed cycles where each cycle starts from its origin, passes through every vertex in the graph exactly once, and returns to its origin, with each edge appearing in only one cycle. In a computing cluster, vertices correspond to computing devices, and edges correspond to communication connections between computing devices. Hamiltonian decomposition can construct multiple circular communication paths. These circular communication paths can be used for the simultaneous propagation of key-value tokens in subsequent steps, thereby fully utilizing existing communication connections and reducing the amount of data that needs to be transmitted on each path, thus improving communication efficiency and network resource utilization.

[0058] After identifying multiple orthogonal directed loops, data allocation and iterative operations can be performed for the current attention computation. Therefore, in step S420, a query sequence segment and a block of the original key-value cache can be allocated to each computing device.

[0059] In this disclosure, the query sequence segments and the original KV cache blocks can be allocated in any suitable manner. In one embodiment, the KV cache can be partitioned and allocated according to Equation (1) as described above, along with the partitioning and allocation of the query sequence segments aligned thereto. In other embodiments, other partitioning forms may also be used, such as special partitioning forms that take into account causal masks (see Equation (2) and its related description), etc. However, to maintain a balance between computation and communication, it is preferable to divide the query sequence and KV cache equally and allocate query sequence segments and original KV cache blocks of the same length to each computing device.

[0060] After acquiring multiple directed loops and completing the initial allocation of query sequences and KV caches, the iterative process can begin, unlike... Figure 3 The iterative process of the vanilla Ring Attention method shown in this disclosure involves parallel data transfer across multiple directed loops. To this end, step S430 may include: in each subsequent iteration, each computing device simultaneously transfers and receives KV cache sub-blocks across multiple directed loops, where the KV cache sub-blocks are obtained by partitioning the original KV cache blocks. Here, "subsequent" refers to the operation after the data allocation step S420, and this operation involves multiple iterations. In a system composed of multiple computing devices, assuming there are n computing devices (where n is an integer not less than 3, and preferably, n is an integer not less than 8 and divisible by 4), each directed loop contains these n computing devices (i.e., each Hamiltonian ring contains n vertices). To ensure that the data transferred in the loop can be obtained by all n devices, the iterative process needs to be performed n-1 times. Therefore, each subsequent iteration can correspond to n-1 loop operations after the data allocation is completed. As mentioned earlier, it is generally believed that completing the overall attention calculation requires an nth iteration process so that each computing device can complete the attention calculation between the locally allocated query sequence segment and the KV cache block obtained in the (n-1)th iteration.

[0061] In different embodiments, the multiple directed loops used to pass KV cache sub-blocks can be all the directed loops obtained in step 410 above, or a subset of these directed loops. Each computing device can divide the allocated KV cache blocks to obtain multiple KV cache sub-blocks, and each KV cache sub-block participates in the circular transfer of one loop in the multiple directed loops.

[0062] To fully utilize bandwidth, in a preferred embodiment, the KV cache sub-blocks can be passed and received using each directed loop obtained in step S410. For ease of explanation, the obtained directed loops can be assumed to be m directed loops, where m is an integer not less than 2. Each computing device can divide its allocated original KV cache block into m KV cache sub-blocks, and in the first iteration: each computing device passes its m KV cache sub-blocks to the next computing device in the corresponding loop of the m directed loops. That is, in the first iteration: each computing device passes its m KV cache sub-blocks to the next computing device connected after it in the m directed loops. In other words, for each KV cache sub-block possessed by a computing device, the computing device passes the KV cache sub-block to the next computing device connected after it in the directed loop corresponding to that KV cache sub-block. For example, if the original KV cache block assigned to rank i is t[i], for instance, when m=4 (meaning there are four loops ring 0, ring 1, ring 2, and ring 3), t[i] is divided into t[0,i], t[1,i], t[2,i], and t[3,i]. In the first iteration, t[0,i] is passed to the next vertex of the i-th vertex in ring 0, t[1,i] is passed to the next vertex of the i-th vertex in ring 1, t[2,i] is passed to the next vertex of the i-th vertex in ring 2, and t[3,i] is passed to the next vertex of the i-th vertex in ring 3. Further, step S430 may include: in each iteration after the first iteration, each computing device simultaneously receives the KV cache sub-blocks passed from the previous computing device in this round of iteration on each directed loop, and passes the KV cache sub-blocks received in the previous round of iteration to the next computing device. The following will be illustrated with specific embodiments and references. Figure 5 and Figure 6 The paper elaborates on the partitioning methods of multiple directed loops and the multi-path parallel transmission process of KV cache sub-blocks.

[0063] Therefore, the LLM inference method disclosed herein transforms unidirectional ring communication between multiple computing devices into multiple parallel unidirectional ring communication by utilizing Hamiltonian decomposition. While maintaining the existing advantages of Ring Attention, it makes full use of the existing communication connections of computing devices, thereby improving communication efficiency and increasing the utilization rate of network resources between computing devices.

[0064] As mentioned earlier, with the growth in LLM scale, inference deployment increasingly relies on computing clusters and strategies such as tensor parallelism (TP) and expert parallelism (EP). These parallelization strategies place stringent requirements on the interconnectivity between devices: any pair of devices must support high-bandwidth, low-latency point-to-point (P2P) communication, and multiple communications should be able to execute concurrently without interference to efficiently complete aggregate communication operations such as All-to-All and All-Gather. To meet these requirements, full-pair, full-topology has become the preferred solution for interconnecting networks in cutting-edge AI accelerators. Its physical implementation mainly includes the above-mentioned references. Figure 1 The two categories are full mesh and switch-based.

[0065] The inference scheme disclosed herein is particularly applicable to applications under all-pair, all-topology scenarios and to Hamiltonian decomposition of complete graphs. In this case, step S410 may include: in response to the plurality of computing devices being n computing devices and the communication connections between the plurality of computing devices being fully interconnected communication connections, generating a complete graph K. n As the topological graph, where n is an integer not less than 8 and divisible by 4; and for the complete graph K n Perform Hamiltonian decomposition to obtain n-1 Hamiltonian cycles as the plurality of directed cycles. Specifically, when n is an integer not less than 8 and divisible by 4, the complete graph K can be decomposed in O(n) time complexity using the Hamiltonian decomposition algorithm. n The decomposition process is performed. Based on this decomposition, the n×(n-1) communication links are divided into n-1 Hamiltonian rings with mutually disjoint edges. These rings are then used as mutually orthogonal circular data paths to support concurrent data transmission.

[0066] Figure 5 The diagram shows the complete topological graph of 8 vertices and the 7 orthogonal directed loops obtained by its Hamiltonian decomposition. Figure 5 The eight vertices on the left correspond to eight fully interconnected computing devices, specifically eight computing devices located within a single computing node (e.g., an inference server). Each edge in the graph represents a bidirectional connection between the two vertices and corresponds to the complete graph K8. A Hamiltonian decomposition of the complete graph K8 yields... Figure 5 The Hamiltonian ring shown on the right has 7 (n-1) mutually non-intersecting edges, that is, 7 mutually orthogonal directed rings, which can be regarded as ring 0 to ring 7 from left to right. There are no duplicate edges in these 7 directed rings, so even if they are used to transmit data simultaneously, they will not cause communication interference to each other.

[0067] Figure 6 An example of multi-ring full-collection communication according to one embodiment of the present disclosure is shown. Figure 6In the symbol t[j,i], it represents the KV cache sub-block originally assigned to rank i and passed along ring j in subsequent steps, where i = 0,1,...n-1 and j = 0,1,...m-1.

[0068] Specifically, Figure 6 Corresponding to Figure 5 The example shown illustrates how eight computing devices can form seven mutually orthogonal directed loops, all corresponding to the entire topology. By fully utilizing these seven directed loops (where m = n-1 = 7), the transfer of the KV cache and attention computation can be efficiently completed in eight iterations (where n = 8). It should be understood that the first seven iterations are sufficient for all KV cache transfers, and the eighth iteration (i.e., iter 7 in the diagram) requires only attention computation for each rank, using the KV cache sub-blocks obtained in the previous iteration and the local sequence segment.

[0069] like Figure 6 As shown in the first column, the t[0] originally allocated to rank 0 is divided into seven parts: t[0,0], t[1,0], t[2,0], t[3,0], t[4,0], t[5,0], and t[6,0]. In iteration 0, these seven KV tokens are passed to the next node in the ring from ring 0 to ring 6. Specifically, t[0,0] is passed to rank 1 in ring 0; t[1,0] is passed to rank 3 in ring 1; t[2,0] is passed to rank 7 in ring 2; t[3,0] is passed to rank 6 in ring 3; t[4,0] is passed to rank 2 in ring 4; t[5,0] is passed to rank 4 in ring 5; and t[6,0] is passed to rank 5 in ring 6 (due to...). Figure 6 For simplicity, rings 2 through 5 are omitted; therefore, the transitive relationships on these loops can be found in [reference needed]. Figure 5 In other words, in the first iteration, rank 0 distributed all of its originally allocated t[0] through 7 paths, and each of the other ranks (i.e., rank 1 to rank 7) received a portion of t[0] through different paths.

[0070] Similarly, Figure 6As shown in the first column, the KV tokens originally allocated to rank 1 through rank 7 were also divided into 7 parts and passed to the next node in the ring through rings 0 through 6 in iter 0. Therefore, after iter 0 ends, each rank passes out its originally allocated KV tokens and obtains one part of the originally allocated KV tokens from the other 7 ranks.

[0071] Repeat the above iterative process until iteration 6 (iter 6), where each rank obtains all KV tokens and in iteration 7 (iter 7), each rank completes the attention calculation with all KV tokens, thus completing one attention calculation involved in LLM inference.

[0072] As above Figure 5 and Figure 6 The example shown can be implemented in particular as a sequential parallel scheme within a single node with eight computing devices. In many cases, the computing cluster can also be expanded to multiple nodes, each with the same device configuration, and they can collaborate on computation through communication.

[0073] In multi-node systems, such as existing 8-accelerator server clusters, each high-speed server can be considered a node, interconnected via a high-bandwidth InfiniBand (IB) network. Each server is equipped with eight inference accelerators (e.g., GPUs), and all eight accelerators are fully interconnected through four ultra-high-bandwidth switching chips (e.g., NVSwitch). Each accelerator is also equipped with an IB network interface card (NIC), enabling it to independently and concurrently perform Remote Direct Memory Access (RDMA) data transfers via its own dedicated 400Gb / s link.

[0074] Such multi-node systems can also utilize the methods described above in this disclosure for communication topology modeling, Hamiltonian loop partitioning, and parallel distribution of KV tokens. For ease of understanding, a two-node computing cluster component will be used as an example first. Figure 7 This illustrates an example of the physical connections, communication topology modeling, and acquisition of multiple directed loops in a 2-node computing cluster.

[0075] like Figure 7As shown in the upper left, for ease of understanding, the 8 accelerators in node 1 are directly represented as 8 vertices 0-7, and the 8 accelerators in node 2 are directly represented as 8 vertices 8-15. Every two accelerators within a node can be interconnected via a dedicated high-speed channel (e.g., an ultra-high 7.2TB / s (bidirectional) aggregated bandwidth communication link NVLink), while every two accelerators across nodes are interconnected via an 800GB / s (bidirectional) aggregated bandwidth communication link from an IB NIC. This is assuming that there is an equivalent bidirectional connection between each vertex (corresponding to the generation of the complete graph K). 16 The communication topology modeling and decomposition scheme of 15 directed loops, namely, K 8×2 If we break it down, each node has 7 × 8 = 56 communication links internally, and 8 × 8 = 64 communication links between nodes. Therefore, the bandwidth of each intra-node communication link is approximately 7200 / 56 ≈ 128 GB / s, while the bandwidth of each inter-node communication link is approximately 800 / 64 ≈ 12.5 GB / s. The inter-node bandwidth is an order of magnitude lower than the intra-node bandwidth, thus limiting the utilization rate of intra-node bandwidth.

[0076] Therefore, in a more reasonable communication topology modeling scheme, it can be like this Figure 7 The topology model of the two connected nodes shown in the upper left corner is two complete graphs K8, interconnected by 8 bidirectional or 16 unidirectional IB communication links (denoted as (K... q -qK q ) p Decomposition, where in this example q=8, p=2, meaning the computing cluster includes p computing nodes, and each computing node includes q computing devices, such as... Figure 7 As shown in the upper right corner. This decomposition scheme is based on the complete graph K within each node. q Hamiltonian path decomposition. As mentioned earlier, Hamiltonian decomposition refers to the process of dividing the edge set of a graph into several Hamiltonian cycles, for example, dividing a complete graph K into several Hamiltonian cycles. q The graph is divided into q-1 Hamiltonian cycles. Hamiltonian path decomposition can be seen as a variant of Hamiltonian decomposition, where each Hamiltonian path passes through each vertex in the graph exactly once, but does not return to the starting point of the path. Through Hamiltonian path decomposition, the edges of the graph can be divided into several non-overlapping Hamiltonian paths, thus providing non-interfering communication paths for data transmission within nodes. Complete graph K q It can be divided into q Hamiltonian paths.

[0077] Since in the previous example the bandwidth between the two nodes is divided by q 2 Then (i.e., equivalent to generating the complete graph K) 2qThe bandwidth between two computing devices across a node (e.g., 12.5 GB / s in the previous example) is much smaller than the bandwidth between two computing devices within a node (e.g., 128 GB / s in the previous example). Therefore, a less aggressive topology modeling approach is adopted, that is, the vertex interconnections within a node are equivalent to a complete graph K. q The interconnections between nodes are equivalent to q communication links. Then, the complete graph K... q The resulting q Hamiltonian paths can be connected end-to-end through q communication links between nodes, thus obtaining q Hamiltonian cycles across nodes, i.e., q mutually orthogonal directed cycles, each containing pq vertices. For example... Figure 7 As shown below, each of the two complete graphs K8 can be decomposed into eight Hamiltonian paths. Connecting each pair of Hamiltonian paths from the two complete graphs K8 yields eight Hamiltonian cycles. Specifically, the eight Hamiltonian paths obtained from the decomposition of the complete graph K8 corresponding to node 0 can be assigned numbers, such as paths P0 to P7. The same numbering can also be assigned to the eight Hamiltonian paths obtained from the decomposition of the complete graph K8 corresponding to node 1, such as paths P0 to P7. Thus, connecting the beginning and end of each path P0 from each complete graph yields a Hamiltonian cycle C0, connecting the beginning and end of each path P1 from each complete graph yields a Hamiltonian cycle C1, and so on, until a Hamiltonian cycle C7 is obtained, thus obtaining eight Hamiltonian cycles as multiple directed loops.

[0078] Since each element in each row or column of the Latin square is distinct, all eight ranks will appear at the start and end points of the decomposed Hamiltonian path, thus fully utilizing all 2×8=16 IB NIC links. Using this modeling and decomposition method, the theoretical bandwidth of each unidirectional intra-node communication link (grey) is approximately... The theoretical bandwidth of each unidirectional inter-node communication link (blue) is approximately The difference is only 28%, far less than that of using K. 8×2 The differences in decomposition significantly improve the utilization of bandwidth within nodes.

[0079] Therefore, step S410 may include: in response to the computing cluster comprising p computing nodes and each computing node comprising q devices with fully interconnected communication connections (at this time, the computing cluster comprises n computing nodes, and n = pq) and the communication bandwidth between computing nodes being within a predetermined threshold range, dividing the plurality of computing devices into p subsets of computing devices (a subset of computing devices comprises q computing devices in a computing node) to generate p complete graphs K. q and adjacent complete graph K qThe topological graph of the graph with q connecting lines, where p is an integer not less than 2 and q is an integer not less than 3 (more preferably, q is an integer not less than 8 and divisible by 4); for each complete graph K q Perform Hamiltonian decomposition to obtain q Hamiltonian paths and assign each of the q Hamiltonian paths a number from 0 to q-1; and process the data from p complete graphs K... q Hamiltonian paths with the same number are connected end-to-end to form a Hamiltonian ring, resulting in q Hamiltonian rings as the plurality of directed loops. Here, the communication bandwidth between computing nodes being within a predetermined threshold range can mean that the difference between the bandwidth of unidirectional communication connections between nodes and the unidirectional bandwidth of fully interconnected communication within nodes does not exceed a predetermined percentage. Figure 7 In the physical connection, the bandwidth of the unidirectional communication connection between nodes is determined by the bandwidth of the IB NIC link (which can be set as B here) and the number of computing devices q within a single node (the theoretical bandwidth is B / 2q). That is, as mentioned above, the theoretical bandwidth of each unidirectional communication link between nodes is approximately

[0080] Furthermore, it can be proven by mathematical induction that the above decomposition scheme can be extended to any number of nodes through a high-bandwidth network, as long as the communication bandwidth between computing nodes is within a predetermined threshold range. At this point, step S410 of communication topology generation and Hamiltonian decomposition can also implement the addition of new nodes and the growth of Hamiltonian rings, and may include: in response to the addition of a new computing node as the (p+1)th computing node to the computing cluster, generating a new complete graph K. n ; Regarding this new complete graph K n The process involves performing Hamiltonian decomposition to obtain q new Hamiltonian paths; then, disconnecting q connections between any two computing nodes in the q Hamiltonian rings, and connecting the new q Hamiltonian paths to the q Hamiltonian rings from the disconnection points, thus forming a new q Hamiltonian ring that incorporates the q new vertices corresponding to the (p+1)th computing node. This decomposition scheme also requires that the communication bandwidth between computing nodes be within a predetermined threshold range; for example, the difference in unidirectional bandwidth between nodes and the difference in unidirectional bandwidth between fully interconnected nodes within a node does not exceed a predetermined percentage.

[0081] For ease of operation, when adding a new node (i.e., the (p+1)th node, corresponding to node p) to a computing cluster with p existing computing nodes and q Hamiltonian paths, it is preferable to disconnect and insert the node from the first node (node ​​0) and the last node (node ​​p-1) of the previous p computing nodes. The decomposition of the p+1 node cluster can then be achieved in two steps: first, disconnect the communication links from rank(8p-8) to (8p-1) to rank 0 to 7; then connect rank(8p-8) to (8p-1) to rank(8p) to (8p+7), and connect rank(8p) to (8p+7) to rank 0 to 7.

[0082] Figure 8 This diagram illustrates an example of Hamiltonian path construction when adding a new node to two existing computing nodes. As shown in the upper part of the diagram, node 0 comprises ranks 0-7, and node 1 comprises ranks 8-15, forming eight Hamiltonian cycles. Each cycle includes a Hamiltonian path from node 0 and a Hamiltonian path from node 1. Now, node 2, comprising ranks 16-23, needs to be added. To do this, as shown in the middle part of the diagram, one cross-node edge in each of the eight Hamiltonian cycles is disconnected (corresponding to the red cross in the diagram). It should be understood that, since p=2, theoretically, any one of the two cross-node edges can be chosen, as these two cross-node edges are communication links from ranks 8-15 to ranks 0-7. Finally, as shown in the lower part of the diagram, ranks 8-15 are connected to ranks 16-23, and ranks 16-23 are connected to ranks 0-7, thus creating eight new Hamiltonian cycles, each comprising 24 vertices (q(p+1) = 8 × (2+1) = 24). It should be noted that although node 2 was incorporated into the input, the requirement that the communication bandwidth between nodes is within the predetermined threshold range is still met, since the theoretical bandwidth of each unidirectional communication link between nodes is determined only by B / 2q (i.e., it is independent of the value of p).

[0083] The communication topology modeling and Hamiltonian decomposition method disclosed herein, which obtains multiple mutually orthogonal directed loops and subsequently performs KV cache sub-block transfer based on these multiple directed loops, is particularly suitable for application scenarios of single servers where the internal computing devices satisfy the fully interconnected condition as described above (i.e., constructing a complete graph KV). n And obtain n-1 directed loops), and application scenarios involving multiple stand-alone machines interconnected through high-bandwidth networks (i.e., (K q -qK q ) p(Decompose to obtain q directed loops). However, it should be understood that the scheme disclosed herein is also applicable to other application scenarios, such as application scenarios where the computing devices inside a single server do not meet the full interconnection condition but can still be decomposed into multiple directed loops, and application scenarios where there are multiple bidirectional paths between two computing devices in a computing cluster.

[0084] For relatively simple interconnection scenarios, step S410 can be implemented by establishing an edge between corresponding vertices when the multiple computing devices are interconnected using the same communication method and / or directly interconnected, and the bidirectional bandwidth between two computing devices is greater than a preset first threshold. For example, when the connections between multiple computing devices are all physically direct, the establishment of edges in the topology graph can be determined directly based on whether the bidirectional bandwidth under the direct connection is greater than the first threshold. Here, the first threshold can be a fixed value or can vary with the amount of data to be processed. In cases where the connections between multiple computing devices are not physically direct but use the same communication interconnection method, for example... Figure 1 When using fully interconnected communication based on switches on the right, the establishment of edges in the topology graph can also be determined directly based on the ratio of the achievable bidirectional bandwidth between each pair of vertices to the first threshold.

[0085] In complex interconnection scenarios involving heterogeneous connections (typically involving cross-node connections), step S410 can be implemented as follows: when the multiple computing devices are interconnected using two or more communication methods, including indirect connections via exchange nodes, determine the number of edges established via exchange nodes such that the difference between the equivalent bidirectional bandwidth of each edge obtained therefrom and the bidirectional bandwidth established via other communication methods is no greater than a second threshold. At this point, the method for dividing the shared bandwidth communication via exchange nodes is deduced based on the clearly defined bidirectional bandwidth between vertices (e.g., the bandwidth under a full-pair, full-topology condition within a node). That is, after the shared bandwidth communication is divided, the equivalent bidirectional bandwidth of each edge should not differ too much from the clearly defined bidirectional bandwidth between vertices (the difference can be quantified based on the second threshold, which in different implementations can be a specific bandwidth difference or a ratio).

[0086] In different implementations, the allocation of the original KV cache blocks can be performed according to different schemes. For ease of explanation, let the multiple computing devices be n computing devices and the multiple directed loops be m directed loops, where n is an integer not less than 3 (preferably an integer not less than 8 and divisible by 4), and m is an integer not less than 2. In a very simple implementation, for example, without using a causal mask, the allocation of the original KV cache blocks can be performed sequentially. In this case, step S420 may include: dividing the KV cache into n KV cache blocks along the sequence length direction, and allocating one of the n KV cache blocks to each computing device as the original KV cache block for that computing device; each computing device sequentially divides its allocated original KV cache block into m sequentially numbered KV cache sub-blocks, which are used for the first iteration of the transmission on the m directed loops. In one embodiment, the allocation of the query sequence segment can be aligned with the allocation of the KV cache blocks.

[0087] For ease of understanding, we will use the example of a sequence length of 560,000 (i.e., 560k) and n = 8, m = 7. First, we can allocate the original KV cache blocks to each device based on Equation (1). At this time, rank 0 is allocated to t[0] = KV[0, 70k); rank 1 is allocated to t[1] = KV[70k, 140k); and so on, until rank 7 is allocated to t[1] = KV[490k, 560k). Unlike vanilla ring attention, since there are 7 directed loops that can transmit KV tokens in parallel, each device needs to further subdivide the allocated original KV cache blocks. For example, rank 0 further subdivides t[0] = KV[0, 70k) into t[0, 0] = KV[0, 10k), t[1, 0] = KV[10k, 20k), t[2, 0] = KV[20, 30k), t[3, 0] = KV[30k, 40k), t[4, 0] = KV[40k, 50k), t[5, 0] = KV[50, 60k), and t[6, 0] = KV[60, 70k). These 7 KV cache sub-blocks are then processed in iteration 0 according to, for example... Figure 5 The data is passed to the next vertex in rings 0 through 6, and similarly, rank 1 through rank 7 can also be divided into KV cache sub-blocks and passed in iteration 0. Thus, in iteration 7, the passing of all KV cache data in rank 0 through rank 7 is completed. It should be understood that, compared to the previous combination... Figure 2In the transmission of KV tokens on a single directed loop, the amount of data transmitted on each unidirectional link of the directed loop decreases from 70k (i.e., S / n) to 10k (i.e., S / nm) in each iteration. This improves communication efficiency and reduces the impact of communication bottlenecks on inference by making full use of the communication links between computing devices.

[0088] However, when using causal masks, simple block allocation strategies (e.g., the sequential allocation strategy mentioned above) can disrupt the balance of computational load. If a simple block placement strategy is used while employing causal masks, then in the i-th iteration, i accelerators will be idle, while only ni accelerators will be performing computations. To achieve load-balanced attention computation when using causal masks, a special block allocation strategy can be adopted. This involves each computing device retrieving KV tokens symmetrically from the head and tail of the KV cache, splitting each of the two retrieved KV tokens, and combining them into two sets before using them for transmission across multiple directed loops. For ease of explanation, we can also assume that there are n computing devices and m directed loops, where n is an integer not less than 3 and m is an integer not less than 2. At this point, step S420 includes: dividing the KV cache into 2n KV cache blocks along the sequence length direction; allocating the i-th KV cache block and the i-th-last KV cache block to the i-th computing device as the original KV cache blocks for that computing device (while, the query sequence segment allocated on each computing device is aligned with the allocated KV cache block), where i = 1, ..., n; the i-th computing device divides the i-th KV cache block and the i-th-last KV cache block into m KV cache sub-blocks each, and pairs the m KV cache sub-blocks of the i-th KV cache block and the m KV cache sub-blocks of the i-th KV cache block for the first iteration of the transmission on the m directed loops. Similarly, in the example where the sequence length is 560k and n = 8, m = 7, the 560k KV tokens are sequentially divided into 16 KV cache blocks, each block including 35k KV tokens. The first and last blocks can be allocated to the first computing device (corresponding to rank 0 here), that is, t[0] = KV[0,35k)∪KV[525,560k), KV[0,35k) can be divided into 7 sub-blocks, which are provided to t[0,0], t[1,0], t[2,0], t[3,0], t[4,0], t[5,0] and t[6,0] respectively. KV[525,560k) can also be... The blocks are divided into 7 sub-blocks, which are also provided to t[0,0], t[1,0], t[2,0], t[3,0], t[4,0], t[5,0] and t[6,0] respectively. Thus, t[0,0], t[1,0], t[2,0], t[3,0], t[4,0], t[5,0] and t[6,0] each include 5k KV tokens from KV[0,35k) and 5k KV tokens from KV[525,560k), which, like the simple allocation strategy mentioned above, also includes 10k KV tokens.Similarly, the second chunk and the second-to-last chunk are assigned to the second computing device (here corresponding to rank 1), until the middle two chunks (i.e., the eighth chunk and the eighth-to-last chunk (the ninth chunk)) are assigned to the eighth computing device (here corresponding to rank 7), and the sub-chunk division, pairwise combination and transfer of each chunk are also performed, thereby ensuring that each rank involves the same amount of computation in each iteration.

[0089] In one embodiment, the above sub-chunk allocation method can be defined by a specific arithmetic formula. Define the KV chunk t[j,i] at rank i, which is transmitted through the j-th directed loop (i = 0, 1,..., n and j = 0, 1,..., m) and can be further divided into two parts:

[0090] Here, n is the number of computing devices in the computing cluster, and S is the sequence length. It can be proved that the KV cache sub-chunks allocated according to the above can achieve perfect load balancing.

[0091] Suppose for a certain rank r0, in a certain iteration iter0, it receives the KV chunk t KV [j0,i0][0] and t KV [j0,i0][1], and at the same time, query chunks t Q [j,r0][0] and reserved on rank r0.

[0092] If i < r0, then all token indices of the query chunk t Q [j,r0][0] and are greater than the token indices of t KV [j0,i0][0], but less than the token indices of t KV [j0,i0][1]. So, only t KV [j0,i0][0] needs to be noted, that is, only half of the KV tokens transmitted to rank r0 via path j0 need to be subjected to attention calculation. If i > r0, then all token indices of KV are less than the token indices of t KV [j0,i0][0] and t all token indices of KV are greater than the token indices of t KV [j0,i0][1]. So, only That is, attention computation is only required for half of the local query tokens allocated on rank r0. Therefore, for any rank r0, its computational load remains the same in any iteration, thus achieving 100% load balancing.

[0093] Furthermore, it's important to understand that while the above description focuses on the partitioning of multiple directed loops and the passing of KV cache sub-blocks during the iteration process, this iteration process also involves attention computation and KV cache management on each computing device. In a computing cluster utilizing the inference scheme of this disclosure, each computing device needs to cache the locally allocated query sequence segment (a query token of length S / n) and KV token pairs of length 2S / n during the iteration process. In each iteration, local attention computation is performed simultaneously with the passing of KV cache sub-blocks, thus achieving parallel communication and computation.

[0094] See back Figure 5 and Figure 6 In iter 0, rank 0 can distribute t[0,0], t[1,0], t[2,0], t[3,0], t[4,0], t[5,0] and t[6,0] on seven rings, while receiving t[0,7] from rank 7 on ring 0, receiving t[1,2] from rank 2 on ring 1, receiving t[2,4] from rank 4 on ring 2, receiving t[3,1] from rank 1 on ring 3, receiving t[4,3] from rank 3 on ring 4, receiving t[5,5] from rank 5 on ring 5, and receiving t[6,6] from rank 6 on ring 6, and performing attention calculation with t[0] on local Q0. In other words, at iter 0, rank 0 needs to cache t[0] and the received t[0,7], t[1,2], t[2,4], t[3,1], t[4,3], t[5,5], t[6,6], which is a KV token pair of length 2S / n, and perform attention calculation of length S / n.

[0095] In iter 1, rank 0 performs local attention computation on the 7 KV cache subblocks received from iter 0, while simultaneously passing them in parallel and receiving 7 new KV cache subblocks from the 7 paths. Therefore, it also needs to cache KV token pairs of length 2S / n and perform attention computation of length S / n. In each subsequent iteration (i.e., from iter 2 to iter n-2), rank 0 needs to cache KV token pairs of length 2S / n and perform attention computation of length S / n. Similarly, other ranks also need to cache KV token pairs of length 2S / n and perform attention computation of length S / n in each iteration. In the nth iteration (i.e., iter n-1), each rank only needs to perform attention computation of length S / n, and finally completes one attention computation involved in LLM inference.

[0096] Therefore, the multi-ring parallel transmission scheme disclosed herein can maintain the same computational and storage intensity and communication computational parallelism as vanilla Ring Attention, while improving communication efficiency and network resource utilization by making full use of existing communication links.

[0097] The communication topology modeling, Hamiltonian decomposition, and multi-loop parallel KV cache sub-block transfer scheme of this disclosure have been described above with reference to the accompanying drawings, and can be implemented as an LLM inference method. This method, considering different network connection conditions between computing devices, constructs multiple orthogonal directed loops through Hamiltonian ring decomposition or, if further included, Hamiltonian path decomposition. These loops are used for the parallel transfer of KV cache sub-blocks in sequential parallelism, thereby improving communication efficiency and network resource utilization while maintaining both communication and computational parallelism and low computational and storage intensity.

[0098] In one embodiment, this disclosure can be implemented as a large language model inference system, including a computing cluster with multiple computing devices for performing the methods described above.

[0099] In one embodiment, this disclosure can also be implemented as a non-transitory machine-readable storage medium storing executable code that, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0100] In one embodiment, this disclosure can also be implemented as a computer program product, including computer program instructions that, when executed by a processor, implement the method described above.

[0101] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. For this purpose, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0103] Various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A large language model inference method, applicable to computing clusters with multiple computing devices, the method comprising: A topology graph is generated based on the communication connection relationship of the multiple computing devices, and Hamiltonian decomposition is performed on the topology graph to obtain multiple directed loops that are orthogonal to each other. Each computing device corresponds to a vertex in the topology graph, and a communication connection between any two computing devices that satisfies a predetermined condition corresponds to an edge in the topology graph. Allocate query sequence segments and raw KV cache blocks to each computing device; and In each subsequent iteration, each computing device simultaneously transmits and receives KV cache subblocks across multiple directed loops.

2. The method as described in claim 1, wherein, The obtained mutually orthogonal directed loops number m, where m is an integer not less than 2. Each computing device divides its assigned original KV cache block into m KV cache sub-blocks, and in the first iteration: Each computing device passes its m KV cache sub-blocks to the next computing device in each of the m directed loops.

3. The method as described in claim 1, wherein, In each subsequent iteration, each computing device simultaneously performs the transfer and reception of KV cache sub-blocks across multiple directed loops, including: In each iteration following the first iteration, each computing device simultaneously receives the KV cache sub-blocks passed from the previous computing device in this iteration on each directed loop, and passes the KV cache sub-blocks received in the previous iteration to the next computing device.

4. The method of claim 1, wherein, A topology graph is generated based on the communication connection relationships of the multiple computing devices, and the topology graph is subjected to Hamiltonian decomposition to obtain multiple directed loops that are orthogonal to each other, including: In response to the fact that there are n computing devices and the communication connection between the multiple computing devices is a fully interconnected communication connection, a complete graph K is generated. n As the topological graph, n is an integer not less than 8 and divisible by 4; and For the complete graph K n Perform Hamiltonian decomposition to obtain n-1 Hamiltonian cycles as the plurality of directed loops.

5. The method of claim 1, wherein, A topology graph is generated based on the communication connection relationships of the multiple computing devices, and the topology graph is subjected to Hamiltonian decomposition to obtain multiple directed loops that are orthogonal to each other, including: In response to the computing cluster comprising p computing nodes, each computing node comprising q computing devices with fully interconnected communication connections, and the communication bandwidth between computing nodes being within a predetermined threshold range, the plurality of computing devices are divided into p subsets of computing devices to generate p complete graphs K. q and adjacent complete graph K q The topology diagram with q connecting lines, where p is an integer not less than 2, q is an integer not less than 8 and divisible by 4, and the plurality of computing devices are pq computing devices; For each complete graph K q Perform Hamiltonian decomposition to obtain q Hamiltonian paths and assign each of the q Hamiltonian paths a number from 0 to q-1; and The p complete graphs K q Hamiltonian paths with the same number are connected end to end to form a Hamiltonian ring, so as to obtain q Hamiltonian rings as the plurality of directed loops.

6. The method of claim 5, wherein, A topology graph is generated based on the communication connection relationships of the multiple computing devices, and the topology graph is subjected to Hamiltonian decomposition to obtain multiple directed loops that are orthogonal to each other, including: In response to the addition of a new compute node as the (p+1)th compute node in the computing cluster, a new complete graph K is generated. q ; For this new complete graph K q Perform Hamiltonian decomposition to obtain q new Hamiltonian paths; and Disconnect q connections between any two computation nodes in the q Hamiltonian rings, and connect q new Hamiltonian paths from the disconnection points into the q Hamiltonian rings to form new q Hamiltonian rings that incorporate the q new vertices corresponding to the p+1th computation node.

7. The method of claim 1, wherein, A topology graph is generated based on the communication connection relationships of the plurality of computing devices, and Hamiltonian decomposition is performed on the topology graph to obtain a plurality of mutually orthogonal directed loops, including at least one of the following: When the multiple computing devices are interconnected using the same communication method and / or directly interconnected, and the bidirectional bandwidth between two computing devices is greater than a preset first threshold, an edge is established between the corresponding two vertices. as well as When the plurality of computing devices are interconnected using two or more communication methods, including indirect connections via exchange nodes, the number of edges established via exchange nodes is determined such that the difference between the equivalent bidirectional bandwidth of each edge obtained therefrom and the bidirectional bandwidth established via other communication methods is no greater than a second threshold.

8. The method of claim 1, wherein, The plurality of computing devices refers to n computing devices, and the plurality of directed loops refers to m directed loops, where n is an integer not less than 8 and divisible by 4, and m is an integer not less than 2. The allocation of query sequence segments and original KV cache blocks to each computing device includes: The key-value cache is divided into n equal blocks along the sequence length, and one block from these n blocks is allocated to each computing device as its original key-value cache block; and Each computing device divides its assigned original KV cache block into m sequentially numbered KV cache sub-blocks, which are used for the first iteration of the transfer on the m directed loops.

9. The method of claim 1, wherein, The plurality of computing devices refers to n computing devices, and the plurality of directed loops refers to m directed loops, where n is an integer not less than 8 and divisible by 4, and m is an integer not less than 2. The allocation of query sequence segments and original KV cache blocks to each computing device includes: Divide the KV cache into 2n KV cache blocks along the sequence length; The i-th KV cache block and the i-th-last KV cache block are allocated to the i-th computing device as the original KV cache blocks for that computing device, where i = 1, ..., n; and The i-th computing device divides the i-th KV cache block and the i-th-last KV cache block into m KV cache sub-blocks each. It then pairs the m KV cache sub-blocks of the i-th KV cache block with the m KV cache sub-blocks of the i-th-last KV cache block for use in the first iteration's propagation along the m directed loops. In this context, the query sequence segment allocated on each computing device is aligned with the allocated KV cache block.

10. A large language model inference system, comprising a computing cluster having multiple computing devices, the computing cluster being used to execute a method based on any one of claims 1 to 9.

11. A computer program product comprising computer program instructions that, when executed by a processor, implement the method as claimed in any one of claims 1 to 9.

Images