A high-efficiency all-to-all communication method for torus topology
The ShiftRing communication method solves the problems of resource waste and startup overhead in all-to-all communication under toroidal topology by using shortest path bidirectional transmission, staged aggregation scheduling, and in-situ forwarding, thus achieving efficient all-to-all communication. It is suitable for training and inference tasks of hybrid expert models and deep recommendation models.
Patent Information
- Application Number
- CN202611244189.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-17
- Publication Date
- 2026-11-06
AI Technical Summary
Existing ring-based all-to-all communication algorithms suffer from problems such as a large number of communication rounds, high startup overhead, and resource waste under toroidal topologies. In particular, they are difficult to perform all-to-all communication efficiently in training and inference tasks of hybrid expert models and deep recommendation models.
A ShiftRing communication method is proposed, which reduces communication startup overhead and additional buffer usage by performing shortest path bidirectional transmission, staged aggregation scheduling and in-situ forwarding in toroidal topology. It is suitable for toroidal topology environments with multiple computing nodes.
It achieves efficient all-to-all communication in toroidal topology, reduces communication overhead and startup latency, reduces memory consumption, is suitable for large-scale distributed training tasks, and improves communication efficiency and GPU memory utilization.
Smart Images
Figure CN123055645A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of aggregate communication, and in particular relates to an efficient all-to-all communication method for torus topology. Background Technology
[0002] In current machine learning, especially deep learning, to improve the efficiency of model training or inference, it is often necessary to utilize multiple computing nodes to jointly complete the same training or inference task. In this process, different nodes undertake parts of the computational task, and after completing their local computations, they also need to exchange and synchronize parameters, features, or intermediate results through the network. Therefore, the need for ensemble communication is widespread in distributed training and inference systems. Among various ensemble communication primitives, all-to-all communication has become increasingly important in recent years. Especially in the training and inference tasks of models such as Mixture of Experts (MoE) and Deep Learning Recommendation Model (DLRM), nodes often need to exchange tokens, sparse features, embedding vectors, or other intermediate results, requiring each node to send its local data to all other nodes and receive data from all other nodes. Compared with communication modes such as broadcast, reduction, and full reduction, the data flow in all-to-all communication is more discrete, the communication relationships are more complex, and it is usually difficult to completely overlap with computation, thus making it more likely to become a performance bottleneck in the entire training or inference process.
[0003] There are various implementations of existing all-to-all communication algorithms. Ring-based all-to-all methods organize all participating nodes into a logical ring, allowing data to be transmitted hop-by-hop between adjacent nodes in the ring to complete the communication task. Besides being directly applicable to ring (i.e., one-dimensional toroidal) topologies, they can also be extended to support multi-dimensional toroidal topologies. This type of method has a simple structure, is easy to deploy, and adapts well to the toroidal interconnect structures widely used in practical computing systems, thus having high practical value in engineering implementation. The bidirectional ring algorithm BidRing and the half-ring algorithm HalfRing are two typical ring-based all-to-all algorithms. BidRing utilizes the bidirectional bandwidth of the ring to improve communication efficiency, while HalfRing further employs shortest path transmission to avoid detours and improve efficiency. However, they both have two shortcomings: first, organizing data blocks for communication in a fine-grained manner results in more communication rounds and a larger cumulative startup overhead; second, a certain amount of buffer space needs to be reserved on each node to temporarily store the data blocks to be forwarded, leading to resource waste.
[0004] Therefore, in toroidal topology, how to reduce communication startup overhead and avoid allocating additional buffers for intermediate forwarding while maintaining stable communication relationships and simplifying implementation has become a key issue in all-to-all communication optimization. Summary of the Invention
[0005] To address the above problems, this invention provides an efficient all-to-all communication method for toroidal topology.
[0006] The present invention provides an efficient all-to-all communication method for toroidal topologies, specifically as follows:
[0007] Applicable Scenarios: This invention presents an efficient all-to-all communication method (ShiftRing) for toroidal topologies. It is suitable for communication environments where multiple computing nodes constitute a toroidal topology, with each participating node being a CPU server, GPU server, or other device capable of computing and communication. In a one-dimensional toroidal topology, all nodes are connected in numerical order to form a logical ring, and each node has direct communication with only two adjacent nodes, one clockwise and the other counterclockwise. The network environment can be either a physical ring interconnection structure or a logical ring structure abstracted from an actual cluster network. In a two-dimensional toroidal topology, communication based on a one-dimensional toroid is required simultaneously in both dimensions. In the case of an M-dimensional torus topology, the communication data is divided into M parts, and the M-dimensional torus topology is also divided into M one-dimensional torus topologies of different dimensions. At the same time, the i-th part of the data communicates on the i-th one-dimensional torus topology, where i=1,2,…,M. After that, the data that has been communicated in the i-th dimension communicates again in the remaining M-1 dimensions. After M rounds of communication, all data is transmitted in parallel in all dimensions, realizing full-to-full communication.
[0008] Initialization: ShiftRing initialization includes establishing the logical ring topology, determining adjacency relationships, and laying out the data in the local buffer.
[0009] Suppose there are N active nodes, numbered 0, 1, 2, ..., N-1; for any node p, its clockwise adjacent node is defined as... Counterclockwise adjacent nodes are defined as Each node holds a copy of the original local data and divides the original data into N logical sub-data blocks according to the target node. The q-th sub-data block represents the data part that needs to be sent to node q. The part of the data that is the same in the target node and the source node is directly kept locally, and the remaining data blocks participate in subsequent communication.
[0010] When a node performs ShiftRing communication, follow these steps:
[0011] Input: The total number of nodes N in the ring topology, the current node number p, and the set of local data blocks held by the current node.
[0012] Output: The set of result data obtained by the current node from all source nodes and with this node as the target.
[0013] Step 1: Read the total number of nodes N and calculate the number of main communication stages K; for a ring topology containing N nodes, the number of main communication stages is taken as... .
[0014] Step 2: On the current node p, divide the local raw data into N logical sub-data blocks according to the target node number, and write these data blocks into the corresponding slots of the local buffer.
[0015] Step 3: Starting from stage 0, execute sequentially up to stage K-1.
[0016] For any stage k, the current node calculates the logical starting offset positions in the clockwise and counterclockwise directions for that stage, respectively: , This is used to determine the range of data blocks that will be transmitted first in this round.
[0017] Step 4: Based on the stage number k, construct a clockwise transmission index list for the current node p. and counterclockwise transmission index list ;
[0018] The larger the stage k, the larger the index range included, and the more data blocks participate in the transmission in this round;
[0019] List and list Each contains k+1 indexes; The i-th index value is , The i-th index value is ,in ;
[0020] Therefore, communication will gradually cover all target data blocks in the order of "far first, then near".
[0021] Step 5: The current node sends the data block corresponding to the clockwise index list to the right neighbor node through the clockwise link, and receives the data continuing to advance clockwise from the left neighbor node; at the same time, it sends the data block corresponding to the counterclockwise index list to the left neighbor node through the counterclockwise link, and receives the data continuing to advance counterclockwise from the right neighbor node.
[0022] Single-hop transmission and reception in both directions are completed in parallel within the same phase.
[0023] Step 6: For new data received in this stage, instead of writing it back to the original fixed location, write it into the slot in the local buffer where the transmission task has been completed, according to the stage number and transmission direction; in this way, the new data can be retained as a result or forwarded as intermediate data in subsequent stages.
[0024] Step 7: If the total number of nodes N is odd, then after the main communication phase ends, all data blocks have completed the transmission on the shortest path, and the current node directly enters the result recovery phase.
[0025] If the total number of nodes N is even, there is another type of data block where the target node is located on the opposite side, and the clockwise and counterclockwise path lengths are both N / 2. For this type of data, an additional round of communication is added after the main communication phase to split the data and transmit it in both directions.
[0026] Step 8: After all communication is complete, treat the local buffer as a circular array, starting from the predetermined logical offset position. .
[0027] By starting to read consecutive data items sequentially, the correct order of all-in-one communication results can be recovered.
[0028] Furthermore, if the application scenario is a two-dimensional toroidal topology, the communication data is generally divided into two parts, and the two-dimensional toroidal topology is also divided into two one-dimensional toroidal topologies of different dimensions. At the same time, a part of the data is communicated in the horizontal direction, and the other part of the data is communicated in the vertical direction. After that, the data that was communicated in the horizontal direction is communicated again in the vertical direction, and the data that was communicated in the vertical direction is communicated again in the horizontal direction, so as to transmit in parallel in two dimensions.
[0029] Furthermore, during initialization, each node also establishes a local receive buffer; the receive buffer in ShiftRing is not only used to store the final received result, but also serves as the carrying area for subsequent intermediate forwarding; therefore, during the initialization phase, it is necessary to reserve corresponding slots for each logical sub-data block and establish a unified offset rule to provide a basis for subsequent in-place forwarding.
[0030] Furthermore, ShiftRing treats the buffer as a circular array and defines a fixed logical starting offset for each node; after communication is complete, the correct order of all-to-all communication results can be recovered simply by reading consecutive data items sequentially from that offset, without needing to perform additional data reordering operations.
[0031] The beneficial technical effects of this invention compared to the prior art are as follows:
[0032] On the one hand, bidirectional transmission via the shortest path reduces communication overhead; on the other hand, phased aggregation scheduling reduces startup latency; and furthermore, in-place forwarding reduces additional memory consumption. Compared to the classic ring-based all-to-all methods BidRing and HalfRing, the ShiftRing method of this invention is more suitable for deployment in large-scale distributed training tasks where all-to-all communication accounts for a high proportion. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of one-dimensional torus (i.e., torus) topology and two-dimensional torus topology.
[0034] Figure 2 This is a schematic diagram of the first stage of all-to-all communication on a 2D toroidal topology.
[0035] Figure 3 This is a schematic diagram of the second stage of all-to-all communication on a 2D toroidal topology.
[0036] Figure 4 This invention describes the complete communication process of ShiftRing on a 5-node ring.
[0037] Figure 5 This is a diagram illustrating the final round of additional communication in an even-numbered node scenario.
[0038] Figure 6 This is a comparative experimental result of the ShiftRing and HalfRing of this invention. Detailed Implementation
[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.
[0040] Work scenario:
[0041] This invention presents ShiftRing, an efficient all-to-all communication method for toroidal topologies. It is applicable to communication environments consisting of multiple computing nodes in a toroidal topology, where each participating node is a CPU server, GPU server, or other device with computing and communication capabilities. One-dimensional and two-dimensional toroidal topology diagrams are shown below. Figure 1As shown. In a one-dimensional toroidal topology, all nodes are connected in numerical order to form a logical ring. Each node has direct communication with only two adjacent nodes, one clockwise and the other counter-clockwise. The network environment can be a physical ring interconnection structure or a logical ring structure abstracted from an actual cluster network. In a two-dimensional toroidal topology, communication based on the one-dimensional toroid is required simultaneously in both dimensions. Specifically: the communication data is generally divided into two parts, and the two-dimensional toroidal topology is also divided into two different dimensions of one-dimensional toroidal topology. At the same time, one part of the data is communicated horizontally, and the other part is communicated vertically. After completion, the data that communicated horizontally is communicated again vertically, and the data that communicated vertically is communicated again horizontally, thus achieving parallel transmission in both dimensions. The two-stage All-to-All communication is as follows: Figure 2 and Figure 3 As shown.
[0042] In a typical training iteration, each node holds a set of data that needs to be distributed to all other nodes. This data can be tokens generated by expert distribution, sparse features in the recommendation model, embedding vectors, or other intermediate tensors that need to be redistributed across nodes. After each node performs all-to-all communication as defined by ShiftRing, each node can obtain data from all other nodes that are targeted at itself, thus providing input for subsequent training computations.
[0043] ShiftRing is particularly suitable for the following scenarios: First, the communication pattern between nodes can be abstracted as a ring; second, all-to-all communication operations occur frequently, and communication latency has a significant impact on training performance; third, the system is sensitive to GPU memory or memory usage, and it is desirable to minimize the additional buffer overhead introduced by intermediate forwarding.
[0044] initialization:
[0045] ShiftRing initialization includes establishing the logical ring topology, determining adjacency relationships, and laying out the data in the local buffer.
[0046] Suppose there are N active nodes, numbered 0, 1, 2, ..., N-1; for any node p, its clockwise adjacent node is defined as... Counterclockwise adjacent nodes are defined as Each node holds a copy of the original local data and divides the original data into N logical sub-data blocks according to the target node. The q-th sub-data block represents the data part that needs to be sent to node q. The part of the data that is the same in the target node and the source node is directly kept locally, and the remaining data blocks participate in subsequent communication.
[0047] During initialization, each node also needs to establish a local receive buffer. Unlike traditional methods, the receive buffer in ShiftRing is not only used to store the final received result, but also serves as the carrying area for subsequent intermediate forwarding. Therefore, during the initialization phase, it is necessary to reserve corresponding slots for each logical sub-data block and establish a unified offset rule to provide a foundation for subsequent in-place forwarding.
[0048] Figure 4 The complete communication process of ShiftRing on a 5-node ring is presented. From Figure 4 It can be seen that ShiftRing has two core features: first, each round of communication utilizes both clockwise and counterclockwise directions simultaneously; second, newly received data is not written back to a fixed static position, but is written to the data slot that has already been sent, so the receive buffer can be continuously reused.
[0049] In practice, ShiftRing first determines the actual transmission direction for each target data block. For a data block sent from node p to node q, the clockwise and counterclockwise distances are calculated separately, and the shorter one is selected as the transmission direction. When the distances on both sides are equal, the data block is split into two parts and sent in clockwise and counterclockwise directions respectively. In this way, all data blocks travel along the shortest path from the start of communication, avoiding the ineffective detours in traditional fixed-direction schemes.
[0050] Subsequently, ShiftRing does not organize each data block as an independent communication, but instead schedules the data blocks in stages according to the shortest path hop distance. Data for farther targets enters the ring in earlier stages, while data for closer targets is gradually incorporated in later stages, increasing the message granularity of each round. With this processing, data already in the ring from the previous stage will continue to advance one hop, while newly added data in the current stage is synchronously injected into the network. The entire process exhibits a bidirectional, parallel, shift-based pipeline communication.
[0051] To more clearly illustrate the execution logic of ShiftRing, the corresponding communication method and algorithm flow are given below.
[0052] When a node performs ShiftRing communication, follow these steps:
[0053] Input: The total number of nodes N in the ring topology, the current node number p, and the set of local data blocks held by the current node.
[0054] Output: The set of result data obtained by the current node from all source nodes and with this node as the target.
[0055] Step 1: Read the total number of nodes N and calculate the number of main communication stages K; for a ring topology containing N nodes, the number of main communication stages is taken as... .
[0056] Step 2: On the current node p, divide the local raw data into N logical sub-data blocks according to the target node number, and write these data blocks into the corresponding slots of the local buffer.
[0057] Step 3: Starting from stage 0, execute sequentially up to stage K-1.
[0058] For any stage k, the current node calculates the logical starting offset positions in the clockwise and counterclockwise directions for that stage, respectively: , This is used to determine the range of data blocks that will be transmitted first in this round.
[0059] Step 4: Based on the stage number k, construct a clockwise transmission index list for the current node p. and counterclockwise transmission index list ;
[0060] The larger the stage k, the larger the index range included, and the more data blocks participate in the transmission in this round;
[0061] List and list Each contains k+1 indexes; The i-th index value is , The i-th index value is ,in ;
[0062] Therefore, communication will gradually cover all target data blocks in the order of "far first, then near".
[0063] Step 5: The current node sends the data block corresponding to the clockwise index list to the right neighbor node through the clockwise link, and receives the data continuing to advance clockwise from the left neighbor node; at the same time, it sends the data block corresponding to the counterclockwise index list to the left neighbor node through the counterclockwise link, and receives the data continuing to advance counterclockwise from the right neighbor node.
[0064] Single-hop transmission and reception in both directions are completed in parallel within the same phase.
[0065] Step 6: For new data received in this stage, instead of writing it back to the original fixed location, write it into the slot in the local buffer where the transmission task has been completed, according to the stage number and transmission direction; in this way, the new data can be retained as a result or forwarded as intermediate data in subsequent stages.
[0066] Step 7: If the total number of nodes N is odd, then after the main communication phase ends, all data blocks have completed the transmission on the shortest path, and the current node directly enters the result recovery phase.
[0067] If the total number of nodes N is even, there is another type of data block where the target node is located on the opposite side, and the clockwise and counterclockwise path lengths are both N / 2. For this type of data, an additional round of communication is added after the main communication phase to split the data and transmit it in both directions.
[0068] Figure 5 This paper demonstrates the execution method of the last round of additional communication in the even-number node scenario. In the communication of four nodes, each node will have a piece of data with the same clockwise and counterclockwise distance. In order to make full use of the bandwidth, this invention divides the data into two parts and performs bidirectional communication, which is the traditional bidirectional ring algorithm.
[0069] Step 8: After all communication is complete, treat the local buffer as a circular array, starting from the predetermined logical offset position. .
[0070] By starting to read consecutive data items sequentially, the correct order of all-in-one communication results can be recovered.
[0071] ShiftRing's in-place forwarding mechanism:
[0072] The key to ShiftRing is not just shortest path transmission, but also how intermediate nodes can complete forwarding without adding extra buffers. Figure 3 In the ShiftRing method shown, the position where the received data is written is not fixed, but changes dynamically as the stage progresses. Only when the original content in a certain slot has been sent will the slot be reused by subsequent new data.
[0073] Therefore, in ShiftRing, the receive buffer serves two roles: storing the received target result and, if necessary, storing intermediate data that needs to be forwarded for the next hop. Once incoming data enters the receive buffer, if it has not yet reached the final target node, it can be sent directly from that slot in the next stage without being copied to a separate forwarding buffer. Thus, ShiftRing achieves in-place forwarding while maintaining the store-and-forward logic.
[0074] ShiftRing results restored:
[0075] After all communication phases are complete, the data in each node's local buffer may not be physically arranged consecutively according to the source or destination node numbers, but its logical layout is determined. ShiftRing treats the buffer as a circular array and defines a fixed logical starting offset for each node. After communication is complete, simply reading consecutive data items sequentially starting from that offset position will restore the correctly ordered full-to-full communication result, without needing to perform any additional data reordering operations.
[0076] Therefore, ShiftRing does not leave complexity to be handled after communication ends, but rather arranges the final result layout synchronously during the communication scheduling and write rule design phases. This feature further reduces tail processing overhead, making the overall communication process more compact.
[0077] Experimental verification:
[0078] To verify the effectiveness of ShiftRing, a prototype of the algorithm was implemented in a PyTorch distributed environment and compared with two baseline methods: the bidirectional ring algorithm (BidRing) and the half-ring algorithm (HalfRing). The bidirectional ring algorithm is a very mature algorithm in industrial applications; while the half-ring algorithm is theoretically the most advanced algorithm.
[0079] Multiple DCU accelerator cards within the same server are directly interconnected via a PCIe 4.0 bus. However, in our experiments, we constructed eight DCUs into a logical one-dimensional ring topology on a single server. Each DCU corresponds to a communication rank, and it was stipulated that DCUs can only send and receive data with adjacent DCUs on the ring (i.e., adjacent rank values). Each scheme implemented its all-to-all communication logic based on a point-to-point communication interface. For different message sizes, 100 all-to-all operations were executed consecutively, and the average completion time was calculated to reduce the impact of system noise.
[0080] from Figure 6 It can be seen that when the message size is small, the communication startup overhead accounts for a relatively high proportion of the total latency. ShiftRing, due to its staged aggregation scheduling, can significantly reduce the number of startups, thus exhibiting the most obvious performance advantage. As the message size increases, data transmission time gradually becomes the dominant factor, and the gap between ShiftRing and HalfRing narrows, but still maintains a stable advantage. Compared to the traditional BidRing, ShiftRing achieves a more significant overall performance improvement by simultaneously reducing both forwarding hops and startup overhead.
[0081] Furthermore, because ShiftRing employs an in-place forwarding mechanism based on the receive buffer, it does not require the allocation of an additional forwarding buffer as in traditional solutions. Therefore, while achieving performance improvements, it also reduces additional GPU memory usage. This demonstrates that ShiftRing not only offers better communication efficiency but is also more suitable for deployment in memory-sensitive deep learning training environments.
[0082] This invention proposes ShiftRing, an efficient all-to-all communication method for ring topologies. ShiftRing is based on all-to-all set communication, maximizing the reuse of adjacent link relationships in the ring topology while jointly designing data transmission paths, stage scheduling methods, and buffer writing methods to reduce the overall overhead of all-to-all communication in ring network environments.
[0083] Unlike existing solutions, ShiftRing does not require all data to be transmitted hop-by-hop in a fixed direction. Instead, it first selects the shorter transmission direction for each data block based on the relative positions of the source and target nodes on the ring, prioritizing data transmission along the shortest path. This directly reduces unnecessary hops and lowers communication costs. Furthermore, ShiftRing does not treat each data block as an independent transmission. Instead, it organizes data blocks from different targets into multiple stages based on transmission distance, prioritizing data from more distant targets and gradually incorporating data from closer targets in subsequent stages. This progressively increasing message granularity reduces the overall number of communication initiations.
[0084] Besides path selection and stage scheduling, another core improvement of ShiftRing lies in its in-place forwarding mechanism based on the receive buffer. In traditional schemes, because received data may overwrite data that has not yet been sent, an additional forwarding buffer is needed to store intermediate results. ShiftRing uses dynamic offset write rules to write the data received in each round into the slots that have already completed their transmission tasks, thus enabling the receive buffer to simultaneously perform the functions of result storage and intermediate forwarding. With this mechanism, ShiftRing can complete multiple rounds of store-and-forward without adding extra buffer overhead.
Claims
1. An efficient all-to-all communication method for toroidal topology, characterized in that, Specifically: Applicable Scenarios: This invention, ShiftRing, is an efficient all-to-all communication method for toroidal topologies. It is suitable for communication environments consisting of multiple computing nodes in a toroidal topology. Each participating node can be a CPU server, GPU server, or other device with computing and communication capabilities. If it is a one-dimensional toroidal topology, all nodes are connected in numerical order to form a logical ring. Each node has direct communication with only two adjacent nodes, one clockwise and the other counterclockwise. The network environment can be a physical ring interconnection structure or a logical ring structure abstracted from an actual cluster network. If it is a multi-dimensional toroidal topology, communication based on a one-dimensional toroid is required in each dimension simultaneously. Initialization: ShiftRing initialization includes establishing the logical ring topology, determining adjacency relationships, and laying out the data in the local buffer; Assume there are N active nodes, numbered 0, 1, 2, ..., N-1; For any node p, its clockwise adjacent nodes are numbered as follows: The counterclockwise adjacent nodes are numbered as Each node holds a copy of the original local data and divides the original data into N logical sub-data blocks according to its target node. The q-th sub-data block represents the data part that needs to be sent to node q. The part of the data that is the same in the target node and the source node is directly kept locally, and the remaining data blocks participate in subsequent communication. When a node performs ShiftRing communication, all nodes execute it synchronously. The node numbered p follows these steps: Input: The total number of nodes N in the ring topology, the current node number p, and the set of local data blocks held by the current node; Output: The set of result data obtained by the current node from all source nodes and with this node as the target; Step 1: Read the total number of nodes N and calculate the number of main communication stages K; for a ring topology containing N nodes, the number of main communication stages is taken as... ; Step 2: On the current node p, divide the local raw data into N logical sub-data blocks according to the target node number, and write these data blocks into the corresponding slots of the local buffer; Step 3: Starting from stage 0, execute sequentially up to stage K-1; For any stage k, the current node p calculates the logical starting offset positions in the clockwise and counterclockwise directions for that stage, respectively: , This is used to determine the range of data blocks that will be transmitted first in this round; Step 4: Based on the stage number k, construct a clockwise transmission index list for the current node p. and counterclockwise transmission index list ; The larger the stage k, the larger the index range included, and the more data blocks participate in the transmission in this round; List and list Each contains k+1 indexes; The i-th index value is , The i-th index value is ,in ; Therefore, communication will gradually cover all target data blocks in the order of "far first, then near". Step 5: The current node sends the data block corresponding to the clockwise index list to the right neighbor node through the clockwise link, and receives the data continuing to advance clockwise from the left neighbor node; at the same time, it sends the data block corresponding to the counterclockwise index list to the left neighbor node through the counterclockwise link, and receives the data continuing to advance counterclockwise from the right neighbor node. Single-hop transmission and reception in both directions are completed in parallel within the same phase; Step 6: For new data received in this stage, instead of writing it back to the original fixed location, write it into the slot in the local buffer where the transmission task has been completed, according to the stage number and transmission direction; in this way, the new data can be retained as a result or forwarded as intermediate data in subsequent stages. Step 7: If the total number of nodes N is odd, then after the main communication phase ends, all data blocks have completed the transmission on the shortest path, and the current node directly enters the result recovery phase; If the total number of nodes N is even, there is another type of data block where the target node is located on the opposite side, and the clockwise and counterclockwise path lengths are both N / 2. For this type of data, after the main communication phase is completed, an additional round of communication is added to split the data and transmit it in both directions. Step 8: After all communication is complete, treat the local buffer as a circular array, starting from the predetermined logical offset position. By starting to read consecutive data items sequentially, the correct order of all-in-one communication results can be recovered.
2. The efficient all-to-all communication method for toroidal topology according to claim 1, characterized in that, If the application scenario is a two-dimensional toroidal topology, the communication data is generally divided into two parts, and the two-dimensional toroidal topology is also divided into two one-dimensional toroidal topologies of different dimensions. At the same time, a part of the data is communicated in the horizontal direction, and the other part of the data is communicated in the vertical direction. After that, the data that was communicated in the horizontal direction is communicated again in the vertical direction, and the data that was communicated in the vertical direction is communicated again in the horizontal direction, so as to transmit in parallel in two dimensions. When the application scenario is an M-dimensional torus topology, the communication data is divided into M parts, and the M-dimensional torus topology is also divided into M one-dimensional torus topologies of different dimensions. At the same time, the i-th part of the data communicates on the i-th one-dimensional torus topology, where i=1,2,…,M. After that, the data that has been communicated in the i-th dimension communicates again in the remaining M-1 dimensions. After M rounds of communication, all data is transmitted in parallel in all dimensions, realizing full-to-full communication.
3. The efficient all-to-all communication method for toroidal topology according to claim 1, characterized in that, During initialization, each node also establishes a local receive buffer. The receive buffer in ShiftRing is not only used to store the final received result, but also serves as the carrying area for subsequent intermediate forwarding. Therefore, during the initialization phase, it is necessary to reserve corresponding slots for each logical sub-data block and establish a unified offset rule to provide a basis for subsequent in-place forwarding.
4. The efficient all-to-all communication method for toroidal topology according to claim 1, characterized in that, ShiftRing treats the buffer as a circular array and defines a fixed logical starting offset for each node. After communication is complete, you can recover the correct order of all-to-all communication results simply by reading consecutive data items in sequence starting from that offset, without needing to perform any additional data reordering operations.