Full reduction communication method based on extensible multi-dimensional full-connection network topology

Abstract

The invention discloses a full reduction communication method based on expandable multi-dimensional full-connection network topology, and belongs to the field of full reduction communication, and the method comprises the steps: constructing the multi-dimensional full-connection network topology; equally dividing the to-be-stipulated data of each computing node according to the number and dimensions of the computing nodes to obtain data blocks; numbering the data blocks of each computing node to obtain a data block group of each dimension of each computing node; exchanging partial data blocks of the data block groups with different dimensions by each computing node and an adjacent node, and performing specification operation on the data blocks with the same dimensions as the local computing node and the target computing node; each computing node exchanges the data blocks of the complete protocol result of the data block groups of different dimensions with the adjacent nodes; all computing nodes obtain all complete protocol data blocks; and splicing the data blocks of the computing nodes into complete data to complete the full-protocol set communication operation. According to the method, the problems of uncontrollable cost, high link load and low hardware utilization rate of the existing method are solved.

Description

Technical Field

[0001] The present invention belongs to the field of all-reduce communication, and particularly relates to an all-reduce communication method based on an extensible multi-dimensional fully connected network topology. Background Art

[0002] A fully interconnected network is a form of network connection, that is, a form in which all nodes are directly connected. Network devices are organized in a mesh topology, and each network node has either a physical circuit or a virtual circuit connected to all other network nodes. The full interconnection provides a large amount of redundancy and has the characteristic of link symmetry, which has received great attention in the field of distributed computing. However, due to its high cost and poor scalability, it is usually used on the backbone, and its application scenarios are limited. Therefore, designing a network topology that has the same high redundancy and symmetry as the fully interconnected network but with controllable costs and scalability is crucial for improving the performance of distributed computing.

[0003] With the development of distributed computing and large-scale deep learning models, the collective communication (COLL) algorithm has played a crucial role in the fields of high-performance computing (HPC) and artificial intelligence (AI). Among them, all-reduce is a key collective communication operation, which is widely used in scenarios such as synchronous update of model parameters and gradient aggregation. In the fields of parallel computing and large model training, all-reduce communication can account for 40% of the total training time of the model when performing distributed gradient aggregation for training deep learning models. Traditional all-reduce algorithms rely on the optimization of network topologies to reduce communication latency and improve bandwidth utilization. A suitable collective communication algorithm can significantly improve the cluster communication efficiency and save the cost of executing tasks. Most of the existing collective communication algorithms are mainly applicable to network topologies such as fat tree topologies, tree topologies, and ring topologies, and will encounter situations such as low hardware utilization and serious performance loss when implemented in multi-dimensional fully connected network topologies. Therefore, how to design an all-reduce algorithm that is fully adapted to multi-dimensional fully connected networks and achieve performance comparable to other collective communication algorithms to fully utilize the network advantages is a very challenging problem.

[0004] The α-β cost model is used to estimate the costs of collective communication algorithms in terms of latency and bandwidth usage, and can guide the algorithm selection for specific collective operations. It divides the communication cost into two parts: startup cost and communication cost. The startup cost is fixed and represents the latency of communication, including the overhead of initiating transmission, link latency, etc.; the communication cost is related to the link bandwidth and is represented by α and β respectively. The time taken to send a message between any two nodes can be modeled as α + sβ, where s is the size of the data transmitted. The performance of the algorithm can be evaluated using this model based on the number of communication steps and the communication volume required for the algorithm to complete. Another algorithm performance metric is the total communication volume, which is the total amount of data transmitted by all nodes in the network to complete the communication task when the algorithm is executed.

[0005] An existing technical solution is the Recursive Doubling (RD) algorithm: for a network with P nodes, the all-reduce operation is performed in two stages: a Reduce-Scatter communication operation followed by an All-Gather communication operation. The Reduce-Scatter stage uses the recursive halving algorithm, and the All-Gather stage uses the recursive doubling algorithm. Both algorithms perform log2P steps of communication operations. Assuming that the data size of each node is 1, each node divides the data into p data blocks {b0, b1,..., b P-1}, and the size of each block is 1 / P. In each step of the Reduce-Scatter stage, the size of the transmitted data is halved, and the distance between the communicating nodes is doubled. Therefore, each node transmits (P - 1) / P of the data in the Reduce-Scatter stage. The working principle of the All-Gather stage is similar, but the communication pattern is opposite. In each step, the size of the transmitted data is doubled, and the distance between the communicating nodes is halved.

[0006] An example of a recursive doubling algorithm is as Figure 1, there are P = 4 nodes in the network. On the left is the object selection for each node's communication step, and on the right is the result at the end of each communication step. The reduction - scattering phase starts from step 1. Each node exchanges data with nodes at a distance of 1: P0 communicates with P1 in the same row, and P2 communicates with P3. Each node sends half of the data required by all nodes, receives the other half of the data required by all nodes, and performs a reduction operation (addition operation in the legend) on the received data. For example, P0 gets the first - half data "12" and "2" from P1 and adds them to its corresponding - position data to get data "19" and "7"; In step 2, each node exchanges data with nodes at a distance of 2. This process continues recursively, and the transmitted data is further halved. Each node sends half of the data after reduction in the previous step, receives the other half, and performs a reduction operation on the received data. This phase has a total of log2p = 2 communication steps, and each node transmits (P - 1)n / P = 3n / 4 of the data. The all - gather phase starts from step 3. Nodes 2 apart exchange their data and splice it. For example, P0 receives the data "21" from P2 and splices it behind its own data "37"; In step 4, nodes 1 apart exchange their own data and the data they received in the previous step. The all - gather phase requires a total of log2P = 2 steps, and each node transmits (P - 1) / P = 3 / 4 of the data.

[0007] Therefore, the recursive doubling algorithm needs to execute 2log2P communication steps to complete the all - reduction communication operation. Each node transmits 2(P - 1) / P of the data, and the total communication volume of the algorithm is 2(P - 1). According to the α - β cost model, the algorithm time is 2log2Pα+(2(P - 1) / P)β.

[0008] The disadvantages of the recursive doubling algorithm are as follows:

[0009] ① It does not consider the actual network topology. If communication between two nodes requires a relay node, there will be a problem of link contention. The more communication hops there are, the more traffic shares the same link, the more congested the link is, and the queuing delay causes the actual time spent by the algorithm to increase significantly, resulting in performance loss;

[0010] ② It is only applicable to the case where the number of network nodes is a power of 2. For non - power - of - 2 node numbers, additional adjustments and communications are required. In a multi - dimensional fully - connected network, there are more cases than just power - of - 2 node numbers, and the recursive doubling algorithm may not be applicable.

[0011] ③ It designs communication based on node numbers, considering the single - port case. For a network with multi - port nodes, the hardware utilization rate is low.

[0012] Another existing technical solution is the ring all-reduce algorithm: The all-reduce is also executed as a reduction-scatter phase and an all-gather phase. For a single-port network with P nodes (only one send and one receive can occur at a time), each node divides its data of size n into P equally sized data blocks. This algorithm performs P-1 steps in each phase. The nodes are arranged in a ring. In the reduction-scatter phase, each node makes P-1 iterations. In each iteration: The node sends a piece of data to its left adjacent node, and at the same time receives a piece of data from its right adjacent node and performs a reduction with the corresponding local data. The data sent and received by each node is different in each iteration. The p-th node initially sends the data at position p and receives the data at position (p+1)%P. Then it continuously processes forward. In each iteration, it sends out the data at the position received in the previous iteration. Finally, each node gets the reduction result of the data at one position; in the all-gather phase, each node does not perform a reduction of the received data with the corresponding local data, but directly overwrites the data at that position. The p-th node initially sends the data at position (p-1)%P and receives the data at position p. In subsequent iterations, it always sends out the data at the position received in the previous iteration until all nodes get the complete reduction result.

[0013] Such as Figure 2 , in a ring network connected by 4-node directed links, each node can only send data to the left node and receive messages from the right neighbor. The initial data of each node is divided into 4 parts as in Example 1. The reduction-scatter phase starts from the first step and is completed in a total of P = 3 steps. Each node receives the data from its right neighbor and adds it to the data at the corresponding position. For example, in the first step, P0 receives the data "4" at position 1 from P1 and adds it to its own data "3" at position 1, getting "7"; in the second step, it sends it to P3 and receives the data "20" at position 2 from P1, adding it to its own data "9" at position 2 to get "29". Until the end of the third step, P0 gets the sum "22" of the initial data at position 4 of all nodes, P1 gets the sum "37" of the initial data at position 0 of all nodes, P2 gets the sum "21" of the initial data at position 1 of all nodes, and P3 gets the sum "38" of the initial data at position 2 of all nodes. Each node transmits (P-1) / P = 3 / 4 of the data. The all-gather phase starts from the third step. Each node sends its newly obtained result to its left neighbor. For example, in the first step, P0 sends the data "22" at position 4 to P3, receives the data "37" at position 0 from P1 and overwrites it on its own position 0. In the second step, it sends the newly obtained data at position 0 in the first step to P3. Each node continues like this until the end of the third step. Each node gets the data at all positions. Each node also transmits data of size 3 / 4 in 3 steps of communication.

[0014] Therefore, the ring algorithm needs to execute 2(P - 1) steps. Each node transmits 2(P - 1) / P of data. The total communication volume of the algorithm is 2(P - 1). According to the α-β cost model, the algorithm time is 2(P - 1)α + ((P - 1) / P)β.

[0015] The disadvantages of the ring algorithm are as follows:

[0016] ① The number of communication steps grows linearly with the number of nodes. It does not have performance advantages in networks with a large number of nodes. Moreover, directly using ring communication in a multi-dimensional fully connected network topology will cause competition between tasks for the links between adjacent groups, resulting in link congestion and performance loss.

[0017] ② In a non-ring network, it is necessary to find a loop that contains all the nodes executing tasks to be applicable. In addition, the ring algorithm is not applicable to multi-port scenarios, so it is difficult to apply to multi-dimensional fully connected networks.

[0018] ③ In the case of a single-port network, for a network with multi-port nodes, the hardware utilization rate is low. Summary of the Invention

[0019] In view of the above deficiencies in the prior art, a fully reduced communication method based on an expandable multi-dimensional fully connected network topology provided by the present invention solves the problems of uncontrollable cost, high link load, and low hardware utilization rate of the existing methods.

[0020] In order to achieve the above invention purpose, the technical solution adopted by the present invention is: A fully reduced communication method based on an expandable multi-dimensional fully connected network topology, including:

[0021] According to the number of computing nodes P, determine the base number n and the dimension d, and construct a multi-dimensional fully connected network topology according to the base number and the dimension.

[0022] Divide the data to be reduced of each computing node equally according to the number of computing nodes to obtain several sub-data; for each sub-data of each computing node, divide it equally according to the dimension to obtain P×d data blocks.

[0023] Number the P×d data blocks of each computing node based on the dimension and the destination computing node, and group them based on the dimension to obtain the data block groups of each dimension of each computing node.

[0024] According to the multi-dimensional fully connected network topology, determine the adjacent computing nodes of each computing node.

[0025] Each computing node exchanges with the adjacent computing nodes of different dimensions respectively Reduce the data blocks with the same number of data blocks as the local dimension and the destination computing node; repeat the data block exchange until there is a data block with a complete reduction result in each dimension's data block group of each computing node; s is the number of data block exchange iteration steps;

[0026] Each computing node exchanges the data blocks of the complete reduction results of the data block groups with the adjacent computing nodes in different dimensions respectively; repeat the exchange of the data blocks of the complete reduction results until all computing nodes obtain all the complete reduction data blocks;

[0027] Stitch the data blocks of each computing node into complete data to complete the all-reduce collective communication operation.

[0028] Furthermore, the number of computing nodes satisfies the following relationship with the base and dimension:

[0029] P = n d

[0030] n ≥ 2, n ∈ Z +

[0031] d ≥ 1, d ∈ Z +

[0032] where P is the number of computing nodes; Z + is the set of positive integers.

[0033] Furthermore, the coordinate numbers of each computing node in the multi-dimensional fully connected network topology are:

[0034] (x d-1 x d-2 …x i x i-1 …x1x0)

[0035] 0 ≤ x i ≤ n - 1

[0036] 0 ≤ i ≤ d - 1

[0037] where x i is the value of the i-th bit in the coordinate number, which is an n-based number; n is the base of the multi-dimensional fully connected network topology; d is the dimension of the multi-dimensional fully connected network topology; i is the dimension.

[0038] Furthermore, when there is a communication link connection between two computing nodes, the Hamming distance of the coordinate numbers of the two computing nodes is 1.

[0039] Furthermore, the numbers of the data blocks of each computing node are:

[0040] Data dim,dst

[0041] Among them, Data is the source data; dim is the dimension to which the data block belongs; dst is the coordinate number of the destination computing node of the data block.

[0042] Furthermore, for each data block of each computing node, when the dimension and the coordinate number of the destination computing node are the same, the positions of the data blocks in the corresponding data block group are the same.

[0043] Furthermore, each of the computing nodes exchanges data blocks with adjacent computing nodes in different dimensions, specifically:

[0044] In the s-th step of data block exchange, each computing node communicates with adjacent nodes in different dimensions to determine the data block groups to be exchanged, and selects data blocks for exchange; the number of iterative steps of the data block exchange satisfies: 1 ≤ s ≤ d.

[0045] Furthermore, the dimension m of the current computing node and the adjacent computing node is the dimension value i where the numerical values in the coordinate numbers of the adjacent computing node and the current computing node are different.

[0046] Furthermore, the dimension of the data block group to be exchanged is (m + s - 1) % d; the m-th element of the coordinate number of the destination computing node of the data blocks to be exchanged is the same as the m-th element of the adjacent computing node; when s > 1, the data blocks to be exchanged are the data blocks reduced in the previous data block exchange.

[0047] Furthermore, each of the computing nodes exchanges the data blocks of the complete reduction results of the data block groups with adjacent computing nodes in different dimensions, specifically:

[0048] In the c-th step of the exchange of the data blocks of the complete reduction results, each computing node communicates with the corresponding computing node in dimension m and exchanges all the data blocks of the complete reduction results in the data block group with dimension (m - c) % d; c is the number of steps of the exchange of the data blocks of the complete reduction results, satisfying 1 ≤ c ≤ d.

[0049] ​The beneficial effects of the present invention are as follows: The multi-dimensional fully connected network topology provided by the present invention can design and build different networks according to two parameters. It can clearly and simply determine whether there is a link connection between nodes according to the node numbers, with lower costs, and can design a network topology with a suitable dimension according to the needs of communication tasks. At the same time, the multi-dimensional fully connected network can also be expanded. The full reduction communication algorithm provided by the present invention can also be used in the multi-dimensional fully connected network topology of powers of 2, making use of all the links in the multi-dimensional fully connected network topology. There is data transmission on each link in each communication step, with a higher hardware utilization rate. Moreover, each node in the full reduction algorithm only communicates with its adjacent nodes, and there is no link contention problem; it realizes the optimal link load, the minimum total communication volume, and fewer communication steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 FIG. is an execution schematic diagram of a recursive doubling algorithm in the related art of the present invention.

[0051] Figure 2 FIG. is an execution schematic diagram of a ring algorithm in the related art of the present invention.

[0052] Figure 3 FIG. is a flowchart of the method of the present invention.

[0053] Figure 4 FIG. is a schematic diagram of the multi-dimensional fully connected network topology in the first aspect of the embodiment of the present invention;

[0054] Figure 5 FIG. is a schematic diagram of data segmentation in the embodiment of the present invention.

[0055] Figure 6 FIG. is a schematic diagram of the first step of the reduction - scatter communication in the embodiment of the present invention.

[0056] Figure 7 FIG. is a schematic diagram of the second step of the reduction - scatter communication in the embodiment of the present invention.

[0057] Figure 8 FIG. is a schematic diagram of the first step of the all - gather communication in the embodiment of the present invention.

[0058] Figure 9 FIG. is a schematic diagram of the second step of the all - gather communication in the embodiment of the present invention.

[0059] Figure 10 FIG. is a schematic diagram of data adjustment in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] The specific embodiments of the present invention will be described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions made using the concept of the present invention are within the scope of protection.

[0061] As Figure 3 shown, in an embodiment of the present invention, a full reduction communication method based on an expandable multi-dimensional fully connected network topology includes:

[0062] Determine the base number n and the dimension d according to the number of computing nodes P, and construct a multi-dimensional fully connected network topology according to the base number and the dimension;

[0063] Divide the data to be reduced of each computing node equally according to the number of computing nodes to obtain a number of sub-data; for each sub-data of each computing node, divide it equally according to the dimension to obtain P×d data blocks;

[0064] Number the P×d data blocks of each computing node based on the dimension and the destination computing node, and group them based on the dimension to obtain the data block groups of each dimension of each computing node;

[0065] Determine the adjacent computing nodes of each computing node according to the multi-dimensional fully connected network topology;

[0066] Each computing node exchanges data blocks with the adjacent computing nodes of different dimensions, and performs a reduction operation on the data blocks with the same dimension and destination computing node as the local; repeat the data block exchange until there is a data block with a complete reduction result in each data block group of each dimension of each computing node; s is the number of iterations of the data block exchange;

[0067] Each computing node exchanges the data blocks with the complete reduction results of the data block groups of the adjacent computing nodes of different dimensions; repeat the exchange of the data blocks with the complete reduction results until all computing nodes obtain all the complete reduction data blocks;

[0068] Stitch the data blocks of each computing node into complete data to complete the full reduction collective communication operation.

[0069] The number of computing nodes satisfies the following relationship with the base number and the dimension:

[0070] P = n d

[0071] n≥2, n∈Z +

[0072] d≥1, d∈Z +

[0073] where P is the number of computing nodes; Z + is a set of positive integers.

[0074] The coordinate numbers of the computing nodes in the multi-dimensional fully connected network topology are:

[0075] (x d-1 x d-2 … x i x i-1 … x1x0)

[0076] 0 ≤ x i ≤ n - 1

[0077] 0 ≤ i ≤ d - 1

[0078] where x i is the value of the i-th bit in the coordinate number, which is an n-ary number; n is the base of the multi-dimensional fully connected network topology; d is the dimension of the multi-dimensional fully connected network topology; i is the dimension.

[0079] When there is a communication link connection between two computing nodes, the Hamming distance between the coordinate numbers of the two computing nodes is 1.

[0080] In this embodiment, according to the first aspect of the present invention, a multi-dimensional fully connected network topology with controllable cost and expandable ability is provided. It includes multiple nodes, and the multiple nodes are connected by communication links to form a multi-dimensional topological structure. The multi-dimensional fully connected network topology structure is determined by two parameters, the base n and the dimension d, where n is an integer greater than or equal to 2, and d is an integer greater than or equal to 1. In one dimension of the network topology, n nodes are pairwise connected, and the nodes at corresponding positions in different dimensions are pairwise connected. The total number of nodes in the network topology is P = n d . Each node in the network topology is uniquely represented by a d-bit n-ary coordinate number, in the form of (x d-1 x d-2 … x i x i-1 … x1x0), where 0 ≤ x i ≤ n - 1, 0 ≤ i ≤ d - 1. If the Hamming distance between the coordinates of two nodes is 1, then there is a link connection between the two nodes. The nodes are computing nodes with computing capabilities, such as devices based on a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU). The computing nodes are connected through switches or optical fiber connections, etc., for data transmission.

[0081] In this embodiment, the multi-dimensional fully connected network determines the total number of computing nodes according to the radix n and the dimension d. In practical applications, a reasonable radix and dimension are designed according to requirements (such as communication delay, bandwidth requirements, etc.), and the link connections are configured accordingly to build the network. Increasing the dimension can expand the network scale. If the network topology has been determined, it needs to be designed according to the physical location or logical relationship of the nodes. For example, in a data center, if the computing nodes are distributed in different cabinets, the cabinets can be regarded as the basis for dimension division, and the nodes in each cabinet are regarded as part of the radix. Suppose there are 4 cabinets in the data center, and each cabinet has 4 computing nodes. Through network wiring and switch configuration, it can be constructed into a multi-dimensional fully connected network with (n = 4) (the connection relationship of the nodes within each cabinet) and (d = 2) (the inside and between cabinets are regarded as two dimensions).

[0082] The numbers of the data blocks of each computing node are as follows:

[0083] Data dim,dst

[0084] Among them, Data is the source data; dim is the dimension to which the data block belongs; dst is the coordinate number of the destination computing node of the data block.

[0085] In this embodiment, step 1: According to the parameters n and d of the multi-dimensional fully connected network topology, it is determined that the number of nodes participating in the all-reduce communication is n d , and it is determined that the number of communication steps required for the algorithm to perform the all-reduce communication is 2d. The data that each node in the network topology needs to perform all-reduce is first evenly divided into n d parts, corresponding to the number of nodes in the network topology. Then, each of these n d parts of data is evenly divided into d data blocks, corresponding to the dimension parameters of the network topology. Thus, the data that each node in the network topology needs to perform all-reduce is evenly divided into n d ×d data blocks, and all data blocks are grouped and numbered according to the dimension and the destination node. The number contains information such as the source node, the dimension group to which the data block belongs, and the destination node coordinates. In subsequent communication steps, each node determines the sending strategy of the data block according to the data block number information and the current communication step number.

[0086] For each data block of each computing node, when the dimension and the coordinate number of the destination computing node are the same, the positions of the data blocks in the corresponding data block group are the same.

[0087] Each computing node exchanges data blocks with adjacent computing nodes in different dimensions respectively. Specifically:

[0088] In the s-th step of data block exchange, each computing node communicates with adjacent nodes in different dimensions respectively, determines the data block group to be exchanged, and selects Exchange the data blocks; the number of iterative steps of the data block exchange satisfies: 1 ≤ s ≤ d.

[0089] In this embodiment, step 2: All nodes perform reduction-scatter communication operations. Each node communicates only with its adjacent nodes. The data blocks grouped by dimension are first sent to the adjacent nodes corresponding to the dimension of the node according to the strategy. Each link transmits 1 / n of a group of data blocks. After each node receives the data blocks in each step, it performs a reduction operation on the data blocks whose dimension group and destination node information in the local data block number are the same. The reduction operations include binary operations such as summation and averaging. In each subsequent communication step, each node will communicate with the adjacent nodes in each dimension to exchange data blocks of different dimension groups until each node has exchanged all dimension groups of data blocks with the adjacent nodes in each dimension. The communication strategy is that each node communicates with the adjacent node in dimension m in the s-th step of the reduction-scatter communication operation, and sends 1 / n of the data blocks reduced in the previous communication step of dimension (m + s - 1) % d. That is to say, the number of data blocks transmitted by each link in each communication step decreases by 1 / n, and each group of data blocks is only allocated to communicate in one dimension in each step until there is a data block with a complete reduction result in each group of data of all nodes.

[0090] The dimension m of the current computing node and the adjacent computing node is the dimension value i where the numerical values in the coordinate numbers of the adjacent computing node and the current computing node are different.

[0091] The dimension of the data block group to be exchanged is (m + s - 1) % d; the The m-th element of the coordinate number of the destination computing node of the data blocks to be exchanged is the same as the m-th element of the adjacent computing node; when s > 1, the data blocks to be exchanged are the data blocks reduced in the previous data block exchange.

[0092] Each of the computing nodes exchanges the data blocks with the complete reduction results of the data block groups of the adjacent computing nodes in different dimensions, specifically:

[0093] In the exchange of the data blocks with the complete reduction results in the c-th step, each computing node communicates with the corresponding computing node in dimension m to exchange all the data blocks with the complete reduction results in the data block group of dimension (m - c) % d; c is the number of exchange steps of the data blocks with the complete reduction results, satisfying 1 ≤ c ≤ d.

[0094] In this embodiment, step 3: all nodes perform a full collection communication operation on the data blocks of the complete protocol obtained in step 2. Each node only communicates with its adjacent nodes, and the communication method is opposite to step 2. In each communication step, the communication strategy is that each node communicates with the adjacent nodes of dimension m in the cth communication step of the full collection communication operation, and exchanges the data blocks of all the complete protocol results of the group of dimension (mc)%d. Each node exchanges n c Each group of data blocks communicates only in one dimension per step. In the entire step, each group of data blocks will traverse each dimension of the node. The received data blocks are concatenated with the local protocol data blocks until all nodes get all the complete protocol data blocks.

[0095] Step 4: Splice the data blocks into complete data according to the original segmentation method. At this point, each node has obtained the reduction result and the full reduction collective communication operation is completed.

[0096] The present invention is based on d The total number of communication steps of the fully reduced communication algorithm for the multi-dimensional fully connected network topology is 2d, and the link load is:

[0097]

[0098] The total communication volume is 2(n d -1), assuming that the data size of each node is 1.

[0099] The multi-dimensional fully connected network topology provided by the present invention can be used to design and build different networks according to the parameters n and d. According to the node number, it can be clearly and simply determined whether there is a link connection between nodes. The total number of links in the network is Compared with the total number of links in a fully interconnected network with the same number of nodes, n d (n d -1) / 2, the number of links is the same only when the dimension is 1. When the dimension is greater than 1, the number of links in the multi-dimensional fully connected network topology is smaller, so the cost is lower, and the network topology with appropriate dimensions can be designed according to the needs of the communication task. At the same time, the multi-dimensional fully connected network can also be expanded, such as Figure 4 The embodiment shows that the number of nodes is doubled by adding one dimension.

[0100] On the other hand, the full reduction communication algorithm provided by the present invention can also be used in a multi-dimensional fully connected network topology of a power of 2, utilizing all links in the multi-dimensional fully connected network topology, and data is transmitted on each link in each step of communication. Compared with the recursive doubling algorithm and the ring algorithm that only use part of the links in each step of communication, the 100% hardware rate is higher; and each node in the full reduction algorithm communicates only with its adjacent nodes, and there is no link contention problem; in addition, in terms of the number of communication steps, the number of communication steps of the recursive doubling algorithm in the multi-dimensional fully connected network topology is 2log2n if it can be used. d , the number of communication steps of the ring algorithm is 2(n d -1), while the number of communication steps of the full protocol algorithm is 2d, which is smaller than the previous two. In terms of link load, the link load of the full protocol algorithm is This is equivalent to evenly distributing the data transmitted by each link in the recursive doubling algorithm and the ring algorithm to all links of each node, so the link load is reduced exponentially. According to the α-β cost model, the algorithm time is Optimal link load, minimum total communication volume, and fewer communication steps are achieved.

[0101] Example 2

[0102] Figure 4 A schematic diagram showing an embodiment of a multi-dimensional fully connected network topology with n=2 is shown. When d=1, there are only 2 1 = 2 nodes, each node is represented by a 1-bit binary number, that is, node 0 and node 1, the Hamming distance between the two numbers is 1, so there is a link connecting them; when d = 2 dimensions, there are 2 2 = 4 nodes, represented by 2-bit binary numbers, namely 00, 01, 10, 11. Two nodes with a Hamming distance of 1 are connected to each other, namely nodes 00 and 01 are connected, nodes 01 and 11 are connected, nodes 10 and 11 are connected, and nodes 10 and 00 are connected. The expansion of 1 dimension to 2 dimensions is equivalent to copying 2 nodes and the links between them in d = 1 dimension on the basis of 1 dimension. The original nodes 0 and 1 become 00 and 01, and the copied nodes become 10 and 11. Then add the links between nodes 00 and 10, and nodes 01 and 11. When d = 3 dimensions, there are 2 nodes in the network. 3 = 8 nodes, that is, a copy of d = 2-dimensional 4 nodes and their links is made on the basis of 2 dimensions, each node is represented by a 3-bit binary number, and links 000 and 100, 001 and 101, 010 and 110, 011 and 111 are added. Optionally, an n d The total number of links in the multi-dimensional fully connected network topology is By setting different parameters n and d, different multi-dimensional fully connected networks can be obtained.

[0103] Figures 5 - 10 shows an execution schematic diagram of a full reduction algorithm based on a 2D fully connected network topology according to an embodiment of the present invention. 2 2 The 2D fully connected network topology of 2 2 is a network where n = 2 and d = 2 in Figure 4 . There are 4 nodes 00, 01, 10, and 11 in the network. Node 00 is connected to 01, node 01 is connected to 11, node 10 is connected to 11, and node 10 is connected to 00. For the full reduction (summation) communication operation, as Figure 5 shown, each node has a data portion of the same size to be processed. For example, data A of node 00, data B of node 01, data C of node 10, and data D of node 11. After performing the full reduction communication, each node will obtain the result SUM = A + B + C + D of the sum of all node data.

[0104] Furthermore, the full reduction algorithm includes a data splitting operation, a reduction - scatter communication operation, a full collection communication operation, and a data adjustment operation. Performing the data splitting operation in the 2D fully connected network topology specifically means: evenly dividing the data of each node into 8 equal parts, and numbering and grouping the data blocks according to the node number and dimension serial number. As Figure 5 shown, the data A of node 00 is divided into 8 data blocks A 0，00 , …, A 1，11 . Taking data block A 1，11 as an example, in the numbering, "A" represents the source data, that is, the data from which node, and the subscript "1,11" indicates that the data block is assigned to dimension 1 group, and its destination node is node 11, that is, node 11 will obtain the reduction result of this data block. The data of other nodes is processed in the same way for their respective data.

[0105] Furthermore, after the data splitting is completed, as Figure 6 and Figure 7 shown, the reduction - scatter communication operation is performed. As Figure 6 , each node communicates with the adjacent nodes in each dimension, and arranges the data transmission of the corresponding dimension group (indicated by the double - headed arrow). For example, node 00 communicates with node 01. The 0th digit of the two node numbers is different, so they belong to the nodes of dimension 0, and thus arrange the data communication of dimension 0 group. The two nodes evenly divide the data of dimension 0 group into the front and back parts. The 0th digit of node 01's number is 1, so node 00 sends data blocks A 0，01 and A 0，11 in dimension 0 group to node 01; while the 0th digit of node 00's number is 0, node 01 sends data blocks B 0，00 and B 0，10The data is sent to node 00, and other nodes arrange the data transmission of dimension 1 group in the same way. After each node receives the data block, it performs a protocol operation on it and the data block at the corresponding position locally. For example, node 00 receives data block B of node 01's dimension 0 group. 0，00 and B 0，10 Then compare them with the A of their own dimension 0 group 0，00 and A 0，10 Regulations, get A 0，00 +B 0，00 and A 0，10 +B 0，10 , after the first step of the protocol-distributed communication, the data blocks of each node are as follows Figure 6 As shown on the right. Protocol-decentralized communication step 2 is as follows Figure 7 , the data of each dimension group of each node is operated in another dimension. For example, node 00 communicates with node 10 of dimension 1. Node 00 sends data block A of dimension 0 group to node 10 of dimension 1. 0，10 +B 0，10 Data block C of dimension 0 group sent to node 10 and received by node 10 0，00 +D 0，00 , and compare the received data block with data block A 0，00 +B 0，00 The complete reduction result of position "0,00" is obtained through protocol. In addition, node 00 communicates with node 01 and performs protocol reduction to obtain the complete reduction result of position "1,00". After the completion of step 2 of protocol-distributed communication, each node in each dimension group obtains a complete reduction result of the data block.

[0106] Furthermore, if Figure 8 and Figure 9 The communication mode is opposite to the reduce-scatter communication. After the reduce-scatter communication, each node will first send a complete reduction result of each dimension group to the node of another dimension, for example Figure 8 In the first step of the full collection communication, the Sum{0,00} of the dimension 0 group of node 00 will be sent to the node 10 of dimension 1, and the Sum{1,00} of the dimension 1 group will be sent to node 01. The data blocks received by the nodes are concatenated with the data blocks at the corresponding local positions. For example, after node 00 receives the Sum{0,10} of node 10, it is concatenated to the end of Sum{0,00}. Figure 9 In the second step of the full network communication, each node sends all the data blocks of the dimension group spliced ​​in the first step to the node of the corresponding dimension. After receiving the data block, the node also splices it with the data block at the corresponding position locally. At this point, each node has obtained the complete reduction result of all data blocks. Figure 10 As shown, finally, the positions of these data blocks are adjusted according to the initial segmentation method to obtain the complete reduction result SUM.

Claims

1. A full-reduce communication method based on a scalable multi-dimensional fully connected network topology, characterized in that: include: According to the number of computing nodes P, the cardinality n and the dimension d are determined, and a multi-dimensional fully connected network topology is constructed based on the cardinality and the dimension; The data to be reduced of each computing node is equally divided according to the number of computing nodes to obtain a number of sub-data; each sub-data of each computing node is equally divided according to the dimension to obtain P×d data blocks; The P×d data blocks of each computing node are numbered based on the dimension and the destination computing node, and grouped based on the dimension to obtain a data block group of each dimension of each computing node; Determine the adjacent computing nodes of each computing node according to the multi-dimensional fully connected network topology; Each computing node exchanges with adjacent computing nodes of different dimensions data blocks, and perform reduction operations with the data blocks with the same dimensions and destination computing nodes locally; repeat the data block exchange until there is a data block with complete reduction results in the data block group of each dimension of each computing node; s is the number of data block exchange iterations; Each computing node exchanges data blocks of the complete reduction result of the data block group with adjacent computing nodes of different dimensions respectively; the exchange of data blocks of the complete reduction result is repeated until all computing nodes obtain all complete data blocks; The data blocks of each computing node are spliced ​​into complete data to complete the full protocol collective communication operation.

2. The all-reduce communication method based on a scalable multi-dimensional fully connected network topology according to claim 1, characterized in that: The number of computing nodes, cardinality and dimension satisfy: P=n d n≥2,n∈Z + d≥1,d∈Z + Where P is the number of computing nodes; Z + is a set of positive integers.

3. The all-reduce communication method based on a scalable multi-dimensional fully connected network topology according to claim 1, characterized in that: The coordinate number of each computing node in the multi-dimensional fully connected network topology is: (x d-1 x d-2 …x i x i-1 …x1x0) 0≤x i ≤n-1 0≤i≤d-1 Among them, xi is the value of the i-th position in the coordinate number, which is an n-base number; n is the cardinality of the multi-dimensional fully connected network topology; d is the dimension of the multi-dimensional fully connected network topology; i is the dimension.

4. The all-reduce communication method based on a scalable multi-dimensional fully connected network topology according to claim 3, characterized in that: When there is a communication link connection between two computing nodes, the Hamming distance between the coordinate numbers of the two computing nodes is 1.

5. The all-reduce communication method based on a scalable multi-dimensional fully connected network topology according to claim 1, characterized in that: The data blocks of the computing nodes are numbered as follows: Data dim,dst Among them, Data is the source data; dim is the dimension to which the data block belongs; dst is the coordinate number of the destination calculation node of the data block.

6. The all-reduce communication method based on a scalable multi-dimensional fully connected network topology according to claim 5, characterized in that: For each data block of each computing node, when the coordinate number of the dimension and the destination computing node are the same, the position of each data block in the corresponding data block group is the same.

7. The all-reduce communication method based on a scalable multi-dimensional fully connected network topology according to claim 1, characterized in that: Each computing node exchanges with adjacent computing nodes of different dimensions. data blocks, specifically: In the sth step of data block exchange, each computing node communicates with adjacent nodes in different dimensions to determine the data block group to be exchanged and selects data blocks are exchanged; the number of iteration steps of the data block exchange satisfies: 1≤s≤d.

8. The all-reduce communication method based on a scalable multi-dimensional fully connected network topology according to claim 7, characterized in that: The dimension m of the current computing node and the adjacent computing node is the dimension value i whose numerical value is different in the coordinate number of the adjacent computing node and the current computing node.

9. The all-reduce communication method based on a scalable multi-dimensional fully connected network topology according to claim 8, characterized in that: The dimension of the data block group to be exchanged is (m+s-1)%d; The mth element of the coordinate number of the destination computing node of the data block is the same as the mth element of the adjacent computing node; when s>1, the exchange is performed The first data block is the data block that was negotiated in the previous data block exchange.

10. The all-reduce communication method based on a scalable multi-dimensional fully connected network topology according to claim 1, characterized in that: Each computing node exchanges data blocks of the complete reduction results of the data block group with adjacent computing nodes of different dimensions, specifically: In the exchange of data blocks of the complete reduction result in the cth step, each computing node communicates with the corresponding computing node on dimension m to exchange all data blocks of the complete reduction result in the data block group with dimension (mc)%d; c is the number of exchange steps of the data blocks of the complete reduction result, satisfying 1≤c≤d.

Images