A global reduction method based on multiple spanning trees
Patent Information
- Application Number
- CN202311102966.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-30
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2043-08-30
AI Technical Summary
虽然近期已有一些方案沿着这一思路进行了设计，但是这些方案在生成树的构造上存在不足，在面对具有任意拓扑结构的异构网络场景时效果不佳，存在延迟过高，带宽利用率不佳等一系列问题
[0057]本发明通过构造更为均衡的生成树，获得最大时延和最小带宽可控、带宽利用率更高的多生成树，并发的全局归约，有效提高网络资源的利用效率，加快全局归约任务的执行效率。
Smart Images
Figure CN117081944B_ABST
Abstract
Description
Technical Field
[0001] This invention pertains to heterogeneous network communication technology, and particularly relates to a global reduction method based on multiple spanning trees. Background Technology
[0002] In recent years, machine learning has achieved rapid development and widespread application in various fields such as image recognition, speech processing, and machine translation. Due to the massive size of training data or the requirement for local storage of privacy data, a machine learning task may employ a data-parallel design, distributing the training task across multiple worker nodes located in data centers in different regions. These worker nodes are often interconnected via heterogeneous network links, forming a network system that may have arbitrary topologies. During training, to ensure the convergence of the global model, nodes need to periodically perform global reduction (AllReduce) to synchronize their respective training results (such as gradients). Utilizing the heterogeneous network bandwidth between nodes to efficiently achieve global reduction is extremely useful for fully utilizing scarce network bandwidth resources, reducing communication time, and improving the efficiency of distributed machine learning training.
[0003] Depending on the communication mode, global reduction can be implemented in various ways, including ring-based and tree-based schemes. Among these, the tree-based scheme has good versatility and wide application. If all worker nodes and their links are abstracted as an undirected graph, in a tree-structured global reduction scheme, all worker nodes form a spanning tree (called the "AllReduce tree"). A set of data undergoing global reduction is first transmitted from the leaves along the tree to the root node, undergoing hop-by-hop "reduction." Finally, the root node obtains the result of global reduction and then "broadcasts" it hop-by-hop along the tree to all nodes. By breaking down the data to be globally reduced into smaller message units and performing hop-by-hop reduction and broadcasting in a pipelined manner, communication efficiency can be improved. Tree-structured global reduction includes two stages: reduction and broadcasting, as described below. Figure 1 , Figure 2 As shown.
[0004] For heterogeneous networks with arbitrary topologies, using only a single spanning tree for global reduction often fails to fully utilize all links and bandwidth within the network. An improved approach is to construct multiple spanning trees and concurrently perform global reduction on different data, thereby maximizing network resources and improving the efficiency of global reduction. Although some solutions have recently been designed along this line of thought, these solutions have shortcomings in spanning tree construction and perform poorly in heterogeneous network scenarios with arbitrary topologies, exhibiting issues such as high latency and poor bandwidth utilization. Summary of the Invention
[0005] To address the above problems, this invention provides a global reduction method based on multiple spanning trees.
[0006] The present invention provides a global reduction method based on multiple spanning trees, specifically as follows:
[0007] Step 1: Based on the remaining bandwidth in the current network, construct a spanning tree with a height no greater than H and an available bandwidth no less than r. If a spanning tree is successfully found, record its structure, allocate a certain bandwidth b to the tree, and update the remaining bandwidth of each edge in the graph. Then continue to try to find the next spanning tree, repeating this process until no new spanning tree that meets the requirements can be found, at which point the process ends.
[0008] Each time a spanning tree is found, starting from a randomly selected node, a suitable edge is continuously selected to expand the tree until all nodes are included. Each selected edge must satisfy the following condition:
[0009] (1) One end of the edge is connected to a node that has been included in the spanning tree, and the other end is connected to a node that has not yet been included, and the available remaining bandwidth is not less than r.
[0010] (2) After adding this edge, the maximum distance between any nodes in the spanning tree does not exceed 2H.
[0011] If multiple edges satisfy conditions (1) and (2), the edge with the largest remaining bandwidth is selected first. If multiple edges with the same bandwidth satisfy the conditions, one edge can be randomly selected. If no edge that meets the requirements can be found to expand the tree before all nodes are included, it indicates that the construction of the tree has failed and the algorithm ends.
[0012] Step 2: By repeatedly calling the algorithm in Step 1, and under the premise that the reduction in overall throughput is within an allowable range, the actual maximum height of the spanning tree is further reduced by binary search.
[0013] Step 3: After obtaining the list of spanning trees TL and the corresponding rate allocation RL, select no more than K trees from them and determine the rate allocation of each tree. In this way, the global reduction task is allocated proportionally, thereby achieving concurrent global reduction.
[0014] Furthermore, the construction process of the spanning tree in step 1 is as follows:
[0015] Input: Network topology G, maximum height constraint H of spanning tree, lower limit constraint r of communication rate.
[0016] Output: A list of constructed spanning trees TL, where the root node of each tree is given; and the corresponding rate assignment RL.
[0017] Variable description: G is an undirected weighted graph. V(G) and E(G) represent the set of all nodes and the set of edges contained in the graph, respectively. Each edge has two attributes: delay and bandwidth, both of which are quantized values.
[0018] S1. Use list TL to record all constructed spanning trees, initialized to empty; use list RL to record the speed of all constructed spanning trees, initialized to empty.
[0019] S2. Use T to record the current spanning tree being constructed, initialized to empty; use V(T) and E(T) to represent the set of nodes and edges contained therein, respectively.
[0020] S3. Randomly select one node from V(G) and add it to tree T to begin the construction of the tree.
[0021] S4. Find all edges in E(G) with bandwidth greater than r, whose two endpoints belong to V(T) and do not belong to V(T) respectively, and form a list of edges EC.
[0022] S5. Use ei to record which edge will be used to expand the spanning tree this time, and initialize it to an invalid value; use di to record the maximum distance between any two nodes in the tree after adding edge ei and the corresponding node to T, and initialize it to 2H.
[0023] S6. If EC is empty, it means that the edge expansion is over and proceed to S11; otherwise, proceed to the next step.
[0024] S7. Pop the edge ej = (u, v) with the largest current bandwidth from EC. If there are multiple edges with the same bandwidth, pop the one with the smaller number first. u belongs to set V(T) and v does not belong to V(T).
[0025] S8. If ei is not an invalid value and the bandwidth of the edge corresponding to ei is greater than the bandwidth of the edge corresponding to ej, proceed to S11; otherwise, proceed to the next step.
[0026] S9. Calculate the distance between any two nodes in the spanning tree T after adding node v and edge ej, and find its maximum value, let's say dj.
[0027] S10. If dj≤di, it means that the edge can be selected to expand T, then ei=ej and di=dj; execute S6.
[0028] S11. If ei is an invalid value, it means that no edge that can be expanded has been found. Execute S12. Otherwise, add the edge and endpoint corresponding to ei to T to realize the expansion, and then execute S4.
[0029] S12. Construction of the current spanning tree T is complete; if T does not contain all nodes in G, it means the spanning tree construction failed, and proceed to S17; otherwise, proceed to the next step.
[0030] S13. Calculate the minimum available bandwidth value for all non-zero bandwidth edges in G, assuming it to be b.
[0031] S14. Use the point in T that is closest to all other nodes as the root node; if more than one node meets the requirement, select the point with the smaller number as the root.
[0032] S15. If TL does not contain a spanning tree with the same spanning tree as T, then add T to TL and add b to RL; if TL already contains a spanning tree with the same spanning tree, then add b to the corresponding tree in RL.
[0033] S16. Reduce the bandwidth of all edges in G used by T by b; execute S2.
[0034] S17, End, return to TL and RL.
[0035] Furthermore, step 2 specifically involves:
[0036] Input: Network topology G, maximum height constraint H of spanning trees, lower limit constraint r of communication rate of a single tree, and the percentage reduction p (0 ≤ h ≤ r) of the total rate of all trees. <p≤1)。
[0037] Output: A list of constructed spanning trees TL, where the root node of each tree is given; and the corresponding rate assignment RL.
[0038] S1. Using the algorithm in step 1, find a set of spanning trees and their bandwidth allocations for G under the constraints of H and r, assuming they are TL0 and RL0.
[0039] S2. Calculate the sum of the rates of all trees in RL0, denoted as tr0.
[0040] S3. Using a binary search approach, find the minimum spanning tree height constraint h, where h≤H, such that under the h constraint, the spanning trees TL and RL obtained by the algorithm in step 1 satisfy that the sum of the rates of RL is not less than tr0*p.
[0041] S4, returns TL and RL.
[0042] Furthermore, step 3 employs mixed-integer linear programming to simplify task implementation and management, specifically as follows:
[0043] Suppose there are n trees in total in TL.
[0044] The 0-1 constant a[i][j] indicates whether the i-th tree uses the j-th edge in G, the constant b[j] indicates the bandwidth of the j-th edge, and the constant Z indicates a sufficiently large positive number.
[0045] Let variable x[i] represent the final rate of the i-th tree, and 0-1 variable y[i] represent whether the i-th tree is selected.
[0046] The model that maximizes the overall speed by selecting no more than K trees from a time chain and ensuring that the speed of each tree is no less than r is described as follows:
[0047]
[0048] The constraints are:
[0049]
[0050]
[0051]
[0052]
[0053]
[0054]
[0055] After solving the problem using a linear programming solver, the trees with y[i] equal to 1 are selected, and the global reduction task is divided proportionally according to the value of x[i] of each selected tree.
[0056] The beneficial technical effects of this invention are as follows:
[0057] This invention constructs a more balanced spanning tree, obtains a multi-spanning tree with controllable maximum latency and minimum bandwidth, higher bandwidth utilization, and concurrent global reduction, effectively improving the utilization efficiency of network resources and accelerating the execution efficiency of global reduction tasks. Attached Figure Description
[0058] Figure 1 This is a schematic diagram of a global reduction model aggregation based on a tree structure.
[0059] Figure 2 This is a schematic diagram of the global reduction model distribution based on a tree structure.
[0060] Figure 3 This is a flowchart of the global reduction method based on multiple spanning trees according to the present invention. Detailed Implementation
[0061] The present invention will be further described in detail below through specific implementation methods.
[0062] For practical purposes, this invention allows distributed applications to specify, based on their requirements, the maximum number K of trees to be generated, the maximum height of each tree not exceeding H, and the minimum allowed bandwidth value r for each tree if selected. In its design, the network between nodes is modeled as an undirected graph, where each edge has latency and bandwidth attributes. The latency of a path is the sum of the latencies of all edges. During tree construction, the latency parameter between nodes is considered as the distance between nodes; the maximum height of the constructed spanning tree with a specified root node cannot exceed the application-required value H.
[0063] The flowchart of a global reduction method based on multiple spanning trees according to the present invention is as follows: Figure 3 As shown, specifically:
[0064] Step 1: Based on the remaining bandwidth in the current network, construct a spanning tree with a height no greater than H and an available bandwidth no less than r. If a spanning tree is successfully found, record its structure, allocate a certain bandwidth b to the tree, and update the remaining bandwidth of each edge in the graph. Then continue to try to find the next spanning tree, repeating this process until no new spanning tree that meets the requirements can be found, at which point the process ends.
[0065] Each time a spanning tree is found, starting from a randomly selected node, a suitable edge is continuously selected to expand the tree until all nodes are included. Each selected edge must satisfy the following condition:
[0066] (1) One end of the edge is connected to a node that has been included in the spanning tree, and the other end is connected to a node that has not yet been included, and the available remaining bandwidth is not less than r.
[0067] (2) After adding this edge, the maximum distance between any nodes in the spanning tree does not exceed 2H.
[0068] If multiple edges satisfy conditions (1) and (2), the edge with the largest remaining bandwidth is selected first. If multiple edges with the same bandwidth satisfy the conditions, one edge can be randomly selected. If no edge that meets the requirements can be found to expand the tree before all nodes are included, it indicates that the tree construction has failed and the algorithm ends.
[0069] The process of constructing a spanning tree is as follows:
[0070] Input: Network topology G, maximum height constraint H of spanning tree, lower limit constraint r of communication rate.
[0071] Output: A list of constructed spanning trees TL, where the root node of each tree is given; and the corresponding rate assignment RL.
[0072] Variable description: G is an undirected weighted graph. V(G) and E(G) represent the set of all nodes and the set of edges contained in the graph, respectively. Each edge has two attributes: delay and bandwidth, both of which are quantized values.
[0073] S1. Use list TL to record all constructed spanning trees, initialized to empty; use list RL to record the speed of all constructed spanning trees, initialized to empty.
[0074] S2. Use T to record the current spanning tree being constructed, initialized to empty; use V(T) and E(T) to represent the set of nodes and edges contained therein, respectively.
[0075] S3. Randomly select one node from V(G) and add it to tree T to begin the construction of the tree.
[0076] S4. Find all edges in E(G) with bandwidth greater than r, whose two endpoints belong to V(T) and do not belong to V(T) respectively, and form a list of edges EC.
[0077] S5. Use ei to record which edge will be used to expand the spanning tree this time, and initialize it to an invalid value; use di to record the maximum distance between any two nodes in the tree after adding edge ei and the corresponding node to T, and initialize it to 2H.
[0078] S6. If EC is empty, it means that the edge expansion is over and proceed to S11; otherwise, proceed to the next step.
[0079] S7. Pop the edge ej = (u, v) with the largest current bandwidth from EC. If there are multiple edges with the same bandwidth, pop the one with the smaller number first. u belongs to set V(T) and v does not belong to V(T).
[0080] S8. If ei is not an invalid value and the bandwidth of the edge corresponding to ei is greater than the bandwidth of the edge corresponding to ej, proceed to S11; otherwise, proceed to the next step.
[0081] S9. Calculate the distance between any two nodes in the spanning tree T after adding node v and edge ej, and find its maximum value, let's say dj.
[0082] S10. If dj≤di, it means that the edge can be selected to expand T, then ei=ej and di=dj; execute S6.
[0083] S11. If ei is an invalid value, it means that no edge that can be expanded has been found. Execute S12. Otherwise, add the edge and endpoint corresponding to ei to T to realize the expansion, and then execute S4.
[0084] S12. Construction of the current spanning tree T is complete; if T does not contain all nodes in G, it means the spanning tree construction failed, and proceed to S17; otherwise, proceed to the next step.
[0085] S13. Calculate the minimum available bandwidth value for all non-zero bandwidth edges in G, assuming it to be b.
[0086] S14. Use the point in T that is closest to all other nodes as the root node; if more than one node meets the requirement, select the point with the smaller number as the root.
[0087] S15. If TL does not contain a spanning tree with the same spanning tree as T, then add T to TL and add b to RL; if TL already contains a spanning tree with the same spanning tree, then add b to the corresponding tree in RL.
[0088] S16. Reduce the bandwidth of all edges in G used by T by b; execute S2.
[0089] S17, End, return to TL and RL.
[0090] Step 2: By repeatedly calling the algorithm in Step 1, and under the premise that the reduction in overall throughput is within an allowable range, the actual maximum height of the spanning tree is further reduced by binary search.
[0091] Input: Network topology G, maximum height constraint H of spanning trees, lower limit constraint r of communication rate, and the percentage reduction p (0 ≤ h ≤ r) of the total rate of all trees. <p≤1)。
[0092] Output: A list of constructed spanning trees TL, where the root node of each tree is given; and the corresponding rate assignment RL.
[0093] S1. Using the algorithm in step 1, find a set of spanning trees and their bandwidth allocations for G under the constraints of H and r, assuming they are TL0 and RL0.
[0094] S2. Calculate the sum of the rates of all trees in RL0, denoted as tr0.
[0095] S3. Using a binary search approach, find the minimum spanning tree height constraint h, where h≤H, such that under the h constraint, the spanning trees TL and RL obtained by the algorithm in step 1 satisfy that the sum of the rates of RL is not less than tr0*p.
[0096] S4, returns TL and RL.
[0097] Step 3: After obtaining the list of spanning trees TL and the corresponding rate allocation (also called bandwidth allocation) RL, select no more than K trees from them and determine the rate allocation of each tree. In this way, the global reduction task is allocated proportionally, thereby realizing concurrent global reduction.
[0098] The first approach is to sort the spanning trees in TL in non-increasing order according to their rate values in RL, and then select the first K spanning trees with rates greater than r (if the number of selectable trees is less than K, then select as many as possible). The global reduction task is then distributed proportionally according to the rate value obtained by each tree, thereby achieving concurrent global reduction (in this approach, in order to make full use of bandwidth, the rate of each tree must not be limited according to the rate in RL).
[0099] The present invention further proposes a method based on an optimization model to find and use no more than K trees from the TL for concurrent global reduction, so as to simplify the implementation and management of tasks and optimize the allocation of tasks among the selected trees.
[0100] Specifically, a mixed-integer linear programming approach is adopted.
[0101] Suppose there are n trees in total in TL.
[0102] The 0-1 constant a[i][j] indicates whether the i-th tree uses the j-th edge in G (1 indicates yes, 0 indicates no), the constant b[j] indicates the bandwidth of the j-th edge, and the constant Z indicates a sufficiently large positive number.
[0103] Use variable x[i] to represent the final allocation rate of the i-th tree, and 0-1 variable y[i] to represent whether the i-th tree is selected (1 means yes, 0 means no).
[0104] The model that maximizes the overall speed by selecting no more than K trees from a time chain and ensuring that the speed of each tree is no less than r is described as follows:
[0105]
[0106] The constraints are:
[0107]
[0108]
[0109]
[0110]
[0111]
[0112]
[0113] After solving the problem using a linear programming solver, the trees with y[i] equal to 1 are selected, and the global reduction task is divided proportionally among the selected trees according to their x[i] values. Simultaneously, data transmission can be rate-limited based on the calculated rate value x[i] of each tree to avoid network congestion.
[0114] This invention addresses the global reduction communication requirements between worker nodes in heterogeneous networks with arbitrary topologies by designing a global reduction method based on multiple spanning trees. The controller, possessing global system information, executes this method to obtain a parallel execution scheme for global reduction and distributes it to each worker node in the system for implementation. By constructing multiple spanning trees, the heterogeneous network bandwidth between nodes is fully utilized, and global reduction operations are performed concurrently, thereby improving efficiency.
[0115] Global reduction is a common communication requirement in many distributed applications, including distributed machine learning. Therefore, this invention can be used to optimize the communication performance of distributed machine learning systems and can also be applied to other distributed systems.
Claims
1. A global reduction method based on multiple spanning trees, characterized in that, Specifically: Step 1: Based on the remaining bandwidth in the current network, construct a spanning tree with a height no greater than H and an available bandwidth no less than r. If a spanning tree is successfully found, record its structure, allocate a certain bandwidth b to the tree, and update the remaining bandwidth of each edge in the graph. Then continue to try to find the next spanning tree, repeating the process until no new spanning tree that meets the requirements can be found. Each time a spanning tree is found, starting from a randomly selected node, a suitable edge is continuously selected to expand the tree until all nodes are included. Each selected edge must satisfy the following condition: (1) One end of the edge is connected to a node that has been included in the spanning tree, and the other end is connected to a node that has not yet been included, and the available remaining bandwidth is not less than r; (2) After adding this edge, the maximum distance between any nodes in the spanning tree does not exceed 2H; If multiple edges satisfy conditions (1) and (2), the edge with the largest remaining bandwidth is selected first. If multiple edges with the same bandwidth satisfy the conditions, one edge can be selected randomly. If no edge that meets the requirements can be found to expand the tree before all nodes are included, it indicates that the construction of the tree has failed and the algorithm ends. Step 2: By repeatedly calling the algorithm in Step 1, and under the premise that the reduction in overall throughput is within an acceptable range, further reduce the actual maximum height of the spanning tree by using binary search. Specifically: Input: Network topology G, maximum height constraint H of spanning trees, lower limit constraint r of communication rate of a single tree, percentage reduction p of the total rate of all trees, 0 <p 1; Output: A list of constructed spanning trees TL, where the root node of each tree is given; and the corresponding rate assignment RL. S1. Using the algorithm in step 1, find a set of spanning trees and their bandwidth allocation for G under the constraints of H and r, assuming they are TL0 and RL0; S2. Calculate the sum of the rates of all trees in RL0, assuming it to be tr0; S3. Use a binary search approach to find the minimum spanning tree height constraint h, where h H, such that under the constraint of h, the spanning trees TL and RL obtained by the algorithm in step 1 satisfy that the sum of the rates of RL is not less than tr0*p; S4, return TL and RL; Step 3: After obtaining the list of spanning trees TL and the corresponding rate allocation RL, select no more than K trees from them and determine the rate allocation of each tree. In this way, the global reduction task is allocated proportionally to achieve concurrent global reduction. A mixed-integer linear programming approach is used to simplify the implementation and management of the task, specifically: Assume there are n trees in total in the TL; The 0-1 constant a[i][j] indicates whether the i-th tree uses the j-th edge in G, the constant b[j] indicates the bandwidth of the j-th edge, and the constant Z indicates a sufficiently large positive number; Use variable x[i] to represent the final allocation rate of the i-th tree, and 0-1 variable y[i] to represent whether the i-th tree is selected; The model that maximizes the overall speed by selecting no more than K trees from a time chain and ensuring that the speed of each tree is no less than r is described as follows: ； The constraints are: ； ； ； ； ； ； After solving the problem using a linear programming solver, the trees with y[i] equal to 1 are selected, and the global reduction task is divided proportionally according to the value of x[i] of each selected tree.
2. The global reduction method based on multiple spanning trees according to claim 1, characterized in that, The process of constructing the tree in step 1 is as follows: Input: Network topology G, maximum height constraint H of spanning tree, lower limit constraint r of communication rate; Output: A list of constructed spanning trees TL, where the root node of each tree is given; and the corresponding rate assignment RL; Variable description: G is an undirected weighted graph, V(G) and E(G) represent the set of all nodes and the set of edges contained in the graph, respectively. Each edge has two attributes: delay and bandwidth, both of which are quantized values. The delay attribute is used to calculate the distance between nodes and the height of the spanning tree. S1. Use list TL to record all constructed spanning trees, initialized to empty; use list RL to record the speed of all constructed spanning trees, initialized to empty. S2. Use T to record the current spanning tree being constructed, initialized to empty; use V(T) and E(T) to represent the set of nodes and edges contained therein, respectively; S3. Randomly select one node from V(G) and add it to tree T to begin the construction of the tree; S4. Find all edges in E(G) with bandwidth greater than r, whose two endpoints belong to V(T) and do not belong to V(T) respectively, and form a list of edges EC. S5. Use ei to record which edge will be used to expand the spanning tree this time, and initialize it to an invalid value; use di to record the maximum distance between any two nodes in the tree after adding edge ei and the corresponding node to T, and initialize it to 2H. S6. If EC is empty, it means that the edge expansion is over and proceed to S11; otherwise, proceed to the next step. S7. Pop the edge with the largest current bandwidth ej=(u,v) from EC. If there are multiple edges with the same bandwidth, pop the one with the smaller number first. u belongs to set V(T) and v does not belong to V(T). S8. If ei is not an invalid value, and the bandwidth of the edge corresponding to ei is greater than the bandwidth of the edge corresponding to ej, proceed to S11; otherwise, proceed to the next step. S9. Calculate the distance between any two nodes in the spanning tree T after adding node v and edge ej to the spanning tree T, and calculate its maximum value, denoted as dj. S10, if dj If di indicates that this edge can be selected to expand T, then ei=ej and di=dj; execute S6; S11. If ei is an invalid value, it means that no edge that can be expanded has been found. Execute S12. Otherwise, add the edge and endpoint corresponding to ei to T to realize the expansion, and then execute S4. S12. Construction of the current spanning tree T is complete; if T does not contain all nodes in G, it means the spanning tree construction failed, and proceed to S17; otherwise, proceed to the next step. S13. Calculate the minimum available bandwidth value for all non-zero bandwidth edges in G, assuming it to be b; S14. Use the point in T that is closest to all other nodes as the root node; if more than one node meets the requirement, select the point with the smaller number as the root. S15. If TL does not contain a spanning tree with the same spanning tree as T, then add T to TL and add b to RL; if TL already contains a spanning tree with the same spanning tree, then add b to the corresponding tree in RL. S16. Reduce the bandwidth of all edges in G used by T by b; execute S2; S17, End, return to TL and RL.